Posted by & filed under aws, CoreOS, DevOps.

If you’re an early adopter like us and have been on the CoreOS bandwagon for more than a few days, you’ve likely run into some Btrfs issues. These issues usually manifest as “no space left on device” errors that cause Docker to completely fail, usually at the most inconvenient time.

While there are some workarounds that range from adding USB sticks to wiping /var/lib/docker, a filesystem that requires manual rebalancing (apparently a kernel update fixes this) is an operational nightmare, especially when CoreOS goes to such great lengths to make things better on the operations side.

After running into the “no space left on device” issue for the second time in two weeks, it was time to move to Ext4. Unfortunately, the current set of AWS machine images are still using Btrfs for their root volumes, which means we’ll have to build our own AMI.

Step 1

Log into the AWS console and click the EC2 link. Click Volumes in the navigation panel followed by Create Volume and proceed to create a volume at least 5GB in size. Take note of the Availability Zone where the volume is being created.

 

Screen Shot 2015-02-28 at 9.51.21 PM

Step 2

Navigate to Instances and click Launch Instance. Launch any Linux AMI (any will do, but I’m using the Amazon Linux HVM AMI) into the same availability zone in which your volume was created. If you choose to launch a hardware assisted virtualization image, you’ll need to first setup a VPC (click Services, VPC, followed by VPC wizard, and follow the instructions to create a VPC with a single public subnet).

 

Screen Shot 2015-02-28 at 9.57.11 PM

Step 3

After your instance launches, navigate back to EC2, click Volumes and find the volume that was created in Step 1. Select the volume and then click the Actions drop-down and choose Attach Volume. Use the instance field to choose the instance you launched in Step 2. If you launched an HVM AMI, enter /dev/xvdf into the device field. For a PVM AMI, enter /dev/sdf. Click Attach.

 

Screen Shot 2015-02-28 at 10.02.33 PM

Step 4

Navigate back to the EC2 panel, select the instance you launched in Step 2 and SSH into the instance. Ensure the volume attached to your instance by entering the command dmesg | grep xvdf or dmesg | grep sdf depending on which virtualization method you chose (HVM or PVM). You should see at least one log entry when the volume was attached. If you don’t see any logs, make sure you’ve logged into the correct instance and that you attached the volume to the right instance.

Build an Ext4 filesystem on the device: mkfs.ext4 /dev/xvdf (or /dev/sdf)

Step 5

Download the coreos-install script from GitHub:
curl -LO https://raw.githubusercontent.com/coreos/init/master/bin/coreos-install and make it executable (i.e. chmod +x coreos-install)

Step 6

Install CoreOS to the device, setting the OEM option to ami:

./coreos-install -d /dev/xvdf -o ami (replace /dev/xvdf as needed)

After this completes, navigate back to Volumes in the AWS console, select the volume, click the Actions drop-down followed by Detach Volume.

Step 7

With the volume still selected, click the Actions drop-down, followed by Create Snapshot. Enter a name and description for the snapshot (i.e. CoreOS Beta). Click Create, and wait for the snapshot to complete.

 

Screen Shot 2015-02-28 at 10.14.32 PM

Step 8

Click the Snapshots link in the navigation menu and select the snapshot that was just created. Next, click the Actions drop-down followed by Create Image. Provide a name (i.e. CoreOS Beta Channel), description, set architecture to x86_64 and Virtualization type to Hardware-assisted, and click Create.

Screen Shot 2015-02-28 at 10.16.42 PM

Step 9

Click the Instances link, select the instance used to install CoreOS onto the device and terminate the instance, as it’s no longer needed.

To confirm Docker is running OverlayFS as its storage driver, run docker info and you should see overlay listed in the output as the current storage driver.

The image that was created in Step 8 should now be available as a private AMI. Launch the AMI with a standard CoreOS cloud-config and enjoy a Btrfs-free installation.

— the behavior-driven marketing platform for developers.

2 Responses to “Use Ext4 and OverlayFS instead of Btrfs on CoreOS”

  1. puf

    its not btrfs which is a problem its coreos
    btrfs works fine with a recent kernel and proper tools.

    Reply
    • tim

      Thanks for your feedback.

      While I can agree that btrfs is moving forward in a positive direction, it’s simply too immature to run in production for most. Having to manually rebalance a filesystem on a regular basis or continuously upgrading it as bugs are fixed isn’t something most people will find acceptable. I look forward to btrfs becoming more mature and being as reliable as Ext3 and Ext4, but it’s got a lot of work ahead of it.

      -Tim

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *