If you’re an early adopter like us and have been on the CoreOS bandwagon for more than a few days, you’ve likely run into some Btrfs issues. These issues usually manifest as “no space left on device” errors that cause Docker to completely fail, usually at the most inconvenient time.
While there are some workarounds that range from adding USB sticks to wiping /var/lib/docker, a filesystem that requires manual rebalancing (apparently a kernel update fixes this) is an operational nightmare, especially when CoreOS goes to such great lengths to make things better on the operations side.
After running into the “no space left on device” issue for the second time in two weeks, it was time to move to Ext4. Unfortunately, the current set of AWS machine images are still using Btrfs for their root volumes, which means we’ll have to build our own AMI.
Log into the AWS console and click the EC2 link. Click Volumes in the navigation panel followed by Create Volume and proceed to create a volume at least 5GB in size. Take note of the Availability Zone where the volume is being created.
Navigate to Instances and click Launch Instance. Launch any Linux AMI (any will do, but I’m using the Amazon Linux HVM AMI) into the same availability zone in which your volume was created. If you choose to launch a hardware assisted virtualization image, you’ll need to first setup a VPC (click Services, VPC, followed by VPC wizard, and follow the instructions to create a VPC with a single public subnet).
After your instance launches, navigate back to EC2, click Volumes and find the volume that was created in Step 1. Select the volume and then click the Actions drop-down and choose Attach Volume. Use the instance field to choose the instance you launched in Step 2. If you launched an HVM AMI, enter /dev/xvdf into the device field. For a PVM AMI, enter /dev/sdf. Click Attach.
Navigate back to the EC2 panel, select the instance you launched in Step 2 and SSH into the instance. Ensure the volume attached to your instance by entering the command
dmesg | grep xvdf or
dmesg | grep sdf depending on which virtualization method you chose (HVM or PVM). You should see at least one log entry when the volume was attached. If you don’t see any logs, make sure you’ve logged into the correct instance and that you attached the volume to the right instance.
Build an Ext4 filesystem on the device:
mkfs.ext4 /dev/xvdf (or
coreos-install script from GitHub:
curl -LO https://raw.githubusercontent.com/coreos/init/master/bin/coreos-install and make it executable (i.e.
chmod +x coreos-install)
Install CoreOS to the device, setting the OEM option to
./coreos-install -d /dev/xvdf -o ami (replace
/dev/xvdf as needed)
After this completes, navigate back to Volumes in the AWS console, select the volume, click the Actions drop-down followed by Detach Volume.
With the volume still selected, click the Actions drop-down, followed by Create Snapshot. Enter a name and description for the snapshot (i.e. CoreOS Beta). Click Create, and wait for the snapshot to complete.
Click the Snapshots link in the navigation menu and select the snapshot that was just created. Next, click the Actions drop-down followed by Create Image. Provide a name (i.e. CoreOS Beta Channel), description, set architecture to x86_64 and Virtualization type to Hardware-assisted, and click Create.
Click the Instances link, select the instance used to install CoreOS onto the device and terminate the instance, as it’s no longer needed.
To confirm Docker is running OverlayFS as its storage driver, run
docker info and you should see
overlay listed in the output as the current storage driver.
The image that was created in Step 8 should now be available as a private AMI. Launch the AMI with a standard CoreOS cloud-config and enjoy a Btrfs-free installation.