After using Jenkins for some time, the natural progression towards cheaper and simpler alternatives kicked in. While not the prettiest thing to look at, Jenkins served us well, but the costs involved with running at least one full-time AWS instance (plus workers) for our CI needs were becoming questionable.
We operate both a private Git repository and Docker registry for Cronally — something Jenkins has very little problem handling since there are extensive configuration options. This is also one of the first hurdles we had to cross while reviewing our hosted CI options. Most CI platforms don’t support custom Git repositories, and some don’t even support non-standard Docker registries. This is unfortunate, since it becomes difficult to evaluate these platforms without paying for private GitHub repositories or a DockerHub account (though this isn’t a problem with non-private repositories).
I decided to take the plunge and migrate one of our projects out of our custom Git repository and into a paid, private GitHub repository which all of the platforms we’ll be evaluating support.
CircleCI
CircleCI was one of the first platforms I reviewed. At $50/mo it’s first paid plan is a bit steep, but fortunately you can run an unlimited number of serialized builds on the free plan.
The first thing you notice about CircleCI is that it looks good. Aesthetically speaking, it’s one of the most pleasing CI platforms I’ve looked at. The layout works well and it’s very responsive.
As with many other CI platforms, CircleCI sports a YAML configuration, so you can easily version your config and an option for encrypted environment variables lets you reference confidential data while keeping it out of source control. Using the YAML configuration, you can override lots of options like the code checkout process, timezone, and artifact targets. I rarely get the feeling that I’m confined to a particular workflow and can basically run my tests and deploy as I see fit.
Docker support is a little unexpected. While CircleCI does indeed have support for Docker, you’re not able to delete images or containers, so if that’s part of your workflow (like it was ours), you’ll need to adjust, or add flags to your build scripts to selectively enable/disable that functionality. Presumably, the reason for this is to prevent users from deleting other users’ images and containers, but such an assumption also leaves an uneasy feeling for the level of isolation that creates (i.e. can another user just enter my container?).
CircleCI builds start very quickly. Occasionally there’s a hiccup where more than 10 seconds pass before a build starts, but that’s rare.
Strangely enough, as of this writing CircleCI still only supports GitHub. As popular as it is, lots of developers still flock to BitBucket for free, private repositories instead of paying for them with GitHub. While there are some differences between the GitHub and BitBucket APIs that affect how automated builds and pull requests can be dispatched, it certainly would appear that CircleCI is missing an opportunity to serve a different type of customer.
SemaphoreCI
SemaphoreCI is a worthy contender in this space. Its layout and UI are significantly different than what we experienced with CircleCI, but I never felt as though I was missing anything.
A big change with SemaphoreCI is the lack of YAML configuration. SemaphoreCI provides a section in its Project Settings area to configure the setup stage and multiple threads for parallelizing tests (the free account includes 2 processors). The configuration options do feel somewhat limited. I wasn’t able to override the source code checkout directory nor could I alter machine-specific settings (i.e. running services, timezone, etc.). Encrypted environment variables are supported through the UI, allowing you to add confidential passwords, AWS keys, etc. to your builds.
Docker support is currently in beta, but so far is fantastic. There aren’t any wacky limitations on removing containers or images like we experienced with CircleCI, so our build scripts work as-is. My assumption is that SemaphoreCI is running Docker inside Docker with special privileges. If that’s the case, there’s a whole new set of security issues that could be raised, but it works the way I would expect. [In the comments below, Aleksandar confirms that they’re not using Docker inside Docker, so my assumption and security concerns have been invalidated]
Free accounts are limited to 100 builds per month, which may be enough for a solo developer or teams that aren’t producing lots of fixes and updates, but at 25 builds per week, a scrappy start-up will most likely require a paid account.
One particular problem we did experience was with Go. SemaphoreCI specifically supports the language and goes to extra efforts to provision a proper Go workspace when that platform is chosen for the build. However, since our build uses Docker, we mount our Go workspace when launching the build container and the symlink created by SemaphoreCI fails to be followed properly. I’m not sure if this is a Docker or SemaphoreCI issue, or perhaps a side-effect of how Docker is implemented, but it’s easily rectified by manually creating a new workspace by copying the source code instead of creating a symlink.
SemaphoreCI now graciously supports both GitHub and BitBucket, making up for some of its lack of configuration capabilities.
Shippable
Shippable has been making a lot of progress towards a more comprehensive platform to include the provisioning of development and testing environments. While we haven’t used that part of their platform, it’s an interesting concept.
Like many other CI platforms, Shippable supports a YAML configuration and has a host of options similar to CircleCI. It also supports encrypted environment variables and unlimited serialized builds in the free version. While we did encounter some performance issues immediately following their v2 launch, most appear to have been rectified and Shippable is a solid platform in this space.
Unfortunately, Docker support is not what we’ve come to expect without purchasing a “dedicated host” option which appears to require you to run another host elsewhere (i.e. AWS, DigitalOcean, etc.). With the standard free version, Docker support is limited to two specific workflows, neither allowing you access to the docker command itself — only taking specific actions based on a few options in the project settings.
While Docker support is less than ideal, Shippable is still a worthy contender for those who have simpler workflows or don’t use Docker at all. It’s supported both GitHub and BitBucket for some time and its YAML configuration makes it a effective compromise for some users.
Our Choice: CircleCI
SemaphoreCI seems to have nailed Docker support, but fails to provide us with a YAML configuration option. Even though we wish CircleCI‘s Docker support more closely matched that of SemaphoreCI, its configuration works well for us and it’s just a pleasure to use. We don’t anticipate running into any issues with serialized builds for a bit, but once we do, the $50/mo fee is still less expensive than our equivalent Jenkins setup would be for similar performance.
Aleksandar
Hi Tim,
Thanks for the great article. Just one thing to add/clarify. We at Semaphore take close care of the security for all of our users, with that in mind we designed our beta platform with Docker support. We don’t actually run Docker inside Docker, since we are also aware of that kind of security issues. 🙂
Could you please give us feedback on the problems you encountered with setting up Go projects, we would appreciate it very much indeed (semaphore @ renderedtext.com)?
tim
Aleksandar,
Thanks for the feedback — I’ve sent an email with the description of the problem.
Cheers!
Tim