In a previous post we discussed how we manage some of our Turret.IO deployments using Jordan Sissel‘s excellent packaging software: FPM. This post is mainly a mini-tutorial on one of many ways to leverage FPM as part of your deployment process.
Like many others, our stack consists of both software built in-house and 3rd party applications such as NGiNX, Redis, Cassandra, etc. In our previous deployment process, provisioning new servers had two major parts: bootstrapping required sofware (i.e. Redis) and then installing our own (WSGI web application, Go binaries, etc.). We needed a way to streamline this and make setup and installation consistent during deployments. By using Chef as part of our bootstrapping process, we could easily download the required RPMs from Amazon S3 and then install each one on the appropriate servers in whatever order was required.
Using this method lets us compile software from source (on identical production servers) with any modifications we need and then package it, significantly reducing our provisioning times (no more unnecessary compiling!) and keeping things organized.
Setup:
Install prerequisites, then install FPM
> [sudo] yum install ruby-devel gcc rubygems > [sudo] gem install fpm
Example 1: Packaging Redis
Download the Redis source
> curl -O -L http://download.redis.io/releases/redis-2.8.13.tar.gz
Extract, build, and test Redis
> tar zxvf redis-2.8.13.tar.gz > cd redis-2.8.13 > make > make test
Install to temporary directory. Some Makefiles (Redis’ does not) support the DESTDIR environment variable that should be used to provide a completely alternate installation directory. Please use that instead of PREFIX if the Makefile supports it.
> mkdir tmp_install > make PREFIX=/path/to/tmp_install install
Create a post_install script to handle any setup work that’s required after the RPM is installed. Our file is named rpm_post_install.sh and it creates the required daemontools run and log files. Be sure to replace [REDIS USER] with the user that will be running the redis binary and have permission to write log files. (Optional)
svc=redis mkdir -p /service/$svc/log echo -e \#\!/bin/sh\\nexec 2\>\&1\\nexec setuidgid [REDIS USER] /usr/local/bin/redis-server > /service/$svc/run echo -e \#\!/bin/sh\\nexec 2\>\&1\\nexec setuidgid [REDIS USER] multilog t ./main > /service/$svc/log/run chmod +x /service/$svc/run chmod +x /service/$svc/log/run
Package with FPM
> fpm -s dir -t rpm -n redis-custom -v 1.0 -C tmp_install -p redis-custom-1.0-ARCH-mycompany.rpm --prefix=/usr/local --after-install rpm_post_install.sh .
You can see all of the available options here: https://github.com/jordansissel/fpm/wiki
FPM:
-s: Sets the ‘source’ to ‘directory’
-t: Sets the ‘target’ to ‘rpm’
-n: Sets the RPM ‘name’ to ‘redis-custom’
-v: Sets the RPM ‘version’ to ‘1.0’
-C: Changes to the specified directory before looking for files to package
-p: Sets the package filename
–prefix: Installs files with the specified prefix
–after-install: Packages and runs the specified script after the RPM is installed
. : List of files to package (if -C is used, the directory is changed first)
If no errors display, you should have the RPM file in your current directory. Install the RPM file as usual:
> [sudo] rpm -Uvh redis-custom-1.0-x86_64-mycompany.com
Replace x86_64 with your architecture
You can make changes and upgrade the package by increasing the version number when running fpm. If you made a mistake and need to reinstall an updated package with the same name, you can do the following:
> rpm -Uvh redis-custom-1.0-x86_64-mycompany.com --replacepkgs --replacefiles
Example 2: Packaging Go binaries
(this assumes you have a working Go installation and have GOROOT and GOPATH setup accordingly)
Our example creates a very basic HTTP server written in Go that responds with some basic text.
Create a project directory:
> mkdir go_serve
Make the directory to hold our source:
> cd go_serve > mkdir -p src/code.mycompany.com/go_serve
Create the following file in src/code.mycompany.com/go_serve/http.go:
package main
import (
"net/http"
)
func MainHandler(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello World!"))
}
func main() {
http.HandleFunc("/", MainHandler)
http.ListenAndServe(":8080", nil)
}
Build it — make sure GOPATH and GOROOT are set accordingly and the ‘go’ executable is on your PATH
> go install code.mycompany.com/go_serve
If the build process completes without any errors, you should have a new directory ‘bin’ with your ‘go_serve’ binary.
Create a post_install script to handle any setup work that’s required after the RPM is installed. Our file is named rpm_post_install.sh and it creates the required daemontools run and log files. Be sure to replace [HTTP USER] with the user that will be running the HTTP server and has permission to write log files. (Optional)
svc=go_serve mkdir -p /service/$svc/log echo -e \#\!/bin/sh\\nexec 2\>\&1\\nexec setuidgid [HTTP USER] /usr/local/bin/go_serve > /service/$svc/run echo -e \#\!/bin/sh\\nexec 2\>\&1\\nexec setuidgid [HTTP USER] multilog t ./main > /service/$svc/log/run chmod +x /service/$svc/run chmod +x /service/$svc/log/run
Package with FPM
> fpm -s dir -t rpm -n go_serve -v 1.0 -C bin/ -p go_serve-1.0-ARCH.mycompany.rpm --prefix=/usr/local/bin --after-install rpm_post_install.sh .
(Note that –prefix includes the full /usr/local/bin path because we’re plucking the binaries right out of the bin/ directory. If we were using a Makefile with DESTDIR or PREFIX, those paths would’ve already been created for us.)
Install the RPM
> rpm -Uvh go_serve-1.0-x86_64.mycompany.rpm
Placing these RPMs in a secure and distributed environment such as S3 provides a quick and easy alternative to hosting a private RPM repository. This process works equally well for packaging web application source code.
While this post mainly focuses on directory sources and RPM targets, FPM supports much, much more (Python modules, .deb files, etc.). Check out https://github.com/jordansissel/fpm for details.
Sign up for Turret.IO – the only email marketing platform specifically for developers
Leave a Reply