Packaging for Deployment

Posted by & filed under Deployments, Go, Web development.

In a previous post we discussed how we manage some of our Turret.IO deployments using Jordan Sissel‘s excellent packaging software: FPM. This post is mainly a mini-tutorial on one of many ways to leverage FPM as part of your deployment process.

Like many others, our stack consists of both software built in-house and 3rd party applications such as NGiNX, Redis, Cassandra, etc. In our previous deployment process, provisioning new servers had two major parts: bootstrapping required sofware (i.e. Redis) and then installing our own (WSGI web application, Go binaries, etc.). We needed a way to streamline this and make setup and installation consistent during deployments. By using Chef as part of our bootstrapping process, we could easily download the required RPMs from Amazon S3 and then install each one on the appropriate servers in whatever order was required.

Read more »

A Faster Turret.IO with Docker

Posted by & filed under Docker, Python, Web development.

Connecting your web application’s server-side logic with a 3rd party service usually introduces some type of a performance penalty. Preventing such issues requires the creation of task queues, worker processes, or other methods that allow slower processes to do their work while maintaining a responsive interface for the user.

Turret.IO supports an API for sending user data that’s used to create email distribution lists based on an arbitrary number of data points provided by a web application. A common problem that’s faced by developers with this usage pattern is that a task queue is almost always necessary to prevent the API calls from introducing additional processing time while a visitor is interacting. Even the most carefully optimized systems may exhibit slower than normal response times given the nature of a web request (DNS, network latency, drive failures, unbalanced load, etc.) so relying on a 3rd party service to respond quickly is simply a bad idea.

Read more »

Daemonize Your Go App

Posted by & filed under Go, Web development.

One of the significant use-cases for Go is writing server applications that need to be running all the time. That means they need to be logging errors and restart automatically if they crash or the entire server restarts.

There are several modern ways to do this (supervisord, Upstart, daemonize, etc) and we don’t use any of them. Instead, we use the over a decade old daemontools (not to be confused with the virtual CD/DVD emulator of the same name).

Daemontools is no longer maintained and in the Public Domain as of 2007. We still use this because it’s fast, predictable, and written in C — which is exactly what we want in a process monitor. It enforces a rule that any application to be daemonized should, itself, run in the foreground by design. This prevents unexpected behavior from applications that implement their own daemonization incorrectly while making it very easy to daemonize virtually any application without handling forking, PIDs, etc. yourself.

Read more »

Cut your Python development time in half

Posted by & filed under Python, Web development.

It’s no secret that we’re Python advocates and believe its cutthroat indentation rules are part of what makes it a great language. We use it to serve our web interface and API and the majority if our maintenance scripts are Python as well. Beginners will find it simple to read and understand. Most commands work just the way you expect them to and when things go wrong, it’s also fairly simple to figure out why. Drop a few print statements throughout your program and see where things go awry.

Print statements can be a quick way to see how far a program’s execution gets before an exception occurs or peek inside a variable, but they require some serious trial and error, re-running your program over and over to review the output — combine this with time-consuming test cases and you can quickly hit a wall.

Read more »

Go: Build Constraints FTW!

Posted by & filed under App Engine, Go, Web development.

We’re continuing to use Google App Engine to host our sign up page (we’ll discuss why in another post) which produced a common problem during our local builds: appengine-specific code would fail when using the standard Go build tools. Because our AppEngine and non-AppEngine code was similar, we wanted to keep the code-base from being fragmented. Enter, Build Constraints.

By adding a simple comment to the top of a file, we can tell both the standard Go tools and the goapp tools which files they may ignore. In our case, we needed to provide access to the AppEngine context but wanted builds to work using both tools:

// build appengine

The above line is placed at the top of our app-engine specific go source file making both the standard Go tools and goapp tools happy.

> GOPATH=~/go-launch/ go test
PASS
ok github.com/turretIO/turret-io-go/tests 3.875s
> ~/go_appengine/goapp test
PASS
ok github.com/turretIO/turret-io-go/tests 4.439s

In addition to the appengine Build Constraint, the tools also recognize the “windows” flag that allows Windows-specific files to ignored or explicitly built.

Cheers!

 

Sign up for Turret.IO — the only email marketing platform specifically for developers