Announcing: Golapa v1.1

Posted by & filed under Email, Go, Web development.

It’s been almost 4 months since our initial release of Golapa and we’ve been using it extensively for managing our Turret.IO signups. Today I’m proud to announce that it now supports Turret.IO as a method for capturing signup data, which also coincides with our release of the Turret.IO Go client library.

Golapa continues to provide a high-performance, customizable launch page (written in Go) for start-ups and runs on Google App Engine. Because Google App Engine offers 28 free instance hours per day, you most likely won’t see any charges until you’re experiencing a modest level of traffic and need additional front-end instances, or have more data requirements (see Google’s App Engine Quota page for details).

Happy launching!

Organizing Go code to be ‘go gettable’

Posted by & filed under Go, Web development.

This topic comes up a lot because Go is a newer language and has some quirks that many developers aren’t used to.

One Go tool that’s particular useful is ‘go get’ — which is used to fetch a remote package and add it to your current Go workspace. Naturally, a Go developer may at some point want his code to be usable by others and thus it should be ‘go gettable’. Here’s where things start to get quirky.

Most law-abiding developers keep their code in one of the major version control systems (Git, Mercurial, Bazaar, etc.) and are generally used to keeping entire projects under version control. The standard Go layout (http://golang.org/doc/code.html#Organization) shows us that we should have 3 top-level directories: src/, bin/, and pkg/. Nested inside src/ are the names of top-level repositories like github.com, launchpad.net, etc. Inside those repository directories are more directories containing the actual Go packages. So, how do we manage a project that contains a multitude of versioned software including our own?

The answer: workspaces

In Go, a workspace is a container that the Go tools use to understand where Go source files are, where to place ‘go gettable’ packages, where to build and install Go programs and libraries, and so on. The workspace itself should not be under version control. A common beginner’s mistake is to create a repository with a src/github.com/github-user/project hierarchy to make the Go tools happy, but then realize that the structure prevents the code from being ‘go gettable’ because the root level of the repository contains the src/ directory instead of Go files.

So what’s the best way to start a new Go project? Carefully follow the instructions here: http://golang.org/doc/code.html#Organization

Pointers

  • When creating a new Go project, create the version-controlled directory structure you expect to use in the future, even if you’re not going to use it yet (i.e. cd ~/go_workspace && mkdir -p src/github.com/git-user/go-project)
  • In the above pointer, go-project is the root of your repository
  • If you publish your code, a developer who runs ‘go get github.com/git-user/go-project’ will be able to use your code

 

Improve Your Code Deployments with FPM

Posted by & filed under aws, Go, Python, Web development.

At Turret.IO, our stack includes software written in both Python and Go. While it’s relatively easy and efficient to deploy Python code that handles the website and API, shipping Go binaries isn’t nearly as elegant. Our initial methods involved deploying the source, compiling on each machine (since Go does compile incredibly fast), and startup scripts handled the rest. As things became more complex and our number of daemon processes increased, this method became tedious and difficult to manage.

FPM to the rescue

Effing Package Management (https://github.com/jordansissel/fpm) is a Ruby script that makes creating files compatible with various package managers incredibly simple (.rpm, .deb., etc.). Feed FPM a source directory, post-init script, and a version tag and in a few seconds you can have an installable package. FPM also supports a handful of source types like Python modules, Ruby gems, even other packages. This all works without having to run through the burdensome process of creating an rpmbuild environment.

How we use it

Incorporating FPM into our build process allows us to compile all of our Go code one time, bundle the necessary init scripts with each binary, and finally build a manageable RPM file. Each application server gets the RPM during our deployment process for installation. This worked well enough for us that we decided to build packages for other services that we used as well, such as NGiNX. After installing to a temporary directory, pass FPM the path and you have a custom, installable NGiNX package.

Using Cassandra with Python and uWSGI

Posted by & filed under Cassandra, Python, Web development.

The latest Python DataStax Driver is a welcomed addition to the available Python clients that support CQL3 with Cassandra. One unfortunate pain point with this client (and possibly others) is the amount of time it takes to connect to a cluster. Because each connection requires the client to fetch the entire cluster’s schema, time-sensitive environments (such as web-based requests) need special care taken to ensure these connections are handled outside of the web requests themselves.

The novice approach (which we also originally took) is to initialize the connection to the cluster on each request. Even a single keyspace with one column family took upwards of 400ms to complete the initial connection — this is without any network access. Obviously having a 400ms baseline without even performing any queries is going to be a problem.

Read more »

Amazon SNS Verification for Python

Posted by & filed under aws, Email, Python.

A great part of Amazon’s Simple Notification Service (SNS) is that it signs each notification, allowing your application to verify the signature before trusting any data inside of it. Of particular interest to us is an Amazon SES option that sends, complaint, hard bounce, and soft bounce messages via SNS rather than email. This allows us to easily parse that feedback and take further action without having to painfully dissect a bounce email and determine the cause.

Instead of using a shared secret, Amazon signs the notification with their private key and provides a URL to access the X509 certificate within each notification. The overall concept is similar to verifying an HMAC signature, except with using a public key infrastructure.

Unfortunately, extracting the public key from an X509 certificate is a bit more difficult than we initially expected (at least in Python, which our notification receiver is written in). Granted, we could’ve used openssl via the subprocess module, but ideally we didn’t want to run an external program just to verify the signature.

After several attempts at using 3rd party libraries, including pyOpenSSL (which had too many issues with our version of OpenSSL), I decided to try M2Crypto. M2Crypto can very easily digest an X509 certificate and spit out the public key, but the actual verification process still eluded me. Some trial and error and searching later, I discovered Arthur Rodrigues’ Verifying X509 Signature in Python which closed the last gap and allowed us to successfully verify the signatures.

See the link above for the proper verification process.