One of the significant use-cases for Go is writing server applications that need to be running all the time. That means they need to be logging errors and restart automatically if they crash or the entire server restarts.
There are several modern ways to do this (supervisord, Upstart, daemonize, etc) and we don’t use any of them. Instead, we use the over a decade old daemontools (not to be confused with the virtual CD/DVD emulator of the same name).
Daemontools is no longer maintained and in the Public Domain as of 2007. We still use this because it’s fast, predictable, and written in C — which is exactly what we want in a process monitor. It enforces a rule that any application to be daemonized should, itself, run in the foreground by design. This prevents unexpected behavior from applications that implement their own daemonization incorrectly while making it very easy to daemonize virtually any application without handling forking, PIDs, etc. yourself.
Daemontools does not follow the common UNIX-like package installation methods that most expect. This will create a /service directory where the run and log scripts will reside.
> mkdir dt_inst
> cd dt_inst
> curl -O http://cr.yp.to/daemontools/daemontools-0.76.tar.gz
> tar zxvf daemontools-0.76.tar.gz
> cd admin/daemontools-0.76
> curl -O http://www.qmail.org/moni.csi.hu/pub/glibc-2.3.1/daemontools-0.76.errno.patch
> patch -p1 < daemontools-0.76.errno.patch
> csh -cf '/command/svscanboot &'
If you didn’t receive any errors, at this point you should have two processes running and svscanboot added to /etc/rc.local. For other ways of starting daemontools, look here.
root 4444 0.0 0.0 4444 856 ? S 01:30 0:00 /bin/sh /command/svscanboot
root 4446 0.0 0.0 4376 1184 ? S 01:30 0:00 svscan /service
Now we’ll add our Go app as a service:
> mkdir -p /service/go_app/log
> mkdir /service/go_app/env
The env directory can be used to create environment variables that will be made available to the Go application via envdir in the run script.
Note: Services begin running as soon as their run scripts exist and are executable.
Create the app run script /service/go_app/run:
# Redirect STDERR to STDOUT
# Use envdir to set any needed environment variables for our app
exec envdir /path/to/our/app/envdir setuidgid goapp_user /path/to/our/go_app
Create the log run script /service/go_app/log/run:
# Redirect STDERR to STDOUT
exec setuidgid goapp_user multilog t ./main
This will create log files in /service/go_app/log/main with the most recent logs being appended to /service/go_app/log/main/current. See the multilog man page for more options.
Make both run scripts executable:
chmod +x /service/go_app/run /service/go_app/log/run
We can check the status of both the app and the log:
> svstat /service/go_app
/service/go_app: up (pid 18124) 15 seconds
> svstat /service/go_app/log
/service/go_app/log: up (pid 18126) 12 seconds
Use the svc command to control the process:
# Take the process down
> svc -d /service/go_app
# Bring the process up
> svc -u /service/go_app
# Kill (restart) the process
> svc -k /service/go_app
# Send the process the TERM signal
> svc -t /service/go_app
See the svc man page for more options
You Go app will be automatically started at boot time, restarted if it crashes, and have its logs rotated without intervention.
Sign up for Turret.IO – the only email marketing platform specifically for developers
You might as well use runit, which has seen some updates.
Runit would be a closer contender for us, but it doesn’t provide anything we feel we’re missing from daemon tools to warrant a switch.
Thanks for the feedback!
I wonder why nobody in the Linux community seems to like launchd? (See http://launchd.info/ and http://en.wikipedia.org/wiki/Launchd). I think the licensing issues should be resolved with the Apache license, right? Its a big mess of different solutions on different Linux distros (systemd, sysvinit, openrc, Upstart etc. See: http://en.wikipedia.org/wiki/Comparison_of_Linux_distributions#Technical)…
My guess would be that since it did have some previous licensing issues (like you pointed out), the fact that it specifically targeted Darwin, and its use of verbose XML configurations all make it more burdensome to adopt. Although it may be worth watching the non-Darwin port on GitHub: https://github.com/rtyler/openlaunchd
Thanks for the feedback!
One important use case for a Golang server is doing HTTP; but in your run script, you doesn’t start the Golang program as root, but as a normal user (setuidgid goapp_user). This is certainly more secure, but it can’t open port 80 anymore. What ddo you do in this cases? Do you run the Golang program as root or do you use some of the various workarrounds to open ports<1024 as normal user, and if, which one?
I haven’t personally tested this, but for cases where you need to open privileged ports, you might be able to use tcpserver: http://thedjbway.b0llix.net/daemontools/uidgid.html
Thank you for your response! I’ve investigated tcpserver; but for my taste, this approach has too much overhead (UCSPI protocol). I decided to use Capabilities instead – just allow the program what i want.
But the Daemontools are really nice to work with (in the KISS sense), so thank you for the tip 🙂