Immutable servers. Also called Phoenix servers, it's hot and hip in the world of cloud services and… actually not all that bad an idea. I used to have a server called Phoenix, so one could say I was ahead of this curve. One would be wrong, but one could say it. 

In reality, I read about this concept a while ago but kind of dismissed it back then as being complicated and difficult to work with. Since then though, I found and became rather interested in an Illumos distribution called SmartOS, in my quest to find a replacement for my aging Solaris server. And it turns out that the immutable server concept and SmartOS are perfect companions.

As I was experimenting with SmartOS datasets I noticed that they were kind of hard to upgrade. Taking a dataset, pointing its pkgsrc repository at a newer version and running pkgin full-upgrade works sometimes but is more likely to cause all sorts of mayhem and chaos instead. By contrast, what I'm used to in Debian is you change your apt sources, run apt-get dist-upgrade and it mostly just works. 

This of course is a bit of a problem: Joyent, the company behind SmartOS if you weren't following, won't maintain all versions of pkgsrc forever and at some point things start getting insecure; you want to keep up-to-date with packages for software that is internet-facing, like your MTA or web server.

So as I was figuring out how to deal with this, there were two important things I ran into: first, Joyent's Machine Image Build Environment (MIBE for short), which is a set of scripts that Joyent uses to build its own SmartOS images. Secondly, a zone configuration setting called "delegated datasets" which is a Solaris Zones feature that allows a zone to manage its own ZFS datasets and in SmartOS gives you a zfs filesystem separate from the image filesystem. 

At that point, things clicked into place and everything made sense: this is how you do immutable infrastructure. And not just that: this is how you do it elegantly.

The benefits are great: you don't edit the configuration of the machine directly so there's never drift (and the machine is never in an unknown state). With traditional configuration management (like Puppet or CFEngine), it's always tempting to make local changes that are neither reflected in the configuration management software nor reset by it, which can cause problems down the line. This approach prevents those problems because upon reprovisioning, all local modifications will with 100% certainty be lost so you know not do that. Upgrades are relatively easy because the images are built from scratch every time, so you always start with a clean slate. And you have a fully-functioning image that you can test properly before you deploy it, because you then deploy that exact image. If you've had a security problem and someone compromised your server, just reprovision and it's good as new; meanwhile you can fix the problem and build a new image that fixes the vulnerability. 

TL;DR: I am now convinced immutable server are the way to go.

It's also the way I've gone, so currently have five custom MIBE images and a fork of another:

• mi-poop-mail: an image running Exim, Dovecot, Clamav and Spamassassin
• mi-poop-webmail: an image that presents a Roundcube webmail interface
• mi-poop-unifi: an image that runs the UniFi controller software from Ubiquiti
• mi-poop-zotonic: an image that runs Zotonic (and serves up this exact website, in fact.)
• mi-poop-irc: an image that runs UnrealIRCd and powers irc.poop.nl, a GIMPnet server.
• mi-dsapid: an image that runs a datasets server that I've modified a bit to use SSL.

They are based on either the base-64 or minimal-64 images from Joyent. I'll explain briefly how this works.

The MIBE environment consists of three parts insofar as what goes into the zone image:

• A directory tree to be copied into the image 
• A list of packages to be installed ('packages' file)
• A script to be executed after the packages are installed ('customize' script)

First, the directory tree is copied into the image filesystem. This is where you can add things like directories that should exist for your software, configuration files, SMF manifests, etc. Also here is where you add scripts that are to be executed by 'zoneinit'. I'll get to that in a bit.

Second, packages are installed. They are read from the 'packages' file and installed one-by-one as if you were pkgin installing them all. Just a shortcut to doing exactly that in the customize script, really.

Last, the 'customize' script is run. This script can do anything really. In my current images, it's almost empty for mi-poop-unifi (only adding a user for it) but for mi-poop-zotonic it's where Zotonic is downloaded and compiled. 

When that is done, MIBE will pack up the image and generate a manifest which you can then use to upload it to a datasets server or install directly with imgadm. 

When you create a zone, it is first provisioned. This is where the zoneinit scripts I referred to earlier come in: they are run upon the first boot of a zone. In these scripts you can enable services, set-up the delegated dataset, etc. I use them to set the timezone, disable ssh, enable services and install the unifi software (to avoid distributing it).

Some of these things can be done from the customize script too, but since these scripts are executed in the running zone, they have access to zone-related information such as its hostname, IP info and the customer/internal metadata. There is a time-out to provisioning zones, so you want to try and do things that take a lot of time in the 'customize' script rather than in zoneinit scripts. That said, anything done before the image is packed up will incur a cost with regards to the image file size. Take that into account and do what makes the most sense. 

I have put my mibe images on github so you can take a look at how you could do immutable infrastructure with SmartOS. There are more mibe images on github but most don't take advantage of delegated datasets which -- if you ask me -- is one of SmartOS's most useful features. The links to the repos are above. They are set up to be as generic as possible with all private data on a delegated dataset, so others can use them too. I also have the prebuilt images available on my datasets server, smartos.poop.nl.