(please note this is my first blog entry post-D. As such and because I’ve yet to talk to any legal folks, I should state that this does not represent any opinion or policy of Dell)

One thing that caught my eye/ear when I could listen/watch was an excellent presentation by Sean Cribbs. Sean holds a special place in my hero worship pantheon for a few reasons:

• I first heard about Riak on one of the first episodes of the ChangeLog show (along with Andy Gross). It and the whole NoSQL thing made sense then
• Sean is a pretty fucking down to earth person. He graciously drove down to one of our local meetups. Uber friendly and an awesome advocate for Basho and Riak.
• He’s a really awesome presenter and if you’ve never had the priviledge of seeing him (live stream or in person), he rocks the mic.

So in Sean’s presentation he’s talking about some changes to the Ruby client library for Riak. Many of the changes make the Ruby library a proper smart client. Read this wiki wiki entry under Connecting to Clusters for some of the features. It’s awesome (especially the transport-related failure handling).

Client libraries in general

I want to say a bit about client libraries. Regardless of what they talk to (though I’ll be talking specifically about database client libraries), this is something many companies get wrong.

Everyone knows I’m not the biggest MongoDB/10Gen fan in the world. I won’t go into detail about the technical reasons behind that. Many others have done a much more eloquent dive into that topic. As much as it’s easy to make fun of MongoDB as being an marketing-driven database, they did get one thing right. They owned their client driver availability. Not only did they own and maintain all the drivers but they largely had the same API across the various languages.

Then again, they had to. Other databases/applications offer a REST-ish interface over HTTP (or plain-text interface like Redis) so they can punt a bit. Got a libcurl port for your language? You’re set. MongoDB has its own protocol and that god-forsaken BSON shitshow.

One of the benefits, however, of a plain-text or HTTP-based protocol is that it’s a pattern we can grok as operators and developers. We load balance webservers. We speak to third-party APIs. It’s not the most EFFICIENT but it’s a known quantity. It’s also, as I said, REALLY fucking easy to add support to your language of choice. No need to FFI some c library or make a binary extension. Any language worth its salt has http client support in stdlib (even if it’s as big a pile of dog squeeze as net/http). Again, most languages also have libcurl support for something better.

Back to smart clients

I largely dislike applications that require smart clients to get the full benefit. As an operations person, I’m USUALLY using a dynamic language like python, ruby or perl to access the system as opposed to directly from the application. This was my biggest gripe with ZooKeeper (as I’ve said many times in the past). It’s also been one of my points of contention with Datomic. If you aren’t on a JVM language, you’re shit out of luck for now. Yes, JRuby makes this billions of times easier for those of us using Ruby but Jython is still not where it needs to be for modern Python.

I also had this problem with Voldemort. Disclaimer this is 2-3 year old data from running Voldemort in production. AFAIK, it’s still the case. For the sake of this discussion, we’re going to ignore data opacity. At the time, the only way to fully access the data in and maintain a Voldemort cluster was from the JVM. I ended up writing quite a bit of JRuby wrapper around StoreClient just to see the data we had in Voldemort.

Riak (and as another example, ElasticSearch) is nice in this regard. It’s HTTP. I can curl it from a shell script. I can use the Ruby library Basho is maintaining. If I’m using a language without ‘official’ support, I can write my own. All the metadata is largely attached to http headers and even monitoring is done via the /ping and /stats urls. Something I didn’t realize until today (thanks to Benjamin Black) is that the stats interface actually exposes stuff I had previously glossed over including cluster topology. This is where the meat of the discussion on twitter today happened.

Operational Happiness

My original statements on this discussion related to using haproxy in front of your Riak cluster. There are several reasons I prefer this but a quick sidebar

Seed Nodes

I’ve had some minor operational experience with Cassandra (nothing to write home about) but one of the things that always bothered me was the idea of ‘seed nodes’. Let me be clear that Cassandra and Riak are pretty much the only two datastores I’d feel comfortable using these days (with the nod going to Riak) in any sort of scalable environment. Postgres has earned its way back to my into my graces but MySQL can insert Louis C.K. euphemism here.

My problem with seed nodes is the idea that I have special nodes. These nodes have to be hard-coded in a config file somewhere or discovered by some other method. I could store them as DNS lookups but now I’ve got to deal with TTLs on DNS. And I’ve got to deal with the fact that DNS doesn’t actually care if the host I’ve been given is actually alive or not.

I could store this information in ZooKeeper but what if I don’t actually have native ZK support in that database? I’ve got to write something that populates ZK when a new node is available and it’s not actually a live check. I still have to test that host first. Yes you should do that anyway but it’s a valid point.

So if I’m storing seed nodes as DNS names in a config file, I can never change those names without either rolling out new code or configs. That might require a restart somewhere. If I’m clever, I could probably make that an administrative hook in my application (think JMX) where I can fiddle the seed list. I can poll a config file for changes. I can do a lot of thing but none of them are “optimal” to me.

Back to haproxy

My prefered method of using Riak is to stick everything behind haproxy. There are several reasons for this but here are a few reasons (note we’re going to assume use of CM at this point):

• Operationally, haproxy nodes are easier to manage than application configs (depending on the application).
• Many times, at different companies, we’ve had to roll our own layer on top of a client library (or even write our own due to licensing issues). Load-balancing is not neccessarily a core application developer competency (and it’s bitten me in the ass before).
• From an operational perspective, I can bring nodes in and out of service in haproxy for maintenance without needing to inform the client or have it waste cycles with stale node detection. It simply will never talk to an out of service backend.

Basically the haproxy crew has already done all the work in load balancing HTTP connections intelligently and they’re pretty damn good at it. I love my developers and I know the folks writing the various libraries I use are smart people but again it goes back to core competency. Note that Basho gets a nod here, however, in that I’m pretty sure that, what with writing webmachine, they grok http pretty well.

I’ve even gone the approach of using haproxy on every system that needs to connect to some backend locally. This totally eliminates the idea of fixed points in the network for connections (at the expense of having to deal with a bit of drift between CM runs). If I wanted to, I could even make multiple riak clusters transparent behind haproxy (though I can only think of a few REALLY specific use cases for that).

Trade offs

Yes there are trade-offs to this approach. As Ben pointed out, the Riak stats interface is really powerful from a topology perspective. A REALLY smart riak client can discovery data layout and make deeply intelligent decisions on that.

With other applications like ElasticSearch I can actually become a full fledged cluster member as a non-data node and actually offload some of the work of the cluster for scatter/gather type operations using the Java library.

With haproxy, I don’t get those types of benefits.

What I do get

Outside of the maintainability of haproxy (which is, again, subjective) I get one benefit I CAN’T get with smart clients that is, unfortunately, neccessary with many customers - a more narrow network allowance.

Enterprises are interesting beasts and still think in terms of traditional tiered application stacks. Nothing wrong with that and it does have SOME security benefits but many deployments I’ve been involved with have official policies that ‘data tiers’ (whatever the hell those are) must be protected in a dedicated network behind an additional firewall. So here’s Riak, a ‘data tier’, that we have to have with as small an ingress as possible. That rules out smart clients of the toplogy-aware variety. So we stick haproxy in the mix. We tell them to use a load balancer. Some folks use an internal F5 HA pair with some sort of VRRP. We’ll set up haproxy + keepalived or some other combination depending.

TMTOWTDI

One of the things that Riak allows is this level of flexibility. I can use HTTP and haproxy. I can use HTTP and smart client. I can use protobufs and haproxy or a smart client. It’s really that flexible. I happen to prefer the haproxy approach for reasons I’ve already mentioned but I totally grok that some folks want a more intelligent client approach. Some folks would argue that there’s a right way and a wrong way but I don’t see it like that. What I see is a datastore that, just like it letting me control the consistency levels I want, let’s me control HOW I access that data.

When I started Noah a few years ago, I had a head full of steam. I had some grand ideas but was trying to keep things realistic. I simply wanted a simple REST-ish interface for stashing nuggets of information between systems and a flexible way to notify interested parties when that information changed.

It started as a mindmap laying in bed one night. It was my first serious project and I had no idea what I was getting in to. If you’re curious, you can read quite a bit of my initial braindumps on the wiki under ‘General Thoughts’. I watched every day as more and more people started following the project.

It was a game changer for me in many ways. Working on Noah was fun and it was rewarding in more ways than one. But real life gets in the way sometimes.

On stewardship

One of the things I’ve learned over the past few years is that for opensource to REALLY thrive, it can’t be a one-person show. I’ve been involved with opensource for most of my 17+ year career. You think I would have learned that lesson before now.

Stewardship is a hard thing. Our arrogance and pride makes us want to keep things close to our chest.

• “I just want to get to a 1.0 release”
• “Things are too in flux right now. It wouldn’t be fair to bring others in”
• “I don’t quite trust anyone else with it yet”
• “Let me just get this ONE part of the API in place first..”

These are all things I said to myself.

What really changed my mind was a few things. Being involved in the Padrino project. Seeing the Fog community grow after Wesley started allowing committers. Seeing Jordan trust me enough to make me a logstash committer before his daughter was born. The biggest trigger was actually one of my own projects - the chef logstash cookbook.

Bryan Berry (FSM bless him) pestered the hell out of me about getting some changes merged in. He was making neccessary changes and fixes. He was evolving it to make it more flexible beyond my own use case. I don’t recall if he asked to be a committer but I gave it to him. The pull request queue drained and he added more than I ever had time for. Not long after, I added Chris Lundquist. Those two have been running it since then really.

I think back to when I got added to the committers for Padrino. It was a rush. It was amazing and scary. Above all it was the encouragement I needed. How dare I deny someone else that same opportunity.

Making that first pull request is hard. To have it accepted is a feeling I’ll keep with me for a long time. I can only hope that some project I create some day will give someone that same confidence and feeling.

So what about Noah

Noah is in the same place Logstash was. I’m not using it and that’s really hurting it more than anything. It’s time to let someone who IS using it take control. I care too much about it to watch it die on the vine. I still believe in what it was designed to do and every single day I get emails asking me if it’s still alive because it’s a perfect fit for what someone needs. The same stuff is STILL coming up on various mailing lists and Noah is a perfect fit. There are companies actively using it even it the current unloved state. Those folks have a vested interest in it.

When I added Chris and Bryan to the cookbook, I sent them an email with what my vision was for the cookbook. I can’t find that email now but I recall only had two real requirements:

• Out of the box, it would work on a single system with no additional configuration (i.e. add the cookbook to a run_list and logstash would work automatically)
• A user never had to modify the cookbook to change anything related to roles (i.e. allow the attributes to drive search for discovering your indexer - hence all the role stuff in the attrs now)

I need to do the same thing for Noah and see where it leads.

Dat list

This list isn’t comprehensive but I think it hits the key points.

Simple

Noah should be simple to interact with. It was born out of frustration with trying to interact with ZooKeeper. Nothing is more simple than being able to use curl IMHO. I can use Noah in shell scripts and I can use it in Java (we had a Spring Configurator at VA that talked to Noah. It was awesome). You should always be able to use curl to interact with Noah. I wish I could find it now but someone once brought up Noah on the ZK mailing list. This led to various rants about how it didn’t do consensus and a bunch of other stuff that ZK did. One of the Yahoo guys (I wish I could remember who) said something in favor of Noah that stuck with me:

Interfaces matter

I know I’m on the right track here because Rackspace just built a product that provides an HTTP interface to ZK. Oh and it does callbacks.

Friendly to both sysadmins and developers

Simplicity plays into this but I wanted Noah to be the tool that solved some friction between the people who write the code and the people who run the code. Configuration is all over the place in any modern stack. Configuration management has come into its own. People are using it but you still see disconnects. Where should this config be maintained? What’s the best way to have puppet track changes to application configuration? I can’t get my developers to update the ERB templates in the Chef cookbook. All of these things are where Noah is helpful.

I still stand by the statement that not all configuration is equal. Volatility is a thing and it doesn’t have to mean the end of all the effort in moving to a CM tool. I wanted to remove that friction point.

I was also immensely inspired by Kelsey Hightower here. I’ve told the story several times of how Kelsey got so frustrated that the developers wouldn’t cooperate with us on Puppet and config files for our applications that he learned enough Java to write a library for looking up information in Cobbler. Cobbler has an XMLRPC api and that was simple enough that he could port his python skills to java and write the fucking library himself. I wanted Noah to be friendly enough that a sysadmin could do what Kelsey did.

Watches and Callbacks

I’ve said this before but one of the most awesome things that ZK has is watches. They have pitfalls (reregister your watches after they fire for instance) but they’re awesome. Noah’s callback system is the thing that needs the most love (it works but the plugin API was never finalized). It’s also one of the most powerful parts that meets the needs of folks that I see posting on various mailing lists.

The idea is simple. When something changes in Noah, you should be able to fire off a message however the end-user wants to get it. I think this is one of the reasons I love working on Logstash so much. Writing plugins is so simple and it’s the gateway drug to anyone who wants to contribute to logstash.

Things I don’t care about

What don’t I care about?

Language

I don’t care about the language it’s written in. If someone wants to take it and convert it to Python or Erlang or Clojure, be my guest. I just want the ideas to live on somehwere. In fact, I’ve rewritten various parts of Noah over the last year privately. Not just experimenting with moving from EM to Celluloid but as a Cherry.py app, in Clojure and I even started an Erlang attempt (except that I know almost NO Erlang so it didn’t get very far).

Name

Honestly I don’t even care about the name. Yeah it’s witty and fits with the idea of ZooKeeper but I have no qualms about adding a link to your project from the Noah readme and recommending people use it instead.

Paxos/ZAB

This was never a requirement for Noah. Noah was specifically designed for certain types of information. If you need that, use the right tool.

Persistence

Let’s be honest. From a simplicity standpoint, it doesn’t get much simpler than Redis. It’s one of the reasons we changed the default logstash tutorial to use Redis instead of RabbitMQ. I know Redis reinvents a lot of wheels that have already been solved but it, along with ElasticSearch, are one of the lowest friction bits of software I’ve dealt with in a long time. Not having external dependencies is a godsend for getting started.

However I’ve also got small experiments privately where I used ZMQ internally and sqlite. I’ve written a git-based persistence for it too.

Riak is also a great fit for Noah and takes care of the availability issue on the persistence side. More on Riak in a sec.

So that’s it

That’s really all that matters. If you want to take ownership of the project, contact me. Let me know and we’ll talk. Who knows. Maybe I’m overestimating the level of interest. Maybe ZK isn’t as unapproachable to people anymore. The language bindings have certainly gotten much better. I just want the project to be useful to folks and I’m getting in the way of that.

What are the other options?

I don’t know of many other options out there. Doozer is picking up steam again as I understand it and it has a much smaller footprint than ZK does. There was a python project that did a subset of Noah but I can’t find it now.

One thing that is worth considering is a project that I found earlier today - zpax. While this is just a framework experiment of sorts, it could inspire you to add your own frontend to it. The same author is also working on DTLS on top of ZMQ.

I’ve thought about ways I could actually do this with Logstash plugins. It’s doable but not really feasible without making Logstash do something it isn’t shaped for.

Another idea that I’m actually toying around with is simply using Riak plus a ZeroMQ post-commit hook so that plugins could be written in a simpler way. Sean Cribbs already took the idea and made a POC 2 years ago based on a gist from Cody Soyland. You wouldn’t have the same API up front as Noah but you could stub that out in some framework and also have it be the recipient of the ZMQ publishes.

Finally you could just use ZooKeeper. Yes it has MUCH greater overhead but you DO get a lot more bang for the buck. There really isn’t anything in the opensource world right now that compares. It also provides additional features that I never really cared about or needed in Noah.

Wrap up

I’m not done in this space. I don’t know where I’m going next with it. Maybe I’ll start from scratch with a much simpler API. Maybe I’ll just run with the Riak idea.

I just want to give a shoutout to the countless people who helped me evangelize Noah over the last few years. It was recommended on mailing lists, twitter and many other places. It meant a lot to me and I only hope that someone will take up the mantle and make it something you would recommend again.

For those of you still using Noah, I hope we can find a home for it so that it can continue to provide value to you.

Thanks.

As part of that redesign, I worked on what we lovingly refer to internally as the “solo installer”. I gave a bit of background on this in a post to the Chef mailing list at one point but I’ll go over it again as part of this post.

Beginnings

To understand why this is something of a departure for us, it’s worth understanding from whence we came. enStratus, like most hosted solutions, has experienced largely organic growth. One of the nice things about a SaaS product is that you have the freedom to experiment to some degree. You might be in the middle of a MySQL to Riak migration and still need two data stores for the time being. You might be in the process of changing how some background process works so you’ve got an older designed system running along side the newer system which is only doing a subset of the work.

With a hosted platform these kinds of things are hidden from the end-user to some degree. They don’t know what’s going on behind the scenes and they really don’t care as long as you’re doing X, Y and Z that you’re being paid to do.

Now for those of you running/developing/managing some sort of SaaS/Hosted solution I want you to take a journey with me. Imagine that tomorrow someone walked into your office and said:

We need to take our production environment and make it so that a customer can run it on-premise. Oh and it has to be able to run entirely isolated and can’t always talk back to any external service.

That’s pretty much the place enStratus found itself. enStratus is a cloud management platform. Not all clouds are public. Maybe someone wants to use your service but has regulatory needs that prevent them from using it as is. Maybe it needs to run entirely isolated from the rest of the world. There are valid reasons for all of these despite my general attitude towards security theater in the enterprise.

Now you’ve got an interesting laundry list in front of you:

• How do you teach someone to manage this organic “thing” you’ve built?
• How do you take not one application but an entire company and its stack and shove it in a box?
• Do you wrap up all the external components (monitoring, file servers, access control) and deal with those?

We aren’t the first company to do this and we won’t be the last. Take a look at Github. They offer a private version of Github. But we’re not just talking about one part of Github - e.g. Gist. We’re talking about the entire stack the company runs to provide Github as we know it.

Unless you design with this in mind, you can’t really begin to understand how difficult of a task this can be. As I understand it, Github finally went the appliance route and offer prefab vms with some setup glue. Please correct me if I’m wrong here.

Early iterations

Obviously you can see that this was/is a daunting task. Original versions of our install process were based around a collection of shell scripts. Because of certain details of the stack (such as encryption keys for our key management system), we had to maintain state between each component of the stack when it was installed. Currently there are roughly 7 core components/services that make up enStratus:

• The console
• The api endpoint
• The key management subsystem
• The provisioning subsystem
• The directory integration service
• The “worker” system
• The “monitor” system

and those are just the enStratus components. You also need RabbitMQ, MySQL and Riak (as we’re currently transitioning from MySQL to Riak). All of these things largely talk to each other over some web service or via RabbitMQ. With one or two exceptions, they can all be loadbalanced in an active/active type of configuration and scaled horizontally simply by adding an additional “whatever” node.

So the original installation process was a set of shell scripts that persisted some state and this “state” file had to be copied between systems. Yes, we could use some sort of external configuration store but that’s another component that we would have to install just to do the installation.

Phase two

One of my coworkers, Greg Moselle was sort of “sysadmin number one” at enStratus. This was in addition to his duties as managing all customer installs. So he did what most of us would do and brute forced a workable solution with the original shell scripts. As enStratus started to offer Chef and Puppet support in the product, Greg gets this wild hair up his ass and thinks:

I wonder if I can rewrite these shell scripts into something a bit more cross-platform and idempotent using chef-solo.

You might be thinking the same thing I originally did that this was largely a bad idea. In my mind we had a workable solution for the interim in the existing shell scripts that had the install of enStratus down to a day or so. Pragmatism right? It’s also worth noting that this was how he wanted to learn Chef…

So off he goes and does what I recommend any new Puppet or Chef user does - exec resources all over the fucking place. Wrap your shell scripts in exec. Hardcode all the fucking things.

Once he did this, then I started working with him on some basic attributes to make them a bit more flexible. Before too long we had a stack of roles matching different components and we had moved everything from cookbook_file to remote_file. It was still a mess of execs but it worked.

But we still had this “state” we had to maintain between runs. This is not going away anytime soon. In production we store this state in attributes and use chef-server. We didn’t have that luxury here.

Then Jim Sander drops in and writes a small setup script that maintains some of that state for us. Basically a wrapper around raw chef-solo. Side note, if you ever need someone to drop some shell scripting knowledge on your ass, Jim’s the man to see. Ask him about his Tivoli days to really piss him off.

At this point, I start working on cleaning up the recipes as a sort of tutorial for folks. I’d pick a particular recipe and refactor it to all native resources and make it data driven. I’d commit these in small chunks so folks could easily see what the differences were easily - stuff like “instead of execing to call rpm, we’ll use the yum provider”.

At this point we’ve got something pretty far evolved from where we were. Now that we’ve got this workable chef-solo repository, I decide to hack out a quick Vagrantfile. The problem was it wasn’t entirely idempotent and we still had some manual steps that had to be dealt with. In addition to finishing up the recipes and ended up rewriting large chunks of the setup script. Now that I had something largely repeatable and localized, we suddenly had a Vagrant setup that folks could use for development. It wasn’t fully automated but it worked. We also still had this shared state thing.

So I set out to refactor the setup script a bit more. What’s important to keep in mind is that the primary use-case for this chef-solo repository wasn’t for Vagrant. This is our “installer”. The interesting part to me is that the improvements to how we do on-premise installs are coming as a direct result of making this work better with Vagrant. There’s a lot of wrapper work tied up in the setup script that wouldn’t need to be done if we used a base box that had more stuff baked in. However not baking stuff in actually gives us a more real-world scenario for installation.

Additionally we needed to be able to somehow pass user-specific configuration settings into the vagrant up process and get those into chef-solo by way of the setup wrapper. We have things like license keys, hostnames and my personal hated favorite - database credentials - that need to be handled in a way that we can make it so a developer can just type vagrant up and be running. If I have to require someone to edit a json file or anything else, the whole thing will fall flat on its face.

So any time we needed something like that, we added support to the setup wrapper and then used environment variables to pass that information in to vagrant.

So how do we vagrant?

We leverage environment variables pretty heavily in our Vagrantfile. If it’s something that someone might need to tune for whatever reason, it’s an environment variable that triggers an option to our setup script.

Current list of tunables

This is just a subset of the tunables we control via environment variables. The majority of these map directly to options for the setup script:

• ES_DLPASS and ES_LICENSE: the basic set of credentials needed to fetch our assets and your personal license key.
• ES_MEM: this is the result of some of our front end developers having less memory than others.
• ES_CACHE: We have an office in New Zealand and bandwidth there is “challenging”. This allows us to cache as much as possible between calls to vagrant up. This not only triggers caching of system packages downloaded but also triggers the prefetch option in our setup script that predownloads all the assets. These assets are all stored in the cache directory of the repository which is not coincidently the value of file_cache_path in chef-solo. Remember that we may not always have external network access during installation so we offer a way to warm the cache directory with as many assets as possible.
• ES_BOX: let’s you specify an alternate base box to use. This is how we test the installer on different distros.
• ES_DEVDIR: shares an additional directory from the host to the vagrant image. This is how development is done (at least for me). I map this to the root of all of my git repository checkouts.
• ES_VAGRANT_NW: Allows you to configure bridged networking in addition to the host-only network we use.
• ES_PROFILE: This directly maps to an option in our setup script for persisting state between runs.

There are other options that are specific to the enStratus product as well but you get the idea.

The setup script

I can’t post the full thing here but I can give you a general idea of how it works and some of the options it supports. This is a “sanitized” truncated version of the help output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

Usage: setup.sh [-h] [-e] [-f] -p <download password> -l <license key> [-s savename] [-c <console hostname>] [-n <number of nodes>] [-m <mapping string>] [-a <optional sourceCidr string>]
-------------------------------------------------------------------------
-p: The password for downloading enStratus
-l: The license key for enStratus

For most single node installations, specify the download password and license key.

optional arguments
------------------
-h: This text
-e: extended help
-f: fetch-only mode. Downloads and caches *MOST* assets. Requires download password and *WILL* install chef
-c: Alternate hostname to use for the console. [e.g. cloud.mycompany.com] (default: fqdn of console node)
-a: Alternate string to use for the sourceCidr entry. You know if you need this.
-s: A name to identify this installation
-n: Number of nodes in installation [1,2,4] (default: 1)
-m: Mapping string [e.g. frontend:192.168.1.1,backend:backend.mydomain.com]

About savename:
---------------
Savename is a way to persist settings between runs of enStratus.
If you specify a save name, a directory will be created under local_settings
will be created. It will contain a YAML file with your settings as well 4 JSON files.

The YAML file is the source of truth for the named installation. The JSON files MAY
be recreated if the contents of the YAML file change. They exist to migrate between systems.
If a save file is found, no other arguments are honored. If you need to change the 
download password or license key, please update the YAML file itself

If you lose this YAML file you will not be able to recover this enStratus installation.
You should save it somewhere secure and optionally version it.

Persisting settings

One of the “gotchas” we have is how do we basically build a node JSON file for chef-solo to use with any information we need to persist. Since we don’t know the state of all the systems involved when we go in, we have to “punt” on a few things. What we end up doing is something we call the savename. If you use this option, the settings you define will be persisted to a directory that git ignores called local_settings. This directory will contain directories named after the above savename parameter. The setup script (written for now in bash) will create a yaml file (easy to do in bash with HEREDOC as opposed to JSON) and also a copy of the generated encryption keys in a plain text file for the customer to store.

The only thing we can count on being on the system up front is the Chef omnibus install (since that’s a requirement). Instead of complicating things with ruby at this point (and chicken/egg issues since the setup script actually installs chef omnibus), we use the erubis binary that gets installed with omnibus to pass the yaml to to a JSON erb template. That generated JSON is the node json with attribute overrides. We actually support multi-node installation in the setup script if you provide a mapping of where certain components are running when calling setup. If you rerun setup using an existing savename parameter, the yaml file is updated (only certain values) and then regenerate the JSON file.

The upshot

The best part of all of this is that we can now say the same process is used when installing enStratus locally in Vagrant, in our dev, staging and production environments (though production uses chef-server) as well as what we install on the customer’s site. We version this repository around static points in our release cycle. We branch for a new release and create tags at given points in the branch based on either a patch release for enStratus itself in that release OR a patch to the installer itself.

It’s not all unicorns pooping rainbows. The process is much more complicated than it needs to be but it’s almost a world of difference from where it was when I started and it was entirely a team effort. This setup allowed us to do full testing to switch entirely off the SunJDK (and the need to manually download the JCE during customer installs) onto OpenJDK. We were able to migrate from Tomcat to Jetty and refactor our build process using this method. I was able to do this work without affecting anyone else. All I had to do when we were ready for full testing was tell everyone to switch branches, run vagrant up and test away.

Special thanks

I want to give a serious shout-out to Mitchell Hashimoto and John Bender for the work they did with Vagrant. Last year I said that no two software products impacted my career more than ElasticSearch and ZeroMQ. This year, without a doubt, Vagrant is at the top of that list.

Addendum

What follows is the sanitized version of our Vagrantfile. If anyone has any suggestions, I’m all ears:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

Vagrant::Config.run do |config|
  if ENV['ES_BOX']
    config.vm.box = ENV['ES_BOX']
  else
    config.vm.box = "es-dev"
    config.vm.box_url = "https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-ubuntu-12.04.box"
  end

  if ENV['ES_VAGRANT_NW'] == "bridged"
    config.vm.network :bridged
  else
    # If you change this address, the conditional logic
    # in console.rb will break
    config.vm.network :hostonly, "172.16.129.19"
  end

  # These entries allow you to run code locally and talk to a
  # "working set" of data services
  config.vm.forward_port 15000, 15000   # api
  config.vm.forward_port 3302, 3302     # dispatcher
  config.vm.forward_port 2013, 2013     # km
  config.vm.forward_port 5672, 5672     # RabbitMQ (autostarts)
  config.vm.forward_port 8098, 8098     # Riak HTTP (autostarts)
  config.vm.forward_port 8097, 8097     # Riak protobuf (autostarts)
  config.vm.forward_port 3306, 3306     # MySQL (autostarts)
  config.vm.forward_port 55672, 55672   # RabbitMQ management interface

  if ENV['ES_MEM']
    config.vm.customize ["modifyvm", :id, "--memory", ENV['ES_MEM']]
  else
    config.vm.customize  ["modifyvm", :id, "--memory", 8192]
  end

  if ENV['ES_DEVDIR']
    config.vm.share_folder "es-dev-data", "/es_dev_data", ENV['ES_DEVDIR']
  end

  if ENV['ES_CACHE']
    puts "Shared cache enabled"
    FileUtils.mkdir_p(File.join("cache","apt","partial")) unless Dir.exists?(File.join("cache","apt", "partial"))
    config.vm.share_folder("apt", "/var/cache/apt/archives", "cache/apt")
  end

  config.vm.provision :shell do |shell|
    ES_LICENSE=ENV['ES_LICENSE']
    ES_DLPASS=ENV['ES_DLPASS']
    ES_PROFILE=ENV['ES_PROFILE'] || "vagrant-#{Time.now.to_i}"

    if ES_LICENSE.nil? or ES_DLPASS.nil?
      puts "You must set the environment variables: ES_LICENSE and ES_DLPASS!"
      exit 1
    end
    ES_CLOUD=ENV['ES_CLOUD']
    ES_CIDR=ENV['ES_CIDR']
    ES_DEBUG=ENV['ES_DEBUG'] || false
    setup_opts = "-l #{ES_LICENSE} -p #{ES_DLPASS} -s #{ES_PROFILE} "
    setup_opts << "-c #{ES_CLOUD} " if ES_CLOUD
    setup_opts << "-a #{ES_CIDR} " if ES_CIDR
    ES_DEBUG ? chef_opts="-l debug -L local_settings/#{ES_PROFILE}/chef-run.log" : ""
    shell.inline = "cd /vagrant; ./setup.sh #{setup_opts}; chef-solo -j local_settings/#{ES_PROFILE}/single_node.json -c solo.rb #{chef_opts}"
  end
  if ENV['ES_POSTRUN']
    config.vm.provision :shell do |shell|
      shell.inline = "chef-solo -j /vagrant/local_settings/#{ES_PROFILE}/single_node.json -c /vagrant/solo.rb -o \"#{ENV['ES_POSTRUN']}\""
    end
  end
end

One thing that I have the hardest time communicating to friends and family who aren’t in the IT industry is the concept of “production” and what it means to be on-call. Even coworkers have a hard time understanding what it means.

The topic recently came up again and the confusion bothered me so much that I resolved to write this blog post as soon as humanly possible.

A few clarifications

I want to clarify a few very important things:

• I’m not whining about what I do
• I love what I do
• I’m not burned out
• I’m not being self-important
• I’ve always had a hard time ‘ranking’ problems. EVERYTHING is important to me.
• I’m not really interested in critiques of what SHOULD have been done. Riak wasn’t around, for instance, when I was managing the financial stuff for instance.
• Yes, rotations are important but not always viable. Luckily we have a solid rotation at enStratus.

What production means to me (and why)

Production environments take many forms. It’s even harder to define. For me, production has always meant “any system, service or component that the business requires to do business”. I’ve worked in several different companies over the last 17 years. In some cases, production was an ERP system or a file server. In other cases, production was a web presence. What’s interesting each of these is that in some cases, production had a time associated with it.

Let’s take a few of these and compare them:

The retail financial company

Long ago I worked for a company that did payday and title loans. We operated over 600+ retail locations from east to west coast. The primary application used by these outlets was a web-based loan management application (websphere + db2). Stores were located across the country and store hours were from 9AM to 6PM (IIRC). From this you might think “production was the web application and it needed to be online from 9AM EST to 10PM EST”. You would be correct at the highest level.

However employees started the day at 8AM (lining up customers to call and what not) and left at 7PM (closing out books for the day). After the last store went “offline”, we began various nightly batch jobs. Being that this was financial in nature, batch jobs were the norm. We also had backups that had to run as we rebuilt our QA database nightly from sanitized production database backups. If the batch jobs didn’t finish in time, we actually had to delay the start of the day for the stores. Our datawarehouse was also loaded from a secondary copy of the database restored from backup. If I recall the main reason for this was that our nightly window was SO crunched for time that we couldn’t even load the warehouse from the main database because we had to start various batch jobs as soon as backup was done.

But that’s just the main system. We had ancillary services as well. None of the retail outlets had access to the internet except through a squid proxy. There were print servers that did server-side check printing. In the backoffice, we had collections and other things that depending on the reports that came out of the data warehouse. We had nightly backups of the LAN stuff. DNS servers that the stores had to use. VPN concentrators. ALL of this had the same SLA as production.

All told, I recall the final number for any business hours outage as costing us something like $100k for 15 minutes of downtime.

The Learning Management System

This system was used by a charter school system in the state of Ohio. It was an online classroom system that provided education for at-risk students. This WAS the school for these kids. Obviously it had normal school hours but since the students had no physical textbooks of any kind, the system HAD to be online for something as simple as homework. As with the previous setup, there were all sorts of ancillary services that we had to have available. All of the static content was shared across all of the tomcat servers via a SAN (OCFS2 - I have scars). It was backed by MySQL. We still had to do backups. We had to maintain connectivity. Everything we had was ‘production’. Since we had developers in other countries on different hours, we burned what money we had when the development environment or our SVN repo was offline. That was production too.

Web applications in general

To those in the industry I’m not telling you anything new. But to those not in the know, the internet doesn’t have office hours. Yes, you can gauge where your largest userbase is but take a system like enStratus.

It manages cloud resources for people all across the world. For many of these people, the only access they have to their cloud account is VIA enStratus. Take AWS for instance. enStratus is responsible for detecting outages and replacing components in the infrastructure for these companies or autoscaling to meet some demand. If enStratus is offline, these actions are NOT being taken on behalf of the user. The biggest fear for me is that enStratus is offline when AWS is having an outage. Some customers are paying us for this use case alone. Mind you in the last few outages, not even enStratus could fix the problem because of control plane issues. One thing enStratus can do is scale across multiple clouds so even AWS control plane issues are no excuse.

enStratus production itself is a pretty complex beast. The stack in general is designed to be fairly “SOA”. We use RabbitMQ pretty heavily for workers. However we’ve had some issues in the recent past where our workers were getting OutOfMemory exceptions. We run multiple workers (obviously) but in this case an OOM on one worker would eventually translate into OOMs across all the workers. When all the workers OOMd, they would stop processing messages from the queue. When that happened, RabbitMQ could eventually tank from units of work waiting to be picked up. We never had this happen, mind you but that was the end game.

This meant we had to be diligent on these OOMs. All the time. 24x7x365.

Luckily this problem was fairly short-lived but until the bug was identified and fixed (it happened to be related to an edge case with S3 bucket sizes), we had to be on guard for these OOM exceptions.

What does it mean?

The thing that I want to get across is that “production” is DIRECTLY related to the bottom line of the business. If “production” is offline, customers can’t use the system. Customers are unhappy. If customers are unhappy they eventually go elsewhere. If customers go elsewhere the company loses money. If the company loses money eventually the company lays people off. This isn’t rocket science. We talk about complex systems and cascading failures. This is a cascading failure that means someone doesn’t have a job at Christmas.

Yes, I take it that seriously. When I talk about production, that’s what I mean.

On-call

Now that you know what production means and what impact it has, I shouldn’t need to say much about being “on-call”. Yes, I’m on-call now and then. Yes, just like a doctor. Sometimes, depending on the company, I’ve been the only person on-call. Ever. No rotation. No help. Just me. The one person keeping shit running all the time so that customers (internal or external) aren’t impacted by the slightest glitch in the system. Yes we should build more resilient systems and we strive to do that. However, tech debt is a thing. It’s not always immediately an option.

So when I’m on-call it means I’m the person responsible for production as defined above and all the baggage that comes with it.

Someone once said to me “Most of us work nights and weekends, although we try to balance it when we can.”

While I appreciate the sentiment, I don’t just work nights and weekends, I work ALL the time. I have to be able to respond at a moments notice to a production issue when I’m on-call. Think about that for a moment. I have to essentially be 15 minutes from a working internet connection and may need to sit in front of a computer for an unspecified amount of time until the issue is resolved. I can’t just go to bed or decide that I’ve worked enough for the day.

The joys of working remote

I’ve been pretty fortunate in the past several years to be in a situation where I can support production from pretty much anywhere. I need at most a 3G connection and my laptop. I can VPN into production and fix most any problem. Yes, I always bring my laptop on “vacation”. If I don’t have a decent signal, then there’s a chance I’ll have to drive to the McDonald’s 15 minutes away to use the wifi. I try not to be on-call when I’m on vacation but sometimes it’s not always an option.

I just got back from visiting my in-laws Up North in Michigan. The only time I was able to get a single bar of 3G was late at night when enough subscribers stopped using the cell towers. Up until a year or so ago, there was no option for any sort of broadband internet in the area.

I’ve finally gotten my in-laws to understand much of what I’ve written here. They’ll be wiring the cottage for cable internet so that I can take the family up for longer periods of time. Yes, I’ll be “working” on vacation but I’d rather work a bit on vacation and stay longer than have to cut the trip short just so I can get back.

Poor poor me

As I said at the start, please don’t misinterpret what I’m saying as whining or some sort of god complex. This is the field I’ve chosen. I love what I do and I take that responsibility very seriously. As we start to build more resilient systems and accept that failure WILL happen and design for it, things will only get better. I still remember the first time I was able to replace a problematic system from scratch in 20 minutes via configuration management and call it a night.

As enStratus migrates its backend to Riak DS and builds out secondary and tertiary datacenters, I look forward to losing a data node not being the end of the world. DevOps über alles and all that.

So dear friends and family, the next time someone tells you they’re on-call for production….cut ‘em some slack.

First time I’ve had to do this but: the following is my personal opinion and in no way reflects any policy or position of my employer

A journey through time

I remember when EC2 was first unleashed. At the time I was working at Roundbox Media (later Roundbox Global - because we had an office in Costa Rica!). I was asked frequently if we could possibly host some of our production stuff there.

It was pretty much a no go from the start:

• No persistent IPs
• No persistent storage

Sure we could bake a bunch of stuff into the AMI root but the ephemeral storage wasn’t big enough to hold a fraction of our data. Still, we leveraged it as much as possible. We ran quite a bit of low risk stuff on it - test installs of our platform and demo sites for customers.

After I left RBX in Feb of 2008, I didn’t get an opportunity to work with AWS for a year or so and by then quite a bit had changed. If Amazon does one thing really well, it’s iterating quickly on its service offerings.

So why is EBS a bad thing?

For Amazon, EBS is NOT a bad thing. It was probably one of the smartest business moves they made (along with Elastic IPs). They could now claim that EC2 was JUST like running your own kit - you have a SAN! you have static IPs!

The problem is it’s not.

The nature of block storage

Anyone who’s dealt with any sort of networked filesystem knows the pains it can cause with certain application profiles. Traditional databases are notorious for expecting actual local storage and real block devices. It amazes me the number of people who put up with the pain of running a database in something like vmware using virtual disks hosted on an NFS device.

The point is the block devices have specific semantics and presumptions.

With EBS you’re promised a tasty block device that your OS can address as if it were local disk. Only it’s not….

Latency

Let’s get the biggest elephant out of the way. EBS is a block device to the OS but under the hood it’s using the network. It may or may not be shared with non-block device traffic but it’s still subject to network latencies. God I hope that EBS at least gets its own port on the host side…

Shared

There’s a whole lot of sharing going on here to:

• local bandwidth from the physical server where your instance is to a given EBS subsystem (array, CEC, whatever)
• aggregate bandwidth from all pysical servers talking to a given EBS subsystem
• disk I/O itself on a given EBS subsystem

I don’t know how the connection from server to EBS is done. I would hope at least there are bonded ports or multiple uplinks/multipathing going on. I would REALLY hope that network I/O and Disk I/O are not on the same channel. Regardless, you’re still sharing whatever the size of that connection is with everyone else on the physical server your instance is on if they’re using EBS as well.

And the physical EBS array where your volume is? Depending on the size of your EBS volume, you’re dealing with network I/O on that unit’s connection from an unknown number of other customers. And to top it off, you’re not just sharing network bandwidth, you’re sharing disk bandwidth as well. There are still spindles under there folks. Sticking an API in front of it doesn’t change the fact that there is spinning rust under the covers.

Above ALL of that, you’ve got competing workloads - sequential vs random read.

Sure, just stick your root OS volume on that. That’s a great idea.

Mixed messages

To me, however, the biggest problem with EBS is not the latency. It’s not the shared resources. It’s not even taking something that is fundamentally locality oriented and trying to shoehorn it into something distributed.

It’s the fact that it sends the wrong damn message. I’ve said this before, I’ll say it again and I’ll stand by it.

Unless you are willing, able or have designed your applications to have any single part of your infrastructure - connectivity, disk, node, whatever - ripped from under you with no warning whatsoever, you should not be running it on Amazon EC2.

By providing EBS, Amazon sends the message that “you can treat this just like your own datacenter”. Just use EBS and you can treat it just like a SAN. Look, we have snapshots!

Hell, I get pissy when folks refer to instances as “boxes” and talk about them like they’re something they physically own. Stop trying to map physical datacenter analogies to AWS. It won’t work and you’ll be disappointed.

You want to know the real kicker? You should be designing like this ANYWAY. Yes, you have much greater control over failure points when you run everything yourself. You have much greater control over resource sharing and I/O profiles. That doesn’t remove the need to design for failure. How far you take it is up to you (and realistically your budget) but when you’re running on AWS, you need to be much more attentive to it.

For the record

I still think AWS and public clouds are awesome. I really do. I think private clouds are just as awesome. The flexibility they offer is almost unmatched but that flexibility comes at a price - performance hits, multiple layers of abstraction and other things.

So the question I found myself asking today is “Does monitoring suck less than it did a year ago?”. I’d have to be an idiot to say anything other than “yes”.

I didn’t set out to start a “movement”. I’m not someone who handles compliments very well. Mainly because I suffer from a bad case of imposter syndrome. The other side of this is that I know that I’ve done next to nothing to make it any better. Meanwhile the real heros are people releasing code every day. People rethinking how we think about monitoring, trending, alerting and everything else. Companies like Etsy, Netflix, Yammer and countless more are sharing the deep squishy bits of how they do things and are releasing code to back it up.

What’s gotten better?

I think the biggest thing that’s gotten better is that people are really starting to leverage advances we’ve had in ancillary tooling in recent times. Not that any of these ideas are new (message busses existed long before RabbitMQ) but the barrier to entry is much lower.

I still feel like what drove this was a byproduct of configuration management uptake. As it became easier to stamp out servers, the rate of change in our infrastructure surpassed what tools were original designed to deal with. As we started having more time to think about what we wanted to monitor (because we weren’t spending all our time building hand-crafted artisan machines), we started to feel the pain more. We wanted our monitoring systems to work as smoothly as the rest of the kit but it didn’t.

Combine that with:

• We now have data storage engines of varying complexity for storing time-series data. And with greater resolution and the ability to change that resolution.
• We have tooling that can read that timeseries data and represent it in dynamic ways.
• We revisted the idea of push vs. pull and leave/join of components in our infra.

The world is a brighter place because of this and to everyone who had something to do with it - whether discussing on a mailing list, talking on IRC, tweeting, writing code or whatever - thank you.

What’s coming down the pipe

As a side effect of this monitoring thing, people ask my opinion a lot. Like I said, I’m still weirded out by this. Stepping back, though, and just geeking out on things, I see some really cool stuff in the future. Here’s just a subset of ‘stuff’ I’ve been thinking about:

Presence-based Discovery

I couldn’t think of a better way to describe it but the idea is simply (or not so simply) that by virtue of coming online, a system is saying “I wish to be monitored in this way”. This is pretty dependent on configuration management for this to go smoothly, imho.

I mentioned this in a post on the devops toolchain but it goes something like this:

• new node comes online
• new node registers its presence in some way (I’m kind of keen on the XMPP idea) with a notification of services it offers
• centralized system is monitoring (har har) this presence system and starts monitoring the system based on predefined criteria at a “well-known” endpoint
• optionally the system can dictate what it wants monitored

Obviously this would all be very painful without some sort of configuration management system. However, it’s very easy for me in my base group or role for a system to say “Install W, register via X, listen on Y for active checks and publish everything to Z endpoint”. What W,X,Y and Z are is irrelevant. We can cookie cutter this stuff. FWIW this is nothing new. We’re just seeing “consumer-grade” options that are usable by everyone.

Push vs Poll

I’ve said many times that poll-based monitoring is dead. That’s a bit of hyperbole. What’s dead is the idea that we can only check X every N times over K period. This is a hold-over from ineffecient polling mechanisms that would crumble under too-frequent polling as well as systems that weren’t able to handle being polled that often. I see polling moving from “check host X every 5 minutes for memory usage” to “watch this bus for memory usage stats and if there’s not anything in 2 minutes, make sure the world is okay”. We’ll always need the outside-in checks of things but that’s much less intensive than polling ALL the things.

We’re so close with tools like Graphite now which can accept arbitrary metrics from anywhere with no need to preconfigure it to accept them. There are some concerns here around bad data being injected from unauthorized sources. As we automate more and make decisions based on this data, we need to be aware of it. Another discussion for another day though something akin to the way mcollective does trust is probably in order.

Self-service and Culture

This is a big one too and I think will cut down on many complaints that people have around even something like Nagios.

We have to be able to say “You know what? We don’t need to monitor that. Let’s disable that check”. If something is unactionable, then why the hell are you alerting on it? This is where decoupling the trending/visualization from alerting can be so powerful. I’m currently rebuilding our monitoring setup to do most checks based on data in Graphite. Why? Because if I flip the relationship around, I’ve now got to deal with the alert question before I can even get the information. Instead of alerting on data and then storing it as an afterthought (perfdata anyone?) let’s start collecting the data, storing it and then alerting based on it.

This also provides for options around self-service. Not everyone needs to know about the disk space on nodeX. Only the people who can fix it do. Maybe your database folks want to get alerts on queries taking longer than N. As an operations person, you can’t do anything about that and certainly not at that moment (in most cases). You’re just going to push it down the line ANYWAY. And do you really want to have to deal with changing thresholds on behalf of someone?

I’m also a big fan of the idea that components in your infrastructure - applications, os, whatever - self-host a pubsub type of endpoint where users can get realtime information about the system themselves. I do this with every logstash install I setup where possible. Every remote logstash agent I’ve setup in enStratus also provides a pubsub 0mq socket that you can use to live tail log information from that host broken down in topic keys around metadata.

Application health by default

I’ve fawned over Coda Hale’s “Metrics” talk several times. I’m far from a fanboy but “Metrics” gets it right. This ties into self-service quite a bit. Developers need to be free to instrument the application without creating “yet another place to look”. Metrics does this so well with the idea of pushing instrumentation out of the application (oh look - push again!) and into Graphite or whereever is appropriate. And if you aren’t ready for push yet, you can still poll via JMX.

The idea has been ported to multiple other languages at this point. There no excuse not to deeply instrument your applications. If you have an application that CAN’T be instrumented properly, maybe you should consider a different application?

Applying science and common sense

The last thing that I see as being a step forward is we start applying science to our process. No more -w 60,60,60 -c 75,75,75 canned thresholds. We start thinking about our thresholds. Maybe we do away with them entirely as static constructs. Instead we apply event processing and historical data to build thresholds that are intelligent.

We start looking at the shape of our infra. Is it REALLY important that you get woken up at 3AM because a Riak node is down? Not if it’s just one but maybe if it’s two depending on your cluster size. Maybe both of those nodes were in the same rack. Okay that’s bad.

We start to consider context and start applying science! We step out of the shamanistic ages of monitoring (who the hell still sets swap space to double physical memory and is 75% of a 1TB volume in use really something that can’t wait?) where “We’ve always done it this way”. Start thinking like the Apple 1984 commercial, whip out your hammer and smash your preexisting notions around what constitutes an alert.

What I’m building

Right now I’m spending most of my time being pragmatic. I’m still adding new checks to Nagios but the information is coming from different sources now. I’m dumping data into graphite via logstash and doing checks on that. Collectd is now pushing directly to graphite as well. Nagios is becoming less and less of a factor. When I finally strip it down to its bare essentials, I’ll have a better idea what gap needs to be filled. It’s starting to look like riemann at that point.

I still want to tackle this presence based idea in some form. Even if presence is just a signal to run chef-client. At my previous company, we used Noah for this. I’ve not yet had time to decide if Noah is the right fight here.

What others are building

What’s more important is what others are building. There are too many to list but I’m going to give it a shot off the top of my head. Please don’t take offense if your project isn’t here.

• Sensu
• Graphite-tattle
• Cepmon
• Logstash
• Librato
• PagerDuty
• Umpire
• alerting-controller
• Graphite
• Statsd
• Logster
• Metrics
• Incinga (yes, they’re starting to diverge from Nagios)
• ZeroMQ
• Chef
• Puppet
• Zookeeper
• Riemann
• OpenTSDB
• Ganglia
• TempoDB
• CollectD
• Datadog
• Folsom
• JMXTrans
• Pencil
• Rocksteady
• Boundary
• Circonus
• GDash

Probably the best bet is to head over to the monitoringsucks tool repo on github. I can’t do the awesomeness of what people are doing justice here.

and so many more.

Of all the problems to fix in chef or puppet, the diffusion and drift of state that occurs in idiomatic usage seems highest priority.

Now for sure what spawned this comment was something unrelated but it got me thinking. Oddly enough Tim Dysinger was either poking around in my head or just had the same idea:

Devops tools should move towards an active assertion of state (instead of passive/polling). This the next-level.

Tim and I hooked up via Skype and bantered about this stuff back and forth. We were on the same wavelength. That’s pretty cool because Tim is pretty fucking smart (and he was able to explain Maybe Monads to me over dinner).

My thoughts on the subject

What follows is something of a brain dump on what both Cliff and Tim had to say. However I’m going to be scoping in the context of security because

• Beaker gave a presentation at Gluecon today (warning! bigass pdf)
• I work with David Mortman who is one of the folks I say “gets it” w.r.t configuration management and security
• Security was the FIRST context that came to mind
• I was lucky enough to be involved with Mark Burgess, Jeff Blaine and Nick Silkey about a very similar topic where Mark said

Good point. I wonder why folks often tear down a perfectly good machine and rebuild it instead of fixing what is broken.

What I’m going to say isn’t new to anyone and smarter folks than I are already working on this (I’m sure) but this is the Internet. I get to babble too!

On Drift

So what exactly IS the problem here? What’s configuration drift and how the hell does it even happen?

The problem here is that, as Tim said, configuration management systems aren’t assertive enough. Look at how a typical CM client run behaves:

• Hey guys, cm is running
• Oh look, this file doesn’t look like it’s supposed to
• /me changes file
• File looks good
• Hey guys, cm isn’t running anymore

That last line is part of the problem. I’ve talked about my Noah project to largish groups of folks (both Puppet and Chef users) a few times now and the answer to the question of “Do you leave puppet (or chef) running in the background?” has always been “No”. There are plenty of valid reasons for this but this is what I would consider the primary cause of drift at the node level. Now maybe this isn’t EXACTLY what Cliff was talking about. I’m not quite on his level so I sometimes misinterpret but when I heard ‘drift of state that occurs in idiomatic usage’, this was what came to mind.

The thing is that these tools are designed to verify state of a resource at the point they run

• Does this file look right? No! Fix it.
• Is this service running? Yes! Cool.

And then they go away. They don’t manage the state of those resources until they next inspect them. The act of managing those resources is not in response to those resources changing but in response to a user ASKING them to be checked. I would even wager that when a user runs chef-client the first thing on her mind isn’t “I sure hope chef fixes my sudoers file” but “I need chef to update my Nagios configs again”. The incorrect state of the sudoers file isn’t really even thought about. That’s because we shove that stuff into some “base” role or group. Something that’s applied to all nodes in our infrastructure. We don’t think of a node as being a “managed sudoers” node. We think of it as a “web server”.

Additionally, because we aren’t in a constant state of verification about these resources, we may have drift that occurs across nodes of different types whilst they share a common base block. Sure I just ran my CM tool to update my Nagios server but what about my web servers? I don’t want to run it there because I KNOW nothing has changed in the web server role.

To me, this is the “idiomatic” usage Cliff spoke about. The tools encourage us to think in terms of composition and reusable patterns but the final function of the node is the way we classify it. Mind you, the answer here is really to run your CM tool in the background but that still doesn’t take us to the next level. We’re still exposed to drift even if it’s for a short period of time. What’s worse is these tools operate by default with a splay value. This actually makes the drift exposure even worse as you can’t even guarantee that it will run at the interval specified.

I first heard about CFEngine when Mark talked about “Anomoly Detection” at LISA ‘04 in Atlanta. My mind was blown but I could never get past the idea that I couldn’t dictate state immediately. The idea (partially a naive understanding on my part) that systems would not become X when I said “Become X” bothered me. The idea that systems have a personality that needs to be respected bothered me.

The point here is that when I want a system to look like X, I want it to look like X right then. I want it to STAY looking like X and I don’t want it to try and account for localized variations. I might feel differently if I were managing a network of servers that were essentially treated like desktops.

But does drift really matter?

Yes and no. If you’re living the cloud life, probably not. The reason is that resources tend to have a short shelf life. If I’m autoscaling via ‘the cloud’ to meet capacity demands then it’s highly likely that those systems won’t be around long enough to drift that far. In the land of configuration management, drift is largely time driven. The longer systems stay around, the greater the chance for drift.

However if you’re running physical hardware that you don’t tear down regularly, then drift is likely to become more pronounced. Interestingly enough there’s a psychological factor at play here. Systems become like pets instead of cattle. We become attached to them. “Oh that’s just db1 acting up again. You know it’s like the oldest one in the fleet”. People start forgetting that the system will fail (Everything fails. Embrace failure!). The start storing one-off scripts on there. Maybe it’s core kit and while it’s managed with Puppet, it’s not frequently touched.

Here’s another interesting point. As time progresses and modules, cookbooks, recipes, bundles, promises whatever are more infrequently touched, the confidence in them goes down. I’ve frequently found myself saying “Shit…I wrote that code like…I don’t even fucking remember when. I have no idea what would happen if I ran it now.”. This uncertainty eventually leads to me MANUALLY COMPARING the current state of resources that would be modified with the versions that would be generated. I’ve even copied comment blocks wholesale from one-off changes I’ve made into ERB templates just to ensure that a service restart didn’t happen.

How fucked is that? Pretty fucked, Alex.

This partially leads into a quote from John Allspaw on the dangers of OVER automation:

Some people, when confronted with a problem, think “I’ll use more automation!” Now they have Three Mile Island problems.

and is even discussed in Patrick McDonnell’s talk at ChefConf.

I don’t agree with John 100% on his take but I can totally understand his perspective. Maybe I’m just more optimistic around exactly how far we can automate.

So where does security fit into this?

Here’s yet another quote from Tim Dysinger on this topic:

If a super-user logs in and changes the sshd_config, you could have the service change it back before he even exited vi. they’d then find a warning email in their in-box. if they tried it again, even on another box, it could send a warning to the team lead and lock the user out.

Mind you security is only one aspect of this thought process. The thing that makes it applicable is that the security domain has already tackled this problem a bit. The problem is it still requires human response. We have tools like Tripwire, Samhain and OSSEC that do active inspection of state but the response is left up to a human. Additionally they’re cumbersome to configure (even with CM). What’s missing is the ‘glue’ between the two problem spaces.

In my head I envisioned something much like Tim described. In fact I even thought about a way that existing tools could be leveraged. It’s not pretty but it’s possible. The idea here, if it wasn’t already blindingly obvious, is that the response to an event that the security tool recognizes should be to run configuration management to correct the errant state.

There are a few problems with this approach that should also be immediately obvious:

• CM is currently an all or nothing approach. Something like the idea behind ‘partial run lists’ in Chef could sort of address this
• If we start down the path of partial CM, we now have to take into account dependencies.
• We have no way to express this. We lack primitives with which to build this logic
• Security is still somewhat in the dark ages. Many security decisions are still binary in nature

What’s also missing in this picture is something to identify patterns. So now we’re bolting three tools together - the tripwire component, our CM tool and some sort of CEP. But again, even if we had this wondertool nirvana, how do we express it?

Let me be clear that I firmly believe that configuration management is absolutely a part of the security story. I stand by my assertion that consistent and repeatable configuration of a system from base OS to in-service is the foundation. Being able to express in code what state a system should have means that you never have to think “Did I forget to disable that apache module?” or “Did I make sure and disable root logins over SSH”. Where we find gaps in this is how we assert the negative without going insane.

Denied unless explicitly allowed is the mantra I followed for years when I was responsible for security. Ask me some time WHY I got out of security and why I don’t have ulcers anymore.

The questions I find myself asking are:

• If we use our CM system as the source of truth, can we sanely infer policy based on our CM codebase?
• If we use the network as the source of truth, can we sanely infer policy based on our neighbors? Should we even trust our neighbors?
• Do we even have the language to express what we mean and is it flexible and primitive enough to be used in composition? You can only go so far with “trusted”, “untrusted”,”mauve” and “taupe”.

Wrap up

As I said, the original discussion was around configuration drift. I realize I went off on a tangent about security but that was intended as an example. I do believe that folks are working on this idea of “active assertion of state” as Tim puts it. I just wanted to brain dump my take on it. I don’t know that it can be solved with even the most flexible of CM tools. I do feel like it’s going to have to be a new generation of tool that takes these ideas into account and includes them in a ground-up design.

Netflix still does operations

Regardless of what words Adrian uses, Netflix still does operations. John Allspaw summed it up pretty well in this tweet:

and here are the things, he mentions:

• Metrics collection
• PaaS/IaaS evaluation/investigation
• Automation (auto-build, auto-recovery)
• Fault tolerance
• Availability
• Monitoring
• Performance
• Capex and Opex forecasting
• Outage response

So what does Adrian get wrong?

These are just a few things that jumped out at me (and annoyed me)

However, there are teams at Netflix that do traditional Operations, and teams that do DevOps as well.

Ops is ops is ops. No matter what you call it, Operations is operations.

Notice that we didn’t use the typical DevOps tools Puppet or Chef to create builds at runtime

There’s no such thing as a “DevOps tool”. People were using CFengine, Puppet and Chef long before DevOps was even a term. These are configuration management tools. In fact Adrian has even said they use Puppet in their legacy datacenter:

yet he seems to make the distinction between the ops guys there and the “devops” guys (whatever those are).

There is no ops organization involved in running our cloud…

Just because you outsourced it, doesn’t mean it doesn’t exist. Oh and it’s not your cloud. It’s Amazon’s.

Reading between the lines

Actually this doesn’t take much reading between the lines. It’s out there in plain sight:

In reality we had the usual complaints about how long it took to get new capacity, the lack of consistency across supposedly identical systems, and failures in Oracle, in the SAN and the networks, that took the site down too often for too long.

We tried bringing in new ops managers, and new engineers, but they were always overwhelmed by the fire fighting needed to keep the current systems running.

This is largely because the people making decisions are development managers, who have been burned repeatedly by configuration bugs in systems that were supposed to be identical.

The developers used to spend hours a week in meetings with Ops discussing what they needed, figuring out capacity forecasts and writing tickets to request changes for the datacenter.

There is no ops organization involved in running our cloud, no need for the developers to interact with ops people to get things done, and less time spent actually doing ops tasks than developers would spend explaining what needed to be done to someone else.

I’m glad to see this spelled out in such detail. This is what I’ve been telling people semi-privately for a while now. Because Netflix had such a terrible experience with its operations team, they went to the opposite extreme and disintermediated them.

Imagine you were scared as a kid by a clown. Now imagine you have kids of your own. You hate clowns. You had a bad experience with clowns. But it’s your kid’s birthday party so here you are making baloon animals, telling jokes and doing silly things to entertain the kids.

Just because you aren’t wearing makeup doesn’t make you any less of a clown. You’re doing clown shit. Through the eyes of the kids, you’re a clown. Deal with it.

Netflix is still doing operations. What should be telling and frightening to operations teams everywhere is this:

The Netflix response to poorly run operations that can’t service the business is going to become the norm and not the exception. Evolve or die.

Please note that I don’t lay all the blame on the Netflix operations team. I would love to hear the flipside of this story from someone who was there originally when the streaming initiative started. It would probably be full of stories we’ve heard before - no resources, misalignment of incentives and a whole host of others.

Adrian, thank you for writing the blog post. I hope it serves as a warning to those who come. Hopefully someday you’ll be able to see a clown again and not get scared ;)

I’ve addressed this topic a bit before here. At the time it was addressing specifically dynamic languages. However the post that Kohsuke wrote (and the post that inspired it) have led me to a new line attitude.

Don’t bother trying to get your packages into upstream vendor distros

Wait. What? Let’s step back a sec

Let me clarify something first. System packages are a good thing. The hassle has always been with BUILDING those packages. It was simply easier to build the software on the machine and install to /usr/local/ than to try and express anything more than the most moderately simple application in RPM or DEB build scripts:

• If what you are packaging has dependencies not shipped with the OS, now you’ve got to package those
• If your dependency conflicts with a vendor-shipped version, you’re screwed.
• If your dependency is a language runtime, give up.
• If your dependency is a specific version of python, just go into another line of work.
• If it’s a distro LTS release, just don’t bother

Ahh but we can work around this!

Yes, you’re right. We now have tools like fpm that take the pain out of it! Maven has had plugins that generate rpms and debs for you for a while now. Things are looking up! Let’s just use those tools.

So now you think, I’ll just get these things submitted to Debian….

KABLOCK

I could rant a bit about Debian’s packaging policy but it’s addressed in the posts above. So maybe the Fedora people are more flexible?

WAT

So here we have the two major distros that won’t even consider your package unless you give the end-user the “freedom” to make your application unusable. Essentially you are told if you want your package to be included in upstream then you have to make sure they can swap out libfunkytown.so.23 with libfunkytown.so.1.

But maybe your application doesn’t work on that version. So maybe you think, I’ll just vendor ALL the things and shove it into /opt or /usr/local? Yeah that doesn’t fly either (for various reasons).

The point is that you’ll probably never be able to get your package included upstream because you’ll never be able to jump through the hoops to do it.

So stop trying

I know, I know. It would be awesome if you could tell users to just yum install kickass or apt-get install kickass but it’s not worth it for several reasons as enumerated above.

Distributions are not your friend. One could argue that its not thier job to be your friend. I would even agree with that argument. The distros have (or at least SHOULD have) an allegience to their user base. My argument is that position is directly opposed to your needs as a software provider.

Things you should not do

• Waste your time trying to ensure that your software works on some busted as old version of libfunkytown that won’t get upgrade for 7 years.
• Waste your time breaking your application into 436 interdependent subpackages just to please upstream
• Ignore the prexisting dependency management ecosystem of your language of choice (especially if it works)

Things you should do

• Use your language’s preexisting dependency management system to collect all your dependencies
• Rebar, bundle, virtualenv, mavenize, fatjar whatever ALL the dependencies
• Use FPM or some homegrown script to create a monolithic rpm or deb of your codebase that installs to /opt/appname
• Make these packages available to your users on your download site
• Alternately, create a repo and repo config file they can use to stay up to date

You will be happy. Your users will be happy. The distros can go lick themselves. We have reached something of a crossroads. As I argued in the previous post, the concept of a distribution is becoming somewhat irrelevant. Distros are more concerned about politics and making statements and broken concepts like software that doesn’t need upgrading for 7 years (or even 2 years) than providing a framework and ecosystem that encourages developers to target software at it.

If someone takes up the noble cause of trying to get your software included upstream, I would go so far as to make it plainly clear on whatever communication you have that you simply cannot support an unofficial repackaging of your software. Be polite. These are still your potential userbase. Simply state that those were not created by you and that the official packages are here.

A case in point

What I’m suggesting you do is not unheard of and honestly is the most tenable long term path for your users. Look at projects like Vagrant, Chef and Puppet among others. All of these tools are “owning their availability” the right way and are arguably providing better end user experiences than getting included in upstream could provide. In fact the experience of official packaging is above and beyond trying to do it yourself. As it should be.

please note that I’m drifiting into deeply unfamiliar territory for me. Someone once told me the best way to learn about something is to write about it. Keep that in mind when making comments?

One of the more interesting parts of maven is the dependency graph and concepts like transitive and (god forbid) circular dependencies. These problems aren’t exlcusive to java, mind you. See bundler for Ruby.

A bit on graphs

Graph is a fairly overloaded term. In the context of this discussion I’m talking about graph theory (insofar as I can grok it). Specifically I want to talk about it in the context of IT operations.

Graphs are nothing “new”. Programmers have binary trees. Network geeks have OSPF. Puppet and Git are fans of the DAG (directed acyclic graph). These are all rooted in the same place no? You have nodes and edges. It’s math all the way down. Unfortunately I suck at math.

If the topic interests you at all, wikipedia has a good couple of articles worth reading. Seeing as I’m far from a domain expert, I can’t vouch for the quality:

• Graph Theory
• Graph (mathematics))
• Glossary of graph theory

How can graphs apply to IT operations

I’ve said for a while now that I feel like there’s something fuzzy on the horizon that I can’t make quite make out and it involves orchestration and graphs. I’m still not clear on how to express it but I’ll try.

Anyone who has ever used Puppet or Git has dabbled in graphs even if they don’t know it. However my interest in graphs in operations relates to the infrastructure as a whole. James Turnbull expressed it very well last year in Mt. View when discussion orchestration. Obviously this is a topic near and dear to my heart.

Right now much of orchestration is in the embryonic stages. We define relationships manually. We register watches on znodes. We define hard links between components in a stack. X depends on Y depends on Z. We’re not really being smart about it. If someone disagrees, I would LOVE to see a tool addressing the space.

Justin Sheehy did an awesome high level presentation on distributed systems, databases and the like at Velocity last year. While the talk was good, one thing that stuck out with me was his usage of the Riak logo:

During the presentation he would zoom out of the logo and replace it with the same logo. It expressed the idea of moving up the stack. Macro versus micro. I have the same feeling about where orchestration is going.

Express yourself

Currently we do a great job (and have the tools) to express relationships and dependencies at the node level:

• webapp needs container
• container needs java
• container needs system user

Going a level higher, we even have some limited ability to express relationships between nodes:

• Load balancer needs app servers
• App server needs database

We’re not quite as good at this part yet but people have workarounds. I use Noah for this. enStratus also handles this very well.

But we’re still defining those relationships manually.

When we get to this next level up, things get REALLY fuzzy. As people start to (re)discover SOA, we now have stacks that have dependencies on other stacks. Currently we use tools like Zookeeper to broker that relationship. But we still have to explcitly manage it.

The level of coupling here isn’t the problem. You can mitigate failure in one stack as it relates to another stack. Fail fast and fall back to sane/safe defaults. Read any article about how Netflix architects to get an idea.

What’s missing?

What I feel like we’re missing is a way to express those relationships and then trigger on them all the way up and down the chain as needed. We’re starting to get into graph territory here.

We must we be able to express and act on changes at the micro level (I changed a config, I must restart nginx) and even at the intranode level (something changed in my app tier, need to tell my load balancer) but now we need a way handle it at that macro level. Not only do we need a way to handle it but we must also be able to calculate what is impacted by that change.

• If I have this internode change, does it affect the intranode relationship?
• If I have an intranode change, does it affect the intrastack relationship?

It seems to me that a graph of SOME kind is the best way to express this. I just can’t quite make it out. Does current graph technology even handle that subgraph relationship? Excuse the pun but where do we draw the line? Are there multiple lines?

Maybe this isn’t an issue. Maybe through resilience engineering we simply keep that “intrastack” dependency as loose as possible so that we don’t have this problem?

I’ve had a lot of people ask me what’s so hot about ZeroMQ. It’s hard to explain but I really would suggest you read the excellent zguide. The best way I can describe it is that it’s sockets on steroids. Sockets that behave the way you would expect sockets to behave as opposed to the way they do now. Just open a socket!.

Inputs and Outputs

I’m only going to touch briefly on inputs and outputs. They were discussed briefly previously and I have a full fledged post in the wings about it.

They essentially work like the other implementations (AMQP and Redis) with the exception that you don’t have a broker in the middle. Let me show you:

[Collector 1] ------ load balanced events ----> [Indexer 1, Indexer 2, Indexer 3, Indexer 4]
[Collector 2] ------ load balanced events ----> [Indexer 1, Indexer 2, Indexer 3, Indexer 4]
[Collector 3] ------ load balanced events ----> [Indexer 1, Indexer 2, Indexer 3, Indexer 4]
[Collector 4] ------ load balanced events ----> [Indexer 1, Indexer 2, Indexer 3, Indexer 4]

As you can see we’re doing a pattern very similar to before. We want to send events of our nodes over to a cluster of indexers that do filtering. The difference here is that we don’t have a broker. Not big deal, right? One less thing to worry about! You don’t have to learn some new tool just to get some simple load balancing of workers. This works great…..until you need to scale workers.

Even using awesome configuration management, you’ve now got to cycle all your collectors to add the new endpoints. This means lost events. This makes me unhappy. It makes you unhappy. The world is sad. Why are you doing this to us?

Luckily I’ve been authorized by the Franklin Mint to release the source code to an enterprise class ZeroMQ broker that you can use. Not only is it enterprise class but it has built-in clustering. You can grab the code here from github.

Here are the configs for the logstash agents (output.conf is collector config, input.conf is indexer config):

output.conf:

1
2
3
4
5
6
7

input { stdin { type => "stdin" } }
output {
  zeromq {
    topology => "pushpull"
    address => ["tcp://localhost:5555", "tcp://localhost:5557"]
  }
}

input.conf:

1
2
3
4
5
6
7
8
9

input { 
  zeromq {
    type => "pull-input"
    topology => "pushpull"
    address => ["tcp://localhost:5556", "tcp://localhost:5558"]
    mode => "client"
  }
}
output { stdout { debug => true }}

Action shot

Here’s a shot of our fancy clustered broker in action (click to zoom):

As you can see the two events we sent were automatically load balanced across our “brokers” which then load balanced across our indexers.

What have we bought ourselves?

Obviously this is all something of a joke. All we have done is point our collectors at other nodes instead of directly at our indexers. But realize that you can create 2 fixed points on your network with 8 lines of core code and use those as the static information in your indexers and collectors. You can then scale either side without ever having to update a configuration file.

I dare say you can even run those on t1.micro instances on Amazon.

Oh and if you don’t like Ruby, write it in something else. That’s the beauty of ZeroMQ.

Filters

The thing that has me most excited is the addition of ZeroMQ as a filter to logstash. As you’ve already seen, ZeroMQ makes it REALLY easy to wire network topologies up with complex patterns. In the inputs and outputs we’ve exposed a few topologies that make sense. However there’s another topology that we had not yet exposed because it didn’t make sense - reqrep.

REQ/REP

reqrep is short for request and reply. The reason we didn’t expose it previously is that it didn’t really make sense with the nature of inputs and outputs. However after talking with Jordan, we decided it actually DID make sense to use it for filters. After all, filters get a request -> do something -> return a response.

If it’s not immediately clear yet how this makes sense, I’ve got another example for you. Let’s take the case of needing to look something up externally to mutate a field. You COULD write a Logstash filter to do this ONE thing for you. Maybe you can make it generic enough to even submit a pull request.

Or you could use a ZeroMQ filter:

1
2
3

input { stdin { type => "stdin-type" } }
filter { zeromq { } }
output { stdout { debug => true } }

Here’s the code for the filter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

require 'rubygems'
require 'ffi-rzmq'
require "json"

context = ZMQ::Context.new
socket = context.socket(ZMQ::REP)
socket.bind("tcp://*:2121")
msg = ''
puts "starting up"
while true do
  socket.recv_string(msg)
  modified_message = JSON.parse(msg)
  puts "Message received: #{msg}"
  # Simulate using an external data source to 
  # to something that you need
  case modified_message["@source"]
  when "stdin://jvstratusmbp.lusis.org/"
    puts "Doing db lookup"
    sleep 10
    modified_message["@source"] = "john's laptop"
  end
  puts "Message responded: #{modified_message.to_json}"
  socket.send_string(modified_message.to_json)
end

By default, the filter will send the entire event over a ZeroMQ REQ socket to tcp://localhost:2121. It will then take the reply and send it up the chain to the Logstash output with the following results:

Alternately, you can send a single field to the filter and have it to work with:

1
2
3

input { stdin { type => "stdin-test" } }
filter { zeromq { field => "@message" } }
output { stdout { debug => true }}

and the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

require 'rubygems'
require 'ffi-rzmq'
require "json"

context = ZMQ::Context.new
socket = context.socket(ZMQ::REP)
socket.bind("tcp://*:2121")
msg = ''
puts "starting up"
while true do
  socket.recv_string(msg)
  puts "Recieved message: #{msg}"
  modified_message = "this field was changed externally"
  puts "Modified message: #{modified_message}"
  socket.send_string(modified_message)
end

and the result:

Many people have been asking for an exec filter for some time now. Dealing with that overhead is insane when coming from the JVM. By doing this type of work over ZeroMQ, there’s much less overhead AND a reliable conduit for making it happen.

Here’s just a few of the use cases I could think of:

• Artifically throttling your flow. Just use a sleep and return the original event.
• Doing external lookups for replacing parts of the event
• Adding arbitrary tags to a message using external criteria based on the event.
• Moving underperforming filters out of logstash and into an external process that is more performant
• Reducing the need to modify configs in logstash for greater uptime.

Wrap up

All the ZeroMQ support is currently tagged experimental (hence the warnings you saw in my screenshots). It also exists in the form described only in master. If this interests you at all, please build from master and run some tests of your own. We would love the feedback and any bugs or tips you can provide are always valuable.

Edit and update

Evidently my testing missed a pretty critical usecase - pubsub. It doesn’t work right now. Due to the way we’re doing sockopts works for setting topics. However we don’t have a commensurate setting on the PUB side. I’ve created LOGSTASH-399 and LOGSTASH-400 to deal with these issues. I am so sorry about that however it doesn’t change the overall tone and content of this message as pair and pushpull still work.

A little history

In January of this year, Jordan merged the first iteration of ZeroMQ support for Logstash. Several people had been asking for it and I had it on my plate to do as well. Funny side note, the pull request for the ZeroMQ plugin was my inspiration for adding plugin_status to Logstash.

The reason for wanting to mark it experimental is that there was concern over the best approach to using ZeroMQ with Logstash. Did we create a single context per agent? Did we do a context per thread? How well would the multiple layers of indirection work (jvm + ruby + ffi)?

Brice’s original pull request only hadnled one part of the total ZeroMQ package (PUBSUB) but it was an awesome start. We actually had two other pull requests around the same time but his was first.

A week or so ago, I started a series of posts around doing load balanced filter pipelines with Logstash. The first was AMQP and then Redis. The next logical step was ZeroMQ (and something of a “Oh..and one more thing..” post). Sadly, the current version of the plugin was not amenable to doing the same flow. Since it only supported PUBSUB, I needed to do some work on the plugin to get the other socket types supported. I made this my weekend project.

Something different

One thing that ZeroMQ does amazingly well is make something complex very easy. It exposes common communication patterns over socket types and makes it easy to use them. It really is just plug and play communication.

However it also makes some really powerful flows available to you if you dig deep enough. Look at this example from the zguide

Mind you the code for that is pretty simple (ruby example) but we need to enable that level of flexibility and power behind the Logstash config language. We also wanted to avoid the confusion that we faced with the AMQP plugin around exchange vs. queue.

Jordan came up with the idea of removing the socket type confusion and just exposing the patterns. And that’s what we’ve done.

Configuration

In the configuration language, Logstash exposes the ZeroMQ socket type pairs in the using the same syntax on both inputs and outputs. We call these a “topology”. In fact, out of the box, Logstash ZeroMQ support will work out of the box with two agents on the same machine:

Output

1
2
3
4
5
6

input {
  stdin { type => "stdin-input" }
}
output {
  zeromq { topology => "pushpull" }
}

Input

1
2
3
4
5
6

input {
  zeromq { topology => "pushpull" type => "zeromq-input" }
}
output {
  stdout { debug => true }
}

Opinionated

Because any side of a socket type in ZeroMQ can be the connecting or binding side (the underlying message flow is disconnected from how the connection is established), Logstash follows the recommendation of the zguide. The more “stable” parts of your infrastructure should be the side that binds/listens while they ephemeral side should be the one that initiates connections.

Following this, we have some sane defaults around the plugins:

• Logstash inputs will, by default, be the bind side and bind to all interfaces on port 2120
• Logstash outputs will, by default, be the connect side
• Logstash inputs will be the consumer side of a flow
• Logstash outputs will be the producing side of a flow

The last two are obviously pretty “duh” but worth mentioning. Right now Logstash exposes three socket types that make sense for Logstash:

• PUSHPULL (Output is PUSH. Input is PULL)
• PUBSUB (Output is PUB. Input is SUB)
• PAIR

It’s worth reading up on ALL the socket types in ZeroMQ.

By default, because of how ZeroMQ will most commonly be slotted into your pipeline, it sets the default message format to the Logstash native json_event.

You can still get to the low-level tuning of the sockets via the sockopts configuration setting. This is a Logstash config hash. For example, if you wanted to tune the high water mark of a socket (ZMQ_HWM), you would do so with this option:

zeromq { topology => "pushpull" sockopts => ["ZMQ::HWM", 20] }

These options are passed directly to the ffi-rzmq library we use (hence the syntax on the option name). If a new option is added in a later release, it’s already available that way.

Usage of each topology

While I have a few more blog posts in the hopper around ZeroMQ (and various patterns with Logstash), I’ll briefly cover where each type might fit.

PUBSUB

This is exactly what it sounds like. Each output (PUB) broadcasts to all connected inputs (SUB).

PUSHPULL

This most closely mimics the examples in my previous posts on AMQP and Redis. Each output (PUSH) load-balances across all connected inputs (PULL).

PAIR

This is essentially a one-to-one streaming socket. While messages CAN flow both directions, Logstash does not support (nor need) that. Outputs stream events to the input.

ZeroMQ has other topologies (like REQREP - request response and ROUTER/DEALER) but they don’t really make sense for Logstash right now. For the type of messaging that Logstash does between peers, PAIR is a much better fit. We have plans to expose these in a future release.

Future

As I said, I’ve got quite a few ideas for posts around this plugin. It opens up so many avenues for users and makes doing complex pipelines much easier. Here’s a sample of some things you’ll be able to do:

• Writing your own “broker” to sit between edges and indexers in whatever language works best (8 lines of Ruby)
• Log directly from your application (e.g. log4j ZMQ appender) to logstash with minimal fuss
• Tune ZeroMQ sockopts for durability

Current ZeroMQ support only exists in master right now. However building from source is very easy. Simply clone the repo and type make. You don’t even need to have Ruby installed. This will leave your very own jar file in the build directory.

Logstash is a unix pipe on steroids

I hope this post helps you understand what I meant by that

Revisiting our requirements and pattern

If you recall from the post yesterday, we had the following ‘requirements’:

• No lost messages in transit/due to inputs or outputs.
• Shipper only configuration on the source
• Worker based filtering model
• No duplicate messages due to transit mediums (i.e. fanout is inappropriate as all indexers would see the same message)

EDIT

Originally our list stated the requirements as No lost messages and No duplicate messages. I’ve amended those with a slight modification to closer reflect the original intent. Please see comment from Jelle Smet here for details. Thanks Jelle!

Our design looked something like this:

One of the reasons that post was so long was that AMQP is a complicated beast. There was quite a bit of dense frontloading I had to do to cover AMQP before we got to the meat. We’re going to take that same example, and swap out RabbitMQ for something a bit simpler and achieve the same results.

Quick background on Redis

Redis is commonly lumped in with a group of data storage technologies called NoSQL. Its name is short for “REmoteDIctionaryServer”. It typically falls into the “key/value” family of NoSQL. Several things set Redis apart from most key/value systems however:

• “data types” as values
• native operations on those data types
• atomic operations
• built-in PUB/SUB subsystem
• No external dependencies

Data types

I’m not going to go into too much detail about the data types except to list them and highlight the one we’ll be leveraging. You can read more about them here

• Strings
• Lists*
• Sets
• Hashes
• Sorted Sets

How Logstash uses Redis

Looking back at our AMQP example, we note three distinct exchange types. These are mapped to the following functionality in Redis (and Logstash data_type config for reference):

This is a somewhat over simplified list. In the case of a message producer, mimicing direct exchanges is done by writing to a Redis list while consumption of that is done via the Redis command BLPOP*. However mimicing the fanout and topic functionality is done strictly with the commands PUBLISH*, SUBSCRIBE* and PSUBSCRIBE*. It’s worth reading each of those for a better understanding.

Oddly enough, the use of Redis as a messaging bus is something of a side effect. Redis supported lists that are auto-sorted by insert order. The POP command variants allowed single transaction get and remove of the data. It just fit the use case.

The configs

As with our previous example, we’re going to show the configs needed on each side and explain them a little bit.

Client-side/Producer

1
2
3
4
5
6
7
8

input { stdin { type => "producer"} }
output {
redis {
 host => 'localhost'
 data_type => 'list'
 key => 'logstash:redis'
}
}

data_type

This is where we tell Logstash how to send the data to Redis. In the case, again, we’re storing it in a list data type.

key

Unfortunately, key means different things (though with the same effect) depending on the data_type being used. In the case of a list this maps cleanly to the understanding of a key in a key/value system. It’s common in Redis to namespace keys with a : though it’s entirely unneccesary.

As an aside, when using key on channel data type, this behaves like the routing key in AMQP parlance with the exception of being able to use any separator you like (in other words, you can namespace with .,:,:: whatever).

Indexer-side/Consumer

1
2
3
4
5
6
7
8
9

input {
redis {
  host => 'localhost'
  data_type => 'list'
  key => 'logstash:redis'
  type => 'redis-input'
}
}
output {stdout {debug => true} }

data_type

This needs to match up with the value from the output plugin. Again, in this example list.

key

In the case of a list this needs to map EXACTLY to the output plugin. Following on to our previous aside, for data_type values of channel input, the key must match exactly while pattern_channel can support wildcards. Redis PSUBSCRIBE wildcards actually much simpler than AMQP ones. You can use * at any point in the key name.

Starting it all up

We’re going to simplify our original tests a little bit in the interest of brevity. Showing 2 producers and 2 consumers gives us the same benefit as showing four of each. Since we don’t have the benefit of a pretty management interface, we’re going to use the redis server debug information and the redis-cli application to allow us to see certain management information.

redis-server

Start the server with the command redis-server I’m running this from homebrew but you literally build Redis on any machine that has make and a compiler. That’s all you need. You can even run it straight from the source directory:

You’ll notice that the redis server is periodically dumping some stats - number of connected clients and the amount of memory in use.

Starting the logstash agents

We’re going to start two producers (redis output) and two consumers (redis input):

Back in our redis-server window, you should now see two connected clients in the periodic status messages. Why not four? Because the producers don’t have a persistent connection to Redis. Only the consumers do (via BLPOP):

Testing message flow

As with our previous post, we’re going to alternate messages between the two producers. In the first producer, we’ll type window 1 and in the second window 2. You’ll see the consumers pick up the messages:

If you look over in the redis-server window, you’ll also see that our client count went up to four. If we were to leave these clients alone, eventually it would drop back down to two.

Feel free to run the tests a few times and get a feel for message flow.

Offline consumers

This is all well and good but as with the previous example, we want to test how this configuration handles the case of consumers going offline. Shut down the two indexer configs and let’s verify. To do this, we’re going to also open up a new window and run the redis-cli app. Technically, you don’t even need that. You can telnet to the redis port and just run these commands yourself. We’re going to use the LLEN command to get the size of our “backlog”.

In the producer windows, type a few messages. Alternate between producers for maximum effect. Then go over to the redis-cli window and type LLEN logstash:redis. You should see something like the following (obviously varied by how many messages you sent):

You’ll also notice in the redis server window that the amount of memory in use went up slightly.

Now let’s start our consumers back up and ensure they drain (and in insert order):

Looks good to me!

Persistence

You might have noticed I didn’t address disk-based persistence at all. This was intentional. Redis is primarily a memory-based store. However it does have support for a few different ways of persisting to disk - RDB and AOF. I’m not going to go into too much detail on those. The Redis documentation does a good job of explaining the pros and cons of each. You can read that here.

Wrap up

One thing that’s important to note is that Redis is pretty damn fast. The limitation for Redis is essentially memory. However if speed isn’t your primary concern, there’s an interesting alpha project called edis worth investigating. It is a port of Redis to Erlang. Its primary goal is better persistence for Redis. For this post I also tested Logstash against edis and I’m happy to say it works:

I hope to do further testing with it in the future in a multinode setup.

Part three

I’m also working on a part three in this “series”. The last configuration I’d like to show is doing this same setup but using 0mq as the bus. This is going to be especially challenging since our 0mq support is curretly ‘alpha’-ish quality. Beyond that, I plan on doing a similar series using pub/sub patterns. If you’re enjoying these posts, please comment and let me know!

Patrick DeBois hit me up with a common logstash design pattern that I felt warranted a full detailed post.

Warning: This is an image heavy post. Terminal screenshots are linked to larger versions

Requirements

• No lost messages in transit/due to inputs or outputs.
• Shipper-only configuration on the source
• Worker-based filtering model
• No duplicate messages due to transit mediums (i.e. fanout is inappropriate as all indexers would see the same message)
• External ElasticSearch cluster as final destination

EDIT

Originally our list stated the requirements as No lost messages and No duplicate messages. I’ve amended those with a slight modification to closer reflect the original intent. Please see comment from Jelle Smet here for details. Thanks Jelle!

Notes

We’re going to leave the details of filtering and client-side input up to the imagination. For this use case we’ll simply use stdin as our starting point. You can modify this as you see fit. The same goes for filtering. The assumption is that your filters will be correct and not be the source of any messages NOT making it into ElasticSearch.

Each configuration will be explained so don’t stress over it at first glance. We’re also going to explicitly set some options for the sake of easier comprehension.

Client-side agent config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

{
  input {
    stdin { debug => true type => "host-agent-input" }
  }
  output {
    amqp {
      name => "logstash-exchange"
      exchange_type => "direct"
      host => "rabbitmq-server"
      key => "logstash-routing-key"
      durable => true
      persistent => true
    }
  }
}

Config Explained

The amqp output:

name

This is the name that will be provided to RabbitMQ for the exchange. By default, the Bunny driver will auto-generate a name. This won’t work in this usecase because the consumers will need a known name. Remember exchanges are for producers. Queues are for consumers. When we wire up the indexer side, we’ll need to know the name of the exchange to perform the binding.

exchange_type

For this particular design, we want to use a direct exchange. It’s the only way we can guarantee that only one copy of a log message will be processed.

key

We’re going to explicitly set the routing key as direct exchanges do not support wildcard routing key bindings. Again, we’ll need this on the consumer side to ensure we get the right messages.

durable

This setting controls if the exchange should survive RabbitMQ restarts or not.

persistent

This is for the messages. Should they be persisted to disk or not?

Note that for a fully “no lost messages scenario” to work in RabbitMQ, you have to jump through some hoops. This is explain more below.

Running the agent

This same configuration should be used on ALL host agents where logs are being read. You can have variation in the inputs. You can have additional outputs however the amqp output stanza above will ensure that all messages will be sent to RabbitMQ.

Indexer agent config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

input {
  amqp {
    host => "rabbitmq-server"
    name => "indexer-queue"
    exchange => "logstash-exchange"
    key => "logstash-routing-key"
    exclusive => false
    durable => true
    auto_delete => false
    type => "logstash-indexer-input"
  }
}

filter {
  # your filters here
}

output {
  elasticsearch {
    # your elasticsearch settings here
  }
}

Config explained

The amqp input:

name

This is the name that will be provided to RabbitMQ for the queue. Again, as with exchange, we need a known name. The reason for this is that all of our indexers are going to share a common queue. This will make sense in a moment.

exchange

This should match exactly with the name of the exchange that was created before in the host-side config.

key

This should, again, match the routing key provided in the host-side configuration exactly. direct exchanges do NOT support wildcard routing keys. By providing a routing key, you are creating a binding in RabbitMQ terms. This binding says “I want all messages sent to the logstash-exchange with a routing key of logstash-routing-key to be sent to the queue named indexer-queue.

exclusive

As with the exchange in the host-side config, we’re going to have multiple workers using this queue. This is another AMQP detail. When you bind a queue to an exchange, a channel is created for the messages to flow across. A single queue can have multiple channels. This is how our worker pool is going to operate.

You do not want a different queue name for each worker despite how weird that sounds

If you give each worker its own queue, then you WILL get duplicate messages. It’s counterintuitive, I know. Just trust me. The way to ensure that multiple consumers don’t see the same message is to use mutliple channels on the same queue.

durable

Same as the exchange declarition, this ensures that the queue will stick around if the broker (the RabbitMQ server) restarts.

auto_delete

This is the setting most people miss when trying to ensure no lost messages. By default, RabbitMQ will throw away even durable queues once the last user of the queue disconnects.

type

This is the standard logstash requirement for inputs. They must have a type defined. Arbitrary string.

Sidebar on RabbitMQ message reliability

Simply put, RabbitMQ makes you jump through hoops to ensure that no message is lost. There’s a trifecta of settings that you have to have for it to work:

• Your exchange must be durable with persistent messages
• Your queue must be durable
• Auto-delete must not be disabled

EVEN IF YOU DO ALL THESE THINGS, YOU CAN STILL LOSE MESSAGES!

Order matters

I know … you’re thinking “What the F—?”. There is still a scenario where you can lose messages. It has to do with how you start things up.

• If you start the exchange side but never start the queue side, messages are dropped on the floor
• You can’t start the queue side without first starting the exchange side

While RabbitMQ let’s you predeclare exchanges and queues from the command-line, it normally only creates things when someone asks for it. Since exchanges know nothing about the consumption side of the messages (the queues), creating an exchange with all the right settings does NOT create the queue and thus no binding is ever created.

Conversely, you can’t declare a totally durable queue when there is no exchange in place to bind against.

Follow these rules and you’ll be okay. You only need to do it once:

• Start a producer (the host-side logstash agent)
• Ensure via rabbitmqctl or the management web interface that the exchange exists
• Start one of the consumers (the indexer config)

Once the indexer agent has started, you will be good to go. You can shutdown the indexers and messages will start piling up. You can shut everything down - rabbitmq (with backlogged messages), the indexer agent and the host-side agent. When you start RabbitMQ, the queues, exchanges and messages will all still be there. If you start an indexer agent, it will drain the outstanding messages.

However, if you screw the configuration up you’ll have to delete the exchange and the queue via rabbitmqctl or the management web interface and start over.

How it looks visually

There are two plugins you should install with RabbitMQ:

• rabbitmq_management
• rabbitmq_management_visualizer

The first will provide a web interface (and HTTP API!) listening on port 55672 of your RabbitMQ server. It provides a really easy way to see messages backlogged, declared exchanges/queue and pretty much everything else. Seeing as it also provides a very nice REST api to everything inside the RabbitMQ server, you’ll want it anyway if for nothing but monitoring hooks.

The visualizer is an ad-hoc addon that helps you see the flows through the system. It’s not as pretty as the management web interface proper but it gets the job done.

Starting it all up

Now we can start things up

Producers

We’re going to start up our four client side agents. These will create the exchange (or alternately connect to the existing one). If you look at the management interface, you’ll see four channels established:

Management view:

Visualizer view:

Remember that until we connect with a consumer configuration (the indexer) messages sent to these exchanges WILL be lost.

Consumers

Now we start our indexer configurations - all four of them

Now if we take a peek around the management interface and the visualizer, we start to see some cool stuff.

In the managment interface, you’ll see eight total channels - four for the queue and four for the exchange

If you click on “Queues” at the top and then on the entry for our indexer-queue, you’ll see more details:

But the real visual is in the visualizer tab. Click on it and then click on the indexer-queue on the far right

You can see the lines showing the flow of messages.

One thing to make note of about RabbitMQ load balancing. Messages are load balanced across CONSUMERS not QUEUES. There’s a subtle distinction there from RabbitMQ’s semantic point of view.

Testing the message flow

Over in your terminal window, let’s send some test messages. For this test, again, I’m using stdin for my origination and stdout to mimic the ElasticSearch destination.

In my first input window, I’m going just type 1 through 4 with a newline after each. This should result in each consumer getting a message round-robin style:

Now I’m going to cycle through the input windows and send a single message from each:

You can see that messages 4-7 were sent round-robin style.

Testing persistence

All of this is for naught if we lose messages because our workers are offline. Let’s shutdown all of our workers and send a bunch of messages from each input window:

We sent two lines of text per window. This amounts to eight log messages that should be queued up for us. Let’s check the management interface:

Now if we stop rabbitmq entirely and restart it, those messages should still be there (along with the queue and exchanges we created).

Once you’ve verified that, start one of the workers back up. When it comes fully online, it should drain all of the messages from the exchange:

Yep, there they went. The last two messages you get should be the ones from window 4. This is another basic functionality of message queue software in general. Messages should be delivered in the order in which they were recieved.

One last diagram

Here’s a flowchart I created with Gliffy to show what the high-level overview of our setup would look like. Hope it helps and feel free to hit me up on freenode irc in the #logstash channel or on twitter.

This post will eventually make its way into the Logstash Cookbook Site.

I tweeted about this scenario not long after it happened but here’s the gist:

I just need something simple to check on the status of a few jobs and run some SQL statements. I’m a DBA and I can’t get any help from my ops team.

The person who asked this was very friendly and I could sense the frustration in her voice. It frustrates me to no end to hear stories of my tribe being this way to customers.

I thought for a minute because I really wanted to help and the best thing I could think of was Jenkins. Yes, Jenkins.

Reasoning

Let’s look for a minute at what we need from a simple health check system:

• Performing some task on a given schedule
• Ability to run a given command
• Reporting on the output of the given command (success/failure)

Now you might think to your self “Self, this sure does sound a lot like cron”. You’d be right. And that’s EXACTLY what took me down the Jenkins path. There have been numerous posts about people replacing individual cron jobs with a centralized model based on Jenkins. This makes perfect sense and is something of a holy grail. I clearly remember researching and evaluating batch scheduling products many years ago to essentially do just this. If only Jenkins had been around then.

Small disclaimer

While Jenkins is a great low friction way to accomplish this task, it may or may not be scalable in the long run. While Jenkins jobs are defined as XML files and can be managed via an API, it’s still a bit cumbersome to automate.

Getting started

First thing to do is grab the latest Jenkins war. The nice thing about Jenkins is that it ships in such an easy to use format - a self-contained executable war. You can start it very simply with:

1

java -jar jenkins.war

You should probably click around to get comfortable with the interface. It’s pretty difficult to screw something up but if you do, just shutdown jenkins, rm -rf ~/.jenkins and start it back up.

Since this post is geared primarily at someone who probably isn’t familiar with Jenkins, I’m going to go over a few quick basics and key areas we’ll be working with:

Menus

The menu is the section on the left handside. It will change based on your location in the application. If you don’t always see something you’re expecting, you can use the breadcrumb navigation to work your way back. Alternately, you can click on the Jenkins logo to get to the main page.

Main menu

This is the menu you see from the top-level of the Jenkins interface

Job menu

This is the menu you see when you are viewing the main page of a job

Note the “Build History” section at the bottom. This is a list of all builds that have been performed for this job. You can click on a given build to see details about it.

Build menu

This menu is visible when you select a specific build from the “Build History” menu of a Job page

Notice the “Console Output” menu option. This will show you the log of what Jenkins did during a build. If you ever have problems with a build, you should come here and look at what happened.

Auto Refresh

In the interest of eliminating any confusion, we’re going to enable “Auto Refresh” from the link on the top right:

Configuration

For the purposes of this exercise, we won’t do too much configuration. We’re going to take the perspective of the person above. We’ll make few assumptions though in the interest of expidiency:

• The user has no passphrase on the SSH key. While this is probably not true, it makes this demo easier.
• The DB test will be executed locally.
• The local environment is some unix-y/linux-y one. The environment for this post was OS X communicating with Linux VMs

The key to success here is something called a “free-style software project”. This is essentially a blank canvas with very few requirements. I’m aware that the “Monitor an external job” type has been recently added but the steps were a bit too invasive for this particular case.

Our test case

I don’t obviously have the specifics of what the user wanted checked so I’m going to extrapolate from her original statement:

• Run a SQL statement to see if some record is found
• Check for a running process
• Check a log file for some given string

The test database

The test database will be MySQL running on a Linux VM. Getting this going is an exercise for the reader, however here is the DDL and test data we’re using:

1
2
3
4
5
6
7
8
9

--- This is just a sample, folks. Yes I know it's insecure.
create database foo_db;
use foo_db;
create table jobs ( id int not null auto_increment primary key, name varchar(10), depth int);
insert into jobs (name, depth) values ("job_a", 100);
insert into jobs (name, depth) values ("job_b", 0);
insert into jobs (name, depth) values ("job_c", 5);
grant select on jobs to 'jenkins'@'%' IDENTIFIED BY 'password';
flush privileges;

First Job

So we’ll create a new freestyle job called “Check FooDB Backlog”. Click on “New Job” from the main menu.

Once you’ve created the job, the screen gets a bit more hectic. We’re only going to concern ourselves with a few key areas:

• Build Triggers

• Build Steps

• I’ll frequently refer to the ? icon.

Scheduling

Under build triggers, we want to use the “Build Periodically” option. The syntax is akin to cron and there are some additional macros for known intervals. As with any Jenkins option, you can click on the ? icon to the right of the option for inline help.

So we’re going to set up our health check to run every 15 minutes:

Defining the Build Step

Through Jenkins plugins, you can get an insane amount of additional build steps. However, the shipped experience has the stuff we need for now. We’re going to be using the “Execute Shell” option. If you are running Jenkins on Windows, you’ll want to use the “Execute Windows Batch command” instead. You will, of course, need to modify the commands appropriately yourself.

Here’s the body of our build step:

1
2
3

#!/bin/bash -l
CHECK=`mysql -u jenkins -ppassword -h 192.168.56.101 -BNe 'SELECT COUNT(*) FROM foo_db.jobs WHERE depth >= 100'`
exit ${CHECK}

Running the job

Once you click save, you can click “Build Now” on the job menu to give it a test. It should fail:

Let’s modify the job so we can see what success looks like. Click on the “Configure” link in the build menu and modify your build step. Set the threshold in the query to 101. The build should now be blue:

This all works very well if you just want to manually inspect the status however let’s take it a step further. Click on the “Configure” link from the Job menu. Notice at the bottom of the following screen, there’s a section called “Post-build Actions”. The very last option is “E-mail Notification”. You can click the ? to see the default behaviour. Check the box and add your email address:

Getting Notified

Sadly, this isn’t enough to enable email notifications. You’ll need to tell Jenkins an SMTP server it can use. Go back to the main menu and click “Manage Jenkins”.

From here, we’re going to click “Configure System”

Another busy screen! The settings in this section can get you in trouble if you aren’t careful. The most common problem is people attempting to enable security and inadvertently locking themselves out. We’re not worried about that for now. Scroll to the bottom and configure your SMTP server. The settings shown are for gmail and you’ll need to click the “Advanced” button to enable additional settings.

You can select the last checkbox to test that your settings work.

Once that’s done, click save. Now we’re going to rerun the job (Go back to the main menu then click your job from the list to see the “Build Now” option in the job menu. You most likely won’t get an email because the job is passing. Let’s configure our job again and set the threshold back to 100. Save the job and click “Build Now” again from the job menu.

You should get an email that looks something like this:

1
2
3
4
5
6
7
8

Subject: Build failed in Jenkins: Check FooDB backlog
See <http://localhost:8080/job/Check%20FooDB%20backlog/8/>

------------------------------------------
Started by user anonymous
Building in workspace <http://localhost:8080/job/Check%20FooDB%20backlog/ws/>
[workspace] $ /bin/bash -l /var/folders/d6/h7dxb_zj49s8xlj91zd3z6fr0000gn/T/hudson1906255485094144268.sh
Build step 'Execute shell' marked build as failure

There’s not much information in there since our job is swallowing the mysql output and using it as the exit code. You can spice the output however you like it by adding echo statements to the build step. Any output from the job will be included in the email. If you change the thresholds back to a value that you know will pass, you’ll get at least one email when the build recovers. Unless the build starts failing again, you won’t get any emails.

1
2

Subject: Jenkins build is back to normal : Check FooDB backlog
See <http://localhost:8080/job/Check%20FooDB%20backlog/9/>

Second Job

So now we’ve got something handling our DB test. We also needed to check to see if some process was running. Let’s do a simple one to see if MySQL is running. Let’s call it “Check MySQL Running”. Follow the steps for creating a free-style job but this time we’re going to create our build step like so:

1
2

#!/bin/bash -l
ssh 192.168.56.101 'ps -ef' | grep mysqld

Again, we’re going to assume that SSH keys are setup with no password. We’re keeping it simple. Just as in the case of the other job, we should get a blue build status.

Third Job

The third job is the most complex in that we’re going to need to install a plugin for maximum effect. This will have you jumping around a bit but hopefully you’re a bit more comfortable navigating by now. At a high level we’re going to do the following:

• Install a new Jenkins plugin
• Create a new job
• Take note of a new build option
• Configure the plugin globally
• Enable the plugin in our job

Installing a Plugin

We’re going to go back to “Manage Jenkins” (accessible from the main menu) but now we’re going to select “Manage Plugins”.

Once on the plugin screen, click the “Available” tab. This part can be overwhelming. It’s especially confusing since plugins will be listed twice if they fall into multiple categories. However, you only need to mark it once.

The plugin we want is called the “Log Parser Plugin”. If you can’t easily find it, use your browser’s “find on page” (CTRL-F, APPLE-F) to find it.

Check the box and click “Install without Restart”. You should see a screen similar to this:

Back to the job

Now let’s create our final job. Following the same steps as above, create a new job called “Check DHCP Errors”. Again, reaching for a contrived case, I’m going to check my VM’s syslog to see if it had any errors related to DHCP.

1
2

#!/bin/bash -l
ssh 192.168.56.101 'tail -n 5 /var/log/syslog'

Now we could have done this with a grep statement just like above. However I wanted to show installing plugins and the “Log Parser Plugin” actually offers some more flexible options, understands more than just pass or fail and can match multiple items without building overly complex flow into your shell step.

You’ll notice at the bottom we now have an ADDITIONAL option in our “Post-build Actions” - Console output (build log) parsing:

Whenever you install a plugin, where it’s used depends on what it does. In this case, we’re doing post processing of the job run log. We can add a third state via this plugin as opposed to just “Pass” or “Fail” - “Unstable”. Before we can enable it, however, we need to give it some parsing rules.

For now leave the option unchecked and click “Save”

Configuring the new plugin

Go back to the “Manage Jenkins” screen (where you set the Email settings). At the bottom, you should now have an option for Console Output Parsing:

Again, anything you configure in this section is GLOBAL. Luckily you can define various rule sets for parsing and apply them individualy to jobs. This plugin is a bit complex so you’ll probably want to look at the documentation.

We’re going to create a very basic rules file in /tmp on our LOCAL machine (where Jenkins is running) called jenkins-dhclient-rules:

1
2

warn /^.*dhclient: can't create .*: No such file or directory$/
info /^.*dhclient: bound to .*$/

This is telling the log parser that the following line is a “warning”:

Jan 23 01:49:52 ubuntu dhclient: can't create /var/lib/dhcp3/dhclient.eth1.leases: No such file or directory

and that

Jan 23 01:49:52 ubuntu dhclient: bound to 192.168.56.101 -- renewal in 1367 seconds.

is informational. These distinctions are handy for the plugin’s colorized output support.

Now that we’ve created that file, under the plugin settings we want to name it and give it the location to the file:

Back to our job

Finally!

Let’s go back to our new job (Check DHCP Errors) and modify it. We want to enable the parsing plugin in the post-build steps. We’re going to check “Mark build Unstable” for warnings and select our rule. Now save the job. The reason we’re going for warning is that this error is not fatal. Our system still gets an IP address. What we want to do is draw attention to it.

Now if you run the job, you’ll get a yellow ball indicating that the job is unstable. If we were to change the first line of our rule to error and check the appropriate box, the job would have been marked a failure. Something additional this plugin provides is the “parsed console output”. If you click on the job detail and select “Parse Console Output” from the job menu, you’ll actually get a nicer way to see exactly what was wrong:

Again, this is a totally contrived example. Obviously we would fail long before the parsing had the host been down.

Tying it all together

All of these individual jobs are neat but there’s an obvious dependency there. We need to be able to SSH to the host, we need mysql to be running and then we want to query it. We don’t want multiple emails for each failure. We only want the actual failed job to alert us. Let’s chain these jobs together to match that flow.

• Under the “MySQL Running” and “FooDB” jobs, disable the cron schedule. We only want it on the “DHCP” job.
• Under the DHCP job, we’re going to select the Post-build step of “Build other projects”
• Check “Trigger even if build is unstable” since we know it’s going to be unstable.
• In the text area, we want to add our “Check MySQL Running” job
• Under the “Check MySQL Running” job, we want to select “Trigger only if build succeeds” and set our text area to the “Check FooDB Backlog” job.

Now if you run the top-level job (Check DHCP Errors), all of the jobs will run. If any fail, the run will stop there and alert you! Since this is now scheduled, every 15 minutes this entire workflow will be checked.

Additional plugins and tips

Jenkins has a boatload of plugins. It’s worth investigating them to see if they make some given task (like output parsing) easier. Some provide additional notification paths like jabber or irc. Others provide additional build step types in specific languages like Groovy or Powershell. You can also do things like create a “Parameterized Build”. This is especially handy for thresholds. There’s also a very handy SSH plugin that let’s you define hosts globally and keys per host. This helps clean up your build steps too.

One plugin that was recommended is the “Email-ext” plugin. This allows you to REALLY spice up and configure your email notifications.

There’s a plugin for checking a web site for some criteria and plugins for starting virtual machines. There are also plugins for creating a radiator view so you can get a nice big dashboard for just checking the state of jobs at a glance.

The key to remember is that Jenkins is an unopinionated build tool. This flexiblity lends itself to doing off-the-wall stuff (like being a monitoring system or a cron replacement). The trick is translating the concepts and terminology of building software to something that fits your use case.

Additional Credits

I’d like to thank Joe Miller, Ches Martin and R. Tyler Croy for reviewing this post and offering up corrections, tips and advice.

The email and enStratus

I got an email from Ilan Rabinovitch just as things were going down with me headed to enStratus. Since the event was going to be right around the time I started, I pretty much put it out of my mind. Then I realized that my boss was going to be attending. I figured it wouldn’t hurt to ask and it was decided that I would shadow John on his trip to enStratus HQ for my mandatory cultural immersion (translation: does the fat redneck own a winter coat?) and on to SCaLE. While I didn’t make it to Minnesota (diverted to San Jose on business), I did still make it to the conference.

I had an awesome time in San Francisco and San Jose. I got to meet a great bunch of folks and geek out hardcore.

Monitoring sucks

Ilan asked me if I would be willing to be on a panel about the whole #monitoringsucks thing. We were able to score a great panel of folks:

• Simon Jakesch from Zenoss
• James Litton from PagerDuty
• Jody Mulkey from Shopzilla

The event was awesome. I did some minor introductions, got the ball rolling with some questions and then we let the audience take it from there. The participation was AWESOME. We had great questions from the audience and the feedback I got AFTER the fact was mindblowing. One particular post-panel question is worth a blog post in its own right.

One thing that really stood out was this: People just don’t know where to start. The landscape is pretty “cluttered”. Chris Webber brought up a very salient point that I sometimes forget; When we talk about “monitoring”, we’re really talking about multiple things - collection, alerting, visualization, trending and multitudes of other aspects.

I got asked several times in the hallway - “What should I use?” or “What do you think about ?”. My first response was always “What are you using now?”.

I like to think I’m pretty pragmatic. I love the new shiny. I love pretty graphs. I’m a technologist. However, I know when to be realistic. My thought process goes something like this:

Do you have something in place now?

Yes

Why are you looking to switch? Is it unreliable? Is it painful to configure? Basically, if it’s getting the job done and has relatively minor overhead there’s no reason to switch. The pain points for me with monitoring solutions usually come much later. It doesn’t scale or scaling it is difficult. It doesn’t provide the visibility I need. It’s unreliable (usually due to scaling problems). Until then, use what you’ve got and be guard for early signs of problems like check latency going up or missed notifications.

If you have a configuration management solution in place, it probably has native support for configuring Nagios. When you add a new host to your environment, you only need to tell your CM tool to run on your monitoring server. If you’ve done any sort of logical grouping, you’ll have the right things monitored quickly.

No

If you don’t have ANYTHING in place, you need to cover two bases pretty quick:

• Outside-In Checks: is my site up and responding timely?
• Stupid stuff: Are my disks filling up? Is my database slave behind?

For outside in checks, use something quick and easy like Pingdom. For the inside checks, don’t underestimate the power of a cron job. If you want something a bit more packaged, look at monit. It’s dead simple and can get you to a safe place.

A note on visibility

Monitoring tools are great but many times they fall down when you need to diagnose a problem ex post facto. If you went the simple route, you probably don’t have any real trending data. This is where many complaints start to come from folks. You end up monitoring the same thing twice - once for alerting systems like Nagios and another time for your visualization, trending and other engines. When you reach this point, start looking at things like Sensu or all-in-one solutions that, while cumbersome and imprecise use the same collected data - Zenoss, Zabbix, Icinga (originally a fork of Nagios).

The event was recorded (both audio and video) but I have no timeframe on when it’s going to be available but I’ll let you know as soon as it’s up.

The rest of the conference

The rest of the conference was epic as well. Being that this was my first time, I didn’t know what to expect. The thing that most stood out was the number of children. This was probably the most family friendly conference I’ve ever been to. Encouraging stuff. Plenty of events and in fact an entire track dedicated to children.

I didn’t get to attend as many talks as I wanted to. While the facility was really nice, the building is like a faraday cage. My phone spent what little battery life it had just trying to get a signal. I spent quite a bit of time running back to my room to charge up. Chris Webber totally got me hooked on portable chargers.

Juju talk

disclaimer: I’m fully aware that Juju is undergoing heavy active development and is a very young project

One of the talks I attended was on Juju. I was probably a bit harsh on Juju when it was first announced. The original name was much better and the whole witch doctor thing just doesn’t sit well with me.

I also hate the tag line - “DevOps distilled”. It’s marketing pure and simple. I have very little tolerance for things that bill themselves as a “devops tool” or “for devops”.

But more than the name, something about Juju didn’t feel right. After the talk, something still doesn’t feel right. While I don’t like pooping all over someone else’s hard work so writing this part is tough.

Where does it fit?

Right now, I don’t think Juju even knows where it fits. It’s got some great ideas and on any other day, I’d be all over it. The problem is that Juju tries to do too much in some areas and not enough in others.

Parts of Juju are EXACTLY what I see as my primary use case for Noah. The service orchestration is great. The ideas are pretty solid. Juju even uses ZooKeeper under the hood.

Services not servers

Everyone knows that I preach the mantra of “services matter. hosts don’t”

The problem is that in an attempt to be the Nagios (unlimited flexibility) of configuration management, it can’t actually do enough in that area. Because it only concerns itself with services (and the configuration of them), it doesn’t do enough to manage the host. Just because the end state is “I’m serving a web page” doesn’t mean you should ignore the host its running on. Since Juju isn’t designed to deal with that (and actually LACKS any primitives to do it), you’re left with needing to manage a system in multiple places - once with your CM tool and then again with the charms.

Someone said it best when he described Juju as “apt for services”. It’s quite evident that the same broken mentality that apt takes to managing packages is applied to Juju as well. Charms have upgrade and downgrade steps. They’re just as complicated too. Not only is there no standard (since charms can be written in any language) it’s actually detrimental. The reason for a common DSL or language like the ones exposed by CM tools is not some academic mental masturbation. It’s repeatability and efficiency. I can go into a puppet shop and look at a module and know what it does. I can look at most chef recipes (outside of ones that might use a custom LWRP) and know what’s going on.

In the Juju world, a single charm could be written in one spot in Python and another spot in Bash. It pushes too much responsibility to the end user NOT to mess something up. I dare say that idempotence doesn’t even exist in Juju.

A fair shake

Again, I’m going to do some more playing around with Juju. I think it can meet a critical need for folks but I think they need to revisit what problem they’re trying to solve. I appreciate the work they’ve done and I’m totally excited that orchestration is getting the proper attention. The presenters were fantastic.

Other stuff

I attended a really good talk about the history of Openstack and where it’s going. It was great. As someone who is working with openstack professionally now (and had just dealt with some of its warts not 3 days before hand), I found it very valuable. Also congrats to the speaker, Jesse Andrews on the birth of his first child!

I managed to make it to Brendan Gregg’s talk as well. If you ever have the opportunity to hear him speak, you should take it. While I’m not a SmartOS user, the talk was really not about that. I walked out with some amazing insight on how smart people troubleshoot performance problems. Very well done.

The hallway track

Of course the real value in any conference is the hallway track. The chance to interact with your peers. I met so many smart people (some twice because I suck at remembering faces at first - sorry!). Chatting with folks like C. Flores, Jason Cook, Sean O’Meara, Chris Webber, Dave Rawks, Matt Ray, Matt Silvey and so many others that I can’t keep straight in my head. Everyone was awesome and I hope that you were able to get as much out of me as I got out of you.

Thanks again to Ilan for the invitation and for running such an amazing conference.

Also, little known made-up fact: Lusis is Tagalog for “He who eats with both hands”…..

I can say, wihtout a doubt, that 2011 has been the most awesome year both professionally and personally in my life. And I have pretty much everyone else to thank for it.

Some special shoutouts

There are so many folks to thank for this year. I owe so many beers that I can’t keep count. I can’t possibly thank everyone but I want to throw a few special shoutouts to folks.

My wife

Why she puts up with my shit, I’ll never know. Needless to say, without her this year would have been radically different. She managed two toddlers by herself on almost every trip I took. She’s been nothing but encouraging and she also helps keep me grounded by reminding me what’s important.

Patrick DeBois

Thanks for giving me the opportunity to help with the DevOps Days events. Through the events I’ve met some of the most amazing people in the world. The DevOps community is a wonderful group of folks and I would never have met half of them if Patrick hadn’t given me the opportunity to participate.

Vertical Acuity

While I’m sad to be leaving friends behind, VA was amazing in letting me travel so much. Not only that but they trusted and valued my opinion on so many things. Whoever takes my place will be lucky to work with such an awesome group of folks.

John Willis

Not only for being a good friend but for giving me an opportunity to work with him at enStratus.

Luke Kanies, James Turnbull, Jose Palafox and the Puppet Labs crew

Puppet Labs gets double the thanks - for giving me the opportunity to talk about my project at PuppetConf and also for sponsoring me to travel to Goteborg and speak. The whole crew over there is amazing.

Damon Edwards, Alex Honor and DTO Solutions

I’m grateful to DTO for giving me the opportunity to attend Velocity and letting me be a booth babe. Damon and Alex both have been forces of awesome for the DevOps community.

John Christian

John asked me early on to help with the Atlanta DevOps meetups and I’m glad he did. He stands alone in the corporate bullshit world of the financial services industry. He tought me a lot and I want to thank him for it.

Mandi Walls

For being pretty much awesome by listening to my ranting, letting me bounce ideas off her. And for being my sister from another mother.

Seth Chisamore

For the various lunches, talks, introductions and meetup involvement. Local folks rock. Seth rocks.

Jordan Sissel

For being awesome, down to earth and not an asshole. And for all the code. And for giving me the honor of contributing to SysAdvent.

Kelsey Hightower

For helping me navigate my foray into the world of Python (technically that was a few years ago). Also for showing folks how to just get shit done.

Everyone else

I can’t possible fit everyone here’s an abbreviated list of folks, in no specific order, who have impacted me this year off the top of my head. If I leave you off, please don’t take offense. I’m shooting from the cuff here.

Bradley Taylor, Will Farrington, Corey Haines, Tim Dysinger, Joe Miller, Mathias Meyer, Vladimir Vuksan, Adam Jacob, Sean Porter, Tony Arcieri, R.I. Pienaar, Adam Fletcher, Anthony Goddard, Joe Williams, Brad Anderson, Cat Muecke (alas, Cat does not tweet!), Harlan Barnes, Wesley Beary, Mitchell Hashimoto, Wayne Seguin, Dan DeLeo, Josh Timberman, Noah Kantrowitz, Andrew Clay Schafer, Mark Imbriaco, Stephen Nelson-Smith, Gareth Rushgrove, Ian Meyer, Devdas Bhagat, Martin Jackson, Michael Leinartas, Kris Buytaert, Brandon Burton, Al Tobey, Matthew Jones, Julian Simpson, Jason Cook, Jos Boumans, Susan Potter, Thom May, Kit Plummer, Sascha Bates, Bob Martin, Bryan Horstmann-Allen, Benjamin W. Smith, Ches Martin, Jason Dixon, Phil Hollenback, Nate St. Germain, Scott Smith, Sean Cribbs, Andy Gross, Ben Rockwood, James Casey, Les Hazlewood, Allan Ditzel, Marius Ducea, Noah Campbell, Tim Anglade, Corey Donohoe, Matt Simmons, Ernest Mueller, Lindsay Holmwood, Christian Paredes, Brice Figureau, Grig Gheorghiu, Darrin Eden, Shay Banon, Ramon Van Alteren and so many others.

Software that changed my world

I also wanted to give a shoutout to a few projects that pretty much changed how I thought about the software world around me.

ElasticSearch

ElasticSearch has beeen, bar none, in my top two amazing things the past year. Having first heard about it via Logstash, when I started digging in it blew my mind. The one thing that amazed me most about ES was the Zen discovery. It’s like the first time you heard about consistent hashing. It’s one of those things that makes you say “how the fuck did I not think of this first?”. The other thing that I find awesome is that ES not only makes scaling up painless but scaling DOWN (which is the hard part) is just as easy. As a sysadmin, ElasticSearch has been the most pleasant bit of infrastructure I’ve ever had the pleasure of standing up.

0mq

The other thing that amazed me this year was 0mq. 0mq essentially makes the difficult and next to impossible things possible. I am not lying when I say that every project I have floating around in my head is either built around or has a perfect spot for 0mq. Along with ElasticSearch, it has fundamentally changed how I think about software, infrastructure and more.

Apache ZooKeeper

While I’m not a fan of ZooKeeper on several levels, It would be wrong to totally ignore it. For the longest time, ZK stood alone in what it provided. People are building amazing things with it and it inspired me to write Noah.

Logstash

At first I was pretty dismissive of Logstash. Mainly because I didn’t have a real need. Then I started digging in and realized that Logstash is only tangentially about logs. Logstash is kind of what you always wanted a pipe to be. Arbitrary input, arbitrary filtering, arbitrary output. The use cases for logstash are so much greater when you stop thinking about logs and start thinking about moving data.

Erlang, OTP and Riak

While my Erlang only extends to passable reading, via Riak, I found a desire to learn more about it. The biggest thing that stuck in my head and also changed the way I think is the Actor model. For the first time that I can remember, a concept made perfect sense to me. While learning Erlang in earnest is a goal for 2012, I think about how simple and understandable the Actor model was.

Celluloid

Following up on the Actor model, I have mad respect for Tony Arcieri. If you look back at his projects, they all follow a similar theme: improving the concurrency story on Ruby. He’s tenacious and passionate about something that most people would have (and if they’re arrogant dickfaces) laughed at by now. Celluloid brings some amazing functionality to Ruby inspired by Erlang and OTP. I find myself defaulting to it whenever I need to even think about concurrency in Ruby. Even outside of that, it encourages good behaviour with threads and reduces the chances you’ll fuck something up. The companion project, DCell, is something I’m itching to work with as well.

Statsd and Graphite

If this past year was about anything, it was about metrics. Lots and lots of metrics. Etsy pushed out statsd and brought metrics collection to the masses. Coda Hale gave an amazing talk on metrics and released the code to back it up. Shooting in the dark sucks. You need numbers. Collect ALL the metrics.

Final Thoughts

The world of open source and the community around devops is amazing. I learned from and met so many people in 2011. I’m hoping that 2012 is the year that I can give back to them in some small way. Thanks to everyone for making this past year amazing.

I’m a fairly active Twitter user.

Patrick has joked that it’s as if I have Twitter wired directly to my brain. It’s not far from the truth. I like to engage people and normally Twitter is great medium for engaging folks. Unfortunately, the message size limit makes Twitter an imperfect medium for involved discussions.

I know better but sometimes I forget.

Anyway, last friday I realized near the end of the day that I had pretty much gone off the rails. If I wasn’t bitching about Maven and Java, I was involved in random discussions about the SaltStack project. Combine that with normal inane bullshit and I somehow managed to pull off a 60+ tweets. It took a comment by roidrage on IRC to point out that I really needed to calm down.

With that, I declared communication blackout for the weekend. I decided to go to happy hour and spend the weekend just having fun with the family. It was awesome. So sorry for the sheer number of people I managed to piss off on Friday.

Trolling Github

One of the things I also decided to do not stress about making time to hack. I knew that if I got working on one of my projects, I would totally stay distracted thinking about it.

So I went trolling. On Github.

I was just poking around Github, when I saw in my feed that some commits were done to the user’s guide for ZeroMQ. Because I have a serious geek woody for ZeroMQ, I got wrapped up reading the guide and looking at some of the more advanced examples. I’ve got some ideas I want to implement in Ark and Noah that involved 0mq so I figured it would be time well spent.

Now if anyone has bothered to read the zguide, you’ll know that one of the BEST parts is the code samples. Seriously. They have examples for all of the architectures in almost every language. I don’t know a single goddamn person who knows Haxe, but there are examples in the guide for Haxe. You can see an example of what I’m talking about here.

Notice at the bottom the list of examples for languages. If you mouse over the last entry, many times you’ll get multiples highlighted. This means that chunk of highlighted languages doesn’t have any examples written.

I noticed that quite a few of the advanced ones didn’t have Ruby versions. I started back at the beginning of the guide until I found the first one that didn’t have a Ruby example - the interrupt example.

Challenge Accepted

So here I am - resolved not to work on any of my own projects and knowing that I didn’t have time to get involved with something TOO heavy. I decided to fork the guide and start adding missing Ruby examples.

Now I only got two example done the entire weekend. This mainly revolved around how limited my time was but also around getting REALLY comfortable with ffi-rzmq. I wanted to make sure that the examples I wrote had the write mix of idiomatic Ruby and yet explicit enough for someone who didn’t know the specifics of ffi-rzmq.

One that I really struggled with was this one:

https://github.com/imatix/zguide/commit/4c231d1023819152813fad09a45458bd33cb02a9

If you get familiar with the zguide, you’ll see a lot of references to zhelpers. It’s really just a bunch of boilerplate code that helps keep the actual examples to a nice consumable chunk size. There was not a zhelpers for the Ruby examples. I looked at the others to get an idea of what kinds of things were in there. In relation to the identity examples, there was a dump helper that just dumped the contents of a message. If you look at the Python and C examples for dump, you’ll see how they pull the identity of the message out. An interesting comparision, is how the Scala version of dump works.

Instead of focusing on duplicating the strategy employed by the C and Python versions, I went with something that fit how ffi-rzmq works a bit more. I realized that the point was not the content of the helpers so much as the end result, showing that 0mq would generate an identity for a message if one wasn’t explcitly provided.

I’m quite sure that at some point, ZMQ::Message objects will get an attribute accessor to simply return the identity. Right now the code base is under a bit of a refactor.

Call to Action

I really want to encourage others to do something like this. No pressure. Just troll Github. Look at some projects that are interesting. Look at the open issues. Fork, fix a bug or two and make some pull requests. After that, go on your merry way. No obligations. At worst, you’ve spent some time sharpening your skills. At best, however, you’ve made a lasting contribution.

And shit, it doesn’t even have to be a code contribution. If you can grok the project well enough, add some wiki pages.

I dunno. Github just has this amazingly easy flow for contribution. Do unto others and all that.

My previous post covered some of the annoying excuses and complaints that people like to use when discussing deployments. The big take away should have been the following:

• The risk associated with deploying new code is not in the deploy itself but everything you did up to that point.
• The way to make deploying new code less risky is to do it more often, not less.
• Create a culture and environment that enables and encourages small, frequent releases.
• Everything fails. Embrace failure.
• Make deploys trivial, automated and tolerant of failure.

I want to make one thing perfectly clear. I’ve said this several times before. You can get 90% of the way to a fully automated environment, never go that last 10% and still be better off than you were before. I understand that people have regulations, requirements and other things that prevent a fully automated system. You don’t ever have to flip that switch but you should strive to get as close as possible.

Understanding the role of operations

Operations is an interesting word. Outside of the field of IT it means something completely different than everywhere else in the business world. According to Wikipedia:

Business operations encompasses three fundamental management imperatives that collectively aim to maximize value harvested from business assets

• Generate recurring income

• Increase the value of the business assets

• Secure the income and value of the business

IT operations traditionally does nothing in that regard. Instead IT operations has become about cock blocking and being greybeareded gatekeepers who always say “No” regardless of the question. We shunt the responsibility off to the development staff and then, in some sick game of ‘fuck you’, we do all we can to prevent the code from going live. This is unsustainable; counter-productive; and in a random twist of fate, self destructive.

One thing I’ve always tried to get my operations and sysadmin peers to understand is that we are fundamentally a cost center. Unless we are in the business of managing systems for profit, we provide no direct benefit to the company. This is one of the reasons I’m so gung-ho on automation. John Willis really resonated with me in the first Devops Cafe podcast when he talked about the 80/20 split. Traditionally operations staff spends 80% of its time dealing with bullshit fire-fighting muck and 20% actually providing value to the business. The idea that we can flip that and become contributing members of our respective companies is amazing.

Don’t worry. I’ll address development down below but I felt it was important to set my perspective down before going any further.

Technical Debt and Risk Management

Glancing back to my list of take-aways from the last post, I make a pretty bold (to some people) statement. When I say that deploy risk is not the deploy itself but everything up to that point, I’m talking about technical debt.

Technical debt takes many forms and is the result of both concious, deliberate choices as well as unintended side-effects. Some examples of that are:

• Lack of or insufficient testing and associated
• Overreliance on time consuming manual processes
• Shortcuts to meet deadlines - both artifical and real
• Violation of the 10-minute maxim
• Technological choices
• Cultural choices
• Fiscal limitations

All of these things can lead to technical debt - the accumulation of dead bodies in the room as a byproduct of how we work. At best, someone at least acknowledges they exist. At worst, we stock up on clothespins, pinch our nostrils shut and hope no one notices the stench. Let’s address a couple of foundational things before we get into the fun stuff.

Testing

Test coverage is one of the easiest ways to manage risk in software development. One of the first things to go in a pinch is testing. Even that assumes that testing was actually a priority at some point. I’m not going to harp on things like 100% code coverage. As I said previously, humans tend to overcompensate. Test coverage is also, however, one of the easiest places to get your head above water. If you don’t have a culture of committment to testing, it’s hard but not impossible to get started. You don’t have to shutdown development for a week.

1. Start by having a commitment to write tests for any new code going forth.
2. As bugs arise in untested code, make a test case for the bug a requirement to close the bug.
3. Find a small victory in existing code. Create test coverage for low hanging fruit.
4. Plan for a schedule to cover any remaining code

The key here is baby steps. Small victories. Think Fezzik in ‘The Princess Bride’ - “I want you to feel like you’re winning”.

Testing is one of the foundations you have to have to reach deploy nirvana. System administrators have a big responsiblity here. Running tests has to be painless, unobstrusive and performant. You should absolutely stand up something like Jenkins that actually runs your test suite on check-in. As that test suite grows, you’ll need to be able to provide the capacity to grow with it. That’s where the next point can be so important.

Manual processes

Just as testing is a foundation on the code side, operations has a commensurate responsibility to reduce the number of human hands involved with creating systems. We humans, despite the amazing potential that our brains provide, are generally stupid. We make mistakes. Repeatability is not something we’re good at. Some sort of automated and repeatable configuration management strategy needs to be adopted. As with testing, you can make some amazing progress in baby steps by introducing some sort of proper configuration management going forward. I don’t recommend you attempt to retrofit complex automation on top of existing systems beyond some basics. Otherwise you’ll be spending too much time trying to differentiate between “legacy” and “new” servers roles. If you are using some sort of virtualization or cloud provider like EC2, this is a no brainer. It’s obviously a bit harder when you’re using physical hardware but still doable.

Have you ever played the little travel puzzle game where you have a grid of moving squares? The idea is the same. You need just ONE empty system that you can work with to automate. Pick your simplest server role such as an apache webserver. Using something like Puppet or Chef, write the ‘code’ that will create that role. Don’t get bogged down in the fancy stuff the tools provide. Keep it simple at first. Once you think you’ve got it all worked out, blow the server away and apply that code from bootstrap. Red, green, refactor. Once you’re comfortable that you can reprovision that server from bare metal, move it into service. Make sure you have your own set of ‘test cases’ that ensure the provisioned state is the correct one. This will become important later on.

Take whatever server it’s replacing and do the same for the next role. When I came on board with my company I spent many useless cycles trying to retrofit an automation process on top of existing systems. In the end, I opted to take a few small victories (using Chef in this case):

1. Create a base role that is non-destructive to existing configuration and systems. In my case, this was managing yum repos and user accounts.
2. Pick the ‘simplest’ component in our infrastructure and start creating a role for it.
3. Spin up a new EC2 instance and test the role over and over until it works.
4. Terminate the instance and apply the role on top with a fresh one.
5. Replace the old instances providing that role with the new ones and move to the next role.

Using this strategy, I was able to replace all of our legacy instances for the first and second tiers of our stack in a couple of months time. We are now at the point where, assuming Amazon plays nice with instance creation, we can have any role in those tiers recreated at a moment’s notice. Again, this will directly contribute to how we mitigate risk later on.

10 minute maxim

I came up with this from first principles so I’m sure there’s a better name for it. The idea is simply this:

Any problem that has to be solved in five minutes can be afforded 10 minutes to think about the solution.

System Administrators often pride ourselves on how cleverly and quickly we can solve a problem. It’s good for our egos. It’s not, however, good for our company. Take a little extra time and consider the longer term impact of what solution you’re about to do. Step away from the desk and move. Consult peers. Many times I’ve come to the conclusion that my first instinct was the right one. However more often than not, I’ve come across another solution that would create less technical debt for us to deal with later.

A correlary to this is the decision to ‘fix it or kick it’. That is ‘Do we spend an unpredictable amount of time trying to solve some obscure issue or do we simply recreate the instance providing the service from our above configuration management’. If you’ve gone through the previous step, you have should have amazing code confidence in your infrastructure. This is very important to have with Amazon EC2 where you can have an instance perform worse overtime thanks to the wonders of oversubscription and noisy neighbors.

Fuck that. Provision a new instance and run your smoke tests (I/O test for instance). If the smoke tests fail, throw it away and start a new one. It’s amazing the freedom of movement afforded by being able to describe your infrastructure as code.

Getting back to deploys

I would say that without the above, most of the stuff from here on out is pretty pointless. While you CAN do automated and non-offhour deploys without the above, you’re really setting yourself up for failure. Whether it’s a system change or new code, you need to be able to ensure that that some baseline criteria can be met. Now that we’ve got the foundation though, we can build on it and finally adopt some distinct strategies for releases.

Building on the foundation

The next areas you need to work on are a bit harder.

Metrics and monitoring

Shooting in the dark sucks. Without some sort of baseline metric, you authoritatively say whether or not a deploy was ‘good’. If it moves, graph it. If it moves, monitor it. You need to leverage systems like statsd (available in non-node.js flavors as well) that can accept metrics easily from your application and make them availabile in the amazing graphite.

The key here is that getting those metrics be as frictionless as possible. To fully understand this, watch this presentation from Coda Hale of Yammer. Coda has also created a kick-ass metrics library for the JVM and others have duplicated his efforts in their respective languages.

Backwards compatibility

You need to adopt a culture of backwards compatibility between releases. This is not Microsoft levels we’re talking about. This affects interim releases. As soon as you have upgraded all the components, you clean up the cruft and move on. This is critical to getting to zero/near-zero downtime deploys.

Reduce interdependencies

I won’t go into the whole SOA buzzword bingo game here except to say that treating your internal systems like a third party vendor can have some benefits. You don’t need to isolate the teams but you need to stop with shit like RMI. Have an established and versioned interface between your components. If component A needs to make a REST call to component B, upgrades to the B API should be versioned. A needs version 1 of B’s api. Meanwhile new component C can use version 2 of the API.

Automation as a default

While this ties a lot into the testing and configuration management topics, the real goal here is that you adopt a posture of automation by default. The reason for this should be clear in Eric Ries’ “Five Whys” post:

Five Whys will often pierce the illusion of separate departments and discover the human problems that lurk beneath the surface of supposedly technical problems.

One of the best ways to eliminate human problems is to take the human out of the problem. Machines are very good at doing things repeatedly and doing them the same way every single time. Humans are not good at this. Let the machines do it.

Deploy Strategies

Here are some of the key strategies that I (and others) have found effective for making deploys a non issue.

Dark Launches

The idea here is that for any new code path you insert in the system, you actually exercise it before it goes live. Let’s face it, you can never REALLY simulate production traffic. The only way to truly test if code is performant or not is to get it out there. With a dark launch, you’re still making new database calls but using your handy dandy metrics culture above, you now know how performant it really is. When it gets to acceptable levels, make it visible to the user.

Feature flags

Feature flags are amazing and one of the neat tricks that people who perform frequent deploys leverage. The idea is that you make aspects of your application into a series of toggles. In the event that some feature is causing issues, you can simply disable it through some admin panel or API call. Not only does this let you degrade gracefully but it also provides for a way to A/B test new features. With a bit more thought put into the process, you can enable a new feature for a subset of users. People love to feel special. Being a part of something like a “beta” channel is an awesome way to build advocates of your system.

Smoke testing at startup

This is one that I really like. The idea is simply that your application has a basic set of ‘tests’ it runs through at startup. If any of those tests fail, the code is rolled back.

Now this is where someone will call me a hypocrite because I said you should and can never really roll back. You’re partially right. In my mind, however, it’s not the same thing. I consider code deployed once it’s taken production traffic. Up until that point, it’s just ‘pre-work’ essentially. Let’s take a random API service in our stack. I’m assuming you have two API servers in this case.

• Take one out of service
• Deploy code
• Smoke tests run
• If smoke tests fail, stop new code and start old code
• If smoke tests pass, start sending production traffic to server
• If acceptable, push to other server
• profit!

Now you might see a bit of gotcha there. I intentionally left out a step. This is a bit different than how shops like Wealthfront do it. They actually DO roll back if production monitoring fails. My preference is to use something similar to em-proxy to do a sort of mini-dark launch before actually turning it over to end-users. You don’t have to actually use em-proxy. You could write your own or use something like RabbitMQ or other messaging system. This doesn’t always work depending on the service the component is providing but it does provide another level of ‘comfort’.

Of course this only works if you maintain backwards compatibility.

Backwards Compatibility

This is probably the hardest of all to accomplish. You may be limited by your technology stack or even some poor decision made years ago. Backwards compatibility also applies to more than just your software stack. This is pretty much a critical component of managing database changes with zero downtime.

Code related

Your code needs to understand ‘versions’ of what it needs. If you leverage some internal API, you need to maintain support for an older version of that API until all users are upgrade. Always be deprecating and NEVER EVER redefine what something means. Don’t change a property or setting that means “This is my database server hostname” to “This is my mail server hostname”. Instead create a new property, start using it and remove the old on in a future release. Don’t laugh, I’ve seen this done. As much as I have frustrations with Java, constructor overloading is a good example of backwards compatibility.

Database related

Specifically as it relates to databases, consider some of the following approaches:

• Never perform backwards incompatible schema changes.
• Don’t perform ALTERs on really large tables. Create a new table that updated systems use and copy on read to the new table. Migrate older records in the background.
• Consider isolating access to a given table via a service. Instead of giving all your applications access to the ‘users’ table, create a users service that does that.
• Start exercising code paths to new tables early by leveraging dark launches

Some of these techniques would make Codd spin in his grave.

We’re undergoing a similar situation right now. We originally stored a large amount of ‘blob’ data in Voldemort. This was a bit perplexing as we were already leveraging S3 for similar data. To migrate that data (several hundred gigs) we took the following approach:

• Deploy a minor release that writes and new data to both Voldemort and S3.
• Start a ‘copy’ job in the background to migrate older data
• Continue to migrate data
• When the migration is finished, we’ll deploy a new release that uses S3 exclusively
• Profit (because we get to terminate a few m1.large EC2 instances)

This worked really well in this scenario. These aren’t new techniques either. Essentially, we’re doing a variation of a two-phase commit.

Now you might think that all this backwards compatibility creates cruft. It does. Again, this is something that requires a cultural shift. When things are no longer needed, you need to clean up the code. This prevents bloat and makes understanding it down the road so much easier.

Swinging like a boss

Here’s another real world example:

Our code base originally used a home-rolled load balancing technique to communicate with one of our internal services. Additionally, all communication happened over RPC using Hessian. Eventually this became untenable and we decided to move to RabbitMQ and JSON. This was a pretty major change but at face value, we should have been able to manage with dual interfaces on the provider of the service. That didn’t happen.

You see, to be able to use the RabbitMQ libraries, we had to upgrade our version of Spring. Again, not a big deal. However our version of Hessian was so old that the version of Hessian we would have to use with the new version of Spring was backwards incompatible. This is yak shaving at its finest, folks. So basically we had to upgrade 5 different components all at once just to get to where we wanted and NEEDED to be for the long term.

Since I had already finished coding our chef cookbooks, we went down the path of duplicating our entire front-end stack. What made this even remotely possible was the fact that we were using configuration management in the first place. Here’s how it went down:

• Duplicate the various components in a new Chef environment called ‘prodB’
• Push new code to these new components
• Add the new components to the ELBs and internal load balancers for a very short 5-10 minute window. Sort of a mini-A/B test.
• Check the logs for anything that stood out. Validated the expected behavior of the new systems. Thsi also gave us a chance to ‘load-test’ our rabbitmq setup. We actually did catch a few small bugs this way.

Once we were sure that things looked good, we swung all the traffic to the new instances and pulled the old ones out. We never even bothered to upgrade the old instances. We just shut them down.

Obviously this technique doesn’t work for everyone. If you’re using physical hardware, it’s much more costly and harder to pull off. Even internally, however, you can leverage virtualization to make these kinds of things possible.

Bitrot

What should be the real story in this is that bitrot happens. Don’t slack on keeping third-party libraries current. If a third-party library introduces a breaking change and it affects more than one part of your stack, you probably have a bit too tight of a coupling between resources.

Wrap up/Take away

This post came out longer than I had planned. I hope it’s provided you with some information and things to consider. Companies of all shapes, markets and sizes are doing continuous deployment, zero downtime deploys and all sorts of things that we never considered possible. Look at companies like Wealthfront, IMVU, Flickr and Etsy. Google around for phrases like ‘continuous deployment’ and ‘continuous delivery’.

I’m also painfully aware that even with these tricks, some folks simply cannot do them. There are many segments of industry that might not even allow for this. That doesn’t mean that some of these ideas can’t be implemented on a smaller scale.

Why are you still deploying overnight?

I thought this post was particularly apropos for several reasons. I just got back from DevOpsDays EU AND I’m currently in the process of refactoring our deploy process.

I’m breaking this up into two parts since it’s a big topic. The first one will cover the more “theoretical” aspects of the issue while the second will provide more concrete information.

Myths, Lies and other bullshit

Before I go any further, we should probably clear up a few things.

Understand, first and foremost, that I’m no spring chicken in this business. I’ve worked in what we now call web operations and I’ve worked in traditional financial environments (multiple places). If it CAN go wrong, it has gone wrong for me. Shit, I’ve been the guy who dictated that we had to deploy after hours.

Also, this is not a “tell you what to do” post.

So what are some of the myths and other crap people like to pull out when having these discussions?

• Change == Risk
• Deploys are risky
• Rollbacks
• Nothing fails
• SLAs

There’s plenty more but these are some of the key ones that I hear.

Change is change

There is nothing inherent in change that makes it risky, dangerous or anything more than change. Change is neither good or bad. It’s just change.

The idea that change has a risk associated with it is entirely a human construct. We have this false assumption that if we don’t change something then nothing can go wrong. At first blush that would make sense, right? If it ain’t broke, don’t fix it.

Why do we think this? It’s mainly because we’re captives to our own fears. We changed something once, somewhere, and everything went tango uniform. The first reaction after a bad experience is never to do whatever caused that bad experience again. This makes sense in quite a few cases. Touch fire, get burned. Don’t touch fire, don’t get burned!

However this pain response tends to bleed over into areas. We deployed code one time that took the site down. We changed something and bad things happened. Engage overcompensation - We should never change anything.

Deploys are not risky

As with change, a deploy (a change in and of itself) is not inherently risky. Is there a risk associated with a deploy? Yes but understand that the risk associated with pushing out new code is the culmination of everything you’ve done up to that point.

I can’t even begin to count the number of ways that a deploy or release has gone wrong for me. Configuration settings were missed. Code didn’t run properly. The wrong code was deployed. You name it, I’ve probably seen it.

The correct response to this is NOT to stop doing deploys, do them off-hours or do them less often. Again with the overcompensation.

The correct way to handle deployment problems is to do MORE deploys. Practice. Paraphrasing myself here from an HN comment:

Make deploys trivial, automated and tolerant to failure because everything fails.

“Release early, release often” isn’t just about time to market. The way to reduce risk is not to add more risky behavior (introducing more vectors for shit to go wrong). The way to reduce the risk associated with deploys is to break them into smaller chunks.

You need to stop thinking like Subversion and start thinking like Git.

One of the reasons people don’t feel comfortable performing deploys during the day is because deploys are such a big deal. You’ve got to make deploys a non-issue.

Rollbacks are a myth

Yes, it’s true. You can never roll back. You can’t go back in time. You can fake it but understand that it’s typically more risky to rollback than rolling forward. Always be rolling forward.

The difficulty in rolling forward is that it requires a shift in how you think. You need to create a culture and environment that enables, encourages and allows for small and frequent changes.

Everything fails. Embrace failure.

It amazes me that in this day and age people seem to think you can prevent failure. Not only can you not prevent it, you should embrace it. Learn to accept that failure will happen. Often spending your effort decreasing MTTR (mean time to recovery) as opposed to increasing MTBF (mean time between failures) is a much better investment. Failure is not a question of ‘if’ but a question of ‘when’.

Systems should be designed to be tolerant of failure. This is not easy, it’s not always cheap and it can be quite painful at first. Failure sucks. Especially as systems administrators, we tend to personalize a failure in our systems as a personal failure.

The best way to deal with failure is to make failure a non-issue. If it’s going to happen and you can’t prevent it, why stress over trying to prevent it? This absolutely doesn’t mean that you should do some level of due dilligence. I’m not saying that you should give up. What I’m saying is that you should design a robust system that handles failures gracefully and returns you to service as quickly as possible. It’s called fault TOLERANCE for a reason.

SLAs are not about servers

SLAs are in general fairly silly things. Before you get all twisted and ranty, let me clarify. SLAs have value but the majority of that value is to the provider of the SLA and not the person on the other end. SLAs are a lot like backup policies.

Look at it this way. I’m giving you an SLA for four nines of availability. That allows me to take around 50 minutes of downtime a year. Of course you assume that means 50 minutes spread over a year. What you fail to realize is that I can take all 50 minutes at once and still meet my SLA. Taking 50 minutes at one time is much more impacting than taking ten 5-minute outages. What’s worse is I can take that downtime not only in one chunk but I can take it at the worst possible time for you.

The other side of SLAs is that we tend to equate them with servers as opposed to services. The SLA is a Service Level Agreement. Not a Server Level Agreement. Services are what matters, not servers.

When you start to equate an SLA with a specific server, you’ve already lost.

Wrap up and part 2

As I said, this topic is too big to fit in one post. The next post will go into specifics about strategies and techniques that will hopefully give you ideas on how to make deploys less painful.