So it looks like Adrian Cockcroft finally spilled the beans on Netflix (no)Operations and sadly it reads like I expected.

Recently I saw a tweet from Kohsuke Kawaguchi that really got me frustrated.

So anyone who knows me knows I spend an inordinate amount of time bitching about Maven. I don’t know if it’s the type of companies I end up working for or what but I always seem to find myself ass-deep in Maven.

A few days ago I wrote up some notes on how we’re making Logstash better by adding ZeroMQ as an option for inputs and outputs. That night we decided to take it a bit further and add support for ZeroMQ as a filter plugin as well.

Every once in a while, a software project comes along that makes you rethink how you’ve done things up until that point. I’ve often said that ElasticSearch was the first of those projects for me. The other is ZeroMQ.

After yesterday’s post about load balancing logstash with AMQP and RabbitMQ, I got to thinking that it might be useful to show a smilar pattern with other inputs and outputs. To me this, is the crux of what makes Logstash so awesome. Someone asked me to describe Logstash in one sentence. The best I could come up with was:

Logstash is a unix pipe on steroids

I hope this post helps you understand what I meant by that

AMQP in Logstash is one of the most complicated parts of the workflow. I’ve taken it on myself, as the person with the most AMQP experience (both RabbitMQ and Qpid) to try and explain as much as need for logstash users.

Patrick DeBois hit me up with a common logstash design pattern that I felt warranted a full detailed post.

Warning: This is an image heavy post. Terminal screenshots are linked to larger versions

Requirements

• No lost messages in transit/due to inputs or outputs.
• Shipper-only configuration on the source
• Worker-based filtering model
• No duplicate messages due to transit mediums (i.e. fanout is inappropriate as all indexers would see the same message)
• External ElasticSearch cluster as final destination

I mentioned briefly in my previous post that I got quite a few people coming up to me after the panel and asking me for advice on monitoring.

This past week I had the awesome pleasure of participating in my first SoCal Linux Expo. As I later discovered, this was the 10th installment of this awesome event (hence the 10x).

The holidays are a busy time for me. I was hoping to get this written before the end of the year but it didn’t happen.