Every once in a while, a software project comes along that makes you rethink how you’ve done things up until that point. I’ve often said that ElasticSearch was the first of those projects for me. The other is ZeroMQ. Edit and update Evidently my testing missed a pretty critical usecase - pubsub. It doesn’t work right now. Due to the way we’re doing sockopts works for setting topics. However we don’t have a commensurate setting on the PUB side.

AMQP in Logstash is one of the most complicated parts of the workflow. I’ve taken it on myself, as the person with the most AMQP experience (both RabbitMQ and Qpid) to try and explain as much as need for logstash users. Patrick DeBois hit me up with a common logstash design pattern that I felt warranted a full detailed post. Warning: This is an image heavy post. Terminal screenshots are linked to larger versions

After yesterday’s post about load balancing logstash with AMQP and RabbitMQ, I got to thinking that it might be useful to show a smilar pattern with other inputs and outputs. To me this, is the crux of what makes Logstash so awesome. Someone asked me to describe Logstash in one sentence. The best I could come up with was: {% blockquote %} Logstash is a unix pipe on steroids {% endblockquote %}

I mentioned briefly in my previous post that I got quite a few people coming up to me after the panel and asking me for advice on monitoring. I tweeted about this scenario not long after it happened but here’s the gist: {% blockquote %} I just need something simple to check on the status of a few jobs and run some SQL statements. I’m a DBA and I can’t get any help from my ops team.

This past week I had the awesome pleasure of participating in my first SoCal Linux Expo. As I later discovered, this was the 10th installment of this awesome event (hence the 10x). The email and enStratus I got an email from Ilan Rabinovitch just as things were going down with me headed to enStratus. Since the event was going to be right around the time I started, I pretty much put it out of my mind.

The holidays are a busy time for me. I was hoping to get this written before the end of the year but it didn’t happen. I can say, wihtout a doubt, that 2011 has been the most awesome year both professionally and personally in my life. And I have pretty much everyone else to thank for it. Some special shoutouts There are so many folks to thank for this year. I owe so many beers that I can’t keep count.

Last Friday was a pretty crappy day. I’m a fairly active Twitter user. Patrick has joked that it’s as if I have Twitter wired directly to my brain. It’s not far from the truth. I like to engage people and normally Twitter is great medium for engaging folks. Unfortunately, the message size limit makes Twitter an imperfect medium for involved discussions. I know better but sometimes I forget. Anyway, last friday I realized near the end of the day that I had pretty much gone off the rails.

This is part 2 in a post on deployment strategies. The previous post is located here

My previous post covered some of the annoying excuses and complaints that people like to use when discussing deployments. The big take away should have been the following:

• The risk associated with deploying new code is not in the deploy itself but everything you did up to that point.
• The way to make deploying new code less risky is to do it more often, not less.
• Create a culture and environment that enables and encourages small, frequent releases.
• Everything fails. Embrace failure.
• Make deploys trivial, automated and tolerant of failure.

I want to make one thing perfectly clear. I’ve said this several times before. You can get 90% of the way to a fully automated environment, never go that last 10% and still be better off than you were before. I understand that people have regulations, requirements and other things that prevent a fully automated system. You don’t ever have to flip that switch but you should strive to get as close as possible.

I came across an interesting post today via HN. I’m surprised (only moderately) that I missed it the first time around since this is right up my alley:

Why are you still deploying overnight?

I thought this post was particularly apropos for several reasons. I just got back from DevOpsDays EU AND I’m currently in the process of refactoring our deploy process.

I’m breaking this up into two parts since it’s a big topic. The first one will cover the more “theoretical” aspects of the issue while the second will provide more concrete information.

wherein John admits he was wrong about something As I was checking my Github feed tonight (as I’ve been known to do on occassion), I saw an update for a project that I was watching. This project is, for all intents and purposes, a redo of Capistrano with some additional system management stuff on top. From the description: All together mixed to make your life easier. As mentioned before do is a fun mix of capistrano, rake, thor and brew.