blog.lusis.org

So you wanna go on-prem do ya

Sun, 15 May 2016 00:00:00 +0000

If you run a successful SaaS platform, at some point someone is going to come to you with the question: can I run it myself? If you’re considering offering a private version of your SaaS, this post might be for you. At this point, I’ve worked for a few companies that are SaaS vendors. In most cases, we’ve also done some sort of private install. Drivers for private installs There are a lot of reasons people want private installs.

Review Site Reliability Engineering

Sun, 17 Apr 2016 00:00:00 +0000

This is a first for me. I’ve never reviewed a book before. There’s an uncomfortable amount of arrogance I feel in doing so but I thought it was important. Recently the Site Reliability Engineering book came out. This is the description directly from the O’Reilly site: {% blockquote %} In this collection of essays and articles, key members of Google’s Site Reliability Team explain how and why their commitment to the entire lifecycle has enabled the company to successfully build, deploy, monitor, and maintain some of the largest software systems in the world.

How Is Rundeck Formed

Thu, 07 Apr 2016 00:00:00 +0000

I’ve mentioned a few times across various tweets that I’m a big fan of Rundeck. Since a few folks have asked why/how we use it I figured a blog post is better What is rundeck? The way I typically describe rundeck is “a web interface for shell scripts”. That’s a bit unfair but that’s the simplest thing for people to grok. If I recall correctly most of the concepts of rundeck were spun out of a larger product from DTO Solutions called Control Tier.

Aws Apigateway for Fun and Profit

Wed, 09 Dec 2015 00:00:00 +0000

At we’re currently in the process of reconfiguring our monitoring, logging and alerting setup. Obviously this is something near and dear to my heart. At my previous employer we did everything in house due to various constraints. I’ve got a rich set of experiences in this space so my first inclination was to build not buy. However, due to OTHER constraints, building was not a practical solution at present. After much research and evaluation, we finally settled on the following stack:

Current Status

Mon, 12 Oct 2015 00:00:00 +0000

I just realized it’s been almost a year since I wrote a blog post

Terraform modules for fun and profit

Mon, 12 Oct 2015 00:00:00 +0000

I’ve been using it for a while on a smaller scale but a recent project at work had me putting it through its paces and working out the issues I had. A few of my issues in the past have revolved around: reusability composability lifecycle “safety” (i.e. don’t blow away an entire running stack accidentally) scope Scope One of the mistakes I made when first using terraform was trying to “do too much” with it.

A Few Things

Fri, 21 Nov 2014 00:00:00 +0000

I can’t believe I have to even write this but it seems that comprehension is hard and sadly I made the mistake of reading comments when someone posted my systemd redux article on HN. CoreOS Yes I know CoreOS uses systemd. It seems that ONE comment got taken entirely out of context. If you take it WITH the context of the whole post: If you must use Linux And we’ve agreed that systemd is here to stay in Linux Use a Linux “distro” where you don’t care that it’s running systemd CoreOS is not a traditional general purpose Linux.

Systemd Redux

Thu, 20 Nov 2014 00:00:00 +0000

I figured it was about time for a followup on my systemd post. I’ve been meaning to do it for a while but time hasn’t allowed. The end of Linux Some people wrongly characterized this as some sort of hyperbole. It was not. Systemd IS changing what we know as Linux today. It remains to be seen if this is a good or bad thing but Linux is becoming something different than it was.

Software Empathy

Sun, 19 Oct 2014 00:00:00 +0000

Over the past several months, I’ve written a lot of blog posts that were critical of various software. If it wasn’t clear from these (sometimes) incoherent ramblings, there was a common thread that bound them all together Why do we write software? I write software every now and then. When I do, if possible, I put it up on github for folks to consume. I’ve talked about this before. For me, being able to write code and somehow that code helps someone blows my mind.

End of Linux

Tue, 23 Sep 2014 00:00:00 +0000

I’ve done a lot of tweeting about systemd lately. My internal conscience constantly reminds me of John Allspaw saying that twitter is just pretty much perfect for snarky comments (paraphrase). Al Tobey asked me a good question: {% blockquote %} I honestly want to know why you dislike it so much. You clearly know wtf is going on. I haven’t heard a specific technical problem. {% endblockquote %} First mistake is thinking I know wtf is going on.

Berks

Tue, 15 Jul 2014 00:00:00 +0000

Note this post has comments disabled. This is a first for me but if someone wants to ask me anything else, there are multiple personal ways to contact me. Recently I posted a message to the chef-users mailing list. It was a hard post for me to write because I was attempting to be as tactful as possible about the issue. While I think it was the right move and some good came out of it, I now feel like I have to defend myself for some reason.

Feedback on Paas Realism

Sun, 22 Jun 2014 00:00:00 +0000

I’ve gotten quite a bit of constructive feedback on the PaaS realism post and I wanted to aggregate it here. OpenShift I was humbled to see a very awesome post from Luke Meyer who works on the OpenShift team. I think if you are going to deploy OpenShift, his post is very important to read. I wanted to respond to a few points specifically he made (and again say thank you for taking the time to respond/educate me)

Paas for Realists

Sat, 14 Jun 2014 00:00:00 +0000

I realize I was pretty down on PaaS the past couple of days. Lest I send the wrong message, I figure a clarification is in order Before we start, we should define some things. It’s always important to be on the same page: PaaS: Platform as a Service (no distinction between public or private). When talking about public, it’s usually Heroku. Private PaaS: A PaaS run “in-house”. I’m using “in-house” loosely here.

No Konami for Operations

Fri, 13 Jun 2014 00:00:00 +0000

up up down down left right left right b a select start I went on a bit of a rip today about all sorts of technology. I figured I should at least clarify some of it in long form. Vmotion/live migration technologies Vmotion is a scam. I have frequently said that only trivial workloads are safe for vmotion. Here’s the reasoning: live migration realistically requires the workload on the vm being migrated to be quiesced.

Omnibus Redux

Sun, 13 Apr 2014 00:00:00 +0000

I’ve been a pretty big proponent of omnibus. I still think it’s the right way to go but recent changes have removed the primary reason for recommending it I did a lot of evangelism for omnibus last year. Presentations, blogposts, a sysadvent article. It is/was a great tool however it no longer fits the primary usecase. Original workflow Originally the biggest benefit to omnibus (outside of the core of what it did) was the Vagrantfile it generated.

Stop Fighting Distros Part 2

Mon, 23 Sep 2013 00:00:00 +0000

I’m a pretty harsh person. My wife and I have this discussion all the time about how things I say get interpreted. As the communicator, this responsibility lies squarely on my shoulders. So before I start, I don’t “hate” distributions or the packaging format they use or the people doing the work. To this even of you who toil to track countless security reports or maintain some software package in an upstream repository because of your love for that software, here’s to you.

A Historical Compendium of Useless Shit

Tue, 17 Sep 2013 00:00:00 +0000

Yes it started on Twitter and ended up here. Circle of life and all that Fair warning: I’m not doing ANY research for this post. Normally I do but I’m doing this off-the-cuff so to speak. If I fuck up some specific thing, please feel free to correct. This is basically a personal perspective/history And it’s probably full of typos so there’s that So a discussion on twitter started because I brought up Mesos, Docker and OSv.

Go for System Administrators

Sun, 11 Aug 2013 00:00:00 +0000

{% blockquote %} If I never directly touch a Go concurrency primitive, I’m convinced I’m going to write all my cli apps with it just for ease of deployment. {% endblockquote %} This is something I said the other day. I figured it deserved a more detailed blog post. NKOTB Most people who know me professionally know two things about me: I’m fairly pragmatic and somewhat conservative about technology decisions I’m a language tourist This second one is something Bryan Berry attributed to me in an early FoodFight episode.

Devops the Title Match

Tue, 04 Jun 2013 00:00:00 +0000

I’ve noticed an annoying trend recently. I was content to ignore it for a while but now it’s getting almost stupid. That trend is the job title of “DevOps”. I came across an article tonight that confused the hell out of me. It was an interview. The outlet wasn’t a technical one per se but it was a technical interview none the less. This part is what confused me (highlights mine):

Smart Clients

Mon, 13 May 2013 00:00:00 +0000

Currently RICON|EAST is going on in NYC. Tom Santero and the whole Basho crew is doing an awesome job if the content available via the live stream and twitters is to be believed.

Future of Noah

Sun, 20 Jan 2013 00:00:00 +0000

This is probably the most difficult blog post I’ve had to write. What’s worse is I’ve been sitting on it for months. When I started Noah a few years ago, I had a head full of steam. I had some grand ideas but was trying to keep things realistic. I simply wanted a simple REST-ish interface for stashing nuggets of information between systems and a flexible way to notify interested parties when that information changed.

How We Vagrant

Mon, 17 Dec 2012 00:00:00 +0000

People may or may not have noticed but I’ve been largely offline for the past 4 weeks or so. This is because I’ve been in the middle of a pretty heavy redesign of a few key parts of our application stack. This also required me to learn Java so I’ve been doubly slammed. As part of that redesign, I worked on what we lovingly refer to internally as the “solo installer”.

What Production Means

Mon, 09 Jul 2012 00:00:00 +0000

This post is something that’s been brewing for a while. While it may sound targeted in tone, it’s more general than that. Let’s just call it an open letter to family, friends and coworkers around the world. One thing that I have the hardest time communicating to friends and family who aren’t in the IT industry is the concept of “production” and what it means to be on-call. Even coworkers have a hard time understanding what it means.

Why Ebs Was a Bad Idea

Fri, 15 Jun 2012 00:00:00 +0000

Since I just tweeted about this and I know people would want an explaination, I figured I’d short circuit 140 character hell and explain why I think EBS was the worst thing Amazon ever did to AWS. First time I’ve had to do this but: the following is my personal opinion and in no way reflects any policy or position of my employer A journey through time I remember when EC2 was first unleashed.

Monitoring Sucking Just a Little Bit Less

Tue, 05 Jun 2012 00:00:00 +0000

So it’s come to my attention that today is the “anniversary” of when I wrote my first “#monitoringsucks” blog post.

Configuration Drift and Next Gen Cm

Thu, 24 May 2012 00:00:00 +0000

It always starts with a tweet. However it normally doesn’t start with a tweet from Cliff Moon. {% blockquote %} Of all the problems to fix in chef or puppet, the diffusion and drift of state that occurs in idiomatic usage seems highest priority. {% endblockquote %} Now for sure what spawned this comment was something unrelated but it got me thinking. Oddly enough Tim Dysinger was either poking around in my head or just had the same idea:

It Sucks to Be Right

Tue, 20 Mar 2012 00:00:00 +0000

So it looks like Adrian Cockcroft finally spilled the beans on Netflix (no)Operations and sadly it reads like I expected. Netflix still does operations Regardless of what words Adrian uses, Netflix still does operations. John Allspaw summed it up pretty well in this tweet: and here are the things, he mentions: Metrics collection PaaS/IaaS evaluation/investigation Automation (auto-build, auto-recovery) Fault tolerance Availability Monitoring Performance Capex and Opex forecasting Outage response So what does Adrian get wrong?

Why You Should Stop Fighting Distro Vendors

Fri, 16 Mar 2012 00:00:00 +0000

Recently I saw a tweet from Kohsuke Kawaguchi that really got me frustrated. I’ve addressed this topic a bit before here. At the time it was addressing specifically dynamic languages. However the post that Kohsuke wrote (and the post that inspired it) have led me to a new line attitude. Don’t bother trying to get your packages into upstream vendor distros Wait. What? Let’s step back a sec Let me clarify something first.

Graphs in Operations

Tue, 06 Mar 2012 00:00:00 +0000

So anyone who knows me knows I spend an inordinate amount of time bitching about Maven. I don’t know if it’s the type of companies I end up working for or what but I always seem to find myself ass-deep in Maven. please note that I’m drifiting into deeply unfamiliar territory for me. Someone once told me the best way to learn about something is to write about it. Keep that in mind when making comments?

Zeromq and Logstash Part 2

Wed, 08 Feb 2012 00:00:00 +0000

A few days ago I wrote up some notes on how we’re making Logstash better by adding ZeroMQ as an option for inputs and outputs. That night we decided to take it a bit further and add support for ZeroMQ as a filter plugin as well. I’ve had a lot of people ask me what’s so hot about ZeroMQ. It’s hard to explain but I really would suggest you read the excellent zguide.

Zeromq and Logstash Part 1

Mon, 06 Feb 2012 00:00:00 +0000

Every once in a while, a software project comes along that makes you rethink how you’ve done things up until that point. I’ve often said that ElasticSearch was the first of those projects for me. The other is ZeroMQ. Edit and update Evidently my testing missed a pretty critical usecase - pubsub. It doesn’t work right now. Due to the way we’re doing sockopts works for setting topics. However we don’t have a commensurate setting on the PUB side.

Load Balancing Logstash With Amqp

Tue, 31 Jan 2012 00:00:00 +0000

AMQP in Logstash is one of the most complicated parts of the workflow. I’ve taken it on myself, as the person with the most AMQP experience (both RabbitMQ and Qpid) to try and explain as much as need for logstash users. Patrick DeBois hit me up with a common logstash design pattern that I felt warranted a full detailed post. Warning: This is an image heavy post. Terminal screenshots are linked to larger versions

Load Balancing Logstash With Redis

Tue, 31 Jan 2012 00:00:00 +0000

After yesterday’s post about load balancing logstash with AMQP and RabbitMQ, I got to thinking that it might be useful to show a smilar pattern with other inputs and outputs. To me this, is the crux of what makes Logstash so awesome. Someone asked me to describe Logstash in one sentence. The best I could come up with was: {% blockquote %} Logstash is a unix pipe on steroids {% endblockquote %}

Lowtech Monitoring With Jenkins

Mon, 23 Jan 2012 00:00:00 +0000

I mentioned briefly in my previous post that I got quite a few people coming up to me after the panel and asking me for advice on monitoring. I tweeted about this scenario not long after it happened but here’s the gist: {% blockquote %} I just need something simple to check on the status of a few jobs and run some SQL statements. I’m a DBA and I can’t get any help from my ops team.

Scale10x Recap

Sun, 22 Jan 2012 00:00:00 +0000

This past week I had the awesome pleasure of participating in my first SoCal Linux Expo. As I later discovered, this was the 10th installment of this awesome event (hence the 10x). The email and enStratus I got an email from Ilan Rabinovitch just as things were going down with me headed to enStratus. Since the event was going to be right around the time I started, I pretty much put it out of my mind.

2011 in Review

Mon, 02 Jan 2012 00:00:00 +0000

The holidays are a busy time for me. I was hoping to get this written before the end of the year but it didn’t happen. I can say, wihtout a doubt, that 2011 has been the most awesome year both professionally and personally in my life. And I have pretty much everyone else to thank for it. Some special shoutouts There are so many folks to thank for this year. I owe so many beers that I can’t keep count.

Github Trolling for Fun and Profit

Tue, 22 Nov 2011 00:00:00 +0000

Last Friday was a pretty crappy day. I’m a fairly active Twitter user. Patrick has joked that it’s as if I have Twitter wired directly to my brain. It’s not far from the truth. I like to engage people and normally Twitter is great medium for engaging folks. Unfortunately, the message size limit makes Twitter an imperfect medium for involved discussions. I know better but sometimes I forget. Anyway, last friday I realized near the end of the day that I had pretty much gone off the rails.

Deploy All the Things

Tue, 18 Oct 2011 00:00:00 +0000

This is part 2 in a post on deployment strategies. The previous post is located here

My previous post covered some of the annoying excuses and complaints that people like to use when discussing deployments. The big take away should have been the following:

The risk associated with deploying new code is not in the deploy itself but everything you did up to that point.
The way to make deploying new code less risky is to do it more often, not less.
Create a culture and environment that enables and encourages small, frequent releases.
Everything fails. Embrace failure.
Make deploys trivial, automated and tolerant of failure.

I want to make one thing perfectly clear. I’ve said this several times before. You can get 90% of the way to a fully automated environment, never go that last 10% and still be better off than you were before. I understand that people have regulations, requirements and other things that prevent a fully automated system. You don’t ever have to flip that switch but you should strive to get as close as possible.

Rollbacks and Other Deployment Myths

Tue, 18 Oct 2011 00:00:00 +0000

I came across an interesting post today via HN. I’m surprised (only moderately) that I missed it the first time around since this is right up my alley:

Why are you still deploying overnight?

I thought this post was particularly apropos for several reasons. I just got back from DevOpsDays EU AND I’m currently in the process of refactoring our deploy process.

I’m breaking this up into two parts since it’s a big topic. The first one will cover the more “theoretical” aspects of the issue while the second will provide more concrete information.

The Configuration Management Divide

Mon, 22 Aug 2011 00:00:00 +0000

wherein John admits he was wrong about something As I was checking my Github feed tonight (as I’ve been known to do on occassion), I saw an update for a project that I was watching. This project is, for all intents and purposes, a redo of Capistrano with some additional system management stuff on top. From the description: All together mixed to make your life easier. As mentioned before do is a fun mix of capistrano, rake, thor and brew.

Fun With Celluloid

Sat, 13 Aug 2011 00:00:00 +0000

warning! This is a really long post

In the course of rewriting the Noah callback daemon, I started to get really frustrated with EventMachine. This is nothing against EventMachine by any stretch of the imagination. I really like it.

What I was having issues with is making the plugin framework as dirt simple as possible. By using EM, I had no choice but to require folks to understand how EM works. This primarily meant not blocking the reactor. Additionally, through no fault of EM, I was starting to get mired in callback spaghetti.

Actors

I’ve mentioned several times before that I love the actor model. It makes sense to me. The idea of mailboxes and message passing is really simple to understand. For a while, there was project that implemented actors on top of EM called Revactor but it stalled. I started following the author (Tony Arcieri) on GitHub to see if he would ever update it. He did not but I caught wind of his new project and it was pretty much exactly what I was looking for.

Actors have a proven track record in Erlang and the Akka framework for Scala and Java uses them as well.

Celluloid

Migrating From Blogger

Sat, 13 Aug 2011 00:00:00 +0000

Many, many years ago I started blogging. It’s fun and I like to feel self-important by dumping my ideas. I enjoy writing so blogging is cool like that.

Over the past 12 years I’ve gone through various tools to help me do that (in no particular order):

PHPnuke
Postnuke
Moveable Type
Plone
CakePHP
rst2html vim plugin
Blogger
Wordpress
Homegrown shit

With the exception of one, all of these tools were simply a pain in the ass. They required lots of moving pieces, we’re bug ridden and security nightmares. The one tool I enjoyed the most?

rst2html in Vim

The only reason I used those tools was because I hated writing HTML. I hated dealing with styling. About 3 years ago I finally settled on using Blogger. Now I’m changing again.

Monitoring Sucks Watch Your Language

Thu, 21 Jul 2011 00:00:00 +0000

The following post is a recap of what was discussed in the 07/21/11 #monitoringsucks irc meeting

Before I get any further, I just want to thank everyone who attended, either in virtual person or in spirit. There have been so many awesome discussions going around this topic since we started. I am truly priviledged to interact with each and every one of you. I struggle to count myself a peer and I can only hope that I provide something in return.

I mentioned to someone today that I’m literally sick of the current landscape. I consider the current crop of monitoring solutions to be technologically bankrupt. The situation is fairly untenable at this point.

I just installed (after having a total loss of an existing Zenoss setup) Nagios again. I’m not joking when I say that it depressed the everliving hell out of me. The monitoring landscape simply has not kept up with modern developments. At first it was mildly frustrating. Then it was annoying. Now it’s actually a detriment.

Now that we have that out of the way….

Why Monitoring Sucks

Sun, 05 Jun 2011 00:00:00 +0000

(and what we’re doing about it)

About two weeks ago someone made a tweet. At this point, I don’t remember who said it but the gist was that “monitoring sucks”. I happened to be knee-deep in frustrating bullshit around that topic and was currently evaluating the same effing tools I’d evaluated at every other company over the past 10 years or so. So I did what seems to be S.O.P for me these days. I started something.

On Noah Part 4

Fri, 20 May 2011 00:00:00 +0000

This is the fourth part in a series on Noah. Part 1, Part 2 and Part 3 are available as well

In Part 1 and 2 of this series I covered background on Zookeeper and discussed the similarities and differences between it and Noah. Part 3 was about the components underneath Noah that make it tick.

This post is about the “future” of Noah. Since I’m a fan of Fourcast podcast, I thought it would be nice to do an immediate, medium and long term set of goals.

On Noah Part 3

Wed, 18 May 2011 00:00:00 +0000

This is the third part in a series on Noah. Part 1 and Part 2 are available as well

In Part 1 and 2 of this series I covered background on Zookeeper and discussed the similarities and differences between it and Noah. This post is discussing the technology stack under Noah and the reasoning for it.

A little back story

I’ve told a few people this but my original intention was to use Noah as a way to learn Erlang. However this did not work out. I needed to get a proof of concept out much quicker than the ramp up time it would take to learn me some Erlang. I had this grandiose idea to slap mnesia, riak_core and webmachine into a tasty ball of Zookeeper clonage.

On Noah Part 2

Tue, 17 May 2011 00:00:00 +0000

This is the second part in a series on Noah. Part 1 is available here

In part one of this series, I went over a little background about ZooKeeper and how the basic Zookeeper concepts are implemented in Noah. In this post, I want to go over a little bit about a few things that Noah does differently.

On Noah Part 1

Mon, 16 May 2011 00:00:00 +0000

This is the first part in a series of posts going over Noah

As you may have heard (from my own mouth no less), I’ve got a smallish side project I’ve been working on called Noah.