Aws Apigateway for Fun and Profit

At we’re currently in the process of reconfiguring our monitoring, logging and alerting setup.

Obviously this is something near and dear to my heart. At my previous employer we did everything in house due to various constraints. I’ve got a rich set of experiences in this space so my first inclination was to build not buy. However, due to OTHER constraints, building was not a practical solution at present.

After much research and evaluation, we finally settled on the following stack:

I’m not going into reasons for switching in this post or details about WHY these providers were chosen. This is to discuss a specific AWS service and how it can really change the nature of ChatOps and more.


Our monitoring and metrics provider, SignalFx, is still building out its integrations. They have a rich set already and are iterating very quickly but coupled with the migration to OpsGenie, we needed to work out a solution for wiring the two together. Using the power of google, I came across a great post from R.I. Pienaar about leveraging Lambda and the AWS API Gateway products.

In his post, he didn’t really go over the API Gateway configuration so I figured I would document what I “figured out” for others.

The flow

The above post covers most of it but the general idea is:

Now you might wonder why go through all this hassle? OpsGenie can create email integrations and SignalFx can send alerts to email address.

It’s about context.

When email alerts come in to OpsGenie, they’re just “pings”: {% blockquote %} Hey this thing sent us an email and here’s the subject. Log in to the website to see the body {% endblockquote %}

Frankly there’s not much else they can do. SignalFx will also send another email when an alert auto-clears but to OpsGenie, that’s another “ping”. It’s unrelated to the previous email.

Hipchat alerts from OpsGenie

As you can see above this could get painful and I already have issues with “alert fatigue”.

Additional, even if you DID log into the OpsGenie website to look at the details, there’s really not much there:

Emails in OpsGenie

Yeah that’s helpful…

So we need to accomplish two things:

The end result, gives us this:

Hipchat alerts from OpsGenie SignalFx Metadata in OpsGenie OpsGenie Alert History

and that’s MUCH more useful.

How we did it

AWS API Gateway is a PRETTY intimidating thing. There’s lots of very specific terminology and frankly I found the docs not so useful. The general idea is this:

There are also concepts like “stages” which frankly were a bit useless in my case. In the end, after you get all of this wired up, you’ll be given a url that looks something like this:

https://<random id>.execute-api.<region><stage name>/<resource name>

and that’s your webhook url.


In the case of SignalFx, a webhook post looks like this:

  "status":"too high",
  "rule":"500 errors over 10",
  "detector":"ELB 5xx Detector"

which translates to the following model:

  "$schema": "",
  "title": "signalfx-webhook-model",
  "type": "object",
  "properties": {
    "incidentId": {"type":"string"},

The lambda function

I’m still cleaning up the banged out code for the Lambda function to post on github however here’s the relevant bits from the opsgenie.js file I added to R.I.’s code:

  sfxNotification: function(event, config) {
	var result = {
		apiKey: config.api_key,
		alias: event.incidentId,
		note: event.detectorUrl,
		message: "Alert: "+event.rule+" ("+event.detector+")",
description = "Current value
		details: event,
Tags = ["`ok`", "`too high`", "`too low`", "`anomalous`"