Five things learned using terraform to manage cloud infrastructure

HashiCorp’s terraform is a powerful and extensible tool for defining and creating cloud infrastructure in a repeatable way. At Olark we use it to manage a number of different environments on Google Cloud Platform. On the journey from imperative to declarative infrastructure we’ve learned a few things. Here are five that I feel are particularly important. What follows are entirely my own opinions.

Five things learned using terraform to manage cloud infrastructure

Using unbound for private DNS resolution in kubernetes

Workloads running in kubernetes pods commonly need access to services outside the cluster. In heterogeneous architectures where some services run in kubernetes and others are implemented on cloud VMs this often means resolving private DNS names that point to either specific hosts or to internal load balancers that provide ingress to groups of hosts.

Using unbound for private DNS resolution in kubernetes

LittleBigMouse solved my LittleBigProblem

It’s been awhile since I’ve come across a small tool for Windows that solves an irritating problem in an effective and transparent way. In fact I rarely even think about Windows software anymore, unless I’m playing a game. So much of what I do these days is done in linux or out in the cloud (in linux) that I’ve pretty much degraded my Windows expertise from superuser down to somewhere above n00b. So when I recently built a new computer and finally upgraded to Windows 10 from 7 Pro I ran into something highly irritating that I had no idea how to solve.

The issue was that I replaced my aging and amazingly long-lived Dell 2405FPW monitor with a new Dell 4k P2715Q. Placed alongside my other Dell, a 24″ U2412m, it makes for a nice, big workspace, with my main work on the screen in front of me, and lots of other stuff parked on the smaller screen to the left. The problem was mouse movement between the two. The P2715Q is running at 3840 x 2160, and the U2412M is at 1920 x 1200. Windows handles scaling content for the displays quite nicely, but with the 24″ display logically placed in the center of the 27″ display’s left edge, anytime you tried to move the mouse across the top or bottom of the larger screen and over to the smaller you’d run into a wall, and have to move the pointer vertically to find the hidden opening in the fence. Meh.

A little googling and I came across this gem:

LittleBigMouse is a lightweight windows service (14MB and negligible CPU) that handles adjusting the mouse position when you move between monitors with different resolutions. It’s small, it installs quickly, and it just works. It is still described as an alpha application by its author, so YMMV, but it has worked flawlessly for me. One thing I haven’t tried yet is gaming with it running. If I have any issues I’ll post an update, but since the app can be easily toggled on and off I don’t think that will be an issue regardless. Props to the author. You can download it at:

Bot Diary, part 1

Chatbots are the hot topic of conversation among developers and investors alike right now. Advances in natural language processing (NLP) and wide adoption of a few messaging platforms like Facebook Messenger, Slack, Snapchat and Whatsapp have presented developers with a unique opportunity. Would you rather navigate to, say, Nordstrom’s website, find the shoe section, figure out the categories, search, choose a pair, add them to a cart, etc., or would you rather message the Nordstrom’s bot: “I need a pair of brown dress shoes, size 10 1/2, leather, and I like Johnson and Murphy” and have it respond with a selection which you can order from simply by tapping? Yeah, thought so. Me too.

Chatbots have a number of advantages. The chat platform (FB for example) knows who you are so that whole create account/auth business can be dispensed with. You can save your personal details like shipping address and payment methods once, and have them shared with companies whose bots you interact with. You don’t have to learn the organization of a new website with a unique information flow. You don’t even have to remember their domain name, and there are still plenty of opportunities for branding and positive customer interactions.

So bots are cool. How do you build one? I’ve just embarked on some experiments to find that out, and I will be posting a little bit here and there about the experience. I chose to build a bot for Facebook Messenger for some simple reasons: they have 900 million users, and it’s Facebook. Everyone uses Facebook, so bots developed on their framework should have very broad reach. I had never done any development on FB so the first thing I had to do was register as a developer. That literally took one button click and I was in the developer console. First step is to create a page and an app. The page profile photo and title are used as the public identity of the bot once it is deployed, but they don’t have to be public during development. I created a page titled MarkBot and left it unpublished. Next I created the app, skipping the canned templates and just proceeding to the console to create an App ID. Facebook interacts with apps via webhook callbacks, so the first thing I had to do was set one up and verify it. Verification consists of FB calling the hook with a challenge param that you need to send back, so the upshot was I needed to deploy some code somewhere to handle this request.

I have an AWS account so my first thought was just to write a quick flask app and slide the code into a docker container, deploy it to a free tier instance, set up an ELB… yeah even with docker this is all still a lot of work to handle a web callback. I only had to do it once and then I would write makefiles to automate the whole thing, but still… had to be a better way. That better way is AWS Lambda. I heard about lambda when it first came out, of course, but I had given no further thought to it since. I work on real systems, with containers, and servers and stuff! Why would I want to simply define a python function, attach an endpoint to it, and be able to respond to http requests… er, yeah. Turns out lambda is pretty damn awesome.

At a high level you write some code in java, python, or node.js. You can create the code right in the lambda console, or create it as an external package that you upload. For the purposes of this simple verification hook I did the former, but will definitely transition to the latter for the bot message handlers, so I will have more to say about that in future posts. Your code gets called with some defined parameters either when you run it manually, or when certain events occur. The definition of “event” includes a whole bunch of interesting sources, such as S3 buckets changing, CloudWatch logs or DynamoDB streams matching a content filter, RDS database tables getting updated, etc. It’s a long list that includes most of AWS’s services, including the one I was interested in: API Gateway.

API Gateway is basically a programmable http request handler that you can use to set up an endpoint path and define the methods it supports. When you’re done you get a url that you can use to hit the gateway. On the back end API Gateway hands off the requests it receives to either another http service, an AWS service proxy or a lambda function. So that is the other big piece of the puzzle: API gateway + lambda function == instant http service. Getting to that realization was the easy part. As with most of Amazon’s AWS offerings there is a lot of complexity lurking just beneath the surface of the busy-looking console and rapidly outdated web documentation. One confusing thing is that you can approach this build from two independent directions: you can create a lambda function and use the lambda console to create an API Gateway endpoint to attach it to; or you can create an API Gateway endpoint and use that console to attach it to an existing lambda function. I imagine both approaches will get you to the same place. I chose to perform the setup in the lambda console, but still had to jump over to the API Gateway console a couple of times.

The two main things that slowed me down were: understanding the deploy cycle for the API, and mapping querystring parameters into my lambda function. On the deploy point it all boils down to one important fact: changes you make to the API endpoint, such as the mapping changes I’ll touch on below, don’t take effect until you deploy the API. You can change and test but until you deploy the new stuff isn’t visible to consumers. That caused me a few rounds of puzzling over Cloudwatch logs (did I mention that lambda logs to Cloudwatch? Slick.). Once I realized what deploy did it all made sense.

The parameter mapping thing is a little more complex, and I don’t have space left to go into it in detail here. The basic point is that you have API Gateway receiving an http request, and your lambda function expecting an event object (and a context object but that’s another topic), and so you have to tell API Gateway how to map the path params, headers, querystring, form variables, or body into the event. You do this in the method execution section of the endpoint method. There are four steps in the processing pipeline where you can affect how requests are handled: method request, integration request, integration response and method response. Since the Facebook webhook verification uses querystring args and expects a json response I had to make three changes here: add the two querystring params to the method request; add a mapping template to the integration request to transform the params into a json object; and add a mapping to the integration response to transform the string I returned into an application/json response.

Once that was done the call from Facebook succeeded and my webhook was verified. Next step is to understand the interaction model between the chatbot and my API. More on that after I get there.

Validate json models with swagger and bravado

If you design and implement APIs for a living then you’re probably already familiar with swagger. Swagger is a specification for defining API endpoints and the model objects they transact. Once you have a swagger definition of your API written in either json or yaml you can do quite a few useful things with it: generate HTML documentation on the fly; generate client and server code; generate a postman collection for endpoint testing; and more to the point of this piece, use the spec to validate incoming objects at request time, resulting in meaningful error responses to clients.

Validate json models with swagger and bravado

Ten tips for debugging Docker containers

Containers are awesome, but sometimes it can feel like your code has been shut up in a black box, stuck off in the cloud where you can’t see what it’s doing. When your app runs the way it’s supposed to this isn’t a problem: containers come, containers go, everyone’s happy. But then there’s that moment when you push a new version of your image and check the cluster status only to see it crashlooping. What’s going on? In my latest post on I list ten tips that will make debugging your docker containers easier.

Ten tips for debugging Docker containers

Comparing Amazon’s Elastic Container Service and Google Kubernetes

Over the last two years docker containers have completely changed the way I look at software builds, deployment and infrastructure. Orchestration platforms like Amazon’s Elastic Container Service and Google Container Engine, built on their open source kubernetes framework, are closing the gap between what we have been able to do with software in containers, and what we want to be able to do with infrastructure and deployments in the cloud. My first piece on is a comparison of these two “container orchestration” platforms.

Comparing Amazon Elastic Container Service and Google Kubernetes

Using rsyslog to ship logs to a remote logstash server

Wanted to give author dreed a shout out for a truly awesome tutorial on using rsyslog to format and ship logs to a remote logstash server listening on a tcp endpoint. Followed this last night and was able to get our logs from haproxy over to logstash and into elasticsearch all running in Google’s kubernetes container engine. Nice work, dreed. I would have commented on the site but for disqus and having to sign in.