LittleBigMouse solved my LittleBigProblem

It’s been awhile since I’ve come across a small tool for Windows that solves an irritating problem in an effective and transparent way. In fact I rarely even think about Windows software anymore, unless I’m playing a game. So much of what I do these days is done in linux or out in the cloud (in linux) that I’ve pretty much degraded my Windows expertise from superuser down to somewhere above n00b. So when I recently built a new computer and finally upgraded to Windows 10 from 7 Pro I ran into something highly irritating that I had no idea how to solve.

The issue was that I replaced my aging and amazingly long-lived Dell 2405FPW monitor with a new Dell 4k P2715Q. Placed alongside my other Dell, a 24″ U2412m, it makes for a nice, big workspace, with my main work on the screen in front of me, and lots of other stuff parked on the smaller screen to the left. The problem was mouse movement between the two. The P2715Q is running at 3840 x 2160, and the U2412M is at 1920 x 1200. Windows handles scaling content for the displays quite nicely, but with the 24″ display logically placed in the center of the 27″ display’s left edge, anytime you tried to move the mouse across the top or bottom of the larger screen and over to the smaller you’d run into a wall, and have to move the pointer vertically to find the hidden opening in the fence. Meh.

A little googling and I came across this gem:

LittleBigMouse is a lightweight windows service (14MB and negligible CPU) that handles adjusting the mouse position when you move between monitors with different resolutions. It’s small, it installs quickly, and it just works. It is still described as an alpha application by its author, so YMMV, but it has worked flawlessly for me. One thing I haven’t tried yet is gaming with it running. If I have any issues I’ll post an update, but since the app can be easily toggled on and off I don’t think that will be an issue regardless. Props to the author. You can download it at:

Ten tips for debugging Docker containers

Containers are awesome, but sometimes it can feel like your code has been shut up in a black box, stuck off in the cloud where you can’t see what it’s doing. When your app runs the way it’s supposed to this isn’t a problem: containers come, containers go, everyone’s happy. But then there’s that moment when you push a new version of your image and check the cluster status only to see it crashlooping. What’s going on? In my latest post on I list ten tips that will make debugging your docker containers easier.

Ten tips for debugging Docker containers

Comparing Amazon’s Elastic Container Service and Google Kubernetes

Over the last two years docker containers have completely changed the way I look at software builds, deployment and infrastructure. Orchestration platforms like Amazon’s Elastic Container Service and Google Container Engine, built on their open source kubernetes framework, are closing the gap between what we have been able to do with software in containers, and what we want to be able to do with infrastructure and deployments in the cloud. My first piece on is a comparison of these two “container orchestration” platforms.

Comparing Amazon Elastic Container Service and Google Kubernetes

Do I want Windows 10?

I sat down in my office this morning and found a new icon in the system tray notification area of my Windows 7 Enterprise desktop. Right-clicking it showed four options, none of which was “Exit.” Left-clicking it brought up this window…

I’m not sure when Microsoft installed this program, but it must have been last week when the Tuesday update batch hit. None of the actions in the window appealed to me this morning. I don’t know what it means to “reserve” my free upgrade, and I am still not sold on Windows 10. Since I couldn’t get rid of the program (at least not without hunting down the process and killing it, after which it would in all likelihood return on the next restart) I used the notifications manager to hide it.

It’s not that I’m uninterested in Windows 10. On the contrary I’ve been very interested in all of Microsoft’s most recent decisions with their ecosystem, including the move to open source .NET and make the CLR portable across systems. And the truth is that I was a Microsoft platform developer for twenty years. It’s safe to say I have never before been two whole versions behind the current release of the operating system. Why, then, am I still running Windows 7? Should I upgrade to Windows 10?

I’m still running Windows 7 because Windows 8 sucked hard, in my opinion. I installed it. I used it. My wife has it on her laptop and I tried to help her a bit during the acclimation phase, and I thought it was horrible. I’m a software developer and the interface of Windows 8 did nothing other than make everything I already knew how to do cumbersome and difficult, with no compensating benefit that I could detect. I don’t have two 24″ monitors so I can cover them with big tiles.

I felt pretty much the same way about Windows Vista, but for different and more technical reasons. Windows Vista just wasn’t ready. Windows 8 just didn’t make any sense, ready or not. However Windows Vista became Windows 7, easily the best version of the OS that I have used, and a pretty high bar which Windows 8 certainly did not manage to leap over, at least in my view. Now that Windows 8 is becoming Windows 10 is it time to switch?

Even without thinking too deeply about it I’m biased against. The main reason is the deprecation of Windows Media Center. Microsoft is giving up on the desktop entertainment functions of the PC, apparently. But for me WMC has long been my Netflix solution of choice. It looks and works great and my ten year-old Firefly RF remote works awesomely with it. I realize, unfortunately, that this technology is aging, so maybe I should just get over it. What else does Windows 10 offer me, other than correcting the things people saw as mistakes in Windows 8? Let’s have a look at their news release. What are the big features a professional developer like me should care about?

Cortana, the world’s first truly personal digital assistant helps you get things done. Cortana learns your preferences to provide relevant recommendations, fast access to information, and important reminders. Interaction is natural and easy via talking or typing. And the Cortana experience works not just on your PC, but can notify and help you on your smartphone too.

Awesome, a digital assistant. I don’t really need one on my desktop. It might be useful in a mobile context, but I don’t run Windows Phone and I am not likely to anytime soon. Then again my kids all have iPhones with Siri, and I don’t hear them talking to their phones. As far as I know they don’t use Siri for anything. Maybe if I went to San Jose I’d encounter lots of people asking their phones to do things, but I just don’t see it happening around here, at least not in public.

Microsoft Edge, is an all-new browser designed to get things done online in new ways, with built-in commenting on the web – via typing or inking — sharing comments, and a reading view that makes reading web sites much faster and easier. With Cortana integrated, Microsoft Edge offers quick results and content based on your interests and preferences. Fast, streamlined and personal, you can focus on just the content that matters to you and actively engage with the web.

Well, web apps are still a big part of what I do so I will be getting familiar with Edge whether I want to or not. Am I excited about getting Windows 10 so Edge can replace my current browser? Let’s see… it has Cortana integrated… w00t. See above. And a new “reading view!” I’ll be keeping an eye on Edge in terms of standards compliance and performance, of course, and I will be testing web apps on it, but if those are the big draw features I’ll continue to bounce back and forth between Chrome and Firefox as one or the other alternately pisses me off.

Office on Windows: In addition to the Office 2016 full featured desktop suite, Windows 10 users will be able to experience new universal Windows applications for Word, Excel, and PowerPoint, all available separately. These offer a consistent, touch-first experience across a range of devices to increase you productivity …

I’m not even going to bother with that whole quote. I love Windows, but if there was ever a good reason to hate Windows it would have to be related to Office somehow. From a word processor so horribly complicated that no living human can enumerate more than 10% of the feature set to an email cum personal productivity tool that set a new standard for how long a legacy code base can continue to be crammed into ever more ill-fitting skin, there is literally nothing about Office that I like. Been using Google Docs for years and the words “touch-first experience” in the quote above certainly don’t give me any reason to rethink that choice. Yuck.

Xbox Live and the integrated Xbox App bring new game experiences to Windows 10. Xbox on Windows 10 brings the expansive Xbox Live gaming network to both Windows 10 PCs and tablets. Communicate with your friends on Windows 10 PCs and Xbox One – while playing any PC game.

Ok, that’s fine. I’m not a console gamer, and I don’t own an XBox, but this is still pretty cool and if I were a console gamer, or was willing to purchase an XBox to replace Windows Media Center, this might be exciting.

New Photos, Videos, Music, Maps, People, Mail & Calendar apps have updated designs that look and feel familiar from app to app and device to device.  You can start something on one device and continue it on another since your content is stored on and synched through OneDrive.

Wow, now this is what I was waiting for. Not. There are better services available now for all this stuff. Definitely will be meaningful to the thousands of people on Windows Phone, though. Next.

Windows Continuum enables today’s best laptops and 2-in-1 devices to elegantly transform from one form factor to the other, enabling smooth transitions of your tablet into a PC, and back. And new Windows phones with Continuum can be connected to a monitor, mouse and physical keyboard to make your phone work like a PC.

I’m not writing this off. Device convergence has been talked about for a long time, and I certainly hope somebody is able to make it happen. The vision of being able to use one device across different inputs and display form factors is compelling. But the problem Microsoft has is they don’t have the dynamic mobile ecosystem to pull this off and make it relevant to lots of people. Jury is out, but give them credit for the attempt anyway.

Windows Hello, greets you by name and with a smile, letting you log in without a password and providing instant, more secure access to your Windows 10 devices. With Windows Hello, biometric authentication is easy with your face, iris, or finger, providing instant recognition.

Same thing. Not writing it off, despite the ridiculous name. Also not relevant to me sitting here at my desk. So like the Continuum feature it is interesting, but provides no motivation to move me off 7 for my work system.

Windows Store, with easy install and uninstall of trusted applications, supported by the broadest range of global payment methods.

When lots of people are running Windows on their phones this will be a very compelling offering. Which is just a bit like saying that when I win the Powerball I will be living in a beachfront home in Santa Barbara. I understand Microsoft’s dilemma, and I don’t envy them. They have to be relevant to mobile, even though they are barely relevant to mobile. You can’t build the future of your business on desktop computing, even though most of the people who use your current product use it at a desktop, for work.

It’s a tough situation, but the reality for me is that reading through this feature list doesn’t make me wonder whether I should upgrade to Windows 10 on my desktop: it makes me wonder why I am still running Windows on my desktop at all. I also have an Ubuntu development box and switching would be pretty painless for me. The answer is: games and Steam, and to a lesser extent a huge file called outlook.pst. I play games like Battlefield 4 and Planetside 2, and I like a mouse and keyboard for that. And I have ten years of history in my outlook file. I never look at it, but for some reason I haven’t been able to just delete it. If I ever get to that point and also stop playing shooters (which I should do since I basically just get slaughtered by teenagers) that will probably be it for Windows.

So the answer to the original question I posed to myself in the title appears to still be “no.” I’ll be giving Windows 10 a look in a VM at some point, and at some other point, hopefully still well into the future, I am going to be faced with the fact that I just can’t continue running Windows 7. I suspect that what will happen then is my Ubuntu and Windows machines will switch roles. Instead of having Windows drive my two monitors and using NX to access Ubuntu in a window it will be the other way around, and I will be switching into Windows every now and then just to play a game, or to actually try and find something in that huge Outlook file.

Microsoft .NET coreclr on Ubuntu

I’ve said for years that .NET should be open sourced and cross-platform, and that development is finally taking place. Today Microsoft announced a preview of coreclr running on Ubuntu, and this evening I was able to build it on Ubuntu 14.04 running in a docker container.

HelloWorld.exe on Ubuntu 14.04 in a docker container

There were quite a few steps involved, and as this is a preview there is also quite a bit still missing. Notably compilation of managed code on linux (using roslyn) is not available, so after building the coreclr on Ubuntu you have to pop into Windows and build it, and the corefx libraries, then copy a bunch of crap over to your linux system. You also still need mono for some callable wrappers, and nuget to grab a bunch of dependent packages. Still, all in all it feels fairly historic, and coming on the heels of Microsoft’s announcement of their new cross-platform code editor I’d say it’s been a good week for them and those of us who are fans of their tools (whatever platform we find ourselves working on).

If you want to try it yourself the ubuntu:14.04 docker image is a good starting point. Note that you’ll want to install both wget and curl before following the instructions to get coreclr running.


IBM Research report on performance of Linux containers

At Knowledge In Practice we were pretty early adopters of Docker, and after more than six months of use nearly all of our production services are now deployed to Amazon’s EC2 as linux containers. While the lower overhead of containers was a draw,  as a small team the main benefits for us have been ease of deployment and increased environmental stability due to the use of Docker build files to declaratively specify the content of each service’s run-time environment. Launching a new instance of a service is literally as easy as adding one line to the cloudinit script for the instance, then running “docker pull” to get the image we want, and “docker run” to get the container going. Those steps could easily be automated as well. It’s a workflow that’s hard to beat.

Late last month IBM Research released a paper (PDF) comparing the performance of linux containers vs. traditional types of hardware and software virtualization. Not surprisingly containers fare quite well, although the paper notes that both VMs and containers need to be fine-tuned for high I/O workloads. Section 2.3 of the paper provides an excellent quick overview of how containers are implemented in linux using kernel namespaces and cgroups, and in fact I found that part of the document more valuable than the performance comparisons. Well worth a scan, at least, if you have an interest in this technology.

Bunch of Yahoos

Having a great set of developer tools can help make your platform ubiquitous and loved. When Microsoft first launched its Developer Network it revolutionized the way programmers got access to their operating systems, tools, and documentation. They successfully migrated that set of resources to the web and it remains invaluable for Windows developers. If you’ve ever set up access to a Google API, or deployed a set of EC2 resources on Amazon’s AWS cloud infrastructure, you know how impactful a clean, functional web interface is.

By the same token, a clunky, dysfunctional interface can make a platform loathed and avoided. Take Yahoo’s Developer Network and their BOSS premium APIs, for example.

We’re working on a system that needs to geolocate placenames in blocks of free text. This isn’t a trivial problem. There’s been a lot of work done on it, and we’ve explored most of it. During that exploration we wanted to try Yahoo’s PlaceSpotter API. It’s a pay service, but if it works well the cost could be reasonable, and just because we have built our system on free and open-source components doesn’t mean we won’t pay for something if it improves our business.

With that in mind I set out to test it, just as I had previously set out to test Google’s Places API. In that experiment I simply created a Google application under my user name, grabbed the creds, and wrote a python wrapper in about five minutes to submit text queries and print out the results. That’s my idea of a test.

In order to test Yahoo’s PlaceSpotter I needed access to the BOSS API. To get access to the BOSS API I needed to create a developer account. Ok, that’s not an issue. I will happily create a developer account. To create a developer account, it turns out, requires a bunch of personal info, including an active mobile number. Ok, I’ll do that too, albeit not quite as happily because all I want to do is figure out if this thing is worth exploring.

I should note that there is a free way to get to the same Geo data that BOSS uses, and the same functionality, through YQL queries. Maybe I was shooting myself in the foot right from the beginning, but I had no experience with YQL, I needed to move quickly and make some decisions, and I just wanted an API I could fling http requests at. Since the billing is per 1000 queries I had no problem paying for the first 1000 to test with. Not that big a deal.

After creating the account, during which I had to change the user name four times because of the cryptic message that it was “inappropriate” (no, I was not trying to use b1tch as a user name, or anything else objectionable), I finally ended up on a control panel-ish account dashboard. There I could retrieve my OAUTH key (ugh) and other important stuff, and activate access to the BOSS API.

I clicked the button to activate the API, and the panel changed to display another button labelled “BOSS Setup.” Next to that was a red rectangle stating that access to BOSS was not enabled because billing had not been set up. It wasn’t obvious to me that in order to set up billing you have to click “BOSS Setup.” I assumed billing would be at the account level. Well there are billing options at the account level! They’re not the ones you want, and unless the verbiage triggers some warnings as it did for me you might just go ahead and set up your credit card there, only to find it didn’t help.

Not to be deterred, I googled a bit and found that, indeed, I had to click “BOSS Setup.” It would have been nice if they had mentioned that in the red-colored billing alert. So I clicked, entered my login again because, you know, I was using the account control panel and so obviously might be an impostor, and ultimately found the place to enter my payment information. Once that was done, submitted, and authorized I received a confirmation and invoice in my email. Now, I could finally toss a few queries at the API.

Except no. When I returned to the account dashboard the same red-colored billing alert appeared. No access. I am a patient man, some of the time. Maybe their systems are busy handshaking. I waited. Nope. I waited some more. Nope. I gave up and waited overnight, and checked again this morning. Nope. Ok, dammit, I’ll click the “BOSS Setup” button again. I do that and what they show me is the confirmation page for my order again, with an active submit button. But wait… I got an invoice? Was I charged? Will I be charged again? Should I resubmit, or email Yahoo, or call my bank?

Maybe I should just not use the API. Oh, and did I mention that they have a “BOSS Setup” tutorial? It’s a download-only PDF. And 2/3 of it is about setting up ads.

Adding swap to an EBS-backed Ubuntu EC2 instance

Another one of those recipes I need to capture for future use. We were running a bunch of memory intensive processes on a medium (m2) Ubuntu instance on EC2 yesterday, and things were not going well. It looked like some processes were dying and being restarted. Poking around in the kernel messages we came across several events reading:

"Out of memory: Kill process ... blah blah"

Well, damn. There are less than 4GB of usable RAM on a medium, but it should be swapping, right? Wrong. We checked free and there was no swap file. That was my screw up, since I set up the instance and did not realize it had no swap. Turns out that what Amazon considers to be memory constrained instances (smalls and micros, for example) get some swap in the default config, but apparently mediums and larges do not. We decided to rebuild the instance as a large to get more RAM and also another processing unit, and add some swap at the same time.

I wasn’t able to figure out how to configure swap space in the instance details prior to launch, so I searched around and put together this recipe for adding it post-launch. This applies to Ubuntu EBS-backed instances. It should work generally for any debian-based distro, I think. If your instance is not EBS backed you can use many of the same techniques, but you’ll have to figure out the deltas because we don’t use any non-EBS instances. Anyway, to the details.

EBS-backed instances have their root volume on EBS, which is what EBS-backed means. But you don’t want to put swap space on EBS. EBS use incurs I/O charges, and although they are very low, they aren’t nothing. If you were to locate swap on EBS then there would at least be some chance of some process going rogue and causing a lot of swapping and associated costs. Not good.

Fortunately, EBS-backed instances still get so-called “ephemeral” or “instance storage” at launch. Unlike EBS this storage is physically attached to the system, and also unlike EBS it goes away when the instance is stopped (but not when it is rebooted). That’s why it is called “ephemeral.” Something else that is ephemeral is the data stored in a swap file, so it seems like instance storage and swap files are made for each other. Good news: every EBS-backed instance on EC2 still gets ephemeral storage by default. It consists of either a 32GB SSD, or two 320GB magnetic disks, depending on your choices. For instances that do not get swap allocated by default, this storage is mounted at /mnt after startup (either the SSD, or the first of the magnetic disks; if you want the second disk you need to mount it manually).

So before you start make sure your config matches what I’ve described above, i.e. you have an EBS-backed instance with ephemeral storage mounted at /mnt (which you can confirm with the lsblk command), and no swap space allocated (which you can confirm with the swapon -s command).

First thing to do is create a swapfile in /mnt. The one thing I am not going to do is opine on what size it should be, because there is a lot of info out there on the topic. In this example I made the swap 2GB.

# sudo dd if=/dev/zero of=/mnt/swapfile bs=1M count=2048

This command just writes two gigs worth of 0’s to the file ‘swapfile’ on /mnt. Next make sure the permissions on this new file are set appropriately.

# sudo chown root:root /mnt/swapfile
# sudo chmod 600 /mnt/swapfile

Next use mkswap to turn the file full of 0’s into an actual linux swapfile.

# sudo mkswap /mnt/swapfile
# sudo swapon /mnt/swapfile

The first command formats the file as swap space (I actually have no idea what it does at a low level, so ‘format’ might be a wildly incorrect term to use), and the second sets it up as the system swap file. Now you should update fstab so that this swap space gets mounted and used when the system is rebooted. Use your preferred editor and add the following line to /etc/fstab:

/mnt/swapfile swap swap defaults 0 0

Lastly, turn on swapping.

# swapon -a

You can now use swapon -s to confirm that the swap space is in use, and the free or top commands to confirm the amount of swap space. One thing to note is that this swap space will not survive an instance stop/start. When the instance stops the ephemeral storage will be destroyed. On restart the changes made in fstab and elsewhere will still be there, because the system root is on EBS, but the actual file we created at /mnt/swapfile will be gone, and will need to be recreated.


Running haproxy at system startup on Ubuntu

Sometimes I just have to capture this sort of thing because I know damn well I am going to forget it long before the next time it’s needed. We decided to use haproxy as a round-robin matchmaker in front of a cluster of elasticsearch nodes. I installed it on Ubuntu 14.04, configured it, and tested it by running it from the command line. Everything worked. But when I rebooted the instance the proxy didn’t start. I checked the init file and run levels. Everything seemed as it should be… but no dice. To make it worse, executing commands like…

sudo service haproxy start


sudo service haproxy status

… resulted in nothing at all. No errors, no output, nada. I searched around a little bit and stumbled on the following line in /etc/default/haproxy…


Apparently this flag, when set to 0, prevents the init script from running the proxy. The second-hand info I found on the reasoning is that there is no rational default config for haproxy, so they want you to explicitly configure it and then set this flag to 1. So… fine. Makes sense, but I do wish the init script was a little more verbose about what was happening. In any case, once I did this…


… haproxy launched at start as desired.

Elasticsearch discovery in EC2

Elasticsearch out of the box does a good job of locating nodes and building a cluster. Just assign everyone the same cluster name and ES does the rest. But running a cluster on Amazon’s EC2 presents some additional challenges specific to that environment. I recently set up a docker-ized Elasticsearch cluster on two EC2 medium instances, and thought I would put down in one place the main things that I ran into setting it up.

The most fundamental constraint in EC2 is that multicast discovery (unsurprisingly) will not work. EC2 doesn’t allow multicast traffic. This gives you two choices: use unicast discovery, or set up EC2 dynamic discovery. Unicast discovery requires that all nodes have the host IPs for the other nodes in the cluster. I don’t like that idea, so I chose to set up dynamic discovery using the EC2 APIs. Getting EC2 dynamic discovery to work requires changes in three places: the Elasticsearch installation, the configuration file at /etc/elasticsearch/elasticsearch.yml, and finally the instance and security configuration on Amazon. I’ll take these in order.

The best way to integrate EC2 into the Elasticsearch discovery workflow is to use the cloud-aws plugin module. You can install this at any time with the following command:

sudo /usr/share/elasticsearch/bin/plugin -install \

This will pull the latest plugin from the download site and install it, which is basically just extracting files. Note that the version in the command is the latest one. You can check here to see if it is still current. And that’s all there is to that. Adding the cloud-aws plugin enables the discovery.ec2 settings in elasticsearch.yml, which is where we’ll head next.

The Elasticsearch config file is located in /etc/elasticsearch/elasticsearch.yml, and we’ll want to change/add a few things there. First, obviously, give everyone the same cluster name: my_cluster

Another setting that makes sense in production, at least, is to require the cloud-aws plugin to be present on start:

plugin.mandatory: cloud-aws

The next two settings are required for the cloud-aws plugin to communicate on your behalf with the great AWS brain in the sky: NOTMYACCESSKEYATALL ITw0UlDnTB3seCRetiFiPuTItHeR3

Not to digress too much into AWS access management, but if you’ve set things up the right way then the access key and secret key used above will be for an IAM sub-account that grants just the specific permissions needed. The next two settings deal with discovery directly:

discovery.type: ec2 false

The first one just sets the discovery type to use the EC2 plugin. Pretty self-explanatory. The second one disables the use of multicast discovery, on the principle of “it doesn’t work, so don’t try it.” The last two settings we’ll discuss can be seen as alternatives to one and other, but are not mutually exclusive, which requires a bit of explanation.

Basically, we may need to filter the instances that get returned to the discovery process. When the cloud-aws plugin queries EC2 and gets a list of addresses back, it is going to assume they are all Elasticsearch nodes. During discovery it will try to contact them, and if some are not actually nodes it will just keep trying. This behavior makes sense with the multicast discovery process, because if you are not listening for multicast traffic then you don’t respond to it. But the EC2 discovery APIs will return all the instances in an availability zone, so we need some way to identify to Elasticsearch discovery which ones are really nodes.

One way to do this is by using a security group. You can create a security group and assign all the instances that you want in a particular cluster to that group, then make the following addition to the config:

discovery.ec2.groups: my_security_group

This setting tells the plugin to return only those instances that are members of the security group ‘my_security_group.’ Since you will need a security group anyway, as explained below, this is a convenient way to separate them from the crowd. But there can be cases where you don’t want to partition on the security group name. You might, for example, want to have one security group to control the access rules for a set of instances representing more than one cluster. In that case you can use tags:

discovery.ec2.tag.my_tag: my_tag_value

This setting tells the plugin to return only those instances that have the tag ‘my_tag’ with the value ‘my_tag_value.’ This is even more convenient, since it doesn’t involve mucking about with security groups, and setting the tag on an instance is easily done. Finally, as mentioned before these aren’t mutually exclusive. You can use the groups option to select a set of instances, and then partition them into clusters using tags.

And that’s it for the elasticsearch.yml settings, or at least the ones I had to change to make this work on EC2. There are a lot of other options if your specific case requires tweaking, and you can find an explanation of them here. The last thing I want to go into are the necessary steps to take in the Amazon EC2 console with respect to configuration. These fall into two areas: security groups and instance configuration. I don’t want to digress far into the specific AWS console steps, but I’ll outline in general what needs to happen.

Security groups first. You’re going to need all the nodes in your cluster to be able to talk to each other, and you’re going to want some outside clients to be able to talk to the nodes in the cluster as well. There are lots of different cases for connecting clients for querying purposes, so I’m going to ignore that topic and focus on communications between the nodes.

The nodes in a cluster need to use the Elasticsearch transport protocol on port 9300, so you’ll need to create a security group, or modify an existing one that applies to your instances, and create a rule allowing this traffic. For my money the easiest and most durable way to do this is to have a single security group covering just the nodes, and to add a rule allowing inbound TCP traffic on port 9300 from any instance in the group. If you are using the discovery.ec2.groups method discussed above, make sure to give your group the same name you used in the settings.

The last point is instance configuration, and for my purposes here it’s really just setting the tag or security group membership appropriately so that the instance gets included in the list returned from discovery. There are lots of other specifics regarding how to set up an instance to run Elasticsearch efficiently, but those are topics for another time (after I figure them out myself!)

The very last thing I want to mention is a tip for Docker users. If you’re running Elasticsearch inside a container on EC2 your discovery is going to fail. The first node that starts is going to elect itself as master, and if you query cluster health on that node it will succeed and tell you there is one node in the cluster, which is itself. The other nodes will hang when you try the same query and that is because they are busy failing discovery. If you look in /var/log/elasticsearch/cluster_name.log you’re going to see some exceptions that look like this:

[2014-03-18 18:20:04,425][INFO ][discovery.ec2            ]
[MacPherran] failed to send join request to master 
reason [org.elasticsearch.transport.RemoteTransportException:
Node [[MacPherran][aHWxgYAhSpaWlPEXNhs7RA][4fb92271cce6][inet[/]]] 
not master for join request from 

I cut out a lot of detail, but basically the reason this is happening is that, in this example, node MacPherran is talking to itself and doesn’t know it. The problem is caused because a running Docker container has a different IP address than the host instance it is running on. So when the node does discovery it first finds itself at the container IP, something like:


And then finds itself at the instance IP returned from EC2:


From there things do not go well for the discovery process. Fortunately this is easily fixed with another addition to elasticsearch.yml:


This setting tells Elasticsearch that this node should advertise itself as being at the specified host IP address. Set this to the host address of the instance and Elasticsearch will now tell everyone that is its address, which it is, assuming you are mapping the container ports over to the host ports. Now, of course, you have the problem of how to inject the host IP address into the container, but hopefully that is a simpler problem, and is left as an exercise for the reader.