IBM Research report on performance of Linux containers

At Knowledge In Practice we were pretty early adopters of Docker, and after more than six months of use nearly all of our production services are now deployed to Amazon’s EC2 as linux containers. While the lower overhead of containers was a draw,  as a small team the main benefits for us have been ease of deployment and increased environmental stability due to the use of Docker build files to declaratively specify the content of each service’s run-time environment. Launching a new instance of a service is literally as easy as adding one line to the cloudinit script for the instance, then running “docker pull” to get the image we want, and “docker run” to get the container going. Those steps could easily be automated as well. It’s a workflow that’s hard to beat.

Late last month IBM Research released a paper (PDF) comparing the performance of linux containers vs. traditional types of hardware and software virtualization. Not surprisingly containers fare quite well, although the paper notes that both VMs and containers need to be fine-tuned for high I/O workloads. Section 2.3 of the paper provides an excellent quick overview of how containers are implemented in linux using kernel namespaces and cgroups, and in fact I found that part of the document more valuable than the performance comparisons. Well worth a scan, at least, if you have an interest in this technology.

Bunch of Yahoos

Having a great set of developer tools can help make your platform ubiquitous and loved. When Microsoft first launched its Developer Network it revolutionized the way programmers got access to their operating systems, tools, and documentation. They successfully migrated that set of resources to the web and it remains invaluable for Windows developers. If you’ve ever set up access to a Google API, or deployed a set of EC2 resources on Amazon’s AWS cloud infrastructure, you know how impactful a clean, functional web interface is.

By the same token, a clunky, dysfunctional interface can make a platform loathed and avoided. Take Yahoo’s Developer Network and their BOSS premium APIs, for example.

We’re working on a system that needs to geolocate placenames in blocks of free text. This isn’t a trivial problem. There’s been a lot of work done on it, and we’ve explored most of it. During that exploration we wanted to try Yahoo’s PlaceSpotter API. It’s a pay service, but if it works well the cost could be reasonable, and just because we have built our system on free and open-source components doesn’t mean we won’t pay for something if it improves our business.

With that in mind I set out to test it, just as I had previously set out to test Google’s Places API. In that experiment I simply created a Google application under my user name, grabbed the creds, and wrote a python wrapper in about five minutes to submit text queries and print out the results. That’s my idea of a test.

In order to test Yahoo’s PlaceSpotter I needed access to the BOSS API. To get access to the BOSS API I needed to create a developer account. Ok, that’s not an issue. I will happily create a developer account. To create a developer account, it turns out, requires a bunch of personal info, including an active mobile number. Ok, I’ll do that too, albeit not quite as happily because all I want to do is figure out if this thing is worth exploring.

I should note that there is a free way to get to the same Geo data that BOSS uses, and the same functionality, through YQL queries. Maybe I was shooting myself in the foot right from the beginning, but I had no experience with YQL, I needed to move quickly and make some decisions, and I just wanted an API I could fling http requests at. Since the billing is per 1000 queries I had no problem paying for the first 1000 to test with. Not that big a deal.

After creating the account, during which I had to change the user name four times because of the cryptic message that it was “inappropriate” (no, I was not trying to use b1tch as a user name, or anything else objectionable), I finally ended up on a control panel-ish account dashboard. There I could retrieve my OAUTH key (ugh) and other important stuff, and activate access to the BOSS API.

I clicked the button to activate the API, and the panel changed to display another button labelled “BOSS Setup.” Next to that was a red rectangle stating that access to BOSS was not enabled because billing had not been set up. It wasn’t obvious to me that in order to set up billing you have to click “BOSS Setup.” I assumed billing would be at the account level. Well there are billing options at the account level! They’re not the ones you want, and unless the verbiage triggers some warnings as it did for me you might just go ahead and set up your credit card there, only to find it didn’t help.

Not to be deterred, I googled a bit and found that, indeed, I had to click “BOSS Setup.” It would have been nice if they had mentioned that in the red-colored billing alert. So I clicked, entered my login again because, you know, I was using the account control panel and so obviously might be an impostor, and ultimately found the place to enter my payment information. Once that was done, submitted, and authorized I received a confirmation and invoice in my email. Now, I could finally toss a few queries at the API.

Except no. When I returned to the account dashboard the same red-colored billing alert appeared. No access. I am a patient man, some of the time. Maybe their systems are busy handshaking. I waited. Nope. I waited some more. Nope. I gave up and waited overnight, and checked again this morning. Nope. Ok, dammit, I’ll click the “BOSS Setup” button again. I do that and what they show me is the confirmation page for my order again, with an active submit button. But wait… I got an invoice? Was I charged? Will I be charged again? Should I resubmit, or email Yahoo, or call my bank?

Maybe I should just not use the API. Oh, and did I mention that they have a “BOSS Setup” tutorial? It’s a download-only PDF. And 2/3 of it is about setting up ads.

New Year, New Theme

I’ve been slow in posting to this site for a couple of years, now. Time is at a premium these days, or at least, it is when you subtract all the time I spend playing guitar and starting but not finishing games. In the run-up to the symbolic passage of the old year and arrival of the new I gave some thought to the site and whether I wanted to keep it up. I decided that I did, but that the theme I originally chose for it, Palaam, had grown quite dated. This new theme is called Elucidate, and I think it’s pretty sharp. It’s responsive, and scales well on mobile devices. I’ve made some tweaks to it, reducing the size of article titles and adding the social icons at the top, nothing major. I also reorganized some content, but not so much that anyone will notice.

Let me know what you think. The team at work and I are ramping up development on a new system, and we’re in the rare position of being able to sample a lot of technologies and make our own choices with regard to our stack and architecture. I hope to write a lot more about that stuff in the coming weeks, so hopefully the bit on Docker above is just the beginning.

Going Down with the Ship: It’s Aeration, Not Suction

Ok, this will seem a little off the wall. Many of you who know me know that I used to be a professional sailor years ago. I worked on everything from small oyster boats to tug-n-barge combos running 600 feet in length. Sailors are great exchangers of tales, and no tale is more horrible and morbidly fascinating than that of a ship sinking. Such tales often feature, in one way or another, the idea that people who aren’t able to swim far enough away from the vessel risk being “sucked under” as it goes down.

That idea never struck me as very plausible. A ship going down creates a void in the water where its mass used to be, and water will rush in to fill that void, but the idea that some sort of suction could be created that would literally pull you down with the ship never made sense to me. I don’t have the technical chops to say exactly why, but it just struck me as wrong. While watching video of a sinking fishing vessel yesterday I thought of an alternative explanation that seems much more reasonable.

As most people who mess around with boats know, a prop that breaks the surface can no longer effectively propel the vessel. The reason for this is a phenomenon known as cavitation. When the prop breaks the surface it pulls air down and aerates the water around it. Aerated water does not have the mass of non-aerated water, and the prop can’t push against it effectively. For the same reason you cannot swim in aerated water. If I put you into a tank of water and bubble air up from the bottom you will sink, however mightily you flail.

Which brings me to sinking ships. They have a lot of air inside them, and when they go down that air comes bubbling up from all the various openings through which it can escape. You can see that effect pretty clearly in this two-minute video of a small fishing vessel sinking. A much larger ship means a lot more air, which in the process of escaping turns the water above into a aerated froth. And as I said above, you can’t swim in froth. So, I think the reality is that when a ship sinks and you are in the unfortunate position of treading water right above it, you don’t get sucked down. You fall.

LinkedIn: Why I Am Not Returning Your Endorsement

Don’t get me wrong, I love you all: the former colleagues from jobs I held eight or ten years ago; the neighbor I have never actually worked with, but who plays a congenial game of poker and is an all-around nice guy. I appreciate that you took the time to go on LinkedIn and endorse me for everything from “Software Development” (because there is no aspect of it I am not awesome at), to SQL Server, ASP.NET, astrobiology, xenolinguistics, and starship navigation. It’s true, I am a master of all those things.

But there’s no way most of you guys could have known that, because, well, we haven’t spoken in a decade or so. I wish that weren’t the case. I’m terrible at keeping in touch. But then, hey, you didn’t call either. I guess, in a way, this puts us back in communication. You took the time to punch that button on LinkedIn, and that means you were thinking of me, or at least stumbled on my bio pic as an artifact of an otherwise accurate search for people you care about. In either event I’m gratified. But I am not going to return the favor, and I thought you deserved to know why before I clear my inbox.

See, in other than a very few cases to which I have already tended, I just don’t know what you’re good at anymore. Ten years ago I was writing C# code, pulling data out of SQL Server, feeding it to ASP’s renderer, writing form-based web apps. When I code now I write Python, stick my data in Postgresql, and use ajax calls to populate pages. That is, assuming I am not writing Java or Objective C for mobile apps.  The world is completely different. If you’ve worked with me in, say, the last year or two then your endorsement might carry some weight, but not after so many years. And the same goes for any endorsement I make in return.

I think there may be some value in LinkedIn’s system of professional network references, but whatever that value is, it is certain to diminish with time. Endorsements are perishable. And given that, regardless of how many I collect, any potential employer is still likely to want me to put on a nice shirt, schlep in for an interview, and prove I actually know my stuff. It’s inconvenient, but it makes them feel a lot better about paying me, so I go along. Fortunately I don’t expect to have to go through it for some time, but then, that’s what we always think isn’t it?

In the meantime, I wanted you to know that I do honestly appreciate your effort, and that my lack of a return gesture does not have a linear relationship to the respect and regard in which I hold you. It’s just that it is exactly that, a gesture. I don’t mind “liking” crap on Facebook or G+ from time to time. After all, what am I really saying? Not much. But an endorsement for a specific skill should mean something, and carry some weight, and I just don’t think that LinkedIn endorsements do.

Fixing My House (or Maybe Not)

I was standing outside this morning, looking at my driveway, and it got me thinking about the nature of suburban property and the new realities we all face. My driveway, you see, needs to be replaced. It was resurfaced years ago with a thin layer of asphalt that was all the cheap bastard that owned the place before me would pay for. That layer is cracking off, the layer under it is dried out and fragmenting, and perhaps most critically the house has settled over thirty years and a portion of the apron near the garage now grades back toward the foundation. If you’re a homeowner, you know why that’s bad.

My house also needs new bay windows. Bay windows are a bad idea. Any boat builder will tell you that if you go to the trouble of building a watertight box, you ought to be very reticent about cutting big holes in it, and even more reluctant to hang cheaply-built wooden structures off of those holes. But that’s what bay windows are, and mine are rotting and need to be replaced. The list doesn’t end there. We need a new furnace, new windows, some interior drywall work to correct the effects of settling, a fix for a foundation crack, regrading in the back yard to fix a drainage problem, a new front porch and landscaping, and a thinning out of the many trees on the property so we can grow grass and stop our dogs from rolling in mud and wet leaves whenever it rains.

Our house sits on an acre of suburban land in northwestern New Jersey. Since moving in ten years ago we have already replaced the roof, the air conditioning system, the deck, and the garage doors. We’ve landscaped, planted grass and watched it die, and painted everything at least twice, inside and out. We’re on our second refrigerator, our third dishwasher, our second washer/dryer combo, and our second microwave oven. I have stopped counting toaster ovens and coffee makers since the numbers are large enough to be unwieldy. During that ten years, in addition to sloshing joint compound and paint on every vertical surface approximately as fast as my children could smash into those surfaces with hard, penetrating objects, I have fixed all three toilets at least four times, and put hundreds of dollars into minor plumbing, electrical, and carpentry projects.

Over the years I imagine I have spent $30-$40k on the place, and I could easily, and I do mean easily, dump another $50k in without even trying hard. The questions is: why should I? For a long time those of us who purchased single family homes in the suburbs viewed them as a form of wealth. Put in the money for maintenance, and the value of the property would increase and ultimately help pay for your kids’ educations and your own retirement. Like the stock market, gain was all a matter of appreciation. Like many of the public companies whose shares are traded on the market, my property doesn’t earn a dime. On the contrary, it costs me several thousand dollars a month in mortgage payments, utilities, maintenance, and the property tax payments by which I rent the right to stay here from the township we live in. If the value of my property doesn’t increase, then I’m never going to get a return on everything I’ve put into it.

And frankly, it seems unlikely to me that the value will increase. I was born in 1960, and my wife and I represent the very end of the baby boomer generation that bubbled into existence in the twenty years after World War II. We’re in our mid forties and early fifties, and our kids are either out of high school, or a year or two away from that milestone. As my generation passes into retirement, where will the families come from to drive demand for all the suburban single family homes that will be available? How can we expect the values of our properties to increase? It’s hard not to think that the great suburban land boom is over for good. And perhaps this should not be surprising. That boom, which largely consisted of people purchasing more land and more housing than they actually needed in order to gain better quality of life, could only have been sustained on the back of rising middle class incomes.

But middle class incomes have stagnated for well over a decade now. At the same time costs continue to rise at a steady few percentage points every year. It’s a game of economic water torture that can only end with exhaustion. I’ve got a few thousand bucks put away, and as I was standing outside this morning entertaining these thoughts, it suddenly seemed clear to me that the stupidest thing I could possibly do with that money is convert it into a new driveway and bay windows, or something to add “curb appeal.” It’s a fairly nasty Catch-22, because when you don’t put in the money to maintain a property, the decline in value only accelerates. Ultimately all you can do is hope to sell before the bottom, but in our neighborhood we already have one vacant place behind us that the bank can’t get rid of, and this is one of the more desirable areas in a state where the real estate market has always had a lot of resilience.

Faced with these realities, the possibility of a “Detroit-ification” of the American suburban landscape seems all too plausible. It would be nice to follow that statement with a prescription for what to do about it, but I have no idea what to do about it. We need a vigorous economy and rising middle class incomes. That should do it. How do you get back to those things? No idea here, or, I guarantee you, in our national capitol or any of the fifty state capitols. If we’re waiting for the right politician to pop up and rescue us it’s going to be a grim vigil. About the best I can suggest is to have a plan for either selling your property or making some money with it. Plans I have entertained include: starting a microbrewery in my basement, converting my attic into a shirt factory, and simply knocking down the house and landscaping the acre in hopes that, some day, a rich guy will want to incorporate it into his growing estate.

Upgrading my Desktop (and Other Anachronisms)

I’m one of those dinosaurs whose computer still exists as a collection of components mounted in a large aluminum box. Therefore, unlike the mobile device users who are laughing at the crusty immobility of my platform, I don’t have to throw the whole thing out when it gets too slow. I can fix it! Desktop computer builders fix their machines by upgrading them. Virtually every significant problem is a good excuse for an upgrade. It’s a testament to the quality of the components available now that my last upgrade was over three years ago.

I’m currently running an E8500 wolfie on an Asus motherboard with a P43 chipset and 8 GB of ram. The wolfdale has really been struggling lately. I have Visual Studio solutions that take 3-4 minutes to load. That’s unacceptable. I need more cores. I also have a GTS-250 graphics card w/512MB. Similarly unacceptable! Battlefield 3 and Skyrim are coming out. The latest generation of any classic game title also provides a valid excuse for an upgrade, and I have two coming at me. Clearly work must proceed on a component list forthwith.

But wait, it is a damn interesting time to be planning an upgrade. I don’t think I’ve been faced with this many tough choices in a long time…

Processor: I5-2500K vs I7-2600K

The 2600K is roughly $100 more, and clocks just .1 Ghz faster. It has hyperthreading on all four cores, and 2MB more L3 cache. I think that’s about it. Do I need hyperthreading? Four cores is a pretty good upgrade from my dualie already. Do I need four more virtual ones? Will I care about that 2MB of cache? I think I might. Leaning 2600K but not sure.

Chipset: P67 vs Z68

The P67 chipset is the tweakable “geek” version of the last Intel platform. The H67 was the mainstream “never touch the BIOS” version. The H67 supported the Sandy Bridge on-chip GPU, but the P67 did not. Now the Z68 combines the tweakability with the onboard GPU support and some drive-caching technology for SSD owners. Do I care about the onboard GPU? Not really, but it might be nice if my graphics card craps out. The rest of the stuff I don’t care about. Leaning P67.

GPU: 560ti/1GB

No real dilemma here. There’s zero chance I will pop for $300 -$500 for a GTX-570 or 80. Midrange for the win, and this will be a major upgrade over my GTS-250.

Monitor: too many factors to list

The monitor is really driving me nuts. I have a 5 year-old Dell 2405. It’s a slow panel, but it has served me well. I now need more screen space, and so I am adding a second display. The problem is that I detest 1080 line LCDs. Please. I had more lines than that on my NEC 17″ fifteen years ago! As a developer I need those lines. On the other hand I would love LED backlighting. It’s cooler and you get far better dark range colors. I would also like a fast panel. I would also like to match the 24″/1920×1200 of the current monitor, just for the sake of symmetry. Really have no idea which way to go here. Choices seem to be something like Dell’s 24″ S-IPS, which is 16:10 but fairly slow at 8ms, or the larger 25″ or 27″ 1080 (yeccch) displays from Acer or Asus.

I guess it’s nice to have choices.

Steal My Copper, Please

Copper is neat stuff. It’s malleable, ductile, resists corrosion, transfers heat readily, and can be easily soldered. In various forms it is incredibly useful. There are around 50 pounds of copper in an automobile, and 11,000 pounds in a diesel locomotive, for example. One of the less useful forms that copper takes is when it is extruded into long, thin wires, coated in insulating material, and strung between wooden poles to carry analog telephone signals. We still have one pair of these wires coming into our home, and I can’t for the life of me figure out why.

Our local carrier is Embarq, or whatever they are calling themselves these days (DinoComm? DustyLink?). For $30 a month they offer us a hard-wired circuit-switched connection to any local phone number I care to call. That is, assuming I remember the number, and don’t mind dialing it in manually. Thirty bucks a month. If I want to call any of the neighboring area codes, have voice mail, call forwarding, caller ID, etc., all of that is extra. We recently dropped all that stuff on our one remaining line. Before that it was $60/month.

Sixty bucks! That’s nearly half my Comcast bill, for a single lousy voice circuit. Really? I’m still paying these guys why? I can’t figure it out. We don’t even use the line anymore. Nobody uses the line anymore except a couple of marketing bots that hang up as soon as I answer (even the guys who write marketing bots don’t give a crap about telephones anymore). We use email, SMS texting, cell phones, and Skype for virtually all our communications needs. Skype is $29/year and I can call anywhere in the U.S. and Canada, with good quality, with one-click dialing, with video, and I can exchange text messages and files at the same time.

We came very close to cutting the cord completely a few months ago, and just couldn’t quite get there. It’s easy to generate doubt. What about 911? It doesn’t work on Skype but we all have cel phones and it does work on them. What about when the power is out and the computers are down? Well, the phone on the copper line is wireless and needs power too… and we have cel phones. All of the real questions have answers. Still, we didn’t get rid of it, and we’re still donating $30 every month so that Embarq can afford to scrub the rust off their trucks. It must be some sort of cultural nostalgia, or the communications infrastructure equivalent of apron strings.

According to the FCC (Table 2.2 Statistics of Communications Common Carriers) in 2006 there were 2.7 billion kilometers of metallic (mostly copper) wire in the physical plant of the country’s licensed local telephone carriers. I did a double take when I saw that number, and had to think about it a bit before I realized that this stat counts every conductor. If you look at sheathed metallic cable, most of which will have multiple conductors, the figure is 6.1 million kilometers. I googled around a bit on some stats for how much copper that represents, and calculated that it’s in the category of “metric ass ton.” If you want to get more specific, have fun with it. However you look at it, there’s a lot of metal hanging from wooden poles so that forgotten autodialers sitting in some dusty office closet can call me and hang up as soon as I answer.

Copper today is selling for $3.30 or so on the New York Exchanges. Next week in this space: the Post Office.

Jersey Sore

I’ve lived in New Jersey now, off and on, for nearly twenty years, but I’m not a native. I was born in Michigan and have at various times lived in New York, Indiana, Rhode Island, Maryland, California, New Hampshire, and the British Virgin Islands. None of the places I’ve lived has been perfect. They all have their good and bad points. When it comes to New Jersey, though, nobody seems to acknowledge the good. From comedians to columnists to Internet forum posters the standard line is that New Jersey consists of a single strip of industrially polluted concrete running from New York to Philadelphia, studded with chemical plants, refineries, landfills, and tanning salons. Now, thanks to MTV’s “Jersey Shore,” everyone also thinks that the open space between the salvage yards and tattoo parlors is crammed with orange-tanned goombahs sporting oily hair and neck chains.

If that’s your view of New Jersey, then rejoice. Your ignorance quotient is about to be attenuated significantly. You’re about to learn that not only is New Jersey nothing at all like the popular stereotypes would have it, it is in fact one of the most beautiful places in America. From the mountainous Highlands of the northwest, to the fertile rolling hills of the midlands, and the vast undeveloped tracts of pinelands in the southeast, New Jersey combines a little slice of everything that is best about our continent. I won’t ask you to take this on faith. In order to make my point I’ve combed through thousands of pictures of the state that I have taken over the years, and selected fifty that I think show a New Jersey most of you don’t know exists. This is the New Jersey that I know, and I’d like you to know it too.

When you’re through enjoying the images, consider this: all that beauty, the pines, the highlands, the hunt country, the Delaware River, Delaware Bay, miles of shoreline, all of it, is within 1-2 hours of New York City and Philadelphia, and 3-4 hours from Baltimore and Washington, DC. In other words, we win. We have our share of problems, to be sure. Taxes are too high, and we have a lot of challenges in terms of how to maintain economic growth and rebuild infrastructure… but then so do the states most of you live in, and our Governor doesn’t have a secret second family. Yet. So the next time you see some Jersey-hater hating, point them toward this post so they can get some education. Better yet, come visit yourself. Then you’ll know what I’m talking about.

Re-run TV?

I got a kick out of this. It was revealed this week that the share of downstream Internet traffic generated by Netflix customers’ streaming movies reached thirty percent in the last measuring period. Thirty percent. But as eye-opening as that figure is, it’s not what I got a kick out of. In some bit of reporting associated with that announcement I learned the following little gem: the pet nickname by which people in Hollywood sometimes sneer at Netflix is “Re-run TV.” I find this genuinely funny. Twenty-five or -six million people (including me) are streaming content from Netflix, comprising thirty percent of all downstream traffic, but Hollywood can still look down on them because they’re just “re-runs.”

In the world these guys grew up in, the one in which their business model was based on total control of content and delivery, there was for each piece of programming a “first run” during which the media biggies allowed people to watch it once, assuming they could be in front of the delivery device at the appointed time. Subsequent performances were “re-runs” for which the media corps were paid big bucks by smaller networks and independent broadcasters. Run, and re-run. And since everything on Netflix has been seen before, why hell it’s all just a bunch of re-runs. In their world, once, and in their dreams now, the viewing public flocks to them en-masse for the must-see content, and once that content has been seen they might agree to dribble it out bit by bit to other, clearly inferior outlets.

Meanwhile, on Planet Reality, I got to watch five seasons of Lost, all of Battlestar Galactica, Firefly, four seasons of Rescue Me, Weeds, Big Love, Torchwood, and dozens upon dozens of documentaries and movies, including most recently all the best, campiest Bond flicks from the sixties. Some of the movies I’ve seen before. Most of the television I haven’t. It’s all “first run” to me, and delivered to my computer, in my office, or on either of our two TVs, when I want to watch it. More importantly, the only way any of the networks can get anywhere near having twenty-six million people care what they are doing on a given night is to get two cute royal kids to marry each other. Hard to pull that off regularly.