Docker build files: the context of the RUN command

Docker is a game changing tool that is simplifying server dependency management in a wide variety of applications. For many of these applications simply spinning up a new container and installing a few things may be sufficient. Once the container is completed and tested you commit it to an image, push the image to your Docker repo, and you’re ready to pull and run it anywhere.

If your image has a lot of working parts and a more complicated install, however, this workflow is probably not good enough. That’s where a DockerFile comes in. A DockerFile is basically a makefile for Docker containers. You use a declarative syntax to specify the base image to build from, and the steps to take to transform it into the image that you want. Those steps usually include executing arbitrary shell statements using the RUN command.

The format of the RUN command is simply “RUN the-command-text”. Initially you might be tempted to look at RUN as essentially a shell prompt, from which you can do anything you would at an interactive shell, but that isn’t quite the way things work. For example, have a look at this minimal DockerFile:

# A minimal DockerFile example
FROM my-image
RUN mkdir /home/root/test
RUN touch /home/root/test/test.txt
RUN cd /home/root/test
RUN rm test.txt

This seems pretty straightforward: start with the base my-image, then create a directory, create a file in that directory, cd into that directory, and finally remove the file. If we try to execute this file using “docker build”, however, we get the following output:

Uploading context 2.048 kB
Uploading context 
Step 1 : FROM mn:saucy-base
 ---> 69e9b7adc04c
Step 2 : RUN mkdir /home/root
 ---> Running in d4802792515c
 ---> 4be0a443060a
Step 3 : RUN touch /home/root/test.txt
 ---> Running in 27aee53a2a17
 ---> b67284690b98
Step 4 : RUN cd /home/root
 ---> Running in 58d5fedeee98
 ---> 3a5826ad206c
Step 5 : RUN rm test.txt
 ---> Running in 02f11782a5e7
rm: cannot remove 'test.txt': No such file or directory
2014/01/31 15:11:34 The command [/bin/sh -c rm test.txt] returned a non-zero code: 255

The reason why the rm command was unable to find test.txt is hinted at by the output above the error. In particular, note the following:

Step 4 : RUN cd /home/root
 ---> Running in 58d5fedeee98
 ---> 3a5826ad206c
Step 5 : RUN rm test.txt
 ---> Running in 02f11782a5e7

Every instance of the RUN command that Docker processes gets applied to a new container that resulted from the changes created by the previous command. That’s what “Running in 58d5fedeee98” tells us. The command is being executed in the container with that ID, which is clearly different from the ID of the container in which the next command runs.

What this means is that the context of each RUN command is essentially a new instance of the shell, and any previous non-persistent changes like setting the current working directory are lost. The following revised DockerFile shows one way around this issue:

FROM my-image
RUN mkdir /home/root/test
RUN touch /home/root/test/test.txt
RUN cd /home/root/test;rm test.txt

Now the command that sets the working directory and the command that removes the file execute in the same context. If we re-run the build command we get the following output:

Uploading context 2.048 kB
Uploading context 
Step 1 : FROM mn:saucy-base
 ---> 69e9b7adc04c
Step 2 : RUN mkdir /home/root
 ---> Running in 633dd0266b8e
 ---> 7b2a80409513
Step 3 : RUN touch /home/root/test.txt
 ---> Running in d8122e2fb2ec
 ---> 70d091a60051
Step 4 : RUN cd /home/root;rm test.txt
 ---> Running in 68589850d97c
 ---> b88df827ad5f
Successfully built b88df827ad5f

One other quick note: when you build a container from a DockerFile containing many steps, a lot of intermediate containers are generated. The way to avoid having to manually delete them is to use the -rm flag to build:

sudo docker build -rm=true - < DockerFile

This will remove all the intermediate containers, as long as the script completed successfully. If any of the commands in the script failed, then it will leave all those containers behind. In that case, the easy way to get rid of them is:

sudo docker rm $(sudo docker ps -a -q)

Thanks to Dan Sosedoff for the tip.

Docker: fat container vs. skinny container

For the last week I’ve been evaluating some infrastructure technologies for our new platform at work. Most recently that effort has focused on using Docker for dependency management and deployment, as well as ElasticSearch, Logstash, and Kibana for log flow handling. Since our embryonic system does not yet produce much in the way of log data, I turned to an existing web spider framework I had built that generates tons of it. Well, according to my standards of scale, anyway. Running the spider against a typical list of 30,000 target domains yields close to a gig of log data. At maturity our new system will likely generate more than this, but the spider makes a great initial test of the logging pipeline I have in mind.

I brought Docker into the mix because we deploy onto AWS, and the promise of automating the build and deployment of independent containers to the server (continuous integration) was too good to pass up. As I began to work on the structure of the container for the spider framework I ran into a decision which will eventually confront everyone who is designing a container-based deployment: whether to make a single fat container with all the necessary internal dependencies, or several skinnier containers that work together. For example, the spider framework has the following major components:

  • The python spider scripts
  • Redis
  • Logstash
  • Elastic Search
  • Kibana
  • Supervisor

The supervisor daemon runs the spiders. The spiders take input from, write work output to, and log messages into Redis. Logstash reads log messages from Redis and indexes them in Elastic Search, and Kibana queries Elastic Search to provide log visualization and analysis.

Looking at that list there are a few different ways you could can it up. The most granular would be having each component run in its own container with just its own dependencies. That would also be the best option from a future scalability perspective. If you needed to cluster Redis, or Elastic Search, it would be a lot easier to do if everything ran separately. Docker makes it pretty easy to specify the network links between containers in a stable, declarative manner. So this could be manageable.

On the other hand, for at least this test iteration I am also attracted to the idea of a single “appliance” container that has everything needed by the framework, with the proper ports exposed so I can connect to it from the outside to monitor and control the operation. In that case configuring a new server at AWS would be a simple matter of installing Docker, pulling the image, launching the container, and then connecting to it from a browser. For simplicity’s sake I find this prospect attractive, and since I am currently using Logstash with the embedded Elastic Search and Kibana instances, I decided to try this approach first.

Probably the main thing you need to get around in this scenario is the fact that Docker wants to run one process per container. It launches that process when the container starts, and when the process exits the container exits. In my case I have at least nine processes I need to run: four instances of python running the spider script, two Redis daemons, and the daemons for Logstash, Elastic Search, and Kibana. This situation isn’t uncommon for Docker deployments. There is a lot of discussion on the web, usually leading to the overarching question that is the subject of this post: do you really want to run a bunch of stuff in one container? Assuming you do, there are a few ways to go about it.

One thing to note about Docker (and LXC containers in general, I think) is that they aren’t virtual machines. They don’t boot up, per se, and they don’t start daemons through rc or initctrl. Even so, it is nice to have core daemons running as services because you get some nice control and lifecycle semantics, like automatically respawning if something faults and a daemon crashes. You can do this by installing them as services, and then starting them manually from a launch script when the container runs. So the command to run your container might look like:

sudo docker run -d markbnj/spider /usr/local/bin/launch.sh

And then the launch.sh script looks something like:

/etc/init.d/redis_6379 start
/etc/init.d/logstash start

Not quite good enough, though, because this script will exit, and then the container will exit. You have to do something at the end to keep things running. In my case I need to get redis and logstash spun up, and then launch my spiders. That’s where supervisor comes in. The last line of my launch script will be a call to run supervisord, which will launch and control the lifecycles of the four spider instances. The container will remain open as long as supervisord is running, which will be until I connect to it and tell it to shut down.

As of last night I had everything but Supervisor set up. I need to do some more reading on parsing json with Logstash, and I need to write some code in the spiders to change the logging method to use the Redis queue. After that I will be able to deploy the last pieces, convert my recipe into a dockerfile I can automate the container build from, and then test. If it all works the way I intend then I will be able to simply launch the container and the spiders will start working. I will be able to connect to port 6379 to monitor the redis work output queue, and 9200/9292 to query log data. Pretty neat stuff.

Using Docker to turn my server app into an appliance

After messing with Docker enough to get comfortable with the way it works, I started thinking about a project I could do that would actually make my life easier right now. I have this server application that consists of a number of pieces. The main part is a python script that scans urls for information. I can run as many of these as I want. They get their target urls from redis, and write their work results back out to it as well. Currently they log to files in a custom format, including the pid so I can tell which instance was doing what.

When I want to run these things on AWS I open up a terminal session, load the initial redis database, then use screen to run as many instances of the spider as I want. Once they’re running I monitor output queues in redis, and sometimes tail the logs as well. I’d like to can this whole thing up in Docker so that I can just pull it down from my repo to a new AWS instance and connect to it from outside to monitor progress. I’d like the container to start up as many instances of the spider as I tell it to, and collect the log information for me in a more usable way.

To do this my plan is to change the spider as follows: I will rewrite the entry code so that it takes the number of instances to run as an argument, and then uses the process class to launch the instances in separate processes. I will also rewrite the logging code to log messages to a redis list instead of files. I will probably do this by launching a second redis instance on a different port, because the existing instance runs in aof mode, and that’s overkill for logging a ton of messages. Lastly I am going to install logstash and kibana. I’ll tell logstash to consume the logging messages from redis and insert them into its internal Elastic Search db, and I’ll use kibana to search and visualize this log data.

Redis, logstash, and kibana will all be set to run as daemons when the container starts, and the main container command will be a shell script that launches the spiders. The Docker image will expose the two redis ports, and the kibana web port. If all goes as planned I should be able to launch the container and connect to redis and kibana from outside to monitor progress. I have a quick trip to Miami at the start of this week, so I won’t be able to set this up until I get back, but when I do I’ll post here about my experiences and results.

Python: If you have Docker, do you need virtualenv?

I’ve been working in Python a lot over the last year. I actually started learning it about three years ago, but back then my day gig was 100% .NET/C# on the back-end, and just when I decided to do something interesting with Python in Ubuntu a series of time-constrained crises would erupt on the job and put me off the idea. Since that time my employer and my role have changed, the first totally, and the second at least incrementally. At KIP we use Python for daily development, and run all our software on Centos on Amazon Web Services. As such I got to dive back in and really learn to use the language, and it’s been a lot of fun.

One of the de facto standard components of the Python development toolchain is virtualenv. We use it for Django development, and I have used it for my personal projects as well. What virtualenv does is pretty simple, but highly useful: it copies your system Python install along with supporting packages to a directory associated with your project, and then updates the Python paths to point to this copied environment. That way you can pip install to your heart’s content, without making any changes to your global Python install. It’s a great way of localizing dependencies, but only for your Python code and Python packages and modules. If your project requires a redis-server install, or you need lxml and have to install libxml2-dev and three or four other dependencies first, virtualenv doesn’t capture changes to the system at that level. You’re still basically polluting your environment with project-specific dependencies.

Dependencies in general are a hassle. Back when I started messing with VMs a few years ago I thought that technology would solve a lot of these problems, and indeed it has in some applications. I still use VMs heavily, and in fact my daily development environment is an Ubuntu Saucy VirtualBox VM running on Windows 7. But VMs are a heavyweight solution. They consume a fixed amount of ram, and for best performance a fixed amount of disk. They take time to spin up. They’re not easy to move from place to place and it’s fairly complicated to automate the creation of them in a controllable way. Given all these factors I never quite saw myself having two or three VMs running with different projects at the same time. It’s just cumbersome.

And then along comes Docker. Docker is… I guess wrapper is too minimizing a term… let’s call it a management layer, over Linux containers. Linux containers are a technology I don’t know enough about yet. I plan to learn more soon, but for the moment it’s enough to know that they enable all sorts of awesome. For example, once you have installed Docker and have the daemon running, you can do this:

sudo debootstrap saucy saucy > /dev/null
sudo tar -C saucy -c . | sudo docker import - myimages:saucy

The first command uses debootstrap to download a minimal Ubuntu 13.10 image into a directory named saucy. It will take a little while to pull the files, but that’s as long as you’ll ever have to wait when building and using a Docker image.

The second command tars up the saucy install and feeds the tarball to docker’s import command, which causes it to create a new image and register it locally as myimages:saucy. Now that we have a base Ubuntu 13.10 image the following coolness becomes possible:

sudo docker run -i -t -name="saucy" -h="saucy" myimages:saucy /bin/bash
root@saucy#

This command tells Docker to launch a container from the image we just created, with an interactive shell. The -name option will give the container the name “saucy,” which will be a convenient way to refer to it in future commands. The -h option causes Docker to assign the container the host name “saucy” as well. The bit at the end ‘/bin/bash’ tells Docker what command to run when the container launches. Hit enter on this and we’re running as root in a shell in a self-contained minimal install of Ubuntu. The coolest thing of all is that once we got past the initial image building step, starting a container from that image was literally as fast as starting Sublime Text. Maybe faster.

So we have our base image running in a container. Now what? Now we install stuff. Add in a bunch of things the minimal install doesn’t include, like man, nano, wget, and screen. Install Python and PIP. Create a project folder and install GIT. Whatever we want in our base environment. Once that is done we can exit the container by typing ‘exit’ or hitting control-d. Once back at the host system shell prompt the command:

sudo docker ps -a

…will show all the containers we’ve launched. Ordinarily the ps command only shows running containers, but the -a option causes it to show those that are stopped, like the one we just exited from. That container still has all of our changes. If we want to preserve those changes so that we can easily launch new containers with the same stuff inside, we can do this:

sudo docker commit saucy myimages:saucy-dev

This just tells Docker to commit the current state of the container saucy to a new image named myimages:saucy-dev. This image can now serve as our base development image, which can be launched in a couple of seconds anytime we want to start a new project or just try something out. I can’t overemphasize how much speed contributes to the usefulness of this tool. You can launch a new Docker container before mkvirtualenv can get through copying the base Python install for a new virtual environment. And it launches completely configured and ready to run commands in.

Given that, I found myself wondering just what use virtualenv is to me at this point? Unlike virtualenv Docker captures the complete state of the system. I can fully localize all dependencies on a “virtualization” platform that is as easy to use as a text editor in terms of speed and accessibility. Even better, I can create a “DockerFile” that describes all of the steps to create my custom image from a knwon base, and now any copy of Docker, running anywhere, can recreate my container environment from a version controlled script. That is way cool.

Now, the truth is, there are probably some valid reasons why the picture isn’t quite that rosy yet. The Docker people will be the first to tell you it is a very young tool, and in fact they have warnings about this splashed all over the website. There are issues with not running as root in the container. I haven’t been able to get .bashrc to run when launching a container for a non-root user. There are issues running screen due to the way the pseudo-tty is allocated. And if you’re doing a GUI app, then things may be more complicated still. All this stuff is being worked on and improved, and it’s very exciting to think of where this approachable container technology might take us.

Pretty Much Everyone Should Read This

We have this divide in my house. I’m pretty much a typical introverted developer who prefers email and text messages, avoids the phone, and will only use my mobile while driving if something powered by a warp engine lands on the freeway in front of my truck. My wife and daughters, on the other hand, live for social interaction, and think nothing of communicating nearly constantly while driving. Like many other modern drivers, they have convinced themselves that their use of “hands-free” devices renders this mode of communicating safe. Several studies have shown that they, and a lot of other people, are wrong about this. Now we have another.

In a study released today the AAA Foundation for Traffic Safety showed that mental distractions stemming from attempting to communicate with other humans while operating a motor vehicle are just as dangerous whether you’re holding the phone or not. Rating distractions on a numerical scale for risk, the study found that simply listening to the radio was a risk level of 1, which is minimal. Talking on a mobile phone, whether hands-free or handheld, rated a 2. Responding to in-vehicle systems and prompts such as those from navigation or entertainment systems provided the most dangerous distractions of all, meriting a 3 on the AAA scale.

Years ago when I was working on the water I was aboard a large sailing vessel whose captain, a master of ocean-going oil tankers who moonlighted as a square-rig sailor, had very strong opinions about chatting on duty. If you had the helm and he caught you talking to a shipmate you were in for a stern rebuke, at least, and might be dismissed from the wheel and sent to another station. For a large ship, keeping a stable course is a matter of safety and significant amounts of money, and what our captain knew was that people who are focused on a conversation cannot remain attentive to the physical and mental requirements of piloting a vessel.

And they can’t pay attention to driving a car either. There have been enough studies done, and enough research released, that there shouldn’t be any remaining debate here. When you’re driving you’re in charge of a couple of thousand pounds of metal moving at a significant speed. That’s a lot of energy, and it can have devastating consequences when it is not properly controlled. The only responsibility you have when behind the wheel is to exert the proper control to move that chunk of metal around safely. Any conversation, at all, detracts from your ability to do that. End of story. And I really do mean, end of story. Because there is no technology that can change this fundamental characteristic of our brains. Until cars can safely drive themselves, we have to put the phones away and do the driving for them.

So-so On Metro

Back at the beginning of last summer Microsoft did a public preview of the Windows 8 user interface, and the world got a look at Metro apps for the first time. A number of people in the software business, myself included, came away with the impression that, at least as far as the mobile platform was concerned, the company was backing away from XAML and Silverlight in favor of HTML5 and javascript. Some of my colleagues felt that this was an overreaction. Certainly it was an early announcement, and there was very little information to go on. As a developer who had invested a fair bit of my career in the .NET framework and related technologies including WPF and Silverlight, I was of course interested to know whether I was going to have to throw that away and learn a new framework to write apps for Windows 8 mobile devices, but in general agreed that the best course was to wait and see.

With the release of the Windows 8 consumer preview back at the end of February wait mode was over, and it was time to see. A day or two after it was made available I cleared enough time to get it installed into a VirtualBox VM along with the Visual Studio 11 beta with support for Metro application development. That process wasn’t entirely smooth, which isn’t totally surprising given that both the operating system and development environment are pre-release software. I still can’t run reliably with the Oracle Guest Additions installed, so no widescreen for me. But eventually I was able to get it working well enough to start playing around with Visual Studio. After building a few tutorials and my own “hello world” style Metro apps, and after using Metro on my desktop for a couple of weeks, I’ve been able to come to some early conclusions.

The first and most urgent question, for us WPF/Silverlight developers anyway, is whether Microsoft threw our favorite stack under the bus. Well, yes and no. The .NET framework, WPF, and Silverlight are all alive and well, and .NET 4.5/C# 5 even contains some cool new toys, such as language support for async programming. However, WPF and Silverlight have no role in Metro app development. In fact the .NET framework has only a supporting role in Metro app development. This is because Metro apps don’t run on the CLR the way WPF and Silverlight programs do. Instead they run on something called the Windows Runtime. The WRT is a sandboxed execution environment for Metro apps. I almost said “mobile apps” but in fact Metro plays a big role in Windows 8 on the desktop too, which I’ll talk about more below. The WRT looks a lot like .NET in certain ways – you create UI markup in XAML or HTML and code-behind classes in C#/VB/C++/javascript – but there are a lot of differences too. So in the end, yes, you’re going to have to learn a new framework to write Metro apps. It doesn’t look like a huge hurdle for .NET developers, but there it is.

So with that out of the way, is it worth learning to write Metro apps? It will be worth it if there is a lot of demand for them. Metro apps can run on mobile devices like phones and tablets, and on the Windows 8 “Metro desktop” (for lack of a better term), so those are the markets where demand will have to arise. We can dispense with the last case first. When I first started using Windows 8 I specifically wanted to keep an open mind about Metro on a desktop machine, but having used it for a couple of weeks now my mind is becoming more and more closed. I don’t see the advantage of most of the Metro UI metaphors in a desktop environment, especially without touch. You end up moving the mouse around a lot more, and virtually every aspect of the design aesthetic for Metro apps is hugely wasteful of screen real estate in an environment where you have lots of it. I have two 24″ LCD monitors, and turning that real estate into a vast field of some cheerful color populated by a few tiles with really big fonts in them doesn’t strike me as a good return on my hardware investment.  Windows 7 is so good, and Windows 8 is in most ways such a minor delta from it,  that there is no way I would voluntarily upgrade just to get Metro. I suspect most businesses will see it the same way, and most consumers will get it on a new PC, if they get it at all.

If I’m right about that then what remains is mobile devices. I’m nowhere near enough the expert on those environments to predict whether Microsoft has a chance of driving adoption. But I know enough about business and markets to think they need a knockout to take substantial share away from competitors as entrenched as iOS and Android. I’ve developed iOS and Android apps, and have used both platforms on smartphone devices, though not tablets. I really don’t see what advantages Metro has in usability. Over the long run, though, I think Metro has a chance against Android, simply because the Android ecosystem is so fragmented. Apple has shown over the years, as has Microsoft for that matter, that tight control leads to a more polished user experience. Based on that I wouldn’t be surprised to see Metro gain some adoption over the next few years. It will continue to be a niche player for some time, however, and that means that developers who put time into learning WRT and Metro development will be speculating on its future success. Will that speculation deliver a big payoff? I guess it’s time to wait and see again.

Upgrading my Desktop (and Other Anachronisms)

I’m one of those dinosaurs whose computer still exists as a collection of components mounted in a large aluminum box. Therefore, unlike the mobile device users who are laughing at the crusty immobility of my platform, I don’t have to throw the whole thing out when it gets too slow. I can fix it! Desktop computer builders fix their machines by upgrading them. Virtually every significant problem is a good excuse for an upgrade. It’s a testament to the quality of the components available now that my last upgrade was over three years ago.

I’m currently running an E8500 wolfie on an Asus motherboard with a P43 chipset and 8 GB of ram. The wolfdale has really been struggling lately. I have Visual Studio solutions that take 3-4 minutes to load. That’s unacceptable. I need more cores. I also have a GTS-250 graphics card w/512MB. Similarly unacceptable! Battlefield 3 and Skyrim are coming out. The latest generation of any classic game title also provides a valid excuse for an upgrade, and I have two coming at me. Clearly work must proceed on a component list forthwith.

But wait, it is a damn interesting time to be planning an upgrade. I don’t think I’ve been faced with this many tough choices in a long time…

Processor: I5-2500K vs I7-2600K

The 2600K is roughly $100 more, and clocks just .1 Ghz faster. It has hyperthreading on all four cores, and 2MB more L3 cache. I think that’s about it. Do I need hyperthreading? Four cores is a pretty good upgrade from my dualie already. Do I need four more virtual ones? Will I care about that 2MB of cache? I think I might. Leaning 2600K but not sure.

Chipset: P67 vs Z68

The P67 chipset is the tweakable “geek” version of the last Intel platform. The H67 was the mainstream “never touch the BIOS” version. The H67 supported the Sandy Bridge on-chip GPU, but the P67 did not. Now the Z68 combines the tweakability with the onboard GPU support and some drive-caching technology for SSD owners. Do I care about the onboard GPU? Not really, but it might be nice if my graphics card craps out. The rest of the stuff I don’t care about. Leaning P67.

GPU: 560ti/1GB

No real dilemma here. There’s zero chance I will pop for $300 -$500 for a GTX-570 or 80. Midrange for the win, and this will be a major upgrade over my GTS-250.

Monitor: too many factors to list

The monitor is really driving me nuts. I have a 5 year-old Dell 2405. It’s a slow panel, but it has served me well. I now need more screen space, and so I am adding a second display. The problem is that I detest 1080 line LCDs. Please. I had more lines than that on my NEC 17″ fifteen years ago! As a developer I need those lines. On the other hand I would love LED backlighting. It’s cooler and you get far better dark range colors. I would also like a fast panel. I would also like to match the 24″/1920×1200 of the current monitor, just for the sake of symmetry. Really have no idea which way to go here. Choices seem to be something like Dell’s 24″ S-IPS, which is 16:10 but fairly slow at 8ms, or the larger 25″ or 27″ 1080 (yeccch) displays from Acer or Asus.

I guess it’s nice to have choices.

Keep the Stars, I’d Like a One G Drive

I have to admit, I’m a complete space geek. That’s one reason why I was thrilled to read this week about the Defense Advanced Research Projects Agency’s grand initiative to seek the fundamental science of interstellar travel, with the goal of developing the enabling technologies by 2111, one hundred years from now. This is good stuff. We need it, by God. In the middle of the last century John Kennedy challenged us to land a man on the moon, and that vision sustained America, in many ways, through some dark and divisive days. In this time of credit downgrades, defaults, unemployment, inflation, foreclosures, the snickering of our enemies, and the bickering of our politicians, an initiative that calls us to envision the greatness we might achieve in a century is more than welcome.

That said, I think I would have suggested a more reasonable target than the stars. I grant that “The Stars!” is a far more potent rallying cry than “A One Gravity Constant Acceleration Propulsion System!” Nevertheless, that’s what I would have suggested, and that’s where I believe our energy and resources ought to be focused. The stars may someday be accessible to us, but a high specific impulse one gravity drive would open up the entire solar system as soon as it was available. The math of constant acceleration is a wonderful thing to behold.

Start from a dead stop and begin accelerating at one gravity, or 9.8 meters per second, squared. In 10 seconds you are travelling 98 meters per second. In 30 seconds you’re moving at 294 meters per second, or 17 kilometers per minute. By the end of your first hour of one-gravity acceleration you’ve reached the blistering speed of 35,280 meters per second. That’s more than 126,000 kilometers per hour.  If you can keep this rate of acceleration up for a week you’ll be moving at more than 5000 kilometers per second. In about 354 days you’ll hit C, the speed of light in a vacuum. Well, actually you won’t, or maybe you will; we really don’t have any idea. But with a one-g drive we could find out in a year. A few years more could put a probe powered by such a drive in the Alpha Centauri system.

As cool as that sounds, the benefits of a one gravity drive don’t derive from streaking across interstellar space at the speed of light. The benefits lie right here in the solar system. Currently getting around the local neighborhood is extremely difficult for us. The reaction engines we have available produce many times one gravity of acceleration, but can only operate for very short periods at high rates of fuel consumption. That’s good, because we need such engines to get off the ground. Obviously if you light off a one-gravity drive at ground level the thing will just sit there and look imposing, and perhaps hum a bit. But once in space these low specific impulse (fuel inefficient) reaction motors become a liability.

Everything in the galaxy is in orbit around something else, riding around and around the inside of a deep well of space created by the gravitational fields of massive objects. The moon orbits Earth, which orbits the Sun, which orbits the galactic center, which orbits something much vaster that I am unfamiliar with. The center of a cluster or something. A spacecraft, once it is free of Earth, is another orbiting object in this vast dance. But, because it has power, it can do something no other object in the universe can do: it can decide to change its orbit. It can do this by applying forces to add to or subtract from its velocity. To put it simply, if you are in an orbit, and you add velocity, you move to a higher orbit. If you subtract velocity you move to a lower.

Because we can only burn our reaction motors for very short times, a human spacecraft that wants to travel from Earth to Mars, for example, has to use something called a Hohmann Transfer Orbit. To put it simply, because I am incapable of putting it other than simply: the spacecraft begins at Earth, sharing Earth’s solar orbit. It is then accelerated until it is in a higher solar orbit that meets with the orbit of Mars at some point in the future. It then coasts until it reaches the orbit of Mars, and decelerates to match Mars’ orbital velocity. Voila! And it only took five years! Yeah, Hohmann transfers are slow. First, you have to delay even starting until the objects are in relative positions that make the voyage practical. Then you have to boost yourself into a high orbit, switch off the motors, and just ghost along for years.

With a one-gravity constant acceleration drive the trip would be slightly more expeditious. You could calculate a trajectory through the solar gravity field that intercepted the object you wished to reach, taking into account deflection from other gravity fields, and then point yourself in the right direction and light off the drive. You accelerate at 1 G until you’re halfway there, and then turn around and decelerate at 1 G for the remainder of the voyage. The moon would be hours away. Earth to Mars? Something like three days. Mars to Jupiter? Maybe ten. Now that’s cool. Such a drive would instantaneously render the entire solar system open to human exploration and exploitation, and would lead to further advances that might one day actually take us to another star.

Exciting, for sure. Achievable? Make no mistake: a one-gravity constant acceleration drive of high specific impulse is magical fantasy. Perhaps it is slightly less so than a lightspeed drive, a warp drive, a transporter, or a holodeck, but it is fantasy just the same. Still, if we’re going to set a very high, very long term goal, I think this is the one to shoot for. We may very well never get to another star. Even if the challenges of distance, time, and velocity can be overcome, the remaining technical challenges of fuel consumption, heat dissipation, hull integrity, radiation, communication, and a hundred other problems might very well be enough to keep us home.

By contrast, we already know pretty well how to survive and navigate our own solar system and return home safely. We simply lack the power to do it in a reasonable time. If we were to gain that power, it would be of far more immediate benefit to the human race than reaching another star system. It would open the vast resources of every corner of the solar system, and if there is other life to be found in the galaxy, mastery of our own system would set the stage for someday encountering them on more equal terms. The stars might make a more compelling objective, but a one-gravity drive is really the key to a human future in space.

Ineffectual Ventures

Here’s a question: what do you do after you’ve made it? I don’t mean had a couple of wins and put a few bucks in the bank. I mean absolutely, unequivocally, ball-out-of-the-park made it? Dr. Nathan Myhrvold, the former Chief Technology Officer at Microsoft, has definitely made it. He leveraged his massive intellect and first-class education into a career at the hottest startup of his era, and in the process made hundreds of millions of dollars, at least. So what do you do after you’re done with game seven of the series and emerge the big winner?

For Myhrvold’s former boss Bill Gates, his friend Warren Buffet, and many other very wealthy individuals, the answer has been philanthropy. For some, like Steve Jobs, it’s a second turn at the reigns of the horse you rode across the finish line in the first race. Others are serial entrepreneurs. Not a few have embraced the greatest challenges of our time. Paul Allen and Elon Musk build spaceships. For many of the newest mega winners, such as Mark Zuckerberg, the question remains to be answered. What about Dr. Myhrvold?

According to a recent investigative piece by NPR, the answer appears to be, at least in part: buy up a huge number of patents and go fishing for dollars. Myhrvold’s company Intellectual Ventures was ostensibly founded to be an incubator for innovation, and indeed it is involved in a lot of interesting projects, many a good bit more relevant to humanity than the world’s most expensive cookbook. The company holds around 1,000 patents on original work, which is pretty damn impressive. On the other hand it has purchased over 30,000 patents for it’s portfolio, and has allied with a shadowy network of employee-less patent trolls (many located in the same building full of empty offices in the Texas town of Marshall) to monetize those intellectual properties through the usual mechanism of threatened litigation.

I don’t need to address here whether software patents are a good or bad thing in themselves. I instinctively don’t like them, but on the other hand I’m a believer in the idea that software is analogous to machinery. If it’s machinery, then why shouldn’t it be patentable, as machines have long been? The question is troubling to me, but it is deserving of its own debate. The issue here is not whether software patents are right or wrong, but rather that the process by which they are currently granted is completely broken. There are patent trolls out there right now suing for license fees on ideas as basic as popping up a tooltip when the mouse is moved over a rectangular area of the screen. The NPR piece talks about a guy who literally has a patent on just about everything everyone does online. Myhrvold’s company used to own it.

It doesn’t take a genius with a doctorate to know that the system is broken, and that patent examiners are unable to accurately judge prior art, or whether what they are seeing is really a unique design for a new machine or process. My own work has been cited in five granted software patents. The articles cited in the patents were all written back in the nineties for journals like Computer Language and Dr. Dobb’s. There’s not one line of original work in any of them. I wasn’t doing original work. I was summarizing technologies for readers who wanted to understand them. I didn’t discover or invent anything, or lay the foundations for anything to be discovered or invented. So what is my work doing in patent disclosures?

That’s the problem, and it’s why I was disappointed to read what Intellectual Ventures has been up to. I have long been an admirer of Dr. Myhrvold, albeit one who will never spend $600+ for his cookbook. I have heard him speak, and read what he’s written, and I don’t believe he can convince himself that the actions of his company are encouraging innovation, or supporting the efforts of small inventors. They certainly could do that. There’s nothing wrong with their model if it is pursued honestly. But they are trolling with a wide net, and catching too many of the wrong kinds of fish.

Patents were originally intended to encourage innovation by offering protected disclosure of methods and processes. Nothing is disclosed when a patent is granted on something basic like tooltips, or remote updating of software, because nothing needed to be disclosed. We all already know how to do those things, which is a pretty clear indicator of prior art. The effect of these patents is simply to erect private tollgates on well-traveled roads. Good software patents might be healthy, but bad software patents are definitely unhealthy, productive of a relationship more parasitic than nurturing. The NPR piece is conclusive, as far as I am concerned, with respect to Dr. Myhrvold’s participation in this business. That’s too bad. With his mind I have no doubt he could make himself another billion or two, and in a much more productive and satisfying way: by inventing something cool.

Windows 8 Hurts My Brain

At D9 Microsoft showed the Windows 8 UI for the first time. All the attention was on the touch-oriented “Live Tiles” interface. Business Insider and some other pubs have taken to calling it “Windows 8 for Tablets.” I don’t know if that’s what it is. About all I can tell is that the Live Tiles interface is supposedly an addition to, not a replacement for, the native Windows UI. That’s good. Getting all touchy-panny with some pretty photos on a ginormous wall screen that costs more than my truck is impressive, but it doesn’t mean much to desktop and laptop users.

Live Tiles on desktops and laptops makes about as much sense as wings on a Hyundai Sonata. Microsoft has said that Windows 8 will run on some different processors, i.e. ARM, Intel system-on-a-chip, whatever. Those aren’t running Alienware gaming boxes. So is this meant to be the UI of the mobile version of Windows 8? Who knows? Is it Windows Phone 8? Apparently not. When asked how developers would write applications for the new Windows 8 UI the company said “HTML 5 and javascript.” Developers and some industry pundits interpreted this to mean no Silverlight for Windows Phone apps. To many people this seemed to make sense, given the expansion to new hardware platforms.

But hardware independence is what .NET, the CLR, and Silverlight are all about. Programmers have invested a lot of time learning the technology stack. When Microsoft released Windows Phone 7 they built the application model and UI on Silverlight for Windows Phone. Developers could leverage everything they new about C#, XAML, and the Silverlight version of the framework to write apps for the mobile platform. Then those apps could be submitted to the shiny new Microsoft app store and start generating iOS-like sales. Now… what? The native mobile interface to Windows 8 will be HMTL + script, but you’ll be able to run Silverlight apps too?

According to some comments I’ve read Microsoft has clarified their position and said that there “will be a place” for Silverlight apps on Windows (Phone?) 8 (for Tablets?). I can’t find any direct quote. On the Silverlight forums the official response has essentially been: we can’t talk now, but don’t believe everything you read, and we’ll say more later. But they’ve said too much already. Why launch a platform and all of the tools to support it, and then make such a radical change in the next version? Has Microsoft figured out that they can’t build a native app ecosystem, and decided to attack Apple’s by promoting HTML 5 and javascript?

I’m not sure, but they’ve at least given me another good reason to continue my efforts with iOS, HTML 5 + JQuery, and Android. Why invest in native Silverlight apps for Windows Phone 7 if Microsoft won’t clearly commit to carrying forward the key components of the platform or building the ecosystem?