Last day at the Nebula Project ( NASA )

Originally written 27.02.2012

There has been a lot of press written up about Nebula. There have been lots of claims made about Nebula. We ( on the team ) often times found ourselves laughing very hard at some of them. Watching openstack start up from before release day till now has afforded me an interesting view on reality. One I am sure most of silicon valley is on a very different planet in regards to.

For instance, there is this bit of hilarity we were laughing at today:

via ( http://www.pistoncloud.com/openstack/ )

Having been one of the primary operations contacts for all of these technologies for the better part of the past year and a half. I can tell you that Piston Clouds claims in this regard are at best a severe perversion of the truth. USASpending did not utilize nova. WWT did not utilize nova and predates nova development completely. And google mars is in the same boat as WWT except that most of google mars wasn’t even ran out of NASA. I won’t drop specifics on what was running those, but it suffices to say the software was not in any reasonable way related to nova or the development of nova. The best one could argue is that, Nebula team members realized, while working on these prior efforts, that all existing open source cloud stacks were pretty much terrible. That realization ultimately leading them down the path that resulted in nova. In that regard that statement could be made to have made sense.

It’s because of the stuff like this, that I wanted to write this post. I wanted people to know about Nebula. In case openstack does become huge and useful… some guy / gal in the future may look back and think… hey I wonder what it was like when this was developed at Ames. This is a quick summary of my time there.

DISCLAIMER

I am not sure if I am allowed to write all this. I did check my NDAs and such. Nothing here is violating NDAs or painting anyone in a negative light ( that certainly is not my intention ). I was a contractor through Dell Services Federal Government ( Formerly Perot Systems ) at NASA Ames with 100% time allocation to the Nebula Project task. I have all the love, respect and admiration for everyone that I worked with at Nebula. If anything in here leads you to believe otherwise, you are mistaken in that belief and I simply failed to express myself well. Nebula will always be in my mind an overwhelming success. And you’ll be hard pressed to prove to me otherwise.

PROLOGUE

I was grabbing lunch at the nearby mall food court while on lunch break. I had been working at JPMC as a contractor with the configuration management team for probably close to a year. My specialization was in HP Server Automation ( aka Opsware ) and many financial automation projects made use of HP SA. My phone rings. I answer. It’s a Dell HR person following up on an interview I had done four months earlier. After that call things sped up and I found myself flying out to SF to meet with the team at NASA.

The decision between working for a bank and working for NASA was an easy one. I mean there isn’t a nerd alive that doesn’t have “work at NASA” on their life dreams list. So, I like many before me, decided to experience something I dreamt of as a kid. And I decided to do it right. I set it up so that I would have two weeks of time to myself before reporting to Ames and starting. During that time I decided I would help a friend on an Art project at Burning Man. I mean if there’s something to get you in the right mindset for following a childhood dream that’s Burning Man. It’s a testament to the infantile desires of child-hearted lunatics with a propensity for the spectacular.

I loved every minute of that trip. Good crazy. Bad crazy. Whatever.

By the time I was on a plane to SFO with all of my earthly possessions sitting in boxes waiting on a destination for the movers from me, I felt I could conquer anything. Which is good, because landing in an unknown city with several days before you have to begin work and a finite budget for finding a new apt and getting out of a hotel can be somewhat stressful.

But, I found a place. And soon enough I was working inside of the Ames Research Center. Home of ridiculously awesome engineering that makes the average US citizen proud of their country. And the place where a scientists dream and hard work can occasionally change the world. Kepler certainly has while I was there. LCROSS just before I arrived. And, eventually even Nebula would leave it’s mark upon the world.

NASA changes the world so often people seem to have simply begun to not even notice it anymore. It’s just the norm for them.

NEBULA TEAM 2010

I am the guy in the Sparkfun sweatshirt.

THE GOOD OLDE DAYS

In the beginning there was Josh McKenty, Jeff Lindsay, and others whom I did not meet while at Nebula. These folks moved on before I arrived so I simply cannot speak to the great things they have done before and since. But the Alumni list of people who have worked on the Nebula project has some pretty successful people on it. All of them are awesome in their own right. By the time I arrived Jesse Andrews was leading the charge to develop and ‘ship’ an open source cloud compute component called “nova”. *Jesse had worked out a deal with rackspace, where in rackspace would open source their cloud file storage and NASA would open source it’s compute controller.

* Correction:

Nova was opensourced *BEFORE* openstack. And Jesse certainly DID NOT
“[work] out a deal with rackspace, where in rackspace would open
source their cloud file storage and NASA would open source it’s
compute controller.” Jesse was on several of the calls with rackspace but
so were the civil servants and you can guess who work worked out the
ideas.
I stand corrected on this point. Obviously outside my scope of knowledge / memory.

Inside NASA we had the Chris Kemp human PR machine running amok as well as James Williams and Ray O’Brien doing amazing work getting Nebula what they needed to be successful.

I am going to stop naming folks for now. Not as a slight but because there are many many more to name and I can’t look them all up.

There is something about being part of a team that is furiously working on something they know will be big and will succeed. Nebula was magical. The CTO of the entire US government Vivek Kundra was betting on our success. Chris Kemp was selling all of NASA on what a huge success Nebula was going to be. And, user after user was being invited into the little club that was Nebula to freely play with the code and have a voice in what it would turn into.

For me, this time was a blur. It took me a long time to catch up with where everyone else was in terms of knowledge on the project. Largely because in those days things were changing all the time and everything was a drop everything and change gears emergency. I know I spent too much time trying to build monitoring scripts and diagnostic tools to help me understand how nova was messing with our machines and how to spot it breaking. Because back then, things broke often.

I was really struggling to keep my head above water in the work load. And the tide was always rising. But, I loved it.

I just wish I had been able to join the team in April rather than September. I missed a good bit of the really early work. As well as getting my name added to the open source licensing agreement ( which turned out to be a big PAIN later on ).

OPEN SOURCE SUCCESS

Nova released before Openstack, and a combination of team members and NASA civil servants worked with rackspace to release openstack. A joining of rackspaces swift object store, and NASA’s nova compute controller.

Austin Texas, October 2010. OpenStack version Austin was released. By February the Bexar version of OpenStack was being released and it was the talk of cloud connect. Things were looking up. Our little project was getting an incredible amount of positive press. Some of it was ridiculous in terms of the claims it made about the software as well as our team. But all in all, people loved it. OpenStack started to pick up steam.

Around this time there were some rumblings internally. I think people were starting to have the conversations about what long term success meant for Nebula. Vivek Kundra’s popularity began to evaporate. And, as usual NASA was feeling the hurt of budget tightening in congress. I remember waiting to find out if we would all be furloughed as Congress failed time and again to set a budget. I can’t speak to the decisions made by Jesse and the Anso Labs contractors. But, by April we had received word of the impending acquisition by Rackspace. From what I have heard, the reason Anso ended up going to rack space was so that they could focus entirely on openstack development and get away from government hoops that were inhibiting their ability to do open source. Something I can confirm do exist.

I want to break in and say this. I don’t have any animosity for the people that left Nebula. Each and every person that left had their own reasons for doing so. And usually they were very good reasons. I think of them all as terrific co-workers and people even today. But when the dominoes began to fall they didn’t stop until there was far too little left.

It began with Anso reducing their involvement in the project. Brian Reed left soon afterwards for a better position elsewhere. Dean Troyer left to later join Rackspace for his own reasons. Chris Kemp set off on his Nebula Inc start up taking with him several key team members. And at the end of it all, the original team had been gutted. While the world was still singing our praises, and NASA was still basking in the glow of another success, we were starting to try to figure out how to pick up the pieces.

RECOVERING FROM SUCCESS

In business, success isn’t a good thing. It’s just a change of state, that will bring about consequences. Some of those consequences are predictable, some are not. Open Sourcing a product doesn’t deprive a company of revenue from that product. Done well, a company can create a market in which they are the foundation and grow it into a huge economic venture. However, when you increase the demand by the general job market for people skilled in this new open source technology, market forces will do their best to decrease your human resources. That’s neither here nor there. But Nebula is something of a cautionary tale in this arena. We lost too many people too quickly to adapt. And because of that we ended up in a bad situation.

I don’t remember much of what happened from late April on through October of 2011. I remember at one point I was sleeping about 4 hours a night. I sometimes didn’t even make it into the office choosing simply to work from my couch rather than waste time putting on clothing. With two people, we stood up a remote container datacenter, and addressed issues on our existing container datacenter. We continued to overcome documentation gaps and knowledge gaps. We made Nebula work again. Then we grew it. Then we began a continuous effort to re-engineer the entire environment to meet the changing needs of NASA. Changing needs that lead us to come up with a deployment model that would dynamically meet any possible need.

I remember we were given awards for that work. Mahendran and I did the lions share of that work. But Dave Engelbrecht, and Bill Endter were absolutely crucial in supporting the effort. Without them Nebula would have simply died under the weight of it’s own upkeep. As it is, it speaks volumes to the skill of all of the Nebula team that four people could maintain and grow the environment even in those early Bexar+ days.

I have had a few moments in my life where I got to see people I admire whom I have worked with run a gauntlet in terms of work and succeed. I think you have to have done it before to understand why people do it. It can be addicting. It’s not about the work. It’s not really even about the success. It’s simply about refusing to let yourself stop long enough to ask if it’s even worth it. Because if you do you might waiver. And if you don’t. It will be worth it.

By December of 2011, I was burned out. Done. We needed more people. We simply didn’t have the resources we once had. And while we had gotten over the hump and taken control of what was a wildly diverse environment through sheer force of will alone. We were facing an added 200 some odd machines, an increasing number of datacenter outages, and an upgrade from bexar+ to a diablo / essex hybrid courtesy our old friends now at rackspace operating as the cloud builders team. In addition to all of that our original nova clusters were reaching 2 and 3 years in age. Hardware failure was becoming a far more common occurrence. We hired new people. And we began a redesign effort that would put us in position to push out new revisions of openstack code, new security profiles, and provide greater degrees of automation across the board all on entirely new systems.

LEAVING AMES

By early February, our team member Dave Engelbrecht had very suddenly passed on. Another team member ended up in the hospital with a heart condition. And everyone was in shock. Nebula was a great project. But it was only great because the people were great. And losing Dave really ripped the heart out of all of us.

Last week I was informed that in the recent NASA budget, Nebula had found itself in very good company. Our funding was being severely cut. And we would be losing most of our key personnel from the contract. Myself included. One part of me was sad. Nebula had been a roller coaster of a project for me. The people I had worked with were amazing. And I had experienced a lot in the past year and a half on Nebula. It feels like it has been a decade since I picked up that phone while eating lunch in Jersey City. One part of me immediately was concerned with the team and their future. But, I really shouldn’t worry. They are all absolutely the pinnacle of awesome. People will be fighting over them. And then I was relieved. Since April of 2010 I have been fighting an uphill battle. I have fought until I couldn’t fight anymore to make Nebula work for the people who started the project. For the organization that I wanted to help since I was a kid. But I don’t need to anymore.

The world now has an open source cloud compute software that isn’t that terribly bad. Nebula helped create the myriad of companies that are now beginning to build themselves around openstack and the people that use it. We created a new micro-economy in IT. And there are now new jobs, new ideas, and exciting times ahead. Nebula is just one more project of NASA’s that just by being around even for a short time changed the world. And I’m a better person for having worked on Nebula in just about every way. I can’t wait to finally contribute to openstack now that I am free of my legal obligation to not do so. And now I just might be able to actually make some changes I’ve been looking to see happen. There is a whole world of possibilities out there now for both me, and Nebula’s kid project openstack.