He explained that I experienced a bug in their dynamic image configuration. When you instantiate a VM, a number of things happen behind the scenes to the base server image. It’s not as simple as copying a directory tree from A to B. A bug was introduced into their code, and they caught the bug and fixed it, but not before it bit some users.
So, Rackspace didn’t intentionally change the server image this time. I apologize for drawing that conclusion.
My November 2011 post about mutating server bits is still correct. We talked about Rackspace’s challenge in balancing “simplicity of use” vs. “power users’ information needs” when a server image changes.
Once again, Rackspace has changed the contents of an already-published server image without any notice to its users.
22 days ago, I provisioned a staging system with Ubuntu 11.10. In upgrading from 11.04, I had the typical difficulties — e.g., removing 11.04 package workarounds, and upgrading some software that we built from sources. When I finished, my Fabric script provisioned my 11.10 servers, and I wouldn’t have to futz with it again until we advanced to Ubuntu 12.04.
So imagine my surprise when I tried re-provisioning our staging system yesterday, and the script threw an oddball installation failure for PostgreSQL, and all the servers had oddball network flakiness.
Rackspace changed their Ubuntu 11.04 (Natty) server image without telling their customers. Our installation scripts unexpectedly broke. In the cloud, the rug can be pulled out from underneath you without warning, even in a very simple setup.
Rackspace has generally been a very good hosting provider. My only significant complaint is with their cloud administrative dashboard — it’s slow, clunky, and often hangs. But we’ve learned to live with it.
When we upgraded from Ubuntu 10.10 to 11.04, we had some typical upgrade pain with our Operations scripts. We had to remove some 10.10 package workarounds, and we switched some software from source builds to packages, because the 11.04 repository’s version was now acceptable.
We got past all that, and moved our systems to 11.04. Since then, re-building servers meant selecting Ubuntu 11.04 as the server image, running our Fabric scripts, and everything working predictably without surprises.
Until November 21…
I’ve got an odd problem. I create a 4 GB VM in the Rackspace Cloud with an Ubuntu 11.04 server image. After it’s created, I can’t ssh to it, and ping returns zero bytes.
I can get to it from the Rackspace dashboard console. But it’s not on the network. Creating a VM without a network is kind of useless.
I first alerted Rackspace to this over a week ago. It’s still present in our VMs, and now impacts our company in a very serious way. Rackspace says their Operations team has to check the host machine to fix this. You’d think this would be easy to isolate and resolve, but….nope.
Does anybody else have this problem?
Update: Rackspace confirmed this is a system-wide problem! Until it’s fixed, after I rebuild a VM I have to ask their customer support to goose the underlying host machine before it’ll respond to the network. Yikes.
Updated at 7/6/2011 1425: Whatdeyaknow, Ubuntu 11.04 just showed up on Rackspace’s server list.
It’s now July, and 11.04 is still not available as a server image at Rackspace Cloud.
I’ve repeatedly asked Rackspace about 11.04′s availability. I always get the same answer back, which is, paraphrasing: “We need to do something to ensure it’s reliable in a VM environment. Please be patient.”
Fine. But, c’mon, it’s been a quarter of a year now.
Does anyone know if Rackspace dropped Ubuntu as a supported operating system, and hopes nobody will notice?
My team (well, there’s just two of us now) is creating the development infrastructure for our big project, which is building my company’s new CMS on Plone. We’ll be Agile, of course, but without zealotry.
- The server is a clean installation, administered by yours truly. I install only necessary software, and use
apt-getfor all of them.
- I read installation instructions before installing software.
- I’m smart.
All went well, until I got to the end of Review Board’s installation instructions. There I found:
Set Up Search
This isn’t strictly necessary (and might be hard on some platforms), but it’ll make your users love you. See SettingUpSearch for details.
I want my team to love me. So, I took the bait.