The benefits of keeping a technology up-to-date far outweighs the costs, unless you’re in an extreme corner case with a very unreliable vendor. Better performance, bug fixes, better capabilities, better support… all good stuff.

We use Solr (with Zookeeper) for many of our search and pivot tasks, and Redis as a Swiss Army Knife. They do different things and have different consistency requirements. You can easily critique any juxtaposition as comparing apples to oranges. I think it’s instructive, because Solr and Redis are both high-performance, production-quality, and powerful tools.

Working on them within the same day, I experienced exact terminal opposites in configuration philosophy!

Let’s meet contestant number 1

Solr is a powerful search engine. Their Cloud feature lets you shard and scale your index, and Solr will do the internal shard and node routing. Or you can direct your queries to the appropriate node for a small performance win. Being short-handed understaffed frugal with our peons worker bees people, we let Solr do the routing. “Here’s a document, store it.” “I want this document.” “Here’s a pivot within a search, do it and assemble the results for me, pronto.” Etc.

Solr nodes are peers, though internally there are leaders and replicas. Solr uses Zookeeper, an Apache technology for distributed persistent configuration. Nodes do the right thing when other nodes come and go.

Let’s meet contestant number 2

Redis is a fast, reliable, straightforward key:value store. Its commands remind me of assembly language opcodes. “Here a key and a value, remember it.” “Give me the value at this key.” “Give me the hashed value at this key.” Etc.

Redis has asynchronous non-blocking master-slave replication. A master can have any number of slaves, and a slave can have any number of slaves. Slaves are read-only by default, though you can make a tree of cascading read/write slaves if you’re game.

Third-party packages exist to do read/write routing, and to even hot failover. For now, we’re keeping things simple and doing our own read/write splitting.

Deployment automation

Because of the relatively few number of servers in our production system, we don’t use a configuration management system (e.g., Chef, Puppet, Ansible) yet. We use Fabric for deployment automation, and we just upgrade in place using Fabric tasks. It’s not as automated as a CM tool, but OTOH it’s one fewer technology with which to grapple. We don’t support hundreds of nodes.

So last week…

We added a Redis slave, which I wanted to codify in our operations fabfile. The slave was for redundancy (though not hot-failover) and future scaling. We send writes to the master, and we read from a load balancer VIP that consists of the master and the slave. I needed to update our fabfile.py to account for this.

We upgraded to Solr 4.0 in January, and then to 4.1. We did it all interactively. We’ve got an awesome Solr consultant advising us, and sometimes doing the work.  The Solr 4.x installation and configuration instructions were documented, but our Fabric scripts hadn’t yet been upgraded. So, I also wanted to codify the installation and configuration in our fabfile.py.

Off I went to upgrade our Fabric script.

First, Redis

Every Redis node has two configuration files. One is unique for the node, and the other has parameters that are common to every node. The former links to the latter. (Of course, you could also just have one (larger) file on each node.)

To set up a slave, you include a SLAVEOF command in your configuration file:

SLAVEOF <master_ip> <master_port>

If your master node has a password, there’s one other command to supply it.

When a node knows it’s a slave, it sends the master a SYNC command. It doesn’t matter if it’s the first time it has connected or if it’s a reconnection. The master then starts background saving, and collects all new commands received that will change the dataset. When the background saving is complete, the master transfers the database file to the slave, which saves it on disk, and then loads it into memory. The master then sends the slave all accumulated commands, and all new commands received from clients that will change the dataset. This command stream is in the format of the Redis protocol itself. (This paragraph is shamelessly copied.)

The most obvious critique is the lack of partial upgrades and timestamps. If a slave loses connection, it re-syncs, to which the master will respond by re-sending the entire database. This is inefficient; there may have been few (or no) database changes since the slave was last synced. OTOH, it’s simple. You might say, foolproof.

Upgrading our fabric script was easy. The “provision redis” task knows when it’s provisioning a master or a slave. (The master is list entry [0].) When provisioning a slave, it uses sed to insert a SLAVEOF into the configuration file.

Then, Solr

Solr is written in Java, so its configuration includes the usual Javaesque parameters to Jetty or Tomcat.

I started codifying our Solr 4.1 installation and configuration into our fabfile. First, we need to edit /etc/security/limits.conf to increase the maximum file descriptors. That was only a couple of sed commands.

Next up was changes to /etc/pam.d/su. Also easy.

Next, for Zookeeper, we have to put this node’s unique identifying number into /etc/zookeeper/conf/myid.

Wait, WTAF? Each Solr node needs a unique identifying number — 1, 2, 3, etc. A node finds its number by looking in /etc/zookeeper/conf/myid. But, erm, every node has a unique IP address. You’re running two instances on the same server? Fine, toss in the port number. Why can’t Zookeeper use that?!

I got a can of Coke and wrote a simple function that returned a node’s unique identifying number. This wasn’t the end of the world, but it was unnecessary.

Then, we have to update /etc/default/zookeeper to adjust JAVA_OPTS.

Then we have to edit /etc/zookeeper/conf/zoo.cfg to insert the server’s IP address in some commands. Holy crap, it has a unique integer and a unique IP address?!

I was now getting a headache.

Are we done? No. Then we run /usr/share/zookeeper/bin/cli_mt to do something. Then /usr/share/zookeeper/bin/zkCli.sh to do something else. Then run zktreeutil. Then check /var/log/zookeeper/zookeeper.log to see if the world has exploded.

Are we done? No. Then we install Jetty, which includes editing/etc/default/jetty to set a smorgasbord of JAVA_OPTIONS and define JAVA_HOME.

Then we install Solr. For this we have to mkdir, jar, cp, and chown various files. Then edit /etc/solr/solr.xml. (Using XML in configuration files is idiotic, but there we are.)

Then we can start Jetty.

But we’re not done. Although Solr’s configuration files have shard information, Solr doesn’t use it. You have to use curl to tell Solr about its shards.

Now are we done? Beats the shit out of me, because I gave up five paragraphs ago.

OMG my head exploded

It seems to me that Solr and Zookeeper’s configuration was designed without any consideration of deployment automation. No thought was given to what an IT staff must do to install and configure this technology. It was designed (I use that word loosely) for an interactive deployment.

Many files must be edited, in the right order, and there no easy markers on which to hang sed commands. Each step can be explained to a human, but is non-trivial to code in even a powerful language like Python. And the total of all of the steps makes you throw up your hands.

In isolation, each step is necessary and reasonable. Although I itch at Javaesque parameters and XML configuration files, I don’t question any one step, except for the “myid”.

But think about how this stuff must be deployed, repeatedly, in real-world situations. When you consider all of what must be done to install and configure Zookeeper/Solr, your head explodes like those poor SOB’s in Scanners. “All” of the configuration isn’t amenable to automation with commonplace tools.

I’m not sure what to do about this. For now, I’ve moved on to another task, and I’ll mull what to do about Solr this week. Maybe just stash the post-edited copies of the configuration files in our pool?

It’s time for conferences to retire the swag bag!

This anachronism should have perished years ago. Anything I want to discover, I can discover on the web 1000x faster than I can in a swag bag. You expect me to flip through the paper collateral? If I’m interested in web-based analytics, Python IDEs, or “live chat” technology for web pages, I’ll search online. I know the swag bag is a tiny and unrepresentative subset of my options. If you’re expecting me to think, “Ooo, I’ll use their product, they’re in my swag bag!” your model of how brains work (well, my brain, anyway, but then again mine’s the archetype) is wrong.

Back in the day, you looked forward to finding the latest copies of software, or interesting trial/demo/crippleware, in a swag bag. Trial copies of Visual Studio, or a Linux distribution, or the like. Or, going way back, new products that you never heard of. None of that is true anymore.

I know swag bag material helps sell sponsor slots, and that’s how conferences defray costs. But they’re still nigh useless, and bad for the environment. It’s a matter of time before vendors wise up to how little return they get on their swag dollar. Conferences should move now to kill this ancient custom!

Every attendee received a Raspberry Pi, Model B unit. Very cool! I didn’t have any interest, so I didn’t claim mine. All the unclaimed units are being donated to charities and schools, so my Raspberry Pi will go to some worthy cause somewhere. I hope some girl or boy is inspired by it.

I wanted to take in some evening events, but except for the first night, my control rods went in around 6:30 pm and I chilled instead. On the plus side, I got lots of sleep and woke up refreshed each morning. In fact, I noticed myself dragging less on the conference’s last day than I normally do at this point in a conference. Getting good sleep is good. Who woulda thunk it?

Odd.

In past years, I posted commentary about the sessions I attended. I won’t do that this year. Don’t feel like it. Not sure why.

tl;dr

• Grades: Drobo customer support: A+. DiskWarrior: F. Disk Rescue 3: A-.
• Don’t consider your Drobo to be hot-swappable. Ever.
• Buy Disk Rescue 3 and have it on hand.
• Run Disk Utility and do a Verify Disk once a month. If that’s too often for you, do it once a quarter.

Gory details

To summarize… My Drobo S (since replaced in their product lineup, I think, by the Drobo 5D) was on my laptop’s USB 3.0 bus. I did a firmware update and *poof* it turned into a brick. By “brick,” I mean that OS X couldn’t mount it and so it was inaccessible.

I acknowledge that I may have caused this by an inopportune power-cycle. I’m ashamed to admit I was doing three things simultaneously when this happened. I think I was careful but my actions are now a blur. So I can’t conclusively say it was the firmware update that triggered it — it might have been my own fumble fingers.

My first recovery action was to try OS X Disk Utility. Its device list highlighted the Drobo in red. “Verify Disk” said it needed repair, but “Repair Disk” failed to repair it. I tried running “Repair Disk” four times in a row (this was one suggestion I received, and I was willing to try anything) but it couldn’t repair it.

Drobo customer support: A+

Plan “b” was contacting Drobo customer support. The support rep was very responsive and supportive. He asked for a diagnostics dump (via the Drobo Dashboard app), then had me try a couple of things, like hooking it up to a USB 2.0 bus and trying a different computer. Alas, none of his suggestions were profitable, because this was diagnosed to not be a Drobo problem per se, but rather a file system format problem.

But I appreciated his responsiveness (within 12 hours each time, and I responded at night after I got home from work) and his being engaged. I could tell I wasn’t getting rote answers selected off a menu by some guy in the Philippines. (Or, if I was, boy is their help-desk software good!!) He asked pertinent questions, advised me on disk repair & recovery software, and “stayed with me” during the entire saga.

If anyone in the Drobo mothership is reading this, my great support rep was “Caleb D.” His communications make me more likely to consider buying a Drobo again.

DiskWarrior: F

Plan “c” was to then buy DiskWarrior, a disk repair application. This was a complete waste of time.

OS X saw the Drobo sufficiently well so that Disk Utility saw it. And Disk Utility saw it sufficiently well to allow Verify/Repair Disk operations. The Repair Disk failed after spewing out lots of diagnostic information, so it at least gave it a shot. I assumed DiskWarrior would be no less capable (and should of course be far more capable) than Disk Utility.

Well, DiskWarrior couldn’t find the Drobo! It wasn’t even listed as a disk on which I could attempt a repair! I tried running it from its stand-alone DVD on my wife’s iMac (my MBP/Retina doesn’t have an optical drive) and attaching the Drobo to the iMac’s USB 2.0 or FireWire buses. Nothing!

I couldn’t fathom how OS X and Disk Utility saw the Drobo, but DiskWarrior — even stand-alone — did not. So, I contacted their customer support.

I got back a response that gave me hope. They felt the problem would be easier to diagnose and handle if we IM’d each other in real-time when I ran DiskWarrior. They gave me a time range when they’d be available, and their IM handle. Super!

I left work early the next day and went home. (I was also sick, so I needed to leave work early anyway. I thought about postponing this and crawling into bed, but as it turns out I didn’t need to.) I got online, IM’d them, and…

Customer support is not on-line.

WTF? I used the IM handle they gave me. They aren’t registered?!?

I emailed them and asked for a functional IM handle. No response! (I then crawled into bed and slept for 18 hours.)

Their product didn’t work. Their product didn’t work off a stand-alone disk. Their customer support blew me off. I’m sorry, but this earns a grade of F.

Data Rescue 3: A-

Plan “d” was my last available option, which was to rescue data off the Drobo. Reading disk data is different from repairing its file structure, and is easier in some ways while harder in others.

Data Rescue 3 comes in downloadable and bootable-CD form. I didn’t need the downloadable CD now but I bought it anyway.

Data Rescue 3 took days to read the Drobo, because it was reading a lot of data block by block. This got me thinking… The Drobo reported its capacity as 16TB, but contained only 12TB of disks. (“Only” 12TB of disks! I remember then 100MB was a universe of disk storage!) Data Rescue 3′s scan would have been faster if it knew this. Data recovery requires knowing details of the data you’re trying to recover — file systems, file types, directory headers, etc. It’s not a stretch that adding knowledge of a Drobo’s disk organization could speed up a scan.

Although it wasted time scanning non-existent blocks, it got the job done! It found all the files and I was able to restore them to my laptop. Yay!

Wrap-up

I’ve re-formatted the Drobo. I promise to not be sloppy about shutting down the Drobo. I promise to run Disk Utility on it once a month. I swear.

In fact, DiskWarrior also doesn’t see the Drobo when it’s run stand-alone off their DVD! I tried this on my wife’s iMac. OS X sees the disk (but does not mount it), Disk Utility lists it and allows me to try a Verify Disk and Repair Disk, but DiskWarrior doesn’t see it. I’ll try their tech support next.

During my migration, the Drobo Dashboard advised me that a Drobo firmware update was available. I did the update, which -boom- bricked my Drobo.

After trying rebooting, power-cycling the Drobo, and plugging it into the other USB socket, I’m at a point where Drobo Dashboard says the Drobo is healthy. But OS X won’t mount it. Disk Utility says:

Unable to bootstrap transaction group 6000: cksum mismatch
No valid commit checkpoint found
The volume xxxxxxx was found corrupt and needs to be repaired.
Problems were found with the partition map which may prevent booting
Error: This disk needs to be repaired. Click Repair Disk.

I then run Repair Disk, and it tells me the same thing! So Repair Disk can’t repair the disk!

I bought DiskWarrior (for $109, I’ll have you know) but it can’t repair a disk that isn’t mounted. They’ll ship me a physical CD-ROM of my purchase, so I can try booting from it. Oh, but wait, my MacBook Pro/Retina doesn’t have a CD-ROM drive!

I am not a happy camper.

We access their API via http calls. They call it a REST API. But like 95% of the “REST” APIs in the world, it’s not REST at all, and in fact nowhere near REST. The term “REST” has been corrupted to be become synonymous with, “web API”.

But whatever. It’s an API accessed with http calls.

One of service calls has a parameter called, “countryCode”, which was documented as an ISO 3166 country code.

I coded up new features that used this API. And I wrote unit tests, and debugged the code on my dev box up to a reasonable point. (It was a “reasonable point” for typical vendors, but not this one, as it turns out.) I then propagated it to the trunk, and uploaded it to our QA system.

As soon as we started system testing, we noticed they didn’t recognize some valid codes. Like the one for United Kingdom (GB).

After thoroughly checking our code for bugs, I submitted a ticket asking them to investigate. Two weeks go by with no response. Then the answer comes back: “Oh, oops. For some countries, we don’t use ISO 3166. Instead, we use our own internal codes. <I won’t bore you with their reasoning, which was nonsensical.> So you need to replace ‘GB’ with ‘UK’.”

I pointed out that this was odd, fraught with peril, and undocumented. What happens when ISO 3166 adds codes that collide with their codes? I received no response.

I sucked it up, coded a dict replacement table, and updated our servers. (I should have manually tested this. But I was pressed for time and hadn’t yet woken up to their level of intellectual rigor.)

Now the calls with “UK” were bombing!

I submitted another ticket. A week later the response comes back: “Oh, ooops. UK won’t work either. You must submit the country in which the company is located. We accept the special strings ‘england’, ‘scotland’, ‘northern ireland’, and ‘wales’; we don’t recognize any symbol for the United Kingdom.”

WTF. WTF to the max. This was absurd in an indeterminate number of ways.

I realized I’d have to create a UK gazetteer to do this. This is the subject of another blog post, but suffice to say I built a city:country UK gazetteer, hacked our code to use it, and uploaded to our servers.

Yeah, I know, I should have manually tested what they told me. I’ve since learned my lesson.

As soon as the new code was on our QA system, I saw that these service calls still didn’t work. So I contacted them again.

A couple of days later the answer comes back: “Oh, ooops. You should use ‘GB’ for the country code after all. We don’t recognize the country names you submitted.”

WHAT. THE. ACTUAL. FUCK.

I counted to 100 and very tartly replied that we started down this path by using the ISO 3166 “GB” code in the first place, and they hadn’t recognized it. Why should it now work? Are you changing your API without any versioning, notice, warning, or documentation changes?

They didn’t reply to this.

I stomped around the office for a few minutes. Then I ripped out the hack code and gazetteer table, so the ISO 3166 “GB” code would be used in our service calls. And again updated our servers.

Presto, the calls now worked as advertised.

And that’s how not to maintain an API.

Here’s an excerpt:

19,000 people fit into the new Barclays Center to see Jay-Z perform. This blog was viewed about 59,000 times in 2012. If it were a concert at the Barclays Center, it would take about 3 sold-out performances for that many people to see it.

Click here to see the complete report.

I’ve spent lots of time at work. We’re doing a big release in January, and there’s no end of bugs to fix, features to tweak, and QA logs through which to grep. It’s not a death march by any means, but it is, I admit, a bit of a grind. Far more time is being spent on technology integration than is ideal, and I had expected.

It’ll be good to get it behind us. I’m looking forward to being able to make some strategic technology and process choices afterward.

I’m looking forward to replacing my 2008-vintage MBP at home with a new model, probably in February.

End of report.

We’re doing some interesting things with name relationships at work, and these present fun development challenges. I’m trying to spend as much time as possible in Emacs, because the less-fun work issues always occur when I’m not coding.

I upgraded our codebase to version 3 of Celery, just to get us off version 2. I’m still hankering to replace Celery, but it must have known it was living on borrowed time because it’s been behaving lately, so I’ve decided to fry some bigger fish. But the moment Celery starts acting up again…

I just turned 55. How the hell did that happen?!?

————————–

Title: Senior Devops Engineer

Reports to: VP Engineering

Location: Seattle, WA

About IP Street
Founded in 2009, IP Street develops and markets software to help corporations, law firms, and research firms better analyze patent-related information. We make IP data easy to get, use, and understand.

Learn more about IP Street by visiting www.ipstreet.com.

Summary
We’re a small company that’s developed a new way to visualize and data-mine intellectual property. You’ll be part of an innovative, high-performance engineering team that’s committed to delivering our product using the best technologies and software development processes.

Our technology stack is almost all OSS, and our server-side code is Python/Django. We use esoteric search technologies. Our QA and production servers are managed servers and VMs, running Linux. We deploy with Fabric.

Most of the job will focus on:

• Monitoring, upgrading, and diagnosing our production system
• Being personally responsible for our production system’s availability and reliability
• Being the technical contact with our hosting providers
• Collaborating with developers to optimize code. And understanding and fixing product bugs yourself
• Direct involvement with new features’ design and performance projections

We’re looking for someone who is hands-on, smart, and energetic. We’ll consider someone who wants to remain an individual contributor, but we ideally want someone who wants to become the Devops lead as the company grows. We don’t know how big this function will become, or what areas of emphasis it will have. But we do know it will expand, and be extremely important to our success, as the user base grows.

So, we’d like this hire to become the Devops lead as the team expands. You’ll be able to set the pace for the group as you build it. And naturally, the future work will include team building, leadership, and management.

This position is ideal for someone who doesn’t have team management experience now, but wants to develop it as their group grows. The rest of the IP Street team will support you in this, if you’ll step up to the challenge when it comes!

Key Responsibilities

• Set up automated monitoring and alerts for our systems, and establish our corporate monitoring practices
• Set up, and/or work with consultants on setting up, master/slave farms for some subsystems, such as Redis, Lighttpd, and PostgreSQL
• Do system and high-level application performance analysis, such as: Monitoring server load over time; adjusting configurations of technologies such as lighttpd, httpd, and memcache; understanding “where the time is going” in problematic code paths; and knowing when to call in a technology expert
• Deploy hot fixes and regular releases. This includes preparing an upgrade plan, ensuring we can rollback, and communicating the schedule and status to the company
• Continually look for ways to improve our system management, e.g. in security, efficiency, and employee workload
• Set up and manage our QA staging system when we enter a release cycle
• Be our first response level for all production problems. Quickly triage them according to severity, probable cause, and workarounds
• Design and implement an operational coverage plan for when we need to expand beyond 5 x 8 support
• Work with the developers to characterize and understand bugs, and fix code bugs yourself

Qualifications

• 7+ years experience in devops. Experience in managing 20+ servers (and you aren’t hesitant about managing more of them)
• 2+ years real-world, serious Python coding experience
• Solid experience in deploying code to, configuring, and working with virtual hosting environments and Linux (Debian family) systems
• Solid experience with modern software engineering techniques and tools
• Excellent judgement in assessing problems. When you encounter a symptom with incomplete information, you’re comfortable choosing a technique or process to deal with it, and improvising as you go
• Familiarity with different approaches to expanding operational coverage
• Points awarded for experience in profiling or tuning databases or search engines (e.g., Postgres, Redis, and Solr), even if only at the system level
• Points awarded for experience with Fabric, Munin, Nagios, configuration management tools, centralized logging services or tools, Nginx, and HAProxy or similar load balancers / reverse proxies
• You would describe your skills as spanning coding, system administration, Linux, and configuring and administering OSS technologies
• You don’t just learn how things work. You also learn why!
• You’re self-sufficient and a self-starter, and confident in setting standards
• You’re eager to build a team from the ground up
• Excellent team and communication skills
• Bachelors Degree or equivalent in Computer Science or Software Engineering

To submit your resume, or for more information, contact john @ this site’s domain.