It's not NoSQL, it's post-relational

Almost five years ago, we were witness to the reinvention of web frameworks. A couple upstarts named Django and Rails appeared at almost the same time, espousing many of the same values. In the typical Gandhi-cycle, they were first ignored, the incumbents fought them, and then they achieved victory over the incumbents. Today, any framework that’s used on new projects is likely to have more than a hint of Django and Rails in it.

Today, we’re seeing the same thing for databases. Something is going on and things are changing. Barring an unprecedented departure of fashion from software development, I’ll look back five years from now and write about some other shift in the development landscape.

But, all shifts like this need a name. Right now, the best we have is NoSQL. The problem with that name is that it only defines what it is not. That makes it confrontational and not amazingly particular to what it includes or excludes.

Damien Katz, the creator of CouchDB, has noted the need for a better name for this storage revolution. Allow me to proffer mine:

Post-relational

What we’re seeing its the end of the assumption that valuable data should go in some kind of relational database. The end of the assumption that SQL and ACID are the only tools for solving our problems. The end of the viability of master/slave scaling. The end of weaving the relational model through our application code.

We’re seeing an explosion in the ideas about how one should store important data. We’re looking at data to see if it’s even worth persisting. We’re experimenting with new semantics around structure, consistency and concurrency.

In the same way that post-modernism is about reconsidering the ways of the past in art and architecture, post-relational is a chance for software developers to reconsider our own ways. Just as post-modernism didn’t invalidate the entire history of art, post-relational won’t invalidate the usefulness of relational databases.

However, it’s likely that those working in some domains will decide that non-relational databases better fit their needs. That’s progress in our field. It’s fun to watch.

Say it with me: post-relational databases, post-relational storage, post-relational thinking, or simply, post-relational.

The Technology Behind Tag Better

I promised you the details on how we built Tag Better, so here we go. This is what I used to build the back-end bits. You’ll have to pester Chris or Alex to get the front-end details.

h2. Sinatra

Technically, it’s a Rails Rumble. Read between the lines of the rules and you’ll see it’s really a Rack Rumble. And so I went with my favorite for prototyping, Sinatra.

Happily, Sinatra had my back the whole time. I never came across anything that stumped me. Further, I didn’t pay any taxes for ceremony I don’t need.

Verdict: perfect tool for the job.

h2. Passenger

I hadn’t used Passenger much before this weekend. I’m pretty happy spooling up app processes in a terminal and watching the logs scroll past. localhost:3000 is my friend.

However, I’m an outlier in this regard. My teammates aren’t as interested in lower-level bits as I am, so I figured that using Passenger is the best bet to help them get the app up and running locally.

The benefit that I didn’t realize we’d get from this is running the same stack locally as on the production server. Besides some virtual host wrangling that Passenger Pane saved me from locally, getting the app up and running was painless.

Verdict: I am quite likely to keep tolerate Apache for that Passenger goodness, especially when I am the operations guy.

h2. Sprinkle + passenger_stack

The moment that I realized we’d have to set up our own server instance was one of brief, abject terror. I knew this could easily expand to fill a lot more of my time than I wanted. Luckily, I was wrong.

Ben Schwartz’s passenger_stack helped me get our Linode slice up far faster than I would have been able to by hand. I cloned his repo, tweaked it to our needs (disabled MySQL, eventually added a CouchDB recipe) and ran it on our server. Several minutes later, we had a working server. Pretty awesome.

passenger_stack uses Sprinkle, which isn’t getting as much play in the server configuration space as Puppet and Chef. Sprinkle does seem really well suited to standing up apps on a few servers. We might want to step up to something heftier once we had more servers, but Sprinkle and passenger_stack are simple to understand and don’t require any supporting infrastructure to use.

Verdict: Not too primitive, not too involved; just right.

h2. CouchDB

When I’m building any app that relies on an API as its primary data source, caching API response data is forefront on my mind. Serving the data locally, rather than making a request every time, means the app feels more responsive. An added benefit is not upsetting the upstream data provider.

I’ve built apps like this that use MySQL as a cache and it just never felt right. I’ve been tinkering with CouchDB and Tokyo Cabinet/Tyrant lately. I decided to go with CouchDB for this one because of the excellent CouchDBX, which makes it easier for those who don’t even know what Erlang is to use CouchDB.

CouchDB ended up working pretty well. While we haven’t really leaned into it, it didn’t present any challenges while I was developing. Using CouchRest with Sinatra worked just fine.

Verdict: It just worked, which is exactly what I needed.

h2. Skipping traditional TDD

OK, so maybe only Jared Diamond would consider this a technology. But skipping the writing of tests to drive my design was pretty helpful. Consider Kent Beck’s flight metaphor. Doing a Rails Rumble is just like the taxi-ing phase. Or a minimum valuable product. Either way, you want to make a small investment towards validating an idea.

Notice I said traditional TDD. To tell the truth, I did write a sniff test script after I had the basic app working. But it wasn’t an xUnit-style test. It’s just a shell script that bangs on the app with fixed parameters. I do have to manually inspect it to make sure nothing is blowing up. What I’m really automating here is the pain of typing out Curl commands.

Verdict: worked great for the original purposes, but I’ll probably add a proper test suite as one of the first post-contest enhancements

So that’s what I think helped make our project go off pretty well. Really, what they did was help me get stuff done and then get out of the way. Isn’t that the best kind of tool?

rufus-tokyo goes 1.0.0

rufus-tokyo 1.0.0 - I’ve been tinkering with Tokyo Cabinet and Tyrant lately. It’s great stuff. Grab this gem and start tinkering!