Curated awesome, the 1st
A bumpy subway wall, loving things for their Unix-y qualities, Kurt Vonnegut looking dapper, the final movement of Dvorak’s Ninth Symphony (originally his fifth), and a music video by Talib Kweli that makes me want to go get my hair cut. Oh, and I can’t leave out the connection between prototyping physical things and applications operating on large data, Ben Scoffield’s take on database taxonomy and a screed on reading one book per week.
(Editor’s note: I recently took to using Tumblr again. For a while, I’ve been curating interesting stuff here. But Tumblr has evolved into a really fantastic application for doing this. So, my policy going forward is to post my stuff here and curate other people’s awesome stuff over there. That said, I’ll probably do “best-of” posts, like this one, to keep you interested and informed.)
Chance Encounter
Another meme-ish “film” by yours truly. This time, the idea is you do something in five seconds, plus a two second intro and a one second outro. Here’s what I came up with:
This is an adaptation of possibly my favorite improvised joke. I deploy this joke when a conversation is interrupted by some disturbance or noisy distraction. Right before the conversation is going to continue, I say “…and that’s how I met the president and the pope on the same day.” Works pretty well.
Funny aside: I found out about this on the Vimeo site, thinking there was a competition this weekend. Turns out, it was last weekend. Figures.
Birthing Born to Run
The birth of Born To Run. On the creation and evolution of the song and album. Great read for Bruce-o-philes.
Representing time in our programs
The time problem is not easy to see in today's mainstream languages because there are no constructs that make time explicit. It is implicit in the system. We don't even know that's what we're doing when we use locks to try to make this work.
I’ve been thinking about how we represent time in programs for a while. The problem is that concurrent programs are all about time, but mostly, we only use two mechanisms to represent it in our programs.
The semi-explicit way is through locks. When we insert locks around some bit of code, we are giving hints to the system that things should only proceed in a certain order. This ordering gives us a notion of time, but it’s not horribly comforting.
The completely implicit way we represent time in our programs is by ordering calls to functions and the lines of code within those functions. Line 10 always executes before line 11, etc.
The problem that Rich Hickey, who has some fantastic ideas about this time stuff, has in the article I quoted is that time is managed manually and implicitly. When you start writing large concurrent programs, this falls apart. We need better constructs to deal with it.
Think of it like the shift from unstructured programming to structured programming to object-oriented programming. At first we just had a long code listing; no functions, just line after line of code. This became mentally untenable, so we shifted to structured, procedural programming. But some of our data was global and it was often hard to tell what functions belong to what data. So we moved to object-oriented programming and encapsulation.
Hopefully Rich Hickey, Simon Peyton-Jones and other functional programming folks can lead us to is a nice way to structure our programs around time. I’m eager to have my brain melted by what they conjure up.
Ain't talkin' 'bout the man
Here’s a fun game. “The Government”:
Try something. Every time somebody complains about the evils or failings of “the government,” strike out “the government” and see what results.
Often, simply striking out “government” reveals a completely different, and far more useful, commentary.
Polyglottin' your data
I think that many of the NoSQL crowd either fail to either recognize, or to properly describe that their preferred databases don’t replace applications like MySQL and Postgres, just as Ruby doesn’t replace Java. Instead, the explosion of these new options for persistence just work better for some domains (and worse for others).
Ben Scofield’s spot-on here. One of the many transitions we’re undergoing is from “I’ll use MySQL most of my career” to “I’ll tinker with a different database for every project over a couple years and then pick and choose the best as time goes on”.
Tons of FP fun
A programming language zoo, a week of FP heaven, rewriting PHP with Haskell and a game for kids of any age to learn the untyped lambda calculus. Did I ever mention it’s a good time to have a fascination with functional programming languages?
iPod Spaceman
(With due apologies to the creators of New Math, the writers of 30 Rock and the lovely iPod people.)
On American political insanity
Still crazy after all these years:
Politicians should tone down the rhetoric. Protesters should read some history before making Hitler comparisons. Talk-show hosts should stop pretending that paranoid nitwits are asking reasonable questions.
The Economist does well to explain the insanity that is propagated by American political media. Reading articles like this help me stay sane. Also: ignoring media with deadlines shorter than a week, and consuming as much constructive satire as possible.
It's not NoSQL, it's post-relational
Almost five years ago, we were witness to the reinvention of web frameworks. A couple upstarts named Django and Rails appeared at almost the same time, espousing many of the same values. In the typical Gandhi-cycle, they were first ignored, the incumbents fought them, and then they achieved victory over the incumbents. Today, any framework that’s used on new projects is likely to have more than a hint of Django and Rails in it.
Today, we’re seeing the same thing for databases. Something is going on and things are changing. Barring an unprecedented departure of fashion from software development, I’ll look back five years from now and write about some other shift in the development landscape.
But, all shifts like this need a name. Right now, the best we have is NoSQL. The problem with that name is that it only defines what it is not. That makes it confrontational and not amazingly particular to what it includes or excludes.
Damien Katz, the creator of CouchDB, has noted the need for a better name for this storage revolution. Allow me to proffer mine:
Post-relational
What we’re seeing its the end of the assumption that valuable data should go in some kind of relational database. The end of the assumption that SQL and ACID are the only tools for solving our problems. The end of the viability of master/slave scaling. The end of weaving the relational model through our application code.
We’re seeing an explosion in the ideas about how one should store important data. We’re looking at data to see if it’s even worth persisting. We’re experimenting with new semantics around structure, consistency and concurrency.
In the same way that post-modernism is about reconsidering the ways of the past in art and architecture, post-relational is a chance for software developers to reconsider our own ways. Just as post-modernism didn’t invalidate the entire history of art, post-relational won’t invalidate the usefulness of relational databases.
However, it’s likely that those working in some domains will decide that non-relational databases better fit their needs. That’s progress in our field. It’s fun to watch.
Say it with me: post-relational databases, post-relational storage, post-relational thinking, or simply, post-relational.
Blame the compiler
Remember when you first started programming? Those early days when you'd take some code out of a book or article, type it out, and then try to make it print silly things or draw funny pictures?
The thing I remember about those days was the temptation to blame the compiler for all the ills of my code. Something doesn't work right and you can't figure out why? Blame the compiler! Of course, this was never right. The compiler is very rarely incorrect.
My goal today is to bring back the joy of blaming the compiler. Of course, Ruby doesn't _have_ a compiler (yet!), so we have to play tricks. But that's part of the fun!
Let's blame the compiler.
See, all you have to do is extend BlameTheCompiler. And then you'll find yourself with one chance in every three executions to say "Hmm, I'm sure I defined that method. Something must be wrong with Ruby." Just like those early days of programming, you're half right. The other half is that you've got a little prankster running in your application.
It's a fun parlour trick. I would not, however, recommend sending code including BlameTheCompiler in as a bug report to the JRuby or Rubinius folks. They wouldn't find it as funny as you or I.
Two cellphones
People with two cellphones worry me.
(More six word stories. Also, an article as such in Wired.)
The Technology Behind Tag Better
I promised you the details on how we built Tag Better, so here we go. This is what I used to build the back-end bits. You’ll have to pester Chris or Alex to get the front-end details.
h2. Sinatra
Technically, it’s a Rails Rumble. Read between the lines of the rules and you’ll see it’s really a Rack Rumble. And so I went with my favorite for prototyping, Sinatra.
Happily, Sinatra had my back the whole time. I never came across anything that stumped me. Further, I didn’t pay any taxes for ceremony I don’t need.
Verdict: perfect tool for the job.
h2. Passenger
I hadn’t used Passenger much before this weekend. I’m pretty happy spooling up app processes in a terminal and watching the logs scroll past. localhost:3000 is my friend.
However, I’m an outlier in this regard. My teammates aren’t as interested in lower-level bits as I am, so I figured that using Passenger is the best bet to help them get the app up and running locally.
The benefit that I didn’t realize we’d get from this is running the same stack locally as on the production server. Besides some virtual host wrangling that Passenger Pane saved me from locally, getting the app up and running was painless.
Verdict: I am quite likely to keep tolerate Apache for that Passenger goodness, especially when I am the operations guy.
h2. Sprinkle + passenger_stack
The moment that I realized we’d have to set up our own server instance was one of brief, abject terror. I knew this could easily expand to fill a lot more of my time than I wanted. Luckily, I was wrong.
Ben Schwartz’s passenger_stack helped me get our Linode slice up far faster than I would have been able to by hand. I cloned his repo, tweaked it to our needs (disabled MySQL, eventually added a CouchDB recipe) and ran it on our server. Several minutes later, we had a working server. Pretty awesome.
passenger_stack uses Sprinkle, which isn’t getting as much play in the server configuration space as Puppet and Chef. Sprinkle does seem really well suited to standing up apps on a few servers. We might want to step up to something heftier once we had more servers, but Sprinkle and passenger_stack are simple to understand and don’t require any supporting infrastructure to use.
Verdict: Not too primitive, not too involved; just right.
h2. CouchDB
When I’m building any app that relies on an API as its primary data source, caching API response data is forefront on my mind. Serving the data locally, rather than making a request every time, means the app feels more responsive. An added benefit is not upsetting the upstream data provider.
I’ve built apps like this that use MySQL as a cache and it just never felt right. I’ve been tinkering with CouchDB and Tokyo Cabinet/Tyrant lately. I decided to go with CouchDB for this one because of the excellent CouchDBX, which makes it easier for those who don’t even know what Erlang is to use CouchDB.
CouchDB ended up working pretty well. While we haven’t really leaned into it, it didn’t present any challenges while I was developing. Using CouchRest with Sinatra worked just fine.
Verdict: It just worked, which is exactly what I needed.
h2. Skipping traditional TDD
OK, so maybe only Jared Diamond would consider this a technology. But skipping the writing of tests to drive my design was pretty helpful. Consider Kent Beck’s flight metaphor. Doing a Rails Rumble is just like the taxi-ing phase. Or a minimum valuable product. Either way, you want to make a small investment towards validating an idea.
Notice I said traditional TDD. To tell the truth, I did write a sniff test script after I had the basic app working. But it wasn’t an xUnit-style test. It’s just a shell script that bangs on the app with fixed parameters. I do have to manually inspect it to make sure nothing is blowing up. What I’m really automating here is the pain of typing out Curl commands.
Verdict: worked great for the original purposes, but I’ll probably add a proper test suite as one of the first post-contest enhancements
So that’s what I think helped make our project go off pretty well. Really, what they did was help me get stuff done and then get out of the way. Isn’t that the best kind of tool?
Tag Better
Yesterday and today, I worked with Alex Bischoff and Chris Griego on a Rails Rumble project. In less than forty-eight hours, we set out to build a web application, using Ruby, from the ground up. While we didn’t boil the ocean, we did come up with something interesting.
Tag Better is the groundwork for an app that enhances one of our favorite, but often neglected, sites: del.icio.us. Personally, I’ve tried other bookmarking tools such as a plain text file, EagleFiler and pinboard.in (which I think has an excellent chance of eclipsing Delicious). However, I’ve found myself back on Delicious, even after a hiatus.
Given Yahoo’s unsteady stewardship of the site, we figured it’s time to take matters in our own hands. Using their API, it’s entirely possible for us to build enhancements on top of Delicious. We decided to do just that.
h2. Triage and tidying
We started out from a gem of an idea that Chris had. When you’re trying to better organize your tags and bundles in Delicious, there are two modes you might want to operate under. The first couple times, you need to do triage. Get stuff roughly into order, and start creating a filing system. Once you’ve got everything under control, you switch to tidying. Now you just need to keep things clean or maybe switch things around that aren’t working well.
We tackled triage first. In specific, we wanted a way to filter down our list of tags, select a bundle, and add some of the filtered tags to that bundle.
h2. Try it out
To get started with Tag Better, you’ll first need to log in with your Delicious username and password (more on that in a moment.)
Once you’re logged in, you’ll see your tag bundles on the left and all your tags on the right. If you click a bundle, the tags that are included in that bundle are highlighted. Clicking a tag toggles its inclusion in the bundle. Changes in Tag Better are immediately reflected in Delicious.
That’s the core interaction. You can also logout, of course. And, if you feel the need, we offer a way to remove your bundle and tag data from our system’s caches.
We were just four short characters away from getting live filtering of your tags via the search box on the top right. Alex and I even got it working, but keyboard cat had already played us out at that point. I promise we’ll add it as soon as judging is over!
A couple other caveats: you might want to create a test bundle (in the Delicious UI) to play with, rather than one you’ve already organized. I don’t think there are problems in the code, but it never hurt to show caution. Also, IE support is probably “interesting”, at this point.
h2. Trust
Even though we need your Delicious username and password to edit bundles on your behalf, we have made sure to never store authentication information on our side. Your authentication data is stored in a cookie on your machine; if you don’t like that, you should probably skip out on using Tag Better as a hosted app. If you delete that cookie, we won’t operate on your data. Further, we’ve added the ability for you to remove your bundle and tag data from our system. That’s the only data that we save on our disks.
I mentioned trust and your authentication data earlier. It’s an important part of an application like this. We don’t want to screw your bookmarks up, and you don’t want us to do skeezy things with your bookmarks. Unfortunately, Delicious does not yet offer a delegated authentication API like Flickr and Twitter do. So, we’ve got two options.
On the one hand, you could trust us. This is up to you. I’d like to think that if you’ve met Alex, Chris or I, then you’re happy to use the application knowing we’ve got more amusing things to do than soil your bookmarks. Further, when the competition is over, we’ll open the source repository up to the public. If you’re a code-review sort of person, go nuts.
On the other hand, you could run it yourself. Again, once the competition is over, we’re going to release the code. Look over the readme, install the dependencies, and run it on your own hardware. Sleep safely at night knowing your password is safely ensconced on your personal machine.
Tomorrow I’ll talk a bit about the technology we used to make Tag Better. Until then, I hope you’ll give it a try. If you have ideas for feedback, leave a comment here!
Fun
Cannonball Adderley:
It's called "Fun". F-U-N, fun. That's something you can do, when everything is mellow.
Here’s to mellow times. Seemed appropriate for a Friday.