Aside
Developers are weird with words
Naming things is hard. Witness things that developers have named and then struggle to explain because words and people are weird:
- TDD sounds like it's about testing, but it's really a design technique.
- BDD sounds like it's about what code does, but it's really a communication discipline.
- Outside-in development sounds like a way to discover the design of software, but it's really a technique for building software using a fractal todo list.
Finishing software ain't easy
When I start work on a project, whether for personal or professional purposes, I have a sense that I need to devote myself to it. That I should figure out how to make it the best it can be, I should only commit the best codes to it; it should be an exemplary piece of work.
On the one hand, this is taking pride in my work, and that’s good! On the other hand, this is ownership, and that’s a little problematic.
The problem with ownership is it leads to irrational decisions. There’s always one more improvement I could make, because this is the thing I own. There is little bit of scope I could add, because I can make this thing really good (at least in my head). There’s one more bug to fix (inexplicably, in my head).
What I’m finding all this points to is, finishing is hard. It’s easy to start a project. It’s challenging to get it out there, and get people to use it. It’s really hard to get it to the point where it’s running itself, you can delegate its operation to others, or it doesn’t need further work.
So that’s a thing I’m trying to figure out. Developing.
The simple problem inside the complex one
A sophisticated solution to a complex problem is fun to find. Its even fun when someone else finds the solution and thoroughly writes it down.
Despite the thrill of recognition for tackling a big complex problem, I find it's more practical to seek the simple problem lurking in and around the big problem. Not all tricky problems are complex. Some are presented in a complex way, some are complex because of restrictions that are easily worked around, and some are complex because of an adjoining social problem. Find the simple problem, or the social problem, and solve that instead. It often works for me.
My favorite kind of solution is one where the solver has taken a problem with a big surface area, found the core of the problem that is 80% of what people care about having solved, and then solved that tiny subproblem. If you want to see masters of this approach, check out the works of Blake Mizerany and Ryan Smith.
Scala and Clojure in terms of city building
The Scala folks are building newer, better cities on top of older cities, which is how things really work (e.g. Paris, Rome, Boston, etc.). The Clojure folks are building Dyson spheres in space, which is a little ambitious and maybe not entirely realistic.
That said, sometimes wouldn’t you rather live in a sci-fi fantasy world rather than on top of layer after layer of archaeology?
Intermediate variables, organizing OO, meeting Grinders half way
I work with Dave Copeland at LivingSocial, but not on the same team. Maybe someday I’ll fix that, but for now I learn a lot from his writings. Herein, a few things worth checking out yourself.
If you ever need to read my code, you’ll eventually come to suspect I have a particular dislike for intermediate variables. You’ll come to suspect this through finding lots of uses of inject and tap, two Ruby methods not everyone is on good terms with. You can imagine I’d side with Dave on the subject of Tap versus intermediate variables. You’d be right, but Dave says it so well, you should read his take on the joy of tap. He also shows how to annoy people with tap-like constructs in other languages. If you’re into combinators, Reg Braithwaite has written about tap in terms of Kestrels.
I’ve been learning a lot about how to think about organizing a non-trivial object-oriented system this year. Gary Bernhardt is doing some fantastic work explaining a hybrid imperative/object/functional system. If you don’t have time to dive into Gary’s entire backlog (it’s worth it to find the time), Dave covers some similar ground describing the only four types of classes in your OO system. Think of these as a post-hoc observation on how many systems seem to evolve; Record objects take root, Service objects reveal themselves (often intertwined amongst other objects), Builders are sprinkled throughout, and there are a few classes hanging out that you wish you’d made immutable. These are handy guides for thinking about and refactoring an existing design. That said, I think it would be overkill to start a design with these archetypes. Caveat: some developers will really dislike organizing a system this way; tread carefully.
I’ve written about the virtues of The Grinder. I know a lot of non-Grinders wish that the Grinder knew more about how to take the code they’ve made to work and improve it so that it is more malleable in the future. Making it Right: Technical Debt vs. Slop sets out a good mindset on how this can happen. Think before you type, write a test, make it work, and then tidy it up with future malleability in mind. From there, non-Grinders need to meet Grinders in the middle is in shrinking the feedback cycle. When tests are too much effort to write or take too long to run, Grinders fall back to their old habits. When making it right involves too many intermediate steps with nothing to show for it, Grinders move on to the next thing. When a non-Grinder learn to be less precious with our work, or a Grinder learns to take a moment to round off the sharp corners on their work, you end up with a much stronger team. Fight for it.
A pithy take on development vs. operations
The essential, face-palming difference between too many development teams and too many operations teams is thus:
Development: “I know how it works, but I don’t know how to make it work”
Operations: “I know how to make it work, but I don’t know how it works”
Be the solution, friends.
Future lies
It’s easy to delude yourself when writing software. Do these tests really describe what the application does? Does the documentation really describe how the system works now? Is this comment an accurate assertion on the state of affairs in the application?
My experience is that there’s little to solve this problem besides discipline. Always double check that you haven’t invalidated something that was written down in the margins. If there’s a way to encode something in code instead of prose, do it.
Vigilance against future-lies is an ever-mindful challenge.
Too eager to add code
I’m a little too eager to add code. If there’s a mess that needs refurbishing, rather than refactoring, I’m too quick to create a parallel world that is nice and tidy like I’d like it. Problem is, I don’t come back to the code in want of refurbishing enough. I know I should rejigger it to use the new shiny bits. For some reason, call it inertia, I don’t.
This is a shot across my own bow. Prefer refactoring to refurbishing. Prefer refurbishing to jumping into something new. Prefer shipping code to all of the previous tactics.
Stand on the shoulders of others' REST mistakes
Like all API design, putting a REST API on your app is tricky business that most people learn through lots of mistakes. So stand on the shoulders of other peoples mistakes! Thus, REST worst practices:
In the REST world, the resource is key, and it’s really tempting to simply look at a Django model and make a direct link between resources and models — one model, one resource. This fails, though, as soon as you need to provide any sort of aggregated resource, and it really fails with highly denormalized models. Think about a Superhero model: a single GET /heros/superman/ ought to return all his vital stats along with a list of related Power objects, a list of his related Friend objects, etc. So the data associated with a resource might actually come out of a bunch of models. Think select_related(), except arbitrary.
Mistaking the app’s internal model with what API users want to work with was the mistake I made on the first API I wrote.
Any big API is going to need to have dedicated servers that just serve API applications: the performance characteristics of large-scale APIs are so different from web apps in general that they almost always require separately-tuned servers.
This is how I prefer to roll my APIs lately. At the least, they should be a separate set of controllers. If you can extract a completely different application even better.
Represent dat API
Rails is missing an abstraction when it comes to building REST APIs, in my opinion. Requests route through controllers, controllers call models or services to obtain the right objects. And then…you awkwardly try to bang a JSON object together with an ERB template? It gets awkward quickly.
There’s a lot of experimentation in the wild attempting to figure out what works well here. You can bang out a bunch of presenter classes. You can describe and compose representations. You can go resource oriented.
I came across one yesterday that immediately caught my eye. You could just use lambda to implement a bunch of functions that present, decorate, or map objects from one representation to another. To borrow an example:
# Define a base representation
UrlsPresenter = lambda do
{
'self' => "#{Gauges.api_url}/me",
'gauges' => "#{Gauges.api_url}/gauges",
'clients' => "#{Gauges.api_url}/clients",
}
end
# Compose the base representation with more data
UserPresenter = lambda do |user|
{
'id' => user.id,
'email' => user.email,
'name' => user.name,
'urls' => UrlsPresenter.call
}
end
# Pass an object to the presenter and convert it to JSON
UserPresenter[user].to_json
I love that this one adds no machinery and no state. Input, function, output. With just lambda, you can describe a bunch of transformations and string them together into meaningful and interesting pipelines. I’m experimenting with this now, hoping to find an interesting way that functional programming approaches can make it simpler to build APIs with Rails.
Four essential topics of 2011, in charts
The Year In 4 Charts: Planet Money does an excellent job collecting four economic charts (themselves chosen from three collections of best-of charts). I’m a dilettante as far as economics and economics go, but these charts do a great job of rolling up what seemed to have been the essential stories of the year.
A picture can nullify a thousand talking points, no?
Making a little musical thing
After software development, music is probably the thing I know the most about. My brain is full of history, trivia, and a modest bit of practical knowledge on how to read notation and make music come out. That said, I haven’t really practiced music in several years. I’ve been busy nerding out on other things, and I’ve grown a bit lazy. Too lazy to find people to play with, too lazy for scales, too lazy to even tune a stringed instrument. Very, very lazy.
Long story short, I’ve been wanting to get back into music lately, but I want to learn something new. Something entirely mysterious to me. Given my recent fascination with hip-hop, I’m eager to try my hand at making the beats that form the musical basis of the form.
There are a lot of priors to cover (tinkering with various sequencers, drum machines, and synthesizers; steeping myself in sample culture; listening to the actual music and understanding its history), but I just made a short, mediocre little beat and put it on the internet. Herein, I reflect on making that little musical thing:
- I’m sure that, if I get serious about this, I’ll need real software like Ableton or Logic. But for my tinkering, it turns out GarageBand is sufficient. The included software instruments aren’t amazing or even idiomatic samples (no TR808, no “Apache” break included), but with a little bit of tinkering, they produce results.
- Laying a drum track down that is little more than a fancy click track helps to get started. GarageBand has a handy feature where you can define the a number of bars as a loop and then record multiple takes, review them, and discard the takes you don’t want.
- What an app lacks in samples you can make up in effects. Throwing a heavy dose of echo and a ridiculous helping of reverb made an otherwise pedestrian drum track way more interesting.
- I didn’t go into this with anything in my head that I wanted to make real. For the drum track, I ended up with a pretty typical beat. A little quantization made it end up sound better and more interesting than it really is. This process, manual input with some computer-assisted tweaking, produced way better results than the iOS drum machines I’ve used in the past.
- Tapping out the bass-line took a little more time than the drums. I didn’t have anything “standard” in my head, so I doodled a bit. This is where the “takes” gizmo in GarageBand came in really handy. Record a bunch of things, decide which one is most interesting, clean it up a little, throw an effect or two on it to make it more interesting, on to the next track.
- In retrospect, lots of effects is maybe a crutch. I don’t have enough taste yet to tell.
- With the drums and bass down, it’s time to adorn the track with a melody or interesting hit for effect. I added one subtle thing, but couldn’t think of anything I liked that was worth making prominent. If I were actually trying to use this beat for something, I’d keep digging. But for my first or second beat, it’s not a big deal.
I wanted to jot down my thoughts because I’d like to write more about making and understanding music, but also because I keep meaning to write down what I find challenging and interesting as I start from a “beginner’s mind” in some craft or skill. And so I did.
You’re six hundred words into this thing now, so I’ll reward you, if we could call it a reward, with “An Beat”.
Crafting lightsabers, uptime the systems, a little Clojure
Herein, some great technical writings from the past week or two.
Crafting your editor lightsaber
Vim: revisited, on how to approach Vim and build your very own config from first principles. My personal take on editor/shell configurations is that its way better to have someone else maintain them. Find something like Janus or oh-my-zsh, tweak the things it includes to work for you, and get back to doing what you do. That said, I’m increasingly tempted to craft my own config, if only to promote the fullness and shine of my neck beard.
Uptime all the systems
Making the Netflix API More Resilient lays out the system of circuit breakers, dashboards, and automatons Netflix uses to proactively maintain API reliability in the face of external failures. Great ideas anyone maintaining a service that needs to stay online.
List All of the Riak Keys, on the trickiness of SELECT * FROM all_the_things-style queries in Riak, or any distributed database, really. The short story is that these kinds of queries are impractical and not something you can do in production. The longer story is that there are ways to work around it with clever use of indexes and data structures. Make sure you check out the Riak Handbook from the same author.
A little bit of Clojure
Introducing Knockbox introduces a Clojure library for dealing with conflict resolution in data stored in distributed databases like Riak. If you’re working with any database that leaves you wondering what to do when two clients get in a race condition, these are the droids you’re looking for. I would have paid pretty good money to have known about this a few months ago.
Clojure’s Mini-languages is a great teaser on Clojure if, like me, you’ve tinkered with it before but are coming back to it. This is particularly useful if you’ve seen some Lisp or Scheme before, but are slightly confused by what’s going on with all the non-paren characters that appear in your typical Clojure program. Having taken a recent dive into the JVM ecosystem, I have to say there’s a lot to like in Clojure. If your brain understands static types but thinks better in dynamic types (mine does), give this a look.
I occasionally post links with shorter comments, if you’d like a slightly more-frequent dose of what you just read.
Quality in the inner loop
In software, this means that every piece of code and UI matters on its own, as it’s being crafted. Quality takes on more of a verb-like nature under this conception: to create quality is to care deeply about each bit of creation as it is added and to strive to improve one’s ability to translate that care into lasting skills and appreciable results.
When I wrote on “quality” a few months ago, I was thinking of it as an attribute one would use to describe the outer loop of a project. Do a bunch of work, locate areas that need more quality, but a few touches on those areas or note improvements for the next iteration, and ship it.
But what Brad is describing is putting quality into the inner loop. Work attains “the quality” as it is created, rather than as a secondary editing or review step. Little is done without considering its quality.
I’m extrapolating a bit from the letter of what Brad has written here, but that’s because I’ve been lucky enough to work with him. Indeed Brad’s work is of consistently high quality. Hopefully he’ll write more specifics about how quality code is created in the future (hint, Brad), and how much it relates to Christopher Alexander’s “quality without a name”.