Less beautiful code, more code that works in production

Developers should care and feed for their systems, especially when they’re in production. Alerting, logging, and metrics are the tools you need. Further, you need to use actual production errors as a feedback loop to push you to write more fault tolerant code.

I’ve struggled with that part in the past. It’s tempting to think I can just code along, building beautiful code. Beautiful code is entirely different from production code. Handling errors, timeouts, partial failures, logging, instrumentation: it’s not exactly design pattern stuff. To wit:

Fault-tolerant code is ugly. It requires conditionals. It won’t look like code in programming books, screencasts, or blog entries. This is the way it has to be, I’m sorry.

Production is all that matters. Highly intelligent words from Dave Copeland. The beauty of code, as it appears in my text editor, is not as important as how it works when it runs on servers. New mantra!

Uncertainty, feedback, confidence, a happy ending

Code

Yesterday, I wanted to setup a quick feedback loop for writing some production code. But, I wasn’t entirely sure what that code should look like. Further, I hadn’t worked in the code base for several months, so I wasn’t sure how the new code would need to interact with other objects.

This is a scenario that often gets me in trouble. I end up spending too much time contemplating how things might be without the feedback of what’s possible and plausible.

What I ended up doing:

  • Start a new test case for the code. I hardly know what it should do, so I ended up with it "should do things".
  • Write the kind of code I would run in a REPL to get started writing the production code. This was a service client, so I did things like setup a connection, invoke a request, and handle a response.
  • Write a test assertion about what I think the side-effect or result of the production code should be. For this service client, I wanted to make sure a useful thing was returned.
  • Run the test, see it fail, learn some things about the REPL-y code I wrote. Since I’m writing Ruby, my feedback rate goes way up here. Running a single test, which I can do with a keystroke in my editor, is way faster than switching over to an IRB, reloading code, and finding the right incantation to run again.
  • Iterate on the code in the test case until I have something that passes. Think, type, feedback. Observe, orient, decide, act. Coding is fun.
  • Once I have passing code, start rearranging so its closer to what I want as actual production code. Replace dummy values with real ones. Figure out how to remove any cheats I may have put in place for expediency.
  • Once I know what the messages are, extract them into helper methods on the test case class. Now I’m starting to do actual design. After each change, I run the tests. If I break the test, I go back and fix it before I proceed. This is Legitimate Refactoring. Feels good.
  • Once I know what the objects are, I go ahead and create their file in the production code, move the code out of the test and into the production code, and make sure the original test still passes. Now I switch back to a more typical test-driven design cycle; write a test case, watch it fail, make it pass, refactor, repeat.

This is sort of like the Saff Squeeze, but for exploratory coding rather than defect isolation. It kept me from spinning my wheels on design and organization before I even knew what the forces are that affect said design and organization. It also makes a new, strange, or messy code base less intimidating; I can incrementally integrate with the existing code as I’m ready to dive further into it.

Give this a try some time. It’s a great confidence booster.

Use what you got

How Shopify scales Rails was one of my favorite talks at Big Ruby. Therein, John Duff talks through what Shopify’s Rails stack looks like and why it works for them. What impressed me most was that Shopify has been running basically the same stack since they started. Ruby, Rails, MySQL, Memcached. They added Redis in for queuing at one point.

I’m so easily tempted and fascinated by The New Shiny, it’s refreshing to read about a shop getting by with The Old Trusted instead. There’s a lot to be said for using solid, well-known tools for as much as you can and working with the strengths and weaknesses those tools afford you. It’s a lot closer to craftsmanship than the temptation of science fiction New Shiny tools, and less likely to turn your guts inside out when something goes sideways.

Pet peeve 74: whenever I slip and focus on complaining about who and what instead of thinking about how and why to solve the problem. This is doubly frustrating because I enjoy solving puzzles (i.e. problems) more than I enjoy annoying people.

It’s so, so easy to kvetch. It feels good. But, it’s so much more useful to figure out why the problem happened and how to solve it or break the problem done to solvable subproblems.

Being human: it’s tricky.

My Rite of Spring overfloweth

Music

Today’s the hundredth anniversary of premier of hometown favorite Igor Stravinsky’s Rite of Spring. The National Public Radios are all over this. NPR Music and WQXR are collecting a bunch of articles they’ve done in the run-up to the anniversary. My favorites: a visualization of the score and a list of essential recordings. The latter features this amazing image of Stravinsky, which I’ve already posted twice today and will continue posting it until it just doesn’t feel right anymore.

Igor Stravinsky in a giant coat, a hat, and sunglasses

Indeed, Igor Stravinsky will be on that suit and tie.

If you’re playing along today, WQXR is streaming twenty-four hours of nothing but the Rite. Highly recommended, if you have the means.

Entropy and anti-entropy on your codebase

Entropy and Evolution of a Codebase goes well with Your Application is on Fire:

If you imagine the modules in a codebase like cells in a Game of Life automaton, you could see cells fading from healthy blue to sickly red, then transmitting their disease elsewhere. Very occasionally, you’ll see a cell or cluster of cells brightening to health as a developer restructures that area. Mostly, the codebase will decay until it must be discarded or rewritten.

Michael Nygard reflects on changes that increase and decrease the entropy of an application, what tends to introduce those changes, arrives at this wonderful metaphor for the thinking about the health of an application over time, and then offers several ways to manage entropy in a codebase. This is one that will stick in your head for a while.

If you haven’t read Michael Nygard’s Release It, you’re missing out on a lot of great stories and useful ideas on how to maintain production software.

Sandi’s Rules

Code

One day, Sandi Metz was pressed by a team she was working with to produce a simple set of rules that would lead to better code quality. After consideration, she came up with the following surprisingly numerical guidelines: classes can’t grow beyond one hundred lines of code, methods can’t grow beyond five lines of code, methods can only take four parameters (hash options count), and controllers can only send one message to one object.

What emerges from these rules is a pretty pragmatic lens on how to practice object oriented design with Ruby and Rails without falling into the tarpit of more radical approaches. You don’t end up needing to worry about fast tests, decoupling from the framework, presenter, conductors, mediators, adapters, ad nauseum.

More critically, you aren’t beset with decision fatigue. You don’t have to survey the landscape of helper gems and bolt-on approaches to writing Rails applications. You can start with the way Rails wants you to write applications: logic and data encapsulated in models, behavior encapsulated in controller actions, easy sharing of data between actions and views with instance variables, etc.

A wingding

You start with what Rails wants; when you find yourself violating one of Sandi’s rules, then you apply design and refactoring techniques. If a model does too much, extract common behaviors into an encapsulated object. If a controller action knows about too many things, move that into an object and call it from the action. You can go a really long way using only the Replace Method with Method Object refactoring until you get to Kent Beck’s notion of a Composed Method.

That bears repeating. In big letters.

Most Rails applications can be made better by relentless application of Method Object and Composed Method.

Thoughtbot has written about what this looks like as it emerges. It doesn’t have to involve a lot of pattern names, new frameworks, framework asceticism or half-baked conventions. You write classes and methods when needed. When they get too big, you refactor, rinse, and repeat.

A wingding

Sandi’s book, Practical Object Oriented Design with Ruby, is great not because it covers new territory or casts a new light on the subjects of object oriented programming and Ruby. It’s great because it shows how you work with the tools provided by OO and Ruby. It’s great because it shows how you go from less-good code to more-good code.

It’s really great because its short and easy to read. This is the book that enthusiastic, “enlightened” programmers can hand to the less energetic to help them understand how to write better code. They can do so without worrying that the book is hard to understand or a beast to read. It’s as close to a magic pill for clue enhancement as anyone has yet come.

Your application is on fire

Code

Six easy pieces on thinking about sustainable code

Your application is on fire. Something is consuming it, a process converting fresh and shiny code into rigid, opaque legacy code.

Seriously, your applications is on fire. You should look into it.

Are you going to fight the fire? Will you keep throwing logs into it, hoping that you can corner it at some point down the line? Do you draw a line in the sand and start containing the fire and eventually suffocating it?

The fire is every change you make to your application in haste. The fire is every design decision that is bolted onto one part of the application without care for how it fits into another part of the application. The fire is every stopgap put in place to fix a short-term problem that isn’t revisited once the short-term has elapsed.

Once time is not the most critical concern, you stop and think about how to make the application better in the long term. You consider what worked about the stop-gap solution, how to design your application with care, how to iterate on improving the architecture of your application.

You fight the fire by trading time for quality. Knowing how to solve your problems, knowing what is important about your application and what is consequential, you start to express a design. You write down principles that worked for you and write down an architecture that joins the principles and the design in the middle. That’s where your application lives, once it’s not on fire.

The reason green field software projects are appealing is that they aren’t on fire. A green field is lush with possibility, without the constraint of decisions already made. More so, a green field is not bound by decisions implicitly accepted but never considered.

The unconsidered decision is how the fire starts. A class here, a subsystem there. Soon you’ve got a system but no design, emergent or otherwise. You’ve got a ball of mud, but the mud is on fire.

That fire is sustained by every time you bolt something else on. It’s sustained by implementing the feature or fixing the bug without care for the program as a whole. It’s fed by making progress without making care.

The fire is warm, but eventually it will consume your application. No one wants to work on the burnt out wrecks inside of it.

Everyone can smell it, but no one wants to go near it. Firewalls are erected, teams establish practices that allow them to avoid working on or near the fire.

So it burns on.

One day, you decide enough is enough. No more fuel on the fire. Every change to the application has to meet standards: does this make the code better, does this make the design better? This here is the design; adhere to the why’s, use your creativity for the what’s and how’s.

That day is the day you start thinking for the long term. In the language of economics, its when you internalize external costs. You accept that the cost of maintaining this application is worthwhile, so you should mitigate that cost, optimizing input effort for output value.

You can’t maximize cost by just getting the work done. You can’t maximize value by building grand structures. You optimize cost for value by continuously making decisions that mitigate or reduce the fire.

Internal combustion engines are made of fire. Despite being powered by flame, combustion engines are pretty mundane things. They became mundane because they have a design and a standard of engineering that contains that fire, harnesses it, and turns it into a mechanism for converting fuel into energy.

When do we harness and focus our code, instead of letting it burn inefficiently and out of control? The moment we do is when we stop being firefighters and start being the wizards we set out to be when we started writing software.

What makes longevity?

Music

A joke for a late-night variety show monologue may only be funny for one day (e.g. a joke about a celebrity). A newspaper article may lose relevance in days or weeks. A TV show might feel dated a couple years after its run ends (e.g. most problems on Seinfeld could be solved with a smartphone). Computer programs don’t fare well over time either (though there are exceptions).

The best songs demonstrate better longevity. Beethoven and Bob Dylan still work today. There will always be something amazing about “Good Vibrations”, at least to the trained ear.

Even the rap trope of yelling the year the song was recorded has a timeless quality to it; it serves as a marker for the state of affairs. “Nineteen eighty nine!” is the first thing shouted in Public Enemy’s “Fight the Power”, marking the historical context for what Chuck D is about to tell us.

Why is this? I suspect it underlies the act of making music. Besides hit factory music, i.e. ear worms you hear on the radio, a musician’s goal is to make something expressive. Expressiveness often leads to qualities that give a piece endurance; timelessness, nostalgia, high quality. Expressiveness is less often an objective for jokes, headline journalism, or television.

That enduring quality, it’s tricky. It happens in film, television, and books too. But, for me, there’s something about music that has a more direct emotional connection. They vibrate my ear drum and work their way directly to a part of my brain containing “the feels”. I hear a good song and I’m immediately thinking about why I enjoy it so much, what makes it so good, or when I heard that song and connected it to an experience.

Maybe that’s why music is such a big deal in our culture. Really, really good music connects in a way beyond “haha that’s funny” jokes or “huh, that’s interesting” writing.

Is it possible to write expressive non-fiction or an enduring computer program? It seems like the answer is yes, but the answers are outliers. Hofstader’s Godel, Escher, Bach and Knuth’s TeX come to mind. For every GEB or TeX, there are thousands of less interesting works.

Further, the qualities of an enduring, expressive, and yet functional work seem somewhat at odds with the pragmatics of the daily act of writing or programming. A lot more perfectionism, experimentation, and principle goes into these works than the typical news article or application.

And yet, for every “Good Vibrations”, there’s probably a thousand commercial jingles composed, elevator tunes licensed, ringtones purchased, and bar bands playing “Brickhouse” yet again. Perhaps music is just as prone to longevity as writing, film, or programming but has a far longer timeline on which its easier to see what really worked. In fifty years perhaps, if we’re lucky, we’ll start to learn what is really amazing in film and software.

The downsides of live music

Music

I am a giant music nerd. I listen to a ton of music, I think about music a lot, and I often seek out new music via Twitter and Rdio. Besides a dislike for showtunes and reggae, I’m a pretty open-minded listener.

Yet, it is exceedingly rare that I seek out live music. When I do, I’m that concert goer who is only buying tickets for long-established acts. In the past several years, I’ve seen Paul McCartney, Ray Wylie Hubbard, Lucinda Williams, Steely Dan, Ben Folds, and Ryan Adams. The youngest of these started their career twenty years ago.

What’s up with that? Well, I simply don’t like live music. I’ve got reasons.

Performances don’t start on time

Having performed in jazz bands, orchestras, stand-up showcases, and improv shows, I’ve come to accept as axiom that performances just don’t start on time. There’s lots of good reasons for this. Everyone wants to get as many people into the seats as possible to make the show better, to improve the audience experience, or simply to make an extra buck.

The reasons that performances at live music venues don’t start on time come down to selfishness. The performers didn’t arrive on time, the stage wasn’t not have been set up in time. Worse, these have a knock-on effect. Once a performance is behind, it only gets more behind. There’s no shortening the break between bands or reducing the time between doors opening and the first band playing.

This leads me to the most inane reason live music is not on time: selling beer. “Doors open at 8 PM” almost universally means that you can walk in at 8 PM, but you can count on not seeing any live music until 9 PM. The opening act for the opening act is the selling of booze. I’ve got better ways to spend my time than standing around for an hour staking out a spot just so the venue can sell beer.

Standing for a couple hours sucks

Whether it’s standing in line for a roller coaster or waiting through beer-time, an opening act, and the changing of the stage, standing around is the worst. Fatigue and boredom set in; you’re taken out of the experience of enjoying music played in front of a lot of people. Sore feet and knees do not an enjoyable listening experience make.

Thank you, venues with seats, and thank you, crowds that don’t feel the need to show their enthusiasm by standing upright. You make live music a much more civil, enjoyable experience.

Crowds of people are the worst

Suppose you get a good room, with good sound, and a great performer. You’ve still got to tend to the other people in the room. The drunk heckler, the people calling out songs, the tall person blocking your view. That’s all after you stood in line to get in, waited to go to the bathroom, or put out of mind the guy who lit up next to you.

Opening acts

Opening acts. They’re a necessary but inconsistent evil. Sometimes you’ll see a really good one. One of the best bands I saw at a Dallas radio station “festival” was on the third stage. One of the worst bands I ever saw was an opener that was sufficiently uncertain of their own skills that the majority of the between-song banter was insults at the audience and counter-heckling gone bad.

I salute events that eschew the opening act and cut straight to the main performer. Give the people what they want.

It’s too loud

I don’t know why, but live music is universally an assault on my ear drums. I’ve been at concerts where I could feel the music moving inside my pants. That seems a bit excessive to me.

Beyond the personal discomfort, there’s nothing about loudness that makes music better. If everything is loud, nothing is loud. Sustained loudness is boring.

Short bouts of loudness; that’s interesting. The juxtaposition of the opening arpeggios of “Wouldn’t It Be Nice” with the wall-of-sound that follows is really nice. The way “Thunder Road” or Bolero grow into something loud and great is what makes them interesting. The amazing loudness of the opening of Also Sprach Zarahustra contrasted with the nearly non-existent quietness of the second movement is genius.

Don’t turn it up because you can. Turn it up because you mean it.

Drums are a lie

Let’s talk about the actual music again. In particular, drums. Drums, my friend, are a lie. They do not sound like you think they do. What you hear on the radio and on albums are the results of trained sound engineers using microphone and equalizer tricks to make drums sound decent.

This is problematic for two reasons. First off, drums are really loud in the hands of an enthusiastic player. Often quite a bit louder than your typical amplifier. Thus, it’s guaranteed you’re going to hear a lot more drums than guitars, horns, strings, or vocals at a live music event. I take that back; I guarantee you that you will not be able to hear strings at any music club you ever go to, but I’ll come back to that.

The more problematic aspect of a drum kit is you’re going to hear raw drums when you go to a live music event. Very little microphone tricks or equalizer cleverness; the drums may not even be isolated. That means the snare is going to sound like a can of beans getting hit with a stick. The toms will sound like someone banging on an empty box. The cymbals are going to sound like a mad person beating on pots and pans.

If you’re anything like me, you’re not going to enjoy them drums.

It sounds terrible

Drums are not the only problematic instrument. In my experience, most live clubs have very poor sound. Even if it’s not too loud, the mix is wrong, you can’t hear the melody, you can’t hear the singing, or the overall sound is distorted.

Assuming that clubs don’t exist merely to move booze (not a big leap in reasoning, I know), I don’t understand how this is the situation. If you want to be a part of a music scene, a good sound system and someone who can operate it seems like par for the course.

I am happy to note that, if you’re lame like me and only go to see performers who have been around the block dozens of times, you’re going to have a much better listening experience. Less prominent musicians are starting to tour with just one accompanying performer and that person is not playing a drum kit. The A-list performers have really good drummers (Paul McCartney’s drummer is a blast to watch) and the sound engineers on the tour are excellent. This makes for a far more enjoyable, balanced sound.

There’s little mystery

This one is rather personal, though I’ve spoken with musicians who feel the same way. If you know how to make music, watching the performance of music can be boring. A song that you can listen to and quickly pick up the structure and details of isn’t all that exciting. Even if it is, you can see the musicians enjoying the performance of the song and just wish you were up there playing and not down here watching.

I do enjoy watching very talented performers do their thing. Someone who mixes music with a good stage show or interesting banter between songs is fun to watch. The Rolling Stones are interesting to watch because Mick Jagger is such a good showman, Charlie Watts seems so apathetic, and Keith Richards is, well, Keef. I’ve really enjoyed seeing Hayes Carll and Lyle Lovett because the stories they tell are great and their banter between songs is amusing.

Genres I don’t know how to make are also fascinating. Hip-hop is not a thing I really know how to make, so that’s fun. Jazz and classical can be fun because I know how they work but didn’t reach the level where I could really make it. My new rule is, whenever Rite of Spring is performed, I need to be there; it’s relatively short (about forty minutes), really awesome, and I’m certain I would not be able to perform it with an orchestra without ruining it for everyone else.

Maybe I’m doing it wrong. Perhaps my heuristics for trying to time a concert so I arrive as the opening act is finishing require tweaking. I should definitely remember to bring earplugs more often. It’s entirely possible I’m just a grumpy guy.

But: I’m not the guy who tells you about the hippest new musical thing. I’m probably not the guy who’s going to catch your favorite band. I’m the guy who goes to see Paul McCartney out of reverence and because my wife and I both like him. I’m the guy who listens to an album as a long-form idea. I’m the guy who wants to understand the history and creation of a thing. That’s just the nerd I am: I understand music over time, not over the course of an evening.

Ed. this originally ran in The Internet Todo List for Enthusiastic Readers. You should check that out. It was pointed out that I’m a bit of an old man. In spirit, this is absolutely true. Also worth noting: I’m going to see Paul McCartney again this week, so I must not entirely hate live music. Human inconsistencies, eh?