Clips from unfinished pieces

On the crux of America’s challenges:

Part of the American experiment is answering the question, “how can we best take advantage of abundance?” Beginning with manifest destiny and evident in the machinations of Wall Street, one of the story lines of America is the quest to make sure resources of all kind are abundant and generating wealth. But we’re arguably at a pivot point. Our money and energy don’t go as far as they used to.

How do we make the transition from resource abundance to resource scarcity?

On helping people troubleshoot the Gowalla API:

While this level of self-documentation is quite helpful, sometimes people have questions on the developer list. For this, I’ve found that asking people to show me whatever it is they’re trying to do using curl is invaluable. It’s a win-win situation. Often, dropping down to a lower-level tool like curl helps to focus your thinking and makes silly error obvious. If it doesn’t become obvious to the API developer, they mail the list with the command they think should work. At that point its either obvious to me and I tell them what to change, or I have a nice, isolated test case from which I can easily try to reproduce their problem.

Who gets screwed when a borrower declares bankrupcty?

Is it possible that bankruptcy-declaring-borrowers are screwing lenders in aggregate? I find it really hard to believe that the banking industry, with its legion of lobbyists and regulatory capture, that any group of uncoordinated individuals could screw the banks.

On the other hand, there was lots of screwing on the part of the banks that led to the financial crisis. Whether it was predatory lending, relying on moral hazard to double down on terrible bets, or asinine compensation structures, the financial industry did something very human. They violated social norms. Except, corporations of this size don’t have social norms. They have only market incentives; when the executives, board members, and majority shareholders look at the books, the numbers devoted to “doing the right thing” are probably a rounding error.

On tail recursion and compilers:

Fact of life: modern processors don’t execute your code in the order the compiler spits it out.

If your code has, for instance, two adds followed by an if statement, it’s pretty likely that second add is going to be executed concurrently or after the conditional. In the world of computer architecture, they call this out-of-order execution, and it’s just another service your hard working processor offers to make sure your code runs faster than you ever intended it to.

On shorter cycles of production and the need to get past perfectionism:

Our modes of production are causing us to change how we produce. More and more mediums, be it journalism or software, are produced on shorter timelines. This is leading us to optimize production such that we can bang the content or code that matters into templates that mostly work, but have a tolerance for the rough edges where things don’t work.

On Barack Obama’s 2010 State of the Union speech that preceeded the health care debate:

Just for grins, I went and read the GOP response to the State of the Union. While they had some vague counterpoints policy-wise, it read mostly as subtle and useless jabs combined with carefully-constructed language to console their base. The GOP is a cynical, gutless organization.

On refactoring and deleting code:

People often say that they would miss having a refactoring browser in languages like Ruby, JavaScript, or anything that is reasonably dynamic. My glib response to this sort of comment is invariably “well, the best refactoring I know is to select the code to modify, hit delete, and start over.” Let’s take that apart.

I’ve observed that, despite our best intentions, we are often loathe to change code that we suspect is working, or that we suspect we don’t know why it’s there. And so, like the planet on which we live, applications accrete into Katamari balls of overly-coupled code that is bound only by locality. Cutting this Gordian knot is often the first step in reclaiming a project.

Deleting code is the knife with which we can attack this problem. Many will acknowledge the goodness of deleting code; it is, quite nearly, a virtue unto itself. I’ve observed that some of the best developers I know are always on the lookout for ways they can obviate code. So, by way of a strawman, I hope you see that I’m quite correct in this regard.

Michael Feathers on how code grows

Festering Code Bases and Budding Code Bases:

Some teams produce what I call a festering code base. In a festering code base, the team changes the code primarily by adding code to existing methods and adding methods to existing classes. The results are predictable. Classes and methods grow malignantly, eventually becoming thousands of lines long.

Better teams produce budding code bases. Developers create new classes and methods and delegate work outward. Periodically, they collapse structure back into a simpler form, but the dominant trend is to grow the code by creating new structure.

I’d never put much thought into how code bases grow in the past. Feathers has some interesting ideas here about the characteristics of good and not-so-good growth and how languages and tools might promote good growth.

A rambling, regurgitated thought on process

Elevator pitch: I’ve found that if you want to divert a productive team into an hour or two of semi-fruitless banter, ask how the team should use Git, Pivotal Tracker, and Capistrano to manage incoming work, verify it, and deploy it to production. In reality, you should ignore all the corner-cases and figure out what will enable you to push really small chunks of work with great frequency.

Ed. What follows isn’t novel, but it was a useful change in perspective for me, so I decided to share.

I’ve been thinking a bit about software processes lately. Despite great variation in telling you how to do so, most processes seem to focus on to do more stuff faster.

Lately, the notion of doing less has a lot of interest. Lean startups are the new-new thing and Getting Real is the old new thing; both preach getting more done and delivering value by doing less and analyzing the results more.

There are two kinds of “do less” a software developer can engage in. In the past I’ve been a little too focused on how I can take on fewer responsibilities from other parties. Literally doing less by scoping down features, putting off decisions, and focusing on things that seem like they really matter. I sometimes feel like I’ve become too eager to do less, making myself something of a cranky coder/slacker. But I digress

Recently, I’ve been trying to tackle doing less in my habits of creating software. How can I write less code to implement a feature, not in the minimalist sense, but in the “how do I just get it to kinda work sense”? How can I take less time between starting something and getting some form of it out in the wild? How can I make my code less coupled so there are fewer changes to make when I decide it needs to do something else? How can I make this less coupled to data storage so that putting it out requires less deployment effort? How can I make changes that are less likely to cause long-term regressions? How can I make it less effort to rollback bad changes?

When I look through the lens of accomplishing more by doing less, a lot of popular software methodology seems like dead weight. Rather than trying to find a process that addresses every team member’s own scars and affections, both perceived and imaginary, it seems most useful to imagine the smallest ruleset that won’t result in uncontrollable entropy and put it into action. If something starts to hurt, imagine the simplest new rule and put it into play.

The goal, as stated above, is to get to the point where you make really tiny, maybe imperceptible changes, and push them really frequently. Everything that stands in the way is the enemy.

Who are we that make software?

We who spend all of our time in front of a computer involved in the production of software are often quick to pigeon-hole ourselves. You probably self-classify as a developer or designer, maybe an engineer or artist if you got a college degree and think highly of it.

But like many other things, it’s all messy now. I’d say I spend sixty percent of my time doing general “developer” stuff, twenty-five percent doing something one could approximately call “engineering”, and split the rest between marketing, business, and design.

Does self-identifying with any one of these roles limit how we think or approach doing what we do?

Sloppy classifications

Consider these heuristics for placing people into categories:

You build things that face other people
You are making things that are constrained by rulesets defined largely by Newtonian mechanics
You are making things where trade-offs between aesthetics and affordances are made
Other people build things on top of your things

None of these are useful at all. Were you to provide any one as a definition of what an engineer or designer is, you could probably get some heads nodding. So there’s something appealing about each of these statements. But none of them provide a pleasing definition or guideline for when you’re doing engineering, development, or design.

Part of the answer to these classifications is that we all do everything. Developers strive to build software that fits within the aesthetic of the code around it or their own personal aesthetic. Designers operate within the limitations of human perception and cognition. Engineers are constrained by both of these but will throw either out in a heart beat to improve upon the efficiencies that are important to the project at hand.

We’re all hybrids

The notion of developing designers and designing developers is by no means new. A few examples:

But consider Kent Beck, renowned for his work building and thinking about the process of building software. He often talks about the design of software, considering trade-offs, aesthetics, and affordances just like a designer does. But he’s also been spending a lot of time recently iterating on businesses, trying out new ideas, and writing about the process and essence of converting an idea into a sustainable business.

Or consider Shaun Inman. He’s writing games as a one-man show. He splits his time between producing the music, drawing pixel art, and coding up collision detection systems. That’s a pretty neat cocktail of talents.

If you’ve ever bikeshedded a design discussion or suggested how a feature might work, you’re a hybrid. Ever refer to yourself as a specializing generalist? That’s a hybrid.

Directed thought

If you’ve self-classified one way or the other, there are little things you might do that have large effects on your thinking. You socialize with those who are like classified, use the tools of that classification, and concern yourself with the classic problems that consume those working in your area of specialization.

If you’re not careful, you could box yourself in too much, become too specialized. While there are opportunities for well-chosen, tightly-focused specializations, they are few and far between. Specializing generalists are the order of the day.

Where do we get if we acknowledge that we’re all hybrids now? Suppose you’re aiming for a balance of sixty percent developer, twenty percent engineer, and twenty percent designer. Is it worth going whole-hog learning Emacs or Photoshop? Or is it better to learn less-capable but lower learning-curve tools like TextMate and Acorn? Should such a person concern themselves with the details of brand design and the implementation of persistent data structures, or is it more important to grasp those topics in a conversational manner?

Is it a better use of Shaun Inman’s time to dissect a Mahler symphony, do an expansive study of pixel art, or review the mechanisms Quake III used for detecting collisions? Is it a better use of Kent Beck’s time to build software and write about that process, to talk to people and integrate their problems into his way of developing software, or iterate on business ideas and share those experiences?

Here’s the motivational part

So now that all of this is forehead-smackingly clear (right?!), where do we go from here? Personally, I’m using the idea to guide how much effort I put into teaching myself new tricks. I probably won’t go on a six-month algorithms kick anytime soon, but I might spend six months learning the pros and cons of various database systems or application frameworks. I’d love to spend a month just tinkering with typography, color, layout, and other visual design topics. I probably won’t sweat it if Emacs or Photoshop don’t integrate into my daily work too well, or prove impenetrable to my mind, since those tools imply workflows that aren’t top priority to me.

But that’s me; where should you go? If you don’t already have a good idea of what kind of hybrid you are, start noting how much time you spend on various sorts of tasks and think about whether you’d like to do more or less of them. Then, start taking action to realize a course correction.

You can be whatever kind of hybrid developer you want, it’s just a matter of putting in the time and effort.

Code re-use as technical debt

I have extremely mixed feelings about code re-use. I think it’s largely a red herring, never working out as well as developers would hope. After all, developers are like golfers; always optimistic about how well an approach will
work or how far down the fairway their ball landed.

But here’s a real stab in the side of code re-use: in many cases, it’s tantamount to technical debt. Embrace technical debt:

For example, early on at IMVU, we incorporated in tons of open source projects. This was a huge win (and we were delighted to give credit where it was due), because it allowed our initial products to get to market much faster. The downside was that we had to combine dozens of projects whose internal architectures, coding styles, and general quality varied widely. It took us a long time to pay off all the debt that incurred – but it was worth it.

Using someone else’s code will help you keep moving now, but you stand a good chance of needing to rewrite it later.

That’s not to say it’s all bad. By the time you know you need to replace someone else’s code, you’ll have learned about the domain it covers and how you need to solve that problem in your domain.

Keep it in mind: just because you can drop someone else’s code into your app and use it, doesn’t mean it’s all roses and butterscotch.

Interviewing to seek values

Adam Wiggins, per usual, is on to something. Values:

Sharing values is the most important part of effective collaboration. If you don’t have significant overlap on values between you and your teammates, you’re going to have a tough time getting anything accomplished.

I’m starting to think that figuring what the other person puts a premium on is the most important part of a technical interview. Is the other person passionate in the same way you are? Are the things they obsess over complimentary to what you would rather gloss over? If the answer to these questions is yes, you’ll probably make awesome things together.

Software development requires empathy

If You Want to Write Useful Software, You Have to Do Tech Support:

It seems so obvious: if you want to develop software that’s useful to people, you’ve got to talk with them. But too many developers take the anti-social approach and consider customer support to be beneath their status. Besides, talking with customers would distract them from important code-slinging.

I have to remind myself, almost every day, that one of the the most important qualities I can possess as a developer is empathy. Primarily for the user, their cognitive load, and what they’re trying to accomplish. But further, for the developer who comes to my code when I’m done, the guy who operates it, and everyone else down the line.

When to do test-driven development

I believe that writing code using testing[1] as a design activity yields long-term benefits that make my life easier.

Though I’m a strong believer, I’ve struggled with TDD in the past. I’ve found I get bogged down in keeping the red-green-refactor cycle going. Sometimes I have to work with code that is lacking sufficient tests, but I know I can’t boil the ocean before I proceed to whatever I’m _really_ trying to do with the code. Other times, I’m not sure if I’m testing the right things; I could be missing tests in one place and writing too many tests in another.

Three easy pieces

Last week, I read three insightful pieces and made one discovery of my own that deepened my understanding of the practice of TDD. Allow me to share.

First off, Kent Beck has been exploring the phases in the life of a startup. The earliest stage is proving the idea. He later asserted that when you’re doing stuff like this, you can drop TDD, temporarily. I thought about this and it clicked. If you’re working on a prototype, where you’re trying to explore an idea and see if it works, you don’t want to iterate on the code, as TDD would have you do. You want to iterate on the _idea_. TDD will just slow you down.

On the other end of the spectrum, you’ve got maintenance programming. Once a startup, company or project has proven their idea and shipped a system, you are maintaining software. Keeping it working, living, breathing. For this sort of development, where you’re making small, focused changes without adding significant functionality, Tim Bray pointed out that TDD is critical. Using it to drive the process of fixing bugs, cleaning up the system and adding minor functionality is really handy for figuring out if you’ve broken something in some dark corner. It also helps the next guy to do the same. It’s a win, and I suspect it’s the sweet-spot of TDD.

If you imagine prototyping and maintenance as opposite ends of the software life-cycle spectrum, adding significant new features to existing software probably lies somewhere in the middle. You may need to “poke around” to decide if what you’re doing is right, like when you’re prototyping. But once you’re done, you want some way for others who have to maintain the software (such as yourself) to figure out what it’s supposed to do and whether assumptions have been broken.

Then I read an anecdote by Uncle Bob about how he worked out an ambitious new feature in Fitnesse. He and his pair got a new feature working, celebrated, and called it a night. When he woke up, he realized they weren’t actually done; they still had to clean it up, by writing tests.

Oh. Duh.

This was the missing link, for me. Sometimes, you need to iterate on the idea first. If skipping the tests helps you, so it goes. Once you’ve got the idea working (and committed!), then start writing tests[2]. Once you’re happy with the structure and coverage of your code, you commit again and _then_ push it to your peers[3].

The crux of my revelation is this: you get the benefits of TDD-as-a-design-activity by doing it. _When_ you do it is immaterial. You just *have* to do it.

My own revelation is blindingly obvious in retrospect. If the going gets tough, proceed in this order: get it to work, write some tests for it, _then_ clean up the code. Sometimes you can break this cycle if what you’re working on doesn’t take too much cognitive capacity. But if you overflow your mental buffer, you have to break it down into steps and work through the cycle. Failing to realize this was one of the causes of me bogging down in TDD.

Context is everything. Always.

Adding context to answer the question of when you start writing tests is something I haven’t found much writing on until recently. I’m increasingly finding that considering the situation is a great ninja-move in my quest towards writing beautiful, useful code.

fn1. Call it a test, example, behavior, or story. Whatever.

fn2. Jim Weirich did a great presentation on how to backfill tests on existing code.

fn3. Pardon the Git-centric terminology[4].

fn4. If you are not already, please start using Git immediately.

Meaningful work

Going to meaningful work:

Just like being awake is more than just having your eyes open, going to work should be more than just being at a workplace trading time for money. It should be meaningful. But where does meaning come from? Of course, it comes from ourselves. We put meaning into things, and share our meanings with others, and teach each other how to build meaning out of what is in front of us.

Buster’s on to something here. He’s articulating one of the qualities I find in the best developers: what they do has meaning and matters to their personality. They are working to make things that result in greater happiness for themselves and others. Their passion is manifest in the quality of their code.

You should also check out Buster’s personal site. He’s got a neat info-graphics, personal data-mining thing going on there.

Breaking with tradition

For the like-minded aficionados of the non-traditional: A Redis implementation of Twitter, designed for learning about non-relational datastores and Clojure’s creator Rich Hickey on state (bonus track: Jonas Boner on the same). Derek Sivers on the fatal determinism of declaring a goal or as I like to say, “the reason you find so many weblogs with one post promising to post a lot”. Michael Heilemann on the lack of good fiction in contemporary games.