The joy of logs

Logs Are Streams, Not Files:

But a better conceptual model is to treat logs as time-ordered streams: there is no beginning or end, but rather an ongoing, collated collection of events which we may wish to view in realtime as they happen (e.g. via tail -f or heroku logs --tail) or which we may wish to search in some time window (e.g. via grep or Splunk).
Work on an app with a couple dozen servers, a handful of databases, and several moving parts and you start to realize that logs are one of your best friends. They're useful for troubleshooting, performance monitoring, and just knowing how your application works in reality, under real traffic.

I’ve tinkered with building deeper APIs for logging within applications and services, but I think Adam Wiggins is on the right path here (not the first time either). Logging should be as simple as possible in applications. All the smarts for aggregating, searching, and extracting interesting information should happen after the data is collected. Using standard out instead of files is a fantastic idea too.

I'm Corgi-internet famous

Me + Kitty

OCD: Obsessive Corgi Disorder, Kitty putting the moves on my man. Also on Men and their dogs. I’m corgi-internet famous! Photo by Courtney.

Noel Rappin's Advice on TDD

Testing Advice in Eleven Steps. My favorite:

At any given moment, the next test has some chance of costing you time in the short term. The problem is it’s nearly impossible to tell which tests will cost the time. Play the odds, write the test. Over the long haul, the chance that the tests are really the bottleneck are, in my experience, quite small.

All eleven are pretty handy for those like myself who still feel like they have a lot to learn about TDD, BDD, et. al.

Organizing and decoding problems

My favorite sort of problem involves the interactions between individuals, groups of people, and mechanical rules generated by individuals and groups. Speculating on how the rules were shaped by the experiences of the individuals and groups is a fun game to play when presented with a curious set of circumstances. Conversely, my least favorite problems are those generated by the institutional brain damage of certain kinds of groups. Software and systems shaped by regulations and the particulars of monetary exchange are tedious at best.

I find it quite amusing when answers aren’t “clean” and technological systems aren’t the best solution. Often, no amount of analysis and brilliant coding can make an improvement. What is needed is to understand what people do, why they do it, and persuade them to do otherwise.

This brings me to economics, specifically behavioral economics. If you take away all the math, economics is largely about how people behave in aggregate. It’s a useful tool in understanding how to work with systems that involve people. But there’s more to economics than explaining how people interact in markets.

Russ Roberts, in the process of explaining how economics is not useful as a mechanism for answering questions like “did the stimulus work?” or “when will the housing market recover?”, gets to what really interests me about economics and finance:

"Is economics a science because it is like Darwinian biology? Darwinian biology is very different from the physical sciences. Like economics it is a very useful way to organize your thinking about complex phenomena. But it is not a predictive or very precise science or whatever you want to call it."

The crux of the biscuit, for me, is right in the middle. Economics is a useful way to organize an intricate and interconnected problem and figure out how to reason about it. In casually studying economics and finance over the couple years, I’ve come to form a better mental model about how the world works.

Its possible that this mental modeling is what I do when I’m coding. I’m poking how some code works, looking at its inner workings, trying to understand how it interacts with the code around it. I sift through revision logs to see how it got to where it is today, talk to others on the team about the code and why it ended up one way and not another. I’m organizing and modeling the code in my head, building up a model that describes .

Lately, I’ve been describing my work as tinkering with lines of code until numbers appear on the screen in the right order. This is invariably greeted with a comforting-to-me response like “I could never do that”. But I really enjoy this. I’m not just debugging my code; I’m sharpening the way the program is organized in my head. Once I’ve got my head around it, I decode that organization into words, code, drawings, etc.

With programs, there is some system of rules, forces, and interactions which describe how the code works or doesn’t. Economics, too, describes a system of rules, forces, and interactions that predicts how a puzzle of human beings operate. Organizing and decoding these technical and social puzzles is great fun.

Workouts make focus and discipline

It is probably obvious I have something of a crush on working out. I am proud I have reached a point where being fit is part of my lifestyle and that I see more and more results. But my crush with exercise goes beyond what I see in the mirror.

As important as the physical aspects are, I get the most out of the mental aspects. Working out mirrors what I do in coding in a couple ways. Both require staying on my grind almost every day. Both require the focus to push myself and resist complacency.

It’s easy to spend too much time tinkering with social media or pick the lighter weights. It’s harder to ignore distractions and focus on the task in front of me, whether it’s code or heaving kettlebells.

When I do manage to focus I benefit twice. Immediately I feel great because I challenged myself and moved my work or workout forward. Later, I realize I’m getting better at putting the work in every day and tuning out that which is not important to what I’m trying to do.

The TV infomercial benefits of working out, ripped extremities and a strong core, are a tertiary benefit to developers and creative types. Even the decrease in neck fat, the ability to lift heavy things, and reduced tendency to get winded are auxillary. The real benefit that people who make things get from working out are increased focus and discipline.

Driven to drawing monsters

From my notebook:

Monster
I don't recall what I was working on the time, but it seems that the interaction between many Gowalla models loading from cache via a get class method and something about memcached was causing enough trouble to drive me to draw a little monster.

Clips from unfinished pieces

On the crux of America's challenges:

Part of the American experiment is answering the question, "how can we best take advantage of abundance?" Beginning with manifest destiny and evident in the machinations of Wall Street, one of the story lines of America is the quest to make sure resources of all kind are abundant and generating wealth. But we're arguably at a pivot point. Our money and energy don't go as far as they used to.

How do we make the transition from resource abundance to resource scarcity?

On helping people troubleshoot the Gowalla API:

While this level of self-documentation is quite helpful, sometimes people have questions on the developer list. For this, I've found that asking people to show me whatever it is they're trying to do using curl is invaluable. It's a win-win situation. Often, dropping down to a lower-level tool like curl helps to focus your thinking and makes silly error obvious. If it doesn't become obvious to the API developer, they mail the list with the command they think should work. At that point its either obvious to me and I tell them what to change, or I have a nice, isolated test case from which I can easily try to reproduce their problem.

Who gets screwed when a borrower declares bankrupcty?

Is it possible that bankruptcy-declaring-borrowers are screwing lenders in aggregate? I find it really hard to believe that the banking industry, with its legion of lobbyists and regulatory capture, that any group of uncoordinated individuals could screw the banks.

On the other hand, there was lots of screwing on the part of the banks that led to the financial crisis. Whether it was predatory lending, relying on moral hazard to double down on terrible bets, or asinine compensation structures, the financial industry did something very human. They violated social norms. Except, corporations of this size don't have social norms. They have only market incentives; when the executives, board members, and majority shareholders look at the books, the numbers devoted to "doing the right thing" are probably a rounding error.

On tail recursion and compilers:

Fact of life: modern processors don't execute your code in the order the compiler spits it out.

If your code has, for instance, two adds followed by an if statement, it's pretty likely that second add is going to be executed concurrently or after the conditional. In the world of computer architecture, they call this out-of-order execution, and it's just another service your hard working processor offers to make sure your code runs faster than you ever intended it to.

On shorter cycles of production and the need to get past perfectionism:

Our modes of production are causing us to change how we produce. More and more mediums, be it journalism or software, are produced on shorter timelines. This is leading us to optimize production such that we can bang the content or code that matters into templates that mostly work, but have a tolerance for the rough edges where things don't work.

On Barack Obama's 2010 State of the Union speech that preceeded the health care debate:

Just for grins, I went and read the GOP response to the State of the Union. While they had some vague counterpoints policy-wise, it read mostly as subtle and useless jabs combined with carefully-constructed language to console their base. The GOP is a cynical, gutless organization.

On refactoring and deleting code:

People often say that they would miss having a refactoring browser in languages like Ruby, JavaScript, or anything that is reasonably dynamic. My glib response to this sort of comment is invariably "well, the best refactoring I know is to select the code to modify, hit delete, and start over." Let's take that apart.

I've observed that, despite our best intentions, we are often loathe to change code that we suspect is working, or that we suspect we don't know why it's there. And so, like the planet on which we live, applications accrete into Katamari balls of overly-coupled code that is bound only by locality. Cutting this Gordian knot is often the first step in reclaiming a project.

Deleting code is the knife with which we can attack this problem. Many will acknowledge the goodness of deleting code; it is, quite nearly, a virtue unto itself. I've observed that some of the best developers I know are always on the lookout for ways they can obviate code. So, by way of a strawman, I hope you see that I'm quite correct in this regard.

Focus, momentum, and accomplishment

Lately, I’m a little fascinated by the interplay between focus, momentum, and accomplishment. Focus is the feeling of flow, the world around you fading away. Momentum is the feeling that you’re moving things along, getting stuff done. Accomplishment is stepping back, looking at what you’ve done, and feeling like you did something good. These factors play out differently in the three activities to which I decide much of my time.

When I’m building software, I’ve found that momentum leads to focus, accomplishments yield momentum, but focus does not necessarily yield accomplishments. Yet, the most exciting moments seem to come when I’m focused and building momentum. Those times when my attention is fragmented but I’m working through a list of stuff that needs to get done doesn’t feel like accomplishment, but it does seem to get the job done.

Anecodotally, it seems that, by definition, Monday is the day of the week that lacks focus but still possesses some form of accomplishment. It’s the sacrificial lamb of the work week.

When I’m reading, writing, and organizing information, I’ve found that momentum is not much of a factor. It’s all about focus. Stick my head into a cluster of ideas, focus on what I’m reading, how I’m thinking about it, or how I’m writing it back out. Keep the problem state in my head, admitting as few distractions as possible. Come up for air periodically, review my progress, and decide to either continue or move on to other things.

Of these three activities, I’ve found that I’m most methodical about how I use momentum when I’m thinking. It’s sort of a rate limiter; after I’ve read something particularly dense, I’ll immediately do something banal like catch up on social medias. Not sure if this tendency is enhancing or diminishing my thinking.

I spend several hours a week training. The coffee shop (thinking) and the gym (training) are my third place(s). Recently, it feels like I’ve leveled up. I had a few consecutive workouts where I was more focused than I’ve ever been whilst exercising; no worries about my breathing, or if I was pushing myself too little or too much. From this focus I found momentum. A couple minutes on the treadmill would turn into tens of minutes, then a half hour. One set of strength exercises would blend into the next. I was able to push myself to higher levels of accomplishments. It felt awesome.

Interestingly enough, I’ve found that I need a day off from each of these activities. As awesome as it sounds to have the kind of intensity that lets one make awesome things every day of the week, I’ve found it necessary to relax and take time away from coding, thinking, and training. As it turns out relaxation is an important part of a balanced diet of focus, momentum, and accomplishment.

A conversation between fictional engineers in a fictional world

A hypothetical conversation that may have occurred between two non-existent engineers working on the second Death Star in the completely fictional Star Wars universe.

Engineer #1: Hey Bob, I was perusing the blueprints for this “Second Deathstar” this morning. Pretty impressive stuff.

Engineer #2: Thanks Hank. I’m pretty proud of it.

Engineer #1: And you should be! Had one question though. There was something in the request-for-proposals that mentioned some flaw in the previous one where a snub fighter could drop a torpedo through a vent and blow the whole thing up, yeah?

Engineer #2: Yep! Don’t you feel bad for the poor schmuck who made that decision?

Engineer #1: Haha, that’s a good one Bob. So you fixed that right?

Engineer #2: Oh, definitely. All the exhaust ports are small enough the only thing falling in there is a grain of sand.

Engineer #1: Nice thinking! So, my real question is, what’s this giant opening you can fly a large freighter through? And why does it lead right to the station’s giant fusion reactor that sits in a room big enough to fly in circles in said large freighter?

Engineer #2: Oh, that? Well, the passage from that room to the surface is where I’m going to run all the pipes and wiring that I forget about until the last second. I figure once I’m done patching everything together, no pilot would be able to fly through there, even in a snub fighter.

Engineer #1: And the giant room?

Engineer #2: Oh, you know clients. Always deciding they want something really impressive at the last minute. I figured I’d just leave a little extra room in case they come up with something at the last minute.

Engineer #1: Haha, right again Bob. Clients are such idiots.

Simple Ruby pleasures

I think I first discovered the joy of take and drop in my journeys through Haskell. But it appears that, since 2008 at least, we have had the pleasure of using them in Ruby too.

Need the first or last N elements from an Enumerable. Easy!

[sourcecode language=“ruby” light=“true”] ary = (1..100).to_a ary.take(5) # => [1, 2, 3, 4, 5] ary.drop(95) # => [96, 97, 98, 99, 100]

range = (1..100) range.take(5) # => [1, 2, 3, 4, 5] range.drop(95) # => [96, 97, 98, 99, 100]

hsh = {:foo => 1, :bar => 2, :baz => 3} hsh.take(1) # => [[:bar, 2]] hsh.drop(2) # => [[:foo, 1]] [/sourcecode]

The real magic is when you use take along with other Enumerable goodies like select and map. Here’s one of my personal favorites amongst the code I wrote in 2010:

[sourcecode language=“ruby” light=“true” highlight=“12,13,14”] class QueryTracer < ActiveSupport::LogSubscriber

ACCEPT = %r{^(app|config|lib)}.freeze FRAMES = 5 THRESHOLD = 300 # In ms

def sql(event) return unless event.duration > THRESHOLD callers = Rails. backtrace_cleaner. clean(caller). select { |f| f =~ ACCEPT }. take(FRAMES). map { |f| f.split(":").take(2).join(":") }. join(" | ")

# Shamelessly stolen from ActiveRecord::LogSubscriber
warning = color(&quot;SLOW QUERY&quot;, RED, true)
name = '%s (%.1fms)' % [event.payload[:name], event.duration]
sql  = event.payload[:sql].squeeze(' ')

warn &quot;  #{warning}&quot;
warn &quot;    #{name} #{sql}&quot;
warn &quot;    Trace: #{callers}&quot;

end

end

QueryTracer.attach_to :active_record [/sourcecode]

This little ditty is awesome because:

  • It's super-practical. Drop this in your Rails 3 app, tail your production log, see the slow queries, go to the method in your app calling it, and fix it. Easy.
  • It only activates itself when it's needed. Queries that execute quickly return immediately.
  • No framework spelunking required. Rails 3's notification system handles all of it. Rails' backtace cleaner gizmo even makes the backtraces much nicer to read.
  • It chains methods to make something that reads like a nice, concise functional program.
For more Enumerable joy, read up on each_cons.

The political empathy gap

The structural problems in American political discourse are legion. Polarized interests, corporate and special interests, the news/hype cycle, and pundits who serve no real political purpose but exist only as media entities. All of it conspires to misdirect discussions of where our country is, where it needs to go, and how to get there.

Underlying all that is a serious problem. There are two sides to every issue (or so we’re told), and they cannot fathom each other’s position. Empathy is not a characteristic of American politics.

This manifests itself everywhere. Dismissive rhetoric, talking past each other, insulting jabs, moral grandstanding, dehumanizing the other side, binary reasoning, and violent propaganda. Were it the case that conservatives and progressives could understand each others goals, fears, and dreams, these problems would exist on the fringe rather than the mainstream.

Intellectual empathy is a hard thing to come by. We all want to be winners, firmly on the side of the virtuous good. To accept that there is something valid in one’s debate opponent is challenging. To go further and accept ambiguity and uncertainty is even more difficult. It would seem that a majority would prefer comfort in wrongness rather than face up to a world where their side is not that of the virtuous good.

I’m not sure how one learns these things other than seeking out new ideas and opinions. Even then, there’s the intuition to sort the wheat from the chaff, the practice from the principle, the solid thinkers from the eccentrics. Political empathy is perhaps (but hopefully not) the endpoint on an intellectual journey that a pop culture is ill suited to embark upon.

I can’t say that my grasp on the problem is good enough to suggest solutions. Perhaps humor, perhaps better education, perhaps breaking bread, perhaps more journalists growing a spine. I should hope it happens soon, because I’m getting pretty tired of how cynical I am of what politics has become.

Now witness the power of this fully functional domicile!

About a year ago, our garage door was not feeling well. It wouldn’t go down on it’s own. To close it required my loving embrace. Specifically, I had to lean against it as the door goes down the tracks. I’m sure I looked like the “mystic” trope in the “action” movie who, at a crucial moment in the action, resurrects the hero so he can go whoop the bad guy.

This is the part about home ownership that sucks the most. I had a strong suspicion this would work out one of two ways. The repair person will come out, look at it for two minutes, bang on something with a hammer a few times and everything works. Easy, cheap. Or, the repair person will look at it, tinker with it, and pronounce the motor dead, to the tune of a few hundred dollars. It ended up being somewhere in between; I think some kind of specialized garage door grease was involved.

I avoid these things because, through hundreds of logical and unintended decisions, I have chosen the life of an extremely specialized knowledge worker. I enjoy making sense of complicated systems that involve people, computers, and uncertainty. I am nearly useless with everything else.

Luckily, ours is an age where that specialization works out pretty well. I am fortunate that I can pay others to worry about typical “male” tasks like changing the oil on our cars or painting the bedroom (cue applause from the economists in the audience). That said, sometimes the interactions with those folks feels a little odd. I usually have to prefix them with the disclaimer that I am “the least handy dude on the planet”; it’s important to set expectations.

Another thing I’ve learned about home ownership is that repairing my garage door will almost certainly be worth every penny I pay for it. I know this because some time ago, we got around to having someone fix our toilet. It would run for quite some time after you flush it. At least thirty minutes. I let this linger because it was easier to not think about spending money on it that it was to do something about it. But, for not too much money, a professional (read: not me) fixed everything. The toilet flushed and stopped running mere moments later.

The psychological release this brought me was incredible. I found myself noticeably happier that our house was less broken. It was like I got a new toy. And exciting, rejuvenated, toilet toy!

…and that’s the worst part about home ownership. That you can sit down and write more than five hundred words on the joy of home repairs. I’ve been working with some folks that are several years younger than I am. These events are interesting to me, but I know that if I were to breach the topic with them, it would confirm to them that I am indeed the lamest trigenarian they’ve ever met. I know this because in college, I worked with older dudes and the summer I spent listening to them talking about the pH balance of their pools was notably tedious.

So here we are, five hundred words later, telling the whole internet I’m the lamest trigenarian they may ever meet. The good news is, everything in my house works. We got it inspected, and we know this with certainty. I highly recommend it if you have the means.

The ear is connected to the brain

Some measure their work in tomatoes. I measure mine in albums and songs.

When it comes time to get stuff done, I match my ambitions to my energy level. Big, uncertain things when I’m full of spark and optimism. Tiny, predictable things when I’m sapped and ready to check out. The coherence, duration, and novelty of what I’m listening to takes me in a different creative directions. I use Pandora, my own iTunes library, and Rdio to match my creative mood to a musical mood.

Sometimes, I don’t know where I’m going. I may not even know where I am. At best, I know how I feel. This is when Pandora shines. If I feel funky, I go with a Billy Preston or Meters mix. If I’m feeling peppy, I go with RJD2 or Steely Dan. If I’m feeling brooding, it’s DJ Shadow or Explosions in the Sky. Pandora finds my way musically, I find my way creatively. Teamwork.

I’m an album guy. I like to queue up a coherent piece of music and listen to it all the way through, in order. A good album rewards this. It starts with a bang, proceeds through a middle section where things might get slow, take on minor key, or both. Things clear up, maybe with a light and organized middle piece. The finale brings it all together and ends with something that sticks in my head.

Ideally, I’d create this way too. Start at the beginning. Find the crux of the problem space, explore some solutions. Work to some kind of apex where I’ve got all the problems solved and all my ducks in a row. Clean up the rough edges, tidy everything up, and ship it as the finale.

Duration is important. A short album (Treats by Sleigh Bells, 32 min.; Beethoven’s 5th Symphony, 31 min.) is great for tackling a compact but rewarding task in a small timeframe. A medium length album (The Suburbs by Arcade Fire, 60 min.; Game Theory by The Roots, 51 min.) is good when I’m starting to feel my groove. When I’m feeling more ambitious, a double album is in order (The Beatles White Album, 96 min., Mahler’s 5th Symphony, 72 min.).

My wildcard is Rdio. Rdio is a blessing for me as a music service. It is very much based on systemic album listening, as opposed to Pandora’s serendipitous song discovery. I turn to Rdio when I know I want to focus, but feel uncertain about where my creative travels will take me. Rdio’s full of music I think I might like, but I’m not yet sure if I really want to make it part of my collection. When I’m listening to Rdio, I’m tinkering with ideas, seeing if I want to own them now or stow them away for later.

Music is important to my craft, even though it’s not a touch point. I bought almost twice as much music in 2010 as I did in 2009. I started paying for both Rdio and Pandora. Text editors (I spent significant time in three of them last year), notebooks (I finished two and started a third), and various arcane tools are important to my work making code, organizing ideas, and shipping useful software. Getting my music right is perhaps just as crucial.