My magic ball predicts that OO and FP are going to take something of a “romantic comedy” path of evolution.

Act I. OO and FP are introduced at dinner parties and they could not seem more dissimilar and hilarious arguments ensue. No one goes home together. Despite the initial miss, the end of Act I finds OO and FP separately talking to friends about how they want the same things.

Act II. OO and FP run into each other at the coffee shop, and then again at the gym. OO is reading a book on ideas that FP loves. One of their friends invites them both to a bar, they get a little sauced and end up making out a bit. OO starts wearing FP’s jacket around town, even finding it a little comfortable. Towards the end of Act II, OO and FP are a bonfide thing, both borrowing ideas from each other. It’s pretty cute.

Act III. Open with a fight between OO and FP. It seems they just can’t come to agree on some important topic like mutability or the nature of behavior and state. Unfortunate and emotional words are uttered. The internet is abuzz with talk of the drama. They go back to their respective friends and rant about the shortcomings of the other. But, late at night, OO finds that not having FP around is less awesome than having FP around. OO cooks up a cooky plan to get FP back into their life. Hilarity, and a little awkwardness ensue. In the end, FP and OO go great together and we end with a montage of “everyone lived happily after” and see a clip that alludes to an OO/FP baby on the way.

If you’re playing at home, we’re already in Act II. Ruby and Python borrow various ideas on iteration from FP languages. We might be towards the end of Act II; Scala is very much wearing ML’s jacket around town. Surely there will be fallout at some point, someone ranting about how OO FP hybrids are too large, too poorly designed, too complicated, etc. The dust will settle, and someone will build an even better OO FP hybrid. Act III will play repeatedly until no one thinks of languages as OO FP hybrids, they just think of them as another language.

Then something different from OO or FP will become obviously useful and this whole romantic comedy will play again. It’s the way of Hollywood, and the way of software development. Everything old is new again; everything new is old again. Rinse, repeat.

I’ve noticed some of the sharpest developers I know are doing one or both of these things:

• Rediscovering object oriented design. Practicing evolving a design, often driven by the pain points illuminated by automated tests. Thinking about coupling and cohesion. Trying to encapsulate the right behaviors and find decide which principles are the most appropriate to the languages and systems they’re using.

• Rediscovering functions. Applying functional programming to everyday programming problems. Using the features of functional languages as an advantage to build concurrent and distributed systems. Finding the differences in functional design and writing more idiomatic code.

The first is a cyclical thing. It happened in Java, it happened in .NET, it’s happening in Ruby now. People come to a language for what makes it different, write a lot of stuff, and keep bumping into the same problems. They (re-)discover OO, start refactoring things and shaping their systems differently. Some people dig it, others dismiss it as too much effort or ceremony. A new thing comes along, and it all happens again.

The second is harder for me to read. I’ve spent a fair amount of time studying FP, though I have yet to apply it to production software. Despite that, I have come across a lot of good ideas that are already part of the code I work with daily, or that I wish was part of the code I work with. FP has good answers to composing systems, reasoning about state, and handling concurrency. It has often suffered from a lack of pragmatism, overly dense literature, and rough tooling. The ideas are worth stealing, even if they haven’t broadly succeeded.

Both of these trends are crucial to moving the practice of software development forward. We need to keep rediscovering and sharpening old ideas whilst experimenting with new ideas to find out which ones are good and which ones less so.

Little-d distributed systems: the accidental sort. You built a program, it ran on one server. Then you added a database, some caches, perhaps a job worker somewhere. Whoops, you made a distributed system! Almost everything works this way now.

Big-D distributed systems: you read the Dynamo paper, maybe some Lamport papers too, and you set out to build on the principles set forth by those who have researched the topic. This is mostly open source distributed databases, but other systems surely fall under this category.

Ph.D distributed systems: you went to a top CS school, you ended up working with a distributed systems professor, and you wrote a system. You then graduated, ended up at Google, Facebook, Amazon, etc. and ended up writing more distributed systems, on a team of even more Ph.D’s.

If you’re building a little-d distributed system, study the patterns in the Big-D distributed systems. If you’re building a Big-D distributed, study what the Ph. D guys are writing. If you’re a Ph. D distributed system guy, please, write in clear and concise language! No one knows or cares what all the little greek symbols are, they just want to know what works, what doesn’t work, and why.

You need to protect a piece of data, like a counter or an output stream, from getting garbled by multiple threads.

Three choices, hot shot:

• Explicit locks (aka mutexes): acquire a lock around the “critical section”, munge the data, release the lock. You have to manage the lock yourself. Multiple threads accessing the lock will not run concurrently anymore.
• Implicit locks (aka monitors): annotate methods that modify important data. The monitor library manages the lock for you. Threads still serialize around the lock, reducing concurrency.
• Atomic objects (aka compare-and-swap): use data structures that take advantage of runtime or processor semantics to guarantee that competing threads never interfere with each other. No locks! Much less serializing! Not broadly applicable, but I highly recommend them when you have the means.

Mutexes, aka lock “classic”

Mutexes are the lowest level of locks, at least in Ruby. They are the ur-locks, the most primitive of locks; everything is built on top of them. With any luck, you won’t ever need to use them directly, but it helps knowing how they work.

Eighty percent of what you need to know is synchronize. You create a lock, and then you use it to protect a piece of code that would go sideways if multiple threads hit it at the exact same time. Here’s a little class that locks around printing to standard output:

class Output

  def initialize
    @lock = Mutex.new
  end

  def log(msg)
    @lock.synchronize { puts msg }
  end

end

Using Output#log instead of puts will prevent the output of your multithreaded program from getting jumbled and confused by everyone writing to stdout at the same time. You could manually lock and unlock a Mutex if you had special needs.

Let’s talk counters

For the next couple examples, we’re going to implement a counter. Multiple threads will update said counter, so it needs to protect itself. Here’s how we use the counter:

    require 'thread'

    CORES=2
    ITERS=1_000

    threads = CORES.times.map do |n|
      Thread.new do
        ITERS.times do |i|
          out.log("Thread #{n}: Iteration: #{i} Counter: #{counter.value}") if i % 100 == 0
          counter.incr
        end
      end
    end

    threads.each(&:join)
    p counter.value

My Macbook Air has two real cores (don’t believe the hype!) and we’ll increment the counter a thousand times in each thread. Every hundred times through the loop, we’ll show some progress. At the end, we join each thread and then print the value of our counter. If all goes well, it will be CORES * ITERS.

All would not go well with this naive implementation:

class WildCounter

  def initialize
    @counter = 0
  end

  def incr
    @counter = @counter + 1
  end

  def value
    @counter
  end

end

If two threads execute incr at the same time, they will misread @counter or unintentionally overwrite a perfectly good value that was incremented behind their back.

We could protect this counter with a mutex, but I want to show you two other ways to go about it.

Monitors, aka intrinsic locks

Turns out, a well-designed class will tend to isolate state changes to a few methods. These “tell, don’t ask” methods are what you’ll likely end up locking. It would be pretty rad if you could just wrap a lock around the whole method without having to create variables and do a bunch of typing, don’t you think?

Those are a thing! They’re called monitors. You can read a bunch of academic stuff about them, but the crux of the biscuit is, a monitor is a lock around an entire instance of an object. You then declare methods that can only execute when that lock is held. Here’s a counter that uses a monitor:

require 'monitor'

class MonitorCounter

  def initialize
    @counter = 0
    # No idea why this doesn't work inside the class declaration
    extend(MonitorMixin)
  end

  def incr
    synchronize { @counter = @counter + 1 }
  end

  def value
    @counter
  end
end

It doesn’t look too much different from our naive counter. In the constructor, we extend Ruby’s MonitorMixin, which imbues this class with a lock and a synchronize method to protect mutator methods. (Ed. if anyone knows why the extend has to happen in the constructor instead of in the class declaration, I’m extremely stumped as to why!)

In incr, where we do the dirty work of updating the counter, all we need to do is put the actual logic inside a synchronize block. This ensures that only thread may execute this method on any given object instance at a time. Two threads could increment two counters safely, but if those two threads want to increment the same counter, they have to take turns.

A brief note on terminology: many Java concurrency texts refer to monitors as “intrinsic” locks because, in Java, they are part of every object. Mutexes are referred to as “extrinsic” locks because they aren’t tightly associated with any particular object instance.

Atomics, aka “wow that’s clever!”

It turns out that, in some cases, you can skip locks altogether. Amazing, right?!

Unfortunately, Ruby doesn’t have core support for atomic objects. Fortunately, Charles Nutter’s atomic library provides just that. It exploits operations provided by the underlying platform (the JVM in the case of JRuby, atomic compare-and-swap operations on Intel in the case of Rubinius) to implement objects that are guaranteed to update within one processor clock cycle. These operations work by taking two parameters, the old value and the new value; if the current value matches the old value, it’s safe to update it to the new value. If it doesn’t match, the operation fails and you have to try again.

Phew! Now you know a lot about atomic processor operations.

“Show me right now, Adam!” you say. Much obliged.

require 'atomic'

class AtomicCounter

  def initialize
    @counter = ::Atomic.new(0)
  end

  def incr
    @counter.update { |v| v + 1 }
  end

  def value
    @counter.value
  end

end

Luckily, Atomic encapsulates all the business of comparing and swapping and knowing about how to use atomic instructions. It maintains the value of the object internally and handles all the swapping logic for you. Call update, change the object in the block, and go on with your life. No locks necessary!

If that doesn’t make you love modern computer hardware, you are a programmer who does not know joy.

Tread carefully

Congratulations, you are now somewhat conversant on the topic of locking in concurrent Ruby programs. You know what the tools are, but, unfortunately, I haven’t the space to educate you on all the ways you are now equipped to shoot yourself in the foot. If you’re curious, you can read up on deadlock, livelock, starvation, priority inversion, and all the failure cases for dead processes left holding a lock.

The principle I try to follow, when I’m presented with a problem that needs locking, is to ask if I can work around the need for locking somehow. Could I use a Queue or atomic? Could I isolate this state in one thread and obviate the need for the lock? Is this state really necessary at all?

To anti-quote Ferris Buehler’s Day Off, when it comes to adding locks, “I highly unrecommend it, if you have the means”.

It’s easy to delude yourself when writing software. Do these tests really describe what the application does? Does the documentation really describe how the system works now? Is this comment an accurate assertion on the state of affairs in the application?

My experience is that there’s little to solve this problem besides discipline. Always double check that you haven’t invalidated something that was written down in the margins. If there’s a way to encode something in code instead of prose, do it.

Vigilance against future-lies is an ever-mindful challenge.

I’m a little too eager to add code. If there’s a mess that needs refurbishing, rather than refactoring, I’m too quick to create a parallel world that is nice and tidy like I’d like it. Problem is, I don’t come back to the code in want of refurbishing enough. I know I should rejigger it to use the new shiny bits. For some reason, call it inertia, I don’t.

This is a shot across my own bow. Prefer refactoring to refurbishing. Prefer refurbishing to jumping into something new. Prefer shipping code to all of the previous tactics.

People want to see action and progress, no matter how small. They want to hear about milestones and rave reviews. Even if you’re not adding new users and customers rapidly, you can still show momentum within the company and product. And if product updates aren’t forthcoming, hopefully you can be forthcoming about why. There are many different ways to make and measure progress, the point is to share them with your community regularly.

Traction by pal Brian Bailey. He's talking about how to get a new app off the ground, but this applies to any kind of business. Communication is winning.

Building an army of robots, Kyle Kneath on GitHub's internal tools. The closing line of this deck is "Surround yourself with beautiful software". One of the most compelling things I've looked at this year.

Architecting for change. Complex systems and change:

1. Distributed systems are inherently complex.
2. The outcome of change in complex systems is hard to predict.
3. The outcome of small, frequent, measurable changes are easier to predict, easier to recover from, and promote learning.

I’d have thought all the useful things to say about Etsy were said, at this point, but I’d have thought wrong!

There’s a good saying about designing distributed systems that goes something like “avoid it as long as possible”. I think these three guidelines are worth adding to that saying. Iterate, examine, repeat. Don’t make big, tricky changes. In fact, large change you can’t recover from are nearly impossible to make anyway, so route around them entirely.

The last bit, “promote learning”, is great too. I follow distributed systems and database designers on Twitter and see tons of great papers and ideas in the exchange. More than that, always teach your teammates about the distributed systems you’re building. The more they know about the design and constraints of the system you’re making, the easier it is for them to work with those systems. If you can’t teach someone to use your system, you probably don’t understand it well enough.

Read up on Thread and Queue and ready for more multi-threaded Ruby reading? Aaron Patterson has written up how the thread safe option works in Rails and some tradeoffs involved in removing the option and making thread safety the default. It’s not as complicated as you might think!

The only rub I can see is that, as far as I can tell, he’s talking about making this the default for production mode. Making it the default for development mode isn’t tenable if you want to keep class reloading around, which almost everyone does. It’s just a hunch, but running without thread-safety in development seems weird when its the default in production. But, some teams run YARV in development and JRuby in production, so maybe I’m just making up things to worry about.

Building a concurrent system isn’t as hard as they say it is. What it boils down to is, you can’t program by coincidence. Here’s a list of qualities in a strong developer:

• Comfort in thinking about the state(s) of their program
• Studies and understands the abstractions one or two layers above and below their program
• Listens to the design forces on their code

Happily, that’s all you need to get started writing code running in multiple threads. You don’t need a graduate degree, mathematical tricks, a specially-ordained language, or membership in the cult of writing concurrent programs.

Today, I want to give you a starting point for tinkering with and understanding concurrent programs, particularly in modern Ruby (JRuby and Rubinius 2.0).

Work queues, out-of-process and in-process

Lots of apps use a queue to get stuff done. Throw jobs on a queue, spin up a bunch of processes, run a job worker in those processes. Simple, right? Well, not entirely. You’ve got to store those jobs somewhere, make sure pulling jobs out of it won’t lose critical work, run worker processes somewhere, restart them if they fail, make sure they don’t leak memory, etc. Writing that first bit of code is easy, but deploying it ends up being a little costly.

If process concurrency is the only available trick, running a Resque-style job queue works. But now that thread concurrency is viable with Ruby, we can look at handling these same kind of jobs in-process instead of externally. At the cost of some additional code and additional possible states in our process, we save all sorts of operational complexity.

Baby’s first work queue

Resque is a great abstraction. Let’s see if we can build something like it. Here’s how we’ll add jobs to our in-process queue:

Work.enqueue(EchoJob, "I am doing work!")

And this is how we’ll define a worker. Note that I’ve gone with call instead of perform, because that is my wont lately.

class EchoJob
def call(message)
puts message
end
end

Simple enough. Now let’s make this thing actually work!

Humble beginnings

require 'thread'
require 'timeout'
module Work
@queue = Queue.new
@n_threads = 2
@workers = []
@running = true
Job = Struct.new(:worker, :params)

First off, we pull in thread, which gives us Thread and our new best friend, Queue. We also need timeout so we have a way to interrupt methods that block.

Then we define our global work queue, aptly named Work. It’s got a modest amount of state: a queue to store pending work on, a parameter for the number of threads (I went with two since my MacBook Air has two real cores), an array to keep track of the worker threads, and a flag that indicates whether the work queue should keep running.

Finally, we define a little job object, because schlepping data around inside a library with a hash is suboptimal. Data that represents a concept deserves a name and some structure!

A public API appears

  module_function
  def enqueue(worker, *params)
    @queue <;<; Job.new(worker, params)
  end
def start
@workers = @n_threads.times.map { Thread.new { process_jobs } }
end

This is the heart of the API. Note the use of module_function with no arguments; this makes all the following methods attach to the module object like class methods. This saves us the tedium of typing self.some_method all the time. Happy fingers!

Users of Work will add new jobs with enqueue, just like Resque. It’s a lot simpler in our case, though, because we never have to cross process boundaries. No marshaling, no problem.

Once the queue is loaded up (or even if it’s not), users then call start. This fires up a bunch of threads and starts processing jobs. We need to keep track of those threads for later, so we toss them into a module instance variable.

The crux of the biscuit

  def process_jobs
    while @running
      job = nil
      Timeout.timeout(1) do
        job = @queue.pop
      end
      job.worker.new.call(*job.params)
    end
  end

Here’s the heart of this humble little work queue. It’s easiest to look at this one from the inside out. The crux of the biscuit is popping off the queue. For one thing, this is thread-safe, so two workers can pop off the queue at the same time and get different jobs back.

More importantly, @queue.pop will block, forever, if the queue is empty. That makes it easy for us to avoid hogging the CPU fruitlessly looking for new work. It does, however, mean we need to wrap the pop operation in a timeout, so that we can eventually get back to our loop and do some housekeeping.

Housekeeping task the first, run that job. This looks almost just like the code you’ll find inside Resque workers. Create a new instance of the class that handles this job, invoke our call interface, pass the job params on. Easy!

Housekeeping task the second, see if the worker should keep running. If the @running flag is still set, we’re good to continue consuming work off the queue. If not, something has signaled that it’s time to wrap up.

Shutting down

  def drain
    loop do
      break if @queue.empty?
      sleep 1
    end
  end
def stop
@running = false
@workers.each(&:join)
end
end

Shutting down our work queue is a matter of draining any pending jobs and then closing out the running threads. drain is a little oddly named. It doesn’t actually do the draining, but it does block until the queue is drained. We use it as a precondition for calling stop, which tells all the workers to finish the job they’ve got and then exit their processing loop. We then call Thread#join to shutdown the worker threads.

All together now

This is how we use our cute little work queue:

10.times { |n| Work.enqueue(EchoJob, "I counted to #{n}") }
Process jobs in another thread(s)
Work.start
Block until all jobs are processed
Work.drain
Stop the workers
Work.stop

Create work, start our workers, block until they finish, and then stop working. Not too bad for fifty lines of code.

That wasn’t too hard

A lot is made about how the difficulty of concurrent programming. “Oh, the locks, oh the error cases!” they cry. Maybe it is trickier. But it’s not rocket science hard. Hell, it’s not even monads and contravariance hard.

What I hope I’ve demonstrated today is that concurrent programming, even in Ruby with all its implementation shortcomings, is approachable. To wit:

• Ruby has a great API for working with threads themselves. You call Thread.new and pass it some code to run in a thread. Done!
• Ruby’s Queue class is threadsafe and a great tool for coordinating concurrent programs. You can get pretty far without thinking about locks with a queue. Push things onto it from one thread, pull things off from another. Block on a queue until you get the signal you’re waiting for. It’s a lovely little abstraction.
• It’s easy to tinker with concurrency. You don’t have to write a giant program or have exotic problems to take advantage of Ruby for concurrency.

All that said, as I was writing this post up, some shortcomings in this example script jumped out at me. Output from the workers can appear out of order (classic concurrent program challenge), we can drain the queue while new work is still arriving (easily solved, but not with queues) and sleep loops (like in drain) are inelegant. If you want to read ahead, locks and latches are the droids you’re looking for.

I hope you see the ease with which we can get started doing concurrent Ruby programming by learning just two new classes. Don’t fear the threads, friend!</

It’s long past time to call Chronologic a project at it’s end-of-life. About a year ago, it went into serious use as the storage system for social timelines in Gowalla. About six months ago, the Gowalla servers were powered down; no epilogue was written. In my work since then, I haven’t had the need for storing timelines and I haven’t been using Cassandra much at all. So, what purpose can Chronologic serve now that it’s not actively moving bits around in production?

A pretty OK Ruby/Cassandra example application

If you’re curious about how to use Cassandra with Ruby, Chronologic is probably an excellent starting point. This is doubly so if you’re interested in building your own indexes or using the lower-level driver API instead of CQL. If you’re interested in the latest and greatest in Cassandra schema design, which you should be, Chronologic won’t help you learn how to use CQL, secondary indexes, or composite columns.

Chronologic is also an acceptable take on building service-oriented, distributed systems with Ruby. It is a good demonstration of a layered, if not particularly OO, architecture. That test suite is fast, and that was nice.

A source of software archaeology

Chronologic started out as a vacation hack that Scott Raymond put together in the winter of 2010. I picked it up and soft launched parts of it in early 2011. As Gowalla went into a drastic product push in the summer of 2011, the development of Chronologic accelerated drastically. A couple other developers started making contributions as Chronologic become a bigger factor in how we built our application and API.

Within the branches and commits, you can probably see design decisions come and go. Dumb bugs discovered and fixed. Sophisticated bugs instrumented, fixes attempted, final solutions triumphantly committed. An ambitious software historian might even glean the pace of development within the Gowalla team and infer the emotional rollercoaster of a big product push through the tone and pace of commits to Chronologic.

An epilogue for Chronologic

I’m a little sad that Chronologic couldn’t become a more general thing useful to a lot of people. I’m a lot sad that, by the time Gowalla was winding down, I was sufficiently burnt out that I wanted little to do with the Chronologic code. All that said, I’m very glad that Scott Raymond encouraged me to work on it and that the team at Gowalla worked with me as I blundered through building my first distributed, high-volume system. It was stressful and challenging, but I’m proud of the work I did and what I learned.

As teams grow and specialize, I’ve noticed people tend to take on characters that I see over and over. Archetypes that seem to go beyond one project and apply to each team I work on over time. Today I want to talk about one of those archetypes: the Grinder.

The Grinder isn’t the smartest or most skilled guy on your team. They don’t write the prettiest code, they aren’t up-to-date on the state of the art, and the way they use tools can seem simplistic. Often, The Grinder doesn’t even push working code; there are often tiny bugs lurking, or even syntax errors. They push or deploy this code perhaps dozens of times a day. At first glance, The Grinder is a Terrifying Problem.

What sets The Grinder apart from your garden variety mediocre developer is that The Grinder is an expert at making progress. They move rapidly, they upset things, and then they get it working. The Grinder is an indispensable part of your team because they’re a bit cutthroat. They’re not worrying about the coolest new tech or design approaches the intelligensia are raving about. They’re just thinking, “how do I get this into production, get feedback, and get on with the next thing?”

The Grinder is an indispensable part of your team because they balance out the thinkers and worriers. While they’re asking “can we?” and “should we?” the Grinder is just getting it done. Grinders expand the realm of possibility by taking a journey of a thousand steps. They don’t invent a jetpack or hoverboard first; they just go with what they have.

The Grinders I’ve known are typically humble, kind people. They know how to operate their tools to get stuff done, and that’s mostly good enough for them. They’re not opposed to hearing about new techniques, but they want to know how it’s going to help them push code out faster. They are not particularly phased by brainy tech that appeals to novelty.

Pair a Grinder with a thinker who values how their skills complement each other and you can make a ton of progress without making a huge mess. A team of all Grinders would eventually burn itself out. Grinders stop when the feature is done, not when the day is over or their brain is out of gas. Grinders need thinkers to encourage them to regulate their work pace and to help them understand how to make rapid progress without coding themselves into a corner.

It’s not hard to recognize the Grinder on your team; it’s likely even the non-technical people in the company know who they are and recognize their strengths. If you’re a thinker who is a little flabberghasted that the Grinder approach works, take some inspiration from how they do what they do and ship some stuff with them.

If it’s late or the Grinder has been working long hours, tap them on the shoulder and tell them they do good work. Send them home so they can sustain it over weeks and months without running themselves down. A well-rested, excited Grinder is one of your team’s best assets.

AC/DC writes songs that are fundamentally very strong. They aren’t the most touching, artistically composed songs. But they’re very solid songs. They hold together well, you can sing along, they don’t ramble on longer than they should.

How robust is AC/DC’s songwriting?

[youtube=www.youtube.com/watch

You can throw bagpipes into one of their songs and it still holds up just fine. That’s solid songwriting.

My Tuesdays typically look like this: write/hack for my weblog, work, lunch, work, short run, and then hack with other Austin nerds at Houndstooth Coffee. As it happens, I did OK on the write/hack, awesome at my first work sprint, OK at my second work sprint, OK on my run, and I’m currently kicking ass in my evening hacks on Sifter.

They can’t all be winners. If you’ve got enough fires going, one is bound to get hot on any given day. Push through the little disappointments to reach those moments of awesomeness.

As of 2012, the major forces operating on the legislation of the US government are, unscientifically speaking:

• 60% path dependence
• 20% regulatory capture
• 10% marginal progress
• 9% political posturing

Everything else, I’d guess, is a rounding error that fits in that last 1%.

Path dependence, in short, means that once you decide to do something, it’s difficult to unwind all the decisions that follow that original decision. Once you build a military-industrial complex, farm subsidy program, or medical program for the elderly, it becomes increasingly difficult to stop doing those things. You’re invested.

Regulatory capture is a phenomenon where a regulated industry, say telecom, becomes sufficiently cozy with the institutions regulating them that they can manipulate the regulators to ease the boundaries they must operate within, or even impose rules making it difficult for new competitors to enter the industry. To some extent, the prisoners run the asylum. Barring an extremely jarring event, like a financial emergency, the regulated can grow their profit margins, comfortable knowing that competitors are increasingly unlikely. More often, regulatory capture is about protecting the golden egg. Path dependence, in the form of subsidies and existing contracts, often goes hand-in-hand with regulatory capture.

Marginal progress is exactly what politicians are not rewarded for. They are rewarded for having strong ties to those with strong ties, for saying the right things, and staying out of the public eye. Politicians don’t enhance their career by doing what they tell their voters they seek to do.

Political posturing is what legislators are rewarded for. If they fail to accomplish what they’ve told voters they will do, they can always blameshift it away: not enough political will, distasteful political advesaries, more pressing problems elsewhere.

This seems cynical, but I’ve got my reasons:

• I find it helps to understand the forces at play before you try to figure out what to invest optimism in.
• Understanding a system is the first step towards making meaningful changes within it.

Actually, that’s a pretty good way to summarize my approach to understanding the systems of the world: understand the forces, learn the mechanisms, figure out how this system is interconnected to the other systems. The interconnections are the fun parts!

I’m going to pick on Resque here, since it’s nominally great and widely used. You’re probably using it in your application right now. Unfortunately, I need to tell you something unsettling.

¡DRAMA MUSIC!

There’s an outside chance your application is dropping jobs. The Resque worker process pulls a job off the queue, turns it into an object, and passes it to your class for processing. This handles the simple case beautifully. But failure cases are important, if tedious, to consider.

What happens if there’s an exception in your processing code? There’s a plugin for that. What happens if you need to restart your Resque process while a job is processing? There’s a signal for that. What if, between taking a job off the queue and fully processing it, the whole server disappears due to a network partition, hardware failure, or the FBI “borrowing” your rack?

¡DRAMA MUSIC!

Honestly, you shouldn’t treat Resque, or even Redis, like you would a relational or dynamo-style database. Redis, like memcached, is designed as a cache. You put things in it, you can get it out, really fast. Redis, like memcached, rarely falls over. But if it does, the remediation steps are manual. Redis currently doesn’t have a good High Availability setup (it’s being contemplated).

Further, Resque assumes that clients will properly process every message they dequeue. This isn’t a bad assumption. Most systems work most of the time. But, if a Resque worker process fails, it’s not great. It will lose all of the message(s) held in memory, and the Redis instance that runs your Resque queues is none the wiser.

¡DRAMA MUSIC!

In my past usage of Resque, this isn’t that big of a deal. Most jobs aren’t business-critical. If the occasional image doesn’t get resized or a notification email doesn’t go out, life goes on. A little logging and data tweaking cures many ills.

But, some jobs are business-critical. They need stronger semantics than Resque provides. The processing of those jobs, the state of that processing, is part of our application’s logic. We need to model those jobs in our application and store that state somewhere we can trust.

I first became really aware of this problem, and a nice solution to it, listening to the Ruby Rogues podcast. Therein, one of the panelists advised everyone to model crucial processing as state machines. The jobs become the transitions from one model state to the next. You store the state alongside an appropriate model in your database. If a job should get dropped, it’s possible to scan the database for models that are in an inconsistent state and issue the job again.

Example follows

Let’s work an example. For our imaginary application, comment notifications are really important. We want to make sure they get sent, come hell or high water. Here’s what our comment model looks like originally:

    class Comment

      after_create :send_notification

      def send_notification
        Resque.enqueue(NotifyUser, self.user_id, self.id)
      end

    end

Now we’ll add a job to send that notification:

    class NotifyUser
      @queue = :notify_user

      def self.perform(user_id, comment_id)
        # Load the user and comment, send a notification!
      end

    end

But, as I’ve pointed out with great drama, this code can drop jobs. Let’s throw that state machine in:

    class Comment
      # We'll use acts-as-state-machine. It's a classic.
      include AASM

      # We store the state of sending this notification in the aptly named
      # `notification_state` column. AASM gives us predicate methods to see if this
      # model is in the `pending?` or `sent?` states and a `notification_sent!`
      # method to go from one state to the next.
      aasm :column => :notification_state do
        state :pending, :initial => true
        state :sent

        event :notification_sent do
          transitions :to => :sent, :from => [:pending]
        end
      end

      after_create :send_notification

      def send_notification
        Resque.enqueue(NotifyUser, self.user_id, self.id)
      end

    end

Our notification has two states: pending, and sent. Our web app creates it in the pending state. After the job finishes, it will put it in the sent state.

    class NotifyUser
      @queue = :notify_user

      def self.perform(user_id, comment_id)
        user    = User.find(user_id)
        comment = Comment.find(comment_id)

        user.notify_for(comment)
        # Notification success! Update the comment's state.
        comment.notification_sent!
      end

    end

This a good start for more reliably processing jobs. However, most jobs happen to handle the interaction between two systems. This notification is a great example. It integrates our application with a mail server or another service that handles our notifications. Talking to those things is probably something that isn’t tolerant to duplicate requests. If our process croaks between the time it tells the mail server to send and the time it updates the notification state in our database, we could accidentally process this notification twice. Back to square one?

¡DRAMA MUSIC!

Not quite. We can reduce our problem space once more by adding another state to our model.

    class Comment
      include AASM

      aasm :column => :notification_state do
        state :pending, :initial => true
        state :sending # We have attempted to send a notification
        state :sent    # The notification succeeded
        state :error   # Something is amiss :(

        # This happens right before we attempt to send the notification
        event :notification_attempted do
          transitions :to => :sending, :from [:pending]
        end

        # We take this transition if an exception occurs
        event :notification_error do
          transitions :to => :error, :from => [:sending]
        end

        # When everything goes to plan, we take this transition
        event :notification_sent do
          transitions :to => :sent, :from => [:sending]
        end

      end

      after_create :send_notification

      def send_notification
        Resque.enqueue(NotifyUser, self.user_id, self.id)
      end

    end

Now, when our job processes a notification, it first uses notification_attempted. Should this job fail, we’ll know which jobs we should look for in our logs. We could also get a little sneaky and monitor the number of jobs in this state if we think we’re bottlenecking around sending the actual notification. Once the job completes, we transition to the sent state. If anything goes wrong, we catch the exception and put the job in the error state. We definitely want to monitor this state and use the logs to figure out what went wrong, manually fix it, and perhaps write some code to fix bugs or add robustness.

The sending state is entered when at least one worker has picked up a notification and tried to send the message. Should that worker fail in sending the message or updating the database, we will know. When trouble strikes, we’ll know we have two cases to deal with: notifications that haven’t been touched at all, and notifications that were attempted and may have succeeded. The former, we’ll handle by requeueing them. The latter, we’ll probably have to write a script to grep our mail logs and see if we successfully sent the message. (You are logging everything, centrally aggregating it, and know how to search it, right?)

The truth is, the integration points between systems is a gnarly problem. You don’t so much solve the problem; you get really good at detecting and responding to edge cases. Thus is life in production. But losing jobs, we can make really good progress on that. Don’t worry about your low-value jobs; start with the really important ones, and weave the state of those jobs into the rest of your application. Some day, you’ll thank yourself for doing that.

Writer’s block gets all the attention. It robs the inspired and stunts the progress of those with a deadline to beat. It’s a starting problem.

At some point, I learned all the tricks for overcoming the start, for getting past the blank canvas. Now, I find myself challenged by the converse. I have a finishing problem. I’m always convincing myself that I’m not done.

How do I get a bunch of words to feel like a cohesive essay? What’s needed to ship this code? How do I get this awesome joke to fit into one little tweet?!

It’s an ongoing challenge. Even if the essay, code, or joke I’m working on isn’t throwing me curveballs, my head can jump in and impose one. Not eloquent enough, has a potential bug, too obtuse. My brain can come up with any number of ways to convince me that I shouldn’t call the thing done.

Here are some things I’ve been trying to outthink my brain:

• Before I sit down to make something, decide what the goal for the session is. Am I trying to get started, explore a new direction, edit or refactor something, or push through the details needed to finish?
• When I start something, outline it. What is the beginning, middle, and end of the thing? What is the result? What are the materials (example code, a demonstrative screenshot, a funny picture), and do I need to acquire or create them?
• Put up a little resistance when the temptation to start something new strikes. Consider whether it’s an exploration or a creation. Can I easily turn it into something I can publicize (on my weblog, GitHub, etc.), or is it an intermediate or even throwaway product?

I’m not sure if any of these will prove reliable finishers. Your mileage may vary.

Here’s my desktop folder. I shoved all my previously unfinished projects in another folder and wiped the slate clean.

Game on.

Lots of folks consider case expressions in Ruby a code smell. I’m not ready to write them off just yet, but I know a good replacement for some uses of case when I see it. Rad co-worker David Copeland’s Lookup Tables With Lambdas is one of those replacements. For cases where a method takes a parameter, throws it into a case, and returns a value, I can replace all that lookup business with a hash lookup. To carry the metaphor through, the hash is the lookup table. Rad.

Where it gets fun is when I need to do some kind of dynamic lookup in the hash. Normally I wouldn’t want to do that when the Ruby interpreter parses my hash literal. If I reach into my functional programming bag of tricks, I recall that lambdas can be used to defer evaluation. And that’s exactly what David recommends. If I’ve got database lookups or logic I need to embed in my tables, Ruby’s lambda comes to the rescue!

This approach works great at the small-to-medium scale. That said, I always keep in mind that a bunch of methods manipulating a hash, using its keys as a convention, is an encapsulated, orthogonal object begging to happen. Remember, it’s Ruby; we can make our objects behave like hashes but still do OO- and test-driven design.

I was afraid that David Chelimsky was going to take away my toys! Consider, explicit use of subject in RSpec considered a smell:

The problem with this example is that the word “subject” is not very intention revealing. That might not appear problematic in this small example because you can see the declaration on line 3 and the reference on line 6. But when this group grows to where you have to scroll up from the reference to find the declaration, the generic nature of the word “subject” becomes a hinderance to understanding and slows you down.

I’m so guilty of using subject heavily. Even worse, I’ve been advocating it to others too. In my defense, it does lend a good deal of concision to specs and seemed like a golden path.

Luckily, David isn’t taking away my toys. He’s got an even better recommendation: just use a method or let with a intention-revealing name. Here’s his example:

describe Article do
  def article; Article.new; end

  it "validates presence of :title" do
    article.should validate_presence_of(:title)
  end
end

This is, now that I’m looking at it, way better. As this spec grows, you can add helpers for article_with_comments, article_with_author, etc. and it’s clear right on the line that helper is used what’s going on. No jumping back and forth between contexts. Thumbs up!