It’s handy to know a lot about programming langauges, patterns, “best” practices, and anecdotal experience in applying those ideas. But premature application of ideas can lead to its own bad experiences and new anecdotal evidence. How can you apply ideas without falling into the premature architecture/optimization trap?

Follow the smells.

You know what a bad method looks like. You know what slow code looks like and how to find it. You can tell when a class is doing too many things. Follow those smells.

Once you’ve smelled it, you’ve dealt it. Kidding!

Once you’ve found a culprit that is imminently making your life as a developer harder, you have to explain it to someone else. Tell them why it’s slow, badly coupled, or too much architecture. Once they’re convinced, then you can reach into that deep knowledge of languages, patterns, practices, and experience to fix the problem.

You commit the code, push it to production, and do a happy dance. You’re not done.

Now you need to share it. Write down what you did and share it with your team, lest the problem happen again. Show your co-workers how to find the problem elsewhere and quickly dispatch it.

If you do it this way, you’re helping yourself three times:

1. You fixed the original problem
2. You showed your colleagues how to fix the problem or how to avoid creating the problem
3. You showed your colleagues how to make measurable, if minor, progress in making your application better without the need for drastic projects

Follow the smells. Fix the smell. Put the fix in everyone else’s brain. Repeat.

When doing object-oriented programming and following SOLID principles, there is sometimes a concern that classes will proliferate and you end up with a Kingdom of Nouns.

I find it more concerning when there is a proliferation of concepts. Too many layers, too many patterns, too many frameworks. The best designs I’ve seen provide just one or two concepts that manage to tie everything together without breaking orthogonality.

The great thing about Rails and Sinatra was that they boiled previously conceptually heavy domains down to a simpler, better set of ideas. In doing so, they move you up the hierarchy of needs, so to speak, and you still end up inventing your own concepts and nouns. More on that soon!

A lot is made about how difficult it is to write multi-threaded programs. No doubt, it is harder than writing a CRUD application or your own testing library. On the other hand, it’s not as difficult as writing a database or 3D graphics engine. The point is, it’s worth learning how to do. Skipping the hubris and knowing your program will have bugs that require discipline to track down is an enabling step to learning to write multithreaded programs.

I haven’t seen much written about the experience of writing a concurrent program and how one designs classes and programs with the rules of concurrency in mind. So let’s look at what I’ve learned about designing threaded programs so far.

The headline is this: only allow objects in consistent states and don’t rely on changing state unless you have to. Let’s first look at a class that does not embody those principles at all.

class Rectangle
  attr_accessor :width, :height

  def orientation
    if width > height
      WIDE
    else
      TALL
    end
  end

  WIDE = "WIDE".freeze
  TALL = "TALL".freeze
end

Just for fun, mentally review that code. What are the shortcomings, what could go wrong, what would you advise the writer to change?

For our purposes, the first flaw is that new Rectangle objects are in an inconsistent state. If we create an object and immediately call orientation, bad things will happen. If you’re typing along at home:

begin
  r = Rectangle.new
  puts r.orientation
rescue
  puts "whoops, inconsistent"
end

The second flaw is that our object allows bad data. We should not be able to do this:

r.width = 100
r.height = -20
puts r.orientation

Alas, we can. The third flaw is that we could accidentally share this object across threads and end up messing up the state in one threads because of logic in another thread. This sort of bug is really difficult to figure out, so designing our objects so it can’t happen is highly desirable. We want to make this sort of code safe:

r.height = 150
puts r.orientation

When we modify width or height on a rectangle, we should get back an entirely new object.

Let’s go about fixing each of these flaws.

Encapsulate object state with Tell, Don’t Ask

The first flaw in our Rectangle class is that it isn’t guaranteed to exist in a consistent state. We go through contortions to make sure our databases are consistent; we should do the same with our Ruby objects too. When an object is created, it should be ready to go. It should not be possible to create a new object that is inconsistent.

Further, we can solve the second flaw by enforcing constraints on our objects. We use the “Tell, Don’t Ask” principle to ensure that when users of Rectangle change the object’s state, they don’t get direct access to the object’s state. Instead, they must pass through guards that protect our object’s state.

All of that sounds fancy, but it really couldn’t be simpler. You’re probably already writing your Ruby classes this way:

class Rectangle
  attr_reader :width, :height

  def initialize(width, height)
    @width, @height = width, height
  end

  def width=(w)
    raise "Negative dimensions are invalid" if w < 0
    @width = w
  end

  def height=(h)
    raise "Negative dimensions are invalid" if h < 0
    @height = h
  end

  def orientation
    if width > height
      WIDE
    else
      TALL
    end
  end

end

A lot of little things have changed in this class:

• The constructor now requires the width and height arguments. If you don’t know the width and height, you can’t create a valid rectangle, so why let anyone get confused and create a rectangle that doesn’t work? Our constructor now encodes and enforces this requirement.
• The width= and height= setters now enforce validation on the new values. If the constraints aren’t met, a rather blunt exception is raised. If everything is fine, the setters work just like they did in the old class.
• Because we’ve written our own setters, we use attr_reader instead of attr_accessor.

With just a bit of code, a little explicitness here and there, we’ve now got a Rectangle whose failure potential is far smaller than the naive version. This is simply good design. Why wouldn’t you want a class that is designed not to silently blow up in your face?

The crux of the biscuit for this article is that now we have an object with a narrower interface and an explicit interface. If we need to introduce a concurrency mechanism like locking or serialization (i.e. serial execution), we have some straight-forward places to do so. An explicit interface, specific messages an object responds to, opens up a world of good design consequences!

Lean towards immutability and value objects whenever possible

The third flaw in the naive Rectangle class is that it could accidentally be shared across threads, with possibly hard to detect consequences. We can get around that using a technique borrowed from Clojure and Erlang: immutable objects.

class Rectangle
  attr_reader :width, :height

  def initialize(width, height)
    validate_width(width)
    validate_height(height)
    @width, @height = width, height
  end

  def validate_width(w)
    raise "Negative dimensions are invalid" if w < 0
  end

  def validate_height(h)
    raise "Negative dimensions are invalid" if h < 0
  end

  def set_width(w)
    self.class.new(w, height)
  end

  def set_height(h)
    self.class.new(width, h)
  end

  def orientation
    if width > height
      WIDE
    else
      TALL
    end
  end

end

This version of Rectangle further extracts the validation logic into separate methods so we can call it from the constructor and from the setters. But, look more closely at the setters. They do something you don’t often see in Ruby code. Instead of changing self, these setters create an entirely new Rectangle instance with new dimensions.

The upside to this is, if you accidentally share an object across threads, any changes to the object will result in a new object owned by the thread that initiated the change. This means you don’t have to worry about locking around these Rectangles; in practice, sharing is, at worst, copying.

The downside to this side is you could end up with a proliferation of Rectangle objects in memory. This puts pressure on the Ruby GC, which might cause operational headaches further down the line. Clojure gets around this by using persistent data structures that are able to safely share their internal structures, reducing memory requirements. Hamster is one attempt at bringing such “persistent” data structures to Ruby.

Let’s think about object design some more. If you’ve read up on domain-driven design, you probably recognize that Rectangle is a value object. It doesn’t represent any particular rectangle. It binds a little bit of behavior to a domain concept our program uses.

That wasn’t so hard, now was it

I keep trying to tell people that, in some ways, writing multithreaded program is as simple as applying common object-oriented design principles. Build objects that are always in a sensible state, don’t allow twiddling that state without going through the object’s interface, use value objects when possible, and consider using immutable value objects if you’re starting from scratch.

Following these principles drastically reduces the number of states you have to think about and thus makes it easier to reason about how the program will run with multiple threads and how to protect data with whatever form of lock is appropriate.

A future is a financial instrument (a thing you invest in) where you commit to paying a price today to receive something tomorrow. The price could go up or down tomorrow, but you’re locked into today’s price. Price goes up, you profit; price goes down, you eat the difference.

A feature is a thing that software does. For our purposes, we’ll say it’s also work that enables a feature: setting up CI, writing tests, refactoring code, adding documentation, etc. The general idea behind software development is that you should gain more time using a feature than the time you spent implementing it.

The Enterprise-D is a fictional space ship in the Star Trek: The Next Generation universe. It can split into two spaceships and is pretty well armed for a ship with an exploratory mission.

Today, Geordi and Worf (middle management) are recalibrating the forward sensor array. It takes them most of the day, but they get the job done. Captain Picard is studying ancient pan-flutes of the iron-age Vulcan era. Data (an android), as an experiment on his positronic net, is trying to learn how to tell an Aristocrat joke.

Tomorrow, in a series of events no one could predict, our friends find themselves in a tense situation with a Romulan Bird of Prey. Luckily, Worf detected it minutes before it decloaked, thanks to the work he and Geordi had performed the day before. This particular Bird of Prey is carrying ancient Romulan artifacts dating back to their own iron age. Amazingly, Picard is able to save the day by translating the inscriptions, which aren’t too different from Vulcan pan-flutes, and prevents an ancient doomsday weapon from consuming the Bird of Prey and Enterprise alike.

Data’s Aristocrat joke is never used. That’s good, because this is a family show.

Our friends on the Enterprise are savvy investors who look at their efforts in terms of risk and reward. They each invest time today into an activity (an instrument, in financial terms) which they may or may not use tomorrow. We can say that if they end up using the instrument, it pays off. We can then measure the pay-off of that instrument by assigning a value to the utility of that instrument. If the value of the instrument exceeds the time they invested in “acquiring” it, there is a profit.

Geordi and Worf’s investment was clearly a profit-bearing endeavor. Few other uses of their time, such as aligning the warp crystals or practicing Klingon combat moves, could have detected an invisible ship before it uninvisibles itself. In Wall Street terms, Geordi and Worf are getting the fat bonus and bottle of Bollinger champagne.

Picard’s investment seems less clear cut. It did come in handy in this particular case, but it probably wasn’t the only activity that would have saved the day. He could have belted out some Shakespeare or delegated to one of his officers to reconfigure the deflector dish. We’ll mark Picard as even for the day.

Data totally blew this one. His Aristocrat joke went unused. Even if he had used it, the best outcome would be that it’s a lame, sterile groaner that only ends up on a DVD extras reel. Data is in the red.

In terms of futures, we can say that the price of working on the foward sensor array went up, the price of pan-flute research was largely unchanged, and the price of Aristocrat jokes plummeted. Our friends on the Enterprise implicitly decided what risks are the most important to them and hedged against three of them. Some of them even came out ahead!

I’m working on software. Today, I can choose to do things on that software. I could 1) start on adding a new feature, 2) shore up the test suite, or 3) get CI setup and all-green. Respectively, these are futures addressing 1) the risk of losing money due to missing functionality, 2) losing money because adding features takes too long to get right, or 3) losing money because things are broken or not communicated in a timely manner.

Like our Enterprise episode, it’s hard to value these futures. If I deliver the feature tomorrow and it generates more money than the time I put into implementing, testing, and deploying the code, we’re looking at a clear profit. Revenue minus expense equals profit, grossly speaking.

Shoring up the test suite might make another feature easier to implement. It might give me confidence in moving code around to facilitate. It could tell me when I’ve broken some code, or some code is poorly designed and holding me back. But, these values are super hard to quantify. Did I save two hours on some feature because I spent one hour on the test suite yesterday? Tricky question!

Chore-ish tasks, like standing up a CI server or centralizing logs, are even harder to quantify. Either one of these tasks could save hours and days of wasted time due to missed communication or troubleshooting an opaque system. Or, they might not pay off at all for weeks and months.

I’m going to start writing down what I worked on every day, guess how many hours I spent on it, and then revisit each task weekly or monthly to guess if it paid out. Maybe I’ll develop an intuition for risk and reward for the things I work on. Maybe I’ll just end up with a mess of numbers. Almost certainly, I will seem pretty bookish and weird for tracking these sorts of things.

You should look bookish and weird too. Let me know what you find. I’ll write up whatever we figure out. Maybe there’s something to this whole “finance” thing besides nearly wrecking the global economy!

My magic ball predicts that OO and FP are going to take something of a “romantic comedy” path of evolution.

Act I. OO and FP are introduced at dinner parties and they could not seem more dissimilar and hilarious arguments ensue. No one goes home together. Despite the initial miss, the end of Act I finds OO and FP separately talking to friends about how they want the same things.

Act II. OO and FP run into each other at the coffee shop, and then again at the gym. OO is reading a book on ideas that FP loves. One of their friends invites them both to a bar, they get a little sauced and end up making out a bit. OO starts wearing FP’s jacket around town, even finding it a little comfortable. Towards the end of Act II, OO and FP are a bonfide thing, both borrowing ideas from each other. It’s pretty cute.

Act III. Open with a fight between OO and FP. It seems they just can’t come to agree on some important topic like mutability or the nature of behavior and state. Unfortunate and emotional words are uttered. The internet is abuzz with talk of the drama. They go back to their respective friends and rant about the shortcomings of the other. But, late at night, OO finds that not having FP around is less awesome than having FP around. OO cooks up a cooky plan to get FP back into their life. Hilarity, and a little awkwardness ensue. In the end, FP and OO go great together and we end with a montage of “everyone lived happily after” and see a clip that alludes to an OO/FP baby on the way.

If you’re playing at home, we’re already in Act II. Ruby and Python borrow various ideas on iteration from FP languages. We might be towards the end of Act II; Scala is very much wearing ML’s jacket around town. Surely there will be fallout at some point, someone ranting about how OO FP hybrids are too large, too poorly designed, too complicated, etc. The dust will settle, and someone will build an even better OO FP hybrid. Act III will play repeatedly until no one thinks of languages as OO FP hybrids, they just think of them as another language.

Then something different from OO or FP will become obviously useful and this whole romantic comedy will play again. It’s the way of Hollywood, and the way of software development. Everything old is new again; everything new is old again. Rinse, repeat.

I’ve noticed some of the sharpest developers I know are doing one or both of these things:

• Rediscovering object oriented design. Practicing evolving a design, often driven by the pain points illuminated by automated tests. Thinking about coupling and cohesion. Trying to encapsulate the right behaviors and find decide which principles are the most appropriate to the languages and systems they’re using.

• Rediscovering functions. Applying functional programming to everyday programming problems. Using the features of functional languages as an advantage to build concurrent and distributed systems. Finding the differences in functional design and writing more idiomatic code.

The first is a cyclical thing. It happened in Java, it happened in .NET, it’s happening in Ruby now. People come to a language for what makes it different, write a lot of stuff, and keep bumping into the same problems. They (re-)discover OO, start refactoring things and shaping their systems differently. Some people dig it, others dismiss it as too much effort or ceremony. A new thing comes along, and it all happens again.

The second is harder for me to read. I’ve spent a fair amount of time studying FP, though I have yet to apply it to production software. Despite that, I have come across a lot of good ideas that are already part of the code I work with daily, or that I wish was part of the code I work with. FP has good answers to composing systems, reasoning about state, and handling concurrency. It has often suffered from a lack of pragmatism, overly dense literature, and rough tooling. The ideas are worth stealing, even if they haven’t broadly succeeded.

Both of these trends are crucial to moving the practice of software development forward. We need to keep rediscovering and sharpening old ideas whilst experimenting with new ideas to find out which ones are good and which ones less so.

Little-d distributed systems: the accidental sort. You built a program, it ran on one server. Then you added a database, some caches, perhaps a job worker somewhere. Whoops, you made a distributed system! Almost everything works this way now.

Big-D distributed systems: you read the Dynamo paper, maybe some Lamport papers too, and you set out to build on the principles set forth by those who have researched the topic. This is mostly open source distributed databases, but other systems surely fall under this category.

Ph.D distributed systems: you went to a top CS school, you ended up working with a distributed systems professor, and you wrote a system. You then graduated, ended up at Google, Facebook, Amazon, etc. and ended up writing more distributed systems, on a team of even more Ph.D’s.

If you’re building a little-d distributed system, study the patterns in the Big-D distributed systems. If you’re building a Big-D distributed, study what the Ph. D guys are writing. If you’re a Ph. D distributed system guy, please, write in clear and concise language! No one knows or cares what all the little greek symbols are, they just want to know what works, what doesn’t work, and why.

You need to protect a piece of data, like a counter or an output stream, from getting garbled by multiple threads.

Three choices, hot shot:

• Explicit locks (aka mutexes): acquire a lock around the “critical section”, munge the data, release the lock. You have to manage the lock yourself. Multiple threads accessing the lock will not run concurrently anymore.
• Implicit locks (aka monitors): annotate methods that modify important data. The monitor library manages the lock for you. Threads still serialize around the lock, reducing concurrency.
• Atomic objects (aka compare-and-swap): use data structures that take advantage of runtime or processor semantics to guarantee that competing threads never interfere with each other. No locks! Much less serializing! Not broadly applicable, but I highly recommend them when you have the means.

Mutexes, aka lock “classic”

Mutexes are the lowest level of locks, at least in Ruby. They are the ur-locks, the most primitive of locks; everything is built on top of them. With any luck, you won’t ever need to use them directly, but it helps knowing how they work.

Eighty percent of what you need to know is synchronize. You create a lock, and then you use it to protect a piece of code that would go sideways if multiple threads hit it at the exact same time. Here’s a little class that locks around printing to standard output:

class Output

  def initialize
    @lock = Mutex.new
  end

  def log(msg)
    @lock.synchronize { puts msg }
  end

end

Using Output#log instead of puts will prevent the output of your multithreaded program from getting jumbled and confused by everyone writing to stdout at the same time. You could manually lock and unlock a Mutex if you had special needs.

Let’s talk counters

For the next couple examples, we’re going to implement a counter. Multiple threads will update said counter, so it needs to protect itself. Here’s how we use the counter:

    require 'thread'

    CORES=2
    ITERS=1_000

    threads = CORES.times.map do |n|
      Thread.new do
        ITERS.times do |i|
          out.log("Thread #{n}: Iteration: #{i} Counter: #{counter.value}") if i % 100 == 0
          counter.incr
        end
      end
    end

    threads.each(&:join)
    p counter.value

My Macbook Air has two real cores (don’t believe the hype!) and we’ll increment the counter a thousand times in each thread. Every hundred times through the loop, we’ll show some progress. At the end, we join each thread and then print the value of our counter. If all goes well, it will be CORES * ITERS.

All would not go well with this naive implementation:

class WildCounter

  def initialize
    @counter = 0
  end

  def incr
    @counter = @counter + 1
  end

  def value
    @counter
  end

end

If two threads execute incr at the same time, they will misread @counter or unintentionally overwrite a perfectly good value that was incremented behind their back.

We could protect this counter with a mutex, but I want to show you two other ways to go about it.

Monitors, aka intrinsic locks

Turns out, a well-designed class will tend to isolate state changes to a few methods. These “tell, don’t ask” methods are what you’ll likely end up locking. It would be pretty rad if you could just wrap a lock around the whole method without having to create variables and do a bunch of typing, don’t you think?

Those are a thing! They’re called monitors. You can read a bunch of academic stuff about them, but the crux of the biscuit is, a monitor is a lock around an entire instance of an object. You then declare methods that can only execute when that lock is held. Here’s a counter that uses a monitor:

require 'monitor'

class MonitorCounter

  def initialize
    @counter = 0
    # No idea why this doesn't work inside the class declaration
    extend(MonitorMixin)
  end

  def incr
    synchronize { @counter = @counter + 1 }
  end

  def value
    @counter
  end
end

It doesn’t look too much different from our naive counter. In the constructor, we extend Ruby’s MonitorMixin, which imbues this class with a lock and a synchronize method to protect mutator methods. (Ed. if anyone knows why the extend has to happen in the constructor instead of in the class declaration, I’m extremely stumped as to why!)

In incr, where we do the dirty work of updating the counter, all we need to do is put the actual logic inside a synchronize block. This ensures that only thread may execute this method on any given object instance at a time. Two threads could increment two counters safely, but if those two threads want to increment the same counter, they have to take turns.

A brief note on terminology: many Java concurrency texts refer to monitors as “intrinsic” locks because, in Java, they are part of every object. Mutexes are referred to as “extrinsic” locks because they aren’t tightly associated with any particular object instance.

Atomics, aka “wow that’s clever!”

It turns out that, in some cases, you can skip locks altogether. Amazing, right?!

Unfortunately, Ruby doesn’t have core support for atomic objects. Fortunately, Charles Nutter’s atomic library provides just that. It exploits operations provided by the underlying platform (the JVM in the case of JRuby, atomic compare-and-swap operations on Intel in the case of Rubinius) to implement objects that are guaranteed to update within one processor clock cycle. These operations work by taking two parameters, the old value and the new value; if the current value matches the old value, it’s safe to update it to the new value. If it doesn’t match, the operation fails and you have to try again.

Phew! Now you know a lot about atomic processor operations.

“Show me right now, Adam!” you say. Much obliged.

require 'atomic'

class AtomicCounter

  def initialize
    @counter = ::Atomic.new(0)
  end

  def incr
    @counter.update { |v| v + 1 }
  end

  def value
    @counter.value
  end

end

Luckily, Atomic encapsulates all the business of comparing and swapping and knowing about how to use atomic instructions. It maintains the value of the object internally and handles all the swapping logic for you. Call update, change the object in the block, and go on with your life. No locks necessary!

If that doesn’t make you love modern computer hardware, you are a programmer who does not know joy.

Tread carefully

Congratulations, you are now somewhat conversant on the topic of locking in concurrent Ruby programs. You know what the tools are, but, unfortunately, I haven’t the space to educate you on all the ways you are now equipped to shoot yourself in the foot. If you’re curious, you can read up on deadlock, livelock, starvation, priority inversion, and all the failure cases for dead processes left holding a lock.

The principle I try to follow, when I’m presented with a problem that needs locking, is to ask if I can work around the need for locking somehow. Could I use a Queue or atomic? Could I isolate this state in one thread and obviate the need for the lock? Is this state really necessary at all?

To anti-quote Ferris Buehler’s Day Off, when it comes to adding locks, “I highly unrecommend it, if you have the means”.

It’s long past time to call Chronologic a project at it’s end-of-life. About a year ago, it went into serious use as the storage system for social timelines in Gowalla. About six months ago, the Gowalla servers were powered down; no epilogue was written. In my work since then, I haven’t had the need for storing timelines and I haven’t been using Cassandra much at all. So, what purpose can Chronologic serve now that it’s not actively moving bits around in production?

A pretty OK Ruby/Cassandra example application

If you’re curious about how to use Cassandra with Ruby, Chronologic is probably an excellent starting point. This is doubly so if you’re interested in building your own indexes or using the lower-level driver API instead of CQL. If you’re interested in the latest and greatest in Cassandra schema design, which you should be, Chronologic won’t help you learn how to use CQL, secondary indexes, or composite columns.

Chronologic is also an acceptable take on building service-oriented, distributed systems with Ruby. It is a good demonstration of a layered, if not particularly OO, architecture. That test suite is fast, and that was nice.

A source of software archaeology

Chronologic started out as a vacation hack that Scott Raymond put together in the winter of 2010. I picked it up and soft launched parts of it in early 2011. As Gowalla went into a drastic product push in the summer of 2011, the development of Chronologic accelerated drastically. A couple other developers started making contributions as Chronologic become a bigger factor in how we built our application and API.

Within the branches and commits, you can probably see design decisions come and go. Dumb bugs discovered and fixed. Sophisticated bugs instrumented, fixes attempted, final solutions triumphantly committed. An ambitious software historian might even glean the pace of development within the Gowalla team and infer the emotional rollercoaster of a big product push through the tone and pace of commits to Chronologic.

An epilogue for Chronologic

I’m a little sad that Chronologic couldn’t become a more general thing useful to a lot of people. I’m a lot sad that, by the time Gowalla was winding down, I was sufficiently burnt out that I wanted little to do with the Chronologic code. All that said, I’m very glad that Scott Raymond encouraged me to work on it and that the team at Gowalla worked with me as I blundered through building my first distributed, high-volume system. It was stressful and challenging, but I’m proud of the work I did and what I learned.