Uncategorized
Draw your software
Better Code Design through Pictures:
Looking at a picture like this reveals so much that is missing when only looking at Emacs or Vim. Classes that violate the Single Responsibility Principle may become obvious because they’re related to too many other classes. Cyclical dependencies might be identified. Even class names may be brought into question. These discoveries are not very obvious when writing code, but they were remarkably obvious once we threw the structure up on the whiteboard.I almost always have some kind of notebook and pen by my side so that I can doodle words and shapes. Having a whiteboard nearby is even better.
Find the classes lurking in your ActiveRecord models
This advice is going on a year old, but it’s still some of the best around. If you’ve got ungainly ActiveRecord objects that are doing way more than abstracting your data model, you are missing classes in the design of your application. Chances are, one of the objects Bryan describes here is what you might want to extract.
Developers are weird with words
Naming things is hard. Witness things that developers have named and then struggle to explain because words and people are weird:
- TDD sounds like it's about testing, but it's really a design technique.
- BDD sounds like it's about what code does, but it's really a communication discipline.
- Outside-in development sounds like a way to discover the design of software, but it's really a technique for building software using a fractal todo list.
20% of programming is duh
Sometimes I think 20% of programming is staring at a problem for thirty minutes and thinking there must be a really simple solution right in front of me and then, eureka and then, duh.
Note that it sometimes takes multiple sittings to reach that simple solution. Last night, for example, no eureka. Step away, try again, repeat. Brains are weird.
Oh, the complexities you'll know
Carried complexity is the bane of your application. When you add something to software, you incur the cost of doing the work plus the risk that the change will break something in already deployed versions of the software. But you also possibly incur the cost of understanding that addition later, probably in the context of other additions, and then trying to figure out what’s broken, slow, or needs to change to add this other new thing.
Software developers are most adept at identifying and resisting complexity when it comes from the product or business guys. Adding this feature will take a few days, fixing this bug will take a few hours, generating that report will take a week. This change might sound good for getting that new customer, but the way it interacts with the other features will confuse lots of people in the long run.
You’ve probably thought or said all of these things in the past.
It’s important to realize we put it upon ourselves too. Are there two ways of solving the same problem in your project, team, or organization? Are you migrating from one framework/queue/architecture to another? Those are complexities that you are carrying around in your head. They aren’t inherently bad, but they are costly, and it is important to weigh the pros and cons before you commit to them. When you choose to incur complexity, make sure it’s buying you something and not just shuffling the bits around a little bit.
Finishing software ain't easy
When I start work on a project, whether for personal or professional purposes, I have a sense that I need to devote myself to it. That I should figure out how to make it the best it can be, I should only commit the best codes to it; it should be an exemplary piece of work.
On the one hand, this is taking pride in my work, and that’s good! On the other hand, this is ownership, and that’s a little problematic.
The problem with ownership is it leads to irrational decisions. There’s always one more improvement I could make, because this is the thing I own. There is little bit of scope I could add, because I can make this thing really good (at least in my head). There’s one more bug to fix (inexplicably, in my head).
What I’m finding all this points to is, finishing is hard. It’s easy to start a project. It’s challenging to get it out there, and get people to use it. It’s really hard to get it to the point where it’s running itself, you can delegate its operation to others, or it doesn’t need further work.
So that’s a thing I’m trying to figure out. Developing.
The simple problem inside the complex one
A sophisticated solution to a complex problem is fun to find. Its even fun when someone else finds the solution and thoroughly writes it down.
Despite the thrill of recognition for tackling a big complex problem, I find it's more practical to seek the simple problem lurking in and around the big problem. Not all tricky problems are complex. Some are presented in a complex way, some are complex because of restrictions that are easily worked around, and some are complex because of an adjoining social problem. Find the simple problem, or the social problem, and solve that instead. It often works for me.
My favorite kind of solution is one where the solver has taken a problem with a big surface area, found the core of the problem that is 80% of what people care about having solved, and then solved that tiny subproblem. If you want to see masters of this approach, check out the works of Blake Mizerany and Ryan Smith.
Scala and Clojure in terms of city building
The Scala folks are building newer, better cities on top of older cities, which is how things really work (e.g. Paris, Rome, Boston, etc.). The Clojure folks are building Dyson spheres in space, which is a little ambitious and maybe not entirely realistic.
That said, sometimes wouldn’t you rather live in a sci-fi fantasy world rather than on top of layer after layer of archaeology?
Tools for software in the large
When software becomes successful, software often becomes large. More features, more support systems, more infrastructure, more people, etc. Therefore, software “in the large” seems like a good problem to solve: how can you work at the same pace with one person, one application, and one server when you reach a hundred people, ten applications and a thousand servers?
Right now, I think the only tool we have for reducing the overhead of large systems and organizations is language. Frameworks and libraries can help make larger applications plausible, but they don’t resist the forces that make a large application hard to work with. Tooling and process, like source control or CI, only help us keep large software going; they don’t slow us down on the path to producing huge software.
Better programming languages make it possible to produce software that does more things in a more concise statement. To some extent better “API language” makes that possible too, but it relies on the power of the host language.
Am I overlooking other tools that make software in the large a manageable endeavor?
Less beautiful code, more code that works in production
Developers should care and feed for their systems, especially when they’re in production. Alerting, logging, and metrics are the tools you need. Further, you need to use actual production errors as a feedback loop to push you to write more fault tolerant code.
I’ve struggled with that part in the past. It’s tempting to think I can just code along, building beautiful code. Beautiful code is entirely different from production code. Handling errors, timeouts, partial failures, logging, instrumentation: it’s not exactly design pattern stuff. To wit:
Fault-tolerant code is ugly. It requires conditionals. It won’t look like code in programming books, screencasts, or blog entries. This is the way it has to be, I’m sorry.Production is all that matters. Highly intelligent words from Dave Copeland. The beauty of code, as it appears in my text editor, is not as important as how it works when it runs on servers. New mantra!
Uncertainty, feedback, confidence, a happy ending
Yesterday, I wanted to setup a quick feedback loop for writing some production code. But, I wasn’t entirely sure what that code should look like. Further, I hadn’t worked in the code base for several months, so I wasn’t sure how the new code would need to interact with other objects.
This is a scenario that often gets me in trouble. I end up spending too much time contemplating how things might be without the feedback of what’s possible and plausible.
What I ended up doing:
- Start a new test case for the code. I hardly know what it should do, so I ended up with
it "should do things"
. - Write the kind of code I would run in a REPL to get started writing the production code. This was a service client, so I did things like setup a connection, invoke a request, and handle a response.
- Write a test assertion about what I think the side-effect or result of the production code should be. For this service client, I wanted to make sure a useful thing was returned.
- Run the test, see it fail, learn some things about the REPL-y code I wrote. Since I'm writing Ruby, my feedback rate goes way up here. Running a single test, which I can do with a keystroke in my editor, is way faster than switching over to an IRB, reloading code, and finding the right incantation to run again.
- Iterate on the code in the test case until I have something that passes. Think, type, feedback. Observe, orient, decide, act. Coding is fun.
- Once I have passing code, start rearranging so its closer to what I want as actual production code. Replace dummy values with real ones. Figure out how to remove any cheats I may have put in place for expediency.
- Once I know what the messages are, extract them into helper methods on the test case class. Now I'm starting to do actual design. After each change, I run the tests. If I break the test, I go back and fix it before I proceed. This is Legitimate Refactoring. Feels good.
- Once I know what the objects are, I go ahead and create their file in the production code, move the code out of the test and into the production code, and make sure the original test still passes. Now I switch back to a more typical test-driven design cycle; write a test case, watch it fail, make it pass, refactor, repeat.
Give this a try some time. It’s a great confidence booster.
Use what you got
How Shopify scales Rails was one of my favorite talks at Big Ruby. Therein, John Duff talks through what Shopify’s Rails stack looks like and why it works for them. What impressed me most was that Shopify has been running basically the same stack since they started. Ruby, Rails, MySQL, Memcached. They added Redis in for queuing at one point.
I’m so easily tempted and fascinated by The New Shiny, it’s refreshing to read about a shop getting by with The Old Trusted instead. There’s a lot to be said for using solid, well-known tools for as much as you can and working with the strengths and weaknesses those tools afford you. It’s a lot closer to craftsmanship than the temptation of science fiction New Shiny tools, and less likely to turn your guts inside out when something goes sideways.
Entropy and anti-entropy on your codebase
Entropy and Evolution of a Codebase goes well with Your Application is on Fire:
If you imagine the modules in a codebase like cells in a Game of Life automaton, you could see cells fading from healthy blue to sickly red, then transmitting their disease elsewhere. Very occasionally, you’ll see a cell or cluster of cells brightening to health as a developer restructures that area. Mostly, the codebase will decay until it must be discarded or rewritten.Michael Nygard reflects on changes that increase and decrease the entropy of an application, what tends to introduce those changes, arrives at this wonderful metaphor for the thinking about the health of an application over time, and then offers several ways to manage entropy in a codebase. This is one that will stick in your head for a while.
If you haven’t read Michael Nygard’s Release It, you’re missing out on a lot of great stories and useful ideas on how to maintain production software.
Sandi's Rules
One day, Sandi Metz was pressed by a team she was working with to produce a simple set of rules that would lead to better code quality. After consideration, she came up with the following surprisingly numerical guidelines:
- classes can't grow beyond one hundred lines of code
- methods can't grow beyond five lines of code
- methods can only take four parameters (hash options count),
- controllers can only send one message to one object.
What emerges from these rules is a pretty pragmatic lens on how to practice object oriented design with Ruby and Rails without falling into the tarpit of radical approaches. You don't end up needing to worry about fast tests, decoupling from the framework, presenter, conductors, mediators, adapters, ad nauseum.
More critically, you aren't beset with decision fatigue. You don't have to survey the landscape of helper gems and bolt-on approaches to writing Rails applications. You can start with the way Rails wants you to write applications: logic and data encapsulated in models, behavior encapsulated in controller actions, easy sharing of data between actions and views with instance variables, etc.

You start with what Rails wants; when you find yourself violating one of Sandi's rules, then you apply design and refactoring techniques. If a model does too much, extract common behaviors into an encapsulated object. If a controller action knows about too many things, move that into an object and call it from the action. You can go a really long way using only the Replace Method with Method Object refactoring until you get to Kent Beck's notion of a Composed Method.
That bears repeating. In big letters.
Most Rails applications can be made better by relentless application of Method Object and Composed Method.
Thoughtbot has written about what this looks like as it emerges. It doesn't have to involve a lot of pattern names, new frameworks, framework asceticism or half-baked conventions. You write classes and methods when needed. When they get too big, you refactor, rinse, and repeat.

Sandi's book, Practical Object Oriented Design with Ruby, is great not because it covers new territory or casts a new light on the subjects of object oriented programming and Ruby. It's great because it shows how to work with the tools provided by OO and Ruby. It's great because it shows how to go from less-good code to more-good code.
It's really great because it's short and easy to read. This is the book that enthusiastic, "enlightened" programmers can hand to the less energetic to help them understand how to write better code. They can do so without worrying that the book is hard to understand or a beast to read. It's as close to a magic pill for clue enhancement as anyone has yet come.
Your application is on fire
Six easy pieces on thinking about sustainable code
Your application is on fire. Something is consuming it, a process converting fresh and shiny code into rigid, opaque legacy code.
Seriously, your applications is on fire. You should look into it.
Are you going to fight the fire? Will you keep throwing logs into it, hoping that you can corner it at some point down the line? Do you draw a line in the sand and start containing the fire and eventually suffocating it?
The fire is every change you make to your application in haste. The fire is every design decision that is bolted onto one part of the application without care for how it fits into another part of the application. The fire is every stopgap put in place to fix a short-term problem that isn't revisited once the short-term has elapsed.
Once time is not the most critical concern, you stop and think about how to make the application better in the long term. You consider what worked about the stop-gap solution, how to design your application with care, how to iterate on improving the architecture of your application.
You fight the fire by trading time for quality. Knowing how to solve your problems, knowing what is important about your application and what is consequential, you start to express a design. You write down principles that worked for you and write down an architecture that joins the principles and the design in the middle. That's where your application lives, once it's not on fire.
The reason green field software projects are appealing is that they aren't on fire. A green field is lush with possibility, without the constraint of decisions already made. More so, a green field is not bound by decisions implicitly accepted but never considered.
The unconsidered decision is how the fire starts. A class here, a subsystem there. Soon you've got a system but no design, emergent or otherwise. You've got a ball of mud, but the mud is on fire.
That fire is sustained by every time you bolt something else on. It's sustained by implementing the feature or fixing the bug without care for the program as a whole. It's fed by making progress without making care.
The fire is warm, but eventually it will consume your application. No one wants to work on the burnt out wrecks inside of it.
Everyone can smell it, but no one wants to go near it. Firewalls are erected, teams establish practices that allow them to avoid working on or near the fire.
So it burns on.
One day, you decide enough is enough. No more fuel on the fire. Every change to the application has to meet standards: does this make the code better, does this make the design better? This here is the design; adhere to the why's, use your creativity for the what's and how's.
That day is the day you start thinking for the long term. In the language of economics, its when you internalize external costs. You accept that the cost of maintaining this application is worthwhile, so you should mitigate that cost, optimizing input effort for output value.
You can't maximize cost by just getting the work done. You can't maximize value by building grand structures. You optimize cost for value by continuously making decisions that mitigate or reduce the fire.
Internal combustion engines are made of fire. Despite being powered by flame, combustion engines are pretty mundane things. They became mundane because they have a design and a standard of engineering that contains that fire, harnesses it, and turns it into a mechanism for converting fuel into energy.
When do we harness and focus our code, instead of letting it burn inefficiently and out of control? The moment we do is when we stop being firefighters and start being the wizards we set out to be when we started writing software.
Learning from a dropped refactoring
You don’t have to deploy every bit of code you write. In fact, it’s pretty healthy if you throw away some of the code you write before you even think about committing it.
Case in point: I sat down to refactor some Sifter code this weekend. I wanted to experiment with an idea about how Rails apps should be organized. I moved some code around, stepped back, and decided the idea behind my refactoring wasn’t sufficiently well cooked and stashed it instead of committing it.
I think that one can follow SOLID principles without radically decoupling your app from Rails and especially ActiveRecord. Instead, you let AR objects do what they’re good at: representing data. You push behavior, as much as possible, out into objects. What you should end up with is a handful of models that encapsulate the tables in your database and a bunch of classes that encapsulate the logic and behavior of your application; whether these classes live in app/models
or somewhere else is a matter of personal taste.
My goal is to extract an object that encapsulates user authentication behavior; a class that is mostly concerned with conditions and telling other objects to do things. I extracted a class focused on “remember me” session behavior. It looked something like this:
class UserAuthentication < Struct.new(:user)
def remember_token?
user.remember_expires_at.present? && (Time.now.utc < user.remember_expires_at)
end
def remember_me(offset=10.years)
time = offset.from_now.utc
user.touch_remember_token(time) unless remember_token?
end
def forget_me
user.clear_remember_token!
end
end
Then, for compatibility, I added this to the model:
def authentication
UserAuthentication.new(self)
end
delegate :remember_token?, :remember_me, :forget_me, to: :authentication
The principles I’m following are thus:
- Don’t expose AR’s API to collaborators. Therefore,
UserAuthentication
must call methods onUser
rather than directly update attributes and save records. - Encapsulate behavior in non-model classes. Therefore,
User
shouldn’t know when or how to manipulate “remember me” data, only expose an API that handles the mechanics.
UserAuthentication
has meaning, but doesn’t really own any behavior. It’s a bit envious of the User
model. That “envy” hints that its behavior really should live in the model.
Further, using delegate
means the User
model’s API surface area isn’t actually reduced. Granted, this is a stopgap measure. I should really hunt down all invocations of the delegated method and convert them to use UserAuthentication
instead.
At this point, it feels like my refactoring isn’t making forward progress. I’ve rearranged things a bit, but I don’t think I’ve made the code that much better.
As I alluded earlier, I decided to stash this refactoring for later contemplation. Maybe next time I should start from the callers of the APIs I want to refactor and drive the changes from there. Perhaps my conjecture about decoupling ActiveRecord data from application behavior needs tuning.
I beat the “ship lots of stuff!” drum a lot, but I’m happy with this result. Learning to strike the balance between shipping the first thing you type out and the thing your future self won’t hate is an undervalued skill. It takes practice, and that’s just what I did here. Plus, I parleyed it into a blog post. Everyone wins!
What I wish I'd known about rewrites
I can’t say enough good things about How to Survive a Ground-Up Rewrite Without Losing Your Sanity. Having been party to a few projects like this, a lot of this advice rings true to me. Let me quote you the good parts!

You must identify the business value of the rewrite:
The key to fixing the "developers will cry less" thing is to identify, specifically, what the current, crappy system is holding you back from doing. E.g. are you not able to pass a security audit? Does the website routinely fall over in a way that customers notice? Is there some sexy new feature you just can't add because the system is too hard to work with? Identifying that kind of specific problem both means you're talking about something observable by the rest of the business, and also that you're in a position to make smart tradeoffs when things blow up (as they will).
The danger of unicorn rewrites:
For the Unhappy Rewrite, the biz value wasn't perfectly clear. And, thus, as often happens in that case, everyone assumed that, in the bright, shiny world of the New System, all their own personal pet peeves would be addressed. The new system would be faster! It would scale better! The front end would be beautiful and clever and new! It would bring our customers coffee in bed and read them the paper.
Delivering value incrementally is of the greatest importance:
Over my career, I've come to place a really strong value on figuring out how to break big changes into small, safe, value-generating pieces. It's a sort of meta-design -- designing the process of gradual, safe change.
But “big bang” incremental delivery is accidental waterfall:
False incrementalism is breaking a large change up into a set of small steps, but where none of those steps generate any value on their own. E.g. you first write an entire new back end (but don't hook it up to anything), and then write an entire new front end (but don't launch it, because the back end doesn't have the legacy data yet), and then migrate all the legacy data. It's only after all of those steps are finished that you have anything of any value at all.
Always keep failure on the table:
If a 3-month rewrite is economically rational, but a 13-month one is a giant loss, you'll generate a lot value by realizing which of those two you're actually facing.
I really wish I’d thought of applying “The Shrink Ray”, an idea borrowed from Kellan Elliot-McCrea:
We have a pattern we call shrink ray. It's a graph of how much the old system is still in place. Most of these run as cron jobs that grep the codebase for a key signature. Sometimes usage is from wire monitoring of a component. Sometimes there are leaderboards. There is always a party when it goes to zero. A big party.
Engineer your migration scripts as idempotent, repeatable machines. You’re going to run them a lot:
Basically, treat your migration code as a first class citizen. It will save you a lot of time in the long run.
Finally, you should fear rewrites, but developing the skill to pull them off is huge:
I want to wrap up by flipping this all around -- if you learn to approach your rewrites with this kind of ferocious, incremental discipline, you can tackle incredibly hard problems without fear.
Whenever I talk to people with monolithic applications, slow-running test suites, and an urge to do something drastic, I want to mind-meld the ideas above into their brains. You can turn code around, but it takes time, patience, and a culture of relentless improvement and quality to make it happen.
Exemplary documentation: size and purpose
There’s a lot to say about programmer-focused software documentation. It’s more crucial than many developers think, so it is often neglected. Even when its not neglected, it’s often an after-thought. I’ve noticed there are three kinds of documentation I’m interested in.
When I first come across some software, I want short and focused examples of how I can use it for my own purposes. I’m not looking for a lot of theory or exposition; show me the benefit. If I can’t quickly see how the software works and makes my life easier, I’m very likely to discard it. In other words, I want shorter, “tweet-sized” documentation that sells me on the sizzle right away.

If I come back to some software, I often want to learn the whole thing in one sitting. I want a longer document that I can read through in a serial fashion to learn most or all of the concepts and details about using the code. It should cover the domain ideas of the software, the individual APIs, and how it all works together to make something. To continue the metaphor, I want a well-written, “Instapaper-length” document worthy of reading in a comfy chair.

After I start using something, I will often want to return to it to remember how to do specific things or to figure out if a task is possible at all. This is when I lean most on traditional API documentation. One to three paragraphs, easily searched are the ideal here. Kind of like the “Tumblr-post” of documentation.
I’ve yet to find all three of these qualities in the documentation for a single piece of software. Finding that software has done a really good job at one of them is delight enough. I can’t imagine how excited the world, at large, would be if something were to have all three. There would be a lot of rejoicing.
Web design for busy coders
Here it is: I'm somewhere between horribly afraid and way-too-smart to seriously attempt front-end web work. Browsers are not the software whose bugs I am interested in knowing about.
That said, putting information on the web that doesn't look like utter dross is a kind of required literacy in our field. While bravely dipping my toes back into the front-end waters, I recently found some great tricks. Rediscovered, probably, but I'm not sure where the idea originally came from.
Most important: design in greyscale. Color is hard and can lead to tinkering. My goal is to get in and out of the front-end bits quickly, so tinkering is the enemy. Greyscale is one dimensional, greatly simplifying matters. Give important information higher contrast and less important information (and "chrome") less contrast. Now you're done thinking about color.
Almost as important: use a fixed-with font. As a programmer, you look at them everyday, so it's a touchpoint of comfort. Pick a font you don't use in your editor all day, just so you can stare at something different for a while. Copy and paste a "font stack" from the aptly named fontstacks. Make important things big and unimportant things small. Now you're done thinking about type.
The key to avoiding browser dragons, it seems, is to skip horizontal layout, i.e. pull quotes, text wrapped around images, etc. It's pretty easy to use CSS if you only run things down the left side of the page. All the depth and despair of CSS is in trying to get things to appear off the left margin. Don't do that. Leave it to people who know how browsers work and how to manage their gnarly bugs. Now you're done thinking about layout.
It's tempting to think you should make your code examples look really nice. Don't worry about it; highlighting code is of marginal value. You'll never be satisfied with how it looks. The human mind is capable of reading code without a rainbow spectrum of colors. Spend time on writing about the code, not on polishing the colors and how its highlighted.
With all of those things out of the way, your way is clear to think about the really important things. What do you need to say, how do you structure the message, what do you leave out, how do you organize all the information? That's the essence of publishing on the web, not the accidental complexity of making things look interesting.
Hypermedia chicken, web browser egg
A lot of the hypermedia philosophy is centered around the idea that API clients should work a lot like web browsers and plain-old Hypertext Markup Language. They should follow hyperlinks, leverage media types, cache data when they can, and intelligently take advantage of the meaning of hyperlinks whenever possible.

The problem with that is, web browsers are way more capable than HTTP clients developers are using to build API clients with.
Here are some things web browsers have become pretty good at:
- GET and POST requests
- Following links and redirects
- Discovering data structures via HTML forms and submitting data using that schema
- Using headers to negotiate content types
- Caching data when possible and expiring those caches
- Handling streaming data and chunked responses
- A bunch of stuff I'm probably forgetting
Here are some things the HTTP client in your typical standard library are good at:
- GET, POST, PUT, and DELETE requests
- Following redirects (if you manage to set the right options)
What's at play here is a chicken and egg problem. Client developers can't build on hypermedia principles until they are working at the level of hypermedia abstractions. Arming them only with HTTP requests and the ability to choose their encoding and schema poison is too low level.
Protocols like HAL or Collection+JSON could light a path to solving this problem. Rather than dealing in pure data, services and consumers work with data annotated with hypermedia-like semantics for traversing data structures and creating new data. If these protocols can gain traction, it's "simply" a matter of getting HTTP clients into widespread use (read: standard libraries in stable releases of your-favorite-programming-language) that are as good at HTTP as web browsers are. At that point, API providers and API consumers could start using hypermedia principles to build APIs for those who aren't interested in the mechanics of hypermedia.
Until then, it seems to me that putting hypermedia principles front-and-center is suboptimal. It's akin to telling someone who wants to use your website that they need to understand MVC patterns first. It's only going to discourage and confuse them.