Developers should care and feed for their systems, especially when they’re in production. Alerting, logging, and metrics are the tools you need. Further, you need to use actual production errors as a feedback loop to push you to write more fault tolerant code.

I’ve struggled with that part in the past. It’s tempting to think I can just code along, building beautiful code. Beautiful code is entirely different from production code. Handling errors, timeouts, partial failures, logging, instrumentation: it’s not exactly design pattern stuff. To wit:

Fault-tolerant code is ugly. It requires conditionals. It won’t look like code in programming books, screencasts, or blog entries. This is the way it has to be, I’m sorry.

Production is all that matters. Highly intelligent words from Dave Copeland. The beauty of code, as it appears in my text editor, is not as important as how it works when it runs on servers. New mantra!

How Shopify scales Rails was one of my favorite talks at Big Ruby. Therein, John Duff talks through what Shopify’s Rails stack looks like and why it works for them. What impressed me most was that Shopify has been running basically the same stack since they started. Ruby, Rails, MySQL, Memcached. They added Redis in for queuing at one point.

I’m so easily tempted and fascinated by The New Shiny, it’s refreshing to read about a shop getting by with The Old Trusted instead. There’s a lot to be said for using solid, well-known tools for as much as you can and working with the strengths and weaknesses those tools afford you. It’s a lot closer to craftsmanship than the temptation of science fiction New Shiny tools, and less likely to turn your guts inside out when something goes sideways.

Entropy and Evolution of a Codebase goes well with Your Application is on Fire:

If you imagine the modules in a codebase like cells in a Game of Life automaton, you could see cells fading from healthy blue to sickly red, then transmitting their disease elsewhere. Very occasionally, you’ll see a cell or cluster of cells brightening to health as a developer restructures that area. Mostly, the codebase will decay until it must be discarded or rewritten.

Michael Nygard reflects on changes that increase and decrease the entropy of an application, what tends to introduce those changes, arrives at this wonderful metaphor for the thinking about the health of an application over time, and then offers several ways to manage entropy in a codebase. This is one that will stick in your head for a while.

If you haven’t read Michael Nygard’s Release It, you’re missing out on a lot of great stories and useful ideas on how to maintain production software.

Data point: a few of the infrastructure pieces out of Twitter have been implemented in low-level, heavy metal C and they’re optimizing on individual machines instead of architecture. Today, twitter/fatcache, a memcached-on-SSDs:

To understand why network connected SSD makes sense, it is important to understand the role distributed memory plays in large-scale web architecture. In recent years, terabyte-scale, distributed, in-memory caches have become a fundamental building block of any web architecture. In-memory indexes, hash tables, key-value stores and caches are increasingly incorporated for scaling throughput and reducing latency of persistent storage systems. However, power consumption, operational complexity and single node DRAM cost make horizontally scaling this architecture challenging. The current cost of DRAM per server increases dramatically beyond approximately 150 GB, and power cost scales similarly as DRAM density increases.

It’s fascinating to observe Twitter’s architectural growth from the outside. They quickly exceeded the capacity of typical MySQL setups, then of Ruby and Rails, then memcached alone. They’ve got distributed filesystems, streaming distributed processing pipelines, and distributed databases. Now they’re optimizing down to the utilization of their hardware, taking advantage of the memory-like latencies of SSDs. When you start caring about power and the size of your index entries, you’ve reached a whole new level of Maslow’s hierarchy of scaling.

If trends continue and Twitter is a leader in how large-scale distributed systems are implemented, watch out. Twitter led many of us to Scala, ZooKeeper, and their own inventions like Storm and Finagle. Gird your programming and scaling fashion loins, because you’re about to learn a lot more about malloc, ERRNO, and processor architecture than you ever wanted to know!

We’ve had Sifter’s repo hooked up to Code Climate for a couple months now and I’m really loving it. Garrett and I have both found it fun to kill duplication or refactor away complex code. A decent test suite enables this, but Code Climate is very much the compass that points you right to the trouble spots. The Code Climate blog is a great read too, consistently featuring thought-provoking ideas on how to think about making better code.

I love tools like flog and flay for quick smell detection. If you are like me and too lazy to configure CI and code metrics, Code Climate’s easy setup and awesome trending are well worth a look.