Developers should care and feed for their systems, especially when they’re in production. Alerting, logging, and metrics are the tools you need. Further, you need to use actual production errors as a feedback loop to push you to write more fault tolerant code.
I’ve struggled with that part in the past. It’s tempting to think I can just code along, building beautiful code. Beautiful code is entirely different from production code. Handling errors, timeouts, partial failures, logging, instrumentation: it’s not exactly design pattern stuff. To wit:
Fault-tolerant code is ugly. It requires conditionals. It won’t look like code in programming books, screencasts, or blog entries. This is the way it has to be, I’m sorry.
Production is all that matters. Highly intelligent words from Dave Copeland. The beauty of code, as it appears in my text editor, is not as important as how it works when it runs on servers. New mantra!
How Shopify scales Rails was one of my favorite talks at Big Ruby. Therein, John Duff talks through what Shopify’s Rails stack looks like and why it works for them. What impressed me most was that Shopify has been running basically the same stack since they started. Ruby, Rails, MySQL, Memcached. They added Redis in for queuing at one point.
I’m so easily tempted and fascinated by The New Shiny, it’s refreshing to read about a shop getting by with The Old Trusted instead. There’s a lot to be said for using solid, well-known tools for as much as you can and working with the strengths and weaknesses those tools afford you. It’s a lot closer to craftsmanship than the temptation of science fiction New Shiny tools, and less likely to turn your guts inside out when something goes sideways.
Entropy and Evolution of a Codebase goes well with Your Application is on Fire:
If you imagine the modules in a codebase like cells in a Game of Life automaton, you could see cells fading from healthy blue to sickly red, then transmitting their disease elsewhere. Very occasionally, you’ll see a cell or cluster of cells brightening to health as a developer restructures that area. Mostly, the codebase will decay until it must be discarded or rewritten.
Michael Nygard reflects on changes that increase and decrease the entropy of an application, what tends to introduce those changes, arrives at this wonderful metaphor for the thinking about the health of an application over time, and then offers several ways to manage entropy in a codebase. This is one that will stick in your head for a while.
If you haven’t read Michael Nygard’s Release It, you’re missing out on a lot of great stories and useful ideas on how to maintain production software.
Data point: a few of the infrastructure pieces out of Twitter have been implemented in low-level, heavy metal C and they’re optimizing on individual machines instead of architecture. Today, twitter/fatcache, a memcached-on-SSDs:
To understand why network connected SSD makes sense, it is important to understand the role distributed memory plays in large-scale web architecture. In recent years, terabyte-scale, distributed, in-memory caches have become a fundamental building block of any web architecture. In-memory indexes, hash tables, key-value stores and caches are increasingly incorporated for scaling throughput and reducing latency of persistent storage systems. However, power consumption, operational complexity and single node DRAM cost make horizontally scaling this architecture challenging. The current cost of DRAM per server increases dramatically beyond approximately 150 GB, and power cost scales similarly as DRAM density increases.
It’s fascinating to observe Twitter’s architectural growth from the outside. They quickly exceeded the capacity of typical MySQL setups, then of Ruby and Rails, then memcached alone. They’ve got distributed filesystems, streaming distributed processing pipelines, and distributed databases. Now they’re optimizing down to the utilization of their hardware, taking advantage of the memory-like latencies of SSDs. When you start caring about power and the size of your index entries, you’ve reached a whole new level of Maslow’s hierarchy of scaling.
If trends continue and Twitter is a leader in how large-scale distributed systems are implemented, watch out. Twitter led many of us to Scala, ZooKeeper, and their own inventions like Storm and Finagle. Gird your programming and scaling fashion loins, because you’re about to learn a lot more about
ERRNO, and processor architecture than you ever wanted to know!
The Web We Lost, on the web of ad-hoc, bottom-up social networks before the pendulum swung fully towards centralized networks like MySpace, then Friendster, and now Facebook, Twitter, and friends. I’m glad Anil Dash is pointing out that great things were happening before social networks were massively financed operations and the delightful things that were different when people ran the system from the bottom up.
Owning and operating your data is obviously better than letting someone trade on it. But, there are missing pieces for users:
- Where do I host my corner of the social network? Putting content on the web without someone else to run it is still strictly nerd stuff.
- How do I find my friends? The advantage of a centralized network is its easy to make global observations, like analyzing social graphs for recommended links.
- What are the checks against bad actors? Comments and trackbacks were fantastic for weblogs, until spammers figured out how to turn them into toys for boosting pagerank.
I don’t think any of these are insurmountable. But, decentralization is hard! Can we pull it off? I’d love to see it happen.
We’ve had Sifter’s repo hooked up to Code Climate for a couple months now and I’m really loving it. Garrett and I have both found it fun to kill duplication or refactor away complex code. A decent test suite enables this, but Code Climate is very much the compass that points you right to the trouble spots. The Code Climate blog is a great read too, consistently featuring thought-provoking ideas on how to think about making better code.
I love tools like flog and flay for quick smell detection. If you are like me and too lazy to configure CI and code metrics, Code Climate’s easy setup and awesome trending are well worth a look.
Try thinking about living and creating a little differently today. Advice for beginners: push through the shortcomings of your early work until your ability catches up with your taste. Slow down, lead life at a slower pace every now and then, it’s good for you. Stop telling us how much everything sucks; not everyone makes the same decisions and trade-offs you would when they create something.
How to be good at anything. In short: do it, get feedback, study how to improve, repeat.
Something I’ve found, through crossfit, is that if I have any strong suit it’s not quitting. Seems trite now that I write it, but it occasionally helps to state the obvious.
Once I’ve decided an activity is worthwhile, I’m pretty good at sticking with it no matter how silly I feel doing it. I’m not the strongest, the fastest, or the best eater. I’m not the smartest, the funniest, or the most charming. But I’ve made progress in life by showing up every day and trying to do a little better than the previous day.
The bottom line: pick a few things to do well, do them, and don’t quit.
What is 'better' code? Dave Copeland on the qualities readable, changeable code exhibits. Of the attributes he identifies, I think number of paths (ABC complexity) is the most important for reading code and fan-in/out is the most important thing to manage for easily changed code.
Petrushka chord. Two major chords played a half-tone apart. So, it sounds good, except it sounds grating. It’s a motif throughout Stravinsky’s ballet Petrouchka. Ergo, like everything Stravinsky, get in my ears! Listen to it and learn more about the chord from the awesome “Feynman of classical music” (I just made that up) Leonard Bernstein.