Link
Less beautiful code, more code that works in production
Developers should care and feed for their systems, especially when they’re in production. Alerting, logging, and metrics are the tools you need. Further, you need to use actual production errors as a feedback loop to push you to write more fault tolerant code.
I’ve struggled with that part in the past. It’s tempting to think I can just code along, building beautiful code. Beautiful code is entirely different from production code. Handling errors, timeouts, partial failures, logging, instrumentation: it’s not exactly design pattern stuff. To wit:
Fault-tolerant code is ugly. It requires conditionals. It won’t look like code in programming books, screencasts, or blog entries. This is the way it has to be, I’m sorry.Production is all that matters. Highly intelligent words from Dave Copeland. The beauty of code, as it appears in my text editor, is not as important as how it works when it runs on servers. New mantra!
Use what you got
How Shopify scales Rails was one of my favorite talks at Big Ruby. Therein, John Duff talks through what Shopify’s Rails stack looks like and why it works for them. What impressed me most was that Shopify has been running basically the same stack since they started. Ruby, Rails, MySQL, Memcached. They added Redis in for queuing at one point.
I’m so easily tempted and fascinated by The New Shiny, it’s refreshing to read about a shop getting by with The Old Trusted instead. There’s a lot to be said for using solid, well-known tools for as much as you can and working with the strengths and weaknesses those tools afford you. It’s a lot closer to craftsmanship than the temptation of science fiction New Shiny tools, and less likely to turn your guts inside out when something goes sideways.
Entropy and anti-entropy on your codebase
Entropy and Evolution of a Codebase goes well with Your Application is on Fire:
If you imagine the modules in a codebase like cells in a Game of Life automaton, you could see cells fading from healthy blue to sickly red, then transmitting their disease elsewhere. Very occasionally, you’ll see a cell or cluster of cells brightening to health as a developer restructures that area. Mostly, the codebase will decay until it must be discarded or rewritten.Michael Nygard reflects on changes that increase and decrease the entropy of an application, what tends to introduce those changes, arrives at this wonderful metaphor for the thinking about the health of an application over time, and then offers several ways to manage entropy in a codebase. This is one that will stick in your head for a while.
If you haven’t read Michael Nygard’s Release It, you’re missing out on a lot of great stories and useful ideas on how to maintain production software.
Twitter's optimizations
Data point: a few of the infrastructure pieces out of Twitter have been implemented in low-level, heavy metal C and they’re optimizing on individual machines instead of architecture. Today, twitter/fatcache, a memcached-on-SSDs:
To understand why network connected SSD makes sense, it is important to understand the role distributed memory plays in large-scale web architecture. In recent years, terabyte-scale, distributed, in-memory caches have become a fundamental building block of any web architecture. In-memory indexes, hash tables, key-value stores and caches are increasingly incorporated for scaling throughput and reducing latency of persistent storage systems. However, power consumption, operational complexity and single node DRAM cost make horizontally scaling this architecture challenging. The current cost of DRAM per server increases dramatically beyond approximately 150 GB, and power cost scales similarly as DRAM density increases.
It’s fascinating to observe Twitter’s architectural growth from the outside. They quickly exceeded the capacity of typical MySQL setups, then of Ruby and Rails, then memcached alone. They’ve got distributed filesystems, streaming distributed processing pipelines, and distributed databases. Now they’re optimizing down to the utilization of their hardware, taking advantage of the memory-like latencies of SSDs. When you start caring about power and the size of your index entries, you’ve reached a whole new level of Maslow’s hierarchy of scaling.
If trends continue and Twitter is a leader in how large-scale distributed systems are implemented, watch out. Twitter led many of us to Scala, ZooKeeper, and their own inventions like Storm and Finagle. Gird your programming and scaling fashion loins, because you’re about to learn a lot more about malloc, ERRNO, and processor architecture than you ever wanted to know!
Wherein I heart Code Climate
We’ve had Sifter’s repo hooked up to Code Climate for a couple months now and I’m really loving it. Garrett and I have both found it fun to kill duplication or refactor away complex code. A decent test suite enables this, but Code Climate is very much the compass that points you right to the trouble spots. The Code Climate blog is a great read too, consistently featuring thought-provoking ideas on how to think about making better code.
I love tools like flog and flay for quick smell detection. If you are like me and too lazy to configure CI and code metrics, Code Climate’s easy setup and awesome trending are well worth a look.
Hit it, don't quit
How to be good at anything. In short: do it, get feedback, study how to improve, repeat.
Something I’ve found, through crossfit, is that if I have any strong suit it’s not quitting. Seems trite now that I write it, but it occasionally helps to state the obvious.
Once I’ve decided an activity is worthwhile, I’m pretty good at sticking with it no matter how silly I feel doing it. I’m not the strongest, the fastest, or the best eater. I’m not the smartest, the funniest, or the most charming. But I’ve made progress in life by showing up every day and trying to do a little better than the previous day.
The bottom line: pick a few things to do well, do them, and don’t quit.
The qualities of better code
What is 'better' code? Dave Copeland on the qualities readable, changeable code exhibits. Of the attributes he identifies, I think number of paths (ABC complexity) is the most important for reading code and fan-in/out is the most important thing to manage for easily changed code.
Get in my ears, you dissonant chord
Petrushka chord. Two major chords played a half-tone apart. So, it sounds good, except it sounds grating. It's a motif throughout Stravinsky's ballet Petrouchka. Ergo, like everything Stravinsky, get in my ears! Listen to it and learn more about the chord from the awesome "Feynman of classical music" (I just made that up) Leonard Bernstein.
Working with Ruby's GVL
Visualising the Ruby Global VM Lock. A nice commit-by-commit look at how extensions for Ruby 1.9 work with the GVL, what that looks like as tests run, and how to release the GVL to allow for better parallelism.
How Ruby IO is formed
Ruby's IO Buffering And You! Jesse Storimer screencasts his way through what happens when you read and write to files and sockets in Ruby, explaining the behavior and spelunking through Rubinius' implementation of IO. You'll learn stuff. If you want to learn even more stuff, check out Jesse's new book Working with TCP Sockets. Jesse is fantastic at describing Unixy things concisely; you'll like it.
A handful of useful project mantras
You could do a lot worse than following the heuristics set out by this Software Architecture cheat sheet. The tip I need to follow more often is "Is There Another Way"; I frequently get way too caught up in my first idea, which is usually too simplistic or requires too much architecture. The tip I often try to guide people towards is "What If I Didn't Have This Problem?"; routing around problems or trying to reduce them to problems that require less code is a super-powerful judo chop.
bitly's nsq has some good ideas
NSQ is a realtime message processing system designed to operate at bitly's scale, handling billions of messages per day.It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.
No SPOFs and reliable message delivery, without relying on something like ZooKeeper, is a big claim. They have some novel approaches to these problems.
First, they run an intermediary daemon, nsqlookupd, between the producers/consumers and the actual queues. These daemons monitor all the available queue servers and tell the clients what to connect to. No configuration of actual queue servers is known to applications. They then run multiple lookup daemons, which are stateless and don’t need to agree with each other in order for the system to operate properly.
Reliable message delivery is provided with at-least-once message delivery semantics. They require all consumers to de-duplicate messages or restrict their operations to idempotent operations. Not exactly legacy friendly, as many applications are coded with the assumption of a closed, one-shot world. But. Idempotence: I highly recommend it if you have the means.
If you need to prevent losing messages due to the FBI stealing your servers, which is something you definitely need to account for, you can set up redundant pairs of servers and rely on deduplication/idempotence to make sure you’re only processing messages once, even if you consume them multiple times.
In summary: lots of good ideas here. Perhaps some of them could be applied to how people are using Resque?
When to Sinatra, when to Rails
On Rails, Sinatra, and picking the right tool for the job. Pedro Belo, of Heroku fame, finds Rails is way better for pure-web apps and Sinatra is way better for pure-API apps. Most of it comes down to Rails has better tooling and Sinatra is better for scratching itches, which happens a lot more in APIs than applications. I’m not ready to pronounce this the final word, but what he’s saying lines up with much of my experience.
That said, you can get pretty far with a Rails API by segregating it from your application. That is, your app controllers inherit from ApplicationController and your API controllers inherit from ApiController. This keeps the often wildly different needs of applications and APIs nice and distinct.
Common sense code checks
Etsy’s Static Analysis for PHP. This isn’t as complicated as you might think. While Facebook’s HipHop is used, and is quite sophisticated, a lot of this is just common sense. Trigger code reviews when oft-misused functions are used or when functions that involve security things are introduced.
This stuff is great for an intern or new team member to get a quick win with. So next time you bring someone onto your team, why not turn them loose on these kinds of quick, big wins?
Hello, you beautiful fixed-width font
Pitch. Not quite a programmer’s font, but holy cow is it gorgeous.
I love the thought put into this type; the creator actually tried to recreate the artifacts of type created by physically striking paper. Turned out that took away from the font, but it’s delightful that he went that deep in considering what a fixed-width font should feel like.
The history of fixed-width, typewriter-esque fonts is fantastic too. Even if you’re not typography-curious like myself, you should read the whole thing and not just look at the fantastic specimens.
Know a feedback loop
TDD is one way to create a feedback loop for building your application. Spiking code out and then stabilizing it is another:
For most people, TDD is a mechanism for discovery and learning. For some of us, if we can write an example in our heads, our biggest areas of learning probably lie elsewhere. Since ignorance is the constraint in our system, and we’re not ignorant about much that TDD can teach us, skipping TDD allows us to go faster. This isn’t true of everything. Occasionally the feedback is on some complex piece of business logic. Every time I’ve tried to do that without TDD it’s stung me, so I’m getting better at working out when to do it, and when it’s OK to skip it.
TDD helps me a lot when I have an idea what the problem looks like. Spiking out a prototype and backfilling tests helps me when I don’t know what the problem looks like.
You’re possibly different in how you approach problems. If you’re flying more by the seat of your pants, or you aren’t including the composition and organization of the code in your feedback loop, I will probably insist you work on something that isn’t in the core layers of the application. That’s cool though; as long as you have any feedback loop that will nudge you towards better discovering and solving the core problem, we’re cool.
The test-driven astronaut
Don't Make Your Code "More Testable", make the design of your program better. Snappy test suites are all the vogue, but that misses the point of even writing tests: create a feedback loop to know when your program works and when your program is organized well. Listen carefully to the whispers in your code; if you're spending all your time writing tests or shuffling code instead of adding features, improving features, or shipping features then you're falling to the siren song of the test-driven astronaut.