Adam Keys is Thinking

Architecting for change. Complex systems and change:

1. Distributed systems are inherently complex.
2. The outcome of change in complex systems is hard to predict.
3. The outcome of small, frequent, measurable changes are easier to predict, easier to recover from, and promote learning.

I’d have thought all the useful things to say about Etsy were said, at this point, but I’d have thought wrong!

There’s a good saying about designing distributed systems that goes something like “avoid it as long as possible”. I think these three guidelines are worth adding to that saying. Iterate, examine, repeat. Don’t make big, tricky changes. In fact, large change you can’t recover from are nearly impossible to make anyway, so route around them entirely.

The last bit, “promote learning”, is great too. I follow distributed systems and database designers on Twitter and see tons of great papers and ideas in the exchange. More than that, always teach your teammates about the distributed systems you’re building. The more they know about the design and constraints of the system you’re making, the easier it is for them to work with those systems. If you can’t teach someone to use your system, you probably don’t understand it well enough.

Read up on Thread and Queue and ready for more multi-threaded Ruby reading? Aaron Patterson has written up how the thread safe option works in Rails and some tradeoffs involved in removing the option and making thread safety the default. It’s not as complicated as you might think!

The only rub I can see is that, as far as I can tell, he’s talking about making this the default for production mode. Making it the default for development mode isn’t tenable if you want to keep class reloading around, which almost everyone does. It’s just a hunch, but running without thread-safety in development seems weird when its the default in production. But, some teams run YARV in development and JRuby in production, so maybe I’m just making up things to worry about.

Zach Holman is doing some interesting code anthropology on the Ruby community. Consider Aggressively Probing Ruby Projects:

Hopper is a Sinatra app designed to pull down tens of thousands of Ruby projects from GitHub, snapshot each repository into ten equidistant revisions, run them through a battery of tests (which we call Probes), and hopefully come up with some deeply moving insights about how we write Ruby.

There are plenty of code metric gizmos out there. At a glance, Hopper takes a few nice steps over extant projects. Unlike previous tools, it has a clear design, an obvious extension mechanism, and the analysis tools are distinct from the reporting tools. Further, it’s designed to run out-in-the-open, on existing open source projects. This makes it immediately useful and gives it a ton of data to work with.

For entertainment, here’s some information collected on some stuff I worked on at Gowalla: Chronologic and Audit.

Do you miss the ability to take a bunch of paper, books, and writing utensils and spread them out over a huge desk or table? Me too!

Light Table is based on a very simple idea: we need a real work surface to code on, not just an editor and a project explorer. We need to be able to move things around, keep clutter down, and bring information to the foreground in the places we need it most.

This project is fantastic. It’s taking a page from the Smalltalk environments of yore, cross-referencing that with Bret Victor’s ideas on workspace interactivity. The result is a kick-in-the-pants to almost every developer’s current workflow.

There’s a lot to think about here. A lot of people focus on making their workflow faster, but what about a workspace that makes it easier to think? There’s a lot of room to design a better workspace, even if you’re not going as far as Light Table does.

There’s a project on Kickstarter to fund further development of Light Table. If you write software, it’s likely in your interest to chip in.

Some nice people at UserVoice took the time to jot down how they manage their product. Amongst the lessons learned:

Have a set amount of time per week that will be spent on bugs

We have roughly achieved this by setting a limit on the number of bugs we’ll accept into Next Up per week. This was a bit contentious at first but has resolved a lot of strife about whether a bug is worthy. The customer team is now empowered (or burdened) with choice of choosing which cards will move on. It’s the product development version of the Hunger Games.

This, to me, is an interesting juxtaposition. Normally, I think of bugs as things that should all be fixed, eventually. Putting some scarcity of labor into them is a great idea. Fixing bugs is great, until it negatively affects morale. Better to address the most critical and pressing bugs and then move the product ball forward. A mechanism to limit the number of bugs to fix, plus the feedback loop of recognizing those who fix bugs in an iteration (they mention this elsewhere in the article), is a great idea.

11 Resources for Learning Unix Programming:

I tend to steer clear of the thick reference books and go instead for books that give me a look into how smart people think about programming.

I have a soft spot in my heart for books that are way too long. But Jesse’s on to something, I think. The problem with big Unix books is that they are tomes of arcane rites; most of it just isn’t relevant to those building systems on a modern Unix (Linux) with modern tools (Java, Python, Ruby, etc.).

Jesse’s way of learning you a Unix is way better, honestly. Read concise programs, cross-reference them with manual pages. Try writing your own stuff. Rinse. Repeat.

Today it's Node. Before it was Rails. Before it was PHP. Before it was Java. Cogs Bad:

There’s a whole mindset - a modern movement - that solves things in terms of working out how to link together a constellation of different utility components like noSQL databases, frontends, load balancers, various scripts and glue and so on.  Its not that one tool fits all; its that they want to use all the shiny new tools.  And this is held up as good architecture!  Good separation.  Good scaling.

I’ve fallen victim to this mindset. Make everything a Rails app, solve all the problems with Ruby, store all the data in distributed databases! It’s not that any of these technologies are wrong it’s just they might not yet be right for the problem at hand.

You can almost age generations of programmers like tree rings. I’m of the PHP/Rails generation, though I started in the Linux generation. A few years ago, I thought I could school the Java generations. But it turns out, I’ve learned a ton from them, even when I was a bit of a hubristic brat. The Node generation of developers will teach me more still, both in finding new virtuous paths and in going down false paths so I don’t have to follow them.

That said, it would be delightful if there was a shortcut to get these new generations past the “re-invent all the things!” phase and straight into the “make useful things and constructive dialog about how they’re better” phase.

Zach Holman's slides on patterns GitHub uses to scale their team Ruby Patterns from GitHub's Codebase:

Your company is going to have tons of success, which means you'll have to hire tons of people.

My favorites:

• Every project gets a script/bootstrap for fetching dependencies, putting data in place, and getting new people ready to go ASAP. This script comes in handy for CI too.
• Try new techniques by deploying it only to team members at first. The example here was auto-escaping markup. They started with this only enabled for staff, instead of turning it on for everyone and feeling the hurt.
• Build projects within projects. Inevitably areas of functionality start to get so complex or generic that they want to be their own thing. Start by partitioning these things into lib/some_project, document it with a read me in lib/some_project and put the tests in test/some_project. If you need to share it across apps or scale it differently someday, you can pull those folders out and there you go.
• Write internal, concise API docs with TomDoc. Most things only need 1-3 lines of prose to explain what’s going on. Don’t worry about generating browse-able docs, just look in the code. I heart TomDoc so much.

These ideas really aren’t about patterns, or scaling the volume of traffic your business can handle. They’re about scaling the size of your team and getting more effectiveness out of every person you add.

Noel Rappin encourages all of us to use our development tools efficiently. If your editor or workflow aren’t working for you, get a new tool and learn to use it.

I’ve been working with another principle lately: minimize moving parts. I used to spend time setting up tools like autotest, guard, or spork. But it ended up that I spent too much time tweaking them or, even worse, figuring out exactly how they were working.

I’ve since adopted a much simpler workflow. Just a terminal, a text editor, and some scripts/functions/aliases/etc. for running the stuff I do all the time. I take note when I’m doing something repeatedly and figure out how I can automate it. Besides that, I don’t spend much time thinking about my tools. I spend time thinking about the problem in front of me. It makes a lot of sense, when you think about it.

I say you should “own” your tools and minimize moving parts because you should understand how they all work together and how they might change the behavior of your code. If you don’t own your tools in this way, you’ll end up wasting time debugging someone else’s code, i.e. a misbehaving tool. That’s just a waste of time; when you come across a tool that offends in this way, put aside a time block to fix it, or discard it outright.

An API Ontology: if you were curious about what the difference between an RPC, SOAP, REST, and Hypermedia API are, but were afraid to ask. In my opinion, this is not prescription; I don't think there's anything inherently wrong with using any of these, except SOAP. Sometimes an RPC or a simple GET is all you need.