Consider:

1. Google's intricate and massive data center operations, wherein Google is not only leading the pack in building distributed computing and database infrastructure, but building massive operations to run those systems.

<li>"People who are really serious about software should make their own hardware." Alan Kay, creator of Smalltalk, object-oriented programming, and many other things that are good in your software development life, said that.</li>

I think Alan Kay’s quote could be rewritten for modern times to say, “Those who are really serious about large applications should make their own datacenters”. Assembling hardware, putting it into racks, putting the racks into data centers, and building out your own data centers are the analog of building your own hardware for software service companies these days.

You probably don’t need expert-level knowledge of these disciplines to write software today (e.g. I know diddly-squat about how packets find their way through the modern internet). You will, however, have a leg up if you’re aware of the possibilities and know how to take advantage of them. Even if you’re building applications with modest capacity needs, knowing how to set up a failover database and when to pay for physical hardware instead of virtualized hosts is a thing that will make your customers and clients happy.

More concisely: when it comes to running your software on hardware, one size does not fit all; know how and when to tailor your application to the hardware, no matter what size your application wears.

You have to invent the right thing. Some things you might invent:

1. A solution to a problem. Nothing novel, just an answer for a question. Eg. any Rails/Django/etc. application.

2. An application of some existing techonologies in a novel way. Eg. integrating a library to avoid inventing your own solution.

3. An incrementally better, specific kind of mouse trap. Eg. building on top of existing infrastructure to solve a problem better than any existing solutions.

4. An entirely new kind of mousetrap. Eg. building wholly new infrastructure because you face a high quality, unique problem that you are imminently required to solve.

Inventing the wrong thing means you’re operating at the wrong level. If you’re too high, you’re spinning your wheels on problems you hope to have. If you’re too low, you’re spinning your wheels on building something that isn’t sufficient to solve your problems. If you’re at the right level, you’re mostly solving problems you actually face and not solving too my coincidental problems.

This doesn’t mean new problems shouldn’t be tackled and new techonologies should not be invented. It applies mostly to reinventing wheels. That is, a project starts with level 1, not level 3 or 4. Apply a technology and improve it before you push the edge. In fact, you must push the limits of an extant technology before level 4 is the right answer. No skipping allowed.

Don’t let imposter syndrome lead you to the wrong technology decision. I’ve tried to build at the wrong level in the past because I felt like I had to fit in with the level of what others working on larger systems were building. Wrong answer.

It’s OK to build a scooter instead of a spaceship if all you need to do is go pick up the mail.

Remote teams are hard. Not impossible hard, but running uphill hard. It’s hard because people are used to interacting face-to-face. Given the opportunity, they’ll interact with those around them rather than those in virtual spaces.

The trick, I think, is to make a better shared space for a remote/local team than the physically shared space they already have. A space that is just as fluid, fun, and useful as a physical space and available anytime, everywhere is more compelling because it affords its occupants (aka team members) more hours in their day (no commuting, flexible hours) and permits all sorts of non-traditional work locations (coffee shops, trains, sofas at home, a summer trip to Europe).

Decoupling work from location and time is a big deal. I hope more companies, in software and outside of it, attempt to solve it.

One black-and-white perspective on building software is that part of it is about mechanics and part of it is about science. The mechanics part is about wiring things up, composing smaller solutions into bigger ones, and solving any problems that arise in the process. The science part is taking problems that don’t fit well into the existing mechanisms and making a new mechanism that identifies and solves all the right puzzles.

You could look at visual and interaction design in the same way. The mechanical part is about using the available assets and mechanisms to create a visual, interactive experience on screens that humans interact with. The science is about solving a problem using ideas that people already understand or creating an idea that teaches people how to solve a problem.

The mechanical case is about knowing tools, when to use them, and how they interact with each other. The scientific case is about holding lots of state and puzzle in your head and thinking about how computers or people will interact with the system.

I’ve observed that people end up all long the spectrum. Some specialize on mechanics, others on science. The rare case that can work adeptly on both sides, even if they’re not the best at either discipline, is really fun to watch.

We nerds are trained from an early age to argue on the internet, hone our logical skills, and engage with people based on data instead of empathy.

twitter.com/gotascii/…

It’s so hard to divorce reason, emotion, and making progress on a project. Letting a logical inconsistency go is harder than forcing someone to see the flaw in their reasoning. Getting angry or worked-up feels more powerful than a supportive attitude. There are so many disasters to avoid, it’s hard to not to force everyone to listen to all the things you’ve been burned by previously and how you want to avoid them at all costs.

Take a deep breath. Fire up your empathy muscles. Figure out how to say “yes” to the work of your teammates while using your experience to guide that work to an even better place. This is what they call “constructive teamwork”.

The year is winding down, and its time to reflect on the 2011 that was. The year took me into the dark abyss of the American housing market and back out the other end. Somehow I ended up in Austin, a little lighter in the pocket-book to show for it. It saw the most exciting, and ultimate, year of that which was Gowalla. I got in pretty good shape, and then got into pretty mediocre shape. I read a lot, coded a lot, wrote a bit, and learned a lot about everything. ‘Twas a tough year, but all’s well that ends well, or so they say.

Things I read

The most interesting fiction I read this year was Neuromancer. This one was probably more jaw dropping before The Matrix came out, but it was still interesting. That said, it reads like the Cliffs Notes version of Neal Stephenson.

The best non-fiction I read was Godël’s Proof. It’s a short, clear explanation of his approach to computability. Even if you’re medicore at math and proofs, like myself, this one will stick.

The best technical book I consumed this year was Smalltalk Best Practice Patterns. First off, I love Kent Beck’s concise but powerful writing style. Second off, this book is like discovering that someone wrote down a really good theory of the elements of software decades ago and no one told you about them. Third, you should get a copy of this book, look for the used ones.

Things I made this year

I piled a lot of the things I learned about infrastructure and shipping software at Gowalla into Mixing a Persistence Cocktail. How to think about scaling, how to ship incrementally, overcoming THE FEAR. It’s all there.

I large chunk of my time at work on Chronologic. I presented on it too. Then I open sourced it. We deployed it at Gowalla and it held up, but not without some rough spots. I presented on those too.

I did a lot of open source tinkering this year. A lot of it is half-baked, but at least it’s out there. That was a major personal goal for the year, so I’m glad I at least stuck my neck out there, even if I’m not rolling in kudos. Yet!

Things I wrote this year

Modulo a summer lull, I ended up doing a good bit of writing this year. The crowd favorites were Why metaprogram when you can program?, The Current and Future Ruby Platform Cassandra at Gowalla, and Your Frienemy, the ORM. My personal favorites were The ear is connected to the brain, Post-hoc career advice for twenty-something Adam, and How to listen to Stravinsky’s Rite of Spring.

Of course, working at Gowalla this year was quite the ride. I wrote about that too, sometimes rather obliquely. Relentless shipping, The pitfalls of growing a team, The guy doing the typing makes the call, Skip the hyperbole, Sleep is the best, and Don’t complain, make things better were all borne of things I learned over the course of the year.

If I had to pithily summarize the year, I’d tie it together under change. Change is good, challenging, frustrating, and inevitable. Better to change than not, though!

I’ve said all this stuff before, but I came across some nice writing that highlights people doing it. I’m repeating it because it’s important stuff.

Step one, get on that grind. Making things is about consistently making progress. Consistently making progress is about showing up every day and moving the ball forward. Progress can take different forms, and sometimes won’t even feel like progress at all. The crux of the biscuit is to make the time to do the things that need doing in order to produce the thing you’re excited about making.

Questlove, band leader of The Roots and pretty much my favorite music nerd of all time, spends most of his waking hours thinking about, rehearsing, or performing his music. A typical day for him is 11 AM - 7 PM at 30 Rock rehearsing for Late Night with Jimmy Fallon or writing new music, 8 PM - 2 AM spent performing or DJing, and late nights winding down by studying their performance from that day’s show or doing some crate digging (cool kid speak for listening to obscure stuff in your record collection).

Step two, simplify. You just can’t devote the mental energy to awesome stuff if your brain is going in multiple directions. Close as many of the social medias, chats, emails, and alarm klaxons as possible. If you’re an organized person, clear your workspace; if you’re a clutter person, just roll with your clutter[1]. And, of course, think critically about what you’re consuming and using. If a tool, book, TV show, or application isn’t pulling its weight helping you do or think awesome things, show it the door.

Matt Gemmell on simplicity:

More importantly, I also believe in simplifying my life, offline and online, to let me focus on doing what I want to do - whether that’s writing code, writing words, or helping other people with their work. To do that, I have to reduce the ambient noise.

Step three, stop. Think. You can’t grind and simplify all the time. Your brain needs room to breathe. If you ever wondered why you do your best thinking and problem solving in your dreams or in the shower, I’ll tell you why: those places have no computers, TVs, or internet. Every week, you need to get away from your computers, music, and distractors. Go someplace novel and interesting; a coffeeshop, a park, a busy boulevard, a quiet trail, whatever makes your brain happy. Take a notebook or whatever you can physically think on. Now use that time to take apart what you’re working on, think about how it works, and figure out how to make it work better.

Jacob Gorban on thinking time:

In this state, we may become so reactive to the tasks that need to get done that we just don’t stop, take a step back and reflect on the whole situation. We may just forget to think deeply, strategically about the business and even about the work tasks themselves.

Your brain will thank you for the chance to stop and think. You’ll feel better when you remove the extra crap that’s distracting you. You’ll glow inside when you put the time in every day to make things and end up with something awesome.

[1] Sorry, I’m not a clutter person, I can’t help you here.

I previously noted that adding people to a team is tricky, doing so quickly doubly so. A nice discussion popped up around how to do so effectively. So, to cover the other side of the team-growing coin, here are some ideas on what helps when adding people to your team:

• When you integrate people, do it purposefully and deliberately. (Jeff Casimir)
• Grow the team slowly. Pair the new person with a mentor. Task the new person with the change that a cultural, process, or technological change that the team agrees upon as part of the recruiting and hiring process. (Myself)
• Pairing can help. Jeff mentioned pairing in the context of teachers. If you’re already doing pairing, I bet it helps a lot of these team growth issues.
• Document your culture (Jeff), present said document as new people join the team. Even better, document your culture online as part of your team’s outward face and recruiting efforts (Brian Doll). Works great for GitHub.
• Announce the hire with an interview-style announcement rather than a short bio (Brian Doll).
• Go over the top when celebrating bring on a new team member (Jeff).
• Jeff noted that in education, they have the advantage that all new people start at the same time in August. You can use this to batch celebrate/integrate new team members.
• Never stop the process of integrating your new team members (Brian). When you stop, people notice. As the saying goes, if it hurts, do it more.
• Job titles can be a cancer (Brian). If you’re constantly bringing on “senior developers”, what is there to celebrate?
• The E-Myth Revisited is mostly about entrepreneurship (Jeff), but it devotes a lot of space to focusing on roles instead of jobs. This makes it easier to bring people on with less focus on titles and more on what they will actually do. Brian notes that roles are great for lowering your bus number and encouraging team ownership of the product.

Culture is hard

Looking at all of these ideas, it strikes me that maybe it’s not adding to a culture that’s tricky; maybe it’s defining and maintaing a culture that’s really challenging. I often find it difficult to draw the line between the personalities on a team and the explicit and implicit culture that is the aggregate of those personalities and their actions. Getting a bunch of people on the same page and deciding what the culture is would prove challenging, as is any activity with a group of people.

Subtract the notion of adding new people to a team, and the above ideas are all about defining and maintaining a culture. That’s something worth thinking about as you start a team. What do you value, how do you present yourself, how do you get stuff done? Once those questions are answered, you have a starting point for your culture. Then it’s a matter of “gardening” that culture so that everyone, new team members and veterans alike, learn it and evolve it.

Thanks to Brian and Jeff for a great conversation, they both get internet gold stars. I’m just the guy who curated it and typed it all in later.

I’m a sucker for good portmanteau. “Devops” is a precise, but not particularly rewarding concatenation of “development” and “operations”. What it lacks in sonic fun, it makes up in describing something that’s actually going on.

For example, the tools that developers build for themselves are taking cues from the scripts that the operations team hobbles together to automate their work. In the bad old days, you manually configured a server after it was racked up. Then there was a specific load out of packages, a human-readable script to work from, a disk image to restore from, or maybe even a shell script to execute. Today, you can take your pick from configuration management systems that make the bootstrap and maintenance of large numbers of servers a programmatic matter.

It’s not just bringing up new servers that developers are dabbling in. Increasingly, I run across developers who are really, really interested in logging everything, using operational metrics to guide their coding work, and running the deploys themselves. In some teams, the days of “developers versus operations” and throwing bits over walls is over. This is a good.

You devop and don’t know it

Even if you don’t know Chef or Puppet, even if you never ssh into a database server even once, even if you never use the #devop hashtag or attend a like-marketed conference, you’re probably dabbling in operations. You, friend, are so devops, and you don’t even know it.

You use a tool or web app to look at the request rate of your application or the latency of specific URLs and you use that information to decide where to focus your performance efforts. You watch the errors and exception that your app encounters and valiantly fix them. Browsers request images, scripts, and stylesheets from your site and you work to make sure they load quickly, the site draws as soon as possible, and users from diverse continents are well served. You run deploys yourself, you build an admin backend for your app, you automate the processes needed to keep the business going. You consult with operations about what infrastructure systems are working well, what could improve, and what tools might serve everyone better.

All of these things skirt the line between development and operations. They’re signs of diversifying your skillset, better helping the team, and taking pride in every aspect of your work. You can call it devops if you want, but I hope you’ll consider it just another part of making awesome stuff.

Here we are, in the waning months of 2011. Ruby and its ecosystem are a bit of an incumbent these days. It’s a really great language for a few domains. It’s got the legs to become a useful language for a couple of other domains. There are a few domains where I wouldn’t recommend using it at all.

Ruby’s strong suit

Ruby started off as a strong scripting language. The first thing that attracted non-tinkerers was a language with the ease-of-hacking found in Perl with the nice object-oriented features found in Java or Python. If you see code that uses special globals like $! and $: or weird constants like ARGF and __DATA__ and it mostly lacks classes and methods, you’re looking at old-fashioned scripting code.

As Ruby grew, it got a niftier way of doing object-oriented programming. Developers started to appreciate it in the same places they might use Java or Smalltalk. A few of the bravest started building production systems using a nice object-oriented language without the drawbacks of a high-maintenance type system (Java) or the isolation of an image (Smalltalk). This code ends up looking a little like someone poking Ruby with their Java brain; they’re not using the language to its fullest, but they’re not abusing it either.

Out of the OO crowd exploded the ecosystem of web frameworks. There were a few contenders for a while, but then Rails came and sucked the air out of the competitive fire. For better or worse, nearly everyone doing web stuff with Ruby was doing Rails for a few years. This yielded buzz, lots of hype, some fallings out, some useful forward progress in the idioms of software development, and a handful of really great businesses. At this point in Ruby’s life, its interesting properties (metaprogramming, blocks, open classes) were stretched, broken, and put back together with a note pointing out that some ideas are too clever for practical use.

As Ruby took off and more developers started using it, there was a need for integration with other systems. Thus, lots of effort was put into projects to make Ruby a part of the JVM, CLR, and Cocoa ecosystems. Largely, they delivered. At the end of 2011, you can use Ruby to integrate with and distribute apps for the JVM and OS X, and maybe even Windows. This gave Ruby credibility in large “enterprisey” shops and somewhat freed Ruby from depending on a single implementation. The work to make this happen is non-trivial and thankless but hugely important even if you never touch it; when you see one of these implementers, thank, hug, and/or bribe them.

Ruby could go to there

WARNING Prognostication follows WARNING, your crystal ball is possibly different than mine

Scala, a hybrid functional/object-oriented language for the JVM, is a hot thing these days. A lot of people like that it combines the JVM, the best ideas of object-oriented programming, and then swizzles in some accessible and useful ideas from the relatively untapped lore of functional programming (FP). So it goes, Ruby already does one or two of these things, depending on how you count. The OO part is in the bag. Enumerable exposes a lot of the same abstractions that lie at the foundation of FP. If you’re using JRuby, you’re getting many of the benefits of the JVM, though Scala does one better in this regard right now. Someone could come along and implement immutable, lazy data structures and maybe a few combinators and give Ruby a really good FP story.

Systems programming is traditionally the domain of C and C++ developers, with Java and Go starting to pick up some mindshare. Think infrastructure services like web servers, caches, databases, message brokers, and other daemon-y things. When you’re hacking at this level, control over memory and execution is king. Access to good concurrency and network primitives is also important. Ruby doesn’t do a great job of providing all of these right now, and Matz’s implementation might never rank highly here. However, one of the promising aspects of Rubinius is that they’re trying very hard to do well in terms of performance, concurrency, and memory management. If Rubinius can deliver on those efforts, offer easily hacked trapdoors to lower level bits, and encourage the development of libraries for network and concurrent programming, Ruby could easily turn into a good solution for small-to-medium sized infrastructure projects.

Distributed systems are sort of already in Ruby’s wheel house and sort of a stretch for Ruby. On the one hand, most Ruby systems already run over a combination of app servers and queue workers, storing data in a hodgepodge of browser caches, in-heap caches, and databases. That’s a distributed application, and it’s handy to frame one’s thinking about building an application in terms of the challenges of a distributed system: shared state is hard to manage, failure cases are weird and gnarly, bottlenecks and points of failure are all over the place. What you don’t see Ruby used for is implementing the infrastructure underneath distributed applications. Hadoop, Zookeeper, Cassandra, Riak, and doozerd all rely on both the excellent concurrency and network primitives of their respective platforms and on the reliability and performance those platforms provide. Again, given some more progress on Ruby implementations and good implementations of abstractions for doing distributed messaging, state management, and process supervision, Ruby could be an excellent language to get distributed infrastructure projects off the ground.

Unlikely advances for Ruby

Embedded systems, those that power your video game consoles, TVs, cars, and steroes, rely on promises that Ruby has trouble keeping. C is king here. It provides the control, memory footprint, and predictability that embedded applications crave. Rite is an attempt to tackle this domain. The notion of a small, fast subset of Ruby has its appeal. However, developers of embedded systems typically hang out on the back of the adoption curve and are pretty particular about how they build systems. Ruby might make in-roads here, but it needs a killer app to acheive the success it currently enjoys in application development.

Mobile apps are an explosive market these days. Explosive markets go really well with Ruby (c.f. “web 2.0”, “AJAX”, “the social web”), but mobile is different. It’s dominated by vendor ecosystems. Largely, you’ve got iOS with Objective-C and Cocoa, and Android with Java and, err, Android. Smart developers don’t tack too far from what is recommended and blessed by the platform vendor. There are efforts to make Ruby play well here, but without vendor blessing, they aren’t likely to get a lot of traction.

Place your bets, gentlemen

Tackling the middle tier (object/functional, distributed/concurrent, and systems programming) is where I think a lot of the really promising work is happening. Ruby 1.9 is good enough for many kinds of systems programming and has a few syntactic sugars that make FP a little less weird. JRuby offers integration into some very good libraries for doing distributed and concurrent stuff. Rubinius has the promise to make those same libraries possible on Ruby.

Really sharpening the first tier (thinking about how to script better, getting back to OO principles, fine tuning the web development experience, improving JRuby’s integration story) is where Ruby is going to grow in the short term. The ongoing renaissance, within the Ruby community, of Unix idioms and OO design is moving the ball forward; it feels like we’re building on better principles than we were just two years ago. The people who write Ruby will likely continue to assimilate old ideas, try disasterous new ones, and trend towards adopting better ways of building increasingly large applications.

When it comes to Ruby, go long on server-based applications, hedge your bets on systems infrastructure, and short anything that involves platforms with restricted resources or vendor control.

Relentless Quality is a great piece. We should all strive to make really fantastic stuff. But I think there’s a nuance worth observing here:

Sharpen the edges, polish the surface and make it shine.

I’m afraid that some people are going to read more than the Kneath intends here. Quality does not mean perfection. Perfection is the enemy of shipping. Quality is useless if it doesn’t ship. Quality is not an excuse for not shipping.

Quality is a subjective, amorphous thing. To you, it means the fit and finish. To me, it means that all the bugs have been eliminated and possible bugs thought about and excised. Even to Christopher Alexander, quality isn’t nailed down; he refers to good buildings as possessing the “quality without a name”.

To whit, this shortcoming is pointed out in the original essay:

Move fast and break things, then move fast and fix it. Ship early, ship often, sacrificing features, never quality.

Scope and quality are sometimes at odds. Schedules and quality are sometimes at odds. There may come a time when you have to decide between shipping, maintaining quality, and including all the features.

The great thing about shipping is that if you can do it often enough, these problems of slipping features or making sacrifices in quality can fade away. If you can ship quickly, you can build features out, test them, and put that quality on them in an iterative fashion. Shipping can’t cure all ills, but it can ease many of them.

Kneath is urging you to maintain quality; I’m urging you to ship some acceptable value of quality and then iterate to make it amazing. Relent on quality, if you must, so you can ship relentlessly.