Adam Keys is Thinking

I’ve been programming, as a full-time job, for more than ten years. I started doing Ruby, and Rails, nearly ten years ago. I’ve been at LivingSocial for a two and a half years now. If you add these numbers up you will find that I am, improbably, an old hand.

As I try to explain and contextualize our organization, technology, and culture inside LivingSocial, I often catch myself delicately dancing around saying “it’s always been this way”. It sucks to hear this. It’s easy for this to sound like a deflection. “Don’t worry yourself, just accept it and let me get back to whatever I was doing before you asked that foolish question.” Crawl back to your desk and grind away.

Beginner’s mind is crucial and fleeting, so every person new to our communities and teams are invaluable. I want to extract as much information about how we’re confusing or mired in our legacy of stupidity. A beginner’s mind can help us improve our organizations, but only when it’s open and the experts aren’t marked as institutional damage to route around instead of engage with.

So I’m explaining organizational history and I find myself hand waving around “it’s always been this way, bask in despair!” At this point I (try to) profusely apologize and restart my explanation or answer. I need to get at that beginner’s mind before it becomes jaded in foregone conclusions.

But sometimes I don’t stop. I forget. I get wrapped up in accurately and concisely explaining how it got this way and why it’s this way. I forget to actually answer the question or suggest how it might be made better.

Allow me apologize to everyone out there on the internet. Sometimes it might seem like I’m saying “it’s always been this way”, but really I got so wrapped up in giving a vigorous history lesson that I lost my train of thought. It might help to restate the question. I’m an old hand, after all, with a frail brain.

I came up in the days when Microsoft and Linux were mortal enemies. Back when “Borg Bill Gates” was the icon for stories about MS on Slashdot. Back when Slashdot was the cutting edge.

Thus, I’m always slightly surprised when I come across some really solid work done by Microsoft. The people at Microsoft are capable of really great work, but it often doesn’t escape The System in place between the hands on keyboards and the public face of the company.

Microsoft Research, in particular, produces surprisingly good research papers. The recent paper on Orleans is a great example.

Orleans is an opinionated “virutalized” actor framework MS developed that runs on their .NET and Azure frameworks:

• The design is single-threaded, small timemeslices, no preemption; not unlike Go’s goroutines (when considered on only one host)
• Orleans has its own runtime for activating, deactivating, locating, and dispatching to actors
• Semantics end up looking like a combination of queue-workers, actors, and Promise-based systems

Implementing Orleans instead of a three-tier (web, app, DB) system allowed them to eliminate the need for a caching layer once they reached scale. Actors encapsulate state and caching thereof instead.

• Virtualized actors become analogous to virtual memory, raising the level of abstraction programmers can work at
• Virtualized actors don’t require management by operators or developers; failure and load-balancing behavior is managed by the runtime instead of prior specification
• Currently implements at-least-once message semantics, which works for developers; considering added only-once message delivery

Orleans is currently in use in the Halo 4 presence and statistics services. I bet you’ll see clones and shallow carbon copies of these ideas in a language or conference near you. Read the paper, it’s quite accessible as distributed systems papers go. There are some nice ideas in there.

Ed. Let me know if you like reading notes on papers I’ve read. I’d like to do it more often if folks find it useful.

RailsConf 2014 having wrapped up a few weeks ago, now seems like a good time to try and unpack what I saw, heard, and talked to others about. Bear in mind I skipped RailsConf 2013 (but I’ve been to all the others), so I may construe something as new that I simply missed last year.

I’m going to break it down, as is my puzzle-solving wont, by technology and people.

Technology

Lots of people are excited about a few central topics.

JavaScript. They want to build ambitious apps, with Ember, Lineman, and other tools. Developers seem somewhat concerned that building apps this way is swimming upstream. Or that it’s early days and things are changing quickly. Or they think it’s awesome to work in a growing field. It’s all of those things.

Service-oriented Architecture. I went to talks about extracting services, designing services, implementing authentication/authorization across services, and how to write the client side of your various services. The talks felt like they were past the “look at this novel thing!” phase and into the “well here’s the nitty gritty” phase. I didn’t happen upon any sessions of the “here’s how this thing punched me in the face!” sort, which is the kind of thing I, personally, want to learn about right now.

Getting outside of Rails. Beyond SOA, some folks are going off the golden path and finding some success. I attended one good talk on adopting ideas from Domain Driven Design and hexagonal architecture. I found Ernie Miller’s ideas about how he’s worked with Rails' flaws interesting, even though I often didn’t agree.

People

The conference started with David’s keynote throwing barbs at the attitude some people display towards TDD (an ambiguous acryonym to start with). Many speakers opened with, in my opinion vapid, jokes about this. Other people seemed to take personal offense that TDD might not be everything. These reactions were, I found, not particularly interesting. The more interesting ones were those who found it as a challenge to consider how they build applications, either through taste and intuition or through reasoning and engineering.

There’s always, at least in conversations I find myself in, some question of where Rails is now. Relative to other technologies, is it a leader, a follower, a player, a has-been? Every year, it’s more of a safe assumption that Rails is a given, that it’s not going to up and disappear. Still, there’s a desire to keep it fresh, avoid stagnation, and above all avoid becoming the demons (J2EE, .NET, PHP, etc.) that people used before Rails. A lot of this discussion seems to be happening around the size and frameworkness of Rails. There has always been pockets of interest around things that are smaller than Rails (like Sinatra, ROM, etc.) More interesting, there is some defense of largeness now, especially in the context of Ember. This is the most interesting, and at times tedious, debate, for me personally.

The attendance of RailsConf continues to grow more diverse. The inclusiveness efforts of the organizers and community at large seem to be bearing fruit. I saw more ladies and more minorities than I have at any conference of this size. Further, the crowd felt less startup-centric than years past. Plenty of folks from startups, sure, but also people at different kinds of businesses: small, big, hardware, software, commerce, marketing, non-profit, etc. What’s more, it was easier than ever to find myself in a conversation with people not entirely like myself. That’s a quite good thing.

Lots of people are concerned about Rails, its community, and such. There’s a vibe, not unlike 2008 or 2009 when some were seeking other ways to build Ruby web apps, that perhaps the framework, the community, and the leaders thereof don’t have all the answers. That may sound damning, but it’s a pretty healthy attitude. The problems developers face and the way we build our programs are, as ever, changing as we’re building them. The tension is out there, but having been through most of a technology hype cycle with Rails, I’m not worried that the community won’t find the resolution of that tension.

Conjecture: thinking about releasing software versions to users/partners/the public in dotted version numbers, e.g. “1.0, 2.1, 5.3” is a symptom of misshaped thinking. This line of thinking seems framed by the notion that only a dozen major versions will ever be released. Certainly, some software works that way: shrink-wrap software, video game consoles, etc.. Most notably, dotted release versions are very useful for software used to build software.

However, its increasingly true that software doesn’t really work until it reaches “version” two-hundred something. But maybe that’s another problematic frame of thinking…

Glenn Vanderburg on teaching developer how to use Ruby testing APIs:

For example: I hate it when APIs (or languages, or whatever) are presented as "it's just English!" That doesn't give anyone anything useful to work with; it's just trying to allay fears, and it replaces a mythical danger (the thing people are afraid of simply because it's unknown) with a real danger: you're telling them they don't need to learn anything, when in fact the opposite is true.

I’ve never liked English-like APIs either. Consider that AppleScript is, bizarrely, completely unusable by people who know how to program in any other programming language.

If you think unpack “It’s just English” a bit, it’s easy to see why you should run away screaming from anything describing itself as being English-like in a good way. English is a human language riddled with inconsistencies. Many, if not most, English speakers learned it when they were young and the brain is almost useless for anything that isn’t language acquisition. Those who have learned English later in life become literate through lots of work, a healthy dose of necessity, and probably several years where their English was good enough for humans to understand but utterly confusing to a computer.

Just say “false” to “It’s just English”, coders!

I’m currently jazzed by the idea of full stack writing:

Hi is what we call a “full stack” writing and publishing platform. Just what is a writing stack? Capture. Write. Publish. is our summary of it, but really it breaks down into five parts:

• Sudden inspiration!
• Capture
• Draft
• Publish
• Converse

Some platforms provide tools for parts of the stack. Hi gives you tools for the full stack.

All the pancakes.

Love the breakfast food metaphor! Hi’s tools and community integration seem pretty nifty. That said, the break down of inspire, capture, write, converse is what I can’t get out of my head. I don’t think you need special tools to approach it with that breakdown. WordPress, Jekyll, Medium, whatever. Always be writing, publish small ideas frequently (a thing I need to do more often!), develop the ideas that stick in your head or that lead to great conversations, repeat. Don’t let the layout of your blog template or the name of the tool limit your writing. It’s an easy mental trap to fall for!

A great thing about writing is that it focuses and sharpens one’s thoughts. The great thing about writing in public is that your thoughts, when passed through the brains of others, sometimes yield even better thoughts. The greatest thing about writing is when you hit publish feeling confident about your position and end up with your opinion flipped once the conversation is over.

So it went with A tale of two Rails view that a few hours after I’d published it, my mind was changed.

On the one hand, you can take a permissive stance on views and ivars. Dave Copeland nicely laid this idea out. If you are responsible about minimizing sharing ivars between actions and views, you have a chance. In this case, that means sharing one or two ivars, no more. Placing a little trust in your fellow developers lets you put off the day when you need to isolate state behind helper methods or other restrictive abstractions.

On the other hand, you can take a contractual stance on views and require all data passing between actions and view to do so through a specific object. Tony Pitale’s SimplestView does just this. It unifies views and partials into one name, “templates”, and then provides backing objects (somewhat confusingly called views). Actions create the backing objects and templates consume them. Nothing can leak through, no hunting down the origin of an ivar needed in this template but not yet defined in that one.

Somewhere in the middle are ideas about building a bridge between the action and the view. One could use responds_with as said bridge, but, for me, that felt like it played better with APIs than traditional views[1]. A possible middle ground is to use something like decent_exposure, or this interesting spin on it by Nathan Ladd. I like that Nathan’s approach is basically sugar over the pattern of encapsulating action state behind controller helper methods which are shared between views. I’ve been using the helper method approach so far, but it’s a little awkward to test and confusing for those used to sharing ivars.

If you’re sick of the baby and the bath water, you might find a more extreme approach interesting. Focused Controller and poniard do away with Rails conventions entirely. You still write controllers, but how you load data and model actions is entirely different. Personally, these are interesting sources of ideas. I’m not sure I’d jump on them for a production application.

Of all these approaches, I’m most intrigued by SimplestView. It seems like the Minimum Viable Departure from Rails conventions that could possibly solve the problem of sharing state and defining a contract on that state. That said, I’m likely to try Dave Copeland’s approach first. I like that it’s about talking with teammates, reviewing each other’s work, and making smart decisions. I’m finding that no amount of framework is as good at helping people write good code as review, feedback, and iteration.

Why do I prefer to avoid referencing instance variables in view?

I needed to clarify this personal principle to a teammate recently. It was one of those things I’ve internalized so strongly that it was initially difficult for me to do so; one of those things that have become so true to me that teaching someone else about it requires getting past the “well, obviously it’s the best, that’s why” phase of mental tantrum.

An entirely made-up, possibly oversimplified Rails view fragment:

<p>Hi, my name is <%= @user.name %></p>

This is a by-the-book Rails view. Set an ivar in an action, use it in views (and helpers), you’re done, get back to thinking about building product.

It’s easy to iterate with this. It’s easy to see data “passed”¹ into the view by scanning for the @ sigil; your editor probably does this already. It’s easy to move markup around in templates with a simple cut-paste.

If you stick with ivars long enough, you’re going to end up with two kinds of misadventures.

Most commonly, you’ll wonder why @user is nil for some view or edge case. Maybe you forgot a filter or didn’t set it in an action? Backend developers are sometimes equipped with the curiosity and knowledge to fix this themselves. For front-end developers or those new to the system, this kind of error is basically “womp-womp sad music go interrupt someone who can help you”.

This leads to the second misadventure: where did this @user thing come from²? Maybe it was set in a helper, or a filter, or an action? Well now you’ve painted yourself into a weak spot of a language like Ruby. Your tools probably can’t point you directly at the line of code where a variable came into being. You can do some clever grep’ing around³, probably. At best, you know the system well enough to get to the right file and find it, or there’s some convention you can use to intuit where it might be set.

Of course this way is better, right?

<p>Hi, my name is <%= current_user.name %></p>

Well, not initially. Already you have to pick a good name for a method because you’re probably going to use it all over the place. Then you have to find a good place to put that method: on a helper method? on a helper object? on a model?, i.e. now you’re making decisions, which is a thing Rails tries to shield you from wherever possible. Personally, I find naming things quite entertaining and hardly one of the hardest problems in computer science⁴.

A marginal benefit comes from reducing the entry points that the name current_user came to be a thing in this view: it’s a method name, a local variable, or a view-local variable passed into the template⁵. Thus the search space for “where did this thing come from” is way smaller for typical Rails templates and manageably smaller for unreasonable Rails templates.

This way pays off once the application gets past the point where new code tidily fits into models, views, controllers, or helpers. At that point, you need objects, messages, and experience at building with objects and messages (successes, stalemates, and abject failures)⁶. If you’re competent at messages (i.e. method calls) in Ruby, you can at this point experience a “this is Unix, I know this!"⁷ moment and work with this method call like you would any other method/function invocation in a computer program.

Making this investment into a method call yields another long-term benefit: simplicity in experimentation and testing⁸. I can poke at helper methods and helper objects in a Rails console without breaking a sweat. If that feedback loop doesn’t get the job done, I can write tests against helpers and (some) controller methods to iterate on figuring out why something doesn’t work the way I think it should.

It’s amazing that blogging about programming is so popular. Programming involves a lot of tradeoffs. Tradeoffs make for wordy articles that don’t leave you thinking “yeah, those other guys are SOOOO WRONG” or “YEAHHHH I’m so right”.

If I’ve written this properly, hopefully you felt both of those feelings. Maybe you reminisced about a day when you thought view ivars were great and then regretted it. Perhaps it’s easier to see why you’d start an app or feature off using ivars and then refactor them to method calls later.

With instance variables in views, as with many other grey areas of Rails, the most useful long view is this: you’re always right, and you’re always wrong, it just depends on which “you” (past you, present you, legacy project you, greenfield project you) is being observed.

Adding new functionality to software is really exciting. I love poking around the edges of a system, figuring out what’s going on, and looking for that “obvious” place where the first change should happen. But sometimes, it’s hard to know if I’m making the right change. How Should This Work?

The temptation when changing an existing system is to implement the desired behavior within the structure of the current abstractions. Repeat this without adjustment, and you’ll quickly end up contorting existing concepts or working around legacy behaviors. Conditionals pile up, and shotgun surgery becomes standard operating procedure.

Even if you code test first, you can make a mess of a system. What you end up with is a system that moves from local maximum to local maximum while the time of the test suite grows unbounded. There are worse things that could happen, but no one’s going to jot this one down as a “best practice” either.

The counterforce to this temptation is the red-green-refactor cycle. Look at how the system works and figure out how the next bit of work might change how the system works. Refactor to simplify the act of making a change, or make the change and refactor afterwards to a better design.

Software can grow by accretion, but it stays malleable when the team culture is one that balances continuous growth with continuous cultivation of good design.

First, read all of this excellent distillation of distributed systems by Jay Kreps,  The Log: What every software engineer should know about real-time data’s unifying abstraction. Now, consider this.

There’s a moment, when you’re building services and web APIs, when you think you’ve pretty much got it under control. You’ve got an endpoint for every query, a resource for every workflow. All the use-cases seem to be under control. And then, the question appears:

“How can I get access to all the updates to all the data? You know, for [REASONS].”

For APIs exposed to external developers over the web, this is where you’d reach for web hooks or PubSubHubbub. It’s not the best solution, but it works. If you’re building an internal system, you could  use the same approaches, or…you could build a log.

No not that kind of log. An event log, like LinkedIn did with Kafka for their internal systems. Every time your data model changes, every create, update, or delete, you drop an event with all the metadata related to the change. The event goes into some kind of single-producer, multiple consumer queue. Then all the clients that want to know about all the changes to all the things can read events off the queue and do whatever it is they need to for those important REASONS.

If you find this intriguing, this is a lot like replication in database systems. Definitely read LinkedIn’s article on this, and definitely read up on how your database of choice handles replication. And if you’ve built this before and have a good answer to initially populating “replicas” of a database, let me know; I haven’t come up with anything better than “just rsync it”.

Brent Simmons is keeping a diary as he works through implementing sync for Vesper, an iOS note-taking app. Building this sort of thing isn’t easy; cf. it took Cultured Code multiple years to implement it for Things. Thus it’s pretty neat that Simmons is breaking it down into understandable chunks:

• Syncing Tags, on reasoning about identity in your data model
• Immutability, Deleting, and Calculated Properties, on tackling deleted data, aka the hardest kind of write in a distributed system
• Sync tokens and efficiency, on converting your system into an event log
• Merging Notes, on sync semantics and dealing with clocks

If you think you need to implement a synchronization system for your application, try to find a shortcut so that you don’t. If you can’t find a shortcut, you could do worse than starting with these notes.

So, Adam from a few years ago, you think you can build a distributed system?

Designing a concurrent system, one that runs across multiple processors using shared memory threads, is difficult because someone can move your cheese. One moment you’ve got food on your plate, and the next moment it’s on someone else’s plate.

Designing distributed systems, one that runs across multiple computers over networks that occasionally do weird things, is difficult because you can’t assume that the message will get through. One moment you’re telling your friend about an awesome new album and the next moment they don’t even know who you are anymore.

In both scenarios, you can’t assume the simplest case. You have to discard Occam’s razor; it’s entirely possible a perfectly devious alien is tinkering with your system, swallowing messages or preventing threads from running. For every operation in your system, you have to ask yourself what could go wrong and how will my system deal with that?

The first jarring thing about these sorts of problems is that you’re probably not using a framework. There are libraries to help you with logging, instrumentation, storage, coordination, and low-level primitives. There aren’t a lot of well-curated collections of opinions expressed as code. There’s no Rails, no Play, no Django, no Cocoa. There’s hardly even a Sinatra, a Celery, a JDBC.

That means you’re going to be designing, for real. Drawing on whiteboard, building prototypes and proofs of concepts. Getting feedback from as many smart people as you can corner. Questioning your assumptions, reading everything you can find on the topic. Looking for tradeoffs that decrease risk and simplify your problem domain.

Accept that you’re unlikely to ever get it entirely right. Those who are really good at it are more like Scotty than Spock. They’re surrounded by levers and measurements, tweaking and rebuilding the system as they go. It’s a fun puzzle!

rd.io/x/QFWiK0E…

Look, Prince, I’m not sure if I can agree with you regarding “Breakfast Can Wait”. Cereal? Toast? English Muffins? Sure, those things can wait. Migas? Pancakes? Bacon? These things cannot wait. And don’t get me started on eggs; wait too long and delicious eggs can go rubbery and gross all too quickly.

In Cocoa, you can poke inside object graphs (and more!) using dotted strings:

 NSDictionary *bestOfBeatles = @{@"paul": @{@"beatles": @"Hey Jude", @"solo": @"Jet"}};

NSString *solo = [bestOfBeatles valueForKeyPath:@"paul.solo"];
NSLog(@"Paul's Best solo song is: %@", solo);

Hey, did you notice that Objective-C sprouted some really handy literals for arrays and dictionaries lately? One could reasonably extrapolate that Apple has an interest in not breaking people’s fingers when they build for iOS and OS X.

There are libraries, e.g. Hashie, that make uniform access into object graphs, like you might get from an API response, as close to idiomatic Ruby as you probably want. Key paths, as implemented in Cocoa’s Key-Value Coding APIs, takes a different approach, using strings to define paths to traverse an object graph. I don’t really like “string programming”, but it’s an interesting approach, so let’s see where it takes us. Let’s add something like “key paths” to Ruby!

Hark, an implementation

Enumerable is an awesome thing about Ruby. Let’s build key path traversal on top of that. The goal is something close to the Objective-C snippet above:

 best_beatles = KVC.new({"paul" => {"beatles" => "Hey, Jude", "solo" => "Jet"}})
solo = best_beatles.for_path("paul.solo")
puts "Paul's best solo song is: #{solo}"

Turns out this isn’t hard to implement, even with a little error handling thrown in:

require 'delegate'

class KVC < SimpleDelegator

  def for_path(path)
    segments = path.split('.')

    segments.inject(self) do |value, s|
      if !value.respond_to?(:has_key?)
        raise ArgumentError.new("Expected #{value} to respond to has_key?")
      end

      if value.has_key?(s)
        value[s]
      else
        raise KeyError.new("Missing key #{s}")
      end
    end
  end

end

OK, first off, did you realize that SimpleDelegator makes it ridiculously easy to wrap objects with extra behavior? It sure does! Second, you could easily write this with a while loop or even a simple each, but why have local variables hanging around when you could use inject instead?![1] Third, this has more conditionals than I’d like, but it fits within 80 columns[2] so I’ll take it.

The pros and cons of using key paths

I’ve got this code, based on a whim. I’m not even sure it’s a good idea. Maybe it is!

• It’s simpler to specify a traversal into a deep object graph with a string than with a bunch of key lookups or method chains. It’s slightly clearer too.
• “Selector APIs”, like XPath or jQuery’s extensions to CSS, are an idea many developers have already wrapped their head around. I wouldn’t want to go anywhere near the complexity of those APIs, but an opinionated set of operations on an object graph might not get too complex.

Maybe it sucks?

• Enumerable is pretty great, why wouldn’t you use that?
• Programming by banging strings together is terrible.

Judgement

I can’t really tell you, definitively, if you should try this or not. I’ve barely even used KVC in Cocoa! It’s a nifty idea, though, and it’s pretty rad that you can implement something like it in Ruby so trivially.

Now you know!