It’s Always Been This Way…Huh?

I’ve been programming, as a full-time job, for more than ten years. I started doing Ruby, and Rails, nearly ten years ago. I’ve been at LivingSocial for a two and a half years now. If you add these numbers up you will find that I am, improbably, an old hand.

As I try to explain and contextualize our organization, technology, and culture inside LivingSocial, I often catch myself delicately dancing around saying “it’s always been this way”. It sucks to hear this. It’s easy for this to sound like a deflection. “Don’t worry yourself, just accept it and let me get back to whatever I was doing before you asked that foolish question.” Crawl back to your desk and grind away.

Beginner’s mind is crucial and fleeting, so every person new to our communities and teams are invaluable. I want to extract as much information about how we’re confusing or mired in our legacy of stupidity. A beginner’s mind can help us improve our organizations, but only when it’s open and the experts aren’t marked as institutional damage to route around instead of engage with.

So I’m explaining organizational history and I find myself hand waving around “it’s always been this way, bask in despair!” At this point I (try to) profusely apologize and restart my explanation or answer. I need to get at that beginner’s mind before it becomes jaded in foregone conclusions.

But sometimes I don’t stop. I forget. I get wrapped up in accurately and concisely explaining how it got this way and why it’s this way. I forget to actually answer the question or suggest how it might be made better.

Allow me apologize to everyone out there on the internet. Sometimes it might seem like I’m saying “it’s always been this way”, but really I got so wrapped up in giving a vigorous history lesson that I lost my train of thought. It might help to restate the question. I’m an old hand, after all, with a frail brain.

Microsoft’s Orleans, a good ideas

I came up in the days when Microsoft and Linux were mortal enemies. Back when “Borg Bill Gates” was the icon for stories about MS on Slashdot. Back when Slashdot was the cutting edge.

Thus, I’m always slightly surprised when I come across some really solid work done by Microsoft. The people at Microsoft are capable of really great work, but it often doesn’t escape The System in place between the hands on keyboards and the public face of the company.

Microsoft Research, in particular, produces surprisingly good research papers. The recent paper on Orleans is a great example.

Orleans is an opinionated “virutalized” actor framework MS developed that runs on their .NET and Azure frameworks:

  • The design is single-threaded, small timemeslices, no preemption; not unlike Go’s goroutines (when considered on only one host)
  • Orleans has its own runtime for activating, deactivating, locating, and dispatching to actors
  • Semantics end up looking like a combination of queue-workers, actors, and Promise-based systems

Implementing Orleans instead of a three-tier (web, app, DB) system allowed them to eliminate the need for a caching layer once they reached scale. Actors encapsulate state and caching thereof instead.

  • Virtualized actors become analogous to virtual memory, raising the level of abstraction programmers can work at
  • Virtualized actors don’t require management by operators or developers; failure and load-balancing behavior is managed by the runtime instead of prior specification
  • Currently implements at-least-once message semantics, which works for developers; considering added only-once message delivery

Orleans is currently in use in the Halo 4 presence and statistics services. I bet you’ll see clones and shallow carbon copies of these ideas in a language or conference near you. Read the paper, it’s quite accessible as distributed systems papers go. There are some nice ideas in there.

Ed. Let me know if you like reading notes on papers I’ve read. I’d like to do it more often if folks find it useful.

Aaron Patterson

Unpacking RailsConf 2014

RailsConf 2014 having wrapped up a few weeks ago, now seems like a good time to try and unpack what I saw, heard, and talked to others about. Bear in mind I skipped RailsConf 2013 (but I’ve been to all the others), so I may construe something as new that I simply missed last year.

I’m going to break it down, as is my puzzle-solving wont, by technology and people.

Technology

Lots of people are excited about a few central topics.

JavaScript. They want to build ambitious apps, with Ember, Lineman, and other tools. Developers seem somewhat concerned that building apps this way is swimming upstream. Or that it’s early days and things are changing quickly. Or they think it’s awesome to work in a growing field. It’s all of those things.

Service-oriented Architecture. I went to talks about extracting services, designing services, implementing authentication/authorization across services, and how to write the client side of your various services. The talks felt like they were past the “look at this novel thing!” phase and into the “well here’s the nitty gritty” phase. I didn’t happen upon any sessions of the “here’s how this thing punched me in the face!” sort, which is the kind of thing I, personally, want to learn about right now.

Getting outside of Rails. Beyond SOA, some folks are going off the golden path and finding some success. I attended one good talk on adopting ideas from Domain Driven Design and hexagonal architecture. I found Ernie Miller’s ideas about how he’s worked with Rails’ flaws interesting, even though I often didn’t agree.

People

The conference started with David’s keynote throwing barbs at the attitude some people display towards TDD (an ambiguous acryonym to start with). Many speakers opened with, in my opinion vapid, jokes about this. Other people seemed to take personal offense that TDD might not be everything. These reactions were, I found, not particularly interesting. The more interesting ones were those who found it as a challenge to consider how they build applications, either through taste and intuition or through reasoning and engineering.

There’s always, at least in conversations I find myself in, some question of where Rails is now. Relative to other technologies, is it a leader, a follower, a player, a has-been? Every year, it’s more of a safe assumption that Rails is a given, that it’s not going to up and disappear. Still, there’s a desire to keep it fresh, avoid stagnation, and above all avoid becoming the demons (J2EE, .NET, PHP, etc.) that people used before Rails. A lot of this discussion seems to be happening around the size and frameworkness of Rails. There has always been pockets of interest around things that are smaller than Rails (like Sinatra, ROM, etc.) More interesting, there is some defense of largeness now, especially in the context of Ember. This is the most interesting, and at times tedious, debate, for me personally.

The attendance of RailsConf continues to grow more diverse. The inclusiveness efforts of the organizers and community at large seem to be bearing fruit. I saw more ladies and more minorities than I have at any conference of this size. Further, the crowd felt less startup-centric than years past. Plenty of folks from startups, sure, but also people at different kinds of businesses: small, big, hardware, software, commerce, marketing, non-profit, etc. What’s more, it was easier than ever to find myself in a conversation with people not entirely like myself. That’s a quite good thing.


Lots of people are concerned about Rails, its community, and such. There’s a vibe, not unlike 2008 or 2009 when some were seeking other ways to build Ruby web apps, that perhaps the framework, the community, and the leaders thereof don’t have all the answers. That may sound damning, but it’s a pretty healthy attitude. The problems developers face and the way we build our programs are, as ever, changing as we’re building them. The tension is out there, but having been through most of a technology hype cycle with Rails, I’m not worried that the community won’t find the resolution of that tension.

About version numbers

Conjecture: thinking about releasing software versions to users/partners/the public in dotted version numbers, e.g. “1.0, 2.1, 5.3″ is a symptom of misshaped thinking. This line of thinking seems framed by the notion that only a dozen major versions will ever be released. Certainly, some software works that way: shrink-wrap software, video game consoles, etc.. Most notably, dotted release versions are very useful for software used to build software.

However, its increasingly true that software doesn’t really work until it reaches “version” two-hundred something. But maybe that’s another problematic frame of thinking…

English-like programming languages, like, yuck

Glenn Vanderburg on teaching developer how to use Ruby testing APIs:

For example: I hate it when APIs (or languages, or whatever) are presented as “it’s just English!” That doesn’t give anyone anything useful to work with; it’s just trying to allay fears, and it replaces a mythical danger (the thing people are afraid of simply because it’s unknown) with a real danger: you’re telling them they don’t need to learn anything, when in fact the opposite is true.

I’ve never liked English-like APIs either. Consider that AppleScript is, bizarrely, completely unusable by people who know how to program in any other programming language.

If you think unpack “It’s just English” a bit, it’s easy to see why you should run away screaming from anything describing itself as being English-like in a good way. English is a human language riddled with inconsistencies. Many, if not most, English speakers learned it when they were young and the brain is almost useless for anything that isn’t language acquisition. Those who have learned English later in life become literate through lots of work, a healthy dose of necessity, and probably several years where their English was good enough for humans to understand but utterly confusing to a computer.

Just say “false” to “It’s just English”, coders!

Find your principles for editing programs

Some folks from GitHub sprung the Atom text editor on the internet yesterday. Releasing a product into a market defined by saturation, settling for brokenness, and zero-cost alternatives is a bold move. I applaud them for jumping into it. I’m eager to see where they end up, both product-wise and technically.

If you’re wondering whether Atom is the right thing for you, it might help you to know I went through a sort of quest several years ago to decide what was vital to me when editing text:

Writing these, thinking deeply about the kinds of problems I did and did not want to solve with my text editor and what that meant for my workflow, was exciting. I learned a lot about finding my own principles.

I ended up, to my great surprise, choosing Vim. Most importantly, that decision stuck. I haven’t gazed upon the possibly greener grasses of other text editors since I committed to my principles and workflow. Whether it turns out Atom is your thing or not, thinking about the principles of how you want to work with computer programs is a thing you might benefit from.

Counterpoint: Rails and instance variables

A great thing about writing is that it focuses and sharpens one’s thoughts. The great thing about writing in public is that your thoughts, when passed through the brains of others, sometimes yield even better thoughts. The greatest thing about writing is when you hit publish feeling confident about your position and end up with your opinion flipped once the conversation is over.

So it went with A tale of two Rails view that a few hours after I’d published it, my mind was changed.

On the one hand, you can take a permissive stance on views and ivars. Dave Copeland nicely laid this idea out. If you are responsible about minimizing sharing ivars between actions and views, you have a chance. In this case, that means sharing one or two ivars, no more. Placing a little trust in your fellow developers lets you put off the day when you need to isolate state behind helper methods or other restrictive abstractions.

On the other hand, you can take a contractual stance on views and require all data passing between actions and view to do so through a specific object. Tony Pitale’s SimplestView does just this. It unifies views and partials into one name, “templates”, and then provides backing objects (somewhat confusingly called views). Actions create the backing objects and templates consume them. Nothing can leak through, no hunting down the origin of an ivar needed in this template but not yet defined in that one.

Somewhere in the middle are ideas about building a bridge between the action and the view. One could use responds_with as said bridge, but, for me, that felt like it played better with APIs than traditional views[1]. A possible middle ground is to use something like decent_exposure, or this interesting spin on it by Nathan Ladd. I like that Nathan’s approach is basically sugar over the pattern of encapsulating action state behind controller helper methods which are shared between views. I’ve been using the helper method approach so far, but it’s a little awkward to test and confusing for those used to sharing ivars.

If you’re sick of the baby and the bath water, you might find a more extreme approach interesting. Focused Controller and poniard do away with Rails conventions entirely. You still write controllers, but how you load data and model actions is entirely different. Personally, these are interesting sources of ideas. I’m not sure I’d jump on them for a production application.

Of all these approaches, I’m most intrigued by SimplestView. It seems like the Minimum Viable Departure from Rails conventions that could possibly solve the problem of sharing state and defining a contract on that state. That said, I’m likely to try Dave Copeland’s approach first. I like that it’s about talking with teammates, reviewing each other’s work, and making smart decisions. I’m finding that no amount of framework is as good at helping people write good code as review, feedback, and iteration.


  1. I know it works with normal views, but I didn’t like that it nudged me down the path of using conditionals. YMMV.  ↩
At sign

A tale of two Rails views

Why do I prefer to avoid referencing instance variables in view?

I needed to clarify this personal principle to a teammate recently. It was one of those things I’ve internalized so strongly that it was initially difficult for me to do so; one of those things that have become so true to me that teaching someone else about it requires getting past the “well, obviously it’s the best, that’s why” phase of mental tantrum.


An entirely made-up, possibly oversimplified Rails view fragment:

<p>Hi, my name is <%= @user.name %></p>

This is a by-the-book Rails view. Set an ivar in an action, use it in views (and helpers), you’re done, get back to thinking about building product.

It’s easy to iterate with this. It’s easy to see data “passed”1 into the view by scanning for the @ sigil; your editor probably does this already. It’s easy to move markup around in templates with a simple cut-paste.

If you stick with ivars long enough, you’re going to end up with two kinds of misadventures.

Most commonly, you’ll wonder why @user is nil for some view or edge case. Maybe you forgot a filter or didn’t set it in an action? Backend developers are sometimes equipped with the curiosity and knowledge to fix this themselves. For front-end developers or those new to the system, this kind of error is basically “womp-womp sad music go interrupt someone who can help you”.

This leads to the second misadventure: where did this @user thing come from2? Maybe it was set in a helper, or a filter, or an action? Well now you’ve painted yourself into a weak spot of a language like Ruby. Your tools probably can’t point you directly at the line of code where a variable came into being. You can do some clever grep’ing around3, probably. At best, you know the system well enough to get to the right file and find it, or there’s some convention you can use to intuit where it might be set.


Of course this way is better, right?

<p>Hi, my name is <%= current_user.name %></p>

Well, not initially. Already you have to pick a good name for a method because you’re probably going to use it all over the place. Then you have to find a good place to put that method: on a helper method? on a helper object? on a model?, i.e. now you’re making decisions, which is a thing Rails tries to shield you from wherever possible. Personally, I find naming things quite entertaining and hardly one of the hardest problems in computer science4.

A marginal benefit comes from reducing the entry points that the name current_user came to be a thing in this view: it’s a method name, a local variable, or a view-local variable passed into the template5. Thus the search space for “where did this thing come from” is way smaller for typical Rails templates and manageably smaller for unreasonable Rails templates.

This way pays off once the application gets past the point where new code tidily fits into models, views, controllers, or helpers. At that point, you need objects, messages, and experience at building with objects and messages (successes, stalemates, and abject failures)6. If you’re competent at messages (i.e. method calls) in Ruby, you can at this point experience a “this is Unix, I know this!”7 moment and work with this method call like you would any other method/function invocation in a computer program.

Making this investment into a method call yields another long-term benefit: simplicity in experimentation and testing8. I can poke at helper methods and helper objects in a Rails console without breaking a sweat. If that feedback loop doesn’t get the job done, I can write tests against helpers and (some) controller methods to iterate on figuring out why something doesn’t work the way I think it should.


It’s amazing that blogging about programming is so popular. Programming involves a lot of tradeoffs. Tradeoffs make for wordy articles that don’t leave you thinking “yeah, those other guys are SOOOO WRONG” or “YEAHHHH I’m so right”.

If I’ve written this properly, hopefully you felt both of those feelings. Maybe you reminisced about a day when you thought view ivars were great and then regretted it. Perhaps it’s easier to see why you’d start an app or feature off using ivars and then refactor them to method calls later.

With instance variables in views, as with many other grey areas of Rails, the most useful long view is this: you’re always right, and you’re always wrong, it just depends on which “you” (past you, present you, legacy project you, greenfield project you) is being observed.


  1. I’m not using “passed” as scare quotes here. Rails’ choice to take ivars from actions and teleport them into views is oft villified, but I find it more useful to think of them as something that just is. They are incredibly useful at first, but some developers will long for an explicit contract (i.e. a parameter list) between action and view. 
  2. If you’re an object design enthusiast, functional programming aficionado, or Rails contrarian you may be crafting an amazing retort about sharing state in weird ways at this point. Please review the previous footnote. Yes, you’re basically right. No, that’s not going to change how Rails works. 
  3. Regexes, two problems, etc. 
  4. Actual hard problems in computer science: systems connected by unreliable (i.e. any) network, laws and regulations written by humans, working with money. 
  5. When it comes to variables in Rails templates, a short guide by Adam Keys; method names: friend, ivars: foe, local variables: foe, view-local variables (passed via render explicitly or by the framework): both! 
  6. This is the concise version of a seems-really-important-to-me idea I need to express in more words. Remind me to write this, should I forget about it, internet! 
  7. What up, Ariana Richards
  8. You knew the testing thing was coming, didn’t you? I mean it’s like Chekov said about guns in stories: if method calls are mentioned in the first act, you have to mention TDD in the last act. If you headdesk’d, consider that you might be annoyed by TDD because you keep giving yourself a head injury when it’s mentioned. 

Grow and cultivate

Adding new functionality to software is really exciting. I love poking around the edges of a system, figuring out what’s going on, and looking for that “obvious” place where the first change should happen. But sometimes, it’s hard to know if I’m making the right change. How Should This Work?

The temptation when changing an existing system is to implement the desired behavior within the structure of the current abstractions. Repeat this without adjustment, and you’ll quickly end up contorting existing concepts or working around legacy behaviors. Conditionals pile up, and shotgun surgery becomes standard operating procedure.

Even if you code test first, you can make a mess of a system. What you end up with is a system that moves from local maximum to local maximum while the time of the test suite grows unbounded. There are worse things that could happen, but no one’s going to jot this one down as a “best practice” either.

The counterforce to this temptation is the red-green-refactor cycle. Look at how the system works and figure out how the next bit of work might change how the system works. Refactor to simplify the act of making a change, or make the change and refactor afterwards to a better design.

Software can grow by accretion, but it stays malleable when the team culture is one that balances continuous growth with continuous cultivation of good design.

Logs from Finland

Not that kind of log

First, read all of this excellent distillation of distributed systems by Jay Kreps,  The Log: What every software engineer should know about real-time data’s unifying abstraction. Now, consider this.

There’s a moment, when you’re building services and web APIs, when you think you’ve pretty much got it under control. You’ve got an endpoint for every query, a resource for every workflow. All the use-cases seem to be under control. And then, the question appears:

“How can I get access to all the updates to all the data? You know, for [REASONS].”

For APIs exposed to external developers over the web, this is where you’d reach for web hooks or PubSubHubbub. It’s not the best solution, but it works. If you’re building an internal system, you could  use the same approaches, or…you could build a log.

No not that kind of log. An event log, like LinkedIn did with Kafka for their internal systems. Every time your data model changes, every create, update, or delete, you drop an event with all the metadata related to the change. The event goes into some kind of single-producer, multiple consumer queue. Then all the clients that want to know about all the changes to all the things can read events off the queue and do whatever it is they need to for those important REASONS.

If you find this intriguing, this is a lot like replication in database systems. Definitely read LinkedIn’s article on this, and definitely read up on how your database of choice handles replication. And if you’ve built this before and have a good answer to initially populating “replicas” of a database, let me know; I haven’t come up with anything better than “just rsync it”.