Find your principles for editing programs

Some folks from GitHub sprung the Atom text editor on the internet yesterday. Releasing a product into a market defined by saturation, settling for brokenness, and zero-cost alternatives is a bold move. I applaud them for jumping into it. I’m eager to see where they end up, both product-wise and technically.

If you’re wondering whether Atom is the right thing for you, it might help you to know I went through a sort of quest several years ago to decide what was vital to me when editing text:

Writing these, thinking deeply about the kinds of problems I did and did not want to solve with my text editor and what that meant for my workflow, was exciting. I learned a lot about finding my own principles.

I ended up, to my great surprise, choosing Vim. Most importantly, that decision stuck. I haven’t gazed upon the possibly greener grasses of other text editors since I committed to my principles and workflow. Whether it turns out Atom is your thing or not, thinking about the principles of how you want to work with computer programs is a thing you might benefit from.

TextMate's beautiful and flawed extension mechanism

This is about how TextMate’s bundle mechanism was brilliant, but subtly flawed. However, to make that point, I need to drag you through a dichotomy of developer tools.

Composition vs. context in developer tools

What's the difference between a tool that developers work with and a tool developers often end up working against? Is there a useful distinction between tools that seem great at first, but end up loathed as time goes on? Neal Ford has ideas. Why Everyone (Eventually) Hates (or Leaves) Maven:

I defined two types of extensibility/programability abstractions prevalent in the development world: composable and contextual. Plug-in based architectures are excellent examples of the contextual abstraction. The plug-in API provides a plethora of data structures and other useful context developers inherit from or summon via already existing methods. But to use the API, a developer must understand what that context provides, and that understanding is sometimes expensive.

Composable systems tend to consist of finer grained parts that are expected to be wired together in specific ways. Powerful exemplars of this abstraction show up in *-nix shells with the ability to chain disparate behaviors together to create new things.

Ford identifies Maven and IDEs like Eclipse as tools that rely on contextual extension to get developer started with specific tasks very quickly. On the other hand, a composable tool exchange task-oriented focus for greater adaptability.

Contextual systems provide more scaffolding, better “out of the box” behavior, and contextual intelligence via that scaffolding. Thus, contextual systems tend to ease the friction of initial use by doing more for you. Huge global data structures sometimes hide behind inheritance in these systems, creating a huge footprint that shows up in derived extensions via their parents. Composable systems have less implicit behavior and initial ease of use but tend to provide more granular building blocks that lead to more eventual power.

And thus, the crux of the biscuit:

Contextual tools like Ant and Maven allow extension via a plug-in API, making extensions the original authors envisioned easy. However, trying to extend it in ways not designed into the API range in difficultly from hard to impossible...

Contextual tools are great right up to the point you hit the wall of the original developer’s imagination. To proceed past that point requires a leap of one or two orders of magnitude in effort or complexity to achieve your goal which the original developer never intended.

Bundles are beautiful

Ford wrote this as a follow-on to a piece Martin Fowler wrote about how one extends their text edior. It turns out that the extension models of popular text editors, such as VIM and Emacs, are more like composable systems than extension-based systems.

All of this is a extremely elaborate setup for me to sing the praise of TextMate. Amongst the many things it got very right, TextMate brilliantly walked the line between a nerdy programmer’s editor and an opinionated everyday tool for a wide range of developers. It did this by exposing its extension mechanism through two tools that every developer knows: scripts and regular expressions.

To add functionality to TextMate, you make a bundle. A bundle is a convention for laying out a directory such that TextMate knows the difference between a template and a syntax definition through a convention. This works because developers know how to put things in the right folder. There were only ever five or so folders you needed to know about, so this was a simple mechanism that didn’t become a burden.

To tell TextMate how to parse a language and do nifty things like folding text, you wrote a bunch of regular expressions. If I recall, there were really only a few placeholders to wedge in these regular expressions. This worked great, as most languages, though the “serious” tools use lexers and parsers, are amenable to low-fidelity comprehension with a series of naive pattern matches. The downside was that languages that didn’t look like C were sometimes odd to work with.

In my opinion, the real beauty of TextMate’s bundles was that all of the behavioral enhancement was handled with shell scripts. Plain-old Unix. You could write them in Ruby, Python, bash, JavaScript, whatever fit your fancy. As long as you could read environment variables and output text (or even HTML), you could make TextMate do new things. This led to an absolute explosion of functionality provided by the community. It was a great thing.

Downfall

Interestingly enough, TextMate is essentially a runtime for bundles. This is how VIM and Emacs are structured as well. TextMate just put a nicer interface around that bundle runtime. However, the way it did so was its downfall, at least for me.

Recall, a few hundred words ago, the difference between composable and contextual extensions. A contextual extension is easy to get going, but comes up short when you imagine something the creator of the extension point didn’t imagine. The phenomenal thing about TextMate was how well it chose the extension points and how much further those extension points took the program than what came before it. I’d estimate that about 90% of what TextMate ever needed to do, you could do with bundles. But the cost to find that last 10%, it was brutal.

Eventually, I bumped up against this limitation with TextMate. I wanted split windows, I wanted full-screen modes, I wanted better ctags integration. No one could add those (when TextMate wasn’t open source, circa 2010) because they required writing Objective-C rather than using TextMate’s extension mechanism. And so, I ended up on a different editor (after several months of wandering in a philosophical desert).

The moral of the story

If possible, you should choose a composable extension mechanism (a full-blown language, probably) and use that extension mechanism to implement your system, ala Vimscript/VIM and elisp/Emacs. If you can’t do that, you can get most of the benefit by doing a plugin API, but you have to choose the extension points really, really well.

The Cadence and Flow of Editing Programs

I figured out why my trists with other editors often end up back at TextMate. It sounds a bit like this:

Tap-tap-tap-tap-tap-tap; TAP; tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap; TAP; TAP; tap-tap-tap-tap-tap-tap; TAP.

When I’ve used vi and its descendants, it sounds like this:

Tap-tap-tap-tap-tap-tap; taptaptap; tap-tap-tap-tap-tap-tap; tapTAP TAP! tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap-tap. tapTAPTAPtapTAP TAP!

And Emacs sounds like this:

Tap-tap-tap-tap-tap-tap; tapTAPtapTAP. tap-tap-tap-tap-tap-tap;tap-tap-tap-tap-tap-tap;tap-tap-tap-tap-tap-tap; tapTAP TAP; TAP TAPtapTAPtapTAPTAP. tapTAPtapTAP!

Lest you fear I’ve created some Ook-like language for describing shortcuts in any known editor, let me explain what’s going on here.

Cadence

Emacs is, at it’s core, a Lisp machine with a text editing language wrapped around it. Every interaction with Emacs invokes a function. Handily enough, the function that adds an “a” to the file you’re editing is bound to the a key on your keyboard. Oddly enough, the function that writes the file you’re editing out to disk is bound to the combination of hitting control and x at the same time, followed by control and s at the same time. Getting them out of order matters. Control-s followed by Control-x does something entirely different.

So when you use Emacs, you type a bit, and then you run some command. Maybe you save the file, or switch to editing another file, or go to peruse a directory. So you tap for a while and then you stop tapping, move your hands every so slightly to mash the control, or alt keys and then tap some other key, usually emphatically. The most commonly used key combinations end up being hit even more emphatically. Sit in a room full of developers using Emacs, listen closely; every once in a while, you’ll here everyone save almost simultaneously and go back to a furry of lower-case tapping.

Vi is slightly different from Emacs in that it is built up from two Unix commands: one for editing single lines of text, and another for moving between said lines of text. Thus, the cadence of a vi user is slightly different. Staccato taps followed by a bang as they switch from line editing to line navigation; more staccato taps, this time oddly spaced as they move between lines and place the cursor to begin their next fury of editing; another burst of staccato text entry; a quick and emphatic tap to take them out of editing mode and then a quick but punctuated trio of taps as they invoke the command that saves the file out, a sequence of finger movements so ingrained in the vi users brain that it appears as more of a gesture than a triplet of discrete key presses.

Here’s a project idea for pranksters: stand in a room full of people using vi and Emacs, listen for the really emphatic taps, and trip the room’s breaker right before they all finish their emphatic save commands. Cackle as chaos ensues.

The space between the taps

A roomful of vi-users, Emacs-users, and TextMate users is a homogeneous mess of clackity-clackity to the untrained ear. Most accomplished programmers are touch typists, so what you’re likely to hear is an undifferentiated stream of rapid-fire tapping. But if you’ve used these editors enough, and wasted enough time thinking about the aesthetics they represent, you can hear the differences in the punctuation as commands are invoked by arcane combinations and sequences of keystrokes.

In Vi and Emacs, there is a concise sequence of keys you can mash to do a regular expression search, move down three lines, go to the second sentence on that line, and replace the word under the cursor with “bad-ass text editing programmer, do not offend”. It is, in part, this power that attracts, fascinates, and empowers their users.

TextMate can do this, sure. But there is very little in the way of support from the editor to do it. You mostly have to put your eye on the piece of text you wish to edit and use some primitive motion keystrokes to get the cursor where you want it. Then you use those same keystrokes to highlight the text to replace, this time holding down a modifier key, then you type in the text you want. TextMate, compared to its programmers editor brethren, is a language of grunts and chuffs next to the sophisticated Latin or French of vi and Emacs.

Flow

TextMate is unsophisticated next to the extensibility and conceptual unity of Emacs, or the pure practicality of vim. So why do I keep coming back to it?

It keeps me in flow.

This is a very personal answer. I’m not saying you can’t achieve a flow-state with vi or Emacs. I’m saying that while I like the idea of those editors, understand the aesthetic, and enjoy watching skilled operators using them, I get lost in the punctuation when I use them. I either forget what punctuation I should use in some text editing scenario, or I have a nagging doubt that there is some better punctuation I could be using instead.

If vi is about navigating lines and editing those lines; Emacs is about invoking Lisp functions on files containing text, then TextMate is about primitive but direct manipulation of the text in a file. There’s very little conceptual overhead. You don’t need to know how the editor is enhanced in order to understand how to operate it. You don’t need to know when to put yourself in different modes of operation to make things happen. You just think of what you want the text to look like, you move the cursor around and you type on the keyboard.

It ain’t much, but I (often) call it home.

Breaking My Habits For Editing Programs

I’m a Unix guy, by upbringing. My first formative experiences in software development were on an early, Linux 1.x version of Debian. I’d used Windows, but always came back to Linux. When OS X got good enough around 10.2, I switched to something that didn’t require so much tinkering, so I could make more useful stuff.

Software development on Unix has skewed towards focusing on tools, languages, and text editors for quite some time. IDEs and browsers on Unix are a messy, foreign thing (just like everything else in Unix). Thus, I’ve long favored the terminal-and-editor style of development.

I’ve decided that now is the time for me to try something different. I like text editors and directly manipulating text, but I can see why some people feel naked without an IDE. The ability to pop-up a level and make a more broad-stroked transformation to a program is appealing. Having code navigation and semantic awareness baked in has lots of potential.

I’ve probably said grumbly things about RubyMine in the past, but I think now is the time to give it a go. Worst thing that could happen is that I don’t like it and I go back to the infinite tinkering of Emacs or the 85% perfect experience of TextMate.

I’ll let you know how it goes.

I originally wrote that a few months ago, at the apex of my editor neurosis.

I did give RubyMine a try, and I like some parts of it. It’s code navigation is pretty nice, it does an admirable job of integrating with the unique ecosystem of tools that a Ruby developer uses to manage their environment, and it does an excellent job of grokking TDD with test/unit and RSpec. RubyMine is a step in the right direction. I suspect that if I had muscle memory for IntelliJ, it would be the way to go.

But, I have muscle memory for TextMate and Emacs, and I have an affinity for being close to my tools. RubyMine felt one step disconnected from both my muscle memory and my tools. That’s quite an accomplishment; most IDEs feel several steps removed the tools and seem to discourage developing finger-memory in favor of menu-memory. I’ll give RubyMine another try in a year, probably, see how it’s coming along. But in the mean time, it’s great to see that there is a vendor out there tackling the challenge that is tools for Ruby.

A Personal Journey in Editing Programs

Over the past year or so, I’ve spent a bit of time tinkering with text editors. It feels like I woke up one morning and was simply dissatisfied with what I was currently using. It certainly wasn’t disgust, because I largely like the tools I use. But I felt it was likely there were better tools out there, and that I should give them an honest try. The grass on the other side might be greener.

So I went through liasons with VIM and a couple with Emacs. The first left me feeling disconnected, like I had ideas but I couldn’t even make them appear on the screen, let alone run. My experiences with Emacs are better, but have always felt somewhat awkward. I suspect that at some point, I could be a full-blown Emacs person, but right now, I’m just an aspirational Emacs guy.

I’ve used a myriad of editors in my time. I started with jed, a small-ish Emacs clone, then graduated to full-on Emacs. By way of viper, I migrated to VIM. Then I picked up BBEdit, because I wanted more direct manipulation of text, and less of a never-ending learning curve. I was pretty quick to jump on TextMate, seeing a great fusion of the Mac and Unix aesthetics.

I still think TextMate is as close as it gets to an ideal situation. And yet, it falls short enough that I tinker with VIM and Emacs, two editors that can easily be labeled powerful but extremely lacking in the visual and conceptual aesthetics departments.

I suspect my mismatch with most of these editors is due to something about my habits, the way I like to work with programs, and the kinds of programs I work with. Emacs’ notions of major and minor modes doesn’t play well with markup files with three different languages embedded within. VIM is efficient for those who have internalized it and hostile to everyone else. TextMate is easy to get started with and pleasant to extend up to a point, but seems to have fallen victim to its creator’s perfectionism.

For a long time, I’ve adhered to the philosophy that one should choose one text editor and learn as much about it as possible. Increasingly, I’m starting to think that there is no panacea, no "one true editor". TextMate is great for working on web apps and easy to extend. Emacs is really wonderful for functional programming languages, especially those with REPLs and languages that are LISP-shaped. And some kinds of development demand an environment more like Smalltalk browsers than text editors.

My journey for editing bliss probably won’t end anytime soon. In the future, I suspect that it’s going to be a spectrum of tools depending on the work I’m doing. It could be that the days of personal text editing monoculture are over.

A Brief Survey of the History of Editing Programs

Software developers spend a lot of time working with code. Over the past half-century of doing so, we’ve invented a lot of mechanisms that make that task easier and reduce the amount of friction between an idea and a computer executing that idea.

A quick review of the approaches to working with code reveal some areas where tool makers have devoted a lot of effort:

  • Editors — we’ve gone from paper tape to punch cards, from flipping switches on a panel to editing files one line at a time, and finally to the point where (most of us) work with one or more files by directly manipulating the text with a combination of keystrokes. When you get down to it, the experience of editing code in Emacs, TextMate, Visual Studio, or Eclipse are quite similar.
  • Environments — many software developers today use some kind of full-screen tool that puts a friendly face on all the tools they need to create, edit and deploy their software. These days, it is most commonly an IDE like Visual Studio or XCode. But way back in the day, Smalltalk people used a clever piece of technology called a browser, and it’s not too much of a stretch to call Emacs a Lisp browser.
  • Tools — we’ve come a long way in fifty years. We started writing machine code directly, then we grew assemblers, linkers, and compilers. Today, few developers will go through their career without using a debugger, profiler, lint tool, or applying an automated refactoring. There are a lot of development tasks that can be offloaded directly to the computer, leaving the programmer free to worry about important things like why anyone would ever choose three-space tabs.

Finally, let’s not forget that the past half-century of software development has seen more than its fair share of programming languages. These languages express a myriad of ideas and the ones that combine them nicely and support their users well have left a lasting impression in the evolution of how we express our ideas so that computers can run them.

It seems we are nearing an inflection point with regard to how we use tools to create programs. As the keyboard-and-mouse give way to the display-and-finger(s), there’s an opportunity to interact with and modify programs in new ways. I suspect that the future of editing programs has something to do with merging editors with environments and making the tools as pervasively used as typing is today.

Those who think with their fingers

In the past couple of years, I’ve discovered an interesting way to think about programming problems. I try to solve them while I’m away from the computer. Talking through a program, trying to hold all the abstractions in my head, thinking about ways to re-arrange the code to solve the problem at hand whilst walking. The key to this is that I’m activating different parts of my brain to try and solve the problem. We literally think differently when we talk than when we write or type. Sometimes, the notion you need is locked up in other parts of your brain, just waiting for you to activate them.[1]

But sometimes, when I’m doing this thinking, there is something I really miss: the feel of the keyboard under my fingers and the staccato of typing. If there’s an analog to “I love the smell of napalm in the morning”, it’s “I love the sound of spring-loaded keys making codes”.

With the release of the iPad, it’s quite likely that a large percentage of the population can start to eschew the traditional keyboard and pointer that have served them in such a mediocre fashion for so long. On the other hand, you can take my keyboard from my cold dead hands. I really like typing, I’m pretty good at it, and I feel like I get a lot done, pretty quickly, when I’m typing.

Last year, I decided I would give other text editors a try. I stepped out from my TextMate happy place to try VIM. I knew this part of the experiment wasn’t going to work because when I felt like I’d gone through enough reading, tutorials and re-learning of VIM, I sat down to tap out some code. And…nothing. I felt like I was operating the editor instead of letting the code flow from my brain, through my fingertips, onto the display. It was as if I had to operate through a secondary device in order to produce code.

Sometimes it seems that developers think with their fingers. I’m not sure what the future of that is. We’ve created environments for programming that are highly optimized for using ten fingers nearly simultaneously. How will that translate to new devices that focus on direct manipulation with at most two or three fingers at a time. Will new input mechanisms like orientation and acceleration take up the slack?

Will we finally let go off the editor-compiler-runtime triumvirate? Attempts to get us out of this rut in the past have proven to be folly. I’m hoping this time around the externalities at the root of past false starts are identified and the useful essence is extracted into something really compelling.

In the mean time, it’s going to be fun trying the different ways to code on an iPad as designers and developers try new ways to make machines do our bidding.

1 If this intrigues you, read Andy Hunt’s Pragmatic Thinking and Learning it’s excellent!