All posts by Adam Keys

About Adam Keys

Telling a joke. Typing.

How to understand Saturday Night Live

Comedy, haha

Saturday Night Live is a changing thing. It’s not new like it was in the seventies, it’s not a powerhouse like it was in the nineties, it may not be the training camp for NBC sitcoms anymore. Despite that, its still a big dog in the worlds of comedy and pop culture. Every time I hear or read “SNL was better when…”, I cringe a little. As far as I can tell, this isn’t true.

Everyone’s got their favorite cast. Ferrell/Shannon/Sanz/Oteri, Fey/Poehler/Rudolph, Hartman/Carvey/Myers. It seems largely to depend on whenever you started watching SNL or when you were a teenager or in college. So when someone says “SNL isn’t relevant any more”, I mentally substitute “I liked SNL better with the cast I watched first.”

Manhattan Skyline

Over the years of watching and reading about SNL, I’ve come to understand that the show is very much about the people on camera, but Lorne Michaels is the show. The only time the show has been in consistent decline was when Michaels wasn’t around in the early eighties. For the past thirty years, claiming SNL was on the down seems to be more of a sport than a rational argument.

Since Michaels’ return, the show is subject to fractal cycles. Each night, some skits kill and some skits bomb. Generally, the front of the show is better than the back; if you stay tuned after “Weekend Update”, you should count yourself a long-time fan, willing to see some weird and/or flat skits, or asleep on the couch.

If you zoom out to look at how a season flows, you’ll again find shows that are really great and some that aren’t. My theory is that this entirely depends on the quality of the host. A mediocre host seems to bring middling material out of the writers and performers. A good or high-profile host seems to bring good-but-not-great material and pretty good performances. One of the darling hosts, like Alec Baldwin and the more recent Jon Hamm, brings the A-game material from the writers and performers play up to the occasion.

Interestingly, musical guests can bring a certain electricity too. Paul Rudd is a capable host, but pairing him with Paul McCartney led to a show that was pretty electric. I defy you to watch that episode and tell me SNL just isn’t as good as it used to be.

Zooming out to look at successive seasons, you see the same sort of up-and-down. Will Ferrell’s first season was good, but not great. He definitely left at his sketch comedy peak, and the show was briefly weaker for it. But right on his heels came the Fey/Poehler/Rudolph powerhouse. There’s an ebb and flow as casts come together, hammer out a few good seasons, and then move on to other stages.

That’s how I understand SNL. Perhaps I’m seeing cognitive biases towards the show through my own cognitive biases. I think it’s still a relevant benchmark of American pop culture.

Context is data to burst your bubbles

Design

Designing with context:

Context is a slippery topic that evades attempts to define it too tightly. Some definitions cover just the immediate surroundings of an interaction. But in the interwoven space-time of the web, context is no longer just about the here and now. Instead, context refers to the physical, digital, and social structures that surround the point of use.

Great design is built around people, not devices or software. Applying responsive design or native app UX is a tool, not a solution. Instead, we should design software that solves a problem for a real person (not a power-user or one of our colleagues) given the devices available to them and the context of use they’re in.

A high information density display is no good to a parent trying to get their kids out the door. Documentation based on video tutorials is no good for someone riding a bus. A developer troubleshooting a service bottleneck needs to know more than the average response time.

As both designers of user experiences and developers of software, we need to get away from the desk and out amongst those we’re building for. It’s too easy to build for ourselves and our friends. We need to consider how others approach and use what we make. Armed with that context, we can design a solution for everyone, and not just those we share a bubble with.

Hyperthreading illustrated

Code

I’m fond of saying hyperthreading is a lie. It’s true though; a dual hyperthreaded core is nowhere near as awesome as a four real cores. That’s more provocative than useful, so let me draw you some pictures.

If you zoom way out, a single core, dual cores, and a single hyperthreaded core look like this:

20130216-134550.jpg

Note how the hyperthreaded core is really a single core with an extra set of registers and instruction/stack pointers. The reason hyperthreading is a lie is you can’t actually run four processes, or even four threads, at the same time. At best, you can run two threads with two others ready in the wings.

I’m a dilletante of processor architecture at best, but I think I can explain why chip designers would do this.

20130216-134900.jpg

My best guess as to why you would design and release a hyperthreaded core is to increase the number of instructions you can retire (i.e. mark as completely executed) per clock cycle. Instructions retired per cycle is one of the primary metrics processor architects use for judging a design.

The enemy of instructions retired per clock cycle is memory accesses and branch mispredictions. When a processor has to go access a cache line or worse, something out in main memory, it has nothing to do but wait. When a branch (i.e. a conditional or loop) is incorrectly speculatively executed (how awesome is it that processors start executing code paths before they even know if its the right thing to do?) they end up in the same predicament. Cache misses and branch mispredictions are at best a recipe for some overhead, and at worst a recipe for waiting around on main memory.

Hyperthreading attempts to solve this problem by keeping extra programs (threads or processes) waiting in the wings, ready to start executing as soon as another pauses due to needing something from main memory. This means our execution units, the things that actually do math and execute the logic of a program, are (almost) always utilized and retiring instructions. And that gets us to our happy place of a higher instructions retired per clock cycle.

Why not just throw more execution units in and have real cores ready to work? I’m not sure, there must be something about how modern processor pipelines work that I don’t know which makes it too expensive to implement. That said, hyper threading (as I understand it) is a pretty clever hack for improving the efficiency of a processor.

TextMate’s beautiful and flawed extension mechanism

Code

This is about how TextMate’s bundle mechanism was brilliant, but subtly flawed. However, to make that point, I need to drag you through a dichotomy of developer tools.

Composition vs. context in developer tools

What’s the difference between a tool that developers work with and a tool developers often end up working against? Is there a useful distinction between tools that seem great at first, but end up loathed as time goes on? Neal Ford has ideas. Why Everyone (Eventually) Hates (or Leaves) Maven:

I defined two types of extensibility/programability abstractions prevalent in the development world: composable and contextual. Plug-in based architectures are excellent examples of the contextual abstraction. The plug-in API provides a plethora of data structures and other useful context developers inherit from or summon via already existing methods. But to use the API, a developer must understand what that context provides, and that understanding is sometimes expensive.

Composable systems tend to consist of finer grained parts that are expected to be wired together in specific ways. Powerful exemplars of this abstraction show up in *-nix shells with the ability to chain disparate behaviors together to create new things.

Ford identifies Maven and IDEs like Eclipse as tools that rely on contextual extension to get developer started with specific tasks very quickly. On the other hand, a composable tool exchange task-oriented focus for greater adaptability.

Contextual systems provide more scaffolding, better “out of the box” behavior, and contextual intelligence via that scaffolding. Thus, contextual systems tend to ease the friction of initial use by doing more for you. Huge global data structures sometimes hide behind inheritance in these systems, creating a huge footprint that shows up in derived extensions via their parents. Composable systems have less implicit behavior and initial ease of use but tend to provide more granular building blocks that lead to more eventual power.

And thus, the crux of the biscuit:

Contextual tools like Ant and Maven allow extension via a plug-in API, making extensions the original authors envisioned easy. However, trying to extend it in ways not designed into the API range in difficultly from hard to impossible…

Contextual tools are great right up to the point you hit the wall of the original developer’s imagination. To proceed past that point requires a leap of one or two orders of magnitude in effort or complexity to achieve your goal which the original developer never intended.

Bundles are beautiful

Ford wrote this as a follow-on to a piece Martin Fowler wrote about how one extends their text edior. It turns out that the extension models of popular text editors, such as VIM and Emacs, are more like composable systems than extension-based systems.

All of this is a extremely elaborate setup for me to sing the praise of TextMate. Amongst the many things it got very right, TextMate brilliantly walked the line between a nerdy programmer’s editor and an opinionated everyday tool for a wide range of developers. It did this by exposing its extension mechanism through two tools that every developer knows: scripts and regular expressions.

To add functionality to TextMate, you make a bundle. A bundle is a convention for laying out a directory such that TextMate knows the difference between a template and a syntax definition through a convention. This works because developers know how to put things in the right folder. There were only ever five or so folders you needed to know about, so this was a simple mechanism that didn’t become a burden.

To tell TextMate how to parse a language and do nifty things like folding text, you wrote a bunch of regular expressions. If I recall, there were really only a few placeholders to wedge in these regular expressions. This worked great, as most languages, though the “serious” tools use lexers and parsers, are amenable to low-fidelity comprehension with a series of naive pattern matches. The downside was that languages that didn’t look like C were sometimes odd to work with.

In my opinion, the real beauty of TextMate’s bundles was that all of the behavioral enhancement was handled with shell scripts. Plain-old Unix. You could write them in Ruby, Python, bash, JavaScript, whatever fit your fancy. As long as you could read environment variables and output text (or even HTML), you could make TextMate do new things. This led to an absolute explosion of functionality provided by the community. It was a great thing.

Downfall

Interestingly enough, TextMate is essentially a runtime for bundles. This is how VIM and Emacs are structured as well. TextMate just put a nicer interface around that bundle runtime. However, the way it did so was its downfall, at least for me.

Recall, a few hundred words ago, the difference between composable and contextual extensions. A contextual extension is easy to get going, but comes up short when you imagine something the creator of the extension point didn’t imagine. The phenomenal thing about TextMate was how well it chose the extension points and how much further those extension points took the program than what came before it. I’d estimate that about 90% of what TextMate ever needed to do, you could do with bundles. But the cost to find that last 10%, it was brutal.

Eventually, I bumped up against this limitation with TextMate. I wanted split windows, I wanted full-screen modes, I wanted better ctags integration. No one could add those (when TextMate wasn’t open source, circa 2010) because they required writing Objective-C rather than using TextMate’s extension mechanism. And so, I ended up on a different editor (after several months of wandering in a philosophical desert).

The moral of the story

If possible, you should choose a composable extension mechanism (a full-blown language, probably) and use that extension mechanism to implement your system, ala Vimscript/VIM and elisp/Emacs. If you can’t do that, you can get most of the benefit by doing a plugin API, but you have to choose the extension points really, really well.

Austin’s startup vibe

Expanded ideas

It’s different from other towns. What's the Difference between Austin and San Francisco?:

Austin offers you more options, but greater variety means that, on the whole, Austinite’s don’t focus as intensely as in San Francisco.  Austin’s defining characteristic (part of it’s slacker culture) is a belief that intensity isn’t always the best thing. Austin believes in variety and moderation. This affects the startup community. Austin, the city, will let you pick and choose from its buffet line, and then admire the smorgasbord you put together. Your lifestyle is a work of art in Austin, and I think the culture rewards you for how you live as much as what you do, often moreso.

In my few visits to San Francisco, I’ve found that I cannot wrap my Texan brain around that town. Trying to really understand its startup culture with just a few visits to the city and Palo Alto is similarly folly. But I did notice the intensity that SF has. It’s not a bad way to describe the town.

That said, I think this is a pretty decent encapsulation of Austin. Austin is a slower town (slower even than Dallas) and revels in the variety of activities available to its people. The Austin tech community is more about smaller groups and individuals too. It’s not (always) about aim-for-the-stars startups or working for large companies using large technology from large vendors. It’s as much about a few people on a team or individuals hacking something out while enjoying their city, family, and friends.

Obviously, I dig that a lot.

Update: I should mention that, while it’s popular to write Austin off as a slacker town, there’s a lot of people dedicated to their work and craft here. It’s not all tacos and BBQ. The events I go to most often are frequented by people who using their evenings to learn something new or talk shop while they’re making something. That is, I think, the most important factor of a startup community: the more people who are putting their evenings into making things, the more likely those things will end up awesome and grow into a business-like organism.

Senior VP Jean-Luc Picard, of the USS Enterprise (Alpha Quadrant division)

Nerdery

If you’re working from the Jean-Luc Picard book of management, a nice little Twitter account of Picard-esque tips on business and life, we can be friends. Consider:

Picard management tip: Be willing to reevaluate your own behavior.

And:

Picard diplomacy tip: Fighting about economic systems is just as nonsensical as fighting about religions.

But I’m not so sure about this one:

Picard management tip: Shave.

If you’re playing from home, the fictional characters that have most influenced my way of thinking are The Ghostbusters (all of them) and Jean-Luc Picard. I also learned everything I need to know about R&B from The Blues Brothers.

SoundCloud, micro-services, and software largeness

Code

From a monolithic Ruby on Rails app to the JVM, how Soundcloud has transitioned to a hybrid approach with Ruby apps intermingling with Scala and Clojure apps. I think some of their idea of what is idiomatic Rails and how to operate Ruby are not exactly on center. But, their approach to the problem of a large Rails app is right on: break it up into “micro-services” that, if you don’t like the code, you can rewrite quickly if necessary.

Lest you fear this is yet another “Rails doesn’t scale!” deck, they do make a key observation. “Rails, PHP, etc. are a very good choice to start something”. Once you get past “starting” and “growing” to “successful and challenging”, you’ll face the same level of challenge no matter what you choose: Ruby or Java, MySQL or Riak. All the technologies we have today are challenged when they grow large.

So don’t let applications and services get large. Easy to say; hard, but worthwhile, to practice.

Those Who Make, by hand

Erudition

Those Who Make is a series about people who craft. Physical things, by hand, that don’t come out the same every time. I love watching people make things, and I doubly love hearing their passion for whatever it is they’re making. Even more enlightening, this is a very international series. It’s not all hipster shops in San Francisco, Portland, and Brooklyn; it’s everywhere.

This is delightful stuff.

How coffee is made in a colorful shop in another country, shot in the “Vimeo style” (is this a thing?): that will always get me.

Thoughts on “Being a Senior Engineer”

Expanded ideas

On Being a Senior Engineer made the rounds late last year. Before I finished reading it, I felt it was pointing me down a path I hadn’t realized was there but needed to go down. It’s the kind of “yes, this!” writing that I often end up ineptly giving people a link to without the ability to explain why they should care or how amazing it is.

I chewed on the original article for a few months, following the links, re-reading it. Basically, I’m trying to completely consume this idea of the responsibilities and abilities of a mature engineer. Below, a bunch of quotes that struck a chord with me and follow-up ideas.

I expect a “senior” engineer to be a mature engineer.

Mature engineers seek out constructive criticism of their designs.

Here’s an example of how I try to apply this: attempt to hold all the options (designs, causes, etc.) in your head. This is doubly important if you have identified a design or project plan as infeasible, but it appeals to those who don’t have the whole thing in their head. Empathy and understanding of other points of view is crucial.

Being able to write a Bloom Filter in Erlang, or write multi-threaded C in your sleep is insufficient. None of that matters if no one wants to work with you.

A thousand times yes! I have often felt that internet culture lionizes those who are quick and merciless in cutting down those that don’t agree or can’t code as prodigiously as the original poster. A senior/mature engineer is not soley defined by typing the most code per day.

Be the engineer that everyone wants to work with.

Please, if you ever see me not being that engineer, tell me!

…they have a responsibility to others to make themselves interpredictable. In general, mature engineers are comfortable with working within some nonzero amount of uncertainty and risk.

This is from a section on making estimates. It’s hard to make estimates, because they feel like binding contracts. If you’re working with the right people, it’s OK, they’re not a contract. Make a guess and help others you work with understand the level of entropy involved in your project reaching a milestone at a specific date.

This code looks good, I’m proud of myself. I’ve asked other people to review it, and I’ve taken their feedback. Now: how long will it last before it’s rewritten? Once it’s in production, how will its execution affect resource usage? How much so I expect CPU/memory/disk/network to increase or decrease? Will others be able to understand this code? Am I making it as easy as I can for others to extend or introspect this work?

  1. The only time is runtime, but a lot of developers focus on the static, build-time properties of their code.
  2. As a corollary, developers become the experts at the “hypothetical” of their code, and the ops team become the experts at the “practical” of their code. This isn’t a good division of labor.

Generosity of spirit is one of our core engineering values, but also a primary responsibility of our Staff Engineer position, a career-level position. These engineers spend the time to make sure that more junior or new engineers unfamiliar with the tech or processes we have not only understand what they are doing, but also why they are doing it.

I’ve found it challenging that I’m so far removed from the struggles of a junior developer that in some ways I don’t even comprehend them anymore. Trying to help those who have come up through Hungry Academy, even just a little, has paid dividends in understanding “junior” programmers and more experienced developers who don’t have my experiences.

They know that they work within a spectrum of ideal and non-ideal, and are OK with that. They are comfortable with it because they strive to make the ideal and non-ideal in a design explicit.

Again: hold all the things in your head, even though you take only one path. For now. It’s software you can and will change your mind.

Further: write software such that doing the right thing is easy, the wrong thing is hard, and amending the shortcomings is possible at a later time.

Being empathetic in this sense means having the ability to view the project from another person’s perspective and to take that into consideration into your own work.

Hold all the people, and their conflicting goals, in your head too. Isn’t engineering fun?

…never go to your boss with a complaint about anything without at least one (ideally more than one) suggestion for a solution. Even demonstrating that you’ve tried working the problem on your own and came up empty-handed is better than an empty complaint.

There will always be things that suck. Complaining about them feels good! Proposing, advocating, and working on solutions is better.

The issue with cognitive biases is that we can be blissfully unaware of when we are interpreting data with our own brains in ways that defy empirical data, and can have a surprising effect on how we get work done and work on teams.

For every time I wonder what cognitive bias I’m currently exhibiting, I’m sure there’s two more times when I have no idea. His list of biases is well worth reading into.

Ten Commandments of Egoless Programming. Yes.

How people feel about technologies, technical decisions, and technical directions is just as important (if not more) than the facts about the details.

People are irrational. Work with it. People have scars learnt from bad experiences. Deal with it. Everyone has succeeded in different ways and made the right and wrong inferences from it. Listen when people talk and speak to what they are excited and concerned about.

Feynman’s mess of jiggling things

Nerdery

Richard Feynman, in the process of explaining rubber bands:

The world is a dynamic mess of jiggling things, if you look at it right!

This simplification delights and amuses me. The great thing is its fractal truth: you can observe our lives at many levels and conclude that they are dynamic jiggling messes.