Like caveats? Try writing about leading teams!

It’s tricky to write about leading software teams. Herein, reflections, not complaints, on pursuing higher software leadership truths. Many of which are riffs on 4 Reasons Writing About Software is Hard:

Writing is actually an incredibly relevant skill for engineering leadership (and engineering in general), but it’s still hard. You can have all the insights in the world, and still struggle to convey your message or find the right audience.

First off, writing is hard so writing about software is hard too. Developing our thoughts beyond “that sounded nice in the shower” is hard but rewarding work.

It’s difficult to translate “this worked” out of the system of people, circumstances, and goals. As noted above, at one scale everything works, at another basically nothing works.

No advice on software development is universal. From the smallest coding details to how we structure our multi-person/week projects, there are no best practices. There’s only “this worked/failed for us when building this particular project”.

Writing about software leadership ends up being a lot of describing the people and scenarios that led to a successful approach. The trick is that getting down to particulars about people and scenarios is either too personal to share on the internet or too specific/proprietary to make sense outside a specific organization. It’s hard to write high-quality leadership ideas without drowning in setup, hedging, or over-generic characters.

It’s tempting but insufficient to suggest “act as a good person would and most things will work out”. Merely acting as a good person does will not get me out of situations where individual best intentions created bad outcomes. Perhaps specifying what a good person does with enough clarity that another person can apply and/or emulate it is a laudable step.

In practice, it seems better to say “act as a good person 95% of the time but do the minimally jerk-y thing 5% of the time to cut Gordian knots created by good intention”. I’m not happy with that trade-off, but it does seem like a necessary part of leading people. I build context so I can trade accumulated trust for impact when the situation requires it. (Hopefully! I often don’t know if I was spending from a surplus or deficit of trust until much later.)

Benchmarking Rails apps in 5 bullets

  1. When in doubt, measure. Twice!
  2. For ad-hoc/napkin estimates, I use Benchmark.ms { …the code… } to size up the performance of Ruby code.
  3. When I want to do The Science to compare approaches, I use benchmark-ips. It works a lot like Benchmark, but does all the cold start, iteration, and math for you. It’s great, thanks Evan!
  4. When it comes to code that interacts with databases (Postgres, ElasticSearch, HTTP APIs, Redis, etc.) it is almost always the case that one big query is far faster than queries inside a loop (e.g., N+1 queries)
  5. Ruby performance is often limited by creating many objects and the time it takes for the garbage collector to find/free them up afterwards. This is sometimes not the case in recent Ruby versions (see #1).

Bonus useful tools:

  • bumbler – profile loading gems from your Gemfile at application boot. In most Rails apps, there are several seconds of savings to find in lazy loading rarely loaded libraries.
  • active-record-query-trace: shows the call-site and last few stack frames for every query in your development log. Super handy for “where is this blasted query coming from?”

The Beatles 🤝 Timeless leadership lessons

The Economist, The Beatles and the art of teamwork:

Take the role of Ringo, for example. When he is not actually playing, the band’s drummer spends most of his time either asleep or looking bewildered. When the other three musicians bicker, Ringo smiles beatifically. To a casual observer, he might appear dispensable. But musically, nothing works without him, and as a team member he softens conflict and bridges divides.

More teams need Ringo(s), those who make it work without soaking up emotional energy.

Managers who think that building esprit de corps requires a separate activity from work—here-comes-the-fun time, set aside for axe-throwing or gif battles or something equally ghastly—are missing a fundamental point. The highest-performing teams derive the greatest satisfaction not from each other, but from the work they do together.

The Beatles weren’t excellent due to vacationing together in India. They were exceptional because of what they built together, both because of and sometimes despite each other. So it goes with teams.

Better know a standard library

Read your current/new language’s standard library. Highly recommended for developers of all experience levels. You’ll pick up the idioms, you’ll discover something useful. You’ll laugh, you’ll cry, you’ll wonder if they’re making a sequel!

My favorites are prelude.hs (which, sadly, does not seem to be the file that Haskell bootstraps itself from any more 🤔) and Rubinius’ implementation of Enumerable in Ruby.

(The reference is from The Colbert Report, which is a fun-but-dated thing to know.)

Offloading fast operations in Ruby by data structure

Noteflakes: A Compositional Approach to Optimizing the Performance of Ruby Apps — the idea is to offload “inner-loop”-type operations from Ruby to C-extensions. The clever twist is this happens via data-structure-as-language. Ruby being Ruby, you can wrap a DSL around the data structure generation to reduce the context switch from Ruby to offloaded operations.

There’s precedent to the approach: if you squint, it’s not unlike offloading the math for computer graphics or machine learning to a GPU. That said, the speed-up is unlikely to be as dramatic.

I hope to hear more of this approach in the future!

Adjacent: “it’s wild how much of the 2021 programming ecosystem is declarative data structures evaluated by recursive functions.”

Don’t be spooky

It’s possibly the best advice for managers I’ve given so far. When you’re communicating with your team, lead with context and reassurance. Never message someone on your team, “let’s talk when you get a minute”. That’s void of information and scary as heck!

I have to remind myself of this when I’m rushing. It’s faster to ping someone to arrange a synchronous talk than it is to write out what I need to say and cover all the bases. But that doesn’t give me license to skip all the context. Broad strokes are okay. An information vacuum is not okay.

Accidental spookiness invites story-crafting. Minds race. Lacking information or context, we tell stories. They often aren’t happy stories, regardless of how good your relationship with the team. We humans are better at convincing ourselves to fear something (survival instinct) than the other way around.

Avoiding spookiness reduces the chance of people telling themselves negative stories. Context and clarity counteract reading the tea leaves and world building. Even more important, it prevents people from pre-gaming the conversation. That way, they don’t prepare for a conversation that happened in their heads, instead of one that’s about to happen. (Avoiding pre-gaming is important on both sides of the conversation, as it turns out.)

A corollary to “don’t be spooky” — deliver constructive but critical feedback as close to the “original sin” as possible. Receiving feedback that you did poorly weeks after the fact is disconcerting. It can lead the recipient to wondering what other things they’re doing poorly but won’t hear about until later. Which leads to story-crafting, and the whole negative cycle starts a-new.

Give your team enough context to pre-game conversation based on the real context, not conjecture. And don’t hold on to feedback for “that perfect time”.

Would you pay more for a noisy computer?

Computers should expose their internal workings as a 6th sense, Matt Webb:

I kinda miss the days when I could hear the hard drive of my computer. If it was taking a while to response (say, when opening a big file), there was a difference between the standard whirr chugga chugga ch-ch-ch chugga seek pattern, and a broken kik kik kik. And you’d have an idea how long loading a file from disk should take, versus the silent “thinking” time afterwards.

The point is not the sound. You barely noticed the sound.

The point is that you felt you like were in psychic communion with the workings of the computer.

1.

Watch and car enthusiast/collectors go wild for these qualities. They go on and on about the shape of the Jaguar E-Type, the roar of an American V-8 engine, the howl of a Formula 1 V-10 engine, the smell of new watch strap leather, the classic looks of an Omega Speedmaster or Rolex Daytona.

And then there’s the “save the manuals” crowd! Despite knowing that modern transmissions are more effective than any human, they desire the involvement of controlling a major interaction between the car’s engine and the rubber meeting the road.

Another crowd revels in and/or yearns to own one of the “last analog” of some sort. The Last Analog Ferrari/Porsche/BMW/etc. It’s a desire for a sensory experience at the expense of outright speed/efficiency, a True Scotsman argument, and a lot of nostalgia. It’s also nice that “analog” cars can’t become haunted my misbehaving computers or software.

2.

Computers (and their distant cousins, the electric car and smartwatch) are left only to simulate these qualities. A smartwatch could display the same information as an Omega Speedmaster and replicate its design down to pixel perfection. It will never catch the light like a physical watch will. Porsche’s Taycan electric car makes its own kind of howl (if you pay slightly more for the option), but it pales to any era of their gasoline engines.

A lot of these watch and car qualities come down to nostalgia. On paper, a CO2 producing, endlessly vibrating machine full of moving parts that are going to break is inferior to the electric car designs of 2021. What we need is a new sort of romance as we transition from moving parts to entirely solid state machines.

Computers could make us feel something. They should tap us once if something works, or tap us a few times when something needs our attention. We should be able to tell they’re working hard, or hardly working, by the sounds of their insides clicking, whirring, or churning. Data and files that have gone untouched for a while should have a faded patina.

In particular, there’s room for moving from Apple Store austerity/minimalism to color and form-follows-function ornamentation.

3.

Cultured Code’s Things is my go-to example. They’ve gone to the trouble of re-implementing large swaths of the UI toolkit Apple provides on their platforms. The attention to detail shines through. Every action has the slightest haptic feedback. Swiping and tapping through the UI has a hint of physical heft, as though there’s a bit of momentum to scrolling around, opening up a task, or flipping over to the navigation “sidebar”. It’s the best use of animation I’ve seen on the platform. It’s still quite austere, but at least there’s a feeling that it’s not a UI floating in Jony Ive’s featureless, zero-gravity void!

I suppose the jury’s out whether patina, “this is my computer, none are like it”, and collectability are out the window too. Maybe this whole crypto-token thing will bear fruit without extending humanity’s use of coal power far beyond its past-due date.

4.

There’s another angle here. To continue the car metaphor, most people don’t want a sports car or muscle car. They don’t even want an SUV with a muscle-car engine. They want something reliable, convenient, and affordable.

Folks mostly want appliances. Solve this problem for me with a minimum of showmanship or drama. That is, they want a quiet and unassuming car or computer. Normal operation should make no sound at all. Worrisome noises should only happen if things worsened after warning the owner and the machine remained quiet.

Solid state computers and cars are unavoidable, as futures go. There’s no reason to go back to combustion, cooling fans, and extensively spinning disks. Computers as appliance are largely here and largely preferred. Watches are in the midst of that transition. Cars are next on the wave of change.

5.

Maybe this all comes down to computers-as-invisible-fabric-of-life vs. computers-as-central-tool. Glass walls that show off the mainframe/server/datacenter room (not unlike glass engine covers on McLaren/Ferrari/Lamborghini supercars!) vs. servers sequestered behind anonymous drywall or offsite completely “in the cloud”.

Someday, we’ll get past computers as we know them now. Then we’ll have “computer-punk”. Like steampunk before it, we’ll imagine life if we’d advanced in knowledge and ethics but not technology.

Probably, if I see that future, I’ll have some fleet of computers, mostly invisible, which I use as my “daily driver” for getting stuff done. But if current trends continue, I’ll spend a few hours a week enjoying a “weekend computer” that is clack-y, slightly anachronistic or nostalgic, and gets noticeably huffy when I ask too much of it.

Math-y and/or word-y

I’m a developer who (formerly) recoiled at math, especially calculus and matrices.

Instead, I thought, I loved the language-y parts of software development. Programming languages, little languages, domain-specific languages. Designing the names, concepts, and relationships in APIs. Domain-driven design, jargon, modeling. I thought of myself as more of a writer-y developer than a math-y developer.

I suspect that I’m not alone amongst programmers in favoring liberal arts over sciences. Further, I suspect that over the past two decades the number of programmers with science-y backgrounds has steadily declined.

But, it turns out I do a lot more math than I thought! SQL is set theory. Layout is math (mostly about centering things 🤷). State machines reduce many kinds of logic problems to simple, but math-y, flowcharts. Almost every bit of programming one does is logic, which is, you guessed it, math.

And that only covers web apps! Computer graphics is a heck of a lot of trigonometry (and whatever is involved in calculating diffusion of light). Machine learning is lots of statistics plus a little calculus. Type checking in compilers is category and/or set theory.

The flip side is also true: developers who think they only enjoy the math-y parts do a lot more language/liberal artsy stuff than they think. They’ve got opinions about language constructs, whether a method name is good or not, if an API is easy to use correctly or not. All “soft” non-math-y constructs!

When it comes down to it, it’s more likely that poor math education let me down than I am fundamentally not a math person. Related: Why Math Class Is Boring—and What to Do About It:

There are two types of people in the world: those who enjoyed mathematics class in school, and the other 98% of the population.

“Rationalize and solve” doesn’t help someone who is venting

If you’re doing the whole servant leadership thing, you’re gonna hear some people venting frustrations. Yihwan Kim, When a 1:1 turns into a vent session:

As an engineering manager, I’m learning that a big part of my job (perhaps my only job) is to help people solve problems. I happen to enjoy solving problems myself. So it’s only natural that when someone starts venting, I want to rationalize the conversation, correct inaccuracies, and discuss actionable next steps.

I always have to remind myself: don’t.

Relatable. It’s easy to react to someone venting by rationalizing and solving. As Yihwan points out, that’s not the card to play here. The win condition for these is to 1) hear your colleague out and 2) help improve the cause of the vent if your teammate wants you to.