The paradox of producing process

The agency to create the system or process you want to work in (axiomatically?) implies you’ll rarely get to work in the system because you’re spending a lot of time working on the system by communicating/iterating/supporting it. 🤷🏻‍♂️


Like caveats? Try writing about leading teams!

It's tricky to write about leading software teams. Herein, reflections, not complaints, on pursuing higher software leadership truths. Many of which are riffs on 4 Reasons Writing About Software is Hard:

Writing is actually an incredibly relevant skill for engineering leadership (and engineering in general), but it’s still hard. You can have all the insights in the world, and still struggle to convey your message or find the right audience.

First off, writing is hard so writing about software is hard too. Developing our thoughts beyond “that sounded nice in the shower” is hard but rewarding work.

It's difficult to translate "this worked" out of the system of people, circumstances, and goals. As noted above, at one scale everything works, at another basically nothing works.

No advice on software development is universal. From the smallest coding details to how we structure our multi-person/week projects, there are no best practices. There’s only “this worked/failed for us when building this particular project”.

Writing about software leadership ends up being a lot of describing the people and scenarios that led to a successful approach. The trick is that getting down to particulars about people and scenarios is either too personal to share on the internet or too specific/proprietary to make sense outside a specific organization. It’s hard to write high-quality leadership ideas without drowning in setup, hedging, or over-generic characters.

It’s tempting but insufficient to suggest “act as a good person would and most things will work out”. Merely acting as a good person does will not get me out of situations where individual best intentions created bad outcomes. Perhaps specifying what a good person does with enough clarity that another person can apply and/or emulate it is a laudable step.

In practice, it seems better to say “act as a good person 95% of the time but do the minimally jerk-y thing 5% of the time to cut Gordian knots created by good intention”. I’m not happy with that trade-off, but it does seem like a necessary part of leading people. I build context so I can trade accumulated trust for impact when the situation requires it. (Hopefully! I often don’t know if I was spending from a surplus or deficit of trust until much later.)


Benchmarking Rails apps in 5 bullets

  1. When in doubt, measure. Twice!
  2. For ad-hoc/napkin estimates, I use Benchmark.ms { …the code… } to size up the performance of Ruby code.
  3. When I want to do The Science to compare approaches, I use benchmark-ips. It works a lot like Benchmark, but does all the cold start, iteration, and math for you. It’s great, thanks Evan!
  4. When it comes to code that interacts with databases (Postgres, ElasticSearch, HTTP APIs, Redis, etc.) it is almost always the case that one big query is far faster than queries inside a loop (e.g., N+1 queries)
  5. Ruby performance is often limited by creating many objects and the time it takes for the garbage collector to find/free them up afterwards. This is sometimes not the case in recent Ruby versions (see #1).

Bonus useful tools:

  • bumbler - profile loading gems from your Gemfile at application boot. In most Rails apps, there are several seconds of savings to find in lazy loading rarely loaded libraries.
  • active-record-query-trace: shows the call-site and last few stack frames for every query in your development log. Super handy for “where is this blasted query coming from?”

The Beatles 🤝 Timeless leadership lessons

The Economist, The Beatles and the art of teamwork:

Take the role of Ringo, for example. When he is not actually playing, the band’s drummer spends most of his time either asleep or looking bewildered. When the other three musicians bicker, Ringo smiles beatifically. To a casual observer, he might appear dispensable. But musically, nothing works without him, and as a team member he softens conflict and bridges divides.

More teams need Ringo(s), those who make it work without soaking up emotional energy.

Managers who think that building esprit de corps requires a separate activity from work—here-comes-the-fun time, set aside for axe-throwing or gif battles or something equally ghastly—are missing a fundamental point. The highest-performing teams derive the greatest satisfaction not from each other, but from the work they do together.

The Beatles weren’t excellent due to vacationing together in India. They were exceptional because of what they built together, both because of and sometimes despite each other. So it goes with teams.


Better know a standard library

Read your current/new language’s standard library. Highly recommended for developers of all experience levels. You’ll pick up the idioms, you’ll discover something useful. You’ll laugh, you’ll cry, you’ll wonder if they’re making a sequel!

My favorites are prelude.hs (which, sadly, does not seem to be the file that Haskell bootstraps itself from any more 🤔) and Rubinius’ implementation of Enumerable in Ruby.

(The reference is from The Colbert Report, which is a fun-but-dated thing to know.)


Offloading fast operations in Ruby by data structure

Noteflakes: A Compositional Approach to Optimizing the Performance of Ruby Apps — the idea is to offload “inner-loop”-type operations from Ruby to C-extensions. The clever twist is this happens via data-structure-as-language. Ruby being Ruby, you can wrap a DSL around the data structure generation to reduce the context switch from Ruby to offloaded operations.

There’s precedent to the approach: if you squint, it’s not unlike offloading the math for computer graphics or machine learning to a GPU. That said, the speed-up is unlikely to be as dramatic.

I hope to hear more of this approach in the future!

Adjacent: “it’s wild how much of the 2021 programming ecosystem is declarative data structures evaluated by recursive functions.”


Don’t be spooky

It’s possibly the best advice for managers I've given so far. When you’re communicating with your team, lead with context and reassurance. Never message someone on your team, "let's talk when you get a minute". That's void of information and scary as heck!

I have to remind myself of this when I'm rushing. It's faster to ping someone to arrange a synchronous talk than it is to write out what I need to say and cover all the bases. But that doesn’t give me license to skip all the context. Broad strokes are okay. An information vacuum is not okay.

Accidental spookiness invites story-crafting. Minds race. Lacking information or context, we tell stories. They often aren’t happy stories, regardless of how good your relationship with the team. We humans are better at convincing ourselves to fear something (survival instinct) than the other way around.

Avoiding spookiness reduces the chance of people telling themselves negative stories. Context and clarity counteract reading the tea leaves and world building. Even more important, it prevents people from pre-gaming the conversation. That way, they don’t prepare for a conversation that happened in their heads, instead of one that’s about to happen. (Avoiding pre-gaming is important on both sides of the conversation, as it turns out.)

A corollary to “don’t be spooky” — deliver constructive but critical feedback as close to the “original sin” as possible. Receiving feedback that you did poorly weeks after the fact is disconcerting. It can lead the recipient to wondering what other things they’re doing poorly but won’t hear about until later. Which leads to story-crafting, and the whole negative cycle starts a-new.

Give your team enough context to pre-game conversation based on the real context, not conjecture. And don’t hold on to feedback for “that perfect time”.


Would you pay more for a noisy computer?

Computers should expose their internal workings as a 6th sense, Matt Webb:

I kinda miss the days when I could hear the hard drive of my computer. If it was taking a while to response (say, when opening a big file), there was a difference between the standard whirr chugga chugga ch-ch-ch chugga seek pattern, and a broken kik kik kik. And you’d have an idea how long loading a file from disk should take, versus the silent “thinking” time afterwards.

The point is not the sound. You barely noticed the sound.

The point is that you felt you like were in psychic communion with the workings of the computer.

1.

Watch and car enthusiast/collectors go wild for these qualities. They go on and on about the shape of the Jaguar E-Type, the roar of an American V-8 engine, the howl of a Formula 1 V-10 engine, the smell of new watch strap leather, the classic looks of an Omega Speedmaster or Rolex Daytona.

And then there’s the “save the manuals” crowd! Despite knowing that modern transmissions are more effective than any human, they desire the involvement of controlling a major interaction between the car’s engine and the rubber meeting the road.

Another crowd revels in and/or yearns to own one of the “last analog” of some sort. The Last Analog Ferrari/Porsche/BMW/etc. It’s a desire for a sensory experience at the expense of outright speed/efficiency, a True Scotsman argument, and a lot of nostalgia. It’s also nice that “analog” cars can’t become haunted my misbehaving computers or software.

2.

Computers (and their distant cousins, the electric car and smartwatch) are left only to simulate these qualities. A smartwatch could display the same information as an Omega Speedmaster and replicate its design down to pixel perfection. It will never catch the light like a physical watch will. Porsche’s Taycan electric car makes its own kind of howl (if you pay slightly more for the option), but it pales to any era of their gasoline engines.

A lot of these watch and car qualities come down to nostalgia. On paper, a CO2 producing, endlessly vibrating machine full of moving parts that are going to break is inferior to the electric car designs of 2021. What we need is a new sort of romance as we transition from moving parts to entirely solid state machines.

Computers could make us feel something. They should tap us once if something works, or tap us a few times when something needs our attention. We should be able to tell they’re working hard, or hardly working, by the sounds of their insides clicking, whirring, or churning. Data and files that have gone untouched for a while should have a faded patina.

In particular, there’s room for moving from Apple Store austerity/minimalism to color and form-follows-function ornamentation.

3.

Cultured Code’s Things is my go-to example. They’ve gone to the trouble of re-implementing large swaths of the UI toolkit Apple provides on their platforms. The attention to detail shines through. Every action has the slightest haptic feedback. Swiping and tapping through the UI has a hint of physical heft, as though there’s a bit of momentum to scrolling around, opening up a task, or flipping over to the navigation “sidebar”. It’s the best use of animation I’ve seen on the platform. It’s still quite austere, but at least there’s a feeling that it’s not a UI floating in Jony Ive’s featureless, zero-gravity void!

I suppose the jury’s out whether patina, “this is my computer, none are like it”, and collectability are out the window too. Maybe this whole crypto-token thing will bear fruit without extending humanity’s use of coal power far beyond its past-due date.

4.

There’s another angle here. To continue the car metaphor, most people don’t want a sports car or muscle car. They don’t even want an SUV with a muscle-car engine. They want something reliable, convenient, and affordable.

Folks mostly want appliances. Solve this problem for me with a minimum of showmanship or drama. That is, they want a quiet and unassuming car or computer. Normal operation should make no sound at all. Worrisome noises should only happen if things worsened after warning the owner and the machine remained quiet.

Solid state computers and cars are unavoidable, as futures go. There’s no reason to go back to combustion, cooling fans, and extensively spinning disks. Computers as appliance are largely here and largely preferred. Watches are in the midst of that transition. Cars are next on the wave of change.

5.

Maybe this all comes down to computers-as-invisible-fabric-of-life vs. computers-as-central-tool. Glass walls that show off the mainframe/server/datacenter room (not unlike glass engine covers on McLaren/Ferrari/Lamborghini supercars!) vs. servers sequestered behind anonymous drywall or offsite completely “in the cloud”.

Someday, we’ll get past computers as we know them now. Then we’ll have “computer-punk”. Like steampunk before it, we’ll imagine life if we’d advanced in knowledge and ethics but not technology.

Probably, if I see that future, I’ll have some fleet of computers, mostly invisible, which I use as my “daily driver” for getting stuff done. But if current trends continue, I’ll spend a few hours a week enjoying a “weekend computer” that is clack-y, slightly anachronistic or nostalgic, and gets noticeably huffy when I ask too much of it.


Math-y and/or word-y

I'm a developer who (formerly) recoiled at math, especially calculus and matrices.

Instead, I thought, I loved the language-y parts of software development. Programming languages, little languages, domain-specific languages. Designing the names, concepts, and relationships in APIs. Domain-driven design, jargon, modeling. I thought of myself as more of a writer-y developer than a math-y developer.

I suspect that I’m not alone amongst programmers in favoring liberal arts over sciences. Further, I suspect that over the past two decades the number of programmers with science-y backgrounds has steadily declined.

But, it turns out I do a lot more math than I thought! SQL is set theory. Layout is math (mostly about centering things 🤷). State machines reduce many kinds of logic problems to simple, but math-y, flowcharts. Almost every bit of programming one does is logic, which is, you guessed it, math.

And that only covers web apps! Computer graphics is a heck of a lot of trigonometry (and whatever is involved in calculating diffusion of light). Machine learning is lots of statistics plus a little calculus. Type checking in compilers is category and/or set theory.

The flip side is also true: developers who think they only enjoy the math-y parts do a lot more language/liberal artsy stuff than they think. They've got opinions about language constructs, whether a method name is good or not, if an API is easy to use correctly or not. All "soft" non-math-y constructs!

When it comes down to it, it's more likely that poor math education let me down than I am fundamentally not a math person. Related: Why Math Class Is Boring—and What to Do About It:

There are two types of people in the world: those who enjoyed mathematics class in school, and the other 98% of the population.


“Rationalize and solve” doesn’t help someone who is venting

If you’re doing the whole servant leadership thing, you’re gonna hear some people venting frustrations. Yihwan Kim, When a 1:1 turns into a vent session:

As an engineering manager, I’m learning that a big part of my job (perhaps my only job) is to help people solve problems. I happen to enjoy solving problems myself. So it’s only natural that when someone starts venting, I want to rationalize the conversation, correct inaccuracies, and discuss actionable next steps.

I always have to remind myself: don’t.

Relatable. It’s easy to react to someone venting by rationalizing and solving. As Yihwan points out, that’s not the card to play here. The win condition for these is to 1) hear your colleague out and 2) help improve the cause of the vent if your teammate wants you to.


One priority is like wind in the sails

It’s true that I can scale myself, teams, and organizations to walk and chew gum at the same time, but it is surprisingly effective to focus on one thing at a time. This is the essence of “priority” — put all my energy into one outcome until it’s done. Then the next one, the next one, etc. as my efforts start to compound.

I, like many folk, do much of my best work in coffee mode. When that deep, coffee mode work aligns with my priority, everything is operating smoothly and life is good. If my priority (singular) changes and I need to go deep on something else, that’s not ideal but not so bad either. As long as I can still focus, things are good. When I’m asked to go deep on two things or pulled to work on deep but unaligned tasks, that’s when things get gnarly.

A useful activity that looks like (and is often called) prioritizing is sorting (“triaging”) a list of potential work by what’s most impactful or important. This is more of a planning activity than a priority exercise. It acknowledges there’s a lot to do and time is finite. The result sends a clear signal: this thing at the top of the list is more important than all the things below it. In particular, any of the top items are more useful to work on than all the things further down the list.

Get better at finishing projects. That’s working smarter, not harder. Finished projects, axiomatically, don’t need prioritizing against other work. Limiting work in progress is difficult to pull off in the moment but a crucial tactic to apply when things get intense.

Possibly controversial: multiple priorities is about the same as having no priorities. The trick that priorities pull is freeing us of the energy-sapping process of deciding what’s most important and what trade-offs to make. A single priority is a note from your past, well-considered self saying “this is more important than all the other things; work on it before anything else”.

Systems of “one priority”:

Multiple priorities make it unclear what expectations are for individuals. Choosing what to work on becomes difficult and a burden. A single priority is a mission, a clarity of work. Get this thing done, declare victory, move on.


Planning focuses our ideas

Planning is essential. But, not too much. Mostly in the next 90-day window (with apologies to Michael Pollan).

Humans are, with few exceptions, awful at planning. It’s impossible to see the future. We rely on our previous experience over data too often. Or, not enough. Or, in the wrong combination for this scenario. Beyond a few days, the world we operate in is too complex, people too hard to predict, and all of it is interconnected in surprising ways.

Even worse, humans easily deceive themselves with plans. It’s so easy to look at basically any kind of ambition or outcome and say “yeah sure, given 6-18 months this seems totally feasible.” (With apologies to our ambition, it probably is not feasible.)

Yet when we account for those hazards, planning is essential (apologies to Dwight Eisenhower). Most things won’t go to plan, but making one forces us to think things through, ahead of time. Outcomes without a plan are worse than outcomes from a plan that has to change when reality punches us in the face (with apologies to Mike Tyson).

I find that periodically looking 90 days into the future to think about what I want to focus on and outcomes I hope to realize is a pretty dang good way of setting myself up for “luck favors the prepared”.

Planning the work is an essential part of doing the work. An ambitious but uncertain 6-month idea becomes an ambitious but plausible plan in steps. Make a 90-day plan by breaking it into chunks. Organize them into a coherent fraction of the 6-month idea. Make trade-offs to decide what’s most important or risky. Start on the first thing. Rinse and repeat.

Iteration is part of planning. Unknowns, predicted and unpredicted, rear their head. Risk turns into caution turns into incidence. Nothing goes exactly to plan. So, we take another swing at the plan, armed with new information.

We’re always smarter than we were last week or last month when we made the plan. Sticking to the plan is foolish. Updating or overhauling the plan makes much more sense than trying to argue with it.

Planning is like writing. They both focus our thinking. An idea that flops when we take it from our brains to the page probably needs more work, whether it’s writing or planning.


Working, directly & small

Omar Rizwan recollects that one of the original selling points of React was that you could consolidate all the HTML, CSS, and JS for a single component in one file. No navigating across large directory trees to find the one line of code that implements the behavior you want. Far less worrying "if I change this am I unwinding a ball of yarn that I will regret?"

(Side note: one of the most powerful tools programmers have is scope. Most things are easier when you're working in smaller and more local scopes. If your HTML/CSS/JS only applies to the file you’re looking at, many things are easy or possible.)

I'd forgotten about this selling point and wish things had evolved differently to support it. In the large, many programming techniques don't work and this one is no exception. But in the small, how wonderful would it be to express everything about an idea you're trying to build in just one place?

Lately, I'm finding that my prototypes and weekend hacks are most successful when I don't try to get "too serious" about them too early. Adding type checkers, linters, even switching contexts to write automated tests are all "too serious". If I give myself a little freedom to work messy and concentrate on the idea, not the code, things go better.

I suspect there's a connection between working messy, generating ideas, and using a framework that doesn't punish/forbid throwing all your code into one file.


Let them go their own way

A mistake many newly minted (and some experienced) engineering managers (EMs) make is listening to their team (good!), discussing potential solutions and tradeoffs (good!), and then telling them how to solve the problem. Whoops!


Hire based on outcomes instead of role descriptions

The first time I hired someone, I wish I’d known it’s much better to think about the outcomes you’re hiring for. With that in mind, work backwards to the experience and skills required for a person to succeed in this role.


The unreasonable effectiveness of checklists

Checklists are a fantastic tool for thinking. This despite the existence of GTD, Kanban, PARA, and any number of ways to organize projects and figure out how to finish them. When I’m starting a project or when the going gets weird, checklists are usually how I end up thinking my way through.


Onboard new teammates with a 90-day plan


My new boss had written up a 90-day plan for me the week before I started. This was perfect timing. I was already starting to put a bow on my current work and my focus was wandering. Now my brain could start working on ideas for the next gig. Plus, I had a much better idea of what I’d start working on and how to make an impact than I did coming out of my interviews. It was one of the better emails I’ve ever received.


Use a tag line that means something

I like that Ember's tagline is about ambition.


The project management corollary to Hofstadter's Law

Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.

Corollary: It always takes more repetitions to tell people what you're doing, how you're going to do it, why you're doing it, how much progress you've made, that you finished doing it, etc. even when you take into account the corollary to Hofstadter's Law.


Towards smaller JavaScript

The JavaScript ecosystem’s gone to a strange place where dense frameworks and complex tooling are the status quo. But, there are data-points suggesting the pendulum could swing back sooner than later:

  • Snowpack 2.0 - download all your deps, import them as modules. Snappy development experience ensues.
  • lit-html - generate DOM without going through React/Vue/etc. intricacies
  • Alpine.js - attach dynamic interactions to elements with data elements describing DOM manipulations
  • htmx - attach dynamic networking to elements with data elements describing AJAX/Websocket/SSE events

Caveat: I haven't tried any of these. But, the trend-line is promising. JavaScript the language, while not perfect, is Pretty Good now. Perhaps the next few years will see the great ideas of the frameworks squeezed into more accessible, less sprawling expressions of those ideas.