The emotional rollercoaster of extracting code
There’s a moment of despair when extracting functionality from a larger library, framework, or program. The idea grows, a seed at first and then a full-blown tree, that the coupling in this functionality isn’t all bad. A lot of people talk only about coupling and leave out cohesion. They aren’t mutually exclusive! When the two are balanced, it’s hard to come up with a reason to start extracting.
On the other hand, sometimes that moment of despair strikes when you start really digging into the domain and realize this chunk of functionality isn’t what you thought it was. Maybe it’s not coherent (see above!) or perhaps the model of the domain isn’t deep enough. This is a pretty good signal to hit the brakes on the refactoring, figure the domain out, and reconsider the course of action.
Feature envy rears its head in extractions too. Patterns of crosstalk between the existing thing and the new thing are a sure sign of feature envy. It’s tempting to say, hey maybe you really need a third thing in the middle. That’s probably making matters worse though.
That said, changing bidirectional communication to unidirectional is usually a positive thing. Same for replacing any kind of asynchronous communication with synchronous. Or replacing lockstep coordination with asynchronous messaging. Envy is tricky!
(I) often encourage starting a new service or application within your existing “mothership”. The trendy way to say this right now is “monorepo all the things” or build a “modular monolith”. I find this compelling because you can leverage a lot of existing effort into operationalizing, tooling, and infrastructure. Once you know the domain and technical concerns specific to the new thing, you can easily extract into its own thing if you need to. The other edge of a monorepolith is that path dependence is a hell of a thing. Today is almost certainly an easier day to split stuff out than tomorrow.
A thing to consider pursuing is a backend-for-frontend service in pursuit of a specific frontend. It doesn’t even have to serve an application. You may have services that are specific to mobile, desktop, apps, APIs, integrations, etc. Each of these may need drastically different rates of change, technical features, and team sets.
Probably don’t split out a service so that a bunch of specialized people can build a “center of excellence” for the rest of the organization to rely upon. This is a very fancy way to say “we are too cool for everyone else and we just can’t stand the work everyone else is doing”. On their best day, the Excellence team will be overwhelmed by the volume of work they have put in front of themselves to make Everything Good. On their worst day, they will straight give up.
If you split something out, realize you’re going to have to maintain it until you replace it. And you’re going to rebuild the airplane while it’s flying. If you’re not really into that, stop now. Just because you can’t stand Rails, relational databases, or whatever doesn’t mean you should jump into an extraction.
What I talk about when I talk about cars
Human design: what went into deciding how a human-facing thing is made? How did they decide to put the infotainment screen there? Why are BMW dashboard lights orange-ish? Who designs gauges and do they know what they’re doing? What the hell is going on with Mercedes dashes? When will we stop using gearshifts? When will the scourge of the PRNDL knob leave us?
Mechanical design: i.e. why is this car the way it is from an engineering point-of-view? Why is the rear-engine 911 unique, special, and kinda dumb? What makes the BMW M1 a weird BMW and yet perhaps the most special? Why do Ferrari engines catch fire so frequently? Do BMW/Audi/Mercedes design their cars as sedans, coupes, or hatchbacks first?
History: What puts some brands, e.g. Ferrari and Porsche over the others? Is Audi interesting? What is the gestalt of Honda or Toyota? How did the Viper come to have a tractor engine and a Lamborghini body? How does a BMW M car become “the next coming of BMW Jesus?”
Emotion: What makes people think a car is special? What kind of person owns a Koenigsegg or Pagani? Why own a Lamborghini? Is it practical to drive a Ferrari touring car? When does an Acura TL make sense for someone who enjoys cars? What will enthusiast cars look like once electric cars are the norm; will we finally enter a world of boring aerolumps?
…amongst other things. So many questions, so many subjective answers. That’s what makes it fun!
More ideas for framework people
A few months ago I wrote about Framework and Library people. I had great follow-up conversations with Ben Hamill, Brad Fults, and Nathan Ladd about it. Some ideas from those conversations:
use a well-worn framework when it addresses your technical complexities (e.g. expose functionality via the web or build a 3-d game) and your domain complexity (e.g. shopping, social networking, or multi-dimensional bowling) is your paramount concern
once you have some time/experience in your problem domain, start rounding off corners to leave future teammates a metaframework that reduces decision/design burdens and gives them some kind of golden path
frameworks may end up less useful as integration surface area increases
napkin math makes it hard to justify not using a framework; you have to build the thing and accept the cost of not having a community to support you and hire from
to paraphrase Sandi Metz on the wrong abstraction: “(Using) no abstraction is better than the wrong abstraction”; if you’ve had a bad time with a framework, chances it was an inappropriate abstraction or you used the abstraction incorrectly
Did you try editing the right file?
The first few years of my career, I edited the wrong file all the time. I could spend hours making changes, wondering why nothing was happening, until I realized I’d been tinkering in the wrong place because I was misreading a file path or not paying close enough attention to control flow.
Fast forward to now, and I’m pretty quick to drop a raise “BLORP” in code I’m tinkering with if things aren’t working like I think they should. All hail puts debuggerering.
However, it turns out I found a new class of this operator error today. I was diligently re-running a test case, expecting new results when the test fixture file I thought was changed was the wrong file. Once I deleted the right file, I was back on my way.
Joyful and grumpy are we who can find new ways to screw up time ever day!
Chaining Ruby enumerators
I want to connect two Ruby enumerators. Give me all the values from the first, then the second, and so on. Ideally, without forcing any lazy evaluations and flat so I don’t have to think about nested stuff. Like so:
xs = [1, 2, 3].to_enum ys = [4, 5, 6].to_enum [xs, ys].chain.to_a # => [1, 2, 3, 4, 5, 6]
I couldn’t figure out how to do that with Ruby’s standard library alone. But, it wasn’t that hard to write my own:
def chain(*enums)
return to_enum(:chain, *enums) unless block_given?
enums.each { |enum| enum.each { |e| yield e } }
end
But it seems like Ruby’s library, Enumerable in particular, is so strong I must have missed something. So, mob programmers, is there a better way to do this? A fancier enumerator-combining thing I’m missing?
Stored Procedure Modern
The idea behind Facebook’s Relay is to write declarative queries, put them next to the user interaction code that uses them, and compose those queries. It’s a solid idea. But this snippet about Relay Modern made me chuckle:
The teams realized that if the GraphQL queries instead were statically known — that is, they were not altered by runtime conditions — then they could be constructed once during development time and saved on the Facebook servers, and replaced in the mobile app with a tiny identifier. With this approach, the app sends the identifier along with some GraphQL variables, and the Facebook server knows which query to run. No more overhead, massively reduced network traffic, and much faster mobile apps.Relay Modern adopts a similar approach. The Relay compiler extracts colocated GraphQL snippets from across an app, constructs the necessary queries, saves them on the server ahead of time, and outputs artifacts that the Relay runtime uses to fetch those queries and process their results at runtime.
How many meetings did they need before they renamed this from “GraphQL stored procedures” to “Relay Modern”?
(FWIW, I worked on a system that exposed stored procedures through a web service for client-side interaction code. It wasn’t too bad, setting aside the need to hand write SQL and XSLT.)
We should get back to inventing jetpacks
I don’t like using services like Uber, Twitch, or Favor. I want to like them, because the underlying ideas are pretty futuristic. But the reality of these services is that the new boss wants to squeeze their not-even-employess-anymore just as badly the old boss did. It feels manipulative, like buying a car. Except I’m abetting the manipulation too. :(
The New Yorker, THE GIG ECONOMY CELEBRATES WORKING YOURSELF TO DEATH:
The contrast between the gig economy’s rhetoric (everyone is always connecting, having fun, and killing it!) and the conditions that allow it to exist (a lack of dependable employment that pays a living wage) makes this kink in our thinking especially clear.
What happens when the gig economy tries to turn a profit? The race downwards will squeeze out all of their contractors until they can replace them all with automated drivers, commoditized personalities, and punitively-low ad revenue sharing rates. This sounds horribly dystopian but I’m pretty sure it’s already happening. See also: when Google kneecapped bloggers as a side-effect of end-of-lifing Reader and changing Pagerank.
The New York Times, Platform Companies Are Becoming More Powerful — but What Exactly Do They Want?
Platforms are, in a sense, capitalism distilled to its essence. They are proudly experimental and maximally consequential, prone to creating externalities and especially disinclined to address or even acknowledge what happens beyond their rising walls. And accordingly, platforms are the underlying trend that ties together popular narratives about technology and the economy in general. Platforms provide the substructure for the “gig economy” and the “sharing economy”; they’re the economic engine of social media; they’re the architecture of the “attention economy” and the inspiration for claims about the “end of ownership.”
After reading this, I started substituting “platform company” for “company building its own monopoly”. And then it all makes sense. Businesspeople say they love free markets, but give any rational-thinking business the chance and they will create so many “moats” and “barriers to entry” that they resemble tiny state enterprises more than a private business. See also: telecoms and airlines.
Anil Dash, Tech and the Fake Market tactic:
This has been the status quo for most of the last decade. But the next rising wave of tech innovators twist the definition of “market” even further, to a point where they aren’t actually markets at all.
Yes, my confirmation bias is burning. Yes, technologists are doomed to recreate the robber-baron past they didn’t study. Yes, we still have time to change this. Yes, our field needs an ethics refresher. Yes, we should get back to inventing jetpacks!
Jeremy Johnson, It’s time to get a real watch, and an Apple Watch doesn’t count:
...watches are one of the key pieces of jewelry I can sport, and while many have no clue what’s on my wrist, those that do… well do. And they are investments. Usually good purchases will not only last forever (with a little love and care), but go up or retain most of their value over time.
When pal Marcos started talking to me about watches, I realized they checked all the boxes cars do, but at a fraction of the price. If cars check your boxes, look into watches. Jeremy’s intro will get you started without breaking the bank.
Feedback: timing is everything
With feedback, like jokes, timing is everything. Good feedback at a bad time won’t do the trick.
I’ve mostly experienced programming feedback through pull requests. This is way better than no feedback. However, since most pull requests occur at the end of work, and not somewhere in the middle, some kinds of feedback are not conducive to pull requests.
Suppose all feedback falls somewhere on two axes: “timeliness” and “depth”. The narrow sweet spot of code review is apparent:
[caption id=“attachment_4302” align=“alignnone” width=“861”]
Pairing and code review are not so similar[/caption]
The sweet spot in the top-right corner is when code review works best: unhurried and in-depth feedback. I’d hesitate to call the lower-right corner of hurried, minimal feedback a code review at all; it’s more like rubber stamping.
I’ve often referred to code review, flippantly, as the worst form of pairing yet invented. I’ve given a lot of code review feedback in the past that was better suited to the synchronous nature of pairing than the very asynchronous nature of code reviews. That said, I feel like pairing is an excellent way to give all manners of feedback in the moment the code is being conceived or written. You can immediately point out possible incorrectness or better designs and talk it out, with the code at hand, with your collaborator.
However, we can’t all pair all the time. Let me show you how I’m trying to better time my feedback when I can’t share it immediately.
A tale of four pull requests
Consider four PR subject lines. Which ones are appropriate for architectural ideas? What about optimization ideas? When is deep refactoring feedback appropriate? Can I look at one of these in an hour when I’m done with my current task?
- “Hotfix Facebook Auth scope”
- “Prevent sending email for failed payment jobs”
- “Add tagging to admin storylines listing”
- “WIP introduce Redis/Lua-based story indexing”
Lately, when I do pull request reviews, I use these guidelines:
- Figure out if this PR seems like it’s a hot patch to production, a quick fix on existing work, a PR landing new functionality, or a work-in-progress checkpoint seeking feedback.
- Bear in mind that hot patches and quick fixes are more time sensitive and need yes/no feedback on correctness more than detailed feedback.
- For hot patches (e.g. “Hotfix FB auth”), I’m only looking for “is this correct” and “will it fix the problem?”; thumbs up or thumbs down and commentary as to what I think is missing to solve the problem. No refactoring ideas. I only touch on performance if I spot a regression.
- For quick fixes (e.g. “Prevent sending email…”), I’m again looking for correctness and timeliness. I might leave ideas for how to improve the performance or cleanliness of the code later. Those kinds of notes are entirely up to the gumption of the other developer, though. I know the low-gumption feeling of wanting only to fix something and get on to the next thing.
- Landing new functionality (e.g. “Add tagging…”) receives a full review cycle. Beyond baseline correctness, I’m trying to view this code through my crystal ball. When some value of
Nis grows, will this code slow down noticeably? Is the code structured so that future changes are easy and obvious? - Work-in-progress checkpoints (“WIP introduce Redis/Lua…”) are open to the full spectrum of feedback. Ideas for how to differently structure data, which APIs to export, how to structure objects, how to name the domain model, etc. are all in play. Pretty much the only thing out of play is anything that feels too close to bike shedding.
- Bear in mind that everyone exists on a spectrum of coding specificity. More seasoned developers are likely open to ideas for restructuring code or considering novel approaches. Less seasoned developers (including seasoned developers new to the team) likely want specific guidance about which changes to make or factors they need to consider.
- Where I may try to respond to hot patches and quick fixes in less than fifteen minutes, I may wait a couple hours before I look at new functionality or WIP reviews.
- The most difficult part with these guidelines is how to handle ideas about refactoring on time-sensitive reviews. I want to hold the line against letting lots of little fixes accrete into a medium-sized mess. I don’t want to discourage ideas for refactorings either; I want them separately so I can act on them when I have the energy to really do them.
In short
Use different tactics when sharing feedback for code review; it’s not pairing. Identify patches, reviews, and full feedback pull requests. Sanity check patches, look for correctness in review, look for design in review. Use GitHub’s review process to indicate your feedback is “FYI” vs. “fix this before merging”. Time-to-response is most important for patches and fixes.
Above all: giving feedback is a skill you acquire with practice, empathy, and maintaining a constructive attitude.
Practically applying Clojure
Fourteen Months with Clojure. Dan McKinley on using Clojure to build AWS automation platform Skyliner:
The tricky part isn’t the language so much as it is the slang.
Also, the best and worst part of Clojure:
When the going gets tough, the tough use maps
This is probably better now that specs and schema are popular. Before, when they were mysterious maps full of Very Important State, reading Clojure code (and any kind of Lisp) was pretty challenging.
Make sure you stick around for the joke about covariance and contravariance. Those type theories, hilarious!
Lessons on software complexity from MS Office
I learned a lot of things from Complexity and Strategy by Terry Crowley:
In Fred Brooks’ terms, this was essential complexity, not accidental complexity. Features interact — intentionally — and that makes the cost of implementing the N+1 feature closer to N than 1.
In other words, the ability to change a product is directly proportional to the size of N (features, requirements, spec points, etc.) for the system that express that product. You may find practices that multiply N by 0.9 so you go a little faster. You may back yourself into a corner that multiply N by 1.1 so you go a little slower. But, to borrow again from Fred Brooks, there is no silver bullet. Essential domain complexity is immutable unless you reduce the size of the domain, i.e. cut existing features.
Not even fancy new technologies are correlated with reducing your multiplier, in the long run:
This perspective does cause one to turn a somewhat jaundiced eye towards claims of amazing breakthroughs with new technologies...What I found is that advocates for these new technologies tended to confuse the productivity benefits of working on a small code base (small N essential complexity due to fewer feature interactions and small N cost for features that scale with size of codebase) with the benefits of the new technology itself — efforts using a new technology inherently start small so the benefits get conflated.
Lastly, this is a gem about getting functionality “for free”:
So “free code” tends to be “free as in puppy” rather than “free as in beer”.
All free functionality eventually poops on your rug and chews up your shoes.
Healthcare is a multiplier, not a consumer good
Adam Davidson tells a personal story about a relative who, with health care, could’ve continued his career. Without that healthcare, he ended up addicted and in jail. What the GOP doesn’t get about who pays for health care:
However, dividing health expenditures into these categories misses an important economic reality: health-care spending has a substantial impact on every other sort of economic activity.
Healthcare isn’t consumption, like buying a TV or going to a movie. It is a Keynesian multiplier. Every dollar the government spends on it means an individual or business can spend more than a dollar on something productive in GDP terms.
UPS and FedEx can’t exist without public roads. Southwest and United Airlines can’t exist without the FAA. Lockheed and Northrop can’t exist without the Air Force. Walmart and McDonald’s can’t exist without food stamps. Entrepreneurs find it harder to start without individual access to healthcare.
Yet Republicans are opposed to the existence of all of these. Perhaps business in America relies on more subsidies and government services than Republicans are willing to admit!
Type tinkering
I’m playing with typeful language stuff. Having only done a pinch of Haskell, Scala, and Go tinkering amidst Ruby work over the past ten years, it’s jarring. But, things are much better than they were before I started with Ruby.
Elm in particular is like working with a teammate who is helpful but far more detail oriented than myself. It lets me know when I missed something. It points out cases I overlooked. It’s good software.
I’ve done less with Flow, but I like the idea of incrementally adding types to JavaScript. The type system is pragmatic and makes it easy to introduce types to a program as time and gumption permit. Having a repository of type definitions for popular libraries is a great boon too.
I’m also tinkering with Elixir, which is not really a typed thing. Erlang’s dialyzer is similar in concept to Flow, but different in implementation. Both allow gradually introducing types to systems.
I’m more interested in types stuff for frontends than backends. I want some assurance, in the wild world of browsers and devices, that my systems are soundly structured. Types buy me that. Backends, I feel, benefit from a little more leeway, and are often faster to deploy quick fixes to, such that I can get away without the full rigor of types.
Either way, I’m jazzed about today’s tools that help me think better as I build software.
Let’s price externalities, America
Hello, America. We have to talk. You are built on top of a mountain of federal (a trillion or so dollars) debt. That debt covers some things we need (roads, health care, social safety nets, education, scientific research) and subsidizes distasteful things (energy companies, military contractors, banks, real estate). You could consider that debt the tip of the iceberg. We can see how it contributes to the annual federal budget in dollars and by percentages. It’s a measurable, knowable thing.
Unfortunately, there’s also a ton of unmeasured debt we are accruing. We have to pay the price for it through social norms and charity. Here’s a hackish list:
- food service is systematically underpaid so we tip them, most often poorly
- the people who clean our hotel rooms are underpaid because they are invisible, unskilled, and often immigrant; depending on what you read, you should tip them but they also say you should hide your valuables from them so which is it, leave money laying around or distrust them not to rub your toothbrush in the toilet?
- we pay a small tax on the amount of gasoline we use, but it is comically low, hasn’t gone up in years, and isn’t enough to pay for the usage of our crumbling roads, bridges, etc. sometimes it's also used to pay for public transit, which is perverse during high gasoline prices if you've studied even rudimentary supply-and-demand
- our children are raised mostly by women who are expected to just do it for free, despite what else they may want to do with their lives
- we let financiers play with our retirement money, in theory because they know how to allocate it, get the occasional Google but more often some business tragicomedy, and in return they get to take a few percent off the top, which ends up being a huge number, for the service basically of them having gone to Harvard or their daddy knew a guy
- millions of people live paycheck to paycheck, go hungry, go into massive debt if life comes at them wrong, etc. all because the Walmarts of the world (and there are way more than just Walmart) pay them next to nothing expecting the federal government to pick up the slack except the federal government has been systematically dismantled over the course of decades by men who fancy themselves smart enough to start The Next Walmart but in fact are barely smart enough to get themselves elected in a fair contest let alone actually lead a congressional district
But I digress and rant. And rant. Economists call these externalities. It’s when you have some accidental cost or benefit that is paid for by a third party, e.g. Walmart paying less than a living wage because the government will pick up the tab through welfare.
Point is, we’re underpaying for a lot of stuff. And that’s fun for some of us. We eat avocado toast, take exciting trips around the world, maybe drive race cars. We sort ourselves out so we don’t see the literally millions of people suffering because we’re not paying what it takes to give everyone a chance at doing better off than their parents or picking themselves up when life knocks them down. And then when some transparently awful populist blowhard runs for president, we’re shocked, just shocked, that he ends up winning.
Bring the higher taxes. Make me pay more to eat out. Charge me more for gas. I don’t mind thinking twice about whether I should subscribe to HBO and Showtime and Netflix. If I can’t go to Disney World as often, so be it.
It’s a small price to pay to have avoided what’s coming over the next four years: an increasingly unequal, unfair world for those of us who aren’t already doing great and white and male. Let me pay more for a greater country where everyone, not just the affluent, seek what it is that makes them happy in life without the fear of illness, bad circumstance, political or racial backlash. Let’s not lord that greater country over people to “motivate them to work harder and escape their current lot in life”. Let’s price the externalities that separate the concerns of the rich from the stresses of the poor and let’s all pay our share.
Universes from which to source test names
A silly bit of friction in writing good tests is coming up with consistent, distinctive names for the models or object you may create. Libraries that generate fake names, like Faker, are fun for this, but they don’t produce consistent results. Thus I end up thinking too hard.
Instead, I like to use names from various fictional-ish universes:
- Wile E. Coyote and the Road Runner: Acme Corp, Ajax, Fleet Foot corp. etc. Bonus: read through the extensive laws and rules of this universe!
- Mickey Mouse universe: you can't go wrong with putting Disney trademarks in your code.
- CIA cryptonyms: when I want my teammates to wonder if they know everything going on with our project.
Hopefully my teammates enjoy these little easter eggs as much as I enjoy looking them up when I need something fancier and less dry than metasyntactic variables.
You should practice preparatory refactoring
When your project reaches midlife and tasks start taking noticeably longer, that’s the time to refactor. Not to radically decouple your software or erect onerous boundaries. Refactor to prepare the code for the next feature you’re going to build. Ron Jeffries, Refactoring – Not on the backlog!
Simples! We take the next feature that we are asked to build, and instead of detouring around all the weeds and bushes, we take the time to clear a path through some of them. Maybe we detour around others. We improve the code where we work, and ignore the code where we don't have to work. We get a nice clean path for some of our work. Odds are, we'll visit this place again: that's how software development works.
Check out his drawings, telling the story of a project evolving from a clear lawn to one overwhelmed with brush. Once your project is overwhelmed with code slowing you down, don’t burn it down. Jeffries says we should instead use whatever work is next to do enabling refactorings to make the project work happens.
Locality is such a strong force in software. What I’m changing this week I will probably change next week too. Thus, it’s likely that refactoring will help the next bit of project work. Repeat several times and a new golden path emerges through your software.
Don’t reach for a new master plan when the effort to change your software goes up. Pave the cow paths through whatever work you’re doing!
Let's not refer to Ruby classes by string
I am basically OK with the tradeoffs involved in using autoloading in Rails. I do, however, rankle a little bit at this bit of advice in the Rails guide to developing engines.
[caption id=“attachment_4216” align=“alignnone” width=“661”]
Screencapture from Rails Guide to Engines[/caption]
In short, when configuring an engine from an initializer, you’re supposed to protect references to autoloaded application classes (e.g. models) by encoding them as strings. Your engine later constantizes the string to a class and everything is back to normal.
A thing I am not OK with is “programming with strings”. As an ideal, strings are inputs from other machines or humans and internal references to code are something else. In Ruby, symbols fill in nicely for the latter. Could I refer to classes as Symbols instead of Strings and live with the tradeoffs?
Well it turns out, oddly enough, that Rails is pretty sparing with Symbol extensions. It has only 120 methods after Rails loads, compared to 257 for String. There are no specific extensions to symbol, particularly for class-ifying them. Even worse (for my purposes), there isn’t a particularly great way to use symbols to refer to namespaced classes (e.g. Foo::Bar).
But, the Rails router has a shorthand for referring to namespaced classes and methods, e.g. foo/bar#baz. It doesn’t bother me at all.
In code I have to work with, if at all possible, I’d rather refer to classes like so:
- Refer to classes by their real ClassName whenever possible given the tradeoffs of autoloading
- When autoloading gets in the way, refer to things by symbols if at all possible
- If symbols aren’t expressive enough, use a shorthand encoded in a string, e.g.
foo/bar#baz - … (alternatives I haven't thought of yet)
- Refer to classes by their full string-y name
But, as ever, tradeoffs.