Smaller barriers to entry, bigger possibilities

James Somer, A Coder Considers the Waning Days of the Craft | The New Yorker:

In chess, which for decades now has been dominated by A.I., a player’s only hope is pairing up with a bot. Such half-human, half-A.I. teams, known as centaurs, might still be able to beat the best humans and the best A.I. engines working alone. Programming has not yet gone the way of chess. But the centaurs have arrived. GPT-4 on its own is, for the moment, a worse programmer than I am. Ben is much worse. But Ben plus GPT-4 is a dangerous thing.

Simon Willison on the same:

I think AI assisted programming is going to shave a lot of the frustration off learning to code, which I hope brings many more people into the fold

We’ve entered the age of AI-powered coding, writing, speaking, and painting centaurs.

If we play our cards right, we will lower barriers to entry and raise the ceiling of possibility to new levels. If more people can create in mediums that are considered specializations now, that might open allow experts to go deeper in their specialization or branch out into areas that were inaccessible without compromising their specialization.

The idea of a dilettante, a person who cultivates an area of interest, such as the arts, without real commitment or knowledge, might become acute or obsolete. We might end up with a new level of “yuck that looks some amateurish and generated”. Or we might end up with reviews like “the artist deftly combines generated and hand-drawn sketches with procedurally generated music modeled on their own previous album I Play Pianos, By Hand, Like Duke Did”.

Of course, we’ve heard this story before and famed economist Keynes is (infamous?) for predicting mechanical automation would all have us exploring our favorite hobbies at this point. So, we gotta play our cards right.

My summer at 100 hertz

Is there a lost art to writing code without a text editor, or even a (passable) computer? It sounds romantic, I’ve done it before, I tried it again, and…it was not that great. 🤷🏻‍♂️

Tooling has improved for ambitious software developers

Tools for working on software in the large1 have improved a lot over since I last considered them ten years ago.

Saying No is the first step

Ryan Holiday, 35 and 34, 36 Lessons on the Way to 36 Years Old:

As part of that, I made the difficult decision to call my publisher to push my next book a year or so. This was a massive clearance on my schedule—several hours a day did not have to be spent researching and writing on a project. Yet it was remarkable how little my life changed. Because tasks expand to fill the space, because it is so easy to say yes to other things. Less demands vigilance and discipline, perhaps even more effort than actually doing stuff.

The reward for saying no feels like saying yes to a more important thing. But you still need decision discipline after that first “no”.

Whenever I’ve heard things like “the key to serene focus and productivity is saying no more often”, it often seems like saying “no, thanks” on projects and potential work is the top of the hill. Like it’s all downhill to doing great work from there. Say no, they say, puts you on easy street to writing the great American novel/album/YouTube channel/large language model.

Alas, saying no is not one weird trick for exercising all of your decision-making and discipline in one crucial moment. Making great stuff requires many small moments of saying no. Say no to looking up that frivolous fact. Say no to the pull of social media. Say no to taking a day off your habit of making great stuff. Say no to that Oreo cookie. ☹️

Have principles, will travel

Pirijan, Kinopio’s Design Principles:

The more unique and definitive your values are, the more useful they’ll be as a decision making tool later on.

These are some of the principles that I use to design (and redesign) Kinopio:

  1. Embrace smallness
  2. Build for fidget-ability
  3. Embrace plain text
  4. A single interface for mobile and desktop
  5. Refine by pruning

A+ examples throughout.

Top of Mind No. 6

I’ve been thinking a lot about setting expectations and goals. I have an idea about setting expectations on how we practice software development in teams and four pillars thereof. They are, broadly: alignment/consensus, accountability/responsibility, transparency/visibility, execution. These seem like four useful touch-points for coaching individuals. More concretely: help teammates drive scope (down, mostly) by setting time expectations and iterating from there.

The other angle on my mind is using subjective measurements to evaluate changes to human systems. That is, don’t ask a person or team to change how they work and immediately hit a numeric benchmark. Instead, ask them how the change is going and rate it from 1-5, worst to best. If the desired outcome is “know what the team is up to on most days”, ask them to write a status report, but don’t specify a number to hit. Instead, use 1:1s to reflect on how the change is impacting their work, look for advantages or shortcomings to the change in process, and decide how to correct course from there.

Updates: LLMs are still promising, but not as much for leadership work. Working incrementally, still underrated.

The point of a commonplace notebook is not to generate immediate enlightenment. Writing a quote or idea now is the tip of the iceberg. The real insight comes when reviewing commonplace notes later and the dots start to connect themselves. That’s when the galaxy brain kicks in and the magic happens.

My promise to you, and the world: I will never call anything a “rig”. No matter how much people want to read about it, or how many hours I invest in it. Even if it is a car or musical equipment or an RV or something that is literally “a rig”. No matter how rigged up it is, I will find a way to dance around the word. Do unto others as they would do unto you, as they say.

Build for the excitement of building

Nice Tietz-Sokolskaya, Write more “useless” software:

When you spend all day working on useful things, doing the work, it’s easy for that spark of joy to go out. … Everything you do is coupled with obligations and is associated with work itself.

You lose the aspect of play that is so important.

Writing useless software is a great way to free yourself from those obligations. If you write something just to play, you define what it is you want out of the project. You can stop any time, and do no more or less than you’re interested in. Don’t want to write tests? Skip them. Don’t want to use an issue tracker? Ditch it. Finished learning what you wanted to? Stop the project if it’s not fun anymore!

Build for self-learning, to tell a joke, to get your mind off something, as methodical practice, or so you can write about it.

You don’t always have to build for lofty open-source principles, entrepreneurial hustle, or because someone told you to!

Related: Whyday is August 19, plenty of time to plan your next frivolous hack.

The pace you’re reading is the right pace for you to read

Ted Gioia, My Lifetime Reading Plan:

IT’S OKAY TO READ SLOWLY

I tell myself that, because I am not a fast reader.

By his accounts, Gioia is a prolific and thorough reader. And yet, a self-proclaimed non-speed-reader.

You don’t have to read super-fast if you’re always reading whatever is right for you at the moment. Doubly so if you’re deeply/actively reading, searching for understanding or assimilating ideas into your own mental arena.

Once more for the slow readers, like myself, in the back: it’s fine, just keep reading!

Inside you, there are two or more brains

David Hoang, Galaxy Brain, Gravity Brain, and Ecosystem Brain:

The Galaxy Brain thinkers are in 3023 while we're in 2023. They relentlessly pursue the questions, “What if?” Imagination and wonder fuel the possibilities of what the world could be.

…

I believe every strong team requires skeptics and realists. Introducing, the Gravity Brain. Don't get it twisted in thinking this is negative—it's a huge positive. If you say "jump," to a Gravity Brain person, they won't say, "how high?" Instead, they say, "Why are we jumping? Have we considered climbing a ladder? Based on the average vertical jump of humans on Earth, this isn't worth our time." Ambition and vision don’t matter if you don’t make progress towards them.

…

Ecosystem Brains think a lot of forces of nature and behaviors. They are usually architects and world builders. When they join a new company, they do an archeological dig to understand the history of society, language, and other rituals.

I’m a natural gravity brain and occasional ecosystem brain. I aspire to galaxy brain, but often get there by of proposing a joke/bad idea to clear my mind and get to the good ideas.

Which one are you?

Aristotle’s ethical means of virtue and vice but for creative work:

  • Winning is the mean between moving the goal-lines to finish and not finishing due to non-constructive goal-lines
  • Quality is the mean between piles of incomplete junk and one or two overwrought ideas
  • Taste is the mean between copying and invention in a vacuum of influences
  • Flow is the mean between distraction and idleness
  • Iteration is the mean between doing it once because you nailed it and doing it once because you gave up

I’m not your cool uncle

I find that playing the “I’m the leader, this is the decision, go forth and do it” card is not fun and almost always wrought with unintended consequences.

A notable exception is teams that fall back to working in their own specialized lanes1 and throwing things over the wall. Really, the issue is throwing things over the wall, working in your lane isn’t inherently harmful2. Playing the leader card and insisting folks work across specialization in their lane or closely with a teammate whose skills complement their own has fixed more things than it complicated. More specifically, the complications were the kind of difficulties a high-functioning team would face anyway, i.e., “high-quality problems”.

Now I wonder what other scenarios justify playing the power dynamic or “I’m not your cool uncle” card to escape an unproductive local maximum.

  1. e.g. back-end, front-end, mobile, etc. ↩
  2. Assuming you’re adept at communicating progress and avoiding rabbit holes ↩

The Textual framework

Textual is a Rapid Application Development framework for Python, built by Textualize.io.

Build sophisticated user interfaces with a simple Python API. Run your apps in the terminal and (coming soon) a web browser.

Works across operating systems and platforms: terminal, GUI, and web (apparently). Uses CSS for styling internally. A wild implementation choice, but probably practical for adoption!

May many more, modern/practical Visual Basic 6/Delphi-like tools bloom. 🤞

Stop writing

George Saunders on getting past self-critical/low-energy writing spirals (aka one of the many forms of writer’s block):

Another thing I sometimes suggest is this: stop writing. That is, stop doing your “real” writing. Yank yourself out of your usual daily routine. Instead, today, off the top of your head, write one sentence. Don’t think about what it should be about, or any of that. Just slap some crazy stuff down on the page. (Or it can be sane stuff. The “slapping down” is the key.) Print it out. Then, go do something else and, over the next 24 hours, do your best not to give that sentence a single thought.

The complete method, in my own words:

  • Start with a sentence, nothing in particular. Just start.
  • Return to that sentence daily, making additions or changes that feel right.
  • As the thing grows from a sentence to a paragraph and so on, spend more time with it. But don’t sweat the magnitude of the output. A trivial punctuation change is “a good day’s work”.
  • Get bolder in the changes. Do what you prefer and don’t worry about why it is you gravitate towards that particular change.
  • Resist the urge to switch from working on an exercise to working your routine/process. Keep growing it instinctively.
  • Eventually you have a whole story/essay/whatever written this way and you’re happy to do something with it.
  • Now you are un-stuck.

(Am I using this post to get past a lull in publishing? Yes.)

Build with language models via llm

llm (previously) is a tool Simon Willison is working on for interacting with large language models, running via API or locally.

I set out to use llm as the glue for prototyping tools to generate embeddings from one of my journals so that I could experiment with search and clustering on my writings. Approximately, what I’m building is an ETL workflow: extract/export data from my journals, transform/index as searchable vectors, load/query for “what docs are similar to or match this query”.

Extract and transform, approximately

Given a JSON export from DayOne, this turned out to be a matter of shell pipelines. After some iterations with prompting (via Raycast’s GPT3.5 integration), I came up with a simple script for extracting entries and loading them into a SQLite database containing embedding vectors:

#!/bin/sh
# extract-entries.sh
# $ ./extract-entries Journals.json

file=$1
cat $file |
  jq '[.entries[] | {id: .uuid, content: .text}]' |
  llm embed-multi journals - \ # [1]
    --format json \
    --model sentence-transformers/all-MiniLM-L6-v2 \ # [2]
    --database journals.db \
    --store

A couple things to note here:

  1. The placement of the - parameter matters here. I’m used to placing it at the end of the parameter list, but that didn’t work. The llm embed-multi docs suggest that --input is equivalent, but I think that’s a docs bug (the parameter doesn’t seem to exist in the released code).
  2. I’m using locally-run model to generate the embeddings. This is very cool!

In particular, llm embed-multi takes one JSON doc per line, expecting id/content keys, and “indexes” those into a database of document/embedding rows. (If you’re thinking “hey, it’s SQLite, that has full search, why not both: yes, me too, that’s what I’m hoping to accomplish next!)

I probably could have just built this by iterating on shell commands, but I like editing with a full-blown editor and don’t particularly want to practice at using the zsh builtin editor. 🤷🏻‍♂️

Load, of a sort

Once that script finishes (it takes a few moments to generate all the embeddings), querying for documents similar to a query text is also straightforward:

# Query the embeddings and pretty display the results
# query.sh
# ./query.sh "What is good in life?"
query=$1

llm similar journals \
  --number 3 \
  --content "$query" |
  jq -r -c '.content' | # [1]
  mdcat # [2]

Of note, two things that probably should have been more obvious to me:

  1. I don’t need to write a for-loop in shell to handle the output of llm similar; jq basically has an option for that
  2. Pretty-printing Markdown to a terminal is trivial after brew install mdcat

I didn’t go too far into clustering, which also boils down to one command: llm cluster journals 10. I hit a hiccup wherein I couldn’t run a model like LLaMa2 or an even smaller one because of issues with my installation.

Things I learned!

  • jq is very good on its own!
    • and has been for years, probably!
    • using a copilot to help me take the first step with syntax using my own data is the epiphany here
  • llm is quite good, doubly so with its growing ecosystem of plugins
    • if I were happier with using shells, I could have done all of this in a couple relatively simple commands
    • it provides an adapter layer that makes it possible to start experimenting/developing against usage-priced APIs and switch to running models/APIs locally when you get serious
  • it’s feasible to do some kinds of LLM work on your own computer
    • in particular, if you don’t mind trading your own time getting your installation right to gain independence from API vendors and usage-based pricing

Mission complete: I have a queryable index of document vectors I can experiment with for searching, clustering, and building applications on top of my journals.

Read papers, work tutorials, the learning will happen

(Previously: Building a language model from scratch, from a tutorial)

I started to get a little impatient in transcribing linear algebra code from the tutorial into my version. In part, it’s tedious typing. More interesting, GitHub Copilot trying to autocomplete the code was sometimes right and sometimes the wrong idiom and actively deciding which was which compounded the tedium. This is a big lesson for our “humans-in-the-loop supervise generative LLMs” near-future. 😬

OTOH, when I got to the part where an attention head was implemented, it made way more sense having read Attention is All You Need previously. That feels like a big level-up: reading papers and working through implementations thereof brings it all together in a big learning moment. Success! 📈

Building a language model from scratch, from a tutorial

I’m working from Brian Kitano’s Llama from scratch (or how to implement a paper without crying). It’s very deep, so I probably won’t make it all the way through the long-weekend I’ve allocated for it.

I’ve skimmed the paper, but didn’t pay extremely close attention to the implementation details. But the tutorial provides breadcrumbs into the deeply math-y bits. No problems here.

I noticed that there are Ruby bindings for most of these ML libraries and was tempted to try implementing it in the language I love. But I would rather not get mired in looking up docs or translating across languages/APIs. And, I want to get more familiar with Python (after almost twenty years of not using it).

I started off trying to implement this like a Linux veteran would, as a basic CLI program. Nonetheless I switched over to Jupyter as it looks like part of building models is analyzing plots and that’s not going to go well on a CLI. And, so I’m not swimming upstream so much.

Per an idea from Making Large Language Models work for you, I’m frequently using ChatGPT to quickly understand the PyTorch neural network APIs in context. Normally, I’d go on a time-consuming side-quest getting up to speed on an unfamiliar ecosystem. ChatGPT is reducing that from hours and possibly a blocker to a few minutes. Highly recommend reading those slides and trying a few of the ideas in your daily work.

I wonder how Vonnegut might have coped with the acceleration of change we cope with in our modern dilemma.

It’s just a hell of a time to be alive, is all—just this goddamn messy business of people having to get used to new ideas. And people just don’t, that’s all. I wish this were a hundred years from now, with everybody used to the change.

– Kurt Vonnegut, Player Piano

He wrote so much about time, traversing it and getting unstuck in it, that I think he might have shrugged and stuck to an earlier motif: So it goes.

Link directly to stations in Apple Music

Generate Apple Music URLs via Apple Music Marketing Tools. Query by song, album, basically anything you can search in the Music.app sidebar.

e.g. WEFUNK radio, long URL: https://tools.applemediaservices.com/station/ra.1461224745?country=us

Purple Current, short URL: https://apple.co/3sCvX8o

Marfa Public radio, 98.9 KMKB: https://apple.co/3sw6LRd

Handy for building Shortcuts to specific stations! Apple still doesn’t provide direct navigation to streams, in the year 2023 🤦‍♂️.