Read papers, work tutorials, the learning will happen
(Previously: Building a language model from scratch, from a tutorial)
I started to get a little impatient in transcribing linear algebra code from the tutorial into my version. In part, it’s tedious typing. More interesting, GitHub Copilot trying to autocomplete the code was sometimes right and sometimes the wrong idiom and actively deciding which was which compounded the tedium. This is a big lesson for our “humans-in-the-loop supervise generative LLMs” near-future. 😬
OTOH, when I got to the part where an attention head was implemented, it made way more sense having read Attention is All You Need previously. That feels like a big level-up: reading papers and working through implementations thereof brings it all together in a big learning moment. Success! 📈
Building a language model from scratch, from a tutorial
I’m working from Brian Kitano’s Llama from scratch (or how to implement a paper without crying). It’s very deep, so I probably won’t make it all the way through the long-weekend I’ve allocated for it.
I’ve skimmed the paper, but didn’t pay extremely close attention to the implementation details. But the tutorial provides breadcrumbs into the deeply math-y bits. No problems here.
I noticed that there are Ruby bindings for most of these ML libraries and was tempted to try implementing it in the language I love. But I would rather not get mired in looking up docs or translating across languages/APIs. And, I want to get more familiar with Python (after almost twenty years of not using it).
I started off trying to implement this like a Linux veteran would, as a basic CLI program. Nonetheless I switched over to Jupyter as it looks like part of building models is analyzing plots and that’s not going to go well on a CLI. And, so I’m not swimming upstream so much.
Per an idea from Making Large Language Models work for you, I’m frequently using ChatGPT to quickly understand the PyTorch neural network APIs in context. Normally, I’d go on a time-consuming side-quest getting up to speed on an unfamiliar ecosystem. ChatGPT is reducing that from hours and possibly a blocker to a few minutes. Highly recommend reading those slides and trying a few of the ideas in your daily work.
I wonder how Vonnegut might have coped with the acceleration of change we cope with in our modern dilemma.
It’s just a hell of a time to be alive, is all—just this goddamn messy business of people having to get used to new ideas. And people just don’t, that’s all. I wish this were a hundred years from now, with everybody used to the change.
– Kurt Vonnegut, Player Piano
He wrote so much about time, traversing it and getting unstuck in it, that I think he might have shrugged and stuck to an earlier motif: So it goes.
Link directly to stations in Apple Music
Generate Apple Music URLs via Apple Music Marketing Tools. Query by song, album, basically anything you can search in the Music.app sidebar.
e.g. WEFUNK radio, long URL: https://tools.applemediaservices.com/station/ra.1461224745?country=us
Purple Current, short URL: https://apple.co/3sCvX8o
Marfa Public radio, 98.9 KMKB: https://apple.co/3sw6LRd
Handy for building Shortcuts to specific stations! Apple still doesn’t provide direct navigation to streams, in the year 2023 🤦♂️.
I got better at estimating projects with intentional practice
I like the idea of practicing1, in the musical or athletic sense, at professional skills to rapidly improve my performance and reduce my error rate. When I was a music major2, I spent hours practicing ahead of rehearsals, lessons, and performance. Until recently, I was unable to conceive of how I might do the same for leadership.
I dropped in on historic races (the newest cars were 30 years old) at Laguna Seca. It’s great to see the track at scale, not as video-game or television camera angle. I watched practice sessions from the grand stand, heard the and watched the cars rip by at competitive speeds.
I walked the paddock and infield. So many good cars to see. Felt the energy of mechanics getting cars, sometimes older than me, ready to go fast on track. I smelled a lot of gasoline and oil.
It was a good day.
This is where we live, for a few days. I’m hoping “little adventures” like this are a nice compliment to vacations. Perhaps the former creates new mental connections and the latter are about disconnecting to consider and build upon existing connections.
“Now a truth,” said the judge. “The main business of humanity is to do a good job of being human beings,” said Paul, “not to serve as appendages to machines, institutions, and systems.”
– Kurt Vonnegut, Player Piano
We’re here to do human things: create for each other, love our friends and family, support our communities, experience and protect our world.
A life serving only algorithms wrought by corporations1, engaged with organizations that care more about themselves than their people, or partaking in eco-cultural regimes that disregard our humanity are not worth pursuing.
-
IOW, don’t throw your whole existence into being a content creator, folks. ↩︎
A vacation is a tool for disconnecting
I’m reflecting1 on travelogue’ing my recent trip to Disneyland. In particular, that it was effective at taking my mind off work while maintaining creative writing and publishing habits.
Lately, I find myself dwelling on work too much or noodling through various puzzles I’m trying to solve. I would rather not bring that with me on vacation. What I removed and what I replaced those thoughts with went pretty well! For my own memory, and perhaps dear reader’s benefit:
Allow plenty of moments to just exist. Listen, enjoy where you are. Engage with the folks you’re vacationing with, presuming they’re not in a phone-bubble. Try to put the phone away after you mindlessly open it out of boredom. Cultivate the moments of boredom, if you can.
Don’t look at feeds like Slack, Discord, socials, etc. I took the extra step of removing Slack from my phone entirely. It’s easier to remove it entirely and re-add each Slack later than to try to ignore particular instances whenever I find myself mindlessly opening the app. Read a lot, ideally longer form things chosen beforehand. Again: ignore the feeds of people hustling or trying to get their ideas into my attention. Think your own thoughts!
Replace all those feeds and work-think with written and photographic journaling. Look for photography opportunities, even if you’re not particularly good at it (yet). Write in a journal, daily, even if you’re not good at it (yet). Write about what you’ve photographed, about whatever you’re thinking while disconnected, about that conversation you just had or the novel thing you just saw/did.
If you want to share via social networks, do it once a day. I posted (via micro.blog and Ulysses) a short reflection on each day. For texture, I included a quad of photos, and wrote a few sentences to reflect on my photography or day so far. Once the post is made, put the computer away and get back to reveling in disconnection.
YMMV, but I hope you try it!
-
With apologies to Craig Mod for standing on the shoulders of A Walk is a Platform for Creative Work. ↩︎
A pretty ideal travel day: start at the pool, wait in the breeze and shade, depart from the open tarmacs of tiny Long Beach Airport.
Not pictured: returning to the searing heat of central Texas. 😬🙃🫠
Many parts of Disneyland are ridiculously easy to photograph.
Particularly at dusk and night, and for folks (like me) with very rudimentary photography skills. I’m pretty convinced this is one of the myriad details the Imagineers wrangle as they design these parks.
Rule #1 is you a) get up early, b) ride stuff early and c) take a break in the middle of the day while (many) folks are at the parks.
Roz is the unsung hero of the Monsters, Inc. franchise. Impeccable delivery, no notes.
One of our favorites is the Animation Academy. It’s a short entertainment, wherein a cast member teaches you to draw a Disney/Pixar character in ten minutes or so. I’m not at all good at sketching, so I think it speaks volumes to the quality of the cast members that you can, maybe, tell that I’m drawing Mickey Mouse here.
1. Rule #2 is you gotta take a picture of this signage, every time.
2. Matterhorn looks better from almost every angle than it feels to ride it (if you’re over forty).
3. It’s a lovely day for retro future and wishing they’d put a people mover in at Disneyland.
4. Spoiler alert.
Rauno Freiburg is building attention to design details
Sitting down and just thinking hard does not magically produce valuable discoveries either. The essence of the word “interaction” implies a relationship between a human and an environment. In my experience, great revelations surface from making something — filling your headspace with a problem — and then going for a synthesising daydreaming walk to stir the pot.
👍🏻 This is the excellence in interactions and attention to detail. Moreover, a restatement of the Hickey/Feynman methods for problem-solving. (Is there a restatement of said methods that doesn’t sound lovely?)
Work side-by-side; the more you can see, the more you can think
Wisen Tanasa, Stop flipping around, put them side-by-side:
This constant flipping is a self-inflicted cognitive load that reduces our productivity. Pay attention to what’s happening in your mind when you flip things around. What was the CSS property I was trying to modify? Flip… flip… What was that JSON structure look like again? Flip… flip… Is it similar to Figma now? Flip… flip… When you catch yourself doing this, stop. Put them side-by-side.
Many of the tasks we have as software engineer is crunching information from one medium to another. What I’d like you to try instead is to pause and make all the information you need to execute your task visible on your screen.
Using splits in my editor (vim, then RubyMine), notes app (Obsidian), and even window manager (Rectangle Pro) are crucial to how I work. Single window, uni-tasking, “focus modes” were coolr in their day. But, most work lends itself to some manner of multi-window, conceptual layout. As Wisen points out, that could be test and production code, designs and code, code and browser/simulator, designs and documentation, or a document and video chat.
I wish I could get even more conceptual layout. I usually do fine with 2-3 vertical splits (i.e. three full-height windows) when I’m coding. But, I look with envy upon designers in Figma/Sketch/etc. when they consider several iterations upon one screen or lay designs out on their canvas to match the spatial/workflow relationships in the actual product. Concepts like Natto, PureData, and Obsidian’s canvas leave me optimistic that in the future I’ll find myself organizing modules and functions however I want, regardless of linear text in source code files on disk.
Magic (AI) is what we don’t (yet) understand
It reflects a sound understanding of the nature of AI — as an uncredited and formless modifier of other technologies. One whose presence is marked by familiar behaviors having slightly magical effects.
– Venkatesh Rao, Magic, Mundanity and Deep Protocolization
The whole magic & AI thing. The most 2023 of vibes!
Yet, it remains to be seen to what extent AI will yield entirely new media, modalities, and consumption levels or if it will merely modify technology we’re already familiar with.
Is generative AI the iPhone (entirely new media/modality/consumption), the iPhone camera (displaces and exceeds existing categories), or the ability to scan QR codes with your camera to reduce transactional friction (a game changer for some that quickly fades into the background)?
Evenings at the Village Gate: John Coltrane (with Eric Dolphy). Five tracks, all excellent. Great bass playing by Reggie Workman and Art Davis. Favorite tracks: “Greensleeves”, “Africa”.
Less but better
The Designer’s Designer’s Watch – A Look Back At Braun And The Rebirth Of A Few Classics
The Braun watches AW 10 and AW 50 embody simplicity. Just as do the clocks and electrical appliances made by Braun, they truly follow the “less is more” dogma that (arguably) stems from the Bauhaus school. Or, as Braun likes to put it: Less but better. Design has been at the core of the brand for seven decades, and its watches and clocks were designed by a team headed by two men: Dieter Rams and Dietrich Lubs.
✋🏻 “Less is more”: easy to say, hard to apply.
👍🏻 “Less but better”: easy to say, easy to tell when you’ve done the first part but missed on the second.