Isaac Clayton

196 Years

2026-04-06T00:00:00+00:00

It’s been a while, I know. I sit here writing today because I feel trapped; a prisonless prisoner who laid bricks until one day he woke up in a cell of his own creation.

Am I overconstrained, overburdened, overpromised? In the event that I carry all the parcels I bear through to the date of their respective deliveries, I’m certain I will emerge on the other side a better person. The constraints seek resolution; the burdens are amenable to carrying; the promises sway, flexible but unyielding.

On the other hand: I am afraid who I will have become if I fumble and lose what I have been entrusted. Similarly, if I hold it all too tight, and the weight pins my body to the cool, damp earth, it is certain that against the ground I will remain: the burdens never shed, the destination never reached. Sometimes I ask, “This destination, is it mine?” Am I living out the dream of another?

People ask things of me and I’m terrified to respond. Terrified that I will sign yet another social contract. I am not a busy person, just overwhelmed. I do not feel the satisfaction some seek when telling others “no”. I march through the work I have to do, and it is done, but I have done nothing. What am I doing?

I’m being dramatic. I should eat something. If there’s any week after which to lighten one’s burdens, this one isn’t bad!

Tomorrow is a new day! I will stay my course and fight a good fight; it does not matter that I win, just that I remain: both true to myself, and the trust that others have placed in me. I need to put on my oxygen mask, unswamp my canoe, start the engine, break the curse, turn the tables, put on my socks and shoes, pull up my big-boy pants, because tomorrow is a new day, and the sun won’t wait to rise for those who waste asleep.

In any case, the sunrise is too beautiful to miss.

Daily reading: Competence as Tragedy</a> </div>

Neopack is live

2026-03-04T00:00:00+00:00

Slowly emerges from the sixth sea of silence.
Hello! I’m back for the time being.
I started working on a handful of projects, and those have been taking up my blogging time. Sorry for the inconsistency.
I wanted to announce that I’ve published the first (and hopefully only) version of Neopack! Neopack is a very small library for quick and schema-agnostic tag-length-value serialization. It should be good for both IDL and RPC type workloads. Here’s a minimal “getting-started”-type example:
[dependencies] neopack = { version = "1.2", features = ["derive"] } </code></pre>
use neopack::{Pack, Unpack}; #[derive(Pack, Unpack)] struct Point { x: i32, y: i32 } let bytes = Point { x: 1, y: 2 }.pack_to_vec().unwrap(); let point = Point::unpack_from_bytes(&bytes).unwrap(); </code></pre> Neopack is what I’d consider to be “done” software. There might be a few minor wrinkles to iron out, but I don’t expect anything major about the library, its API, or the wire format to change anytime soon.
If you’d like to take a look, here are the docs. It is very similar to serde</a> (except it’s only about one kiloline of code) and has zero-copy deserialization:

neopack</code> on docs.rs</a>. Core library, exposes derive</code> feature.</li>
neopack-derive</code> on docs.rs</a>. Derive macros for the Pack</code> and Unpack</code> traits.</li> </ul> One limitation of neopack is that it’s no-std</code>. So the crate doesn’t implement serialization for standard library types, like PathBuf</code>. I considered going the serde</a> route, with the fancy mod</code> for custom serialize/deserialize, but in the name of simplicity I ask that you serialize these types yourself to bytes or a String</code>, and to not include them in serialized structs. If I figure out a solution that makes everyone happy, I’ll do it! That’s all! Thanks for reading. Daily reading: Bob Nystrom, Representing Heterogeneous Data</a> I enjoyed Bob Nystrom’s musings on designing a language with heterogeneous data types, so you might enjoy it too. (Neopack is built on heterogeneous Algebraic Data Types. Because neopack uses TLV for everything, you can fully and dynamically inspect and decode a packed value with an unknown schema! Isn’t that neat?) </div>
Programming as theory building is true now more than ever 2026-02-21T00:00:00+00:00 These past two days have been something of a whirlwind, as I made a couple of important life decisions. The decisions were, in a strange way, very small, but in a “marble rolling down a saddle” sort-of way that might compound and drastically change the side of the saddle my timeline will spill down… we shall see! I’ve repeatedly brought up the paper Programming as Theory Building</a> in conversation with friends this past week, so I figured it would be good to write up the common thread of these conversations and discuss how the ideas in the paper are relevant today. The paper was written in 1985 by Peter Naur. The main thrust of the argument is that when you program, you are doing two things: Producing the artifact of text, or source code, on disk.</li> Producing the model of a system in your mind. This is “theory building”.</li> </ol> Programming, Naur argues, should be framed as theory building. The real value of what is produced is precisely the model that lives in the programmer’s mind. The mental model is what allows you to debug, extend, and collaborate on a codebase; not the source code. Why the theory building angle is useful</h1> I really like the way a few insights that naturally follow as a consequence of thinking of programming as theory building. I’ll run through them now. First, a bug occurs when your mental model of the system’s behavior disagrees with what the code does. As programs do exactly what you write down, a bug isn’t wrong code; it’s doing exactly what you wrote. A bug happens when your code doesn’t match your theory, or your theory doesn’t match the code. Fixing a bug, then, requires either refining your mental model or refining the code. Often it’s both. In this sense, to a customer, if an app doesn’t work the way they expect it to, that’s a bug, to them. UX bugs occur when we do a poor job of theory building with our user. Second, source code is a medium of reflection against which further theory can be built. When you have an abstract notion of a system, you can only hold so much of the theory in your head, or in design documents. Producing source code grounds theory, and allows the programmer to explore and discover the right way to extend their existing theories, so new systems can be developed and features can be implemented. Finally, theory building provides a good framework to think about working on code with others. As the source code is not the artifact, and rather the theory is, a lot of programming on a team boils down to communicating the theory you have built up, and asking good questions to learn the theory your teammates have built up. Different team members hold expert theories for different parts of the codebase. Becase of this, interfaces are clean boundaries where you can communicate a crisp and neatly-packaged theory to a teammate, which packages complexity to become a sharp tool with which to carve whatever code they need to write. Documentation, pair programming, demonstrations are all means through which a body of theory is built and communicated. Pair programming is great for theory building</h1> In high school, I wrote a lot of code on my own. The first time I really had the opportunity to work on a team on projects with a bus factor bigger than one was when I was an intern at Zed</a>, in 2022. I spent a lot of time pair programming with Antonio, who taught me a lot about Rust and software engineering. I now realize that Antonio spent so much time pair programming with me because he was very invested in theory building: both in improving my understanding of the theory behind the editor and the tools we were using, and in improving his understanding of the code I was writing, so he could carry the theory we built together forward after my internship ended. I’m really grateful for this mentorship and I still think back to a lot of the lessons I learned about e.g. async Rust minutiae Antonio patiently taught me along the way. Vibecoding is antithetical to theory building</h1> Now, the AI tie in. Great. Sure. I saw it coming from a mile away. Here it is: Sometimes AI tools can produce a better code artifact; however, better code that was generated doesn’t help you, as the programmer, to build a better theory. Taking a step back, the idea of programming as theory building doesn’t materially change because of AI tools that write code now exist. (A follow-on paper that thoroughly investigates the effects of AI tooling on theory building would be interesting to read.) The issue is that it’s easy to convince yourself that you’ve built theory when you haven’t: It’s easy to fall into the trap or classical fantasy of “texting your nerdy eager-to-please friend an app idea and watching it get written”. First, if you want to be engaged in building theory when using tools, you need to be engaged. If you treat tools as teammates, they will invariably go off and build theory on their own. The model will not go out of its way to teach you its assumptions. You need to understand the theory that has been generated if you wish to extend the artifact of the code. Second, AI tools have no qualms picking interfaces and invariants for you. The interfaces they choose will often be those interfaces that are the most plausible for what you are trying to do. However, these interfaces are not always the best interfaces for whatever it is that you are specifically working on. Sometimes they’re worse. Sometimes they’re better! Regardless of ontological merit, any interface sucks if you don’t understand it. Finally, for both of these issues, it is not enough to simply read and write more documentation, as Naur argued back in 1985. Documentation is a poor substitute for theory. AI generated documentation for AI generated code does not necessarily build the right theory in your mind. Lessons for collaborating on code</h1> With these problems in mind, what can we do? I write here in the context of AI tools that write code, but this is true of collaboration with human teammates as well. (A lot of these ideas are lessons I indirectly learned through Antonio, though we were not using AI generated code at the time.) I’ve learned a few things: First, you should really care about good, pedagogical documentation and good API design. When you design an interface, you have the opportunity to both build your theory of the system and specify the exact interfaces and invariants you require (instead of letting a teammate, or some other collaborator, choose for you). I find that writing system documentation, writing out types, writing an API, writing property-based tests by hand, before letting an AI implement what is left, produces better results. Second, you should pay extra attention and slow down when you realize your theory for something you are working on is slipping. When working with AI collaborators, it is easy for experienced engineers to vibe-code a simple app, because the theory is something they already understand fully. The issue arises when a programmer then starts working on extending or developing new aspects of the system without having the symbolic grounding</a> to close that feedback loop, and develop the theory you require. Finally, you can always ask these tools, as you would a colleague, to be watchful of when your theory is slipping, and to help you understand what you’re missing. AI tools can generate lots of ungrounded theory, yes, but they can also work to answer questions and determine the correctness of your existing theories. One last note</h1> Importantly, while talking about theory building in the context of AI tools, I should mention that I believe it to be bad to “go off to build thought palaces alone”. Humans communicate differently with other people than they communicate with AI. Having real-life people care about what you’re working on, anxiously engaged with your code, provides the human perspective you miss out on when writing software alone (or with a tool). That’s it for today! I probably won’t blog tomorrow because it’s Sunday. If you’d like a story about someone who wrote no code but was great at collaborative theory building, you should read: Daily reading: The Worst Programmer I Know</a> </div> Symbolic grounding 2026-02-18T00:00:00+00:00 After yesterday’s</a> massive post (and given the two assignments I have to do today) I figured I would write something a little shorter today. A little observation, perhaps. In the 70s and 80s</a>, people cared a lot about symbolic reasoning; precise relational systems of facts and rules used to derive logical truths. Any relational system is one concerned with resolving constraints. These ideas fell out of favor because of the powerful general “statistical” methods we have today. From casual usage, I’ve found that writing code with language models pairs very well with property-based testing. Give the model a list of invariants and constraints, along with types and interface definitions, have it generate a ream of tests using a property-based testing library. Verify the tests are correct, then let the model run in a loop against the symbolically grounded system until all constraints are satisfied. You can even tell it to keep track of the search tree in a consistent format as it goes, to know what to try next and where to backtrack to. This turns a fuzzy word problem into an unyielding semantic one; the model acts as a very good prior for performing symbolic search. We’re seeing moves in this direction as newer models employ thousands of little “tool calls” while reasoning, to symbolically ground their reasoning. These systems are trained end-to-end; one day models may be little more than good priors from which to sample belief networks on the fly. I think datalog or relations generally should be built into programming languages. One language does this well: Daily reading: The Flix Programming Language</a> </div> Epistemic and Aleatoric Uncertainty 2026-02-17T00:00:00+00:00 This is a follow-on post to yesterday’s</del>, um, last Friday’s post about curiosity</a> driven approaches to reinforcement learning. I intended to publish this post on the 14th, but it was Valentine’s day weekend and $\text{irl} > \text{blog}$. To summarize: Imagine we have some black-box environment with a state space $s$. At each timestep, we receive an observation $o$ (derived from $s$), and can pick an action $a$. The environment evolves according to a stochastic state transition dynamic $s’ \sim P(\cdot \mid s, a)$ depending on the current state and our action, producing a new observation $o’$. Suppose we don’t have a goal or reward function for our environment; we simply would like to train a neural network to model $P(s’ \mid s, a)$. How can we sample sequences of actions, or episodes, that contain all the information we need to build an accurate model of state transition dynamics? The solution looks something like curiosity: construct a reward signal along the lines of “the number of learnable bits of information per observation”, and train a policy $\pi$ using traditional RL techniques like PPO to improve the policy and sample divers information-dense trajectories. </div> Today I’m going to try to explain the following relation: $$\text{uncertainty} = \text{aleatoric} + \text{epistemic}$$ If that doesn’t make sense now, don’t worry. It should be clear by the end of the post. Our goal is to learn a policy that is capable of exploring an arbitrary environment. In essence, we want to seek out states that are surprising, but not totally random; in other words, we want to find states where we are uncertain, but capable of learning through interaction and observation. Surprisal</h1> Let’s start by quantifying this idea of surprise. In information theory, there is this concept of surprisal, which is defined as: $$h(y) = -\log P(y)$$ Here, $y \sim Y$ is some specific outcome; an event sampled from a (discrete) random variable $Y$. Surprisal $h(y)$ is measured in bits. A likely outcome can be encoded in few bits, and likewise tells you little; an unlikely outcome takes many bits to encode, and conversely tells you a lot. To make surprisal concrete, rolling a 4 on a fair die has probability $\frac{1}{6}$. The surprisal is $h(⚃) = \log_2 6 \approx 2.58$ bits of information. Entropy, total uncertainty</h1> Surprisal is concerned with a single event. If we have a distribution $Y$ over multiple events, is there a way to describe how much information we expect to learn upon measuring the outcome? Yes, and this is exactly entropy $H[Y]$, or “expected surprisal”: $$H[Y] = \mathbb{E}[-\log P(Y)]$$ We use lowercase $h(y)$ for the surprisal of a single event $y$, and uppercase $H[Y]$ for the surprisal over a distribution of events $Y$. The expectation $\mathbb{E}$ here just measures the surprisal of each event, weighted by how probable it is. Mathematically speaking: $$H[Y] = \sum_y P(y) \cdot h(y)$$ You can think about entropy as, “in expectation, how many bits of information will we gain by making an observation?” Conditional entropy</h1> Entropy measures how much information we expect to learn about $Y$… in isolation. Oftentimes, we’ve observed some correlated event $x$. In that case, how much information do we expect to learn from $Y$, given we already know $x$? This is precisely conditional entropy $H[Y \mid x]$: $$H[Y \mid x] = \mathbb{E}[-\log P(Y \mid x)]$$ We write lowercase $x$ because we’re conditioning on a specific input, rather than averaging over all possible inputs, which is what you normally see. Note that this is exactly the same thing as entropy, just over the conditional distribution $P(Y \mid x)$ instead of $P(Y)$. If knowing $x$ makes $Y$ easier to predict, the conditional entropy is lower. If $x$ is independent of $Y$, then $H[Y \mid x] = H[Y]$. Like entropy, we can also write conditional entropy as a weighted sum, but conditioned on $x$: $$H[Y \mid x] = \sum_y P(y \mid x) \cdot h(y \mid x)$$ In our environment, if $S$ is the distribution over next states, and $(s, a)$ is our state action pair, then $H[S \mid s, a]$ is an exact measure of how unpredictable the next state is from a given state-action pair. This measures, in essence, how random state transitions are. For a deterministic environment, $H[S \mid s, a] = 0$. In practice, we observe $o$, not $s$. Reconstructing state from observations is a different filtery, hidden-markov-modelly can of worms. We’ll gloss over this for now, and condition on $s$ directly, but the intractable component of epistemic uncertainty I touch on later comes partly from this gap. Conditional entropy tells us how uncertain $Y$ is given a specific event $x$. But in general, if we know the value of a correlated random variable $Z$, we often want to ask how much of $Y$ it tells us. If measured in bits, this quantity is called mutual information. Mutual information</h1> A coin flip has high conditional entropy regardless of context; you can never learn to predict it. For curiosity, we want our policy to seek states that it finds unpredictable because it has not yet seen them, not because the environment is random and can’t be predicted. Mutual information between $Y$ and some other variable $Z$ measures how many bits of the expected surprisal of $Y$ are accounted for if you already know $Z$. It’s precisely the gap between entropy and conditional entropy: $$I[Y; Z] = H[Y] - H[Y \mid Z]$$ We can draw this as a Venn diagram, which is nicely symmetric: