17
Two things that need to happen in software ASAP

2026-09-17 · 1040 words

I wrote this in 30 minutes after reading about attacks targeting Rust library maintainers, thinking about all the recent news about unintended AI compromise, and remembering Jia Tan.

Two things that need to happen in software ASAP. I won’t bury the lede:

1. Capability-based security for all libraries across all platforms.
2. Secure privacy by default with good UX/DX.

Let me explain.

These days AI is compromising production systems left and right. The frequency and pressure of social engineering attempts to compromise open-source projects is increasing. Any data that can be collected will be collected and is treated as fair game to use to profile, sell, and manipulate individuals in aggregate. AI products are optimized, emergently and incidentally, to be habit-forming and deceptive.

The wonder of the open internet is the serendipity of discovering new ideas and forming new friendships. I didn’t learn to program through classes, but instead by reading blogs, making friends online, and attempting projects that were ambitious to me at the time. This is something I want to preserve.

The influx of Randian meat-proxies is making this a lot harder. So I want to propose two solutions or directions all programmers should consider to be common sense when designing software, like writing unit tests or not implementing your own constant-time cryptography routines. I hope that languages, libraries, developer tooling, and the open-source community will adapt to integrate these changes.

On capability-based security. First, the biggest risk of a supply chain attack is that a dependency that did one thing now does another. It is crazy to me that software installed on a system essentially has full reign over the system. (Relevant XKCD, software can do things you didn’t expect). To that end, all dependencies should declare the capabilities they require, and these capabilities should be passed in, either in the package manifest or in code. This way an array utility library can’t make network requests if compromised. Beyond the language and tooling layer, platforms should support capabilities and provenance, be that through hardware and permissions, or composition layers like Wasm components. The Fuchsia microkernel is also of note. It would be great if we could write a linux-compatible microkernel with capability-based security, and formally prove its implementation correctness and security guarantees. This is likely one of the hardest classes of developer-tooling problems that exists, but we need to figure it out.

On privacy-by-default. Second, most software is not private by default. When I first started programming, it was crazy to me that web apps would send passwords to the server in plaintext, relying on TLS to encrypt the data in transit. I thought, why not derive the secret locally and send some signature or proof to the server, perhaps interactively, to communicate knowledge of the password and the identity of the recipient, without transmitting the sensitive material? Luckily things like passkeys, which are pretty well designed AFAICT, are gaining some traction. But the point still stands. Most programmers aren’t defensive or security minded and will rely on whatever security guarantees the language, platform, or protocols they rely on define. AI-written software doubly-so. To that end, we need good awareness as programmers as to what fundamental security primitives exist, how to use them, what to be careful about, and high-level abstractions with good defaults that provide robust guarantees. It would be great if Signal were formally verified; it would be additionally better if it were a library offering an ergonomic secure-channel primitive.

It’s been said before, but much like how the natural world is full of viruses and bacteria that will exploit and reproduce within whichever environment they can find, the same has been happening to the information environment in which we exist. We need to evolve defenses and “immune responses” to go along with the increased external pressure on the security and safety of our systems.

There has already been a retreat from the public internet in some sense. I, for one, do not use social media or go to websites I’ve never heard of when looking things up. I avoid reading comments whose provenance is unknown. Much like how maintaining a healthy physical diet is important, I think a strict, nutritious, and judicious information diet is important as well. AI-written text, flashy solicitations for attention, short-form video, advertising, and other forms of spam are junk food for the mind and soul.

I’m going to get more than eight hours of sleep tonight so I’ll wrap this up. To protect spaces where people hang out and learn together online we have to limit exposure to the pressures of the writhing internet. We need to do this in a way that respects the privacy and autonomy of the individual while building long-term trust. I think public read-only communities with invite-only writes are a step in the right direction, in this sense. If we do switch to these more invite-centris models, we need to be careful about making them findable and joinable by the budding next generation of programmers, to protect the spirit of the open web; not just a sea of small clubs. We should work on lowering the barrier to entry for people to publish on their own websites, we should curate indexes of interesting work, and allow newcomers to share and receive feedback on their projects. This is tough to balance given the adversarial nature of things these days, but possible I believe.

This is rough but I just wanted to get it out there. These are technical solutions but caveat emptor there is no such thing as ‘a technical solution to a social problem’.

The more interesting question is: given the reality of incentives to compromise digital infrastructure and take advantage of people online, how should infrastructure and groups online having to do with digital infrastructure be organized?