Are we being good ancestors?

— Jonas Salk, referenced from The 10,000-year clock below

The 10,000-year clock will mark time with astronomic and calendric displays and a chime generator designed with the help of Brian Eno that can produce over 3.5 million unique bell chime sequences — one for every day the clock is visited for the next 10,000 years.

Ten thousand years is about the age of modern civilization, so the clock will measure out a future of civilization equal to its past. This assumes our civilization is in the middle of whatever journey we are on — an implicit statement of optimism.

The 10,000-year clock

How do you build a clock that will last 10,000 years? Look at the list given in the article above:

Longevity

10,000-year clocks should display the correct time for the next 10,000 years

• Go slow
• Minimize sliding friction
• Stay clean and dry
• Expect bad weather and earthquakes
• Expect non-malicious human interaction
• Don’t tempt thieves

Maintainability

It should be possible to maintain a 10,000-year clock with little maintenance, using bronze-age technology

• Use familiar materials
• Make it easy to build parts
• Include the manual

Transparency

It should be possible to determine the operational principles of a 10,000-year clock with close inspection

• Allow inspection
• Allow rehearsed motions
• Expect restarts

Evolvability

It should be possible to improve a 10,000-year clock over time

• Separate functions
• Provide simple interfaces

Scalability

It should be possible to build working models of a 10,000-year clock from table top to monumental size using the same design

• Make all parts similar size

Some of these sound similar to software, no?

• Make it easy to build parts
• Include the manual
• Separate functions
• Provide simple interfaces

Indeed, many fundamental principles of nature are true across domains.

This latest release is called κατακλυσμός, and for the savvy transliterator, you can see that means cataclysm!

But its real meaning in Koine Greek is more like flood, which is a better description for this release. The biggest single change, though, is once again another plugin refactor — and we’ll probably do it again. Getting those abstractions right will allow Kai to be designed for the Long Now, and hopefully grow organically into something great that can stand the test of time.

Please note: SOPS/system-level commands modify system state and that and the ISO command is very experimental. Use them with caution on disposable machines as this is still a prototype!

Highlights:

• Refactored the plugin schema (again)
• Lots of ergonomics and UX infrastructure
• Tests for every command implementation!
• Better errors and Nix error bubbling
• system-level commands to control your Nix config!
  • generations/rollback/switch
• ISO create command (for an eventual KaiOS…?)
• SOPS secrets
• backend-scoping and passthrough

Refactoring the plugin schema (again)

Kai is built via plugins similar to how Caddy works. The standard (and currently only useful) plugin, std, is compiled using xkai, which, like xcaddy, just bundles the std package together and provides a generic CLI skeleton around it. As noted in previous posts, getting this plugin structure right is crucial to Kai’s essential goals of being generic, flexible, and modular. The single biggest refactor is changing the schema design to more accurately reflect the separation between commands, blocks, and backend implementations.

Ergonomics

Kai didn’t have much in the way of a help menu, and errors literally just spit out raw Roc constructs (gross!) which made them difficult for humans to read.

Help Menu

Finally, a proper help menu!

kai
A friendly frontend for determinate computing

Usage:
  kai [OPTIONS] <COMMAND> [ARGUMENTS] [--json]

Kai is a tool for providing a simplified interface on top of determinate
systems (mainly Nix) for ease of use. Commands often correspond with a
Kaifile block defining their behavior:

  $ kai shell
  $ cowsay "Hello from Kai!"

  # Kaifile
  shell {
      packages: ["cowsay"]
  }

Commands:
  build     Build an artifact declared in the Kaifile.
  deploy    Build and activate a machine on its declared target.
  image     Build an image from a machine declared in the Kaifile.
  iso       Build a bootable ISO from a machine declared in the Kaifile.
  machine   Build a machine declared in the Kaifile.
  run       Run a task declared in the Kaifile.
  service   Build a service declared in the Kaifile.
  shell     Enter an inline or declared developer environment.
  system    Manage machine-wide state.
  update    Update and lock project dependencies.
  workflow  Run a workflow declared in the Kaifile.
  version   Print version information.

Options:
  -f, --file <PATH>  Use the Kaifile at PATH
      --json         Output JSON Lines
      --no-color     Disable colored output
  -y, --yes          Assume yes for confirmation prompts.
  -h, --help         Print help

Environment:
  KAI_DIR             Project-local workspace directory (default: .kai)

More information: https://github.com/thebrandonlucas/kai

CLI examples on each help menu

Each subcommand has a help menu with its equivalent Kaifile block example if applicable:

kai help build
Build an artifact declared in the Kaifile.

Usage:
  kai [OPTIONS] build [BACKEND] <ARTIFACT> [--json]

Examples:
  kai build <my-artifact>

Kaifile block:
  build my-artifact {
    environment: dev
    run: ["make"]
    output: "result"
  }

Arguments:
  BACKEND  Optional backend name
  ARTIFACT  Artifact name from the Kaifile

Options:
  -f, --file <PATH>  Use the Kaifile at PATH
      --json         Output JSON Lines
  -y, --yes          Assume yes for confirmation prompts.
  -h, --help         Print help

--json Outputs

clig.dev has a great entry on how you should design everything for humans and machines (so tools can be composable), so we have a --json flag that should work for each command:

kai help build --json
{"type":"help","message":"Build an artifact declared in the Kaifile.\n\nUsage:\n  kai [OPTIONS] build [BACKEND] <ARTIFACT> [--json]\n\nExamples:\n  kai build <my-artifact>\n\nKaifile block:\n  build my-artifact {\n    environment: dev\n    run: [\"make\"]\n    output: \"result\"\n  }\n\nArguments:\n  BACKEND  Optional backend name\n  ARTIFACT  Artifact name from the Kaifile\n\nOptions:\n  -f, --file <PATH>  Use the Kaifile at PATH\n      --json         Output JSON Lines\n  -y, --yes          Assume yes for confirmation prompts.\n  -h, --help         Print help"}

I should probably add a --pretty flag or something for humans to be able to use the JSON version if they’d like, but we’ll wait and see if that use case gets justified.

ANSI

We added ANSI coloring infrastructure. Just the very basics, bold for headers, red for errors, etc. The base is there and we can easily add/modify as needed.

Errors

Kai Errors

I think the correct philosophy around error message design is clig.dev’s:

One of the most common reasons to consult documentation is to fix errors. If you can make errors into documentation, then this will save the user loads of time.

Catch errors and rewrite them for humans. If you’re expecting an error to happen, catch it and rewrite the error message to be useful. Think of it like a conversation, where the user has done something wrong and the program is guiding them in the right direction. Example: “Can’t write to file.txt. You might need to make it writable by running ‘chmod +w file.txt’.”

Signal-to-noise ratio is crucial. The more irrelevant output you produce, the longer it’s going to take the user to figure out what they did wrong. If your program produces multiple errors of the same type, consider grouping them under a single explanatory header instead of printing many similar-looking lines.

Consider where the user will look first. Put the most important information at the end of the output. The eye will be drawn to red text, so use it intentionally and sparingly.

If there is an unexpected or unexplainable error, provide debug and traceback information, and instructions on how to submit a bug. That said, don’t forget about the signal-to-noise ratio: you don’t want to overwhelm the user with information they don’t understand. Consider writing the debug log to a file instead of printing it to the terminal.

Make it effortless to submit bug reports. One nice thing you can do is provide a URL and have it pre-populate as much information as possible.

That last one wasn’t considered in this release, but I do love the idea and will add it in the future.

Now, errors that used to look like this:

Program exited with error: PlanningFailed({ backend: "nix", command:
"build", location: None, message: "build requires exactly one config
name", plugin: "std" })

Now, in the new release, look like this:

error: build requires exactly one artifact argument
usage: kai build <ARTIFACT>
example: kai build <my-artifact>

Much better!

Nix Error Bubbling

Since the std plugin in Kai sits on top of Nix, there will always be a subset of errors that Kai won’t be able to capture itself, since there can always be drift between how Kai mirrors the underlying implementation and we’ll never be able to have a perfect one-to-one error map. For example, if the user defines a package foo that is not in nixpkgs, Kai will presently still just write the flake.nix file and try to evaluate it under the hood, resulting in one of those ugly gargantuan Nix errors like:

error:
       … while calling the 'derivationStrict' builtin
         at «nix-internal»/derivation-internal.nix:37:12:
           36|
           37|   strict = derivationStrict drvAttrs;
             |            ^
           38|

       … while evaluating derivation 'nix-shell'

       … while evaluating attribute 'nativeBuildInputs' of derivation 'nix-shell'

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: attribute 'foo' missing
       at /nix/store/…-source/flake.nix:7:15:
            6|       packages = [
            7|               nixpkgs."legacyPackages"."x86_64-linux"."foo"
             |               ^
            8|       ];
       Did you mean one of fio, fjo, folo, foot or fop?

This violates the rules above by showing so much information internal to Nix itself rather than the user’s code, where the violation actually lies. That should be up front and immediately obvious.

The real error is just this:

error: attribute 'foo' missing
at /nix/store/…-source/flake.nix:7:15:
    6|       packages = [
    7|               nixpkgs."legacyPackages"."x86_64-linux"."foo"
     |               ^
    8|       ];
Did you mean one of fio, fjo, folo, foot or fop?

So we attempt to strip this out, but more work/experimentation will need to be done.

Features

import blocks

Kaifiles can now import each other! This is not a whole module system like a programming language, but rather basically just string concatenation under the hood similar to Caddy. We really just needed a way for large Kaifiles to be expanded.

Kaifile imports work like this:

# parts/environment.kai
environment dev {
  packages: ["cowsay", "fortune"]
}

Then, Kaifile:

import "parts/environment.kai"
shell {
  environment: dev
}

system Command and Subcommand Groups

Why should Kai only be used at the project level? What if you wanted to use it to configure your whole system like NixOS does? What if we could start working on combining the UX and flash of Omarchy with the power of NixOS? The first step toward this is to group commands which do this under the system command (i.e. command which manipulate the system). This is the first command which has a few subcommands:

WARNING: These commands are experimental and can modify your system! Advised to use only on throwaway machines.

kai system
Manage machine-wide state.

Usage:
  kai system <COMMAND>

Commands:
  generations  List local system generations.
  rollback     Roll back the current host one system generation.
  switch       Build and activate the declared machine.

One of the most powerful features of NixOS over other operating systems is the ability to rollback your machine’s state. Since most things in NixOS are declarative in a version-controlled file, this provides an enormous advantage over other operating systems which quickly become a tangled layered mess descending into dependency hell. Nix’s rollback allows you to switch between generations of your system’s state if anything goes wrong, or you just liked how one of your previous versions worked better. Paired with a solid UX, I expect this concept to unlock many novel use cases for people that aren’t fully utilizing their NixOS systems. At present, our rollback just goes back to the previous generation, the most useful command to recover from brokenness caused by an upgrade.

The other command added, switch, is what actually does the activation of your Nix derivations to change the state of the software. Under the hood this is nixos-rebuild switch and is probably the most essential and commonly used NixOS command.

Deploy

Another big use-case unlock. If you have a machine in the cloud, it’s extremely convenient to be able to deploy a new config at will. I’ll be adding this to my personal websites — as soon as I can configure them completely using Kai on NixOS machines (more to come!):

kai help deploy
Build and activate a machine on its declared target.

Usage:
  kai [OPTIONS] deploy [BACKEND] <MACHINE> [--json]

Examples:
  kai deploy <machine>
  kai deploy -y <machine>

Kaifile block:
  machine <machine> {
        environment: server
        system: "x86_64-linux"
        target: "root@example.com"
  }

Arguments:
  BACKEND  Optional backend name
  MACHINE  Machine name from the Kaifile

Options:
  -f, --file <PATH>  Use the Kaifile at PATH
      --json         Output JSON Lines
  -y, --yes          Assume yes for confirmation prompts.
  -h, --help         Print help

ISO

WARNING! Experimental, please don’t use on any important machines yet.

Another path to an Omarchy-but-with-Kai is an ISO. With a valid machine config, you should be able to boot into a valid NixOS machine from a Kaifile with this ISO on a flash drive! Eventually, this’ll be spruced up to give a much better UX.

SOPS Secrets

WARNING! Experimental, please don’t use on any important machines yet.

We added sops-nix, a standard tool for safely integrating encrypted secrets in Nix/Git. This is a bit limited as at the moment you have to encrypt the secret — Kai only accepts already-SOPS encrypted secrets and doesn’t handle that itself. However, this unlocks being able to deploy machines with predefined secrets on them.

Backend Escape Hatch

I wanted to allow the NixOS options assignments in flakes resulting from Kaifiles. This is a tricky problem because Nix has a lot of custom options keys for tons of services. Obviously Kai can’t re-hardcode all these. Further, there are some concepts which Kai is tying itself directly to Nix for, like overlays, which Guix doesn’t have (though it does have analogous concepts).

At present, I’m unaware of a good way around the “non-overlapping concepts” problem for backends which should allow the Kai frontend to remain seamlessly usable. So the best approach I could find is to provide an escape hatch for these scenarios:

machine server {
 environment: server
 system: "x86_64-linux"
 users: ["agent"]
 services: ["openssh"]

 backend nix {
   networking.hostName: "kai-server"
   services.fail2ban.enable: true
   networking.firewall.allowedTCPPorts: [22, 443]
 }
}

Everything under backend nix is a restricted set of typed assignments that “lowers” Kaifile code into flake.nix. This is a big flexibility leap forward for Kai which will allow the base cases to remain separately abstracted from the bespoke code in the short term. Long-term, we’ll probably have to find something better.

Global Backend Config

One repo I was attempting to add Kai to did not use the standard nixpkgs branch but rather nixos-unstable-small, a use case I hadn’t accounted for yet. While I’d ultimately like a large set of decisions like this to be configurable by the Kai binary, I think it’s also useful to have it available as a declarative config option. This is of course backend-specific, so we also scope this by its backend:

backend nix {
 packages: "github:NixOS/nixpkgs/nixos-unstable-small"
}

Scoped Architecture Predicates

Similar to the above, this repo also required architecture-specific config. We already had OS-level config e.g.:

on linux {
    ..
}

on macos {
    ..
}

But this doesn’t provide the full host picture, as you also have architectural variation:

on linux x86_64 {
    ..
}

on macos aarch64 {
    ..
}

Future

With all these features, Kai’s surface area is getting bigger. There are a growing number of commands that do very different things. Unix Philosophy says that tools should do one thing well. But the legendary Unix designers had an early-mover advantage in that they got to define much the set of tools that we use today, and therefore they didn’t have to operate as strictly within the set of available tooling. If Kai took the approach of splitting up every concern into sub-binaries, we would both have to ship a binary for every little tool rollback, deploy, switch, generations etc. and you’d have to know that each one of these things was Kai-specific. Nix does some of this with nixos-rebuild etc., but I think in the modern world we don’t want to have to redefine entire systems if we can get the majority of benefits without doing it. Swiss Army knives are closer to the mark for largely capable tools like this.

So I do think packaging things together with the plugin approach is the right one — different plugins can ship different sets of features: we could have a kai-slim, kai-full, one for project-level configs only, one for full OS management, one for Nix, one for Guix, etc.

That said, the plugin system is not being fully utilized: We are neither building out the Guix plugin system nor generalizing std to allow switching between Nix and Guix seamlessly.

I think this will be essential to avoid overfitting ourselves to Nix-only backends. While tempting in the short term, an agnostic design will pay huge dividends over time.

I’m also realizing that Nix can do lots of things and is very flexible. As such, people use it in a thousand different ways, which is something some love and others hate. This directly conflicts with the “one canonical way to do things” approach, which makes it far easier to create standards and form community (which, case in point, Nix really struggles with!).

This is an early canary — and one that we’ll have to pay close attention to mitigating in the future.

Alan Kay’s pithy phrase remains evergreen in my head such that I’m tempted to make it the philosophical motto of the project:

Simple things should be simple, complex things should be possible.

— Alan Kay

Achieving both, though, ain’t easy!

But what great things worth doing ever were?

Onward!

In the beginning all you want are results, in the end all you want is control.

Eskil Steenberg, How I Program in C

Life is long enough, and it has been given in sufficiently generous measure to allow the accomplishment of the very greatest things if the whole of it is well invested.

Seneca, On the Shortness of Life

Most European cathedrals show differences in plan or architectural style between parts built in different generations by different builders. The later builders were tempted to “improve” upon the designs of the earlier ones, to reflect both changes in fashion and differences in individual taste. So the peaceful Norman transept abuts and contradicts the soaring Gothic nave, and the result proclaims the pridefulness of the builders as much as the glory of God. Against these, the architectural unity of Reims stands in glorious contrast. The joy that stirs the beholder comes as much from the integrity of the design as from any particular excellences. As the guidebook tells, this integrity was achieved by the self-abnegation of eight generations of builders, each of whom sacrificed some of his ideas so that the whole might be of pure design. Even though they have not taken centuries to build, most programming systems reflect conceptual disunity far worse than that of cathedrals […] I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas.

Fred Brooks, The Mythical Man-Month

This update might be a bit unusual, as I want to write an update but also some a little more personal about my experience of programming.

There are two things that are true of Kai that were never true of a programming project for me before:

1. I’m building something from scratch as its progenitor, that
2. Is completely open source, that
3. I want to last a long time

I’ve taken serious aim at all but (3) before, but the longer I live the more Karl Popper’s maxim of working on long-term problems speaks to me:

I think that there is only one way to science - or to philosophy, for that matter: to meet a problem, to see its beauty and fall in love with it; to get married to it and to live with it happily, till death do ye part - unless you should meet another and even more fascinating problem or unless, indeed, you should obtain a solution. But even if you do obtain a solution, you may then discover, to your delight, the existence of a whole family of enchanting, though perhaps difficult, problem children, for whose welfare you may work, with a purpose, to the end of your days.

Karl Popper, Realism and the Aim of Science

By taking Kai seriously, by striving to make something that provides as close to a strictly better experience in using computers (to the extent achievable, there’s No Silver Bullet), I am forced to think harder than I ever have about what I’m building and why, the alignment between my highest-level end-goals and day to day tasks, how to make incremental progress every day which is aligned with these goals, how to maintain motivation, etc.

The genuine attempt to build something great is a forcing function for practices which lead to mastery.

As such, I’ve been doing a lot of reading (and more writing than I’ve ever done before!).

Sometimes I make mistakes or take some shortcuts of course. I think it can be very helpful to make something badly at first and just get it out there, share it, and get feedback from other really smart people in the community.

Kai is already starting to get fun, real, and useful, so I’m working to ensure it stays that way for a long time.

There’s been two releases since the last update: προιεναι - 0.0.5 (means “to proceed”) and ετυγχανε - 0.0.6 (means “it happened”), and a lot of changes in them.

So many changes, where to start? Let’s just follow Stephen King’s advice for novels on this one, and just skip the boring stuff.

External Features!

Let’s start with the fun stuff relevant to you, dear reader. Here are the new Kai commands added to the StdPlugin since the last update.

Source

Not all things that a project needs are always in nixpkgs. Sometimes you need to declare an external link to a URL tar or something, as was the case for me with aion (also built in Roc), where I had to declare bitcoin-qr (a QR code NPM package I wrote for adding bitcoin payments to your site) URL as a source as this isn’t on nixpkgs. This is really useful and makes Kai far more flexible!

source bitcoin-qr {
  url: "tarball+https://registry.npmjs.org/bitcoin-qr/-/bitcoin-qr-1.4.1.tgz"
}

build aion {
  environment: cli
  source: "src/aion.roc"
  output: "aion"
}

Machine

The machine block allows us to define full NixOS closures which we can then deploy or activate (for already-extant NixOS machines), which when combined with Kai can run services on startup or have particular packages installed.

Invoke with kai machine <name>

# Build a typed NixOS deployment closure with `kai machine agent`.
environment server {
  packages: ["curl"]
}

machine <name> {
  environment: server
  system: "x86_64-linux"
  users: ["<name>"]
  services: ["openssh"]
}

Image

That’s great, but what if we don’t already have a machine and want to create a bootable NixOS image? You can do that too via the image command! Which builds a generic QCOW2 image from a machine definition. For example, kai image agent turns the agent machine in your Kaifile into a portable image that a cloud provider can import.

I was able to test and verify this works on Digital Ocean via aion (below). The definition looks the same as machine, but it is invoked via kai image <name>:

# Build a generic QCOW2 image from the same machine definition.
environment server {
  packages: ["curl"]
}

machine <name> {
  environment: server
  system: "x86_64-linux"
  users: ["<name>"]
  services: ["openssh"]
}

Service

Services are basically programs that run on startup and just continue running in the background. Caddy is a good example: You just want a web server that’s always on and routing requests to your sites. A crucial prerequisite if Kai is going to be useful in spinning up NixOS machines on desktop or in the cloud.

Example:

# A service turns an executable build artifact into a NixOS systemd service.
environment cow {
  packages: ["cowsay"]
}

build cow {
  environment: cow
  run: ["sh", "-c", "cp \"$(command -v cowsay)\" cow"]
  output: "cow"
}

service cow {
  artifact: "cow"
  secrets: []
  restart: on-failure
}

Secret

A secret is basically an environment variable in either your shell, or machine, etc. This allows people to have the same machine but have behavior change based on the environment variables they set or have them set API keys or whatnot that allow them to use services. For example, I used aion to spin up a machine that ran pi coding agent connected to my ppq.ai API key. So in one command I could spin up a NixOS machine based on a Kaifile that had an agent running on it!

Example:

environment cow {
  packages: ["cowsay"]
}

# Build the "cow" program artifact
build cow {
  environment: cow
  run: ["sh", "-c", "printf '#!%s\\ntest -s \"$CREDENTIALS_DIRECTORY/message\"\\nexec %s \"secret loaded\"\\n' \"$(command -v sh)\" \"$(command -v cowsay)\" > cow && chmod +x cow"]
  output: "cow"
}

# Secret values stay outside the Kaifile. For this example, put
# "classified moo" in /run/kai/secrets/message on the machine.
secret message {
  provision: runtime
}

# Systemd passes the file to the service without exposing its contents.
service cow {
  artifact: "cow"
  secrets: ["message"]
  restart: on-failure
}

Internal Updates

Plugin Overhaul (again)

I once more redid the Plugin system, and it still isn’t quite where I want it to be. Getting this right is really important because it’s the gateway to others adding plugins for Kai to create their own plugins and features outside of std. I won’t yet go into the details because it isn’t quite there yet, but suffice to say that I’ve spent a lot of time making this much more self-documenting better abstracted. My goal is to get to the point where new users can “just dive in” making and experimenting with their own plugins.

Tidy.roc

matklad has a wonderful article on automating desired patterns of behavior for your codebase via a tidy script they made at TigerBeetle. This is a really cool concept and one I wished I’d considered sooner. Basically, imagine you just wrote your own linter, not merely to enforce generic code style rules, but also to enforce invariants that you seek to ensure your custom rules are followed with every commit. This is essentially another additional layer of ensuring conceptual integrity for the project, for whatever invariants you care about.

I started simple in Kai, just adding a Tidy.roc which enforces:

1. Every code module has a top-level explainer comment
2. Limit line length to 80 (some really long strings were getting hard to read)

Tigerbeetle’s tidy.zig does so much more, of course, and some things I’d like to do, like ensuring large blobs don’t get committed (once they’re merged in that history, they’re hard to remove!).

But the real value is that I now have a script that I can edit to analyze my codebase on every commit and ensure nothing has violated a growing list specific list of invariants!

Guix Shell

The Mythical Man-Month talks a lot about the importance of maintaining Conceptual Integrity in a program, and much of my research and thinking around Kai has evolved around this concept. Conceptual Integrity is the idea that your program should be aligned conceptually around what it’s supposed to be doing, and that this drifts as programs grow over time and abstractions get leakier.

One suggestion he has for maintaining conceptual integrity is having multiple implementations as a means to do ensure the design does what you want and is not too tied to your specific implementation. After all, one thing we’re trying to explore with Kai is the bounds of a generic determinate system – we need not and should not repeat Nix’s mistakes or tie ourselves too closely to their architectural mistakes.

So, just as a proof of concept for now that Kai will indeed work with two systems, I added just a single Guix shell command, which proved out that the command/backend/implementation architecture works for both Nix and Guix!

You can compile it with nix run .#xkai -- build plugins/guix/GuixPlugin.roc.

Then run ./kai shell guix.

One day, I hope this flexibility will allow us to surgically replace problematic parts of Nix, in an incremental and backward compatible way, with our own implementation of a determinate system!

Fuzzer

I’ve never fuzzed code before, and I’ll admit to ignorance on a lot of the theory and practice, especially with regards to practical benefit. But with Luke Boswell’s new Roc fuzzer library, and given that it was just a small bit of code surface to run for my parser (which fuzzers are apparently pretty good for), I decided to give it a try. Right now it’s only implemented for a portion of the parsing code, but it’s nice to know I have a way of easily and automatically testing thousands of inputs to the parser!

Dogfooding Kai

So far I’ve created two in-progress greenfield projects with Kai with the robot friend. This has proven a lot of the use case for Kai. But I’ll also be adding it to extant projects like lyceum.quest, conllu.lyceum.quest, and this personal website, to prove that Kai can be used both for hosting and deployments, as well as developer environments and builds. Essentially, Kai should be able eventually to cover most common use cases for things that happen around the code, so that you can spend more time doing what you want to do – focus on the code itself.

Aion

I built a very vibe coded (do not use yet!) tool to dogfood Kai with that I wanted for myself, a way for me to spin up NixOS servers on DigitalOcean at-will which spin up machines according to their Kai configuration. With this I was able, for example, to spin up an agent that “just works” and allows me to build things on that box with environment secret variables, services, packages, and just declarative computer setups in general, via just a single Kaifile!

This feels like a huge milestone, because one of the goals of Kai is to be able to augment/replace the traditional interface of using NixOS. If you can easily spin up simple to read declarative, version-controllable operating systems, it’s a massive game-changer for the headache of maintaining and remembering how you set up software environments. Your operating system setup becomes self-documenting, portable, shareable, trivially modifiable, trivially rollback-able, and now, because it’s not written in pure Nix, understandable!

The greater eventual ambition with this will be to allow people to use something like a KaiOS as a desktop as well, just like you can with NixOS. Then, maybe we can steal some of the Omarchy mindshare and prove to DHH that Nix-for-Desktop can be made easy ;)

There are a lot of knobs to think about before doing this in earnest, but we’ve already come a long way.

It’s completely open source and you can use it with your own Digital Ocean account (they’re one of the few that allow you to spin up custom NixOS servers).

Gump

With all the recent problems with Github lately, I figured I’d try my hand at a (again, very vibecoded) version-control system with a few twists.

I’m calling it Gump (short for git dump!). The goals are:

1. Self-hostable alternative to Github.
2. Allows for portable projects, not just portable code. i.e., issues, pull requests, wiki updates, etc. should all be concepts built into and portable and them selves version-controlled, like Fossil but for decentralized work like git.
3. Treats jj concepts as first-class as opposed to git. Like Nix, most people who’ve spent the time to figure out jj seem to far prefer it as a major innovation over git. Plus, unlike Nix but very much like Kai, jj is completely backward compatible with git and your teammates don’t even need to know you’re using it.
4. Easy to use CLI, usable by agents with configurable permissions.

The main goal, of course, is simply to work on Kai. It’s going to be extremely opinionated and bespoke to my wants, if I keep working on it at all. It’s going to do weird stuff like probably only use public key encryption for auth (the fact that most websites still do “check your inbox for an email code” is insane and backward). Or take matklad’s advice of queueing PR merges by default instead of waiting for the merge button to be green, then reviewing, then clicking merge.

Who knows? Maybe this will even lead to building a jj-based generic magit style editing experience too. There’s so much to improve upon.

There are a lot of problems we forgot because we were all just using git and Github for so long that we forgot to question many basic assumptions.

There are of course a whole host of things Github does really well that would be extremely difficult to replicate, but it’s also been really bad lately and the coding world is waking up to the idea that maybe it was not-so-great idea to make all of open source code in the world dependent on a single company.

I don’t want to distract too much from Kai though, and if I actually start using/depending on this I’ll need to throw the slop code away and rebuild something proper from scratch, but Kai development comes first and foremost and I need to stay focused on the bigger goal, but still, this is a fun idea to pursue.

Bootstrapping and Playing with Kai

You can try Kai without installing it by running nix run github:thebrandonlucas/kai -- version. That is how Aion and Gump keep their custom Kai plugins reproducible without requiring a global installation, while this package is not yet on nixpkgs.

Notes and Learnings

Making Code Smaller

Jonathan Blow has a talk where he has an interesting section about how “deleting code is always a good thing”. I’m not sure I agree completely, but a common theme I’m noticing in all these investigations is that conceptual integrity, and an understanding of the entire system is paramount to maintaining good code.

One aspect of understanding things well is having less things to understand. Deleting code while keeping it readable is one way to do this. This also means deleting anything that feels superfluous, inessential, etc. It means careful abstraction when helpful, and no abstraction when not.

Lines of code I think can generally be a helpful metric for whether code is getting smaller but it’s not the full story. If your code is very DRY but opaque, the true workings of it buried under layers and layers of indirection, what was the point?

Testing Must Be Done Up Front

SQLite creator Richard Hipp says in his excellent Software Should Work talk that the secrets to making SQLite probably the most-used piece of software in the world are:

1. Extreme portability. As Hipp says, “it’s just a file you can email to your friends!”
2. It just works. He primarily attributes this to an extreme approach to testing. In fact, there is far more test code in SQLite than code being tested!

I will admit with some shame that I do not have any test yet other than a single integration test that doesn’t block CI. At first, I just blindly told the bot to add tests “in the style of ‘How to Test’” which was silly and lazy and didn’t solve my essential problem: Helping to know what my code did (free documentation) and more importantly give me confidence that it worked.

The main reason I don’t have much testing is that I’m heavily debating and considering my strategy. How to Test says that tests should be cheap, operate at the boundaries of your interface, operate only on data (i.e. only do CPU work, no I/O), and therefore be fast, that they should be very easy to maintain (or else you won’t want to do them), and more.

That’s a lot of things to think about! Roc thankfully makes much of that easy by having a ! marker to declare effectful and effectfree (pure) code, and so if you just trivially separate the code which performs effects from the code which just operates on pure data (same input always results in same output). This means if you just test the pure functions your tests will run fast. But I was finding that my tests were adding a lot of surface area, that I didn’t understand how they worked or what they were doing, and that they therefore didn’t give me much confidence and in fact hampered my understanding of the code.

So, in the meantime, I wrote a zig build kaifiles command which just runs all the Kaifiles in the examples folder end-to-end. I figured that since the examples demonstrate and run through basically all syntax using the simple cowsay program, having those work and actually seeing their output gives me a lot of confidence that at least the happy path is consistently working. I didn’t block zig build ci with these since they’re neither pure nor fast, which may be a mistake, and that has bitten me (while writing this, I realized the command failed because I hadn’t run it in awhile!), so I may change that despite it being effectful.

But a major next step for me is figuring out and working hard on the right fast test boundary so I can start building more confidence moving forward that Kai doesn’t have many bugs.

I should probably also add an invariant to tidy.roc to verify that every block of Kai’s StdPlugin Kaifiles are used. As we add new blocks, the examples won’t include them all automatically without some rule.

Dependency Cultures

Richard Feldman’s Dependency Cultures talk at Software Should Work really took me out of my Javascript bubble and got me asking deeper questions about what tools I should be using.

There are real downsides to the Tigerbeetle approach of manually maintaining all of your own tools and needing to understand everything about your program. I don’t have a great grasp of assembly or the theory of electricity and yet I can still write code to make computers do things. Tools and dependencies do exist for a reason.

But.

There are two edges to this sword. If you take a short enough view, never having sought or gained a deeper understanding of how the things you use work, you’ll always be spending your time learning things which don’t last and are always changing.

The more time I’ve spent in the room with the creators of great things, they tend to have a deep understanding of the foundational structures of the tools they use and are building on, and as a result are much more versatile and capable. As new technologies slowly die in favor of newer technologies, they are less and less affected, because the deeper down into the foundations you go the less the knowledge you gained becomes irrelevant.

There’s a line between the practical utility of leveraging new technologies and the foundational stability of learning the old ones, and I think in tech by our very nature we’re drawn to the former more than we generally should be.

Often, if you think about the software you love the most that’s lasted the longest, it’s usually by people who spend a lot of time learning deeply.

The Tools You’re Using Are Enough

Relatedly, one of matklad’s articles linked to a John Carmack discussion about “just using the tools you already have”. Since this is being built in a language for which the compiler is being rebuilt, I run into a lot of bugs. So, because:

a) I’m excited to help in some small way contribute to this new language. b) I want to use more reliable, fast tools (who doesn’t?) and Roc’s model fits my concerns well. c) There’s less mental overhead and more conceptual integrity in using one tool for me. d) It’s fun and I get to do a bunch of mini-projects in the language. e) Not all code is equally important. The code that checks your code is less important than the code it’s checking, and I don’t need to anguish over every tiny detail.

Trunk-based Development

This article struck an idea in my mind that I can’t get out: every commit should add value to the codebase. This is hard to think about how to do properly, since features rarely lend themselves to single-commit completion! But there are tricks to do so for bigger features as noted in the article, and even in reviewing agent constructed code the idea of constantly merging things to master has been helping tremendously in avoiding not only cumbersome merge conflicts but just keeping the full program in my head, which is becoming more important as features keep getting added. The rapid development speed increases in adding more features provided by agents means that a lot of discipline is required to monitor them, reign them in, maintain control over what the code should actually be doing, and simplifying it. Agents have a huge bias in favor of bloat, and I’ve actually found it pretty hard to have them keep things simple, small, and easy to understand. Trunk-based development helps by reducing the number of concepts I’m having to keep up with at once.

Thoughts on Commits

Different projects have different philosophies on what a commit is, it seems, which has consequences for how your repo history is written and what history you allow to be merged. Mitchell Hashimoto, for example, thinks 1 PR equals 1 commit. Therefore a commit is the mergeable, PRable unit of change. So, the process would be something like: work on your feature locally, adding WIP commits or whatever as you go, then, amending, rebasing, fixup’ing, etc. when you’re done into a single commit that you push and merge.

According to Hashimoto, that final mergeable product should be +50/-50 ideally (easy to review, small incremental change) but no greater than +500/-500 (beyond that it starts getting difficult for a human to review).

Matklad has a looser definition, in that commit history can show the progress of how the final idea came to be. So a commit doesn’t need to necessarily be an isolated independent unit of work, the merge commit can be that, but it should tell a cohesive, easy to understand story about how you got there (and therefore you should still rebase/amend/fixup your WIP or exploration commits into something comprehensible by later review). He finds this useful as you can more easily trace the why and how for features and bugs when doing git blame, and if you have the merge commits, you already have the “final commit” at which that feature got merged.

I’ve been tending to favor Matklad’s approach on this, and trying to balance/reconcile that with this article’s trunk-based development approach where each commit can still provide independent value.

Also, I just started using jj, the new darling VCS of most who take the time to use it, it seems, and that also requires a huge rethinking of how to do all these things.

Regardless, I’m trying to pay much more attention now to the utility of the history.

AI Use

Matklad’s “Loop Closing” Strategy

Vibe coding has been both great and terrible for me. It’s difficult to find the right middle ground between having the robot do everything for you and doing everything yourself old-school. Obviously typing every for loop yourself should no longer be necessary. But I do think I should at least read every loop, and if I can’t understand the code (which I frequently can’t) that’s a red flag for me. This is becoming increasingly important and increasingly my bar: if I can’t understand the code as I read through it, it’s not good enough. This means that I either need to add comments (I plan on adding many more comments as part of cleaning up this codebase) or name variables something better, or either abstract more or even sometimes abstract less (abstracting can make internals unnecessarily opaque).

But one strategy I’ve been trying that’s been working as a nice middle ground (though I still need a bit more discipline) is:

1. Create a .gitignore’d directory called plans.
2. Inside plans, create a folder for my-feature-plan or my-bug-fix.
3. Inside my-feature-plan, create a spec.md.
4. Describe, in as much detail with as many real and specific examples you can of how you’d like the thing to work, the feature, fix, or refactor.
5. Tell the AI to create a plan.md which follows repository rules and splits the work into small, reviewable, independently value-adding chunks (Using Mitchell Hashimoto’s rule of +50/-50, no more than +500/-500 – a human (me!) has to read all the code after all!). Tell it to give examples of what it intends to do.
6. Review and refine that plan (I always find I underspecify and the AI didn’t get what I meant exactly).
7. Tell it “Do first chunk”, review that chunk, merge into master. Sometimes if I’ve got multiple things going I’ll be lazy and just have it do all the chunks at once but even with all the invariants, CI checks on every commit, and commit rules, I find this still bites me and I have to end up having it redo things. So I’m becoming more convinced that smaller chunks, each reviewed by a human before moving on, is still the way.

Focus, Motivation, Mistakes

Some final thoughts.

Focus Focus Focus!

Focus is extremely important yet really hard. I have to play all sorts of tricks on myself because the internet as it stands is adversarial by default: that is, more helpful than harmful to my overall happiness and effectiveness unless I actively take steps to make it a positive. I shouldn’t have to work to make a computer a net positive for me (which is a big part of what I want Kai to help with). This universal constant in my life should be a net positive by default, or at least trivial to make it one.

Maintaining Motivation

Read this fantastic blog on programming and just a lot of good general life advice.

Sometimes, I’ll find that reading and re-reading my code with the purpose of refactoring just doesn’t sound fun. Or thinking about testing strategy or whatever. And then I don’t want to do it. And then I find myself trying to reach for Yet Another Side Project instead of just making the software I want most (Kai) great.

I’m trying to employ several tricks to get around this, such as:

• Work in smaller and smaller chunks
• Walk away from the computer when tired and, if I still want to feel like I’m making progress, read a book on programming that would help (e.g. The Mythical Man-Month or The Cathedral and the Bazaar) read a blog article, or watch a talk.
• Sharing! Writing blog posts takes a lot of effort, even relatively simple or sloppy, plain-English ones like this one. But even just slight positive feedback from real humans who would like to use this is uber-motivating. And, it helps me think through things at a high-level again and remind myself what I’m really trying to do and forces me to understand the system I built better.
• Frequent releases (Mechanical Habits).
• Working on automating processes. E.g. creating a release script which makes releasing as simple as zig build release ....
• Releasing frequently. Doing “the whole vertical” from top to bottom for every small change or small set of changes, that is, writing code, running it, testing it, releasing it, and writing about it, all frequently, makes the whole process run smoother and be less stressful. If you deploy every small change to prod every day, prod deploys accumulate less cruft, become muscle memory, and each one has a lower surface area for problems. Many eyes make all bugs shallow much faster in that case. Plus, it’s more exciting for both you and people who like your project!
• Dogfood your project into another project. I can’t tell you how exciting it was to just say to an agent: “Build thing X. For anything that isn’t code itself, you are only allowed to use kai. No bash scripts, no python helpers, no nix. Just kai” and have it work! It’s a bit clunky, but in Aion and Gump, you really can do everything you need to do, from develop to deploy, using Kai!

Some other great sources I’m borrowing from are:

• matklad’s How to Test
• Mitchell Hashimoto

Mistakes

I’ve already found myself making a lot of mistakes with Kai:

• Overreliance on AI. It takes a lot of discipline to not just keep building and instead fixing what you already have and trying to make it great. That said, if I spend a whole year working on Kai in isolation, just to rip it apart and throw it away in favor of some new implementation when someone points out a completely better architecture, I won’t feel bad about not trying to achieve perfection on the first go before even sharing it.
• Not reading code enough (Don’t write bugs!). And, as a result, having some of the internals of things like the parser or assembler remain a bit fuzzy in my brain.
• Getting distracted. I think it was an overall positive to vibe code Aion and Gump (and vibing was absolutely the correct decision in their case – anything more would have meant spending an unjustifiable amount of time on two projects I may might 1. throw away or 2. distract from Kai too much).

A lot of upfront work went into ensuring modular design, especially so that new features could be added relatively cleanly under a contract that I liked. The abstractions aren’t perfect yet by any means, and it may still be a little confusing for a newcomer writing their own plugin. We’ll work more on that in some future release.

But I did feel it was good enough to finally build more of the features that would put kai to good use. That said, here are the features currently available in kai. These examples are all available in the latest code release:

As before, you could enter shells with certain programs:

# The simplest possible Kaifile.
#
# Run `kai shell` to enter this environment on Linux or macOS.
shell {
  packages: ["cowsay", "fortune"]
}

But now we have a more generic environment builtin which can be used to describe the same package set across the other new keys. For example: you can now create tasks and run them via kai run <task name>

# Run `kai run moo` to make the cow say a pithy quote!
environment cow {
  packages: ["cowsay", "fortune"]
}

task moo {
  environment: "cow"
  run: ["sh", "-c", "fortune | cowsay"]
}

Try running this with kai run moo!

You can also run tasks together as part of a workflow (think CI). Run the below via kai run cowvolution:

# Workflows compose tasks and builds into one repeatable command.
# Run this one with `kai workflow cowvolution`.
environment cow {
  packages: ["cowsay"]
}

task normalcow {
  environment: "cow"
  run: ["cowsay", "moo"]
}

task antiquitycow {
  environment: "cow"
  run: ["cowsay", "Non nobis solum nati sumus."]
}

task victoriancow {
  environment: "cow"
  run: ["cowsay", "doth mother know you weareth her drapes?"]
}

workflow cowvolution {
  steps: [
    "run normalcow",
    "run antiquitycow",
    "run victoriancow",
  ]
}

What if you want to package things together into a portably build-able derivation? You can! Here’s a little shell script which will combine cowsay and fortune to have the fortune piped into whatever cowsay has to say. Run via kai build <output-name> (in this case wisecow)

# You can build artifacts using the build command!
#
# This one combines `cowsay` and `fortune` to give you a
# `wisecow` binary.

environment cow {
  packages: ["cowsay", "fortune"]
}

build wisecow {
  environment: cow
  run: [
    "sh",
    "-c",
    "printf '#!%s\\n%s | %s\\n' \"$(command -v sh)\" \"$(command -v fortune)\" \"$(command -v cowsay)\" > wisecow && chmod +x wisecow"
  ]
  output: "wisecow"
}

Finally, we have overlays, to help people run things that may not be in nixpkgs (kai itself needs this ability)

# Overlays add packages that are not provided directly by nixpkgs.
shell {
  packages: ["rocpkgs.nightly"]
  overlays: ["github:thebrandonlucas/roc-overlay"]
}

That’s it for now! Stay tuned!

• You had to write custom parsing logic (terrible!)
• They weren’t strict definitionally and the concept of a Backend was unclear and vague (does that just mean nix? What if there is more than one requirement?)
• They were unvalidated. You could “do whatever you wanted” and received no help from xkai when you did something illogical
• Plugin modules themselves were not modular/reusable/easy to make

This release now updates that so the plugins should be much simpler to write, parsing is handled for you, a few validations are done by xkai to help ensure you built it right, and Backend was given a much stronger definition that represented the system much better.

Additionally, I made a custom DSL! Even though the previous embedded roc DSL was pretty thin, I realized I could make things much simpler if I made a custom one. The primary motivation to do it in an embedded language was to inherit things like conditional logic, but I really only needed it for switching between systems (e.g. if system == linux then install ...) and that alone did not justify the extra complexity. Plus, it would require every user to have both a determinate system installed like nix and the roc compiler. Instead, we remove that extra 30MB or so and the final kai binary comes out to about 600KB! This does come at the cost of a lot of gutsy config parsing and validation, which is annoying, but I think the tradeoff is worth it.

But the best part is now, the simplest config.kai (which I’ll soon rename Kaifile) looks like this:

shell {
    pkgs: ["cowsay"]
}

With just this, you’ll be able to define reproducible dev environments! For changing behavior on different systems:

on linux {
    shell {
        pkgs: ["cowsay"]
    }
}

on macos {
    shell {
        pkgs: ["pokemonsay"]
    }
}

Much simpler and straightforward than a flake.nix, that’s for sure!

Onward!

• Its usefulness
• Its modularity
• Its evangelistic efforts
• Its ease of use

I believe this has become orders of magnitude more important in the Age of AI, as code (which was never the most expensive piece of the software development lifecycle anyway) has become incredibly cheap to produce. I also believe, perhaps controversially, that the principles of software development as established by its progenitors are even more relevant, with perhaps some tweaks and modifications.

A recurring theme, whether it be in such classics as the Unix Philosophy or The Mythical Man-Month or The Cathedral and the Bazaar or modern works like Working in Public, is that good software should be modular. Think of the success of LEGO. A wonderful toy that embodies the spirit of open source software. LEGO’s have outlived many a toy because their design allows them endless reconfigurability and modifiability. If grandpa builds a castle from LEGO’s, then dad can add a dragon, and son, having slain it, can finally give the freed princess’ tower a long-overdue remodeling. This means that the efforts of the tower have not been lost, from grandpa to father to son, that there is a deeply human history and personalization to that tower.

Code works the same way. Linux is the undisputed king of infrastructure OS, and will probably overcome Windows and Apple: as the latter continue to get worse, piling on more and more new bloatware on already inferior foundations, their software will become more and more intractable to fix while the former continues to get more featureful, more useable, more battle-tested.

Thus, a modular approach will allow the code to be more flexible, interoperable, changeable, robust, and last much longer.

Making Kai Modular

The three concerns for kai modularity outlined in previous articles are:

• Allowing replacement of any standard command with custom logic
• Allowing commands to be added since we can’t anticipate every use case
• Allowing piecemeal replacement of underlying nix or guix pain points (such as the famously slow nix evaluator) with custom interoperable solutions

I modeled the design after caddy, the best web server technology I’ve ever used, which has great defaults, is fantastically easy to use and configure, and is extremely modular.

I created a separate binary, called xkai (analagous to caddy’s xcaddy) which creates custom kai binaries which execute the behavior specified in a custom Plugin.roc file. For example, the “core library” of kai is StdPlugin which looks like this:

import kai.Plugin as PluginApi

# The standard plugin is pure. kai.roc imports its typed configuration
# constructor; only the generic executor performs the resulting actions.
# TODO: it's explicitly limited to nix for now. This might be long-term
# desirable behavior, but we may want to consider providing guix support
# out of the box as a fallback. Or keep it as a separate Guix plugin/binary.
StdPlugin := [].{
    ShellConfig : { pkgs : List(Str) }

    nix : PluginApi.Backend
    nix = PluginApi.Backend.{ name: "nix" }

  # Only support the nix backend for now
    backends = [nix]

  # We have one command, shell, which just creates a nix dev shell
  # with a flake and enters it.
    shell_command : PluginApi.Command
    shell_command = PluginApi.Command.{
        argv: [],
        backends,
        name: "shell",
    }

  # Whereas `shell` is a portable `Command` variable, `nix_shell`
  # is the actual implementation that tells the `kai` binary resulting
  # from `xkai build` on this plugin which side-effects to perform.
  #
  # Separating these two concepts is important - if we supported, say,
  # `guix shell`, then we could have one command in kai which would run
  # `guix shell` or `nix shell` depending on which system the user wanted
  # to use. So it's very flexible!
    nix_shell : PluginApi.Implementation
    nix_shell = PluginApi.Implementation.{
        actions: [
            WriteConfigUtf8({ path: ".kai/flake.nix" }),
            Exec({
                args: ["develop", "path:.kai#default"],
                command: nix.name,
            }),
        ],
        backend: nix,
        command: shell_command,
        requirement: Program(nix.name),
    }

    shell : ShellConfig -> {
        implementation : PluginApi.Implementation,
        rendered_config : Str,
    }
    shell = |shell_config| {
        implementation: nix_shell,
        rendered_config: StdPlugin.render_nix(
            shell_config.pkgs,
        ),
    }

  # Render the actual flake, which will be written to a managed
  # directory called `.kai`
    render_nix : List(Str) -> Str
    render_nix = |pkgs| {
        package_lines = pkgs.map(
            |pkg| "              nixpkgs.\"legacyPackages\".\"x86_64-linux\".\"${pkg}\"",
        )
        lines = [
            "{",
            "  inputs.nixpkgs.url = \"github:NixOS/nixpkgs/nixos-unstable\";",
            "  outputs = { nixpkgs, ... }: {",
            "    devShells.x86_64-linux.default = nixpkgs.legacyPackages.x86_64-linux.mkShell {",
            "      packages = [",
        ].concat(package_lines).concat([
            "      ];",
            "    };",
            "  };",
            "}",
        ])
        Str.join_with(lines, "\n")
    }
}

# Plugins are pure, so we can do cheap, no I/O tests!
expect {
    module_config = StdPlugin.shell({ pkgs: ["hello"] })
    module_config.implementation == StdPlugin.nix_shell and
        module_config.rendered_config.contains(".\"hello\"")
}

This means that plugin creators need only write this plugin file, run xkai build on it, and they have their own custom kai binary! Which is, of course, how I will also be developing kai.

Thinking through this design has been by far the most difficult aspect of this project so far, and required a lot of research up front. But I am profoundly satisfied with this architecture.

Although kai is still very much a prototype which needs much better testing, interface, and of course, feature work, this is beginning to feel like a very solid and extensible foundation which will allow not just me, but anyone, to easily contribute meaningfully to this in the future.

Onward!

A lot of work went into doing that very basic thing that is a near-zero improvement over nix itself, because so much went into the underlying design that I’m hoping will make Kai a Zero to One tool: an upgrade to the interface of deterministic computers so delectable and powerful that it will make the human experience, tooling interop, and agentic efficiency orders of magnitude better.

Finding a Boundary

One of the main ideas behind Kai is for the user to manage a simple config file, kai.roc, which can serve as a simplified interface for interacting with your determinate system (for the vast majority, Nix under the hood). This is hard, because:

• A) Nix has so many layers of the stack, e.g.:
  • Nix the language, used by:
  • Nix the package manager, used by:
  • NixOS, along with:
  • The nixpkgs package repository

That’s a lot of layers interoperating in a “closed” way because:

• B) Nix built these monolithically, such that it’s very hard to take one piece of that stack and use it for anything useful outside of the broader ecosystem. This means two things:
  • Boundaries are difficult to isolate at the conceptual level (essential complexity baked into the Nix ecosystem).
  • Boundaries are difficult to isolate at the code level (accidental complexity due to years upon years of cruft, ad-hoc development, and no real competition leading to a lack of need for interop).

All of this together forms a natural walled garden: unlike Apple, which intentionally fights its users by hiding code for the sake of (Apple believes) reduced complexity and financial incentives, Nix seems to have created one by accident. While explicitly encoding preferences for open software (via, for example, needing to manually modify configs to set a flag allowUnfree = true) and a philosophy which advocates maximal user control over their systems, in practice it is so complex and abstruse for the average daily user, with so many layers, that anyone who wishes to improve upon or isolate a single piece faces a very effective barrier to general adoption because, in sum:

• A) Users can’t figure out how to use it, and:
• B) Developers don’t have a middle ground between improving on the current ecosystem (thus adopting its current problems and working backward to solve a limited subset of things it can), or creating new systems (which, despite the profundity of the idea, is so difficult to do—see all the many layers above!—that only one has even tried: Guix, and its community is very small).

Enter Kai

Kai proposes a middle way. Kai acknowledges two fundamental facts:

1. Nix dominance
2. Nix “Accidental Complexity”

Nix has the largest package ecosystem in the world. A stunning achievement for a package manager which isn’t the most well used, and an easy testament to the power of how reproducible software allows you to ship. Aside from that, it has seen slow increased adoption over the years despite continually fracturing communities and continuously deprecating APIs: even frontier LLMs get frequently confused about the best way to use Nix, because so much of the documentation teaches deprecated practices! Finally, there are no real competitors: imagine trying to build a completely unique package manager which uses a custom language and build a custom operating system from it while maintaining the largest package ecosystem on Earth, without being a for-profit company. Despite valid criticisms of Nix, this is a staggering achievement! One that is extremely difficult to replicate.

On the other hand, Nix suffers from so many problems which don’t need to exist. I sometimes wonder if its creator, Eelco Dolstra, thought that some enterprisers would take his idea and carry it out to permanently improve the world by eliminating the whole class of software deployment problems, in the way that, say, Rust did for programmers. Maybe he, like Benjamin Franklin when he named the poles of electricity positive and negative, thought that someone would take his discovery and formalize it more appropriately and give them better names, whereas instead, we are just stuck with these terms, and we are still stuck with Franklin’s conceptual mistake of inverting the logical poles. And thus we are still stuck with many of the tradeoffs of Nix that are probably not worth having, despite the scope of the downstream improvements of using it over our current imperative systems being enormous and fundamental. Here we are over twenty years later, still building on top of inferior foundations. A huge portion of this I think is to be blamed on Nix’s design at basically every level.

Taking these into account, Kai proposes to do the following:

1. Be backward compatible with Nix. Indeed, to simply use Nix directly under the hood at first.
2. Remove as much accidental complexity from using Nix as possible. Primarily through simplified UX and useful missing features or abstractions.

Alan Kay’s motto is my North Star here: “Simple things should be simple; complex things should be possible.” That is, make Nix (via Kai) as easy as possible to use while allowing users to do anything they’d normally be able to do in Nix (even modify/create raw Nix files where they find they can’t do something in pure Kai).

This has resulted in me spending a lot of time seeking the appropriate boundaries to expose. After settling on Roc as the config language to use, I posted my idea on the Roc forum, and to my amazement, Luke Boswell created a working prototype of my idea, roc-blueprint, overnight!

This was great, as roc-blueprint provided an example of a general, pure, typed, and portable backend which could then be lowered into a compatible backend of choice (i.e. any reproducible system). In reality there are only two: Nix and Guix. Then this lowered Roc representation (itself another platform) could be used to instantiate and run actual respective commands. For example, here’s a program that provides a portable representation of a Nix shell which contains the pokemonsay program:

app [main!] {
    pf: platform "https://github.com/lukewilliamboswell/roc-platform-template-zig/releases/download/1.0.0/AnZoxzoGPtSGQ15EQh6pBeeaHJ7aizP9MQhK81dES3Uq.tar.zst",
    blueprint: "../../blueprint/package.roc",
}

import pf.Stdout
import blueprint.Blueprint
import blueprint.Requirement
import blueprint.Target
import blueprint.Environment
import blueprint.Nix

pokemonsay : Requirement
pokemonsay = Requirement.new({
    id: "pokemonsay",
    display_name: "Pokemonsay Program",
})

workspace : Blueprint.Draft
workspace = Blueprint.workspace({
    name: "pokemonsay shell",
    target_systems: [Target.X86_64Linux],
    envs: [
        Environment.new({
            name: "default",
            requirements: [pokemonsay],
        }),
    ],
})

nix_config : Nix.Config
nix_config = Nix.config({
    nixpkgs: Nix.github_input(
        "nixpkgs",
        "NixOS",
        "nixpkgs",
        "nixos-unstable",
    ),
    bindings: [
        Nix.bind(
            pokemonsay,
            "nixpkgs",
            ["pokemonsay"],
        ),
    ],
})

main! : List(Str) => Try({}, _)
main! = |_args| {
    valid = Blueprint.validate(workspace) ? |errors| BlueprintInvalid(errors)
    source = Nix.render(valid, nix_config) ? |errors| NixInvalid(errors)
    Stdout.line!(source)?
    Ok({})
}

Cool! But notice any problems?

1. Boilerplate-galore! The actual flake.nix this renders is fewer lines than the config! (21 vs. 54)

# Generated by roc-blueprint and roc-blueprint-nix. Do not edit.
{
  "description" = "Development environments for pokemonsay shell";
  "inputs" = {
    "nixpkgs" = {
      "url" = "github:NixOS/nixpkgs/nixos-unstable";
    };
  };
  "outputs" = { nixpkgs, ... }:
    {
      "devShells" = {
        "x86_64-linux" = {
          "default" = nixpkgs."legacyPackages"."x86_64-linux"."mkShell" {
            "packages" = [
              nixpkgs."legacyPackages"."x86_64-linux"."pokemonsay"
            ];
          };
        };
      };
    };
}

So the resulting config file is more complex than what it is simplifying, with this caveat: we define the portable (blueprint platform) and the non-portable (blueprint_nix platform) in the same file, the benefit being we can reuse blueprint programmatically and theoretically have it port to any number of determinate system protocols under the hood (eventually, one config could boil down to a Nix or Guix system). Another benefit is we get strong typing: unlike flake.nix, the individual pieces are strongly typed and thus linters and checkers could be built upon this to much greater effect. So these are some great benefits, the strong typing and portability alone we may even call Zero to One benefits, but we have only achieved goal (1) above:

Be backward compatible with Nix. Indeed, to simply use Nix directly under the hood at first.

But we’ve made (2) worse, adding complexity for arguably marginal gains for the average user:

Remove as much accidental complexity from using Nix as possible. Primarily through simplified UX and useful missing features or abstractions.

There are also other problems: for example the user must have a compatible Roc compiler installed, and they have to do roc main.roc to run it.

And even if we solve the above, this is an incomplete solution: the platform design is great, but must we really expect every user to just import this into their own programs, and write the logic to lower it?

I thought hard about what to do here, because the pros of this design are real, and I wanted to keep it for the time being as programmers might find this level of control useful, but for the vast majority of people a simpler config is in order. So I wrote one abstraction layer on top of it. Where the previous design looked like this:

[portable blueprint] -> [nix blueprint] -> [nix files]

The current one adds:

[kai.roc config] -> [portable blueprint] -> [nix blueprint] -> [nix files]

The kai.roc sacrifices a bit more detail and control for simplicity and looks like this:

app [config] {
    kai: platform "../../platform/config.roc",
}

import kai.Kai

config : Kai.Config
config = {
    name: "pokemonsay shell",
    shell: {
        pkgs: ["pokemonsay"],
    },
}

Much better! 13 total lines, and only 5 are config that the user needs to consider in the base case! But the idea is that the user will be able to move down and use the tool at any layer of abstraction as suits their needs. I’m not sure whether I’ll keep roc-blueprint at all yet, as the benefits of an intermediate representation between high-level abstraction and low-level control must be weighed against the need for maintenance simplicity, and I’m not sure yet of the value proposition when the simpler model exists.

To achieve this simplicity over roc-blueprint, we make some assumptions on:

• The distinction between a validated blueprint and a Draft workspace
• The Environment concept
• The machine architecture (we remove Target.X86_64Linux from above, for example)

But I think many of these decisions can appropriately be outsourced to under-the-hood management by a CLI tool, which is the second part of the solution.

For example, we don’t have to encode the architecture in the config mandatorily if the CLI tool kai can just save those details in the config (or just default to building for all default/common systems).

Right now, the CLI can only do kai shell. It is dependent on your machine having Nix (and Nix is still hardcoded in) and most egregiously on being x86_64 Linux (sorry!), but this will be modularized away later.

Mitchell Hashimoto built an excellent Zig overlay flake which allows you to get the nightly version of the Zig compiler with a single Nix command. At the time of this writing, Zig 0.16.0 is the latest stable release. Some projects intentionally track Zig’s rapidly changing development branch instead. Ziglings, for example, currently requires a sufficiently recent 0.17.0 development build and does not compile with Zig 0.16.0! Although Zig’s official binary can run on NixOS, integrating it reproducibly and ensuring host linking works correctly requires additional Nix setup. If zig-overlay didn’t exist, I’d have had to roll a custom solution just for myself. An overlay/flake is thus useful for any project not considered stable to allow anyone to have a reproducible/pinnable or just bleeding-edge environment for any version indexed by the overlay. It makes compatibility and environment setup orders of magnitude easier to achieve. No more hunting down binaries! (or hunting down and patching if you’re on NixOS).

The newer and more in flux the project, the more useful this is. And with the proven utility of the zig-overlay template, I decided to build one for an even newer programming language, Roc. At present, Roc releases nightly binaries of the compiler. They are currently replacing the Rust compiler with a new scratch Zig rewrite, which has breaking changes to Roc’s design such that code and platforms targeting the old compiler are usually incompatible with the new compiler, and have not made a stable release of the Zig compiler release yet. Thus, there’s a lot of incompatibility between apps which were built using the old and new Roc compilers, since the new compiler introduces a lot of backward-incompatible language features. So, being a nix enthusiast, I thought it would be helpful to create a roc-overlay modeled after zig-overlay, allowing tracked versions of the nightly compiler to be selected by a developer, or even better, declared as an input in flake.nix and pinned to an exact repository revision in flake.lock for given projects so the developer doesn’t even have to think about which nightly to use.

roc-overlay Contents

roc-overlay provides a Nix flake for the official Roc new-compiler binaries. It installs prebuilt archives and does not build from source. It mirrors zig-overlay in overall structure. Use it like this:

# To get the latest nightly as seen by the repo,
# run one of these:

# Create a shell with the compiler.
nix shell github:thebrandonlucas/roc-overlay
roc version

# Run one-off commands directly from the flake.
nix run github:thebrandonlucas/roc-overlay -- version

Including the overlay in your flake.nix:

   {
     inputs = {
       nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
       roc-overlay.url = "github:thebrandonlucas/roc-overlay";
     };

     outputs = { nixpkgs, roc-overlay, ... }:
       let
         system = "x86_64-linux";
         pkgs = import nixpkgs {
           inherit system;
           overlays = [ roc-overlay.overlays.default ];
         };
       in
       {
         devShells.${system}.default = pkgs.mkShell {
           packages = [ pkgs.rocpkgs.nightly ];
         };
       };
   }

Hopefully you find this overlay useful. Please open an issue if you find anything that needs improvement.

Simple things should be simple, complex things should be possible.

• Alan Kay

nix, and the whole ecosystem surrounding it, is the only major attempt at a top-to-bottom solution to the problem of reproducible software. There is one other that has implemented this (guix), but its mindshare is dwarfed by Nix’s, which is in turn dwarfed by other package managers and Linux distros, which are in turn dwarfed by proprietary programs and operating systems like macOS and Windows.

Nix is, most fundamentally, a means of getting software from machine or environment A to machine or environment B in a deterministic way: that is, as long as you can use nix on both machines to perform the transfer and the target machine has the resources to run the program, you can guarantee that the software will be transferred completely and have all the dependencies it needs to run. No other major package manager does this.

Technically this is a big leap forward, but, given how few use nix, we can’t yet call it that. It’s no secret that nix is notoriously difficult to use. But it seems to me that many of these problems are problems of accidental complexity (problems engineers create through suboptimal design and can therefore fix) rather than essential complexity (problems inherent to the problem itself). Since Nix is very old (older than git!), there has been a lot of churn on tooling, which has resulted in a lot of tooling, a monolithic architecture, split communities, outdated documentation, and many many many third party solutions aiming to solve (and re-solve) the same accidents of complexity.

nix does not do one thing well nor is there one obvious way to do any given thing: simple things are complex and complex things are complex.

That said, at its base, nix is a beautiful idea, and an elegant solution to the software distribution problem. It enables so many positive downstream effects. It makes your package management and operating system declarative, and brings the deterministic, mathematical certainty of a functional language to your computer-using experience. That is power.

Thus, we have a revolutionary tool that basically no one uses because it is very difficult and the ecosystem is fragmented. Therefore I believe that the consequences of making a determinate system like nix easy to use will have a profound impact on how we interact with computers for the better.

As such, this problem is begging for a solution.

Kai - Prototyping a friendlier frontend for determinate computing

I’ve begun working on one such solution which I’m calling kai, short for the Ancient Greek καιρός or “the right time”. At first glance, it is just lipstick on nix, which alone would be great. But with the proper design we can architect it to be much more.

The goal is to:

1. Define a generic protocol which enumerates a set of capabilities any determinate system should have, such as:
   • Reproducible developer shells for teams and projects (probably the most common nix use case now)
   • Ad-hoc shells for testing out software on the fly
   • Build targets allowing you to create virtual machines, or deployments with the specified software and configurations preinstalled, or the ability to create custom ISOs to roll your own operating system configuration!
   • Rollback to previous generations if a new dependency breaks the system
   • Automated garbage collection for dangling dependencies no longer in use
   • etc.
2. Implement the protocol in a CLI tool and config file such that it is compatible with the two full-fledged determinate systems (nix and guix) and any others which later implement the generic protocol, eliminating as much accidental complexity as possible and making them friendly and fun to use. The mappings of the generic protocol to the concrete implementations we call blueprints (thanks Luke Boswell!)
3. Make the tool modular such that users can:
   • Take only those parts of the determinate system which they want (some may only be interested in reproducible shells while others want build systems – there should of course be a “default” option which has everything the average user wants).
   • Modify or create their own implementation of a given tool. For example, tvix spawned out of a desire to make a modular nix implementation since the primary implementation is monolithic, the problem being that if you don’t like the way a given piece works or believe it is inefficient, you cannot easily swap it out for a different implementation and must satisfy yourself with the monolith.
   • Add new modules and commands as desired (for example one may want a kai deploy which handles deploying to their VPS of choice).

This clearly goes beyond “make a better nix frontend”. The motivation is that this design will allow us to experiment and play with designs for a better system while being backward compatible with existing ones. And further, replace components of one of the existing systems (e.g. replacing one of the nix components with a tvix one, but still having the total package management solution be nix-compatible).

It is built in the roc programming language, a high-level functional language which allows us to create platforms (in a lower-level language, zig, in this case) which define behavior tailored to specific use-cases. For example, in our case we create a platform to specify an EDSL (Embedded Domain-Specific Language) to implement the determinate protocol.

So far, kai is just a proof of concept which implements a basic version of each of the three ideas above. The nascent determinate protocol so far has two commands in the CLI, shell and build:

• shell: Enters a shell containing the packages specified in kai.roc by calling the corresponding shell command. This is equivalent to a flake.nix for nix and a manifest.scm for guix. If none exists in the current directory, it scaffolds a kai.roc file with no packages. Example:

  app [config] { kai: platform "../platform/config.roc" }
  
  config = [
      Shell({
          name: "simple-kai-shell",
          # `packages` is a Roc header keyword in this compiler, so shell configs use `pkgs`.
          pkgs: ["git"],
      }),
  ]
  

• build: Builds the specified machine image according to kai.roc. Technically this should be a generic build for any target, but I’ve chosen a very common use case for the time being for simplicity. Example:

  app [config] { kai: platform "../platform/config.roc" }
  
  config = [
      MachineBuild({
          hostname: "kai-example",
          system: "x86_64-linux",
          install: ["git"],
          ssh_keys: [],
          state_version: "25.05",
          image: { format: "qcow2" },
      }),
  ]
  

There is also a third command, blueprint, which allows you to choose the backend which implements the determinate protocol. In our case we have two blueprints which can implement shell and build: nix and guix. So if you blueprint list to see your options, then do blueprint set nix, calling shell will perform nix develop under the hood and build will call the appropriate command to build the machine above.

There is also a .kai directory which manages a config file corresponding to the blueprint. For nix, .kai will contain a flake.nix and our build and shell commands will manipulate and use that under the hood. For guix, the file is manifest.scm.

Next in the plan is to add a bespoke deploy command modularly, such that deploy can mean different things for different people, and we can both (1) show the power of kai to easily deploy machines preconfigured with software, and (2) show how people can extend kai commands to suit their various use cases.

Onward!

…to read the Latin & Greek authors in their original is a sublime luxury … I enjoy Homer in his own language infinitely beyond Pope’s translation of him […] I thank on my knees him who directed my early education for having put into my possession this rich source of delight: and I would not exchange it for any thing which I could then have acquired & have not since acquired.

Thomas Jefferson, Thomas Jefferson to Joseph Priestley, Philadelphia, 27 Jan. 1800

But of the mere facts of history, as commonly accepted, what educated youth of any mental activity does not learn as much as is necessary, if he is simply turned loose into an historical library? What he needs on this, and on most other matters of common information, is not that he should be taught it in boyhood, but that abundance of books should be accessible to him.

The only languages, then, and the only literature, to which I would allow a place in the ordinary curriculum, are those of the Greeks and Romans; and to these I would preserve the position in it which they at present occupy. That position is justified, by the great value, in education, of knowing well some other cultivated language and literature than one’s own, and by the peculiar value of those particular languages and literatures.

John Stuart Mill, Inaugural Address, 1867.

I’m building a new open source digital library for Ancient Greek using AI.

With the rise of LLMs, we have an opportunity to create new tools for preservation, learning, and engaging with ancient texts that would have taken massive amounts of dedicated and skilled work to produce before.

The tools which pioneered the work of bringing Ancient Greek to the internet can be credited to projects like Perseus Digital Library, Scaife Viewer, and others. Building on top of their work in the age of AI can yield results previously unfeasible.

Thus, there is a unique opportunity to initiate a revival of interest in the works of Antiquity in their original language; to let us hear the authors speak again with their own voices.

To this end I created a set of individual skills, coordinated as a pipeline by a meta-skill text-orchestrator, for creating this database automatically. The goal is: given a single prompt like “Add Plutarch’s Lives”, it will search the web, find the original Greek edition, find public domain English translations, download them, process and clean them, segment them into verses (or create a custom versifiable translation if no extant translation aligns well), create interlinear translations, create Latin transliterations for pronunciation, and create treebank data for each word.

This is still only in the “proof-of-concept” stage, and there are many improvements to make, but it has successfully run through the full pipeline on a few smaller sample texts. Therefore text-orchestrator could be an extremely powerful tool for automating database generation from a single prompt. The data it collects is optimized for learning purposes, and the pipeline ends by generating a database artifact, texts.db, which can be directly imported into the lyceum.quest Ancient Greek reader, which is a companion project to this one. It acts as the interface for the data and is my attempt to bring together all the best aspects of Perseus, Scaife, and other readers online, and improve them where possible.

I also show how I ran Karpathy’s Autoresearch to fine-tune an alignment skill which generates English interlinears for every word of the Greek text. We should be able to use this method to fine-tune any skill in the pipeline, but that is reserved for future work.

Table of Contents

• Why
• Goal
• Limitations
• AI Generated Data
• Agent Optimizations and Workflow
• Desired Data
• Generate, Verify, Optimize Loop
• The Text Orchestrator Skill Pipeline
• Future Directions
• Bottlenecks
• Conclusion

Why

I believe this revival is imperative in a time where technology and culture allow and encourage a distancing from our roots which leaves us further untethered from a deep historical understanding of ourselves and our “story” in the world.

As a result nearly all of the institutions for which learning Greek was once a rite of passage in any serious education are gone, and the curious self-driven student is mostly left with fragmented resources for learning.

Further, the attention economy and modern technology constantly competes with any new learner’s focus.

This deadly trifecta of difficult and fragmented learning resources, with a centuries-long cultural cooling on and de-institutionalization of the classics, and deep distraction-technology integration into our personal lives are deep barriers to overcome, but the rise of LLMs can, I believe, help overcome all three.

I take John Stuart Mill’s view that history is fundamentally interesting in and of itself, and if these three barriers are overcome, students would come flocking back to learn.

Therefore, reviving interest in learning ancient languages mandates lowering those barriers to entry and making the learning process as engaging and as easy to dive into what the student is there to learn as possible.

With LLMs, we can do this.

As Dr. Gregory Crane, founder and head of the Perseus Digital Library, notes:

Now we have multiple Large Language Models (LLMs) that can not only translate but also provide reasonable word-by-word grammatical explanations of any source text. My first goal is to provide my students with enough knowledge so that they can begin to exploit this rich and increasingly complex scaffolding.

[…]

LLMs are slow and frontier models are expensive but, given the money, there is no reason we could not generate literal translations for all Classical Greek and Latin literature.

Dr. Gregory Crane, Towards corpus-based learning: exercises with Ancient Greek, 2026.

Goal

My primary goal is three-in-one: Create the largest corpus of publicly available online Greek texts with the richest set of accompanying data to allow us to build the most intuitive, engaging, and useful interfaces possible for both learners and experts.

If this is successful, we can expand to other ancient languages, starting with Latin.

I’ve already built the initial version of the interface at lyceum.quest. See the announcement post for the dashboard here.

The proof-of-concept has been validated in showcasing a taste of what’s possible now and is already one of the most feature-rich interfaces available. It aggregates much of the freely-available data online. It makes available side-by-side Greek-English translations with the Greek fully parsed, such that each word of Greek text becomes interactive. A user can click on a word to see a popup containing definitions from each major Greek database (Perseus, Logeion, and LSJ), see morphology and grammar data, and even see an AI-generated contextual meaning. It also has numerous display enrichment options like color highlighting for inflections and grammar, and an Anki-style spaced repetition deck for memorizing words from a reader’s chosen texts.

We can, however, do so much more.

For example, what I personally felt was missing most from my own learning were Hamilton-style word-by-word interlinear translations, which are rare to find online, as that would fill a primary gap for me when learning. So I had Claude generate them custom for the Iliad, which was pretty token-heavy and expensive. But the value of having disambiguated word meanings was immediately apparent. Further, it’s extremely important to note that for most Greek texts, interlinear translations have never been done. Even for the Iliad itself, I was unable to find a complete interlinear English translation (Hamilton himself, the champion of this method, only did this for a few books of the Iliad).

In short, it takes a tremendous amount of effort even from expert humans to create interlinear translations, which the world would greatly benefit from having. This is just one example of a real gap here that could benefit greatly from AI.

Limitations

A major goal is to do this for as many texts as possible, starting with the most desired texts like Homer, Plutarch, The Gospels, or Meditations. But I was generating these interlinear translations through manual prompting, with crude trial-and-error experimentation. I eventually learned to feed it “ground truth” data (using Hamilton’s Iliad interlinear as a sample) so that it could develop an alignment skill from which it would produce Hamilton-like interlinear translations for a given text. I was able to achieve moderate success with this but still found the method too manual, ad-hoc, and error-prone, and translations would not always disambiguate the word correctly for the given context.

Two things quickly became clear:

1. There was much more we could do than merely interlinear translations.
2. My current approach of prompting AI per-translation was expensive and unscalable.

For (1), I realized I wanted more than interlinears. In one reader view, I wanted:

• Full prose translations (as many public domain as I could find)
• Interlinear beneath each word
• Segmented verses, like the Bible (enabling side-by-side views)
• Transliterations (the Latin rendering for pronunciation help)
• Treebanks containing grammar, inflection, and other morphology data for each word (ideally color coded for easier sight-recognition)

This way, information could be progressively revealed to the reader as it suits them. They can try to learn each word one at a time, learn how it fits within the phrase or sentence, then learn how that sentence fits within the greater narrative. This level of granularity exists at the word level.

This way, the text can truly “come alive” as each word now represents a rich treasure trove of data that a reader can dig into at their preference and to their taste.

So a single Greek text can go from this:

To this:

These are just a sample of the things we can do with static, one-time LLM use. We could also add interactive AI chat elements like highlighting and asking about a passage, like ancientgreek.jean.land does, for example. Or a chat just to talk about a specific work or ask general questions.

Eventually, we could even use AI speech-to-text (STT) and text-to-speech (TTS) features to help students learn and grade their pronunciation, or read passages to them.

AI Generated Data

For now, we will focus on what can be done up-front with AI. The tradeoff we are making at present is that the high costs of doing this work are all in the initial phase. Adding in dynamic AI features can come later. A good static data set for translations/transliterations etc. only needs to be done once and is then available forever.

Some, like ancientgreek.jean.land, have taken the approach of using AI to quickly generate translations on the fly, which is a great novel feature. But this won’t scale and should be long-term unnecessary. We need not spend tokens on any of the work required here any more than is necessary to produce a correct result. Therefore, what’s really needed is a large token budget to:

1. Refine a process for generating the translations
2. Create translations using it

And then a budget for human Greek experts to:

1. Manually audit and correct the data produced by the process itself
2. Optimize ground-truth “holdout” data used by the agents to evaluate the quality of their work by (and automate improvements)

Agent Optimizations and Workflow

LLMs have recently passed a capability threshold in which many programmers find themselves no longer writing code, but exclusively prompting instead. This has led to massive debates on the true efficacy of LLMs: whether the code they produce is truly high-quality, functional, and maintainable, or “slop” masquerading under a flashy veneer. There is significant skepticism in the Classics community as well.

Despite valid misgivings, it appears that with the right workflows AI can produce surprisingly accurate results, which can then be refined by a human expert.

So if AI can translate accurately enough, this turns the problem from one of translation generation to one of validation, and the human cognitive load for validating work is orders of magnitude easier than the act of creating it.

This means we can, perhaps, for the first time in history, have a hope of creating a unified corpus rich interlinears, morphology data, versifications, and more to provide progressively enhanced experiences when engaging with ancient texts, all-in-one, as opposed to the fragmented efforts which exist today.

That said, the current state of the art appears to involve the heavy use of markdown files like SKILL.md and AGENTS.md along with intelligent context management (context being the primary scarce resource for agents). I’ve attempted to use best practices where possible to build a scalable pipeline for Lyceum which can take a simple query like “Add Plutarch’s Lives” and end with a rich set of data which includes the texts, translations, generated interlinear, etc. which are ready to import into the reader.

Desired Data

As I was working through this it soon became clear that we may be able to automate the following:

• Interlinear translation
• Latin transliteration
• New Testament-like verse segmentation of Greek/English for phrase-level learning
• Custom “versified” translation if no English translation aligns well (ex. Homer translations are rarely direct)
• Treebanks

Each of these would have been extremely time-intensive pre-AI, but following a Generate-Verify-Optimize loop (below), this could become a far more feasible task of auditing and verification rather than generation from scratch.

Generate, Verify, Optimize Loop

Using agent skills, if we apply an approach of Generate-Verify-Optimize feedback looping, we can refine texts through a combination of agent-based methods like Karpathy’s Autoresearch and human feedback loops. For any skill we have which generates data, like /skill:create-interlinear, we craft human-expert curated data we can verify against from which the agent can optimize itself.

We save this verification “holdout” data to be used only for evaluation against the LLM’s results at the end of the process, so as not to bias it.

After comparing the generated data against the holdout data, we can see the error rate. There is some ambiguity here, as Greek word-level translations can have multiple correct meanings, even given the same passage or context.

For example, the opening word of the Iliad, μηνιν, could mean “wrath” or “rage”, and if the holdout data only has “wrath”, it could count “rage” as incorrect. Of course, over time the holdout data can be refined to account for multiple correct answers, but the primary downside is negligible as it just means we’re forcing the AI to conform to a specific style.

After verification, based on how effective it was, we can optimize the skill, either by hand or using tools such as Autoresearch.

Human verification is mandatory, but using Autoresearch to improve the skill can be very helpful and save a lot of time skill debugging. Autoresearch can be extremely effective at optimizing to hit a benchmark, but if the holdout data isn’t sufficiently accurate, or diversified enough to be representative of our generic goal of being able to work with any Greek text, we could be optimizing for the wrong solution. Of course, multiple skills could be optimized. This is where human judgement should intervene and manually inspect both the skill to ensure the wording accurately captures our goals, and, more importantly, the holdout data to make sure we’re optimizing the right target.

The current recommended path is probably something like:

1. Human prompts AI to create generation skill to achieve a goal (i.e. “Generate interlinears for a given Greek text”).
2. Human reads and refines the generated skill.
3. Human creates holdout data or runs the skill once to see what sort of data it generates.
4. Human inspects holdout data and refines it to their exact requirements.
5. Begin AI Generate-Verify-Optimize loop with Autoresearch, which will: a. Generate the data by invoking the skill in a subagent. b. Verify the skill’s efficacy. c. Optimize the skill by tweaking in response to failures.
6. Stop after N predetermined rounds or until the correctness threshold is met (e.g. 95% translation match against holdout data).
7. Human re-reads generated skill to ensure no unexpected overfitting.
8. Re-run Autoresearch if desired threshold isn’t met.

Personally, I don’t see a problem with having the AI generate a skill initially: it’s often trivial for AI to lookup SKILL.md best practices and implement them better and faster than a human can. But since the temptation to “let the AI handle it” is strong due to the experience of waiting around for it to complete its tasks, the human-in-the-loop should take extra caution to ensure it’s behaving as expected, truly matching against the holdout as it claims to be, and passing against large and diverse sets of holdout data that accurately capture the breadth of the desired output.

The Text Orchestrator Skill Pipeline

The text-orchestrator pipeline is a “set of skills” skill. Its goal is to take a simple prompt like “Add Plutarch’s Lives” and end with an output/texts directory containing a set of .json files and a query-able database texts.db which contains all the data listed in our desired list above. This texts.db is the same one that lyceum.quest uses.

As subsequent texts are added via text-orchestrator, the skill is instructed to update, not overwrite, texts.db.

text-orchestrator spawns subagents to orchestrate a 10 stage process from a kickoff prompt:

★ = Full LLM judgement required at this step, expensive

Since the stages are sequential, the subagents are spawned for essential context savings. Scripts are also used to make the process deterministic, allowing the LLM to save precious context on making hard-to-script judgement calls, like whether to create a custom translation for versification or use an existing translation that is sufficiently versify-able. The use of scripts also allows us to add tests.

text-orchestrator is a versatile skill which can be used with the following commands:

• /text-orchestrator run [work] [--mode add|apply] [--profile <profile>] — Run the full pipeline for a text
• /text-orchestrator replay [work] --profile <profile> — Replay specific stages on an existing workspace
• /text-orchestrator resume [work] — Resume an in-progress pipeline from where it stopped
• /text-orchestrator step [work] — Execute only the next pending stage, then stop
• /text-orchestrator status [work] — Show pipeline progress and next stage
• /text-orchestrator abort [work] — Mark the pipeline as blocked and stop

The most important of these is perhaps the replay command, which can replay specific stages of the pipeline at will. A dedicated text-replay skill is also provided which can regenerate data from the already-extracted sources. These skills are important since as we iterate on improving the skills and pipeline, we can run replay to improve on the previous results without having to start from scratch.

I’ve tested the current iteration of the full text-orchestrator pipeline on two small text samples so far:

• The first 5 hexameters of the Iliad
• Aesop’s Fable The Crab and the Fox

The interlinear generation run provides cost tracking when run in API mode (faster and better provenance, but much more expensive than a subscription). In the future this will be added to every stage of the pipeline for complete pipeline cost tracking and provenance. Here are the results of the run for the Iliad:

{
  "run_id": "2026-03-24T15:12:41Z-homer-iliad-proem-1-1-5-1",
  "text": "homer-iliad-proem-1-1-5",
  "book": 1,
  "range": "book 1",
  "started_at": "2026-03-24T15:12:41Z",
  "completed_at": "2026-03-24T15:14:15Z",
  "elapsed_seconds": 94.0,
  "seconds_per_word": 3.13,
  "total_latency_ms": 94006,
  "creator_model": "claude-sonnet-4-20250514",
  "skeptic_model": "claude-sonnet-4-20250514",
  "referee_model": "claude-opus-4-20250514",
  "total_words": 30,
  "total_batches": 5,
  "total_input_tokens": 10187,
  "total_output_tokens": 4721,
  "total_cost_usd": 0.248784,
  "cost_per_word_usd": 0.008293,
  "total_challenges": 7,
  "challenge_rate": 0.2333,
  "failed_call_count": 0,
  "model_breakdown": {
    "claude-sonnet-4-20250514": {
      "calls": 10,
      "input_tokens": 6978,
      "output_tokens": 2906,
      "cost_usd": 0.064524
    },
    "claude-opus-4-20250514": {
      "calls": 5,
      "input_tokens": 3209,
      "output_tokens": 1815,
      "cost_usd": 0.18426
    }
  },
  "benchmark_score": null
}

The interlinear generation is an expensive process as it uses adversarial subagents (Creator, Skeptic, Referee) to debate the best English translation for each word. This is the most LLM-intensive and longest-running part of the pipeline.

Here are what the results look like after importing texts.db into the reader:

Here we have lyceum.quest’s reader enabled with “Transliteration” and “Interlinear” toggled on. This sectioning created by the segmentation skill enables us to learn the Iliad section-by-section by adding each section to a spaced-repetition, Anki-style deck built-in to the reader, enabling progressive, direct learning by using the most scientifically-proven memorization algorithm known.

Future Directions

There are several directions we can go from here:

• One of the most obvious things to do is benchmark various models for accuracy. What we need here are Greek experts to create holdout data for each skill, run Autoresearch against it, and measure results, and tweak. My proposal would be to run autoresearch against individual skills, then run autoresearch against the text-orchestration pipeline as a whole (though defining binary acceptance criteria for text-orchestration is a bit “fuzzier” than the other skills).

• As we refine the processes, we can begin to aggregate or generate new types of data. For example, a recent project used AI to successfully transcribe Latin manuscripts never-before uploaded with a 90% accuracy rate. As this improves we will be able to make previously unavailable data available.

• There’s no reason if we get the proper funding that we couldn’t have agents running overnight. Imagine, for example, a Hermes agent running text-orchestrator for a predetermined set of texts continuously, to be imported into the reader and evaluated at the leisure of experts. Upon reaching an acceptable confidence interval, we can leave it running for a larger subset of texts and human-verify in bulk.

• As Dr. Crane mentions in the article “Towards corpus-based learning: exercises with Ancient Greek”, we can eventually use AI on the other end of the spectrum: to create custom experiences per-student (or expert!). For example, a user may be able to highlight a piece of text and ask questions about the grammar, or meaning, etc. AI could tailor lessons to each student such that they can immediately engage with the text while focusing on their specific weak-points.

• Hand-crafted books can be much more information-dense and educationally useful than exist at present. Currently, the best-in-class is widely regarded as the Loeb Library. But these books only put a Greek translation alongside an English translation. For a beginner, this makes it very difficult to dive into the Greek because it’s very hard to know from reading the English where words, phrases, and sentences correspond in the Greek. Thus these books are limited in their utility to inexperienced readers. We currently have the ability to replicate Loeb naively without any AI assistance and indeed the reader at lyceum.quest does this already for multiple translations, allowing the user to even switch between translations side by side as they see fit, which the Loeb Library can’t do. In web form, this isn’t as much of a problem as a reader can choose their level of granularity. But in book form, choosing which information to include per-word becomes a judgement problem. Still, I personally would love to have Greek-English editions with color-coded word-inflections, grammar hints, and interlinear translations. Perhaps readers can even select which information they want their copy of the book to include when ordering it from the website themselves, like a build your perfect book experience tailored to their preferences as they would in the online reader. In short, we can come a long way in making the Greek originals “come alive” by curating a tradeoff between information density and usefulness in a novel print format.

• Lastly, it should be noted that these skills were generated on a NixOS machine, and thus they’re tied to a lot of nix-specific instructions. We will soon need to generalize these skills to other Linux systems.

Bottlenecks

Some bottlenecks to the project are:

1. Expertly hand-crafted ground truth and holdout data. These holdouts should exist across genres (epic, philosophy, history, etc.) and across dialects (Ionic, Attic, Koine, etc.).
2. API token funding. Generating these tokens, especially in multi-stage adversarial loops as exist in text-orchestrator, is token-intensive. The tradeoff we’re aiming to make here is high upfront cost for long-term payoff. Rather than suboptimal on-the-fly generation, it is better to find methods which generate a great translation once, with some degree of human-review acceptable.
3. Skill development and model optimization. Methods like spawning subagents for each stage, or even adversarial subagents within each stage (i.e. Stage 8), can help improve results. Karpathy’s Autoresearch for automating skill improvement against holdout data will be extremely useful here.
4. The models themselves. I’ve only tested this on Claude models (Sonnet/Opus). Benchmarking will likely reveal models optimally suited to different tasks. It should also be noted that the staggering rate of model improvement may render any processes developed to optimize translations irrelevant, and therefore time invested in optimizing the methods should be weighed against acceptable output quality and the knowledge that any methods we develop are likely to have ephemeral use and diminishing returns.

Conclusion

The rise of LLMs has opened a new door to build better educational tools and more comprehensive databases than have ever existed. The opportunities are particularly salient for niche fields ripe for revival. When building resources for ancient Greek in the age of the internet, we were historically constrained by a shrinking cultural interest in the classics and therefore a smaller pool of experts dedicating resources to creating these tools and databases. Now, however, LLMs have reached a point where custom translations, morphology, and other useful data can be generated with a high degree of accuracy, and it is likely that these tools will continue to improve.

With the careful application of frontier models, quality harnesses, and rigorous human expert review, it is now possible to create a unified, comprehensive database of ancient Greek texts and generate useful data from them optimized for learning and researching.

Hallucination is still a major concern, and more work needs to be done to see if we can reliably produce accurate translations at the word, sentence, and paragraph level.

But the results for the tools as they exist speak for themselves, and skew optimistic.

At this point, I’m bottlenecked by time, money, and Greek expertise. I’ve just been using standard Claude and Codex subscriptions to create and test the orchestrations, but with the right resources and team we can do so much more and at an unprecedented scale.

If you like ancient languages, AI, or are simply excited about preserving and expanding our knowledge of history through digital libraries, please consider Joining the Discord, helping audit translations, donating, or spreading the word!

What I’ve covered here only scratches the surface of what’s now possible to improve our digital access to Greek and other ancient languages.

Links

• Lyceum Github
• Lyceum Orchestrator Skill Pipeline
• Lyceum Orchestrator Outputs
• Discord Community
• Donate

I learned ancient Greek as I would have learned a living language.

Heinrich Schliemann, the man who rediscovered the lost city of Troy, quoted from The Story of Civilization Vol. II: The Life of Greece

Months ago, after having visited Rome and delving deeper and deeper into ancient history, I decided I couldn’t tolerate translations for the Ancient authors. I felt compelled to read them in their own words. And this meant that I had to learn to read Greek and Latin. I started on that journey with Ancient Greek. I am still a beginner, but I felt strongly that a tool which met my learning needs was missing. So I used Claude to build a website to bridge the gap: https://lyceum.quest

I began with a top recommendation online: Athenaze. It has given me a great foundation, particularly for grammar, but I quickly became frustrated by a few things:

1. I was learning Greek through a made-up story about an Athenian farmer. It was engaging, but it wasn’t “real” Ancient Greek and it wasn’t a story they wrote, which quickly became demotivating when things got difficult. This isn’t to fault the book, which is great, but I wanted something that put me in direct contact with the texts to make me feel like I was “unlocking” some ancient secret by learning the language.

2. So much of language learning is memorization, and we have a scientifically proven solution to the long-term memorization problem: spaced-repetition, which I wasn’t using, and which is very difficult to implement without the help of software. The strategy I was using of just haphazardly writing down words over and over was inefficient and exhausting.

3. Flipping through glossaries/dictionaries is difficult, and every language-learning book will force you to do this by the nature of that format. Each lesson introduced new words, but if I forgot a previous word somewhere I’d have to find which chapter it was in or flip to the dictionary. Since I didn’t know most words at the start of each chapter because my memorization technique was bad, I found myself spending a lot of time flipping pages. This was further complicated by the fact that each word is inflected, which meant the glossary definitions just had to pick one specific form of that word in the definition. This is confusing to a new student as the dictionary definition often just looks like a completely different word. Eventually, I ended up just typing the words I forgot into an LLM to have it tell me what it was, which was far faster. Double-checking with the glossary I found that the LLM was almost always correct.

4. The answer key to the book wasn’t built in, which seems common practice for Latin/Greek books, likely because they are geared for universities. But for a self-learner, I had the constant sense that I was veering off course with no way to self-correct except to validate my answers via an LLM and online dictionaries. This was also incredibly frustrating.

There are some books that tried to address concern 1, such as Complete Ancient Greek by Gavin Betts and Alan Henry, but they still have to simplify the original for learning purposes and usually do not seem to address 2-4.

I began to search more seriously for ways to just make direct contact with the authors I wanted to read, with some sort of translation to help me engage with the original. After a bit of searching, it seems that the most serious player in this game is the Loeb Library, which produces beautiful little books which have the Greek (or Latin) original on one page alongside a quality translation on the other. As far as I can tell, this is one of the only publishers which even publishes works with original Greek, translation or no translation. I enjoyed these and tried reading side by side, but this too was difficult and fraught with error. The word orders, and sometimes even sentence orders, of the translations are different. While this was a step in the right direction as an aid to learning and made the process much more rewarding, it still felt incomplete to me.

Eventually I discovered the interlinear and/or Hamiltonian method of teaching these languages and stumbled across his translation of the Iliad, which felt like a breath of fresh air. An interlinear translation is one where the translation of the word is placed directly below its appearance in the original. One problem with this is that word order in Greek can be completely different than in English, making it harder to comprehend. So Hamilton rearranges the words of the original in order to make them have a more natural flow in English.

Stumbling on this, it felt like the obvious solution. Here the text came alive to me in my language. I was engaging directly and immediately with my purpose for learning it. Suddenly learning transformed from something tedious to something fun — learning the language through the ancient texts is giving me the sensation of unlocking an ancient secret. Learning through made up stories designed to teach lessons, by contrast, felt like busywork that was obscuring my true goals. It was based on the idea that the vocabulary, not the grammar, was the primary blocker to learning a language. This method was pretty popular throughout the 19th century and was advocated by John Locke and others (as you can read in the intro to Hamilton’s Iliad above), and is still in use today by seminary students and many of the most popular apps for Latin learning (Legentibus, for example, uses interlinears). This was a solution to problems 1, 3, and 4 above all in one.

So after learning the basics of declensions and verbs from Athenaze, I switched to solving problem 2: memorization. I painstakingly added Hamilton’s cards by hand, word for word, into Anki. It was slow-going, but I found after a couple days I could actually read a sentence or two of the real Iliad!. And with each day, more.

Then, while listening to The Story of Civilization, Vol. II: The Life of Greece by Will Durant, I heard the insane story of Heinrich Schliemann. As an eight-year-old boy, his father read Homer to him, and little Schliemann became so enamored with it that he made a promise to his father that when he grew up he would rediscover the lost city. Around age thirty, a successful merchant, figuring he’d acquired enough wealth to fund his dream, he set out to find Troy. To the disbelief of many a scholar at the time, it appears that he actually did find it!. In a footnote, Durant includes Schliemann’s journal where he describes how he learned the language, which was the final nudge I needed to inspire me to build Lyceum:

“In order to acquire quickly the Greek vocabulary,” Schliemann writes, “I procured a modern Greek translation of Paul at Virginie, and read it through, comparing every word with its equivalent in the French original. When I had finished this task I knew at least one half the Greek words the book contained; and after repeating the operation I knew them all, or nearly so, without having lost a single minute by being obliged to use a dictionary….Of the Greek grammar I learned only the declensions and the verbs, and never lost my precious time in studying its rules; for as I saw that boys, after being troubled and tormented for eight years and more in school with the tedious rules of grammar, can nevertheless none of them write a letter in ancient Greek without making hundreds of atrocious blunders, I thought the method pursued by the schoolmasters must be altogether wrong….I learned ancient Greek as I would have learned a living language.”

Everything about this felt right to me: the learning by doing approach, not wasting time (academic methods seem to assume you want to make the study of the language a science, and expect you to learn it formally better than your native tongue. In short, they do not respect your time), the observation on ineffectual tedium, the focus on learning it as a living language. It doesn’t hurt that this is what Troy’s discoverer advocated and that he also happened to know Russian, French, English, Dutch, Spanish, Portuguese, Italian, Swedish, Polish, Latin, Arabic, and his native German by the end of his life.

I now had enough intuition about what I felt was lacking to build the tool I wanted.

Building the Website

I asked myself if I could build a website with a rich interface for reading and engaging with the original texts that is better than what currently exists. So I started building, with the following goals in mind:

1. The reading experience should start by at least achieving parity with current resources (Scaife, Loeb). This meant in practice that I needed English translations (multiple preferred, if available). This alone would let us achieve parity with the other viewers.

2. There are databases which exist to serve different purposes, but they have not been combined into a single intuitive interface. Combine freely-available, scholarly data in one place so that it can be maximally beneficial.

3. The site should tightly integrate spaced-repetition features into texts of the reader’s choice as a proven effective method for memorization.

4. Use AI to do useful upfront work that it would take humans a long time to do (e.g. interlinear text generation). Like in Bitcoin, my primary field, proof-of-work is far easier to validate than to generate. As long as we’re up front about the limitations, there’s no reason we can’t use AI to generate initial interlinear texts which are then refined. I will go into more detail about my thoughts on using LLMs to generate interlinear translations and how I did it here in a subsequent post, but suffice it to say, they can achieve better results than you may expect.

I was aiming for something like a cross between the Scaife Reader (which is difficult to use and more overtly academic) and Legentibus (which has proven very successful, but is more paywall-ed, more focused, and does Latin instead of Greek).

Using Claude to build the site I was able to achieve all of these goals for at least a couple texts each, validating the concept. The one exception is that I need some mechanism and/or scholarly review to validate and correct mistakes in the interlinear texts. Thankfully, these can easily be compared in-site with the scholarly sourced definitions from LSJ, Perseus, and Logeion short definitions at any time for the time being.

Tour of the Site

The homepage gives you an overview of the primary features.

Clicking “Browse All Texts” will take you to the /texts page, where you can search for texts by author or title, similar to Scaife Viewer:

The corpus spans 373 authors and 1,837 works across all sources. Many of these will only contain the original Greek text with no translation for the time being, although I would love to fix that over time. The five texts at the top (Aesop’s Fables, Gospel of John, Odyssey, Iliad, and Meditations) have been given special attention. These texts have interlinear translations and transliterations, and were what was primarily used for building and testing the site. If we click on one, we can see the reader view:

There’s a lot to unpack here. On the “Display Options” card, we have checkboxes which we can use to add morphology information (sourced from Perseus) to each word to help with our understanding of the grammar. For example, there is a “Color by Part of Speech”, “Color Inflection”, and “Show Inflection” option. For each word, there is a popup you can click on to see that word’s definition from various sources (Logeion, Perseus, and LSJ), and grammar. You can also see multiple translations side by side with the original Greek when they are available. Any of these options can be combined to your reading taste. As far as I’m aware, there is currently no Ancient Greek resource which combines dictionary definitions, grammar/morphology, and side by side translations for free.

For paid users, the page gets more interesting and useful. Two new checkboxes unlock in “Display Options”: “Show Interlinear” and “Show Transliteration” both generated by Claude. Transliterations are useful to show the Greek characters in Latin text, and thus help beginners to memorize the pronunciation of the Greek characters:

Interlinear texts, as discussed earlier, give you direct word-by-word translations in context, so that you are no longer left guessing which of the many Perseus/LSJ definitions apply in the context of the book you are reading.

Because these are LLM-generated and not scholarly-reviewed translations, you should compare them against their Logeion/Perseus/LSJ counterparts to be certain, but I have found they are usually accurate for these curated texts. You can compare them easily via the popup:

I am working on ways to better ensure/enhance the accuracy of these AI-generated interlinears, but in the meantime, this built-in cross-validation with the official scholarly sources should suffice.

These interlinear translations, combined with official translations and scholarly-source cross checking, make for a very powerful interface for reading.

There is one other feature that I’m particularly fond of: Anki-style spaced-repetition. Paid users can select a section from any text and add those words to a deck, where they can later practice memorizing them.

Then, clicking on the “Study” tab will take you to:

Now, you will be presented with cards like this:

Front:

Back:

You can even click “View in context” to have it highlight occurrences of that word in the text you added it from!

Finally, we have a dashboard with a few charts to track your learning progress:

Conclusion

Lyceum is a rich modern reader for reading Ancient Greek texts with a novel combination of features to let you access the Greek authors of Antiquity in their native tongue. It is meant as a helpful, intuitive, and engaging resource to other means of learning Greek. It is under active development and new features and fixes will be added regularly.

Ultimately I’m building this because I want it to exist. I am the first customer. I now use the spaced-repetition and interlinear features every day and am learning the words from the Iliad, Odyssey and Aesop’s Fables using it. Every day I can read a little more, and I am genuinely excited by my progress. I will continue to add features I (and others) think are useful, I’m sure there are plenty of mistakes to correct and improvements to be made that I’m missing.

If you want to have a voice in Lyceum’s development or provide feedback to the project please join the Discord or send an email to support@lyceum.quest with any suggestions you have or mistakes you find.

Future goals include:

• Fix any missing or incorrect dictionary definitions, with a user-driven feedback mechanism.
• Add many more interlinear translations and transliterations especially for popular texts.
• A feedback mechanism for evaluating/correcting LLM-generated interlinear translations/transliterations.
• Full English/other language translations for a wider variety of texts (many currently only have the Greek).
• Voice/Audio enabled reads (like Legentibus) for pronunciation.
• Make texts downloadable in nice readable formats. A fun thing might even be to allow PDF generation in whatever display format options you’ve selected for offline reading.
• Some day, the above could evolve into creating books/printouts with rich morphological information per word for certain texts (for example, a book could include interlinear translations, per-word grammar information). No publisher seems to do this as of now.

Gratitude for Previous Work

Ross Scaife for laying the groundwork which led to the Scaife Viewer, which was a huge inspiration. The Vesuvius Challenge, which stirred my imagination with an excitement about rediscovering the past, the Loeb Library for their excellent books, Anki for being the only widely-adopted spaced-repetition software, Athenaze for being the book I started learning Greek with, and Legentibus for setting the gold standard for modern Latin learning and being a great resource to take inspiration from.

Data Sources:

Texts

• Perseus Digital Library Greek texts and English translations from canonical-greekLit repository. 50+ authors, 631 works, from Homer through the Church Fathers.
• Diorisis Ancient Greek Corpus: 820 pre-lemmatized XML texts with word-level morphology, POS tagging, and sentence boundaries. Used to aid in generating contextual meanings.
• First1K Greek Project: Additional Greek texts from the Open Greek and Latin project at the University of Leipzig.
• Chambry Aesopica: Chambry’s critical edition of Aesop’s Fables with Perry numbering, multiple recensions, and scholarly apparatus.
• Townsend Aesop Translation: George Fyler Townsend’s 1867 English translation of Aesop’s Fables, used for parallel reading.

Morphology & Glosses

• Perseus Ancient Greek Dependency Treebank Syntactic annotations (dependency trees) and glosses for select texts including Aesop, Homer, and Attic prose. Used for word-level alignment with translations.

• Diogenes: Morphological analyses for 400K+ Greek word forms. Each entry maps an inflected form to its lemma, part of speech, and full grammatical analysis (case, number, gender, tense, voice, mood, person).

Definitions

• LSJ: 116,000+ dictionary entries from the definitive Ancient Greek lexicon (9th edition, 1940). Full scholarly definitions with citations, etymology, and usage notes. The “gold standard” comprehensive Greek definitions.

Claude AI

• Content: AI-generated contextual glosses that analyze word meaning within specific passages. Used to disambiguate words with multiple meanings (e.g., “λόγος” as “word” vs “argument” vs “reason” depending on context).

[The software deployment problem] is about getting computer programs from one machine to another—and having them still work when they get there.

The Purely Functional Software Deployment Model, Eelco Dolstra

Nix allows you to setup software on your computer in such a way that your setup is reproducible, meaning your setup on machine A can be exactly the same as your setup on machine B – as long as you have Nix.

To most people, learning Nix is a pain, due to the new concepts, sparse and outdated documentation, and community infighting.

But I think Nix can make using computers fun and powerful and less painful, once you learn how to handle its’ edges.

One fun way we can use Nix is to stitch together programs like Legos and have them interact with each other in a reproducible way. Let’s create an example!

First install Nix if you haven’t.

Say we have a fun little bash script like this:

#!/usr/bin/env bash

# filename: pokefortune.sh

message="${1:-}"
pokemon="${2:-slowking}"

# Silence Perl locale warnings.
export LC_ALL=C

# Generate a fortune if user did not pass a message.
if [[ -z "$message" ]]; then
  message=$(fortune)
fi

echo "$message" | pokemonsay -p "$pokemon" -n

It optionally takes in a message and a Pokédex number, and prints out that Pokémon and message using the pokemonsay program. If either the message or the Pokemon aren’t specified, it uses a default Pokemon and the program fortune to generate the message.

Save this in a file pokefortune.sh, make it executable, then run it:

# Make the bash script executable
chmod +x pokefortune.sh

./pokefortune.sh

If you’re like most people, you probably don’t have pokemonsay or fortune installed on your system, so you’ll likely see something like this:

./pokefortune.sh: line 6: fortune: command not found
./pokefortune.sh: line 7: pokemonsay: command not found

Therefore this script, which works on my system because I have these programs installed, isn’t reproducible on your system. Let’s create a Nix derivation to make it so. Create a file called pokefortune.nix and copy the following:

# filename: pokefortune.nix

{
  pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/nixos-24.05.tar.gz") { },
}:
pkgs.writeShellScriptBin "pokefortune" ''

message="''${1:-}"
pokemon="''${2:-slowking}"

# Silence Perl locale warnings.
export LC_ALL=C

# Generate a fortune if user did not pass a message.
if [[ -z "$message" ]]; then
  message=''$(${pkgs.fortune}/bin/fortune)
fi

echo $message | ${pkgs.pokemonsay}/bin/pokemonsay -p "$pokemon" -n
''

Now run:

nix-build pokefortune.nix

This may take awhile, especially if this is your first time running Nix. Let’s look at the output:

nix-build src/scripts/nix/pokefortune.nix
unpacking 'https://github.com/NixOS/nixpkgs/archive/nixos-24.05.tar.gz' into the Git cache...
this derivation will be built:
  /nix/store/vg4zmghqcnhjbs8kqhx04xixvm36d3ik-pokefortune.drv
these 9 paths will be fetched (2.65 MiB download, 26.91 MiB unpacked):
  /nix/store/0vdf5mpd762bw53rgl5nkmhvzq8n4m0d-file-5.45
  /nix/store/77cdqhqprqbciyhzsnzmsk7azbk2xv6r-fortune-mod-3.20.0
  /nix/store/d0i8idmbb4jji9ml01xsqgykrbvm7dss-gnu-config-2024-01-01
  /nix/store/0554jm1l1qw1pcfqsliw91hnifn11w8m-gnumake-4.4.1
  /nix/store/2wp235bg03gykpixd9v2nyxp08w8xq8a-patchelf-0.15.0
  /nix/store/c5x32idp600dklz9n25q38lk78j5vwxb-pokemonsay-1.0.0
  /nix/store/pba53n11na87fs4c20mp8yg4j7qx1by2-recode-3.7.14
  /nix/store/zix67r268ihi4c362zw7c0989z12jmy7-stdenv-linux
  /nix/store/2329271b42wh6b6yhl7jmjyi0cs4428b-update-autotools-gnu-config-scripts-hook
copying path '/nix/store/d0i8idmbb4jji9ml01xsqgykrbvm7dss-gnu-config-2024-01-01' from 'https://cache.nixos.org'...
copying path '/nix/store/c5x32idp600dklz9n25q38lk78j5vwxb-pokemonsay-1.0.0' from 'https://cache.nixos.org'...
copying path '/nix/store/0vdf5mpd762bw53rgl5nkmhvzq8n4m0d-file-5.45' from 'https://cache.nixos.org'...
copying path '/nix/store/0554jm1l1qw1pcfqsliw91hnifn11w8m-gnumake-4.4.1' from 'https://cache.nixos.org'...
copying path '/nix/store/2wp235bg03gykpixd9v2nyxp08w8xq8a-patchelf-0.15.0' from 'https://cache.nixos.org'...
copying path '/nix/store/pba53n11na87fs4c20mp8yg4j7qx1by2-recode-3.7.14' from 'https://cache.nixos.org'...
copying path '/nix/store/2329271b42wh6b6yhl7jmjyi0cs4428b-update-autotools-gnu-config-scripts-hook' from 'https://cache.nixos.org'...
copying path '/nix/store/zix67r268ihi4c362zw7c0989z12jmy7-stdenv-linux' from 'https://cache.nixos.org'...
copying path '/nix/store/77cdqhqprqbciyhzsnzmsk7azbk2xv6r-fortune-mod-3.20.0' from 'https://cache.nixos.org'...
building '/nix/store/vg4zmghqcnhjbs8kqhx04xixvm36d3ik-pokefortune.drv'...
/nix/store/vsv2spw517cwq791fl3f8iymm6hshhyq-pokefortune

Nix fetched the packages we specified in our .nix file: pokemonsay, fortune, and some we didn’t specify: such as file, patchelf, and recode, which one or both of the other two packages depends on. Then it copied them locally, built a binary, and placed it at /nix/store/vsv2spw517cwq791fl3f8iymm6hshhyq-pokefortune/bin. We can confirm this by running it:

/nix/store/vsv2spw517cwq791fl3f8iymm6hshhyq-pokefortune/bin

You should see something like this:

To be reproducible, Nix ensures that it knows about every single dependency needed to create a package at all times. Because of that, you can do fun things like this:

# Quickly enter a shell with the dependencies to create and display images.
nix-shell -p graphviz chafa

# Query the derivation's dependency graph, create a .png from it and display it:
    nix-store --query --graph $(nix-build pokefortune.nix) | \
    dot -Tpng -o pokefortune-dependency-graph.png && chafa pokefortune-dependency-graph.png

Resulting in a view of every single dependency (the “closure”) that our pokefortune program requires:

How cool is that! We can see the whole dependency tree for anything packaged with Nix!

You have just done something very powerful with Nix: You’ve created a reproducible derivation that anyone who has the package manager installed on their system can use, which they couldn’t before.

This is just a taste of what Nix can do, but I hope that the potential is clear. As Farid Zakaria says in his blog post “Learn Nix the Fun Way”:

Hopefully, seeing the fun things you can do with Nix might inspire you to push through the hard parts.

There is a golden pot 💰 at the end of this rainbow 🌈 awaiting you.

• https://fzakaria.com/2024/07/05/learn-nix-the-fun-way

This article is directly inspired by Farid Zakaria’s blog post Learn Nix the Fun Way. Check out his excellent blog here.

Special thanks to Russell Weas and veracius for their input to this article and code

References:

• https://edolstra.github.io/pubs/phd-thesis.pdf
• https://web.archive.org/web/20171017151526/http://aptitude.alioth.debian.org/doc/en/pr01s02.html
• https://fzakaria.com/2024/07/05/learn-nix-the-fun-way
• https://zero-to-nix.com/start/nix-run/

Our doubts are traitors and make us lose the good we oft might win by fearing to attempt

• Shakespeare, Measure for Measure

We need you to dream; then pursue those dreams with vigor. You are a human being. You are a machine with hopes and dreams; a calculator who looks at cold, mute, void, dead nature, and somehow sees God and meaning despite yourself. You are a teetering bulb of dread and dream. You cannot help but see the world in infinite terms. Despite your best efforts, you believe in right and wrong, black and white, good and evil. Nihilism is your greatest lie to yourself, and you feel the nausea it induces when it crawls into your heart and dies, because it is unnatural and anti-life. Are you truly so bored with the ease and comfort that the vast array of the past has struggled, and suffered, and died to provide you with, that you have to cut yourself to remember you can feel?

Shall I be carried to the skies,

On flowery beds of ease,

While others fought to win the prize,

And sailed through bloody seas?

Laura Ingalls Wilder, Little House in the Big Woods, p. 96

Use this life. Determine what is good, or at least, what is not evil, then act as a humble servant of the former, and in direct repudiation of the latter. Your heart will thank you for it, even if the world never does. It is a cooling salve for your depression and melancholy. Though at the cost of greater anxiety. But anxiety is excitement also, and adventurousness, and worth the struggle.

Far better it is to dare mighty things, to win glorious triumphs, even though checkered by failure, than to take rank with those poor spirits who neither enjoy much nor suffer much, because they live in the gray twilight that knows not victory nor defeat.

Theodore Roosevelt, Strenuous Life

You are hopelessly finite; but your actions can and do graze infinity. You can make a dent in the Universe. Some of us little people have. And they who have are frightfully similar to you.

How can any honest person look around at their world and not see that it is a museum of passion projects? A handful of souls through the brief span of history shake off their multitude of excuses, doubts, fears, false friends, oppressive family - shed the dying skin - to chase these impossible dreams, and oftener than we should expect, achieve them. Even in the infested rivers of Calcutta, what is all that trash polluting the river? It is the unmaintained dreams of the past piling on and on into a neglectful present. Plastics and juice boxes and old toys are miracles if you stop them from coalescing into growing piles of grime.

Work passionately to some purpose. Most of those who regret working hard worked hard as slaves to some lower passion like money or cheap status. Those who conduct their work in service of Eternity have a cleaner conscience. Did we hear honest words of regret from Socrates? From Cicero or Cato? From Adams, Hamilton, Jefferson, or Washington? Only in their moments of weakness and fear when their tasks ahead bore relentlessly down on them. But, determining to answer the Call of Duty, they steeled themselves and pressed forward, leaving behind a trail of proud memories which sustained genuine pride in the shade of life.

My depression is the most faithful mistress I have known — no wonder, then, that I return the love.

Soren Kierkegaard, Either/Or: A Fragment of Life

Work toward depression and nothingness, and you will achieve it.

Weekends we spend alone, absorbing Netflix, Youtube, ads. Burning away the candle on video games, where we pretend we are heroes, when real knighthood beckons to the brave. Doomscrolling our ten-second videos of AI-generated sequences of ever-more unintelligible noises and images, then we complain that we can’t afford a house. Watch porn, then complain there are no good women. Work harder and smarter on justifying our inaction, and find enemies to our supposed purposes everywhere. We will never find an enemy sufficient to outsource our self-loathing. In this way we will never be cured of it. We are not taking our problems seriously by failing to recognize ourselves as their root cause. And who can we expect to take us seriously if not ourselves?

You are not jealous of the past. You have more than them if you cared to look. You are jealous of your own better nature which you glimpse buried within yourself manifesting forthrightly in others. Their lived virtue is recognizable to you as your own dormant morality.

You are not jealous of the past, except their Hope for the future. Reclaim the Hope, and you’ll quickly see the thousand conveniences and asymmetric advantages afforded to you not given them.

That hope will be reclaimed with the youthful vigor of active optimism; never by a passive pessimism.

“Friends,” said he, “the taxes are indeed very heavy, and if those laid on by the government were the only ones we had to pay, we might more easily discharge them; but we have many others, and much more grievous to some of us. We are taxed twice as much by our idleness, three times as much by our pride, and four times as much by our folly; and from these taxes the commissioners cannot ease or deliver us by allowing an abatement. However, let us hearken to good advice, and something may be done for us; God helps them that help themselves.

Benjamin Franklin, The Way to Wealth

You have the internet friend! And you’re using it to watch cat videos and porn. What’s wrong with you? How many philosophers through the millenia have dreamed of this moment, where anyone and everyone had access to the voluminous fount of human knowledge! So many of them thought this day would unleash the golden age we’ve finally been waiting for. You can, for next to nothing, read the minds of the greatest thinkers, inventors, and scientists who’ve ever lived. But instead, we discovered that we are easily hackable, we will still use the evil software of companies we know are ill-designing against us, or at best do not care. We sell our souls in as near literal a sense as anyone could have possibly imagined, for a little convenience, a little fun, a little dopamine, and say that it is impractical to live without them, and give up. If we use this great tool for knowledge at all we’re most likely to use it for a petty win in a debate not worth having. For the most part, we’ve shortened our attention spans and become addicted further still to our baser instincts. This is not Utopia. This is a Brave New World, and we, for the most part, feel oppressed by a future we wrongly think we have little power to avoid.

The best way to predict the future is to invent it.

Alan Kay

Knowledge and wisdom are now cheap, and as such, unvalued. Even our educated don’t know much today, because knowledge and understanding are no longer the point. The knowledge of our college graduates is the kind of knowledge any true thinker hates. Commoditized and packaged so they can “get a job” and “have a career”, and rarely develop the mark of a truth-seeker:

So many people today - and even professional scientists - seem to me like somebody who has seen thousands of trees but has never seen a forest. A knowledge of the historic and philosophical background gives that kind of independence from prejudices of his generation from which most scientists are suffering. This independence created by philosophical insight is - in my opinion - the mark of distinction between a mere artisan or specialist and a real seeker after truth.

Albert Einstein, Correspondance to Robert Thorton in 1944

You can adopt the whole human heritage. Do not be bound by superficial alliances. Be bound by vision and destiny. The human race is your anchor. Any giant’s shoulders are shoulders to stand on if they are good and true. I admire Socrates and Pericles, though I am not Greek. Cicero and the Antonines, though I am not Roman. Confucius, Nietzsche, Kant, Petrarch, Da Vinci, what do I care for their allegiances? They painted their minds to us, and now they are mine.

Even within these flawed humans, or any flawed human, we can take the good and discard the bad. No human is really one human. We are instead a hash of blurry archetypes, oscillatingly present or absent from person to person, and within the same person at different times, places, and circumstances.

Every man I meet is in some way my superior, and in that, I can learn of him.

Emerson, Think, Vol. 4-5 (1938), p. 32

Is the state of society not to your liking? Good. Fix it!. What else do you have to do? Complain? Settle? You are not allowed to be angry and do nothing. Society will sink under the festering weight of this accumulated hypocrisy. I do not intend to imply this is easy; swimming against the tide never is. Far from it. Only that it is worth it. It is necessary.

Do you see problems? Well I see an army of problem-solvers; an array of latent potential waiting to unleash its energy, and hold back the tide of societal decay. Your mission in life is waiting for you. What excites you? If you have nothing that excites you, then what bothers you? What do you wish was, that isn’t now? There is your destiny. Only a liar can’t think of a worthy pursuit that will provide him a lifetime of work. Worthy pursuits and things worth doing are ever-present. You are far more limited than they are scarce. If you cultivate reasons enough for Being, imbibe enough meaning and purpose in your daily diet, a panoply of destinies scream and beg for you to take them on.

I’ve heard the cry from other people my age that “everything’s been done”, and all this noise about artificial intelligence isn’t helping young people dream about brighter futures. It seems to be feeding their already weak grasp on the meaning of life. They constantly ask themselves what they will have to do, not realizing the opportunity lying latent in every problem. Every where I look I see little issues, and have reached the point that I lament the fact that I don’t have the time or energy to work on all of them myself, as they would distract too much from an even greater or more exciting mission.

Do you feel that the problems are too big? Start small. Rome wasn’t built in a day. Do you feel unmatched by the impenetrable weight of what you’re fighting? Fight it anyway, and become a salient force to be reckoned with along the way, growing strong from each victory, and stronger still from each defeat. A scattered array of untrained militia defeated the might of the British Empire at its height, with muskets they used to hunt birds and deer. Can you honestly decry that your petty worries are anywhere near that insurmountable? You have been living through the Pax Americana, and you and I spent the whole of it complaining, and if we’re not extremely careful we will be reminded what real pain and suffering is.

I think that there is only one way to science - or to philosophy, for that matter: to meet a problem, to see its beauty and fall in love with it; to get married to it and to live with it happily, till death do ye part - unless you should meet another and even more fascinating problem or unless, indeed, you should obtain a solution. But even if you do obtain a solution, you may then discover, to your delight, the existence of a whole family of enchanting, though perhaps difficult, problem children, for whose welfare you may work, with a purpose, to the end of your days.

Karl Popper, Realism and the Aim of Science

So long as there are problems, this world needs your hidden and unique offerings. So long as there are problems, you can be the solution. Do not deprive us of your secret skills. Meet your destiny head-on. Cast your gaze upward toward the Lights of the Firmament, and dream.

When developing a Bitcoin payment flow, there are multiple ways a user can expect to be able to pay. They might want to pay an on-chain address or Lightning invoice, they may be scanning a QR Code from their phone, copy/pasting from a wallet, or using a WebLN browser extension. Creating an intuitive interface that also captures all the possible ways a user can pay is one of the fundamental UX challenges of developing an application that can receive payments in Bitcoin.

This flexibility opens up many exciting use cases, but often comes at the cost of being able to easily develop a smooth experience for the user. The greater the developer’s cognitive load, the more difficult it is to create intuitive UX.

This project aims to provide everything needed to allow standard Bitcoin & Lightning Network payments out of the box. It handles creating the proper URIs from just an address or invoice, and favors creating unified URIs whenever possible. Styles are highly customizable and images can be embedded. It also includes polling functionality – a callback can be passed as a property of the element to periodically check for payment.

Check it out on Github and feel free to play around with the options and create your own here.

In addition, because it’s a web component, you should just be able to drop it into any framework and have it work, whether it’s pure HTML, Sveltekit, React, making it very versatile. Examples are included in the Github repo.

Here’s a quick demo of how to use it in Sveltekit:

Install:

yarn add bitcoin-qr

Then:

<script lang="ts">
  import { defineCustomElements } from 'bitcoin-qr/loader';

  defineCustomElements();
</script>

<bitcoin-qr
 id="qr"
 width="300"
 height="300"
 bitcoin="bc1p4mqqpxdyg3l0tvpv9xaesw5rsqv7fqpcn9ydawx3e8yr4enmc3qq7m59vd"
 lightning="lno1zrxq8pjw7qjlm68mtp7e3yvxee4y5xrgjhhyf2fxhlphpckrvevh50u0q2a8y2qt9jeu3pj7p3u58rxs98yu93dxjlmy3eqp24qze09anfr6cqszm40686ujqq7t9jsjjf42tl2pryv3ds9f4hm6hysk7lvswdm70ntsqvlgc5nm4px6k3q2wmj4xun784sss2jn00q0w4dhydmln92wvash6jckffu0wmetwptj38f5g09k8pznuk4wq2df046yv0c5m3dakwgfz5ths2ymm3z6vnehvxj6urcarufa7uecwqqst9x4063hdpxq4frmupnsam9ahg"
 parameters="amount=0.00001&label=bitcoin-qr.blu.cx&message=bitcoin-qr.blu.cx"
 image="./assets/bitcoin.svg"
 is-polling="false"
 poll-interval="1000"
 type="svg"
 corners-square-color="#000000"
 corners-dot-color="#000000"
 corners-square-type="extra-rounded"
 dots-type="classy-rounded"
 dots-color="#000000"
 debug="true"
 poll-callback={callbackExample}
/>

I built this about a year ago but didn’t post about it because there was more I wanted to do and test before putting it out there. But there’s always something else to do, isn’t there?

If you find any issues, feature requests, or suggestions for improvement, feel free to create an issue in Github.

Hope this saves you some time!

  environment.etc."nextcloud-admin-pass".text = "PWD";
  services.nextcloud = {
    enable = true;
    package = pkgs.nextcloud30;
    hostName = "10.8.0.2"; # An example of an IP assigned to the device by Wireguard VPN. This way it's accessible from other devices.
    config.adminpassFile = "/etc/nextcloud-admin-pass";
    config.dbtype = "sqlite";
  };

There are a couple of deviations here from the original post. First, I needed to upgrade to pkgs.nextcloud30 instead of pkgs.nextcloud28, as the docs recommend (it’s officially deprecated and nixos-rebuild switch will hassle you about it). You also have to change "PWD" to a password that’s allowed by Nextcloud’s default password policy as mentioned in the article above, which, in my case, just amounted to creating a secure password with a password manager and them using that.

Also, be sure to later enable the TCP ports 80 (HTTP) and 443 (HTTPs):

nixpkgs.firewall.allowedTCPPorts = [ 80 443 ];

I was able to access the Nextcloud UI at 10.8.0.2 after this! However, I got stuck in password hell for reasons I cannot fathom (it seemed like it was loading my password correctly, only to tell me after over 10 seconds that the password was wrong). What I think was happening was that Nextcloud wasn’t accepting the original password I used, which was something stupid simple like password123, and it was failing Nextcloud’s password rules. But then when I went to change it, something about the database was already screwed up or my IP address was just banned and I couldn’t login even after having set a good password. In the end, I just ended up nuking all the Nextcloud setup via sudo rm -rf /var/lib/nextcloud (careful with that command!), and then I re-ran sudo nixos-rebuild switch which re-setup the config with a good password from the start this time. And wouldn’t you know it, things worked! I could login with the default username of root and the password I had set in the config above.

By the way, another neat trick I learned is that you can look up what options exist for a nix service by using the nixos-option command. By running nixos-option services.nextcloud.config, I found you could set an admin username of your choice as well via config.adminuser.

ssh-copy-id sets up the ssh keypair exchange for you automatically by just running it against your username@host-ip like so:

ssh-copy-id root@123.456.789

Then, it’ll ask you for your password, and voila! You should now have ssh setup between your computer and the host such that the next time you run ssh root@123.456.789 it will just auto-log you in, no passwords required!

#!/bin/bash

# Script to copy files from nb drafts to site code, then commit and push

# --- Check environment variables ---
if [ -z "$SITE_GIT_REPO_DIR" ]; then
  echo "ERROR: SITE_GIT_REPO_DIR is not set."
  exit 1
fi
if [ -z "$NB_FINAL_DRAFT_DIR" ]; then
  echo "ERROR: NB_FINAL_DRAFT_DIR is not set."
  exit 1
fi

if [ -z "$NB_FINAL_DRAFT_IMAGES_DIR" ]; then
  echo "ERROR: NB_FINAL_DRAFT_IMAGES_DIR is not set."
  exit 1
fi

if [ -z "$SITE_CODE_MD_DIR" ]; then
  echo "ERROR: SITE_CODE_MD_DIR is not set."
  exit 1
fi

if [ -z "$SITE_CODE_MD_IMAGES_DIR" ]; then
  echo "ERROR: SITE_CODE_MD_IMAGES_DIR is not set."
  exit 1
fi

# --- Copy files ---

# Copy markdown files
echo "Copying files from $NB_FINAL_DRAFT_DIR to $SITE_CODE_MD_DIR..."
cp -r "$NB_FINAL_DRAFT_DIR"/* "$SITE_CODE_MD_DIR"/

# Check if the copy command was successful.
if [ $? -ne 0 ]; then
  echo "ERROR: Failed to copy files from $NB_FINAL_DRAFT_DIR to $SITE_CODE_MD_DIR."
  exit 1
fi

# Copy image files
echo "Copying files from $NB_FINAL_DRAFT_IMAGES_DIR to $SITE_CODE_MD_IMAGES_DIR..."
cp -r "$NB_FINAL_DRAFT_IMAGES_DIR"/* "$SITE_CODE_MD_IMAGES_DIR"/

# Check if the copy command was successful.
if [ $? -ne 0 ]; then
  echo "ERROR: Failed to copy files from $NB_FINAL_DRAFT_IMAGES_DIR to $SITE_CODE_MD_IMAGES_DIR."
  exit 1
fi

# --- Git Commit and Push ---

# Navigate to the Git repository directory
cd "$SITE_GIT_REPO_DIR"

# Check if it's a git repository
if [ ! -d ".git" ]; then
  echo "ERROR: $SITE_GIT_REPO_DIR is not a git repository."
  exit 1
fi

# Add the changed files (only the md and images directories)
echo "Adding files to git..."
git add "$SITE_CODE_MD_DIR" "$SITE_CODE_MD_IMAGES_DIR"

# Check if there are any changes to commit
if ! git diff --cached --quiet --exit-code; then
  echo "Committing changes to git..."
  git commit -m "Update content"

  # Push the changes to the remote repository
  echo "Pushing changes to remote origin..."
  git push origin master

  # Check if the push command was successful.
  if [ $? -ne 0 ]; then
    echo "ERROR: Failed to push changes to remote origin."
    exit 1
  fi

  echo "Successfully updated remote repository."
else
  echo "No changes to commit."
fi

echo "Script completed."
exit 0

Automating Deployments to VPS

I also wanted to automate deployments to the VPS I was using on every push. So essentially, my website would update every time the script ran. So I put this deploy.yml in .github/workflows:

name: Deploy to VPS

on:
  push:
    branches:
      - master

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm install

      - name: Build SvelteKit app
        run: npm run build

      - name: Deploy to VPS
        uses: appleboy/scp-action@v0.1.7
        with:
          host: ${{ secrets.VPS_HOST }}
          username: ${{ secrets.VPS_USER }}
          password: ${{ secrets.VPS_PASSWORD }}
          source: "build/*"
          target: "/var/www/blu"
          rm: true
          strip_components: 1

Note that the VPS_HOST, VPS_USER, and VPS_PASSWORD must be defined repository/environment secrets in Github. Also note the action we’re using at the end to do the actual deploy, appleboy/scp-action. Two parameters were important for what I was trying to do. First, rm: true, which deletes the target folder (/var/www/blu in this case) before uploading the data. In the scenario where I delete or move a file in the new build, the old file would’ve remained present without that parameter set, which I didn’t want.

Second, the strip_components: 1 is also important here. Without it, the build directory was getting placed inside /var/www/blu, instead of the contents of the build directory moving into blu. So strip_components specifies that while I want to use build/* as the source of my files, I only want what’s in it, i.e. build/*, so don’t put anything after the first slash (e.g. build) into /var/www/blu.

That’s it! Now every time I update the content of the site (such as with this post), I just copy it over to final, then run the sync script, and voila!

I’ve been using it fairly happily for awhile, but it booted into a black screen recently. I had a flash drive plugged in and I had also installed new programs using dnf (The Fedora package manager, which is the distro Asahi uses), and after some reddit searching it appears that the problem was caused by something having to do with dnf update. After restarting the computer, it showed the Apple logo, Asahi logo, and Fedora logo in sequence as usual, and then never booted past a black screen. Unfortunately I wasn’t really able to “fix” it, but I was at least able to get into a terminal screen, mount a flash drive, backup my data, and then do a hard reset.

On boot, press and hold FN + CTRL + OPTION + F2. This should take you into a terminal-based login screen. Login with your normal user credentials. Then plugin a flash drive. Identify the device name by typing lsblk. It should appear as /dev/sda or something of the like. You should be able to identify it by its’ file size (e.g. 32GB, etc.). Then, mount the device. First, you should create a mount point using mkdir, such as in /mnt:

Create a mount point:

sudo mkdir `/mnt/mydrive`

Mount the device:

sudo mount /dev/sda /mnt/mydrive

Copy files from your home directory (or wherever you have data stored you want to keep) to the flashdrive:

cp -r ~/* /mnt/mydrive

Unmount and eject the drive when done:

sudo umount /mnt/mydrive
sudo eject /dev/sda

Repartition the Hard Drive

Now, you’ll need to do a complete uninstall/reinstall of Asahi Linux by repartitioning the hard drive on your Mac.

I followed this tutorial video to help me repartition the Mac properly, and it worked like a charm. Then, I was able to go through the same process of reinstalling Asahi again and using my Apple Silicon machine with Linux.

1. I use it on a lot of different computers and often want to add to notes that only exist on computer A while I’m on computer B.
2. Since everything I write on my website is in Markdown, I want to have the ability to have what I place in that notebook automatically update the website (or at least update it after running some command)

Solving Problem 1 will be the focus in this article. In a subsequent post, I plan to detail solving problem 2.

Requirements

I want a system that automatically syncs files across devices that have it installed on and connected to at an interval I define when the machine comes online.

Syncthing

After prompting Perplexity.ai about the issue, two methods which appealed to me are Syncthing and Git. Using Git would’ve been more manual and time-consuming, and since Syncthing is a very popular tool and fully open source, I went with that option.

Installing

Syncthing has a great “Getting Started” page that gave me everything I needed to know to use it. Just download the app on two devices. Select the architecture you’re using from their releases page here. Follow their Getting Started page and you should have all the basics to get up and running.

Helpful Notes

Use Cases

Syncthing is not really meant to be a backup software like rsync, because changes made on one machine will synchronize to the other machines, and therefore if one machine deletes a file it will also be deleted on the other machine. It’s meant to be a sharing software keep documents on different computers synchronized, as the name suggests. Which means that it should really be used for things like media libraries or as an alternative to cloud storage solutions like Box or Google Drive, or when needing to edit documents on multiple machines, as in my case with nb.

Mobile Devices

I wanted one of my devices to be my phone, so just note that for mobile (i.e. by mobile I mean Android or a security-hardened derivative like GrapheneOS — sorry iPhone fans, but their anti open source ethos precludes you from using this one) you have to use a fork of their now deprecated Android app. You can download the apk file here.

Folder Types

There are three types of folders:

• Send Only
• Receive Only
• Send and Receive

This is useful for sharing files while still setting some basic permissions for them. Send Only is great for sharing with people/devices that you don’t want to be able to modify your local copy via their modifications; Receive Only is the opposite — it can only be shared with, and yet changes from the sending folder will not modify changes in the receiving folder.

The FAQ is Very Helpful

Initially when using Syncthing I was confused about a few things (Should I use this for backups? Why does syncing take so long? How does it work? Will it ever connect to anything other than my own machines and can they see my data?). The FAQ answered all my questions very nicely and in detail, so be sure to give that a read.

Note For Obsidian Users

Obsidian is an (unfortunately closed-source) note-taking app that pairs nicely with nb. I use nb on all my desk/laptop machines and use Obsidian on mobile since trying to use a terminal on mobile would be a terrible experience. After installing Syncthing-Fork on your mobile device and syncing the relevant folder, you can open that folder as your Vault in Obsidian. Before making any changes in Obsidian, however, you should disable a setting in Syncthing. Click on the synced folder, scroll down to “Watch for changes”, and disable it. This will make it so changes to that synced directory will only attempt to sync changes to the others on an hourly basis. Obsidian saves files to disk extremely frequently, basically on every keystroke, and this caused Syncthing to behave strangely or simply not sync any changes made from Obsidian to the other devices. This is a bit unfortunate because there’s a possibility that you could update the same file on your mobile device and another machine within that hour, causing a conflict. But fortunately Syncthing will save both versions of the file in a “conflict” file and you will have both versions so that there’s no content loss.

We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.

• T.S. Eliot, from “Little Gidding,” Four Quartets. Originally published 1943.

This past year, I’ve experimented with several different methods of creating and managing a personal website. I didn’t merely want the most convenient solution, or the solution most likely to buy popularity more easily via its platform or network effects. I wanted a solution that was in line with:

• 1. A growing set of principles I want to practice see more of in software I use and create.
• 2. A set of very unique and specific features I desired.

The pursuit of principles and features, of course, give way to the brutal constraints of reality to some degree. I don’t have infinite time. And so the desire to obtain the features I want while maintaining those principles must be balanced with the constraints of time and effort. After experimenting with several iterations of website-building, I have in many ways “arrived where I started”, but now feel more confident, more sure-footed, more “aligned”, and more excited about what even a simple little personal website could be.

First Site — A DIY Compromise

I started out as most Gen-Z developers start, I think. You want to be a developer, you need a portfolio. You want to be a web developer? Your portfolio is your website. And so my first website was just that. I built it during many skipped classes in college, while I was still painstakingly and inefficiently learning HTML, CSS, and Javascript. It was designed to be artsy and beautiful and to get me a job as a web developer. And so I didn’t post or write about anything, just listed what I felt would be most impressive to recruiters. But, after proving to yourself that you can indeed get a job and you can indeed make enough to pay your rent and not be a broke college kid forever, you start to realize there’s more important things in life than just trying to impress recruiters. You realize that a personal website can be so much more. It can be a reflection of yourself.

And so, with that in mind, I built my first “real” attempt at a personal site. I built it in Sveltekit (which was in unstable version 3 at the time), used the MDSveX library so I could write my articles in Markdown. This was a proper blogging site that also listed projects and “About Me” and “Contact” and whatnot, however there were several problems:

• Hosting: It was hosted on Netlify, which meant that there were often cold starts when visiting it. This also meant that I wasn’t hosting the site myself, which wasn’t a big deal to me at the time, but developed into one over the course of the year.
• Features: The site was low on features and bells and whistles, but this was mainly because I just didn’t take the time to implement them. The fact that everything was written in Sveltekit meant that I could have, but I wasn’t taking the time to really think about what I wanted implemented and then come up with a good plan to do it.
• Preprocessing the Markdown: I didn’t quite understand how to set up the preprocessor in MDSveX properly, which does the job of converting the Markdown to HTML when building the site. What I think was happening was this resulted in pages having to convert the Markdown to HTML on the fly during runtime, and so the bigger the article, the slower the page load, but I never truly investigating this since I decided to redo the website from the ground up anyway. To the end user, this just felt like the page was frozen for awhile, which was really annoying UX.
• Content and Organization: I wasn’t sure of how I wanted things organized, and I only had a few articles, blogs, and quotes on there. The way I would write my content was by writing the article in Markdown directly on the site and then just committing and pushing the code to Github, and from there it would build and deploy to Netlify automagically. That was nice, but I would prefer to be able to write the article in another program (so that I could have “draft” versions of it and whatnot) and then just commit + push when ready in a single command/click sort of way.

All these problems contributed to a year of experimentation that left my actual ability to write things a bit in limbo.

Substack

I started out by just wanting something that worked, and worked well, for writing, without having to worry about anything else. Substack is the obvious choice for this. It’s famously anti-censorship, has an intuitive UI, a large audience, most of my own subscriptions worth reading are from Substack, easy subscription tiers and connection to Stripe for payments, easy email subscriptions. What’s not to like?

There was one thing that just ended up being a deal breaker for me. Probably a bizarre requirement to most people, but I just couldn’t in good conscience do it. At the time, I was considering requiring subscriptions and payments for some of the articles I wrote, and on Substack, there’s no option for bitcoin payments. The saddest thing of all is, it appears that there was a way to pay with bitcoin while Substack was using OpenNode’s API, but don’t any longer. I suppose that’s a price of vendor lock-in. It is now Substack’s official policy to not support any bitcoin/crypto payments. I work in the Lightning Network because the payments use case of bitcoin is one of if not it’s most exciting aspect. It’s probably one of the strongest ways bitcoin can positively influence how we use the internet. To only allow credit card subscriptions, and not bitcoin lightning micropayments, for paywalled articles if I did any would be sad and hypocritical of me. There are other reasons of course, like the fact that Substack isn’t open source and that I wouldn’t actually own my own data. The idea that my blog dies when the company dies was another major negative.

So what other options were there that fit my increasingly stringent criteria?

Ghost

Enter Ghost, a website which markets itself as essentially the more principled and feature-rich version of Substack (which I believe they are). They are fully open source and you can run the blog software locally yourself, deploy it on a VPS of your choice, setup your own email provider, etc. If you choose to pay for them to host it for you, it’s only $9 a month and that includes the ability to bring your own domain, email subscriptions, etc. A huge improvement to me, and a project and team that I was (and still am) very excited about.

They even have a marketplace of templates and integrations, and given their open-source spirit I was hopeful they’d have some sort of bitcoin integration on there — and there is one…that you have to pay for. It’s called Scribsat; here’s the Ghost Forum post about it. They charge a monthly fee steeper than Ghost’s hosting fees itself to allow you to accept bitcoin lightning payments on Ghost. That didn’t really fit the bill for me, either, and I don’t want to be forced to depend on another service to accept the payments for me. For what it’s worth, this does seem to be a hot topic on the Ghost Forum, and I hope that someone will take up the call of developing something people can use for this. Ghost is a great tool and company, used by Bitcoiners, and would be powerful to many if properly combined with bitcoin payments.

I considered working on this myself, but there is significant overhead that I just didn’t and don’t have the time (for now) to take up.

For awhile, the lack of bitcoin-based payment options was worth it, and I paid for a full year’s subscription even though I wasn’t sure if I’d use it for long (I didn’t, but I consider it a tithe to open-source software). I loved the writing and publishing experience, which was extremely simple, and there were a good number of default themes to choose from. However, none that I loved, and I found myself longing for more customizability. For most bloggers who can pay a few bucks a month or have the wherewithal to self-host, I would definitely recommend Ghost.

Inspiration Draws From Disparate Sources

Shall I tell you the secret of the true scholar? It is this: Every man I meet is my master in some point, and in that I learn of him.

• Ralph Waldo Emerson, Greatness

My website was hosted on Ghost for a couple months, and it actually allowed me to write an article or two and post a quote here and there, and I was starting to give up on monetizing it (was I ever really doing it for that anyway?).

But around this time I was coming into greater contact with online presences that helped me to think differently, and though they all differ radically in occupation, style, and personal beliefs, they share an energy for creating good things meant to last; a passion for thinking long-term and for following their hearts over easy money and fame. A desire to achieve greatness in some form or other, and to have the revenues of their considerable efforts paid in something deeper than cash.

One of those I stumbled across was Luke Smith, a thoroughly original person, whose unique admixture of Twitter-like meme humor, dead-serious power usage of FOSS-only tech, deep historical, philosophical, and linguistic knowledge, and opinionated take on Christianity fascinated me, even if I didn’t agree with many of his opinions and means of presenting that information. It’s rare to find a person occupying all those particular spheres simultaneously, particularly in technology. From him I learned to ask for more from my software, to seek and value the right to and practice of ownership, whether that’s in owning your own land, owning your own money (bitcoin, being debt-free), or owning your own technology to the highest degree possible. He even removed all credit card-based ways to donate to him on his website, as they weren’t in keeping with his principles. I used his tutorial website on becoming an Internet Landlord by self-hosting a website on your own DNS, for example, to build and host the version of this site at the time of writing. Another thing that is important to him is to have technology be a tool you use, rather than you be its’ tool. To that end, he helped encourage me to try to build and use software that will last, and focus less on the ephemeral. Not an easy pursuit in the software world, open source or otherwise!

Another figure who influenced this blog is Henrik Karlsson, whose article A blog post is a very long and complex search query to find fascinating people and make them route interesting stuff to your inbox had a deep impact on me and the direction of my thinking about how a blog should be made. It is excellent, both the idea and the read.

Zig creator Andrew Kelley, also comes to mind, whose article Why We Can’t Have Nice Things helped me realize the incredible waste that often goes into making products worse when money alone is the goal.

Visakan Veerasamy’s article We Were Voyagers is also an amazing and deeply encouraging read, and while it didn’t directly influence how I built this website, it further influenced the “do something useful to others with your life in a spirit of gratitude” approach I’m trying to take to life in general, and will have an effect on the prose and content of the site if nothing else.

But the final major influence was gwern.net, whose beautiful and carefully crafted website design is matched by its rich content and elegant prose. It’s filled with a hundred little features, optimizations, and design decisions that make it a joy to use. For example, it has custom page previews which allow you to see a preview of where a link will take you when hovering over it. It has tables of contents and footnotes for each article, and the footnotes become sidenotes on wide screens for convenience. He takes the time to design artsy dropcaps for each article. And something I found very interesting that I’ve never seen before, is his extensive use of local archives to fight against linkrot, which is an important problem if we wish to preserve something of ourselves for the future. Read his about page for the site to learn more.

All told, Gwern’s website appears to me an exercise in how to create an optimal personal website for the long-run. He summarizes the fundamental question of building a site from the link above:

What does it take to present, for the long-term, complex, highly-referenced, link-intensive, long-form text online as effectively as possible, while conserving the reader’s time & attention?

This version of my site, though still nascent, is based on a fuzzy mesh of these influences, and I will make the attempt to improve it and make it more resilient and higher quality over time.

Knowing the Place for the First Time

Who was I kidding? I’m a web developer. I should’ve known from the beginning it would feel wrong to me to outsource that task, despite the work it may have taken. So after a year of exploration, I’m back at doing the same thing I did before — building the website in Sveltekit and MDsveX. But hopefully this time better, and having learned a lot about how to practically achieve that in addition to learning my own reasons for trying to do so in the first place.

Before, building my own website was a cool thing to put on my resume. But this time I have my reasons.

What Now?

My primary reason for building this site is to document everything I do that I think might be of use to someone else (or my future self). I want something that I built myself and which I have full control over, whose format I have the freedom to modify or restructure at any time (which I couldn’t do in my previous methods). I want something that allows me to publish articles, blogs, daily journals, quotes I find interesting, projects I’m working on, what have you, in as frictionless a manner as possible while still retaining maximum expressive possibilities. Which, in essence, means I have to build things myself, and that there will be a high up-front cost to doing so.

So be it!

This is one of my favorite applications of micropayments I’ve seen so far. Excitingly, they have an API key in addition to the web interface, which meant that I was able to get neovim working like the Cursor code editor using the avante neovim plugin by customizing my config! So now, I finally have a way to accountlessly, privately, and subscription-less-ly pay for a service I very much want to use. Exciting times! I will gladly pay for this service now for my AI code-editing needs, over paying flat rates for subscription-based things like Cursor. I will try to do a writeup on how to do this soon.

Payjoin is a derivative of Coinjoin, which is an older, more interactive protocol, meaning the user must be heavily engaged with it to use it, which necessarily drives down usability and inhibits adoption. Somehow, despite this, Coinjoin has seen much higher adoption than payjoin so far, despite the more obvious benefits and ease of use with payjoin. A complex, unclear path forward for developers inhibits adoption by wallet software.

Payjoin has been around for a few years, and given:

1. The many benefits listed above
2. The ability for a passive, nonintrusive user interface
3. The simplicity for wallet providers to integrate it

Why is adoption so low?

Especially compared to the far more interactive, difficult to use, and expensive Coinjoin protocol?

In this article, we’ll look at current attacks on Bitcoin privacy, the history of payjoin from the perspective of privacy, how it works and how it can provide so many benefits with no changes to Bitcoin, and the current state of adoption. If payjoin can radically improve privacy, scaling, and fee issues with Bitcoin, the low effort required for wallets to integrate it would be well worth it.

Why Privacy Matters for Bitcoin

Before discussing the importance of payjoin, we must understand the importance of privacy. If you don’t need convincing on this front, feel free to skip down to the history and explanation of payjoin.

In Western democracies, the prime importance of privacy is difficult to communicate, as its benefits still seem invisible to people. It is harder to convincingly explain to someone why privacy matters (especially in the face of higher cost or greater inconvenience) if they’ve never felt the often terrible consequences of bad actors having too much information about them, perhaps because it requires people to think long-term about the consequences of these invasions.

Still, privacy seems to be something more and more people want in theory, but they typically do little to achieve it unless the barriers are extremely low and convenient. Therefore, technologies that wish to protect people’s privacy should be designed to do so with minimal requirements for interaction and inconvenience for the user.

Fungibility

Privacy is not the only problem payjoin can help solve, but it is the problem it was created to solve. The lack of privacy by default on Bitcoin has long been lamented, and one taken very seriously in the Bitcoin community. Bitcoin was designed to be directly peer-to-peer and censorship resistant. But the ability to track owners of future payments once coins have been linked to an identity allows for discrimination. This destroys fungibility – or the degree to which one of a currency’s individual units are indistinguishable from another of the same unit – which is one of the primary properties of a good money.

If buyers can be tracked, not only can coins that are currently owned by illicit actors be rejected, but coins that have ever been used for illicit purposes can be marked as such and be rejected by merchants, nevermind whether the current owners obtained them through perfectly legitimate means. Imagine not being able to buy milk at the grocery store because the dollars you have were once-upon-a-time used to buy drugs, would it be fair to you to be discriminated against by having them say “your money is no good here”? Should you be punished for someone else’s crime? And what would that do to how you treat those dollars? It would make them worthless as your spending power is diminished by owning them, and make some dollars (the “non-tainted” dollars) more valuable than other dollars, which of course shouldn’t make sense. One dollar should always equal one dollar, no matter what individual one dollar bill is used to represent it, or the ability of the dollar to do its job of transferring value is impeded.

The Criminal Myth

It is often said by Bitcoin and privacy detractors that privacy is for criminals. This is the familiar “if you aren’t doing anything wrong, you have nothing to hide” logic, which is easily refuted:

• Few people would be okay with an internet live stream of them taking a shower or using the bathroom being broadcast. Is that because it’s wrong? This is to point out that everyone has something to hide, and hiding things isn’t inherently wrong.
• More broadly, the government is charged with providing the legal basis for what’s wrong, but that definition is always subject to change. If people aren’t free to have privacy, then they really aren’t free at all, as their actions will be severely restricted by perceived social, if not governmental, restrictions. People are judged and attacked for doing perfectly legal things by others all the time. Privacy is the power to choose who you reveal yourself to.

Aside from this naive and self-evidently illogical argument, criminals, unlike most law-abiding citizens, are willing to accept high costs to obtain privacy, so measures to hurt privacy-by-default hurt regular people far more than they hurt criminals. This would be the case even if governments weren’t doing a poor job of actually using their new privacy-restricting measures to actually catch criminals en masse, but instead to “pick and choose” and selectively spy on the citizenry at their will. If a citizen says something those in power don’t like, and what those in power don’t like is always in flux, it can selectively decide to target and hurt them.

Finally, the desire for privacy isn’t merely the fear of government overreach. It is a practical safety or reputational concern. If someone can find out how much money you have and where you live, how hard is it to steal from you? Now consider the number of places you entered your address, payment details, pictures, on the internet. Do you trust every one of those sites to keep your information secure? You shouldn’t, because even the best fail, and criminals are paying large sums of money to hackers who exploit weaknesses for this valuable information.

Privacy and Democracy

A prerequisite to control in any totalitarian state is knowledge of the speech, access to information, and financial activity of its citizens. Without it, it can’t know what to attack, shut down, or manipulate to advance its narrative or further its control. If it doesn’t reliably have access to this information, it becomes difficult to target citizens on a whim. In totalitarian societies of the past, such as the Soviet Union and Nazi Germany, privacy was eroded by ensuring the indoctrination of family members who would report on unfavorable opinions in private conversations, destroying webs of trust in families. When this same erosion of privacy occurs with money, its effects can be even more powerful than with speech. Cutting off the source of funding can be a very effective means of disbanding political dissent.

Bitcoin Privacy is Under Attack

Never waste an opportunity afforded by a good crisis

– Attributed to Nicollo Machiavelli

It is in the name of combating criminals, in this case Hamas terrorists, that new regulation is opportunistically being proposed to make privacy-preserving methods in Bitcoin illegal.

On October 10, 2023, an article was published by the Wall Street Journal reporting that Hamas acquired $130 million dollars of funding by means of cryptocurrency. One week later, senator Elizabeth Warren wrote a letter to President Biden urging him to address questions regarding his administration’s response to the “use of cryptocurrency by terrorist organizations” by October 31, citing the WSJ article as evidence of the urgent need for such regulation. The letter obtained the signatures of 29 of the 100 senators, and 76 members of the House. Curiously, on October 19th, just two days after the letter was sent, the Financial Crimes Enforcement Network published a proposal for regulating cryptocurrency mixing due to the risk of money laundering. It lists, among the methods used to obscure transactions:

Using programmatic or algorithmic code to coordinate, manage, or manipulate the structure of a transaction: This method involves the use of software that coordinates two or more persons’ transactions together in order to obfuscate the individual unique transactions by providing multiple potential outputs from a coordinated input, decreasing the probability of determining both intended persons for each unique transaction.

This definition includes Coinjoin and Payjoin, although using algorithmic code is broad enough to include any transaction, and therefore justify at-will censorship.

But the WSJ article, which provided the sentiment for the letter and attempted to justify the regulation, horribly misinterpreted the data – the actual dollar amount that was connected with Hamas was $450,000. Cryptocurrency is nowhere near a major source of Hamas’ funding. Hamas itself explicitly stated that they did not want funding via Bitcoin due to its traceability.

The irony of the proposed regulation, then, is that its effects will be minimally impactful to the terrorist organization that is the justification for its existence, but maximally impactful to everyone else who wants to use Bitcoin and other cryptocurrencies.

There can be little doubt that the battle for the right to privacy in Bitcoin has arrived in the U.S., and it has predictably masked itself in the guise of national security against foreign adversaries. It’s more important now than ever to understand the need for and use of privacy-preserving techniques on Bitcoin to combat the ongoing attempts to chip away at it.

We must defend our own privacy if we expect to have any.

1. How Transactions Work

To understand the need for payjoin and how it works, it’s crucial to understand how transactions work. Each transaction in Bitcoin is a combination of inputs and outputs. Outputs define which public key or “address” the bitcoin is being sent to. Inputs define which “sources” or previous outputs are used in creating the new outputs for a transaction. It is helpful to think of this as an analogy to how we use different denominations of cash to pay for things. Imagine you need to pay $25 to a restaurant for dinner, plus a $5 tip for the waiter, for a total of $30 (these are your outputs, two different “chunks” of money going to two different people – the restaurant and the waiter).

How would you pay it? Let’s say you have the following bills available (these are your inputs):

• One $20
• Two $10’s
• Six $5’s

To construct this transaction, you could use one $20 bill and two $5 bills, one being for the waiter:

Note one important way in which our cash analogy breaks down: the $20 and the $5 merge into one “bill”. This would be more analagous to melting individual pieces of gold into one bigger piece in order to be able to pay the exact amount required, instead of handing over multiple pieces. Bitcoin allows us to split and merge inputs however we’d like to create the appropriate output.

You might also use two $10 bills and two $5 bills like so:

You could even use your six $5 bills:

Before they’re spent on something, the individual bitcoin “bills” you have are called unspent transaction outputs, or UTXOs. This name is confusing at first, but makes perfect sense if you think about it. They are the “results” (outputs) of transactions that haven’t yet been used up by other transactions. A transaction output that is unspent is a transaction output that you can spend. Therefore, in effect, UTXOs become like the bills of cash in your wallet. After they’re spent, they become inputs to another transaction’s outputs (the cash in someone else’s wallet), and are no longer spendable by you, but the record of the fact that you spent that bill to them remains on the blockchain.

Unlike cash, for a Bitcoin transaction to be valid, we need the approval of the sender. This is done with the sender’s digital signature, which serves as proof of their intent to spend the coin. A valid signature (that is, a signature corresponding to the address of the UTXO) needs to be present on each UTXO that the sender is trying to spend as an input to a new transaction. The presence of a signature “unlocks” the UTXO and signals that it was its owner’s intent to spend it in the proposed transaction.

Here is an example of a real transaction that had 1 confirmation while writing this:

(Link)

As we can see, the above transaction consumes one input and creates two outputs, one being the actual payment, and the other almost certainly being change sent back to the spender. A fee equal to the sum of the inputs minus outputs is also paid to the miner.

This “UTXO model” is very powerful. Since every transaction has inputs and outputs, and since the outputs of one transaction become the inputs of another transaction later, we end up with a long chain of transactions, and are able to track bitcoins’ transfer of ownership from transaction to transaction all the way back to a miner. Since Bitcoin has a limited supply, and derives its crucially non-inflationary properties from this fact, it’s important to be able to audit how much is in circulation or “unspent” at any given time, which the UTXO model allows for.

This is the crux of the privacy problem with Bitcoin. Every transaction has a history. All the bitcoin that came to you and anywhere you send it to can be easily tracked. The system was explicitly designed to support this, albeit not with the intention of tracking individuals. In this system, your only true bet is to never link your identity with your public key, which, in our age of mass surveillance, can be very hard.

The Historical Origins of Payjoin

Satoshi’s Modest Mistake

When Satoshi published the whitepaper in 2008, he recognized the privacy problems that came from the requirement of announcing every transaction to the public, and keeping it private.

He had two recommendations to avoid linking an identity with transactions:

1. Keep public keys anonymous
2. Don’t reuse public keys

This is good advice, but for 1) it is very difficult to keep our identities completely isolated from our payments without extreme caution when making payments online. And for 2) even without public key reuse, if transactions spending from multiple keys are spent together in subsequent payments, it is not too difficult for a tracker to put together which keys belong to which person. These suggestions, even when combined, make for a very difficult to enact and imperfect privacy solution.

After these suggestions, Satoshi then makes a modest mistake, exaggerating the weakness of his own system:

As an additional firewall, a new key pair should be used for each transaction to keep them from being linked to a common owner. Some linking is still unavoidable with multi-input transactions, which necessarily reveal that their inputs were owned by the same owner. The risk is that if the owner of a key is revealed, linking could reveal other transactions that belonged to the same owner.

Satoshi’s assumption, and indeed, all the examples shown so far, presume that all the inputs in a transaction belong to the same owner. In other words, it assumes all the “bills” come from your wallet, which is a fair assumption, but one that isn’t necessarily true. This assumption is called the common input ownership heuristic. It is almost always true for any given transaction, and it is the basis of chain surveillance.

Coinjoin

In early 2013, Gregory Maxwell played a fun game in the bitcointalk.org forum. He provided one of his UTXOs (worth 1 BTC) and an address of his, and asked people to create new transactions using that UTXO as an input. If they proposed a transaction where they sent less than 1 BTC back to him, that meant they were trying to take from him, if more, they were giving to him. If they used the transaction to send exactly 1 BTC back to him, however, they were simply using his address in the transaction to increase privacy by making it look as though it was one of the spender’s UTXOs, even though it wasn’t. When one of Maxwell’s inputs was consumed and sent back to one of his addresses, he posted another one so someone could continue the game. From the perspective of a chain analysis company, this would have the effect of making Maxwell seem rich! Since his address was public, and many UTXOs were being used to construct transactions involving him, anyone analyzing the transactions and making the assumption that all inputs in a transaction were owned by the same person would think Maxwell was transacting with far more bitcoin than he actually was, hence the post’s title: “I taint rich!”.

Of course, this game wasn’t really private, since Maxwell posted his address in a public forum, but it proved a very important and consequential concept. As Maxwell states:

A lot of people mistakenly assume that when a transaction spends from multiple addresses all those addresses are owned by the same party. This is commonly the case, but it doesn’t have to be so: people can cooperate to author a transaction in a secure and trustless manner.

In a later post that same year, Maxwell formalizes this idea into a concept he called Coinjoin:

When considering the history of Bitcoin ownership one could look at transactions which spend from multiple distinct scriptpubkeys as co-joining their ownership and make an assumption: How else could the transaction spend from multiple addresses unless a common party controlled those addresses?

[…]

This assumption is incorrect. Usage in a single transaction does not prove common control (though it’s currently pretty suggestive), and this is what makes CoinJoin possible:

The signatures, one per input, inside a transaction are completely independent of each other. This means that it’s possible for Bitcoin users to agree on a set of inputs to spend, and a set of outputs to pay to, and then to individually and separately sign a transaction and later merge their signatures. The transaction is not valid and won’t be accepted by the network until all signatures are provided, and no one will sign a transaction which is not to their liking.

This means that, in effect, an arbitrary number of people can coordinate to create transactions each contributing and signing their own inputs, without risk of theft by others at any point.

He then highlights another benefit of Coinjoin transactions, which is that transactions can be batched to save on fees by finding someone else who wants also wants to make a payment at the same time as you:

The idea can also be used more casually. When you want to make a payment, find someone else who also wants to make a payment and make a joint payment together. Doing so doesn’t increase privacy much, but it actually makes your transaction smaller and thus easier on the network (and lower in fees); the extra privacy is a perk.

And finally, Coinjoin is a protocol such that, if enough people use it, everyone wins, resulting in privacy gains for everyone:

Such a transaction is externally indistinguishable from a transaction created through conventional use. Because of this, if these transactions become widespread they improve the privacy even of people who do not use them, because no longer will input co-joining be strong evidence of common control.

To provide a concrete example, say we find three people who want to do a coinjoin. Beforehand, they agree to mix 0.1 bitcoin, gaining privacy by having three equal-sized outputs to make it unclear which address owns which coins. The change addresses will be clear to an analyst, but the 0.1 values will not.

The privacy gains aren’t necessarily very strong with only three participants, especially if the other participants de-anonymize in later transactions by tying them to their identity, but this can be mitigated with multiple rounds of coinjoin on those coins or by using larger anonymity sets.

To summarize, Coinjoin is a transaction created with inputs and outputs from multiple parties, such that it is difficult to tell who owns which coins after the transaction.

For a deeper dive into how to actually create a Coinjoin and what tools are available, see this guide.

Coinjoin is one of the most effective and widely adopted privacy solutions for Bitcoin today, but it has some major drawbacks:

1. Interactivity: Coinjoin requires a high degree of interaction from the participants; they need to agree to a uniform output size and must all supply their signatures within a reasonable time. High interactivity requirements create friction for users, and therefore hinder adoption by wider audiences.
2. Centralized coordinators: Wasabi and Whirlpool are currently the most popular methods of Coinjoining. They take fees for conducting the coordination, in addition to the miner fees paid just to participate in the transaction (since Coinjoin transactions have a lot of signature data, they are more expensive). JoinMarket is an example of a non-coordinated service, but this has the tradeoff of increased interactivity.
3. Multiple rounds required for enhanced privacy: To get better privacy, it is often recommended to Coinjoin multiple rounds due to the limitations of small anonymity set sizes, which costs time, increases interactivity, and costs more in fees.
4. Coinjoins look different from normal transactions: Coinjoin transactions have a distinctive, recognizable pattern: multiple inputs from multiple parties resulting in multiple uniformely sized outputs. This means that if your coins have been identified prior to you doing the Coinjoin, the snooper will know you’ve Coinjoined. They may not know where the money went or what you do with it after you Coinjoin, but they know how much you had and that you did a Coinjoin in the first place.

Clearly, this is not an end-all be-all solution for Bitcoin privacy due to these limitations, especially for more passive Bitcoiners who want a privacy-by-default approach.

After a few years, a better solution was on the horizon, one that could be done without any extra steps having to be taken by the transactees, was directly peer-to-peer with no centralized coordinators or marketplaces (therefore saving time and money), and looked no different from normal transactions: Payjoin.

Payjoin has been developed from a series of earlier innovations, let’s take a look at those.

BIP-21

An important user experience (UX) improvement in early Bitcoin was BIP-21. BIP stands for Bitcoin Improvement Proposal, and defines a set of standards that either require consensus changes to the Bitcoin protocol (e.g. hard forks or soft forks) or provide information and methods otherwise useful to interacting with Bitcoin.

BIP-21 is a standard that defines a scheme using URIs to simplify user interaction with Bitcoin, allowing them to pay by clicking links or scanning QR codes. A few query parameters, such as amount, label, and message are also defined so that client software can easily access and parse them for better UX. Here is an example BIP-21 URI with those parameters:

Importantly, the standard is extensible, as custom query parameters can be created and new standards can be built on top of it. For example, in the Lightning Network, the custom parameter lightning can be added in addition to the Bitcoin address so that the user can pay with either method:

This powerful and flexible BIP would prove useful in combination with concepts from Coinjoin.

Pay-to-Endpoint (P2EP)

The earliest mention of the Payjoin concept I could find was by Blockstream, published in August 2018, where the article references a workshop from which the concept emerged. It referred to the resulting idea as Pay-to-Endpoint (P2EP), because it combined the concepts of Coinjoin and BIP-21 to allow a sender and a receiver to collaboratively contribute inputs to a transaction via a BIP-21 compliant endpoint provided by the receiver. Here is an example given in the article of what an endpoint provided by the receiver may look like:

Of note in particular is the p2ep parameter, whose value is an endpoint (in this case a .onion address, but it could just as easily be an http:// address or any other compatible endpoint), which signals to receiving wallets that the sender would like to try a P2EP payment. If that fails, wallets should fall back to the sender paying the address normally, just using the sender’s inputs.

Because the contribution of inputs are collaborative in P2EP, and don’t result in the “coin-tainting” uniform outputs that Coinjoin creates, Payjoin transactions are much more difficult to identify.

The idea was a major step in the right direction, but the idea was still nascent and unformalized, and had some additional complexity to be removed.

Aside - Satoshi’s Pay-to-IP (P2IP)

A variation of this idea, called Pay-to-IP, was actually implemented by Satoshi in the very first version of Bitcoin. There were, however, major privacy concerns with this approach, and so it was abandoned in subsequent versions of Bitcoin.

Bustapay

Later that same month, Ryan Havar emailed a revised version of the idea to the Bitcoin developer mailing list, and formalized it into a BIP which he called Bustapay. This version simplified the initial P2EP protocol and removed some of the complexity in favor of simplicity, the idea being that simplicity would prove essential for adoption.

The Bustapay proposal still had a few major issues that needed refinement, and the protocol was not as refined as it could have been. But it was a further step in the right direction, and its focus on simplicity for the purposes of wallet integration was a key one, especially for the slow-moving and cautious Bitcoin developer ecosystem. Although Bustapay never took off, it was a final precursor to the Payjoin proposal we have today, which is ripe for wallet integration, and positive transformation of on-chain transactions.

The Payjoin Proposal

Finally, by mid-2019, the concepts for Bustapay and P2EP were further refined and added to by Nicolas Dorier, founder of BTCPayServer, and Kukks, with the publication of BIP-78, titled “A Simple Payjoin Proposal”.

With our background of the protocols leading up to payjoin, the meaning and purpose of the opening abstract of the proposal should now be clear:

This document proposes a protocol for two parties to negotiate a coinjoin transaction during a payment between them.

This proposal laid out, in much more rigorous detail than prior methods, how to conduct a coinjoin transaction between a sender and a receiver in a way that breaks the common-input ownership heuristic that is simple, flexible, and cheap.

How Payjoin Works

Let’s say Alice wants to pay Bob 1.1 BTC, and a chain surveillance company sees a transaction like this:

They might assume that Alice paid Bob the 0.5 BTC output and gave the rest to herself as change, like this:

And the vast majority of the time, they’d be right! After all, normally the change is the larger amount, and 0.5 is more “round” and thus likely to be used for payment, than 1.1.

They might also wonder why she used an unnecessary input (the 0.8 and 0.3 BTC inputs would’ve sufficed), but they can never know for sure that this wasn’t a normal transaction, and they can’t know for sure why an extra UTXO was used. She could have just been consolidating a UTXO for easier management later. It could be a payjoin, but even if you assumed it was, which UTXO’s are Alice’s and which are Bob’s? It’s impossible to know. Since most transactions aren’t payjoins, they’ll probably make the faulty assumption that it wasn’t.

Alice, however, is smart and wants to preserve her privacy, and she knows about payjoin, so she asks Bob to contribute one of his own inputs to the transaction. Bob agrees, so he creates a transaction that spends one (or more) of his UTXOs as an input, and proposes it to Alice. If the transaction looks good to Alice, she broadcasts the payjoin transaction to the network. In this case, the transaction actually looks like this:

If chain surveillance made the assumption that both inputs were owned by Alice (as they did in the first example, and currently do in practice), their assumptions about what both Alice and Bob own would be wrong!

It’s also interesting to note that Alice and Bob conducting this transaction helps everybody gain privacy. Because, unlike coinjoin, it looks like a normal transaction, if enough people use payjoin chain surveillance will be unable to trust whether any transaction is normal. By foiling chain surveillance’s attempts at spying on their transaction, Alice and Bob make every transaction a little bit suspect. If there are enough people doing it, it makes them all suspect. Online privacy is often a numbers game, and the more people participate, the better the privacy.

In this example, Alice and Bob collaborated to create a transaction that used both of their inputs to preserve privacy. Now, of course, this whole process can be automated (and is in practice).

In BIP-78, the process is more formally defined as follows:

1. A receiver shows the sender a BIP-21 URI with a query parameter pj= that refers to an endpoint/server people can send partially signed bitcoin transactions (PSBTs) to. The endpoint can be HTTPs, .onion, or any other authenticated encryption protocol. For example:

2. The sender creates a finalized, broadcastable PSBT using only their own inputs that is fully capable of making the required payment, to the receiver’s endpoint. This is called the “Original PSBT”.

3. The receiver modifies the PSBT to include his own inputs, signs his inputs, and gives this back to the sender. He does not modify any of the inputs or outputs of the sender. This is called the “Payjoin Proposal”.

4. The sender verifies the proposal, re-signs her inputs to finalize the transaction, and broadcasts it to the network.

If something goes wrong at any point in the process, such as, for example, the receiver doesn’t have any UTXOs with which to create the Payjoin Proposal, then the receiver simply broadcasts the Original PSBT and it goes though as a normal transaction. Although this uses all inputs from the same owner, again, if enough people are using payjoin, it becomes impossible to make the assumption that the two parties didn’t payjoin, and surveillance will simply have to assume they did and try to find other methods of tracking payments.

Payjoin’s Many Benefits

More Broken Surveillance Heuristics

The common-input ownership heuristic is not the only privacy-destroying assumption that payjoin breaks. BIP-78 identifies two other heuristics that can be used to identify owners:

• Change identification from the scriptPubkey:

  In Bitcoin, scriptPubKey is the “locking script” that specifies the conditions under which Bitcoin can be spent. It’s called scriptPubKey because the locking condition requires a valid signature matching the public key (the address) to unlock it. In other words, only the person who controls the private key to a particular UTXO’s associated public key can unlock it.

  There are multiple types of scriptPubKey’s in use, (P2PKH, P2WPKH, P2SH, P2TR). Typically, wallets use the same scriptPubkey for all transactions, and therefore, the change amount (the amount the sender sends back to themselves after consuming UTXOs as inputs) will likely have the same scriptPubkey, while outputs sent to the receiver are much more likely to have a different scriptPubkey. This means that UTXOs with the same scriptPubkey in a transaction can be identified as likely belonging to the spender, assuming the outputs sent to the receiver are different.

  BIP-78 specifies a method to allow the receiver to only use scriptPubkey‘s matching the spender’s, breaking the above heuristic’s ability to tell which outputs are the change outputs.

• Change identification and payment from round payment amounts:

  Normally, payments made to peers have round amounts since it’s far more natural to transact with round numbers. If Bob is charging Alice (and they aren’t basing the bitcoin price on a direct conversion to a “rounder” fiat price), he’s more likely to charge her an amount like 0.00010000 as opposed to a non-rounded amount like 0.00010231, for simplicity. For transactions in which only one output is round, it’s very likely the case (for now) that the non-round output is the change output.

  Payjoin also allows for this metric to be broken by describing a way for the receiver to add extra round outputs when constructing the proposal.

Asymmetric Gains by Uniting with a Bigger Crowd

As mentioned earlier, one of the main drawbacks with coinjoin from a privacy perspective is that: 1) coinjoins are easily distinguishable from other transactions and 2) few people do them relative to non-coinjoin transactions. This creates problems for the fungibility of Bitcoin, because it’s possible for people to mark coins as tainted, since some may have the ludicrous perception that the desire for privacy implies malicious intent. Of course, as most transactions, or even a threshold of transactions, preserve privacy, then they cease to stand out.

Payjoin appears like any other transaction, and therefore doesn’t draw attention. An external observer has no reason to even look at a payjoin more closely, because the payjoin shows no effort to obfuscate the payment and change amounts.

By appearing like any other transaction, even marginal gains in payjoin adoption mean that everyone’s privacy is harder to invade, because surveillance heuristics quickly become unreliable. Adam Gibson, a foundational contributor to JoinMarket and an expert in Bitcoin privacy, sums it up well:

If you’re even reasonably careful, these PayJoin transactions will be basically indistinguishable from ordinary payments. […] Now, here’s the cool thing: suppose a small-ish uptake of this was publically observed. Let’s say 5% of payments used this method. The point is that nobody will know which 5% of payments are PayJoin. That is a great achievement […], because it means that all payments, including ones that don’t use PayJoin, gain a privacy advantage!

UTXO Consolidation

Clearly, payjoin and it’s predecessors were aimed at solving privacy problems. But there is a nice ancillary benefit to using payjoin, specified right in BIP-78 itself: UTXO consolidation.

Satoshi’s suggestion to use a new address for every received payment results in user’s wallets having many UTXOs to manage. When these UTXOs are all used together as inputs to new transactions (and assuming it’s not a coinjoin or payjoin), these transactions cost more in fees. Fees are calculated based on the size of the transaction in bytes, since block space is the scarce resource, so more inputs equals more fees for a single large transaction.

It should be noted that using payjoin for UTXO consolidation does not necessarily save on fees, because the expense of each UTXO for chain space still needs to be paid at some point. Rather, it spreads the fees out over time and gives an opportunity to batch the UTXO with the payment. Batching makes UTXO consolidation cheaper than doing it as its own transaction. It also makes for easier management of UTXOs, and less data stored on disk. In addition, it is possible for wallets to implement a way for a receiver to specify that they would like to consolidate their UTXOs when mining fees are low in advance, making UTXO consolidation an automatic and smoother process.

Lightning and Payjoin: A Perfect Match

Opening Lightning Channels with Payjoin

The Lightning Network (LN) is a second-layer solution built on Bitcoin that takes transactions off-chain to allow for near-instant, final settlements with far lower fees, tremendously increasing transaction throughput, improving privacy, and allowing for new use cases for Bitcoin such as micropayments. It uses a network of payment channels between nodes to route payments from source to destination. These channels require node operators to lock up “liquidity” (bitcoin that can flow between one node and its channel partner) between their channel partners. How much bitcoin you can spend in a channel is limited by how much liquidity exists on your side of that channel.

Most of the complexity in managing a lightning node comes from opening these channels and managing the liquidity in them. Onboarding is one of the biggest pain points, since there are several steps involved. Let’s say Alice wants to open up a channel with Bob and she has setup a brand-new, unfunded lightning node. Alice has to do the following:

1. Send an onchain transaction funding her newly created lightning wallet with sufficient funds to at least open the channel and wait (at least 10 minutes) for it to be confirmed.

2. Use her lightning wallet software to negotiate a channel opening transaction with Bob and wait for it to be confirmed.

At a minimum, Alice has to pay two fees and wait ~10 minutes per transaction, which can be tedious.

Payjoin can simplify this process and help Alice save money by doing both the funding and the opening transaction at once.

In this scenario, Alice preconfigures her payjoin receiving endpoint with the details of how she’ll open channels: including the amount of bitcoin and which channel partners she’ll open to. Then, using a wallet with payjoin support, someone (presumably Alice) will send a payjoin proposal, negotiate the payjoin transaction with the receiving endpoint, and upon receiving the funds, the endpoint will make the necessary API calls to open a channel with Bob’s node.

In other words, the sender (probably Alice in this case) will send a payjoin proposal to Alice’s receiving payjoin endpoint to create a transaction that pays directly to the 2-of-2 multisignature output between Bob’s node and Alice’s node to construct a lightning channel between them. Which turns the process into a one-step transaction:

One interesting thing to note is that both LN and payjoin currently have a liveness requirement (though, in the case of payjoin at least, perhaps not for long), meaning that participants must be online when transactions take place. This is fairly limiting in comparison to on-chain Bitcoin, which only requires the sender to be online at the time of payment. However, this also allows for the two protocols to pair together quite nicely.

For example, Lightning is great for increasing privacy by taking payments off-chain, and for massively increasing Bitcoin’s ability to be used as a medium-of-exchange (i.e. something you would actually use to buy everyday things) in addition to a store of value. But the requirement to open channels on-chain means that the size of your channels and the people you open channels to leaves an onchain footprint. For reasons already discussed, payjoin can obfuscate and destroy many assumptions made by snoops.

It also makes things simpler because users now only have to make one payment instead of two to open their first channel, faster because they only have to wait for one transaction to be confirmed, and cheaper because they only have to pay one fee. In fact, more than one channel can be opened by these means. What if you could make a list of all the nodes you wanted to open channels to, make a single transaction to a BIP-21 payjoin receiving endpoint, and have them all open at once, automatically, waiting for one transaction to confirm and paying one fee. For all your channels. Well, you can!

There is already a project which implemented this idea, called Nolooking, which allows you to queue up a list of public keys and batch open multiple lightning channels at once! This way, Alice can not only open a channel up to Bob, but she can open up channels to Bob, Carol, and Dina, all the while making just one on-chain payment! Needless to say, the potential for this to simplify Lightning UX is huge. It’s exciting to imagine a future in which Lightning wallets just have payjoin on by default, and the de facto user experience is to just pick your channel partners, make a single Bitcoin transaction, and voila! You now have a lightning node with as many channels as you can afford, and paid one transaction for it. How amazing would that be?

It’s easy to imagine how this could simplify self-custodial lightning adoption. It would be interesting if lightning wallet software could just have a “quick setup” option, where all the user has to do is input how much Bitcoin they can lock up (i.e. how much liquidity they want), set to a default of opening a few, reasonably sized channels with decent tradeoffs for routing and fees. For advanced users, there could be an “I know what I’m doing” option.

Weaknesses

As with any protocol, there are tradeoffs to payjoin.

One of the main issues is the liveness (or online-ness) requirement. The receiver’s payjoin web server must be online at the time the transaction is sent in the current implementation, due to the need for both sender and receiver to negotiate (programmatically, of course) the resulting transaction. This might limit adoption to merchant servers or lightning nodes, as they are the only people with an incentive to always stay online. It would be much more convenient from a user’s perspective if a transaction could be sent at any time irrespective of whether the receiving server is online.

Another less likely but more dangerous weakness is that if a payjoin server (i.e. the receiver’s server) is on an unsecured server, the receiver’s outputs could be modified in flight(before they reach the sender) and stolen.

However, as we’ll discuss later, a solution to the above two problems has already been proposed.

Finally, one weakness of the protocol is simply the fact that it faces adoption barriers as wallets must do the work to integrate it. A particular challenge is that the ideal user interface would be one in which payjoin is just enabled by default. Sending wallets and receiving wallets just try to payjoin without a user necessarily needing to be burdened with setting up that privacy feature. The best privacy is default privacy, since requiring users to take action to defend it means they probably won’t. Therefore, for payjoin adoption to really take off for the average user, it needs to be a seamless experience that they don’t have to think about. Wallets should enable it by default. Remember, it’s built right into the protocol to fallback on regular transactions without requiring any additional action from the user in case the payjoin fails.

Serverless Payjoin

Dan Gould has submitted a draft BIP for a version 2 of payjoin which allows it to be done asynchronously and without using a server. This serverless payjoin would solve both the requirement for a receiver to be online at the time of payment, and the security issues related to this being done on unsecured servers, mentioned earlier. As the always-online nature of payjoin receivers is perhaps the biggest user experience barrier to adoption, the implementation of this BIP could be a huge benefit for payjoin adoption and passive Bitcoin privacy.

The Current State of Payjoin Adoption

As of late 2023, payjoin adoption is still relatively low, but has been steadily increasing since its inception in 2018.¹ Since payjoin is ready to use today and doesn’t require any Bitcoin consensus changes, the only real impediment is for wallets to write the software to adopt it, and the tools to help them do so are getting better every day. Payjoin Dev Kit (PDK) is a new payjoin implementation with modules that wallet software can use to integrate today. It even comes with a payjoin-cli tool that allows you to create payjoins from the command line. The library is written in Rust, but bindings allowing other languages to use it are currently being built.

Wallet Support

BTCPayServer and JoinMarket already support both sending and receiving, although not by default. BlueWallet, Sparrow, Wasabi, and BitMask support sending. A few other wallets support it via an extension, including Bitcoin Core. There are active PRs to integrate payjoin into Mutiny Wallet today. See here for the current state of payjoin adoption.

Payjoin and Bitcoin’s Future

As mentioned earlier, Adam Gibson has proposed that even just 5% of on-chain transactions being conducted with payjoin could have a huge impact on Bitcoin privacy. We just need a threshold number large enough that analysis companies can never safely assume they’re interpreting the transactions correctly. Once their methods of spying on us are broken, then ill-informed, arbitrary, and bad-faith restrictions on Bitcoin privacy imposed by those who neither understand its benefits nor have any interest in preserving our right to privacy will become irrelevant.

And as we’ve seen, due to the many possibilities opened up by payjoin’s flexibility, it is not merely a privacy solution, but an extensible cooperative transaction protocol that allows for solutions such as fee reduction and single transaction multi-channel opening for lightning nodes, among others. The benefits it would provide to Bitcoin could be enormous and achieved today without changing anything about Bitcoin itself.

So what are we waiting for?

Support

If you’d like to support or contribute to payjoin’s development, start by joining the discord, donating, or checking out payjoin.org.

Acknowledgements

I want to give a huge thank you to Dan Gould for reviewing and suggesting very helpful improvements to this article.

Here’s Gibbon on the importance of reading and writing to civilization, and what distinguished the Romans from the “barbarians”, i.e. the Germans:

The Germans, in the age of Tacitus, were unacquainted with the use of letters; and the use of letters is the principal circumstance that distinguishes a civilized people from a herd of savages incapable of knowledge or reflection. Without that artificial help, the human memory soon dissipates or corrupts the ideas intrusted to her charge; and the nobler faculties of the mind, no longer supplied with models or with materials, gradually forget their powers; the judgement becomes feeble and lethargic, the imagination languid or irregular. Fully to apprehend this important truth, let us attempt, in an improved society, to calculate the immense distance between the man of learning and the illiterate peasant. The former, by reading and reflection, multiplies his own experience, and lives in distant ages and remote countries; whilst the latter, rooted to a single spot, and confined to a few years of existence, surpasses, but very little, his fellow-labourer the ox in the exercise of his mental faculties. The same, and even a greater, difference will be found between nations than between individuals; and we may safely pronounce that, without some species of writing, no people has ever preserved the faithful annals of their history, ever made any considerable progress in the abstract sciences, or ever possessed, in any tolerable degree of perfection, the useful and agreeable arts of life.

– Decline and Fall of the Roman Empire, Ch. IX, p. 242

He’s an extremely ancient figure, so much so that no one really knows when he lived, typically placing him somewhere between 1500 to 1000 BC. Zoroastrianism was the religion of the ancient Persians, and dominated much of the Middle East for a millennium, until being overtaken by Islam.

Aside from many of its very interesting ideas about free will, heaven and hell, creation and anti-creation, etc., one idea I found especially refreshing to find in minds that long predate Jesus, Socrates, Confucius, and even the first philosopher – was the rejection of asceticism; the abstinence of sensual pleasures which often also leads to the withdrawal of societal participation and sense of social responsibility. Zoroastrianism apparently differs directly with the founders of many of our modern religions in this regard. To quote Gibbon:

But there are some remarkable instances, in which Zoroaster lays aside the prophet, assumes the legislator, and discovers a liberal concern for private and public happiness, seldom to be found among the grovelling or visionary schemes of superstition. Fasting and celibacy, the common means of purchasing the Divine favour, he condemns with abhorrence, as a criminal rejection of the best gifts of Providence. The saint, in the Magian religion, is obliged to beget children, to plant useful trees, to destroy noxious animals, to convey water to the dry lands of Persia, and to work out his salvation by pursuing all the labours of agriculture.

One may wonder how much pointless human suffering was caused by the elevation of self-inflicted pain to an act smiled upon by God. This isn’t to diminish the merits of well-directed sacrifice – a sacrifice to some social purpose – just sacrifice for sacrifice’s sake.

Continuing with Gibbon:

We may quote from the Zendavesta a wise and benevolent maxim, which compensates for many an absurdity. ‘He who sows the ground with care and diligence, acquires a greater stock of religious merit, than he could gain by the repetition of ten thousand prayers.’

While the rejection of asceticism doesn’t exist without the opposing danger of hedonism, and while I’m sure a closer look at Zoroastrianism would reveal many strange features, it is interesting to see a religion that explicitly celebrates the application of our efforts toward positive social utility. If you’re going to be judged by God, it’s nice to be judged by whether you do good in the world, instead of being judged simply by the degree to which you abstain from participating in its evils.

Check out the Voltage presentation on this topic here.

If you’re interested in learning more about the history of micropayments, check out Amber Case’s article and the Open Index Protocol’s episode

Before Bitcoin

Before the World Wide Web, the internet, or even the personal computer revolution, the concept of making tiny payments in exchange for digital content had already been conceptualized. The term “micropayment” can be traced back to Ted Nelson as early as 1960, when he began brainstorming his ideas for what a globally accessible, computer-based repository of human knowledge could look like, and coined many other terms and ideas that are used on the web today, like “hypertext” and “hypermedia”. While often failing to implement his own vision for the web (and being vocally regretful of how the web took shape), others took many of his ideas and built them into what we know it as today.

In 1992, Tim Berners-Lee, creator of HTTP and HTML, released his second version of HTTP and the first reference to the status codes now in common use. Among them was a code that Berners-Lee and others thought would one day be used to pay for digital content: 402 payment required. Sadly, this status code is officially “reserved for future use”, as the various attempts to make micropayments on the web, from its inception, failed to come to fruition. Over 30 years after the invention of the internet, we’re still waiting for one of its major original visions to be fulfilled.

Berners-Lee founded the World Wide Web Consortium (W3C) in 1994 to guide the development of the web, and micropayments were a major consideration from the start. In 1995, Phillip Hallam-Baker, who wrote a number of RFCs on internet security, drafted the Micro Payment Transfer Protocol (MPTP), which never seems to have been implemented. It offers a number of insights on the nature of micropayments that are as relevant today as they were at the founding of the internet:

There is a large interest in payment systems which support charging relatively small amounts for a unit of information. Here the speed and cost of processing payments are critical factors in assessing a schemes usability. Fast user response is essential if the user is to be encouraged to make a large number of purchases.

A critical limitation of MPTP, however, was that a third party, referred to as a broker, was explicitly required by the protocol.

At the time, there was no way to make a digital payment without a trusted mediator, so any attempt at a protocol for micropayments had to be developed with some sort of custodianship of the money in mind.

The W3C continued to push for micropayments for a while, in 1998 publishing an overview of micropayments and suggesting MPTP as a practical approach and noting:

Micropayments have to be suitable for the sale of non-tangible goods over the Internet […] With the rising importance of intangible (e.g. information) goods in global economies and their instantaneous delivery at negligible cost, “conventional” payment methods tend to be more expensive than the actual product.

This echoes Hallam-Baker’s second major concern of transaction costs imposed by technological or overhead costs of available payment mechanisms. His first concern, the need for a “fast user response”, is often overlooked in discussions about the viability of micropayments. But one astute observer, who would later go on to have a strong influence on the development of Bitcoin, was already thinking in depth about this problem. Nick Szabo, creator of the concept of smart contracts and Bitgold, wrote “Micropayments and Mental Transaction Costs” in 1999. Szabo states that the primary reason for the failure of micropayments to take hold was not due to technological or overhead costs. It was due to the mental transaction cost experienced by the user of having to decide whether a purchase was worth the price for every interaction on the internet.

[…] customer mental transaction costs are significant and ubiquitous, so much so that in real world circumstance cognitive costs usually well outweigh technological costs, and indeed technological resources are best applied towards the objective of reducing cognitive costs. […] Micropayment technology efforts which stress technological savings over cognitive savings will become irrelevant.

A web based on micropayments implies frequent purchases, which implies decision fatigue. It is likely the case that for most micropayments, the mental transaction cost incurred by having to make constant choices about purchases likely outweighs the value of what they’re paying for.

Large companies like Compaq and IBM, and startups like Pay2See, Millicent, iPin, and others, tried their hand at reducing these technological and mental transaction costs for micropayments in the early days, and it was still an assumption that the concept would be a lasting feature from the start.

Perhaps the most notable of these companies, which would go on to have a lasting impact on the Bitcoin community, was DigiCash, spearheaded by David Chaum. Chaum had already formalized many ideas for a blockchain-like data structure and secure digital cash in 1982 before going on to found DigiCash in 1989. DigiCash implemented Chaum’s proposals, allowing users to withdraw money (called “eCash”) from a bank and make untraceable digital micropayments with it. Unfortunately, only one bank ever implemented eCash, and in 1998 the company went bankrupt.

Other micropayment efforts were also dissolving around the same time, and the W3C itself closed their contributions to micropayment activity in 1998.

The dot-com crash was in full swing, and micropayments were one of the ideas that crashed the hardest. It was a good time to be a critic. In 2000, writer Clay Shirky wrote “The Case Against Micropayments” in which he boldly declared:

Micropayment systems have not failed because of poor implementation; they have failed because they are a bad idea. Furthermore, since their weakness is systemic, they will continue to fail in the future.

His primary argument for their fundamental flaw was not technological or infrastructural — but instead echoed Nick Szabo one year prior: decision fatigue. He goes on:

In particular, users want predictable and simple pricing. Micropayments, meanwhile, waste the users’ mental effort in order to conserve cheap resources, by creating many tiny, unpredictable transactions. Micropayments thus create in the mind of the user both anxiety and confusion, characteristics that users have not heretofore been known to actively seek out.

Shirky went on to predict three methods of payment that would become dominant on the web which did not suffer from the decision fatigue problem: aggregation (bundling low-value things into a single higher-value transaction), subscription, and subsidy (getting someone other than the user to pay for the content — today this has manifested itself as the ad model).

By the end of the dot-com crash, Shirky’s predictions were looking more salient. Credit cards, with their infrastructure costs prohibiting payments of less than $1, had become the de-facto method of payment, and passion for the micropayments project was losing steam. What was a self-evident and exciting prospect for the web’s future faded against the backdrop of its increasingly centralized, surveilled, and ad-driven successor — Web 2.0.

Bitcoin and the Centralized Web

We have to trust them with our privacy, trust them not to let identity thieves drain our accounts. Their massive overhead costs make micropayments impossible.

— Satoshi Nakamoto

The idea driving 402 was that it’s obvious that support for payments should be a first-class concept on the web and it’s obvious that there should be a lot of direct commerce taking place on the web […] In fact, what emerged is a single dominant business model which is advertising. That leads to a lot of centralisation, because you get the highest cost per clicks and with the largest platforms.

— John Collison, President of Stripe

Satoshi Nakamoto released the Bitcoin whitepaper near the end of 2008, aptly timed in the midst of the U.S. Housing Crisis. He released the original code for it shortly after. Bitcoin was a massive breakthrough, both in the history of computer science and the history of money, and spurred a new wave of interest in the possibilities of the internet. For the first time, there was a permissionless way to transfer value with an internet native currency, without all the inelegant, bloated infrastructure needed to use credit cards.

For awhile, the price of bitcoin was so small that some people did advocate its use for micropayment systems, despite Satoshi admitting that it wasn’t (yet) a great solution to that problem:

Bitcoin isn’t currently practical for very small micropayments. Not for things like pay per search or per page view without an aggregating mechanism, not things needing to pay less than 0.01.

But the limits imposed by fees didn’t stop people from dreaming about new possibilities enabled by it. Marc Andreessen, creator of the first popular web browser, gave the examples of content monetization and fighting spam:

One reason media businesses such as newspapers struggle to charge for content is because they need to charge either all (pay the entire subscription fee for all the content) or nothing (which then results in all those terrible banner ads everywhere on the web). All of a sudden, with Bitcoin, there is an economically viable way to charge arbitrarily small amounts of money per article, or per section, or per hour, or per video play, or per archive access, or per news alert.

This statement is not true today of course (at least, as far as Layer 1 is concerned), but the fees in 2014 were low enough that it was actually possible to build around the concept of micropayments. One interesting project built around this time was Bitmonet, which allowed users to choose their subscription level by paying 10 cents for just one article, 15 cents for an hour of unlimited access to the website, or 20 cents for a day pass. Unfortunately transaction fees are no longer low enough to be able to allow for arbitrarily small micropayments, and although the issue was clearly on Satoshi’s mind from the earliest days of Bitcoin, it wasn’t designed to specifically address the micropayment problem.

Subscriptions and Ads

Shirky’s predictions for content monetization were pretty accurate, particularly when it came to subscription and ad models.

In the ad model, content was subsidized by an advertiser, typically through a third-party. From 2014 to 2022, Google and Facebook essentially held a duopoly over the online ad market as the third-party mediator between advertisers and content creators. The two companies (and in reality, most of big tech) collected enormous amounts of personal information and simply asked their users to entrust them with the safety of their data, despite numerous breaches. This information is used to show targeted ads for products people are more likely to buy. Companies often refer to this model as “free — with ads”. But in reality, users do pay a price. The ad model forces users to trade two things in exchange for the content:

1. Their data, forcing them to give it to third parties, which, as Nick Szabo famously stated, are security holes.
2. Their attention. There is a reason for the phrase “pay attention”, and it’s well illustrated here. More time users spend on a site with ads, the more money advertisers, ad platforms, and the content creator makes. Thus a creator is economically incentivized to show as many ads as they possibly can without annoying the user so much that they leave the platform. The currency of the “free with ads” web is your attention. You are the product.

What’s clear in the ad model is that the consumer becomes a second-class citizen. Since a layer of abstraction exists between a creator’s revenue and the end user, making a great user experience is not the highest priority. And as more and more consumers use ad-blockers, content creators are forced to get more aggressive with ads, making using the web a more degraded experience for everyone.

Subscriptions also rose in popularity. Customers showed they were much more willing to pay for bulk access to licensed content like movies and music on a regular basis instead of paying to own individual songs. Although a more honest business model, they can also be very problematic when they are the sole option for payment. And in recent years, as these services become increasingly competitive, more and more people find themselves suffering from subscription fatigue. The inability to access just one (or a few) specific news articles, movies, or songs at any given time forces a suboptimal choice of trying to pay in bulk and optimize the most content for a given subscription.

Take streaming services, for example. Today there are so many streaming services, all battling for content licensing, that users end up paying multiple subscriptions to try to capture a greater subset of the movies and TV shows they want. But all they really want is to watch an incredibly small subset of what’s offered by any given service. When they pick a service for a movie or show they want, it often doesn’t stay long, and will be bounced around unpredictably from company to company as licensing expires and gets updated.

News articles are another example. Companies like The New York Times or The Economist entice readers by allowing them to read just a few seconds of an article before blocking the content with a subscription paywall. Even more so with newspapers than with movies, customers are far more likely to want to pay a small amount for a single article of their choice than for a package deal to articles they don’t want.

While subscriptions offer a more straightforward approach than ads, using them in practice often makes for what is an increasingly costly and stressful management game.

When Clay Shirky was writing about the issues of mental transaction costs, he was writing before the mental costs of subscriptions and ads began to weigh down on people as they do today. Bitcoin gave a solution to the problem of an internet-native currency, but slow processing and high fees quickly became a prohibitive problem for a system supporting micropayments. One more major innovation was needed before true implementations of micropayments technology could take off.

The Lightning Network

A decentralized system is proposed whereby transactions are sent over a network of micropayment channels (a.k.a. payment channels or transaction channels) whose transfer of value occurs off-blockchain. — Joseph Poon and Thaddeus Dryja, The Bitcoin Lightning Network

In February 2015, “The Bitcoin Lightning Network” was published, representing perhaps the biggest advancement in Bitcoin since its inception. Through the clever use of Bitcoin properties, a series of payment “channels” could be constructed such that transactions did not have to be announced to the blockchain but could still retain the lack of trust in a third party, using the Bitcoin blockchain itself for dispute resolution. Payments can either be made directly between two channel partners, or they can be routed across a series of partners, so long as there exists a path of connected partners between sender and recipient. Because of this, it was no longer necessary to pay fees to miners for every transaction, and payments could be settled near-instantly. Since the primary reason micropayments weren’t viable on Bitcoin in the first place was due to high fees and slow transaction processing, the Lightning Network (abbreviated LN or called “lightning” for short) opened up a whole new world of possibilities that are just beginning to be explored.

Methods

The smallest unit of bitcoin is a satoshi (often abbreviated to sat). One bitcoin is equal to 100,000,000 sat. At current conversion rates, 1 satoshi is equal to $0.00023, meaning that the smallest payments that can be made (assuming no routing fees) are currently at a granularity of about two ten-thousandths of a penny, a finer granularity than was ever seriously considered by the web’s early founders. Technically, even finer granularity payments of a thousandth of a satoshi (called millisatoshi, abbreviated msat) can be made within the confines of the Lightning Network, but any sub-satoshi amounts will be rounded down if the money is ever moved back to the blockchain. Therefore, all of the methods of payment discussed below can be used to make micropayments, but different mechanisms allow for a variety of novel payment schemes to be set up. I won’t go into the technical details of how the Lightning Network works here. My intention is to provide a high level overview of some ways micropayments can be made with it, and hopefully to spark people’s imaginations to the possibilities.

Invoice

The most basic way to make a payment on the Lightning Network is through the Bolt-11 invoice. This is an encoded string of information containing everything necessary for the payment to be made. The invoice is generated by the recipient and then sent to the payee. Typically, from a user’s perspective, the payment string is encoded as a QR code which the payee can scan.

Benefits:

• Least abstractions from a technological level
• Available by default in LN implementations
• Payments occur directly node to node

Drawbacks:

• A recipient having to first send an invoice can make for awkward UX
• Both nodes must be online for a payment to be made
• Invoices are non-reusable and must be freshly generated every time

Most of the ways to pay on LN today either simply use Bolt-11 invoices or some abstraction of them under the hood.

Keysend

An alternative to invoices are keysend payments, which allow for payments directly between two nodes without an invoice. Senders only need a node’s public key. One crucial downside to keysend is the lack of a proof of payment, making it unsuitable for merchants or other systems that rely on proof of purchase. For content monetization, however, keysend pairs nicely with the concept of value4value: a model which gives content away freely and asks the user to give back the value they felt they received in turn (and requires no proof of payment).

Benefits:

• No invoices required, payment can be sent instantly
• Available by default in LND, Eclair, and Core-Lightning (but must be manually turned on)
• Direct node to node payments

Drawbacks:

• No proof of payment limits scope of use

Possible micropayment use cases:

• Streaming sats (Boosting) while listening/watching/reading content
• Micro tipping
• Boostagrams

Examples:

• Podcast Index/Podcasting 2.0

LNURL

LNURL is a set of standards designed to expand the capabilities of LN by providing a specification for “out-of-band” or third party communication between nodes. LNURL standardizes a whole set of LN-based interactions for developers. For example, the LNURL-Pay spec defines how services can create static QR codes that can be paid multiple times. Under the hood, the service is generating a new invoice every time a payment attempt is made.

Benefits:

• Static pay/withdrawal links
• LN based authentication schemes
• Many more

Drawbacks:

• LNURL operates “out-of-band” meaning that there is a service between LN nodes creating invoices on a node’s behalf. This can mean that the service must be trusted to stay online in addition to the nodes involved in the payment.

Possible micropayment use cases:

• Private, printable, reusable QR code for donations on content
• Reusability allows for payments to be made even when embedded in images and videos
• Withdrawals for content creators who’ve accumulated sats on a given platform
• Donations to protestors: Anyone can hold up a sign a protest movement captured on video, and sympathizers can easily donate. To quote Marc Andreessen from “Why Bitcoin Matters” in 2014:

  Think about the implications for protest movements. Today protesters want to get on TV so people learn about their cause. Tomorrow they’ll want to get on TV because that’s how they’ll raise money, by literally holding up signs that let people anywhere in the world who sympathize with them send them money on the spot. Bitcoin is a financial technology dream come true for even the most hardened anticapitalist political organizer.

Examples:

• Cointrain
• Lightning addresses
• List of Awesome LNURL things

LSAT

Many speculate that [the 402 status code] was intended to be used by some sort of digital cash or micropayment scheme, which didn’t yet exist at the time of the initial HTTP specification drafting.

However, several decades later, we do have a widely used digital cash system: Bitcoin! On top of that, a new network oriented around micropayments has also arisen: the Lightning Network.

Olaoluwa Osuntokun, LSAT: Authentication and Payments for the Lightning-Native Web

Lightning Service Authentication Tokens (LSATs) are a protocol created for authentication and paid APIs that leverage the forgotten HTTP error code 402 discussed earlier in conjunction with the lightning network. LSATs can be thought of like reusable tickets that need to be paid for to gain access to a particular resource. But the powerful thing here is that rules can be applied to modify access to resources on a given site, and the tickets can be reused on subsequent visits to the site.

Benefits:

• Extremely flexible
• Reusable
• Allows for highly granular and fine-grained access to digital resources
• Finally makes use of 402 error code!

Drawbacks:

• Difficult version control
• No current methods for efficient management (Could be solved by browser extension such as Alby)
• Underutilized/underdeveloped despite being around for awhile

Possible micropayment use cases:

• Pay for single video/podcast/song/article and remember it on subsequent visits
• Metered subscriptions of your choice
• Custom expiration dates on content

Examples:

• Aperture
• Boltwall
• lsat-js
• Alby Mixtape
• Nakaphoto

WebLN

WebLN is a standard for abstracting away LN interactions by relying on a client (such as a browser extension) to communicate with a WebLN enabled site. The client must have the ability to interact with the user’s lightning node. For example, a site might use WebLN to request a payment from the user when a button is clicked. When clicked, the user’s client extension will popup, providing whatever payment details the WebLN enabled site sent to the client in a simple to use interface. WebLN greatly simplifies the payment flow for browser based interactions, where user’s no longer have to pull out their phones to make QR code based payments.

Benefits:

• Better UX
• Simple implementation
• No third party required
• Compatible with LNURL

Drawbacks:

• Browser client required
• Few clients in use
• If user doesn’t have a client installed, fallback options (such as LNURL or invoices) must be provided

Possible micropayment use cases:

• Anything an invoice, LNURL, or Keysend can do!

Projects Using Micropayments

Finally, here is a short list of projects using micropayments that I found interesting:

• micropay.ai: Micropayments to generate images using DALLE-2
• WebLN Experiments: A collection of various WebLN demos
• LN VPN: Pay as you go VPN service, pay as little as $0.1 for 1 hour
• LNCal: Allow people to pay you in bitcoin to schedule meetings
• BTCMap: Map of locations accepting bitcoin. Uses Boosting for favored payments
• Lightning Video: Like Youtube, but uses bitcoin micropayments instead of ads
• Sats4Likes: Get micropaid for micro tasks
• PeerTube Lightning Plugin: Pay PeerTube creators with LN
• Stacker News: Reddit-style discussions with micropayment tipping
• Bookmark.org: Pay to archive links
• THNDR Games: Earn bitcoin while playing games

See bolt.fun, the Lightning App Directory, or Lightning Landscape for a larger list of projects being built on Lightning.

Conclusion

Micropayments have arrived on the web, and this article just scratches the surface of what’s possible. While there are significant technological and mental barriers remaining, I hope the above examples show how far we’ve already come – and how much further we can go – for a cleaner, freer, and more expressive internet.