LagomCMS, part 3: the design

On this page

This is part three of four about LagomCMS, the CMS that runs this site. Parts one and two were the research. This is the design, and why each choice went the way it did.

Six requirements

No system I looked at did all six:

  1. Content is files. Readable, diffable, in Git, still openable in fifteen years.

  2. Files still get queried in milliseconds, through an index that's only a cache.

  3. The model lives in code.

  4. Markdown for prose, structured data for what Markdown can't express.

  5. The CMS doesn't render pages.

  6. It's small. Tens of megabytes for a few thousand documents, fast because it does little.

Kirby has the first and slows down past a certain size. Payload has the third and puts content in Postgres. Sanity has the fourth in a proprietary store. None has the sixth, because they all run on runtimes with a floor of a few hundred megabytes.

The name is Swedish: lagom means the right amount. I use it as a rule. When a feature can be argued either way, it stays out.

Runtime: OpenResty and LuaJIT

OpenResty is nginx with LuaJIT built in. Code loads once at startup, so every request arrives to a world already in memory: no framework boot, no autoloader, no plugin discovery. That removes the bootstrap cost by design instead of caching around it.

LuaJIT's FFI calls C by declaring the function signature, with no binding layer and no build step, and the calls get JIT-compiled:

local ffi = require("ffi")
ffi.cdef[[
  int sqlite3_open_v2(const char *filename, sqlite3 **ppDb, int flags, const char *zVfs);
]]
local sqlite = ffi.load("sqlite3")

Everything I needed is a C library: SQLite for the index, cmark for Markdown, libyaml for front matter. And nginx already does TLS, static files, gzip, keepalive and rate limiting properly, so the application only has to handle the API.

The costs I accepted:

  • SQLite is synchronous, so a query blocks the worker. Reads take tens of microseconds, and unbounded scans have to be impossible, not just unlikely.

  • No native inotify, so the file watcher polls a non-blocking descriptor from a timer.

  • LuaJIT is Lua 5.1 plus extensions.

Before committing I spent a week proving it was viable: the four C libraries in a container, SQLite built with FTS5 and JSON, and inotify events crossing a Docker bind mount on Linux. They do.

Borrowed ideas, and where they collided

Nothing here is original. Files as the truth come from Kirby and Statamic. The body as typed blocks from Sanity and Ghost. The model as code from Payload. Hooks from WordPress. Hot reload and plugins as plain code from Neovim and LÖVE. Four of those clashed:

  • Files vs blocks. A gallery can't round-trip through Markdown. So anything Markdown can say stays Markdown, and the rest goes in a typed fenced block, like a lagom:gallery fence with JSON inside. The file stays readable and the API still serves a block array.

  • Files vs concurrent writes. The admin, vim and git pull can all write the same file. Every write carries the hash of the file it started from and gets rejected if the file has changed since. Making that optional would make lost edits the default.

  • Hooks vs isolation. Running each plugin in its own Lua state would cut it off from the request API and async I/O, which is most of what makes a plugin useful. So plugins run in the main state with a restricted environment.

  • Hot reload and the index vs several workers. Each nginx worker has its own memory. Hot reload only runs with a single worker, in development. Production reloads nginx, which takes under 100 ms. For the index, one worker watches the files and writes, and the rest read a shared SQLite file in WAL mode.

What I locked

  • Storage: Markdown with YAML front matter, one folder per document, media beside the document that owns it.

  • Index: SQLite through the FFI, WAL mode, disposable. A second database holds users, sessions and tokens, because those exist nowhere else. Sessions in the index would log everyone out on every reindex.

  • Writes: hash-guarded, atomic (a temp file, then a rename), and deletes go to a trash folder instead of being unlinked.

  • Auth: an httpOnly SameSite cookie for the admin, so no credential ever reaches JavaScript. Scoped bearer tokens for machines.

  • API: REST only. No GraphQL: there's no mature GraphQL server in Lua, and I haven't met anyone here who needs it.

  • Admin: a separate app served as static files, using the same public API as everyone else. No private endpoints.

  • No framework. Lapis offers a Postgres ORM, migrations and templates, and this uses none of them. The router is a couple of hundred lines.

  • Licence: AGPL, so anyone running a modified copy as a service has to share it.

Rejected outright: themes, GraphQL, a visual page builder, multi-tenancy and a plugin marketplace. Instead of themes there's a reference Astro frontend to fork.

The spec came to about four thousand words, with the reasoning written next to every decision. That's so that six months later I can't undo one for a reason I'd already considered and rejected.

Next: what building it proved wrong.