As part of our regular practice at Clear Box, we pause each week with our AI historian to take stock of what we’ve built. This reflective process helps us spot drift and uncover the ideas that live between our best-laid plans.
This one is technically my fault.
I was trying to get ahead on some security work for Strata, our memory-management layer. I’d carved it out into its own lane of work so it could progress independently while I focused on other priorities. On paper, I thought the plan was solid.
But then a few questions came back regarding security and what exactly we meant by privacy. My answer—in all its wisdom—was essentially: “Ultra mega as private as possible.”
Fast-forward a couple of weeks. We’re optimizing the engine after merging the work, and I’m looking at the performance numbers. Something is wrong. Why are we getting a fraction of the tokens we were getting before? What happened?
Turns out, “ultra mega private” had been interpreted as a full ceremony around every single token. Verification, bookkeeping, checks on checks. FOR EVERY. SINGLE. ONE.
Haha, whoops.
Imagine the bouncer checks your ID at the door and lets you into the club. But then, every thirty seconds, that same bouncer comes over to check your ID again while you’re right in the middle of trying to dance and vibe out. Okay… how much safer is this making the club, and when do I actually get to enjoy it?
That was happening inside the engine, in the work we’d added around llama.cpp.
In one test on my NVIDIA RTX 5090, the model took about 13 milliseconds to generate a token. Reading back and hashing its internal state added another 23 milliseconds. We were moving roughly 150 MB off the GPU to verify it again for every single token produced. That one check cost more than the generation itself.
And the overhead didn’t stop there. We were repeatedly scanning cache states and clearing temporary working buffers, then recording proof that we’d cleared them—some of which wasn’t even being consumed during generation. The system was spending real time producing assurances that provided zero value to the person waiting for an answer.
Our goal is to stay within 3–5% of the performance of llama.cpp, the underlying engine, at the version we’re building on. We were paying far more than that for the ultra-mega-secure version of commonllama.
So we stepped back and looked at what security meant through the user’s eyes. What were we protecting? When did that protection matter? What work genuinely belonged in the middle of generating an answer?
Speed, stability, and security all have costs. Nothing exists in a vacuum.
If our protections make the software so unpleasant that nobody wants to use it, we’ve missed something fundamental.
We moved the repeated verification work to the events where it belonged: saving or loading memory, sealing a workspace, and cleaning up after work finished. Protections for private content still matter deeply, but asking a process to constantly prove the state it had just written, token after token, was swallowing the entire experience we were trying to provide.
The part I own is the vague instruction. “As private as possible” sounded like clear direction when I was busy, but it left far too much room for assumptions about what to protect against and how much performance the user should sacrifice. I hadn’t made those tradeoffs explicit enough.
That’s why we pause and look back.
A collection of individually defensible checks can easily add up to a system that nobody would choose to use.
I want the bouncer there—but just let me dance, dude!