As part of our regular practice at Clear Box, we pause each week with our AI historian to take stock of what we’ve built. This reflective process helps us spot drift and uncover the ideas that live between our best-laid plans.
We’d been so focused on proving whether commonllama worked that we hadn’t stopped to ask how fast it actually was. It was that far off my radar.
We were building a diagnostic tool to identify problems at various context depths: 4k, 8k, 16k, and onward toward 131k tokens. At each stage, we ran memory operations to see whether saving, clearing, and restoring context held up as the volume of material grew. It was meant to put the engine through its paces.
Then came the 131k run. It just kept going.
Based on the 4k results and some back-of-the-envelope math, we were already thirty minutes past where I expected it to finish.
Processing naturally slows as context grows, but I wanted to understand exactly how much slower it gets on a specific model and machine. The 4k result alone wasn't enough to tell a user what to expect once they had a substantial conversation or a mountain of documents loaded.
Why weren’t we timing these runs? And where were the standardized tests showing that slowdown across different hardware?
I went looking for those comparisons and realized the collection of results I needed simply didn’t exist.
We already had a tool that repeated the work at different depths. If we measured token throughput alongside the time spent saving and restoring context, we could turn a diagnostic tool into something genuinely useful for people choosing hardware or deciding which models to run.
I remember researching the best CPU and GPU for the money when I was building computers growing up. I’d spend a month poring over PCMark and 3DMark results before making a purchase.
Where is our “AIMark” score?
I wanted that kind of transparency for local AI, but I couldn’t find a repository of numbers to work from.
I really do plan. But sometimes the realization only comes when I step back and see what I’ve already built. Looking back, I can see how a tool we needed for diagnosis was evolving to answer a wider question:
Can someone with no background in AI get something useful from this on the hardware they already own?
Later, I finally connected the PCMark idea to our web server. We could collect and publish the results through our own website, becoming the kind of scorekeepers I’d relied on when comparing PC benchmark results. Other people could run the benchmark and contribute their results, too. More machines would give us more to learn from, and someone else might spot a pattern we’d missed... “help us help you” type energy!
This is part of why we’re building local infrastructure. So much attention goes toward coding performance, while basic questions about everyday use still require digging. Cloud wins by default when local lacks these fundamentals…