As part of our regular practice at Clear Box, we pause each week with our AI historian to take stock of what we’ve built. This reflective process helps us spot drift and uncover the ideas that live between our best-laid plans.
Ummm… why are we accepting these numbers? Something’s borked.
I remembered an earlier test where we’d used fewer CPU threads because the machine was tied up with other work. I can’t recall if we’d assigned it eight or twelve threads at the time, but when I looked at a later run using all 32, it wasn't any faster—in fact, it might have been slower. That didn’t add up.
We paused to design a simple experiment: sweep the thread counts across our development machines. On our Ryzen 9 9950X desktop, the Qwen3 4B test generated about 22 tokens per second with 16 threads. At 32 threads, that fell to about 15 tokens per second. Even eight threads got close to 22 tokens per second. Something about using every logical thread was costing us performance.
That processor has 16 physical cores sharing work across 32 logical threads through simultaneous multithreading, or SMT. Those extra threads share core resources and caches; they don’t add memory bandwidth. Local AI generation is often limited by how quickly it can read model data from memory. More threads can mean more competition for the same resources. That fits what we were seeing: giving the system every available thread made it slower.
We ran similar tests on a Mac mini M4. In a sweep with Qwen3 4B, we fed the same input at four, six, eight, and ten threads, repeating each setting three times for consistency. Eight threads delivered roughly 98% of the peak speed for that input. Those last two threads bought us almost nothing, meaning eight threads gave us a useful sweet spot that left breathing room for other system tasks.
From there, we explored the M4’s performance and efficiency cores, including “cooking” model context and running background tasks on the efficiency cores. I wanted to understand exactly where those jobs belonged and how the machine could continue doing useful work around them without choking.
That opens up a bigger question for me: Why do we always need the fastest AI? What about the most efficient?
My NVIDIA RTX 5090 graphics card is basically a standalone computer that happens to plug into a bigger computer. But in car terms, it gets about 5 mpg and goes from zero to 900 mph (!) in a second and a half.
My Hyundai Tucson is a hybrid. It’s designed for efficiency. In eco mode, it gets about 35 mpg and takes its time getting up to speed. But it still gets me where I’m going. I’m getting a useful, consistent experience while spending far fewer resources to get there.
That’s a design choice I want us to explore with local AI. How close can the everyday experience be while using less? The thread tests gave us a way to quantify how much CPU work actually buys useful performance. Power consumption is another part of that question. Data centers matter, certainly, but I want to understand what we can do efficiently on the machines people already own.
Looking back, it was an odd set of performance numbers that gave us a reason to pause and investigate. We came away with better settings, but also a broader question about what we should be optimizing for. If someone gets a responsive assistant and can keep using their computer comfortably, how much does that last bit of speed really matter?