As part of our regular practice at Clear Box, we pause each week with our AI historian to take stock of what we’ve built. This reflective process helps us spot drift and uncover the ideas that live between our best-laid plans.
A 12B model needs another 24 GB of RAM just for the cache? (Wait, what?!) It uses sliding-window attention. (I was pretty sure that should bring the cache requirements down significantly…)
My AI assistant was calmly presenting this as the reason our Gemma setup wouldn’t work at 131k context. Rather than just telling it “you’re wrong”—which usually just gets a reflexive agreement—I asked it to search the web and remind itself how sliding-window attention is actually supposed to work.
We needed to align on why this looked wrong before we could fix it.
Once we investigated, we found that our integration was still using a default that allocated the sliding-window cache across the full context. So we had a model designed to work with a smaller window, but we were reserving memory as though those layers needed the whole conversation. Ack!
We tried changing that in a throwaway prototype. The sliding-window portion went from over 20 GB to under 1 GB.
Okay… now that makes more sense.
Then we put that change through the memory tests. Saving and restoring worked at 4k, but saving failed at 8k once the window had moved far enough. Our save logic had been treating the start of that moving window as the start of the whole context. Changing the allocation exposed that assumption, so the fix also needed to preserve the right state for saving and rewinding. Another day of testing later, we finally had the problem solved.
Looking back, this is a recurring theme in my work with AI. It can present a problem as a foregone conclusion. My job is to push agents to investigate the missing information so we can see the issue together.
Taking the time to align on the why is part of the work.
In this case, simply accepting that “the model needs too much RAM” would have left both an oversized allocation and a flawed save assumption completely unexplored.