Small Models Can Win
We discovered something interesting: small models were computing uncertainty but we were throwing it away.The Metrology Connection
Metrology is the science of measurement. Every field of engineering obsesses over measurement - tolerances, latency, purity. But in AI, models compute probabilities for every token and we just… ignore them. That seemed wrong.The Simple Idea
What if we used the uncertainty that models already compute? When a model generates text, it knows when it’s unsure. It has alternatives. It has confidence scores. We typically throw all this away. Our entropy-guided refinement just uses what’s already there:- Keep the uncertainty measurements
- Find uncertain parts
- Let the model fix those specific parts
Why It Works
Small models with entropy refinement can approach the performance of much larger models on many tasks. Not because we made them smarter, but because we let them identify and fix their mistakes. It’s selective - only refining when needed. Most of the time the model is confident and correct.Why This Matters
Most AI discoveries come from intuition. Someone notices something odd and follows it.- Attention mechanisms: “What if the model could look at everything at once?”
- Dropout: “What if we randomly turned off neurons?”
- Our work: “What if we used the uncertainty that’s already there?”
Moving the Field Forward
AI Training exists to spread these kinds of insights. Not everyone can afford the latest models or massive compute. But everyone can use better techniques. When we share methods like entropy refinement, we’re not just making individual models better. We’re showing that progress doesn’t always require scale. Small models can win through measurement, through clever use of existing information, through actually paying attention to what the model is telling us.Learn More:Research Paper | Blog Post