Science & Technology (Commonwealth Union) – As artificial intelligence becomes more and more frequently used in every day activities, its accuracy becomes even more significant.
At present the leading reasoning models have something in common with the most outspoken person in a room: they speak with unwavering confidence, whether they’re correct or just guessing. Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have pinpointed this overconfidence to a flaw in how such models are trained—and introduced a solution that improves reliability without sacrificing accuracy.
Their approach, known as RLCR (Reinforcement Learning with Calibration Rewards), teaches language models to report not just answers, but also how confident they are in those answers. In other words, the system evaluates its own uncertainty and assigns a confidence score alongside each response. Tests across multiple benchmarks showed that RLCR cut calibration errors by as much as 90 percent, while maintaining—or even improving—accuracy, both on familiar tasks and entirely new ones. The findings are set to be presented at the International Conference on Learning Representations later this month.
The root of the issue is surprisingly straightforward. Reinforcement learning methods used in modern AI systems reward correct answers and penalize incorrect ones—nothing more nuanced than that. A model that carefully reasons its way to the right conclusion earns the same reward as one that simply guesses correctly. Over time, this encourages models to respond with equal confidence in all situations, regardless of how certain they actually are.
This overconfidence carries real risks. In fields like medicine, law, or finance—where people rely on AI to make decisions—a system that consistently signals high confidence, even when uncertain, can be dangerously misleading. A model that claims to be “95 percent sure” but is correct only half the time is arguably worse than one that admits uncertainty, because it gives users no reason to question or verify its output.
“The standard training approach is simple and powerful, but it gives the model no incentive to express uncertainty or say I don’t know,” explained Mehul Damani, who is an MIT PhD student and co-lead author on the paper. “So the model naturally learns to guess when it is unsure.”
RLCR tackles this issue by introducing one extra component into the reward function: the Brier score, a widely used metric that penalizes differences between a model’s expressed confidence and its true accuracy. As training progresses, the model learns to evaluate not just the problem itself, but also its own level of uncertainty—producing both an answer and a confidence estimate. Answers that are confidently incorrect are penalized, as are correct answers presented with unnecessary doubt.
The theoretical foundation supports this approach: the researchers demonstrated mathematically that such a reward design leads to models that are both accurate and properly calibrated. They validated this on a 7-billion-parameter model using various question-answering and mathematics benchmarks, including six datasets the model had not previously encountered.
The findings revealed a clear trend. Conventional reinforcement learning actually worsened calibration compared to the original model, reducing its ability to judge its own uncertainty. RLCR, however, reversed this effect, significantly improving calibration without sacrificing accuracy. It also outperformed post-training methods where a separate system assigns confidence scores afterward. As co-lead author Isha Puri noted, standard RL doesn’t merely fail to improve calibration—it makes it worse, resulting in models that grow both more capable and more overconfident at the same time.
The team further showed that the confidence estimates formed by RLCR are practically utilizable at inference time. When the models produce multiple candidate answers, choosing the one with the most self-reported confidence, or taking measurement of votes by confidence in a majority-voting scheme, which enhances the accuracy as well as the calibration as compute scales.
