A research consortium has released a benchmark designed to test calibrated uncertainty in language models.
Beyond right and wrong
Each task includes incomplete or conflicting evidence. Models are evaluated not only on the final answer but also on whether their stated confidence matches the available information.
The work could help teams distinguish systems that sound reliable from systems that communicate their limits.