Story
Don't ask an LLM for a confidence score
1 report
1 website
1 discussion
Latest report
Overview
LLM confidence scores often misrepresent the certainty of generated outputs. Relying on these scores can lead to misunderstandings about the reliability of AI responses.
Summary from justinflick.com.
Reports
One report so far. Additional coverage will appear here when it is linked to this story.
justinflick.com
Don't ask an LLM for a confidence score
Read report →
Community reactions
Each reaction summarizes sampled comments from the linked discussion.
Hacker News · justinflick.com
Hacker News discussion ↗
Historical count: 5 · 5 saved in sample · Historical sample
Analysis ready
Read this discussion’s reaction
Based on 5 sampled comments from this thread. Analyzed 2026-07-28.
- Praise: With the right prompting strategy, LLMs can verbalize probabilities that are better calibrated than their own estimates, which opens up new avenues for better decision-making.
- Praise: For many applications, like finance, high or low confidence scores have proven to be accurate about 90% of the time, making them practically useful despite some edge cases.
- Concern: The claim that asking LLMs for confidence scores is completely useless is overly strong, especially when research shows they can be useful under certain conditions.
- Concern: Models often lack a good sense of their own capabilities, leading to unreliable self-assessments when asked about the meaningfulness of their outputs.