Semantic entropy measures uncertainty across the meanings expressed by possible responses, rather than treating every difference in wording as a different answer. In language-model research, it can be estimated by generating several responses to a question, grouping responses that express equivalent answers, and examining how probability is distributed across those groups.

The distinction matters because different sentences can communicate the same conclusion. Conversely, nearly identical sentences can disagree about a date, a cause, or a condition that changes the answer.

Semantic entropy is an uncertainty measure, not a truth test. A model can repeat one mistaken answer consistently. It can also produce several defensible answers because a question is ambiguous. Understanding the measure requires keeping meaning, probability, evidence, and quality separate.

From Shannon entropy to answer meanings

Shannon entropy describes uncertainty over a specified probability distribution. Before calculating it, we need to identify the possible outcomes and the probability assigned to each one.

Consider a simplified question with two possible answers. If each answer has a probability of 50%, uncertainty is greater than if one has a probability of 99% and the other 1%. Entropy describes that difference in concentration.

For a discrete random variable X, the formula is:

H(X) = −∑i pi log2(pi)

  • X identifies what can vary, such as the answer to a question.
  • pi is the probability of the outcome indexed by i.
  • −log2(pi) describes the surprisal of that outcome. Less probable outcomes are more surprising when observed.
  • Multiplying by pi weights that surprisal by how often the outcome is expected.
  • ∑ means to add the contributions across all outcomes.

With base-two logarithms, the result is measured in bits. Two equally likely outcomes have an entropy of 1 bit. A distribution assigning all probability to one outcome has an entropy of 0 bits. Outcomes with zero probability contribute zero to the sum.

These numbers describe the distribution. They do not establish whether its probabilities are justified or whether its favored answer is correct.

Changing what counts as an outcome

For language models, we can examine distributions over next tokens, complete response strings, or groups of responses that express equivalent meanings. These are different objects.

Suppose a simplified model produces four possible responses with equal probability:

  • “The policy took effect in 2022.”
  • “Its effective year was 2022.”
  • “The policy took effect in 2023.”
  • “Its effective year was 2023.”

There are four distinct strings, giving a string-level entropy of 2 bits under these assumptions. But there are only two answer meanings: 2022 and 2023. Each meaning receives half the probability, giving a semantic entropy of 1 bit.

If all four responses instead expressed the same year, their wording could still vary while semantic entropy was zero under that grouping.

Four possible response strings each have probability 0.25. Two wordings answer 2022 and combine to a meaning-group probability of 0.50; two answer 2023 and also combine to 0.50. Entropy across strings is 2 bits; across meaning groups it is 1 bit. These probabilities are illustrative, not sampled research results.
The same assumed distribution viewed at two levels: four strings become two answer meanings. Grouping adds their probabilities; it does not tell us which year is correct. Open the diagram at full size.

For a fixed distribution and a deterministic grouping, merging strings into meaning groups cannot increase entropy. The grouping removes distinctions between wordings. That mathematical relationship does not establish that the grouping preserved every distinction the reader needs.

Back to article navigation

How answers become meaning groups

Semantic entropy depends on deciding which responses count as equivalent answers. Shannon’s formula does not make that decision; a separate method must construct the groups.

Equivalence is usually considered in relation to the question. Two responses may answer the requested fact identically while differing in explanation, tone, or background detail.

Different words can express the same answer

For the question “In which year did the policy take effect?”, these answers may belong together:

  • “It became effective in 2023.”
  • “The effective year was 2023.”
  • “2023.”

Counting each wording as a separate answer would exaggerate disagreement about the year.

Similar words can express different answers

Now consider:

  • “The feature works on version 4 when compatibility mode is enabled.”
  • “The feature works on version 4 whether or not compatibility mode is enabled.”

The sentences share most of their vocabulary, but they disagree about a necessary condition. Grouping them together could conceal uncertainty that matters operationally.

Differences involving negation, quantities, dates, scope, and exceptions deserve particular attention. A small textual change can produce a substantial change in meaning.

Similarity and entailment are not interchangeable

Semantic similarity can help identify related responses, but relatedness is weaker than equivalence. Two statements about the same feature may be very similar while contradicting one another.

Some research methods use bidirectional entailment: approximately, whether each answer implies the other when interpreted with the question. If both directions hold, the answers may be treated as expressing the same answer meaning.

This is useful but fallible. An automated entailment model can miss a qualification, misunderstand specialized terminology, or produce inconsistent pairwise judgments. Turning those judgments into clusters introduces further implementation choices.

Multi-claim answers complicate the grouping

Two responses might agree on a date while disagreeing about why the policy changed. Whether they belong in one group depends on what the measurement is intended to capture.

A year-only grouping would preserve agreement on the date and ignore disagreement about the explanation. A full-response grouping might separate them. Neither choice should remain implicit.

Semantic entropy is therefore easiest to interpret when the task has identifiable answer alternatives. Broad essays and creative drafts do not offer such clean boundaries.

Back to article navigation

How semantic entropy is estimated

At the conceptual level, a meaning group receives the combined probability of the response strings assigned to it. Entropy is then calculated across the meaning groups rather than across the individual strings.

Let M be the random variable whose possible outcomes are meaning groups, and let m identify a particular group. The relationship is:

H(M) = −∑m P(m) log2 P(m)

Here, P(m) is the probability assigned to meaning group m. For a fixed prompt, context, and generation procedure, it combines the probabilities of the response strings assigned to that group:

P(m) = ∑s ∈ m P(s)

Here, s denotes a complete response string; the sum runs over strings assigned to group m. The mathematical form of entropy remains Shannon’s. The important change is what the outcomes represent.

In practice, a language model has an enormous space of possible responses. We generally cannot enumerate every response, determine its meaning, and sum its exact probability. Practical methods estimate the quantity using a limited sample and a specified grouping procedure.

A frequency-based illustration

Suppose ten sampled responses produce these groups:

  • Eight answers say 2023.
  • Two answers say 2022.

Using sample frequencies as probability estimates gives 8 ÷ 10 = 0.8 and 2 ÷ 10 = 0.2. Substituting those estimates gives:

Estimated H(M) = −[0.8 log2(0.8) + 0.2 log2(0.2)]

≈ −[(−0.2575) + (−0.4644)] ≈ 0.722 bits

Both logarithms are negative because their inputs lie between zero and one. The leading minus sign makes the total positive. This value measures uncertainty across the two sampled answer meanings; it is not a 72.2% error probability.

This is an empirical estimate from ten observations—not a complete description of the model’s possible answers. A different sample could yield different proportions or reveal another meaning group. If all ten answers agreed, the empirical estimate would be zero, but unseen alternatives could still exist.

Research implementations differ. Some use model likelihoods when estimating probability mass; frequency-based variants use how often sampled meanings occur. The research paper’s Methods section distinguishes its likelihood-based estimator from its frequency-based “discrete semantic entropy” variant. The ten-response illustration follows the frequency-based approach. Results should be interpreted according to the actual estimator, rather than assuming that every system described as using semantic entropy performs the same calculation.

This article uses base-two logarithms and reports bits. The paper’s natural-log formulation uses nats. The choice of base changes the numerical scale, so values should be converted to the same units before comparison; 1 bit equals ln(2) nats, approximately 0.693 nats.

The generation procedure affects the result

An estimate depends on more than the question alone:

  • Prompt wording: Different instructions can change which answers are generated.
  • Available context: Evidence, examples, and misleading material can shift the response distribution.
  • Sampling settings: Temperature and probability truncation change which responses are likely to appear.
  • Sample size: Small samples can miss less common answer meanings.
  • Grouping method: Different equivalence criteria can merge or separate the same responses.
  • Model and version: Results from one model do not automatically transfer to another.

Repeated deterministic generation may return the same answer every time without meaningfully exploring alternatives. Conversely, deliberately sampling very broadly can reveal variation that would rarely appear under the system’s normal operating settings.

A useful report therefore states the sampling conditions. There is no context-free semantic entropy value belonging permanently to a question.

Back to article navigation

What it can reveal about factual reliability

Semantic entropy has been investigated as a signal for detecting unreliable factual answers. If a model produces incompatible answer meanings across repeated samples, that disagreement may indicate that its answer is unstable.

In a 2024 Nature paper, Sebastian Farquhar and colleagues studied semantic entropy for detecting confabulations: a class of erroneous responses associated with arbitrary or inconsistent generation. Their experiments provide evidence that meaning-level uncertainty can help identify such errors under the evaluated conditions.

That is narrower than establishing a universal hallucination detector. A measure can be useful for predicting errors on tested tasks without detecting every kind of falsehood.

Low entropy can accompany a consistent mistake

A model may repeatedly supply the same wrong date because that association is strongly represented in its training or supplied context. The responses can form one stable meaning group.

The measure then reflects agreement within the generated distribution. It does not independently compare the claim with reality.

Ten samples from the same model are not ten independent sources of factual confirmation.

High entropy can reflect a problem with the question

Consider “When was the policy introduced?” Possible responses might refer to its announcement, approval, publication, or effective date.

Variation could reflect ambiguity rather than fabrication. The useful next action may be to clarify which event the person means.

High semantic entropy can also arise when several answers are genuinely defensible, especially in interpretive or open-ended tasks. Disagreement is a signal to investigate, not a diagnosis by itself.

An uncertainty score is not a calibrated error probability

An entropy value does not directly mean “there is a 30% chance this answer is wrong.” Establishing a relationship between a score and error risk requires evaluation against observed outcomes.

Thresholds suitable for short factual questions may perform differently on technical explanations, multilingual questions, or unfamiliar subject matter. A deployment needs evidence about its own tasks and failure costs.

Back to article navigation

Using the signal in retrieval decisions

Semantic uncertainty can inform adaptive retrieval: deciding whether to seek more information, change sources, clarify the request, or stop. It should not determine those actions alone.

Relevant evidence may concentrate generated responses around one answer meaning. But concentration is not automatically improvement.

  • A reliable, applicable source may resolve a genuine disagreement.
  • A misleading passage may make every response repeat the same false claim.
  • A conflicting authoritative record may increase disagreement while revealing an important evidence problem.
  • A passage from the wrong software release may produce a stable answer that does not apply to the user’s situation.

The useful question is not simply whether entropy fell after retrieval. It is whether the retrieved material supports the necessary claims under the relevant conditions.

Investigate what the responses disagree about

Suppose sampled answers disagree about whether a feature requires an optional setting. Rather than issuing another broad search, the workflow can identify that condition as the next evidence target.

It might then inspect version-specific documentation, expand a passage containing an exception, or ask an authorized operator to check the local configuration.

That approach connects the uncertainty signal to an inspectable information need. Version-aware retrieval addresses applicability; semantic entropy alone does not.

Keep evidence sufficiency separate from answer stability

A system should still ask:

  • Which source supports the proposed answer?
  • Does it concern the requested time, version, and scope?
  • Are conditions and exceptions preserved?
  • Do apparent confirmations share one original source?
  • What remains unverified?

Grounding AI-assisted answers requires this connection to evidence. A stable distribution of generated meanings cannot supply it on its own.

Likewise, reaching a search limit is not the same as resolving uncertainty. A workflow should preserve whether it stopped with sufficient evidence, needed clarification, lacked access, or exhausted its budget.

Back to article navigation

Repeated generation in writing workflows

A writing system may generate several drafts from the same seed and context. This resembles repeated sampling, but whole-article evaluation introduces difficulties beyond factual question-answering.

Two drafts can share a central argument while differing substantially in accessibility, examples, organization, accuracy, and usefulness. A grouping that treats them as one meaning may overlook precisely the qualities an editor needs to assess.

Sampling, selection, and revision are different processes

Repeated sampling
Generates additional candidates under the same specified task and generation procedure. It exposes alternatives but does not, by itself, improve any candidate.
Selection
Evaluates candidates and chooses among them using criteria or human judgment. Its usefulness depends on whether the evaluation can recognize the qualities that matter.
Revision
Changes a draft, prompt, context, or evidence in response to an identified weakness. It changes the conditions of the next result rather than merely requesting another sample.

If an editor adds a missing source or asks for a clearer explanation, the subsequent draft comes from changed conditions. A difference in measured entropy before and after that intervention needs to be interpreted accordingly.

Semantic agreement does not rank drafts

An editor may prefer one of two semantically equivalent drafts because it explains a difficult idea more clearly, preserves an important qualification, or asks less of the reader’s working memory.

Conversely, a draft with a different argument is not necessarily more creative or more useful. It might introduce a valuable perspective—or simply misunderstand the assignment.

Low semantic entropy is not editorial coherence, and high semantic entropy is not creativity. Those judgments require other evidence and criteria.

Editorial review remains responsible for assessing factual support, structure, accessibility, and suitability for publication. Generating more candidates does not transfer that responsibility to a numerical measure.

Back to article navigation

Questions to ask before trusting an estimate

A semantic entropy result becomes more useful when its construction and intended role are explicit:

  1. What uncertainty is being measured? Identify whether the outcomes are answer meanings, whole-response meanings, or something narrower.
  2. What counts as equivalent? Explain which differences are preserved and which are ignored.
  3. How were responses generated? Record the model, prompt, context, sampling settings, and sample size.
  4. How were probabilities estimated? Distinguish sample frequencies from likelihood-based methods and other approximations.
  5. What could the grouping miss? Check conditions, negation, quantities, and subordinate claims.
  6. What does the score predict in this application? Evaluate its relationship to actual errors rather than assuming one.
  7. What decision will it influence? Specify whether the response is clarification, retrieval, abstention, review, or another action.

Semantic entropy offers a useful correction to treating every wording change as substantive disagreement. It can reveal that a model’s apparently varied responses converge on one answer—or that fluent, similar-looking responses contain incompatible claims.

Its boundary is equally important: a language model can produce many answers without producing many meanings. It can also produce one meaning repeatedly without that meaning being correct.

The measure helps locate uncertainty among generated meanings. Determining whether those meanings are supported, applicable, and useful still requires evidence, interpretation, and judgment.

Back to article navigation