The condition is not the signal

A condition of interest is something we want to understand or influence. A signal is information available about it. The two are related, but they are not interchangeable.

A thermostat uses a sensor reading to regulate temperature. That reading represents temperature at a particular location, subject to the sensor’s accuracy and response time. It does not directly describe how comfortable every person in the building feels.

The same distinction appears in software and organizational work:

• Support resolution: closing a ticket records a workflow event; it does not necessarily establish that the person’s problem was resolved.
• Retrieval usefulness: clicking a search result indicates selection; it does not establish that the result answered the question.
• Editorial quality: publishing an article confirms an output; it does not establish that its claims were adequately reviewed.
• Usability: completing a task quickly records elapsed time; it does not establish that the outcome was understood or correct.

These signals are not useless. They answer narrower questions than the broader conditions they may be used to represent.

Direct evidence and proxies

A proxy is an indicator used to stand in for something that is harder to observe directly. Completion time might serve as a proxy for effort. The number of reopened support tickets might serve as a proxy for unresolved problems.

A proxy can be useful when its relationship to the intended condition is understood well enough for the decision being made. Problems arise when that relationship is assumed rather than examined.

The useful question is not simply “Can we measure this?” but “What conclusion does this measurement support?”

Even a direct observation has a limited scope. Seeing that a person reached a control without scrolling supports a claim about that encounter. It does not establish that the layout works for every screen size, input method, or task.

Observation is a design decision

A feedback loop’s view is shaped by several choices: where observation begins, which events are recognized, how they are recorded, and what distinctions survive when the evidence is summarized.

Consider measuring how long it takes to submit a form. Starting the clock at the first keystroke leaves out time spent understanding the instructions. Stopping it when the server accepts the submission leaves out time spent determining whether the submission succeeded.

Neither boundary is automatically wrong. Each defines a different question.

If the question concerns server processing, the narrower boundary may be appropriate. If it concerns a person’s ability to complete the task, that same boundary may exclude important parts of the experience.

The observation needs a working definition

Before comparing measurements, define what they mean:

• Does “completed” mean submitted, accepted, saved, or independently confirmed?
• Does “error” include only failures the software detects, or also mistakes discovered during review?
• Does “time spent” include interruptions, reading, and recovery?
• Does “successful search” mean opening a result or finding information that answers the question?

These distinctions connect to a broader engineering principle: network success is not task success. Evidence that one component worked should not silently become evidence that the whole activity succeeded.

Observation is not yet feedback

Recorded information becomes part of an effective feedback loop when it influences subsequent action. A report that nobody reviews may contain useful evidence without closing the loop.

The interpretation matters too. The same increase in completion time might lead one team to remove a review step and another to investigate confusing instructions. What the loop does depends on how the evidence is understood, not merely on whether it was collected.

An editing workflow example

Imagine an editor who repeatedly has to scroll after returning from a long article to reach the next control. A developer changes the layout to make that control easier to reach.

A narrow evaluation might ask whether scrolling decreased. That is a reasonable starting point, but the intended improvement is probably broader: help the editor continue working without losing orientation or introducing new obstacles.

Several observations could help evaluate the change:

• Reachability: can the editor reach the next control without unnecessary movement?
• Orientation: is it still clear which article and workflow stage are active?
• Keyboard use: does the control remain reachable in a meaningful focus order?
• Content access: does its new position obscure text or become awkward when zoomed?
• Outcome clarity: can the editor still distinguish saving, previewing, and publishing?

This does not require a permanent measurement system for every item. A focused walkthrough, keyboard check, and conversation during ordinary use may provide enough evidence to assess a small revision.

Suppose the editor reaches the control more quickly but now confuses it with a publishing action. The original metric improved, yet the workflow became less safe.

That result does not make reachability an invalid measure. It shows why reachability should not carry the whole judgment.

A sound conclusion might be: “The revised layout reduced scrolling in the observed sessions, but the action labels need revision before we can regard the change as successful.” This preserves the useful finding without overstating it.

For changes affecting interaction, testing with assistive technology can reveal consequences that pointer-based observation alone would miss.

What missing evidence can conceal

What a loop does not observe can matter as much as what it records. Missing evidence is especially easy to overlook when the available numbers look precise.

Counts need context

A decrease from twenty recorded errors to ten sounds like an improvement. But if attempts fell from two hundred to fifty, the recorded error rate rose from 10 percent to 20 percent.

Even a rate requires interpretation. The task may have changed, different people may be attempting it, or the mechanism that records errors may have stopped capturing some failures.

When evidence changes, consider at least three possibilities:

• The underlying condition changed.
• The mix of people, tasks, or circumstances changed.
• The observation process changed.

More than one may be true.

Successful completions leave out unsuccessful attempts

Measuring completion time only for people who finish excludes those who abandon the task. A difficult interface might appear efficient if the people who encounter the greatest barriers never reach the event that stops the clock.

Similarly, an optional survey may reveal valuable experiences without representing everyone affected. People who could not access the survey, did not notice it, or chose not to respond remain outside that evidence.

These limits do not invalidate the observations. They limit the conclusions that can reasonably follow.

Summaries can hide uneven outcomes

An improved average can coexist with a worse experience for some users. A layout might become faster for pointer users while becoming harder to navigate by keyboard.

Where practical and appropriate, examine relevant differences between tasks and interaction conditions. This can often be done through deliberately chosen test scenarios rather than collecting personal characteristics.

The aim is not endless segmentation. It is to avoid letting a convenient summary erase a consequential problem.

When measurement changes behavior

A measurement may begin as a way to understand work and later become a target that shapes it.

If a support team is judged mainly by ticket-closing speed, staff may have an incentive to close difficult cases prematurely. If an editorial process rewards publication count, careful review may appear to be an obstacle rather than part of the work.

This is often discussed through Goodhart’s law: when a measure becomes a target, it can become less reliable as a measure. The practical concern is the changing relationship between the indicator and the purpose it was intended to serve.

No deliberate manipulation is required. People may reasonably adapt to the expectations placed on them. An automated controller can also pursue a poorly specified target without recognizing the broader consequences.

Check what must not worsen

One response is to pair an improvement measure with explicit checks on important consequences.

For the editing example, reduced navigation effort might be the desired improvement. Preserving access to content, keyboard operation, and clear action boundaries would be conditions that must not worsen.

Additional checks do not guarantee success. They do, however, make the decision more honest: “faster” is evaluated alongside the things speed must not sacrifice.

People also need a way to challenge the target itself. A feedback loop may regulate effectively toward a condition that no longer serves the work.

Enough information, not maximum collection

A limited view does not justify collecting everything. More observation can introduce privacy risks, maintenance burdens, and additional noise without improving the decision.

A useful observation plan gathers evidence proportionate to the question and the consequences of acting on it.

A small interface revision may need a few carefully chosen tests. A change affecting many people or an important irreversible action needs stronger evidence and a more deliberate review.

Before adding data collection, ask:

• Which uncertainty would this information reduce?
• What decision could change because of it?
• Could a less intrusive observation answer the same question?
• Who needs access, and how long must the information remain available?

Qualitative evidence belongs in the loop

A person saying “I cannot tell whether my work was saved” provides information that a save-success counter may never capture.

That account does not establish how common the problem is. It can nevertheless reveal a failure mode worth investigating. A single reproducible case may be enough to establish that a problem exists, even when its prevalence remains unknown.

Numbers and accounts of experience can support different parts of the same investigation. Neither should be forced to answer questions it cannot resolve.

This is where distinguishing fact, inference, and uncertainty becomes practical: record what happened, explain what it suggests, and leave unresolved questions visible.

A practical review of a feedback loop

Whether the loop is automated, human-guided, or a combination of both, a short review can clarify what its observations support.

1. Name the condition that matters. Describe the outcome or acceptable range before choosing a convenient metric.
2. Define the signal. State exactly what is observed, when it is recorded, and what its boundaries exclude.
3. Explain the relationship. Identify why the signal is evidence about the condition, including any proxy assumptions.
4. Look for missing cases. Consider abandonment, undetected failures, delayed consequences, and people or tasks absent from the evidence.
5. Check competing consequences. Ask what could worsen while the main indicator improves.
6. Identify the response. Establish who or what can act, under which conditions, and with what authority.
7. Observe after acting. Allow enough time for relevant consequences to appear, then revisit both the change and the measurement.

The observation plan itself may need revision. If users keep reporting difficulties that never appear in the dashboard, the discrepancy is evidence about the dashboard’s limits—not a reason to dismiss their experience.

A dependable feedback loop does not need to see everything. It needs a sufficiently informative view for the actions it takes, a clear account of what remains outside that view, and a way to reconsider its interpretation.

Choosing what the loop can see is therefore also choosing the limits within which it can act responsibly. The measurement remains useful when those limits stay visible.