Human Review Workflows and Confidence Calibration

Human review workflows and confidence calibration convert a Claude agent from a "trust everything" black box into a production-grade decision system where humans inspect what matters and auto-approve what does not. Task statement 5.5 of the Claude Certified Architect — Foundations (CCA-F) exam — "Design human review workflows and confidence calibration" — sits inside Domain 5 (Context Management & Reliability, 15 % weight) but its reach is broader than the weight suggests: every scenario that involves irreversible actions, regulated data, or high-cost errors touches this task statement. Customer-support resolutions, structured data extractions into downstream systems, code generation committed to CI/CD, and multi-agent research outputs all need an answer to the same question: when does a human look at this, and how do we know the model's confidence can be trusted?

This study note unpacks the full human-review surface a CCA-F candidate is expected to design: the architecture of where humans integrate into an autonomous decision chain, the difference between raw model confidence and calibrated confidence, field-level confidence scores versus single-output scores, confidence threshold design that avoids over-escalation, stratified sampling across confidence bands, the counter-intuitive need to spot-check high-confidence items for silent errors, review queue priority and SLA design, reviewer feedback integration for prompt improvement, review UI principles, throughput-versus-quality trade-offs, audit trail structure, a traps section that every community pass report flags, practice anchors tied to the customer-support-resolution-agent and structured-data-extraction scenarios, and an FAQ that closes the loop from design to exam technique.

Human Review Workflow Overview — Where Humans Integrate Into AI Decision Chains

A human review workflow is the set of design decisions that determine which Claude outputs a human inspects before they become ground truth. In a pure-autonomous system, every output is acted on immediately. In a pure-manual system, every output is inspected. Production systems live somewhere in the middle — they auto-approve the majority of cases, route a minority to reviewers, and expose a structured audit trail for the rest. The architect's job is to pick where on that spectrum the system sits, scenario by scenario.

The Three Integration Points

A well-designed workflow chooses between three positions for the human checkpoint:

1. Pre-action review — Human reviews the model's proposed action before it executes against the world. Appropriate when the action is irreversible (refund issued, code merged, record deleted, email sent to a customer).
2. Post-action review — Human reviews the outcome after the action has executed but before downstream consumers see it. Appropriate when the action is reversible and the review is about quality control rather than safety.
3. Sampled review — Only a subset of outputs is reviewed, stratified across confidence bands. Appropriate when volume is high and the cost-of-error is moderate.

Every CCA-F scenario maps cleanly to one of these three positions. A customer-support agent issuing a refund above a threshold must use pre-action review; a structured-data-extraction pipeline writing to a staging table typically uses sampled post-action review; an agent categorizing internal documents for retrieval might use no review at all on obviously high-confidence items.

Why CCA-F Tests This Explicitly

The exam repeatedly asks whether a given scenario should auto-approve, route-to-human, or escalate, and what the trigger should be. Community pass reports cite "picked auto-approve when the scenario had a non-trivial cost of error" as one of the top Domain 5 mistakes. The exam rewards architects who treat review workflows as a first-class design concern, not an afterthought appended to an otherwise-autonomous agent.

Confidence Calibration Concept — Aligning Model Confidence With Actual Accuracy

A Claude model can express confidence — "I am 90 % sure this invoice number is 48293" — but the raw number is only useful to the extent it matches empirical accuracy. Confidence calibration is the practice of ensuring that when the model says "90 % confident," the output is actually correct 90 % of the time on a representative sample.

Calibrated vs Uncalibrated Confidence

An uncalibrated confidence score is a number the model emits that has no verified relationship to real-world accuracy. A model that says "95 % confident" on outputs that are correct only 60 % of the time is severely overconfident and dangerous in a review workflow — the threshold policy built on top of it will auto-approve far too aggressively.

A calibrated confidence score is one that has been empirically validated against a gold-standard dataset. If the model says "95 %," you have measured that it is correct close to 95 % of the time across the confidence bucket. Only calibrated confidence is safe to build threshold policies on.

How to Calibrate in Practice

Calibration is an operational workflow, not a configuration flag:

1. Collect a representative labeled sample (hundreds to thousands of items, ground-truth labels from humans).
2. Run the agent over the sample and capture the model's confidence for each output.
3. Bucket outputs by confidence band (for example, 0.5–0.6, 0.6–0.7, ..., 0.9–1.0).
4. Measure actual accuracy within each band.
5. Plot expressed confidence against measured accuracy; the ideal is a 45-degree line (y = x).
6. Adjust thresholds based on the empirical curve, not the raw model output.

The output of calibration is a mapping table from expressed confidence to trusted accuracy, and it is the prerequisite for any threshold policy downstream.

Field-Level Confidence Scores — Per-Field Uncertainty Rather Than Single Output Score

A common architectural mistake is asking Claude for a single confidence score on a complex output that actually contains multiple independent fields. A structured extraction that produces {invoice_number, vendor_name, line_items, total, due_date} has five independent sources of uncertainty. A single document-level score obscures all of them.

The Field-Level Pattern

Instead of confidence: 0.87 on the whole record, the architecture emits a confidence per field:

{
  "invoice_number": {"value": "48293", "confidence": 0.98},
  "vendor_name": {"value": "Acme Corp", "confidence": 0.95},
  "line_items": {"value": [...], "confidence": 0.62},
  "total": {"value": 1284.50, "confidence": 0.99},
  "due_date": {"value": "2026-05-15", "confidence": 0.74}
}

Now the review policy can route only the uncertain fields to a human — not the whole record. The reviewer sees "please verify line_items and due_date" with the high-confidence fields pre-filled. Throughput goes up, error rates on the flagged fields go down, and reviewer fatigue goes down because they only inspect what is genuinely uncertain.

How Field-Level Scores Are Enforced in the Schema

Field-level confidence is expressed as part of the structured output schema. Using the strict tool use pattern, each field's confidence is a required property of a per-field wrapper object, and the tool definition enforces that the model cannot emit a field without a confidence value. This converts confidence reporting from a soft prompt instruction into a schema contract the model must satisfy.

Why a Single Score Is Wrong for Multi-Field Outputs

A single document-level score averages the uncertainty across all fields. A record where the model is 99 % sure of four fields and 50 % sure of the fifth will often report something like 90 % overall — which is above most auto-approve thresholds and will ship a 50 % likely wrong fifth field into production. Field-level scoring surfaces the risky field instead of smuggling it through.

Confidence Threshold Design — Setting Review Triggers Without Over-Escalating

The threshold policy is the rule that converts a confidence number into a routing decision: auto-approve, route-to-human, or escalate. Setting the threshold badly has asymmetric consequences — too lenient means undetected errors reach production, too strict means reviewers drown in obviously-correct items and the system loses the throughput benefits of automation.

The Two-Threshold Policy

A mature policy uses two thresholds, not one:

• Auto-approve threshold (for example, 0.95) — Outputs at or above this are auto-approved, with optional sampled spot-check (see the high-confidence sampling section).
• Escalate threshold (for example, 0.60) — Outputs at or below this are escalated to a specialist reviewer or deflected entirely (the agent refuses to answer and asks the user for clarification).
• Middle band (between the two) — Outputs land in the standard review queue and are looked at by a first-line reviewer.

Two thresholds let the system reserve expensive specialist time for genuinely hard cases and keep standard reviewers productive on the middle band.

Threshold Sources — Business Cost, Not Gut Feel

Pick thresholds by empirical reasoning about the cost of a false accept versus the cost of a reviewer-minute. If the cost of a missed error is $500 and a reviewer-minute costs $1, even low-probability errors justify a review. If the cost of a missed error is $0.10 and a reviewer-minute costs $1, only high-probability errors justify a review. The threshold is the break-even point between those two costs given the empirically measured false-accept rate at each confidence band.

Over-Escalation Failure Mode

Setting the auto-approve threshold too high (for example, 0.99 in a system where 0.95 is already safe) floods the review queue with items humans cannot usefully evaluate — they see 0.96 outputs that are indistinguishable from the 0.99 outputs they just approved, and reviewer quality drops. Over-escalation is a real failure mode, not a safety win.

Stratified Sampling — Reviewing Representative Samples Across Confidence Bands

Stratified sampling is the practice of drawing a review sample from every confidence band proportionally, rather than reviewing only the lowest-confidence items. It is the most frequently misunderstood concept in this topic area and is a direct exam-trap target.

Why Random Sampling Is Not Enough

If you sample uniformly at random from the output stream, you will see mostly high-confidence items (because most items are high confidence) and your sample will not contain enough low-confidence items to measure the error rate in that band. Conversely, if you sample only low-confidence items, you never observe whether high-confidence items are actually as correct as the model says they are.

What Stratified Sampling Actually Does

Stratified sampling bins outputs by confidence band, then samples a fixed number from each band:

• From the 0.5–0.6 band, sample N items.
• From the 0.6–0.7 band, sample N items.
• ...
• From the 0.9–1.0 band, sample N items.

Every band is observed. You can now measure empirical accuracy in each band, detect calibration drift, and catch systematic errors that only appear in a specific confidence range.

Stratified Sampling as the Calibration Feedback Loop

Stratified sampling is the mechanism by which calibration stays honest over time. Models drift; data distributions drift; upstream tools drift. A stratified sample re-collected monthly tells you whether the 0.95 band is still 95 % accurate or whether it has quietly decayed to 85 %. Without stratified sampling, calibration is a one-time event that goes stale.

High-Confidence Sampling — Spot-Checking Auto-Approved Items for Silent Error Detection

Sampling the high-confidence band specifically is worth calling out because it is the single most counter-intuitive piece of this topic. Most architects look at a 0.98-confidence item and assume it is fine. The exam rewards architects who assume nothing until empirical evidence says so.

Silent Errors

A silent error is a model output that is wrong but carries high confidence. It shows up in the auto-approve path, never flows to a reviewer, and never surfaces as a complaint because downstream systems accept it. Silent errors are the worst kind because nobody notices until a pattern emerges in production — a batch of mis-extracted invoice totals, a cluster of misrouted support tickets, a quiet contamination of a downstream dataset.

How High-Confidence Sampling Catches Them

Periodically, a random subset of auto-approved items is routed to a reviewer as if it were a normal queue item. The reviewer does not know it came from the high-confidence band — they just review it. The measured error rate on this sample is the empirical false-accept rate, and it is the only reliable signal that the auto-approve threshold is still safe.

Sampling Rate Trade-Off

The sampling rate is a cost knob. Higher rate means more reviewer time spent on items that are almost always correct; lower rate means silent errors take longer to detect. Typical production values are 1–5 % of auto-approved outputs sampled, with the rate increasing temporarily whenever the model, the prompt, or the upstream data changes meaningfully.

Review Queue Design — Priority Ordering, SLA, Assignment Routing

Once an output has been routed to a human, the review queue is the operational structure that decides who sees it, when, and in what order. Queue design is an architectural decision, not a ticketing-system detail.

Priority Ordering

Queue items are not equal. A $10 000 refund approval waiting for review should not sit behind a $50 refund approval just because the $50 one arrived first. Priority ordering weighs:

• Business value of the underlying decision (transaction size, customer tier, regulatory sensitivity).
• Confidence distance from threshold — an output at 0.62 (near the middle-band floor) is arguably more urgent than one at 0.85 (comfortably in the middle band) because the former is closer to the escalate cut.
• Upstream deadlines — SLAs inherited from customer promises or downstream process gates.

SLA Design

Every review queue needs an explicit time-to-review SLA. Without one, the queue becomes the backlog where items quietly die. Typical SLAs are tiered: critical items in minutes, standard items in hours, low-priority items in a business day. The SLA is the contract between the autonomous system and its human operators; breaches trigger alerts and staffing adjustments.

Assignment Routing

Not every reviewer is qualified for every item. A specialist reviewer for fraud decisions is wasted on document categorization, and vice versa. Assignment routing maps item attributes (category, priority, required expertise) to reviewer skill profiles. In small deployments a single queue is fine; at scale, per-skill queues are required.

Queue Observability

The queue itself should be instrumented: depth, age of oldest item, throughput per reviewer, approval-versus-correction rate per reviewer. These metrics close the feedback loop between operational reality and architectural decisions — a rising queue depth is an early warning that thresholds are too strict or that the model has regressed.

Reviewer Feedback Integration — Capturing Corrections to Improve Future Prompts

Every reviewer action is a labeled data point. A mature workflow captures it and feeds it back into the system — not as fine-tuning (out of scope for CCA-F) but as prompt improvement, threshold re-calibration, and error-pattern analysis.

Three Kinds of Feedback to Capture

1. Approve without edits — The model was right. Record the item as a positive example in the confidence band it came from.
2. Approve with edits — The model was close but not quite right. Record the delta between model output and reviewer correction. This is the gold feedback — it is the precise thing the model got wrong.
3. Reject — The model was materially wrong or the request was out-of-scope for the agent. Record the rejection reason in a structured category.

The delta from "approve with edits" is the most valuable because it identifies systematic model errors: reviewers repeatedly correcting the same field, the same phrase, the same class of error reveals a prompt-level fix.

Feeding Corrections Back as Few-Shot Examples

When a class of correction emerges (reviewers keep fixing the same type of field), add the before/after pair to the agent's prompt as a few-shot example using the multishot prompting pattern. The model then learns the correction in-context without any training. This closes the loop from reviewer action to prompt improvement within days, not quarters.

Threshold Recalibration From Feedback

If reviewers are consistently approving without edits in the 0.85–0.90 band, the auto-approve threshold can arguably be lowered to 0.85. If reviewers are consistently rejecting or materially editing in the 0.90–0.95 band, the auto-approve threshold should be raised. The feedback-driven threshold adjustment loop is how a static policy becomes a living calibration.

Review UI Considerations — Presenting Context Sufficient for Informed Human Decision

A reviewer cannot make a good decision without seeing the information the model saw. The review UI is therefore an architectural concern, not a front-end implementation detail.

What the Reviewer Must See

• The model's output — The actual proposed action or extracted data, field-by-field.
• The model's confidence — Per field where applicable, plus an overall score.
• The model's reasoning — A brief, structured rationale or, where exposed, the content of an <thinking> block.
• The input context — The source document, the customer message, the code diff, the research query. Whatever the model used as input.
• Relevant history — Prior turns of the same case, prior decisions from the same customer, prior outputs on similar items.
• The decision affordance — Approve, approve-with-edits, reject, escalate. Each action must be one click with optional structured reason capture.

What the Reviewer Should Not See

• Unprocessed raw logs — Dumping the full tool call history overwhelms the reviewer and drops their accuracy.
• Confidence scores without calibration context — A raw 0.87 is meaningless to a reviewer who does not know what 0.87 means in this system. The UI should translate ("the model is correct on items at this confidence level about 91 % of the time based on last month's calibration").

The "Lost in the Middle" Effect Inside the UI

Reviewers exhibit the same attention-decay pattern as models: critical information buried in the middle of a long scroll is often missed. Put the key decision points at the top, summarize long inputs, and visually distinguish the items that need review (the low-confidence fields) from the items that do not.

Review Throughput vs Quality — Balancing Reviewer Load Against Error Rate

Every review workflow has a throughput-quality frontier: at a given reviewer staff level, you can review X items per day at Y% error rate. Pushing to review more items means quality drops (reviewer fatigue, rushed decisions); holding quality constant means items accumulate in the queue.

The Three Levers an Architect Controls

1. Threshold placement — Moving the auto-approve threshold down reduces review volume; moving it up increases review volume.
2. Field-level routing — Routing only the uncertain fields (rather than whole records) cuts review time per item by 40–70 % in practice.
3. Reviewer specialization — Matching item complexity to reviewer skill tier increases throughput at constant quality.

The Economic Calculation

Every threshold decision has a cost: (cost per reviewer-minute × minutes per review × review volume) versus (false-accept rate × cost per false accept × total volume). The optimum is where marginal cost of review equals marginal cost of missed errors. This is not a moral decision — it is an economic one, and the exam treats it that way. Scenarios that frame review as "always more is better" are testing whether you understand the trade-off.

Fatigue as a Silent Quality Killer

Reviewers who process identical-looking items for hours develop automaticity — they start approving without actually looking. Combat this with rotation (mixing item types), mandatory breaks, sampled quality audits on reviewer decisions themselves, and queue throttling to prevent overload.

Audit Trail Design — Recording Human Decisions for Downstream Analysis

An audit trail is the durable record of every decision the system made, who (or what) made it, and what they saw at the moment of decision. It is the prerequisite for any subsequent analysis — compliance, incident investigation, threshold tuning, reviewer calibration, and model regression detection.

What Every Audit Record Should Contain

• Timestamp — When the decision was made, to the second.
• Decision outcome — Auto-approved, approved-with-edits, rejected, escalated, deflected.
• Actor — The model ID (if auto-approved) or the reviewer ID (if human).
• Model output snapshot — The exact output the model produced, including confidence scores.
• Reviewer action delta — If a human edited, the exact before/after delta.
• Input snapshot — Either the input itself or a durable reference to it (content hash).
• Policy in effect — The threshold values, the prompt version, the model version at decision time. Snapshotting these is critical — policies change and historical decisions must be evaluable against the policy that governed them.

Why Policy Snapshotting Matters

If you raise the auto-approve threshold in June, items auto-approved under the old threshold in May are not wrong retroactively — they were correct under the policy in effect at the time. Without policy snapshotting, every threshold change poisons your historical audit data.

Audit Trail as Provenance

The audit trail is also the backbone of provenance tracking in multi-source synthesis scenarios (task 5.6). When a downstream consumer of the agent's output asks "where did this come from?", the audit trail answers — input, model output, reviewer action (if any), policy in effect.

Integrating Review Workflows With the Agentic Loop

The review workflow is not a separate system bolted onto an agent — it is a structural part of the agentic loop's termination logic. When the agent reaches a decision point that exceeds the auto-approve threshold or falls into the escalate band, the loop pauses and the review queue becomes the next step. Once a reviewer acts, the decision flows back into the loop (or into the agent's audit trail) as a tool_result-like observation.

The Pause-and-Resume Pattern

An agentic loop that integrates human review typically uses the lower-level process() SDK entry point (covered in task 1.1) because the loop must pause externally and resume when the reviewer completes the action. run() and stream() do not cleanly express an external approval gate; process() does.

Confidence Scores as Loop-Termination Inputs

Inside the loop, the architect wires confidence into termination conditions. Low confidence on a critical field does not just lower the score — it triggers a specific branch (route to review) that is mechanically distinct from the normal end_turn path. This is the mechanical realization of the review workflow within the loop structure.

Structured Output Enforces the Signal

The confidence field must be a required property of the agent's structured output, enforced via strict tool use. A soft prompt ("please include a confidence") is not enough — the model will occasionally omit it, and your review logic will fall over. Strict schema guarantees the signal is always present.

Plain-English Explanation

Three analogies from different domains make the abstract workflow concepts tangible.

Analogy 1: The Hospital Triage Desk — Threshold Design and Priority Queues

A hospital emergency-room triage nurse does exactly what a confidence-threshold policy does. A patient arrives; the nurse looks at vitals, complaint, and history and assigns one of five acuity levels. Level 1 (life-threatening) goes straight to the resuscitation room — no queue. Level 5 (routine) sits in the waiting area and may wait hours. The middle levels go to exam rooms as they open. The triage decision is made from limited information and the nurse's confidence, but the consequence of the decision is calibrated by the cost of being wrong: sending a Level 1 to the waiting room kills someone; sending a Level 5 to the resuscitation room is wasteful but not fatal. The triage nurse also does periodic audits — a senior nurse samples the triage decisions to check for calibration drift. Some of the audited decisions are the obvious Level 5s (stratified sampling — you have to check the easy ones too, otherwise a Level 1 mis-triaged as a Level 5 will never be caught). Reviewer feedback flows back into training — systematic mis-triage patterns lead to triage protocol updates. The hospital triage desk is a human-review workflow with thresholds, priority routing, stratified audit sampling, and an audit trail — exactly the shape CCA-F is testing.

Analogy 2: The Customs Inspection Line — Stratified Sampling and High-Confidence Spot Checks

An international airport customs line handles thousands of travelers per hour. Most pass through green lanes (auto-approve). A minority are pulled into secondary inspection based on risk signals (route-to-human). A small random sample from the green lane is also pulled aside for inspection despite showing no risk signals — that is high-confidence sampling. The customs officers know that if they only inspect the suspicious-looking travelers, sophisticated smugglers who look exactly like normal travelers will sail through undetected. The random green-lane pull is the only defense against silent errors. Stratified sampling in the customs analogy is picking a proportional sample from every risk band — "we inspect 0.5 % of green lane, 5 % of yellow lane, and 100 % of red lane" — so that every risk tier is observed and the risk-scoring model stays honest. An auditor who only reviewed red-lane cases would never detect that green-lane confidence had drifted. The customs line also keeps an audit trail (every inspection outcome logged with officer ID, time, and rationale), uses tiered SLAs (flight connection times drive priority), and routes specialist cases (agricultural, controlled substances) to specialist officers. It is the customer-support-resolution-agent scenario in physical form.

Analogy 3: The Manuscript Editor — Field-Level Confidence and Reviewer Fatigue

A magazine copy editor reviewing a 3 000-word article does not read every sentence with equal intensity. They skim sentences they are confident are clean (the author is a veteran, the opening paragraph is well-structured) and slow down on passages where something feels off — a tense shift, a fact that should be checked, a name they do not recognize. The editor is doing field-level confidence review: high-confidence spans get a skim, low-confidence spans get full attention, and the uncertain bits are flagged for the author to address. If a copy editor tried to apply equal scrutiny to every sentence in every article, they would either miss issues (fatigue) or produce a handful of articles per week (throughput collapse). Field-level confidence in a Claude extraction is the same idea: tell the reviewer which fields are uncertain so they can focus their attention there and rubber-stamp the rest. When the editor keeps correcting the same mistake across many articles (the author always misuses "comprise"), that pattern flows back as a style-guide update — the reviewer feedback integration loop. And when the magazine does its annual quality audit, it samples articles from the whole year, not just the flagged ones, because the silent errors are the ones that slipped past the first editor unnoticed.

Which Analogy Fits Which Exam Question

• Questions about threshold design and priority queues → hospital triage desk analogy.
• Questions about stratified sampling and silent errors → customs inspection line analogy.
• Questions about field-level confidence and reviewer throughput → manuscript editor analogy.

Common Exam Traps

CCA-F Domain 5 exam items around human review workflows exploit five recurring trap patterns. All five appear in community pass reports as "close but wrong" distractor shapes.

Trap 1: High Confidence Means Correct

"The output has confidence 0.98, so we can skip review." Wrong — high confidence is necessary but not sufficient. Calibration drift, systematic errors, and silent failures all live in the high-confidence band. The correct pattern is auto-approve plus high-confidence sampling, not auto-approve alone. Any answer choice that treats a high confidence score as a terminal trust signal is a trap.

Trap 2: Stratified Sampling Means Only Low-Confidence Sampling

"We review everything below 0.80." That is low-confidence sampling, not stratified sampling. Stratified sampling requires proportional samples from every band — including 0.95+, where silent errors hide. Answer choices that describe "sample only items below threshold" or "sample only uncertain outputs" are incomplete and will be scored wrong. The correct phrase to look for is proportional across bands.

Trap 3: Single Document-Level Confidence Is Enough

"We ask Claude for a confidence score on the whole record and route below 0.90." For multi-field outputs this is wrong. A 0.90 overall can hide a 0.50 field, and that field will reach production unchallenged. Correct design emits field-level confidence and routes at field granularity. Any scenario with a multi-field structured output should trigger this trap recognition.

Trap 4: Always Escalate When In Doubt

"Safer to send borderline cases to a human." Over-escalation is a real failure mode: reviewers fatigue, throughput collapses, and specialist reviewer time is wasted on cases first-line reviewers could have handled. Correct policy uses two thresholds (auto-approve, escalate) with a middle band for standard review, and escalate time is reserved for genuinely hard cases. Answers that describe a single-threshold "review everything below X" are less sophisticated than two-threshold answers.

Trap 5: Reviewer Corrections Are Disposable

"We close the ticket after the reviewer approves." Wrong — the correction delta is the highest-signal training data the system will ever produce, and discarding it means the same error recurs next week. Correct design captures corrections into a structured table feeding prompt refinement, threshold recalibration, and few-shot example addition. Answers that do not mention feedback integration are incomplete.

Practice Anchors

Human review and confidence calibration concepts surface most heavily in two of the six CCA-F scenarios. Treat the following as the design spine for scenario-cluster questions on task 5.5.

Customer-Support-Resolution-Agent Scenario

In this scenario, a support agent autonomously triages and resolves inbound tickets — answering questions, issuing small refunds, updating account settings, and escalating complex cases. Human review shows up in three places: a pre-action review for refunds above a threshold (irreversible financial action), a sampled post-action review on resolved tickets (quality control), and an ambiguity-escalation branch on low-confidence classifications. Expect questions that test whether you correctly route an irreversible high-value action (always review), whether you stratify the post-resolution quality sample (yes, across confidence bands), and whether you capture the corrections from human-reviewed resolutions back into the agent's prompt as few-shot examples. The scenario also exercises the distinction between escalating to a specialist reviewer (low confidence, complex case) and deflecting to the customer (ambiguous input, ask for clarification).

Structured-Data-Extraction Scenario

In this scenario, a batch pipeline extracts structured records (invoices, contracts, forms) from unstructured sources and loads them into a downstream system. Review workflow design is central: field-level confidence is emitted in the extraction schema (enforced via strict tool use), the auto-approve threshold is calibrated against a labeled sample, stratified sampling runs continuously to detect calibration drift, and corrections from reviewers flow back as few-shot examples for the extraction prompt. Expect questions that test whether you emit per-field versus per-record confidence (per-field), whether high-confidence items are ever sampled (yes, regularly), how you schema-enforce the confidence signal (strict tool use, not soft prompt), and whether the audit trail snapshots the policy version in effect (yes, so historical decisions remain evaluable).

FAQ — Human Review Workflows Top 5 Questions

What is the difference between confidence calibration and just asking Claude for a confidence score?

Asking Claude for a confidence score produces an uncalibrated number — a value the model emits based on its internal uncertainty that has no guaranteed relationship to real-world accuracy. Confidence calibration is the operational process of measuring the model's expressed confidence against ground-truth accuracy on a labeled sample, bucketing by confidence band, and producing a mapping table from expressed confidence to measured accuracy. Only calibrated confidence is safe to build threshold policies on. On CCA-F, scenario answers that plug a raw model confidence directly into a threshold policy without describing calibration are incomplete and will be marked wrong when a better answer exists. The correct pattern is measure first, then threshold.

Why does stratified sampling have to include high-confidence items?

Because silent errors — items the model is wrong about but reports as high confidence — only show up in the auto-approve path and never otherwise reach a reviewer. If you sample only below the auto-approve threshold, the high-confidence band is never observed and calibration drift in that band cannot be detected. Stratified sampling draws proportionally from every confidence band (including 0.95+) so that the empirical accuracy of each band is measured continuously. This is the most-tested sub-concept in this topic — answer choices that describe "sample only items below threshold" or "sample only uncertain outputs" are trap distractors, while "sample proportionally across all confidence bands" or equivalent phrasing is the correct answer.

When should I use field-level confidence scores instead of a single overall score?

Use field-level confidence whenever the output contains multiple independent fields. A 0.90 overall score on a record with five fields can hide a 0.50 field that will reach production with no review. Field-level confidence lets the routing policy inspect per field and route only the uncertain fields to a human, auto-approving the rest pre-filled. This improves throughput (reviewers see only the risky fields) without increasing error rate. Enforce the field-level schema via strict tool use so the model cannot silently omit a confidence value; do not rely on soft prompt instructions. The structured-data-extraction scenario on CCA-F directly tests this pattern.

How do I choose the auto-approve and escalate thresholds?

Calibrate empirically against a labeled sample, then pick the thresholds where the marginal cost of review equals the marginal cost of missed errors at that confidence band. The auto-approve threshold is where false-accept rate times cost-of-error equals reviewer-minute cost; above it, auto-approve plus sampled spot-check is cheaper than routine review. The escalate threshold is where a first-line reviewer is unlikely to resolve the item correctly and specialist time is warranted. Between the two, items go to the standard review queue. Setting either threshold by gut feel rather than measurement is a common trap; setting a single threshold instead of two is a less sophisticated design that exam distractors will include as a tempting but inferior choice.

What is the most important thing to capture in an audit trail for a review workflow?

The policy version in effect at decision time — the threshold values, the prompt version, the model version, the tool schema version. Without this snapshot, every future policy change invalidates your historical audit data because you can no longer tell whether a past auto-approval was correct under the policy that governed it. Capture the standard fields too (timestamp, actor, decision outcome, model output snapshot, reviewer delta, input reference), but the policy snapshot is the one most often missed. The audit trail is also the foundation for downstream provenance tracking in multi-source synthesis scenarios (task 5.6), so its design choices propagate beyond task 5.5 into the broader Domain 5 surface.

Further Reading

• Building effective agents — escalation and human-in-the-loop: https://docs.anthropic.com/en/docs/build-with-claude/agentic-loop
• Handle tool calls — tool_result format and error responses: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/handle-tool-calls
• Increase output consistency — structured outputs: https://docs.anthropic.com/en/docs/test-and-evaluate/strengthen-guardrails/increase-consistency
• Strict tool use — schema-guaranteed output: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/strict-tool-use
• Writing tools for agents — Anthropic Engineering Blog: https://www.anthropic.com/engineering/writing-tools-for-agents

Related ExamHub topics: Escalation and Ambiguity Resolution, Multi-Instance and Multi-Pass Review Architectures, Validation and Retry Feedback Loops for Extraction, Information Provenance and Uncertainty in Multi-Source Synthesis.