LLM Requests

Status: Approved Type: Technical (Recommended*)

*Strongly recommended for every service that invokes an LLM (Bedrock/Anthropic). The practices here are distilled from transcript-service, enrich-service, and audit-service production incidents and fixes.

Problem

A service that calls an LLM inherits a dependency that is non-deterministic, expensive, and schema-unaware. Without a shared discipline, every service rediscovers the same failure classes the hard way:

• Wasted spend and latency — prompts assembled with volatile content first defeat the provider's prompt cache; nobody notices because cache hits are not measured.
• Hallucinated identifiers — the model mistypes long, high-entropy strings (UUIDs) it was asked to echo back, corrupting joins downstream.
• Unparseable or mistyped output — the model emits unescaped quotes inside JSON strings, wraps output in markdown fences, stringifies an array field, or omits a required field. Forced tool use fixes the envelope, not the field types — a lesson that cost two production incidents (audit-service #70, #72).
• Enum drift — free-text-ish prompts let the model invent near-miss enum values ("greeting" when the list says "welcome"), causing hard validation failures on some inputs only.
• Blind retries — pipeline-level retries replay content-deterministic failures verbatim and never converge, while the one retry mode that works (re-invoking with corrective feedback) is missing.
• Unreproducible failures — the exact prompt that failed was never persisted, so nobody can replay it to diagnose or to verify a fix.
• Transport surprises — long non-streaming generations idle the TCP connection for minutes; NAT hops kill it and the job dies as an opaque ECONNRESET or a silent Lambda timeout.

Context

When to Use This Pattern

• Building or reviewing any service step that invokes an LLM (currently Bedrock/Anthropic models) — e.g. the analysis steps of enrich-service, the audit step of audit-service, the profile-participants/contextualize steps of transcript-service.
• Designing the prompt, input document, output schema, or error handling for such a step.
• Diagnosing recurring LLM-output failures (validation errors, JSON parse errors, enum rejections).

When NOT to Use This Pattern

• Choosing model ARNs, inference profiles, tagging, or cost attribution — that is AWS Bedrock Inference Profiles.
• Bridging real Bedrock credentials into a LocalStack sandbox — that is Local AWS Sandbox.
• Operating the surrounding async pipeline (idempotency purges, relaunch procedures, transient-vs-deterministic triage of jobs) — that belongs to the owning service's runbooks; this pattern covers the request itself.

Audience & Scope

Engineers and AI agents building or maintaining LLM-invoking service steps. The scope is one request: from prompt assembly to a validated, typed output object. Everything before (job routing) and after (persisting results, callbacks) is the service's own architecture.

Solution

Treat the LLM as an untrusted, expensive, cache-sensitive dependency, and structure every request along its lifecycle:

1. Assemble the prompt        static-first, cache breakpoints at shared boundaries
2. Shape the input            canonical schema at the edge, lossy projection in the prompt
3. Guide the reasoning        reasoning-before-verdict, disciplined enums
4. Force structured output    tool use + single-source schema + field descriptions
5. Validate → repair → retry  deterministic repair first, corrective feedback second
6. Test both contracts        input schema, output schema, mocked unit tests, replayable inputs
7. Operate the transport      timeouts with headroom, keepalive, usage metrics

Each stage has one governing idea:

1. Caching is architecture, not tuning. Order content by volatility and place cache_control breakpoints exactly where the shared prefix ends.
2. The model sees a projection, not the canonical document. Schema-validate at the edge; send a filtered, aliased, model-friendly view — and never make the model copy what it can alias.
3. The schema field order is the chain-of-thought. In structured output the model generates fields in order — put reasoning before the verdict fields.
4. Forced tool use guarantees JSON, not types. Validate every field; describe every field.
5. Repair what is mechanically unambiguous; give feedback for the rest; fail typed.
6. A failure you cannot replay is a failure you cannot fix. Persist the exact model inputs per job.
7. A long non-streaming call is a fragile TCP connection. Configure it deliberately.

1. Prompt Architecture & Caching

Order every prompt from most static to most volatile, and place cache_control: { type: 'ephemeral' } breakpoints at the boundary of what is shared across calls — which is not always the system prompt.

• The system prompt MUST contain only invariant instructions (role, field definitions, output rules) and SHOULD carry a cache breakpoint. audit-service caches its system prompt; it is identical across every audit:

  system: [{ type: 'text', text: system, cache_control: { type: 'ephemeral' } }],
  

• When a step fans out multiple calls over the same document, the shared document belongs in a cached block and only the per-call delta goes after it. enrich-service's map phase sends the same conversation text N times with a different "which events" suffix — so the breakpoint sits on the conversation, inside the user content:

  const userContent: TextBlock[] = [
    { type: 'text', text: buildConversationText(interaction), cache_control: { type: 'ephemeral' } },
    { type: 'text', text: `Analyze ONLY these events: ${requested.join(', ')}` },
  ];
  

• Every invocation MUST log cache effectiveness. All three services emit the same metric line; a broken cache is invisible without it:

  console.log(JSON.stringify({
    metric: 'bedrock_usage',
    modelId,
    input_tokens: usage?.input_tokens ?? 0,
    output_tokens: usage?.output_tokens ?? 0,
    cache_read_input_tokens: usage?.cache_read_input_tokens ?? 0,
  }));
  

2. Input Shaping

• Two contracts: the canonical document and the prompt projection. A document entering the pipeline MUST validate against its published schema (@ontopix/schemas, e.g. customer-interaction@v1.0) at the edge, on ingest — a malformed input rejected early is a cheap failure; the same document rejected inside the LLM step is an expensive one. But the canonical document is not what the model receives. The prompt carries a derived projection: deliberately lossy and model-friendly. A projection MAY drop fields, rename or restructure them, substitute delicate values (identifiers — see below), or re-render the document entirely — enrich-service renders the customer-interaction document as plain-text conversation turns (buildConversationText), not as JSON. The canonical schema governs storage and inter-service exchange; it never dictates the prompt. Projections MUST be pure, deterministic functions, so they are unit-testable and produce byte-identical output for the cache breakpoints of §1.
• Filter before you send. The most common projection: strip fields the model does not need (internal metadata, storage keys, presigned URLs). audit-service passes samples and criteria through filterSampleForPrompt / filterCriteriaForPrompt before serialization. Less input is cheaper, faster, and less distracting.
• Alias high-entropy identifiers. The model MUST NOT be asked to echo UUIDs or other long random strings — it will eventually mistype one. The projection replaces them with short aliases, and the response is translated back afterwards. enrich-service:

  // The model echoes the event id as a result key. Re-typing a 36-char UUID is
  // error-prone, so we expose short aliases (e0, e1, ...) and translate them
  // back here. The real UUID never reaches the model, which removes the
  // id-hallucination failure class at the source.
  const { aliasOf, idOf } = buildEventAliasMap(interaction);
  

• Standardize the language block. When output language is configurable, append one instruction block stating: free-text fields in the target language; field keys, enum values, and IDs in English. All three services use this exact contract.

3. Guiding the Reasoning (Chain-of-Thought)

• Reasoning before verdict — in the schema itself. In structured output the model generates fields in declaration order, so field order is the chain-of-thought. Put the reasoning field before the fields it justifies, and say so in the prompt. audit-service's criterion schema orders criteria_id, reasoning, indicators, and its prompt opens with "Reason first: analyze the sample in relation to this criterion … then evaluate indicators."
• Enum discipline. Every enum field MUST list its values with a one-line definition each, SHOULD include counter-examples for the confusable ones, and MUST state a closed-world tie-break rule. enrich-service's analyze-map.md is the exemplar:

  Strict enum constraint: Every field with enumerated values MUST use one of the listed values, spelled exactly as written. If no value seems to fit perfectly, choose the closest match from the list. Never invent, approximate, or use synonyms — an unlisted value causes a hard validation failure.

  
  and per-value disambiguation such as "urgency alone does not make a message negative — 'I need help urgently' is neutral". The absence of this rule is exactly what produced invented enum values in production (enrich-service#12).
• Co-locate instructions with the data they govern. When each item carries its own evaluation intent, attach it to the item (audit-criteria's per-indicator evaluation_prompt, which the prompt declares authoritative over the generic description) instead of accumulating special cases in the global prompt.

Schema-embedded chain-of-thought vs native thinking

Both boost accuracy through the same mechanism: verdict tokens condition on reasoning tokens generated before them. The reasoning text does not need to be read or stored for the benefit to hold — treat it as disposable unless a downstream consumer wants it. The choice is downstream of the output-integrity choice (§4), and platform-dependent — verify the current constraint before assuming either direction:

• Under forced tool choice on Bedrock, schema-embedded reasoning is the only lever. On our Bedrock invocation path, tool_choice: {type: "tool"} requires thinking disabled, so wherever output is forced (§4) the schema field is the only reasoning mechanism available. This is a platform/mode constraint, not a property of the technique: on the first-party Anthropic API (and Vertex), current models accept forced tool choice with thinking enabled. Re-evaluate this section's premise when the invocation platform or mode changes.
• Schema-embedded reasoning carries evaluation-shaped work well even where thinking is available. It has a structural advantage on multi-item evaluation: reasoning interleaves with verdicts per item (in a 19-criterion audit, each reasoning is generated immediately before the indicators it justifies), whereas native thinking happens up front, thousands of tokens before the last verdict. And when a consumer wants the rationale (audit-service persists it into the result document), the accuracy tokens were already paid for.
• Prefer native thinking for exploratory reasoning (where mode and platform allow): planning, multi-step deduction, ambiguity resolution where backtracking helps. Models are RL-trained to reason in the thinking scratchpad, so on hard problems it outperforms prompted chain-of-thought, which generates in "answer mode" — linear and polished, no self-correction. The costs: on Bedrock it means giving up forced tool choice (tool_choice: auto), reintroducing the "model answered in prose instead of calling the tool" failure mode (the §5 ladder MUST cover it), and thinking tokens are billed as output (depth steered via the effort parameter on current models).
• They compose. Where both are available, private thinking (exploration) and short per-item schema reasoning (local justification) do different jobs at different granularity — keep the reasoning field even when thinking is enabled.

4. Structured Output

Force structured output via tool use with tool_choice, and treat the returned tool_use.input as untrusted data:

• Single-source schema. One Zod schema MUST both generate the tool's input_schema and validate the response at runtime, so the two can never drift:

  export const auditTool = {
    name: AUDIT_TOOL_NAME,
    description: 'Emit the audit result. Call this exactly once…',
    input_schema: z.toJSONSchema(llmAuditOutputSchema, { target: 'draft-7' }),
  };
  // …and at runtime:
  llmAuditOutputSchema.parse(toolUse.input);
  

• Tool use guarantees the envelope, not the fields. The provider guarantees syntactically valid JSON — it does NOT guarantee each field matches its declared type. In production the model has stringified an entire array field (criteria_results as one JSON-encoded string, itself containing unescaped quotes — audit-service#72) and omitted required fields (#69). Every field MUST therefore be validated (§5); a blind as T cast is a defect.
• Describe every load-bearing field. Field-level .describe() on the schema propagates into the tool's input_schema and is the highest-leverage place to steer argument shape — far stronger than prose in the system prompt. Name the exact failure mode you are preventing. Measured on the adversarial sample of #72: first-attempt success went from 0/5 (no descriptions) to 2/3 (with descriptions):

  criteria_results: z.array(criterion).min(1).describe(
    'A native JSON array of criterion-result objects, one per input criterion. ' +
    'Emit as a real JSON array of objects — never as a single stringified JSON document.',
  ),
  reasoning: z.string().describe('Plain text, not JSON. Escape any double quotes inside the text.'),
  

• Guard truncation as a typed error. Check stop_reason === 'max_tokens' and raise a specific error (OUTPUT_TRUNCATED / TruncatedOutputError) before any parsing — a truncated tool input otherwise surfaces as an unrelated, opaque parse error downstream. Size max_tokens from observed output sizes with generous headroom.

5. The Validate → Repair → Retry Ladder

Handle invalid output in strict order of cost, and never skip a rung:

Rung 1 — Deterministic repair (free). Mechanically unambiguous defects are fixed in code, not by re-asking the model: strip unknown keys, fix a single-character alias typo when exactly one candidate matches. Repair MUST fail hard on ambiguity — a repairer that guesses is worse than a failure. enrich-service runs this as a dedicated state-machine step (repair-analysis) with stripUnknownAnalysisKeys and reconcileEventIds, which repairs an unambiguous transposition but throws on a genuine mismatch.

Rung 2 — Retry with corrective feedback (expensive, bounded). When validation fails, re-invoke the model continuing the same conversation: echo the model's invalid tool_use back as the assistant turn, then a tool_result with is_error: true whose content quotes the actual validation error plus the corrective requirements. Bound attempts (3 total in audit-service). Quoting the real error keeps the mechanism generic across failure classes (wrong types, omitted fields):

messages.push({ role: 'assistant', content: [toolUse] });
messages.push({
  role: 'user',
  content: [{
    type: 'tool_result',
    tool_use_id: toolUse.id,
    is_error: true,
    content: buildCorrectiveFeedback(validationErrorDetails),
  }],
});

A plain retry (same prompt, no feedback) MUST NOT be used for validation failures: content-deterministic failures reproduce verbatim — the #72 sample failed 4/4 identical plain re-runs, then recovered on the first corrective retry.

Rung 3 — Fail typed. After the ladder is exhausted, surface a typed, actionable error (ToolInputValidationError carrying the validator details), never a stringly Error. Orchestration layers (Step Functions) MUST retry only transport/throttle errors — application-level validation failures are re-driven inside the invocation layer where feedback is possible, or not at all.

6. Testing the Contract

• Input contract: documents entering the pipeline are validated against their published schema (@ontopix/schemas) at the edge, with unit tests over the validator. Prompt projection functions (§2) are unit-tested as pure functions: canonical document in, expected projection out.
• Output contract: unit tests prove the validator rejects each known failure shape (stringified array, omitted field, unknown enum value) and that the retry ladder re-invokes on failure and succeeds when a later attempt returns valid output — with the Bedrock client mocked. Tests MUST NOT make real model calls.
• Schema-propagation test: assert that field .describe() texts actually appear in the generated tool input_schema — the steering only exists if the propagation holds.
• Persist the exact model inputs per job. Each job SHOULD store the exact documents sent to the model (e.g. {jobId}/input.json, {jobId}/criteria.json in the results bucket, as both enrich-service and audit-service do). This is what makes a production failure replayable: the #72 diagnosis and the fix's end-to-end verification were both exact replays of a failed job's stored inputs against the real model. A failure you cannot replay is a failure you cannot confidently fix.

7. Transport & Operability

• Request timeout below the Lambda timeout, with headroom, so a stalled call surfaces as an actionable client error instead of a silent platform kill. enrich-service: 870 s request timeout under a 900 s Lambda timeout.
• TCP keepalive for long non-streaming generations. A multi-minute InvokeModel sits idle at the TCP level while the model generates; NAT/firewall hops drop idle connections (socket hang up / ECONNRESET). Configure the HTTP handler explicitly:

  new NodeHttpHandler({
    requestTimeout: 870_000,
    connectionTimeout: 5_000,
    httpsAgent: new HttpsAgent({ keepAlive: true, keepAliveMsecs: 1000 }),
  });
  

• Model aliases resolved at runtime. Steps reference capability aliases (fast / medium / high), resolved through an SSM parameter with code defaults, so models can be swapped per environment without a deploy. See AWS Bedrock Inference Profiles for the ARN/billing side.
• No PII in logs. Log field keys, counts, aliases, token usage — never content values. When repair strips values, log which keys, not what they contained.

Implementation

Checklist for a new (or reviewed) LLM-invoking step:

1. System prompt contains only invariant instructions and carries cache_control; volatile content ordered last.
2. Fan-out steps place a cache breakpoint on the shared document block inside user content.
3. bedrock_usage metric (input/output/cache-read tokens) logged on every invocation.
4. Input filtered to what the model needs; documents validate against their @ontopix/schemas schema at the pipeline edge.
5. No UUIDs or high-entropy strings for the model to echo — aliased in, translated back out.
6. reasoning field precedes verdict fields in the output schema; prompt instructs reason-first.
7. Every enum: per-value definitions, counter-examples for confusables, closed-world tie-break rule.
8. Output forced via tool use; one Zod schema generates input_schema and validates at runtime.
9. Load-bearing fields carry .describe() naming the failure mode to prevent.
10. stop_reason === 'max_tokens' raises a typed truncation error before parsing.
11. Validate → repair → retry ladder: deterministic repair (hard-fail on ambiguity) → bounded corrective-feedback retry → typed error. No blind as T, no plain retries for validation failures.
12. Unit tests mock Bedrock and cover each known bad-output shape plus the retry path.
13. Exact model inputs persisted per job for replay.
14. Request timeout < Lambda timeout with headroom; keepalive on the HTTPS agent.
15. Logs carry keys and metrics, never content values.

Consequences

Applies Principles

• Evidence Over Assumptions — cache hits are measured (bedrock_usage), steering effect is measured (0/5 → 2/3), failures are replayed from persisted inputs, never guessed at.
• Security by Design — untrusted-output posture: everything the model returns is validated before use; no PII in logs.
• Simplicity First — remove failure classes at the source (aliasing, filtering) rather than accumulating downstream repairs.
• Code Quality & Maintainability — single-source schemas and typed errors keep the contract in one place.
• Long-Term Thinking — replayable inputs and measured caches keep the system diagnosable as models and prompts evolve.

Examples

• audit-service — forced tool use with single-source Zod schema (src/shared/analysis.ts), field-level .describe() steering, corrective-feedback retry loop and ToolInputValidationError (src/shared/bedrock.ts), cached system prompt, truncation guard. Issues #70 → #72 and PR #73 document the incident lineage this section distills.
• enrich-service — fan-out-aware cache breakpoints (src/shared/analysis.ts), event-id aliasing, deterministic repair step (src/repair-analysis/), truncation as TruncatedOutputError, keepalive/timeout transport config (src/shared/bedrock.ts), exemplary enum discipline (src/prompts/analyze-map.md).
• transcript-service — the profile-participants and contextualize steps consume Bedrock under the same conventions (shared bedrock.ts, prompt-per-step markdown files).

Related Patterns

• AWS Bedrock Inference Profiles — which model/ARN to invoke and how cost is attributed; this pattern governs the request built on top of it.
• Local AWS Sandbox — how sandboxes bridge real Bedrock credentials while emulating everything else.
• Service Structure — where prompts, shared invocation code, and repair steps live inside a service.

References

• Principles vs Patterns — the layer model in the main README.
• Anthropic — Prompt caching — cache breakpoint semantics and TTL.
• Anthropic — Tool use — forced tool choice and structured output.
• audit-service issues #70, #72 — the production incidents behind §4 and §5.