Service Structure

Pattern defining the canonical shape of an Ontopix backend service: repository layout, API contract, internal architecture, configuration, observability, local sandbox, testing, and deployment.

The pattern is descriptive in spirit and lightly prescriptive in detail: it captures the canonical shape every service should take and resolves the small choices where services would otherwise drift, so that any service built against it starts aligned and any service measured against it has a clear target to converge toward.

When This Pattern Applies

This pattern applies to backend services that:

• Run on AWS Lambda (Node.js 22 or newer, TypeScript, ESM).
• Expose an HTTP API through API Gateway v2 with AWS SigV4 authentication.
• Process work asynchronously with a job-create / job-status / callback contract.
• Live in a dedicated repository (one service = one repo).

It does NOT apply to:

• Synchronous request/response APIs.
• Frontend or library repositories — see repository-structure.md for the generic minimum.
• The central platform infra repository.

If you create a service that does not fit this pattern, surface the deviation in AGENTS.md per agents-entrypoint.md and propose a sibling pattern.

Services that already adopt or are inspired by this pattern are listed in the Adoption section at the end of this document.

1. Repository Layout

my-service/
├─ src/
│  ├─ router/                    # API Gateway handler
│  │  ├─ index.ts                # HTTP routing + handler entry point
│  │  └─ validate.ts             # Zod schemas for request validation
│  ├─ <step>/                    # One directory per Lambda function
│  │  ├─ index.ts                # Lambda handler
│  │  └─ prompts/                # (optional) Bundled .md prompts
│  ├─ shared/                    # Cross-Lambda utilities
│  │  ├─ types.ts
│  │  ├─ dynamo.ts               # JobStore
│  │  ├─ s3.ts                   # ResultStore
│  │  ├─ bedrock.ts              # Bedrock client + model resolution
│  │  ├─ callback.ts             # Webhook dispatch
│  │  ├─ logger.ts               # Structured JSON logging
│  │  ├─ env.ts                  # requireEnv / optionalEnv helpers
│  │  ├─ errors.ts               # ServiceError + ErrorCodes
│  │  └─ aws-config.ts           # LocalStack-aware SDK config
├─ tests/
│  ├─ unit/                      # Mirrors src/ structure
│  ├─ integration/               # Optional, against real AWS
│  ├─ e2e/                       # Against local sandbox
│  ├─ fixtures/                  # Static input/output samples
│  └─ helpers/
│     └─ env.setup.ts            # Shared test environment setup
├─ .sandbox/
│  ├─ docker-compose.yml         # LocalStack
│  ├─ setup.sh                   # Idempotent resource creation
│  ├─ orchestrator.ts            # Local Step Functions replacement
│  └─ .env.example
├─ .infra/                       # Terraform — see infrastructure-layout.md
├─ .github/workflows/            # Thin wrappers — see github-actions.md
├─ AGENTS.md
├─ README.md
├─ Taskfile.yaml
├─ CODEOWNERS
├─ .editorconfig
├─ package.json
├─ pnpm-lock.yaml
└─ tsconfig.json

The minimum repository contract from repository-structure.md MUST be honored. The directories listed above are additions specific to services.

2. API Contract

Services expose a small, predictable HTTP surface. The contract below is mandatory for new services and is the canonical version against which existing services SHOULD be aligned.

2.1 Transport & Authentication

• Protocol: HTTP (API Gateway v2, HTTP API — not REST API).
• Authentication: AWS SigV4 (AWS_IAM authorizer) on all endpoints except /v1/health.
• Region: eu-central-1 (GDPR — non-negotiable, see bedrock-inference-profiles.md).

2.2 Versioning

All routes MUST be prefixed with a version segment:

/v1/<resource>

Breaking changes MUST be introduced under a new version prefix (/v2/...). Versioning is reflected in Terraform via an api_version variable.

2.3 Standard Endpoints

Every service MUST expose at least:

The <resources> segment is the plural noun for the service's primary entity (e.g. transcripts, enrichments, audits).

2.4 Job Creation Request

{
  "idempotencyKey": "string",      // ≤ 256 chars, REQUIRED, deduplication key
  "callback": {
    "url": "https://...",          // HTTPS only, public hostnames only in prod
    "events": ["completed", "failed"], // Subset of supported events
    "secret": "string"             // Optional, enables HMAC-SHA256 signing
  },
  "metadata": { },                 // Echoed back; ≤ 64 keys, ≤ 4096 bytes per value
  "payload": {
    "input": { },                  // Service-specific input (typically S3 URLs)
    "config": { }                  // Service-specific configuration
  }
}

Standard limits (Zod-enforced):

• idempotencyKey: 1–256 characters.
• metadata: max 64 keys, key length max 256 chars, each value serialized max 4096 bytes.
• All URLs (callback.url, payload URLs): HTTPS only in production/pre/dev. The validation MUST be relaxed when ENVIRONMENT=local to allow sandbox testing.
• All URLs MUST reject private/reserved IP ranges (anti-SSRF) — see §7.3.

2.5 Job Creation Response

{ "jobId": "uuid-v4" }

Returned with HTTP 202 Accepted. If a request with the same idempotencyKey has already been accepted, the same jobId MUST be returned (also 202).

2.6 Job Status Response

{
  "jobId": "uuid-v4",
  "status": "pending | processing | <intermediate> | completed | failed",
  "metadata": { },                 // Echoed from creation
  "result": { },                   // Present iff status == "completed"
  "error": {                       // Present iff status == "failed"
    "code": "string",
    "message": "string"
  },
  "timestamps": {                  // See §2.7
    "created_at":   "ISO 8601",
    "started_at":   "ISO 8601",
    "updated_at":   "ISO 8601",
    "completed_at": "ISO 8601",
    "failed_at":    "ISO 8601",
    "transitions": { }
  },
  "consumption": [                 // Optional, present once the job has consumed
    {                              //   billable resources. One entry per
      "task_name": "string",       //   resource consumed (e.g. LLM tokens,
      "provider": "string",        //   transcription minutes, third-party
      "service": "string",         //   API calls).
      "variant": "string",         //
      "count": 0,                  //
      "unit": "string"             //
    }
  ]
}

The result schema is service-specific and SHOULD be a typed contract from @ontopix/schemas where one exists.

2.7 Timestamps

Every job status response MUST include a timestamps object that lets clients answer "when was this submitted, when did it start, when did it finish, when did it last move?" without having to poll repeatedly.

"timestamps": {
  "created_at":   "2026-04-30T10:00:00Z",  // mandatory — job accepted by router (POST received)
  "started_at":   "2026-04-30T10:00:02Z",  // mandatory once processing begins
  "updated_at":   "2026-04-30T10:00:45Z",  // mandatory — last status transition
  "completed_at": "2026-04-30T10:00:45Z",  // present iff status == "completed"
  "failed_at":    null,                    // present iff status == "failed"
  "transitions": {                         // optional — full lifecycle trace,
    "pending":      "2026-04-30T10:00:00Z", //   one entry per status the job has
    "processing":   "2026-04-30T10:00:02Z", //   passed through.
    "transcribed":  "2026-04-30T10:00:30Z",
    "completed":    "2026-04-30T10:00:45Z"
  }
}

Rules:

• All timestamps MUST be ISO 8601 in UTC with the Z suffix.
• created_at is always present (it is the moment of the POST).
• started_at is present once the job has transitioned out of pending. While the job is still pending it MAY be omitted or null.
• updated_at is always present and reflects the most recent status transition — clients use it to detect stalled jobs.
• completed_at and failed_at are mutually exclusive and present only in their respective terminal states.
• transitions is optional and recommended for services with multi-step pipelines, where it gives the client (and on-call) a precise per-stage timeline. Each key is a status name; each value is the timestamp the job entered that status.
• When transitions is provided, it MUST be consistent with the top-level fields (created_at == transitions.pending, etc.).

2.8 Consumption Reporting

Services that consume billable resources (LLM tokens, transcription minutes, third-party API calls, etc.) MUST report what they used in the consumption field of the status response. Each entry in the array is a record aligned with the records[] shape of the ConsumptionReport schema (@ontopix/schemas, consumption-report/v1.0-beta1).

Minimum fields per entry (mirroring the schema):

Services SHOULD also forward consumption to the platform Ledger out-of-band, attributed by workspaceId (typically passed in metadata). The consumption field in the status response is the client-facing surface of the same data; Ledger is the system-of-record for billing and cost attribution. The two MUST stay consistent.

2.9 Validation

• Library: Zod for all request validation.
• Layout: schemas live in src/router/validate.ts.
• Output validation: where an output schema exists in @ontopix/schemas, the service MUST validate before returning/storing — fail loudly if the contract is violated.

2.10 Error Responses

All errors return JSON:

{ "code": "ERROR_CODE", "message": "Human-readable message" }

See §9 for the standard error code set and HTTP status mapping.

3. Internal Architecture

3.1 Lambda-per-Function

Each step in the service's pipeline is a dedicated Lambda function with its own directory under src/. There is no shared monolith handler. Typical Lambdas:

• router/ — API entry point. Creates jobs, reads jobs, starts orchestration.
• One per pipeline step (e.g. transcribe, analyze, audit, compute-metrics, …).
• finalize/ — Builds the final result, persists it, sends the success callback.
• handle-error/ — Centralized failure path: marks job FAILED, sends failure callback.

This is not the absence of layering — it is the rejection of the classic controllers/services/repositories/ monolith. Inside each Lambda there are still implicit layers:

1. Handler (index.ts) — validates input, orchestrates.
2. Shared services (from src/shared/) — JobStore, ResultStore, BedrockClient, etc.
3. AWS SDK clients — instantiated through aws-config.ts.

3.2 Canonical src/shared/ Modules

The following modules are canonical. A service MUST NOT have multiple competing implementations of these concerns.

3.3 Dependency Injection

Stores and clients use manual constructor-based DI with optional injection for tests:

export class JobStore {
  constructor(
    private readonly tableName: string,
    private readonly client: DynamoDBDocumentClient = makeDynamoClient(),
  ) {}
}

// Factory for normal Lambda boot
export function makeJobStore(): JobStore {
  return new JobStore(requireEnv("DYNAMODB_TABLE"));
}

This keeps Lambdas free of frameworks while remaining trivially testable via aws-sdk-client-mock or hand-rolled stubs.

4. Job State (DynamoDB)

4.1 Table

One DynamoDB table per service per environment per API version:

• Name: {service}-{environment}-{api_version}-jobs (e.g. my-service-prod-v1-jobs).
• Partition key: jobId (string).
• GSI: byIdempotencyKey on idempotencyKey (string).
• TTL attribute: ttl (epoch seconds).
• Billing: PAY_PER_REQUEST.
• Encryption: managed (AES256).
• PITR: enabled in pre/prod.

4.2 JobStore Operations

The JobStore class is the only path to the table. It MUST expose at least:

• getById(jobId) / getByIdempotencyKey(key)
• create(record) — fails if jobId exists
• setProcessing(jobId) / setStatus(jobId, status, ...) — uses ConditionExpression for optimistic locking against the previous expected status
• setCompleted(jobId, resultRef) / setFailed(jobId, error)

All status transitions MUST use ConditionExpression to prevent race conditions across concurrent Lambda executions.

4.3 Status Lifecycle

The canonical lifecycle is:

pending → processing → <service-specific intermediate> → completed
                                                       ↘ failed

• pending: row created by router, orchestration not yet started.
• processing: orchestration running.
• Intermediate states (e.g. awaiting_analysis, analyzed) are service-specific and SHOULD be lowercase snake/kebab consistent within the service.
• completed / failed are terminal.

4.4 TTL

Differentiated TTL by terminal state:

• completed: 30 days.
• failed: 7 days.

Set the ttl attribute when transitioning to a terminal state.

5. Artifact Storage (S3)

5.1 Bucket

One bucket per service per environment per API version:

• Name: {service}-{environment}-{api_version}-results.
• Lifecycle: matches DynamoDB TTL — expire artifacts after 30 days. Service-specific prefixes (e.g. batch-input/, batch-output/) MAY have shorter expirations.
• Public access: blocked.
• Encryption: AES256.

5.2 Key Conventions

{jobId}/input.json      # Optional, captured input
{jobId}/result.json     # Final result (CustomerInteraction@vX or service-specific)
{jobId}/<step>.json     # Intermediate per-step artifacts

Access goes through the ResultStore class — Lambdas MUST NOT instantiate S3Client directly.

6. Configuration

6.1 Environment Variables (Boot-time)

Required at module load via requireEnv():

• DYNAMODB_TABLE, S3_BUCKET
• STATE_MACHINE_ARN (router only)
• Service-specific secrets references (e.g. ElevenLabs API key ARN)

Optional via optionalEnv():

• ENVIRONMENT — local enables sandbox-specific code paths (e.g. URL validation relaxation).
• AWS_REGION, BEDROCK_REGION.
• DEBUG_PAYLOADS — emits raw request/response bodies in logs.
• LOCALSTACK_ENDPOINT / AWS_ENDPOINT_URL — switches aws-config.ts to LocalStack mode.

6.2 SSM Parameter Store (Runtime)

Configuration that varies by environment but is not a secret SHOULD live in SSM under:

/{service}/{environment}/...

Examples:

• /{service}/{env}/models/{mode}/{alias} — Bedrock model IDs.
• /{service}/{env}/callback/{max_attempts,base_delay_ms,max_delay_ms} — retry policy.
• /{service}/{env}/batch/{poll_interval_seconds,timeout_hours} — batch tuning.

Resolution MUST fall back to a hardcoded default if the SSM parameter is missing, so a fresh deployment is bootable without out-of-band setup.

6.3 Secrets

Secrets MUST live in AWS Secrets Manager, referenced by ARN as a Lambda env var. The Lambda fetches and caches the value at boot.

7. Logging, Errors & Callbacks

7.1 Structured JSON Logging

shared/logger.ts writes one JSON object per line to stdout (CloudWatch ingests it):

{ "timestamp": "2026-04-30T12:00:00Z", "level": "info", "lambda": "router", "action": "create_job", "jobId": "..." }

• All log lines MUST be valid JSON (single line, no multi-line dumps).
• Levels: debug, info, warn, error.
• PII MUST NOT appear in logs.

7.2 Error Model

Errors flow through ServiceError (in shared/errors.ts):

throw new ServiceError(ErrorCodes.VALIDATION_ERROR, "callback.url must be HTTPS", 400);

Standard error codes:

The router MUST translate uncaught errors into INTERNAL_ERROR and never leak stack traces to clients.

7.3 Callbacks (Webhooks)

Services notify clients via HTTP POST to the callback.url provided at job creation.

Payload — at minimum:

{ "jobId": "uuid-v4", "event": "completed | failed | <intermediate>" }

Services MAY include a richer payload (e.g. embed result for completed).

Subscription — clients subscribe to a subset of events via callback.events. Beyond completed / failed, services MAY define optional intermediate progress events (e.g. transcribed, contextualized, profiled). Intermediate events are service-specific and SHOULD be documented in the service's README.

Retry policy — exponential backoff with jitter:

• Default: 5 attempts, base delay 1 s, max delay 60 s, ±10% jitter.
• Per-attempt timeout: 10 s.
• Configurable via SSM (§6.2).

Anti-SSRF — callback.ts MUST reject:

• Non-HTTPS URLs (except in local environment).
• Private IPv4 ranges: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16.
• Loopback (127.0.0.0/8, ::1), link-local (169.254.0.0/16), reserved ranges.
• Hostnames resolving exclusively to the above (resolve before sending).

Optional HMAC signing — if the client provides callback.secret, the service MUST sign the request body with HMAC-SHA256 and send the signature in the X-Webhook-Signature header.

7.3.1 Authentication: HMAC vs IAM SigV4

Two callback authentication mechanisms are available. Choose based on the caller:

IAM SigV4 for internal callbacks

When the callback receiver is an Ontopix Lambda (e.g. a Lambda Function URL or an API Gateway route with AWS_IAM auth), use IAM SigV4 signing instead of a shared secret. This leverages the existing AWS IAM trust model — no secret material to generate, rotate, or store.

Sender side (the service dispatching callbacks):

1. Sign the outbound POST using AWS SigV4 with service lambda (for Function URLs) or execute-api (for API GW routes).
2. Attach the signed headers (Authorization, X-Amz-Date, X-Amz-Security-Token) to the callback POST.
3. Grant the sender's Lambda execution role the appropriate IAM permission (see below).

Reference implementation — sigv4-fetch.ts in ontopix/app:

import { SignatureV4 } from "@smithy/signature-v4";
import { Sha256 } from "@aws-crypto/sha256-js";
import { defaultProvider } from "@aws-sdk/credential-provider-node";
import { HttpRequest } from "@smithy/protocol-http";

export async function sigv4Fetch(url: string, init?: RequestInit): Promise<Response> {
  const parsed = new URL(url);
  const body = init?.body ? String(init.body) : undefined;
  const request = new HttpRequest({
    method: (init?.method ?? "GET").toUpperCase(),
    hostname: parsed.hostname,
    path: parsed.pathname + parsed.search,
    headers: {
      host: parsed.hostname,
      ...(body ? { "content-type": "application/json" } : {}),
      ...((init?.headers as Record<string, string>) ?? {}),
    },
    body,
  });
  const signer = new SignatureV4({
    credentials: defaultProvider(),
    region: process.env.AWS_REGION ?? "eu-central-1",
    service: parsed.hostname.endsWith(".on.aws") ? "lambda" : "execute-api",
    sha256: Sha256,
  });
  const signed = await signer.sign(request);
  return fetch(url, { method: signed.method, headers: signed.headers as HeadersInit, body });
}

Receiver side:

• Lambda Function URL: set authType: "AWS_IAM" on the Function URL resource.
• API Gateway route: set authorization_type = "AWS_IAM" on the route.

Required IAM permission — grant the sender's execution role one of:

For Lambda Function URLs:

{ "Effect": "Allow", "Action": "lambda:InvokeFunctionUrl", "Resource": "arn:aws:lambda:<region>:<account>:function:<receiver>" }

For API Gateway routes:

{ "Effect": "Allow", "Action": "execute-api:Invoke", "Resource": "arn:aws:execute-api:<region>:<account>:<api-id>/*/POST/<route>" }

8. Local Sandbox

The sandbox lets developers run the entire service locally. See sandbox-environments.md (general principles) and local-aws-sandbox.md (LocalStack specifics) for the framework. This section covers the service-specific layer.

8.1 Components

.sandbox/
├─ docker-compose.yml       # Pinned LocalStack version, never `latest`
├─ setup.sh                 # Idempotent: creates DynamoDB, S3, SSM, Lambdas
├─ orchestrator.ts          # Replaces Step Functions
└─ .env.example

8.2 LocalStack Coverage

LocalStack emulates: Lambda, DynamoDB, S3, API Gateway v2, SSM, Step Functions (limited).

The setup.sh script MUST be idempotent — running it twice in a row is a no-op on the second run.

8.3 Local Orchestrator

Step Functions in LocalStack does not reliably support all features used in production (task tokens, async invoke S3 triggers, batch polling). Services MUST ship a TypeScript orchestrator at .sandbox/orchestrator.ts that:

• Polls the DynamoDB jobs table for pending jobs.
• Invokes Lambdas in the correct order via the AWS SDK (LambdaClient).
• Routes to handle-error on any failure.
• Runs in the background (nohup, PID file at .sandbox/orchestrator.pid).

Differences from production are expected and acceptable (sequential vs parallel, no retries vs Step Functions retry policy). The orchestrator's purpose is functional iteration, not behavioral fidelity.

8.4 Non-Emulable External Services

Services that cannot be emulated by LocalStack — for example AWS Bedrock, third-party LLM providers, audio/transcription APIs, payment gateways — MUST use real credentials and real endpoints even in the sandbox. The pattern:

1. Sandbox loads real AWS credentials (via AWS_PROFILE or explicit access keys) into the Lambda env.
2. aws-config.ts switches only emulated services to LocalStack endpoints; non-emulated SDK clients use the real region.
3. The cost of these calls in sandbox is the developer's responsibility.

8.5 Required Sandbox Tasks

See sandbox-environments.md for the full task contract. Service-specific additions that MUST exist:

• sandbox:make-request — submit a job (parameters via env vars: AUDIO_FILE=, JOB_ID=, etc.).
• sandbox:job — fetch job status by JOB_ID.
• sandbox:debug:artifacts — download S3 artifacts for a JOB_ID to a local directory.
• sandbox:debug:logs — tail orchestrator + LocalStack logs.

9. Testing

Three layers, all using Vitest.

9.1 Unit Tests — tests/unit/

• Mirror src/ directory structure.
• Mock AWS clients with aws-sdk-client-mock.
• Mock external HTTP with vi.mock() or nock.
• No network, no LocalStack, no real AWS.
• Setup: tests/helpers/env.setup.ts populates required env vars.
• Coverage target: handler logic + every shared/ module.

9.2 Integration Tests — tests/integration/ (optional)

• Run against real AWS in a developer or CI test account.
• Skipped when required ARNs are not configured.
• Exercise the JobStore/ResultStore against real DynamoDB/S3.
• Clean up created records on teardown.

9.3 End-to-End Tests — tests/e2e/

• Run against the local sandbox (task sandbox:start is a precondition).
• Submit a job, poll for completion, assert on the final result.
• Use static fixtures under tests/e2e/fixtures/ for inputs and expected outputs.
• For LLM-driven services where output is non-deterministic, use threshold-based scoring rather than exact-match assertions (the threshold strategy is per-service and not part of this pattern).
• Gated by env (e.g. INCLUDE_E2E=1) so CI can run unit-only by default.

9.4 Conventions

• Test files: *.test.ts, co-located in tests/<layer>/<mirror>.
• Fixtures: tests/fixtures/ (shared) or tests/e2e/fixtures/ (e2e-specific). Fixtures MUST NOT have a test_ prefix.
• Required commands (see taskfile-contract.md): test:unit, test:integration, test:e2e, test:all, test:cov.

10. Build & Deploy

10.1 Build

• Bundler: esbuild, one bundle per Lambda function.
• Output: dist/<lambda>/index.mjs (ESM, platform node, target matching the active Lambda Node.js runtime).
• External: @aws-sdk/* is bundled (do not rely on the runtime's SDK version).
• Bundled assets: prompts/*.md MUST be copied alongside index.mjs so the prompt loader can read them at runtime.
• Size guard: build MUST fail if any Lambda zip exceeds 40 MB (escalation threshold per lambda-deploy.md).

10.2 Deploy

• Default mechanism: direct zip upload via aws lambda update-function-code — see lambda-deploy.md.
• Terraform manages the function definition (memory, timeout, IAM, env vars) but ignores filename and source_code_hash.
• One placeholder zip is committed at .infra/bootstrap/placeholder.zip for first-time creation.

10.3 Infrastructure

Follows infrastructure-layout.md. Service-specific resources in .infra/:

• DynamoDB jobs table + GSI (§4.1).
• S3 results bucket (§5.1).
• Lambda functions (one per src/<step>/ directory).
• Step Functions state machine.
• API Gateway v2 + routes + integrations.
• IAM roles per Lambda (least-privilege).
• Bedrock inference profiles (where applicable, per bedrock-inference-profiles.md).

All AWS resources MUST be tagged per aws-resource-tagging.md.

11. Taskfile Interface

The Taskfile contract is defined in taskfile-contract.md. A service Taskfile MUST cover the following namespaces:

• setup:* — first-use setup (per decisions/adr-0014-setup-namespace.md).
• lint:* — check, fix, types, format, all.
• test:* — unit, integration, e2e, all, cov.
• build:* — lambda, clean, and (recommended) check-size.
• ci:* — lint, test, build, all — task ci:all MUST mirror CI exactly.
• sandbox:* — start, stop, restart, reset, status, plus the service-specific tasks in §8.5.
• infra:roles:* — init, plan, apply for the deployment-roles layer (manual, admin-only).
• infra:shared:* — init, plan, apply for the shared layer (manual, infrequent).
• infra:* — init, plan, apply, validate, fmt, destroy (with CONFIRM=yes guard) for the service layer.
• deploy:* — dev, pre, pro as appropriate.
• validate:structure — checks repository minimum files exist.

The Taskfile is the single operational interface. Workflows in .github/workflows/ are thin wrappers (per github-actions.md) that call task ci:all, task build:lambda, task infra:plan, etc.

12. CI/CD

CI/CD follows github-actions.md. Service-specific notes:

• validate job: setup → registry login → task ci:all.
• plan job (PRs only): task infra:shared:plan + task infra:plan ENV=....
• deploy workflows: separate per environment, manual dispatch for pre and prod.
• OIDC roles: CodeArtifact-read for build; Terraform-plan for plan; deploy roles for apply. No long-lived credentials.

13. New Service Checklist

When scaffolding a new service:

1. Repository minimum (per repository-structure.md): README.md, AGENTS.md, Taskfile.yaml, .editorconfig, CODEOWNERS.
2. Add src/router/, src/shared/, tests/{unit,integration,e2e}/, .sandbox/, .infra/.
3. Implement the canonical shared/ modules (§3.2).
4. Define API schemas in src/router/validate.ts per §2.
5. Create .infra/deployment-roles/ with CI/CD IAM roles per §14.2; bootstrap with admin credentials.
6. Create .infra/shared/ with DNS and ACM certificate per §14.1; apply with admin credentials.
7. Create .infra/service/ with DynamoDB, S3, Lambda, SFN, API Gateway per §4, §5, and §10.3.
8. Implement .sandbox/setup.sh (idempotent) and .sandbox/orchestrator.ts.
9. Cover Taskfile namespaces from §11, including infra:roles:* and infra:shared:*.
10. Wire CI workflows with path filtering and artifact passing per §14.3.
11. Register the service in AGENTS.md with pattern references and constraints.

14. Deployment Infrastructure

This section defines the layered infrastructure structure and CI/CD pipeline for services. The model separates concerns by lifecycle and access level, enabling automated deployments while maintaining security boundaries.

14.1 Infrastructure Layers

Services organize Terraform into three layers under .infra/:

.infra/
├── deployment-roles/     # Layer 1: CI/CD IAM roles (manual, admin-only)
│   ├── main.tf
│   ├── backend.tf
│   ├── variables.tf
│   └── terraform.tfvars
├── shared/               # Layer 2: Long-lived resources (manual, infrequent)
│   ├── dns.tf
│   ├── backend.tf
│   └── ...
├── service/              # Layer 3: Per-environment resources (CI automated)
│   ├── main.tf
│   ├── variables.tf
│   ├── dev.tfvars
│   ├── pre.tfvars
│   └── prod.tfvars
└── modules/              # Reusable modules (e.g., lambda/)

Layer 1 — Deployment Roles defines IAM roles that GitHub Actions assumes via OIDC. These roles are scoped to the service's resources and MUST NOT be deployed by CI (chicken-and-egg problem). An admin bootstraps this layer once; subsequent changes are rare.

Layer 2 — Shared contains resources that exist independently of any specific environment or API version: the custom domain, ACM certificate, and DNS records. Changes are infrequent and reviewed carefully before manual apply.

Layer 3 — Service contains all per-environment resources. CI deploys this layer automatically on push to protected branches.

14.2 Deployment Roles (CI/CD)

Each service defines two IAM roles for CI/CD, following the trust model from github-actions.md:

Trust Policy Example ({service}-ci-plan):

assume_role_policy = jsonencode({
  Version = "2012-10-17"
  Statement = [
    {
      Effect = "Allow"
      Principal = { Federated = "arn:aws:iam::${account_id}:oidc-provider/token.actions.githubusercontent.com" }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringLike   = { "token.actions.githubusercontent.com:sub" = "repo:${org}/${repo}:*" }
        StringEquals = { "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com" }
      }
    },
    {
      Effect    = "Allow"
      Principal = { AWS = "arn:aws:iam::${account_id}:root" }
      Action    = "sts:AssumeRole"
    }
  ]
})

The second statement allows developers to assume the role locally for testing.

Permission Scoping: The deploy role MUST scope write permissions to resources prefixed with {service}-*. This prevents a compromised pipeline from affecting other services. Example resource patterns:

Resource = [
  "arn:aws:lambda:*:${account_id}:function:${service}-*",
  "arn:aws:dynamodb:*:${account_id}:table/${service}-*",
  "arn:aws:s3:::${service}-*",
  "arn:aws:states:*:${account_id}:stateMachine:${service}-*",
  "arn:aws:iam::${account_id}:role/${service}-*",
]

Bootstrap Process: Deploying the roles layer requires admin credentials (not CI):

# First-time setup (admin)
cd .infra/deployment-roles
terraform init
terraform apply

After this, CI can use the roles for all subsequent operations.

14.3 CI/CD Pipeline

The CI/CD pipeline structure for services:

PR to master:
  ci.yaml → validate → plan-pre → (plan-prod parallel)
                    ↘ plan-prod ↗

Push to master:
  ci.yaml → validate → plan-pre  → deploy-pre → deploy-prod
                    ↘ plan-prod ↗

Artifact Passing: The validate job builds Lambda artifacts and uploads them via actions/upload-artifact. Subsequent jobs download these artifacts, avoiding redundant builds:

# In validate job
- uses: actions/upload-artifact@v4
  with:
    name: lambda-dist
    path: dist/
    retention-days: 1

# In plan/deploy jobs
- uses: actions/download-artifact@v4
  with:
    name: lambda-dist
    path: dist/

Path Filtering: Changes to the deployment-roles layer should NOT trigger the deploy pipeline. Use paths-ignore:

on:
  push:
    branches: [master]
    paths-ignore:
      - '.infra/deployment-roles/**'
      - 'docs/**'
      - '*.md'

Notifications: Deploy jobs SHOULD notify on success/failure. Use Slack for engineering-stream:

- name: Notify Slack (failure)
  if: failure()
  uses: slackapi/slack-github-action@v3
  with:
    method: chat.postMessage
    token: ${{ secrets.SLACK_BOT_TOKEN }}
    payload: |
      channel: "engineering-stream"
      text: "❌ {service} prod deploy failed — ${{ github.sha }}"

14.4 State Management

Each layer has its own Terraform state, stored in the shared S3 backend:

The service layer uses dynamic backend configuration via -backend-config to support multiple environments:

# Taskfile.yaml
infra:init:
  dir: .infra/service
  cmds:
    - terraform init
        -reconfigure
        -backend-config="bucket=ontopix-tfstate"
        -backend-config="key=services/{{.PROJECT}}/{{.ENV}}-{{.API_VERSION}}/terraform.tfstate"
        -backend-config="region=eu-west-1"
        -backend-config="dynamodb_table=ontopix-tflocks"

The service layer's backend.tf is intentionally empty — all values come from -backend-config:

terraform {
  backend "s3" {}
}

Cross-Layer References: The service layer MAY read outputs from the shared layer via terraform_remote_state:

data "terraform_remote_state" "shared" {
  count   = var.enable_custom_domain ? 1 : 0
  backend = "s3"
  config = {
    bucket = "ontopix-tfstate"
    key    = "services/${var.project}/shared/terraform.tfstate"
    region = "eu-west-1"
  }
}

14.5 Resource Naming Convention

All AWS resources follow this naming pattern:

{service}-{env}-{api_version}-{resource}

Examples for enrich-service in pre with API v1:

Why api_version? Including the API version in resource names enables v1/v2 coexistence during migrations. A service can run both versions in production simultaneously, each with its own DynamoDB table and S3 bucket, allowing gradual traffic shift.

The name_prefix local in Terraform centralizes this:

locals {
  name_prefix = "${var.project}-${var.environment}-${var.api_version}"
}

resource "aws_dynamodb_table" "jobs" {
  name = "${local.name_prefix}-jobs"
  # ...
}

14.6 Taskfile Infrastructure Tasks

The Taskfile MUST include tasks for all three layers. The pattern supports an optional INFRA_ROLE_ARN variable for local assume-role, enabling developers to mirror CI behavior:

vars:
  INFRA_ROLE_ARN: '{{.INFRA_ROLE_ARN | default ""}}'
  
tasks:
  infra:roles:init:
    desc: Initialize Terraform for deployment-roles layer
    dir: .infra/deployment-roles
    vars:
      AWS_CREDS:
        sh: |
          if [ -n "$INFRA_ROLE_ARN" ]; then
            aws sts assume-role --role-arn "$INFRA_ROLE_ARN" --role-session-name taskfile \
              --query "Credentials" --output json | \
              jq -r '"AWS_ACCESS_KEY_ID=\(.AccessKeyId) AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey) AWS_SESSION_TOKEN=\(.SessionToken)"'
          fi
    cmds:
      - env {{.AWS_CREDS}} terraform init

  infra:roles:plan:
    desc: Show Terraform plan for deployment-roles layer
    # ... similar pattern

  infra:roles:apply:
    desc: Apply deployment-roles changes (requires admin credentials)
    # ... similar pattern

The AWS_CREDS dynamic variable allows:

• CI: Uses OIDC credentials directly, INFRA_ROLE_ARN is empty
• Local dev: Sets INFRA_ROLE_ARN to assume a role before running Terraform

Required Tasks:

Adoption

The following services adopt or are inspired by this pattern. Entries are informational — the pattern itself remains the source of truth, and services SHOULD converge toward it where they currently diverge.

When adding a new service to this list, link the repository and note any intentional deviations.

Related Patterns

• Repository Structure — minimum repo files
• AI Agent Entrypoint — AGENTS.md requirements
• Taskfile as Contract — Taskfile namespaces and rules
• Sandbox Environments — sandbox principles
• Local AWS Sandbox — LocalStack specifics
• Infrastructure Layout — .infra/ conventions
• AWS Resource Tagging — tag taxonomy
• GitHub Actions — CI/CD wrappers
• Lambda Deploy — direct-upload deploy strategy
• Bedrock Inference Profiles — Bedrock provisioning
• Repository Context Directory — .context/

Related Decisions

• ADR-0002 — Agents entrypoint
• ADR-0003 — Taskfile contract
• ADR-0011 — Lambda deploy strategy
• ADR-0012 — Workflow architecture
• ADR-0013 — Taskfile contract v2
• ADR-0014 — Setup namespace