Cross-Cutting Layer

What the layer provides

Zod artifacts

Zod artifacts are the main contract between phases. Each phase emits a structured JSON artifact that must validate before it is written or passed forward. This gives Testonaut three properties:

• Determinism at the boundary: the next phase receives a known shape, not arbitrary model text.
• Traceability: scenarios, source evidence, links, warnings, and tool calls remain inspectable after the run.
• Versioning: each artifact carries schemaVersion, so future migrations can be handled intentionally.

Main artifacts

Why JSON plus Markdown

Each major pack is written twice: JSON is the machine-readable source of truth for the next phase and for audits; Markdown is the human-readable companion for review, pipeline artifacts, and debugging.

Audit log

The audit log records consequential actions as append-only JSONL — used for governance and review, not as input to future decisions. Default path out/audit.jsonl, overridable with TESTONAUT_AUDIT_LOG. Each line is one event:

{
  "at": "2026-06-17T12:00:00.000Z",
  "action": "pr.raised",
  "actor": "implement-tests",
  "target": "FINT-1056",
  "detail": {
    "status": "raised",
    "url": "https://...",
    "branch": "testonaut/FINT-1056"
  }
}

Reading the audit log

testonaut audit
testonaut audit --target FINT-1056
testonaut audit --action pr.raised --limit 20
testonaut audit --json

Redaction before audit

Audit entries pass through the redactor before they are appended, protecting common secret and PII shapes: OpenAI and Anthropic API keys, GitHub tokens, Slack tokens, AWS access key ids, bearer/basic auth headers, private keys, email addresses, and password / secret / token / api_key style values. Project-specific patterns can be added:

{
  "policy": {
    "redactPatterns": [
      "customer-[0-9]{6}",
      "internal-account-[A-Za-z0-9-]+"
    ]
  }
}

Invalid custom regex patterns are skipped rather than crashing the run.

Cost ledger and budgets

Testonaut tracks LLM spend in an append-only cost ledger at out/cost-ledger.jsonl (override with TESTONAUT_COST_LEDGER). Each entry records the timestamp, calendar day, measured or estimated run cost, issue key, and the CLI command that produced the cost.

How costs are calculated

Testonaut prefers exact cost when the SDK provides it. When exact cost is unavailable, it estimates from token counts and a model price table. The estimator understands uncached input tokens, cached input tokens, Anthropic cache-creation tokens, output tokens, and reasoning tokens where reported. Custom or Azure deployment names can be priced with:

TESTONAUT_MODEL_PRICES='{"my-azure-deployment":{"input":1.75,"cachedInput":0.175,"output":14}}'

Langfuse observability

Testonaut uses OpenTelemetry and OpenInference instrumentation, exported to Langfuse. Telemetry is optional; when disabled, the same workflows run without loading the OpenTelemetry stack. Minimum environment:

OTEL_ENABLED=true
OTEL_PROVIDER=langfuse
OTEL_PUBLIC_KEY=<langfuse-public-key>
OTEL_SECRET_KEY=<langfuse-secret-key>
OTEL_ENDPOINT=https://cloud.langfuse.com
OTEL_PROJECT_NAME=testonaut

If OTEL_SESSION_ID is not provided, ticket workflows use the Jira issue key as the session id — jira:<ISSUE-KEY> — grouping the Researcher, Test Design, Implementation, and Feedback spans for one ticket into one Langfuse session.

What Langfuse receives

Why Langfuse and artifacts are both needed

• Langfuse answers operational questions: how many LLM calls per ticket, which phase spent the most tokens, did cached input reduce cost, how long did each call take.
• Artifacts answer product and audit questions: which Jira comments and Confluence pages were used, which AC produced this scenario, which Xray tests were created, which files changed, which warnings were attached.

Redaction and secret handling

The layer assumes secrets may appear in tool responses, command output, environment-derived values, or reviewer comments. Before text leaves the process, Testonaut applies redaction to logs, telemetry span payloads, audit entries, written artifacts, and PR comments. MCP server configuration also supports environment-variable expansion so API tokens can live in .env or pipeline secret variables rather than in repository config.

Configuration summary

Repository config

{
  "schemaVersion": "1.0.0",
  "product": "example-product",
  "policy": {
    "gates": {
      "acCoverage": "required",
      "riskCoverage": "warn",
      "humanReviewBeforeUpload": true,
      "humanReviewBeforePr": false
    },
    "budgets": {
      "maxCostPerRunUsd": 5,
      "maxCostPerDayUsd": 25
    },
    "redactPatterns": []
  },
  "pr": {
    "mention": "@testonaut",
    "maxIterationsPerPr": 5
  }
}

This config is itself Zod-validated. Invalid JSON or invalid fields fail fast with readable validation errors.

Runtime environment

Failure and safety behavior