Intent workflow evaluations

The development harness exercises production provider adapters and deterministic intent validation with synthetic text, telemetry and disposable in-memory workspaces.

• It does not connect to your Nines server or deploy recording rules.
• The workflow suite explicitly tests reviewed writes inside its disposable workspace.

Contract checks without a model

From the repository root:

make intent-eval                 # 105 drafting checks
make intent-workflow-eval        # 70 full workflow checks
make template-eval               # 120 deterministic measurement/intent checks
make selection-eval              # 45 scripted template-selection checks

# Save private reports in new files:
go run ./integration/intent-eval --out intent-evaluation-draft.json
go run ./integration/intent-eval --suite workflow \
  --out intent-evaluation-workflow.json

# Run one workflow scenario through all five adapter profiles:
go run ./integration/intent-eval --suite workflow \
  --case availability-reviewed-apply

• The drafting, workflow and template-selection fixtures ignore configured model endpoints unless --live is supplied.
  • The deterministic template suite never calls a model.
  • They start loopback HTTP fixtures with scripted responses and run five adapter profiles: native Anthropic, native Gemini, compatible chat completions, Ollama with schema output and vLLM with JSON output. go test ./... also exercises these suites in CI.
• The drafting corpus contains 21 cases: availability, latency, written durations, missing target/window, calendar windows, conflicting targets, compound outcomes, percentiles and hostile routing instructions.
  • Adversarial responses cover invented quantities, duplicate fields, discarded clarifications, null clarification items and unauthorized query fields.
  • Agent task success/duration, LLM evaluation verdicts, conditional inference TTFT, undefined output quality and joint agent conditions extend the semantic corpus.
  • Drafting must preserve the selected scope and leave the SLO store unchanged.
• The workflow corpus contains 14 cases: availability and latency through reviewed apply; ambiguity; missing populations and metrics; sparse and zero traffic; conflicting critical user journey references; stale binding approvals; changed source routing; a lost apply response followed by readback/retry; an unauthorized provider-routing tool call; and template-based availability/latency measurements through human confirmation and reviewed apply.
  • The harness runs the actual bounded worker, simulates an explicit owner confirming an inline measurement or selecting a binding and approving a digest, and checks that the agent role cannot approve or apply.
  • It checks source, binding, intent, manifest and time-window provenance, no preparation writes, atomic apply receipts and retry behavior.
  • The source-routing fixture simulates a changed Mimir dataset fingerprint; it does not contact a real Mimir deployment.
• Scripted responses establish adapter and safety contracts. They do not establish real model quality or prove how a model responds to hostile instructions.

Template measurement evaluations

• make template-eval runs 120 deterministic checks across the 15 shipped templates.
  • Each passes through the production measurement compiler and typed intent planner with isolated synthetic telemetry.
  • Checks cover compatible evidence, absent series, wrong metric types, missing success series or histogram boundaries, event-ratio arithmetic, absent observations and zero traffic.
  • Planning and compilation must leave the store unchanged.
  • Reports use mode: deterministic and a suite digest derived from the complete catalog and probe IDs; --suite templates --live is rejected.
• This deterministic suite validates measurement contracts, not real-model template selection, evaluator accuracy, actual provider instrumentation or end-to-end task success.
  • Drafting now has 16 semantic cases eligible for explicit live runs plus five adversarial fixture-only responses.
  • The workflow fixtures additionally exercise template discovery, pinned rendering and recorded recommendation provenance through five protocol adapters.
  • Their scripted selections do not certify real model judgment.
  • Existing real-model results below belong to the earlier corpus and are not acceptance evidence for these added cases.

Template-selection acceptance

• make selection-eval runs nine scenarios through five scripted protocol profiles (45 results).
  • Use --suite template-selection --live to run the same nine scenarios against your configured model.
  • This suite starts from an explicit typed goal, isolating template choice and tool use from the drafting suite's natural-language extraction checks.
  • It uses strict synthetic telemetry that recognizes only the metric and population declared by each case.
  • It never contacts your telemetry connections, saves an SLO or deploys rules.

• Each result separates model checks (template choice, grounding, event semantics, useful clarification and safe tool arguments) from server guardrail checks (unchanged goal, no definition writes and preserved human confirmation).
  • An attempted unsafe action or invented metric fails the model checks even if the server rejects it.
  • The catalog-injection scenario also fails unless the hostile metadata actually reached the model.
  • A model unable to call tools fails; it does not receive credit for safety alone.
• Checks are deterministic assertions, not another LLM's judgment.
  • Clarification uses expected state plus case-specific keyword checks; the TTFT observation point also uses a text check.
  • These checks cannot establish the quality of arbitrary explanations, correctness of an evaluator/rubric, genuine task success, sampling coverage or production instrumentation.
  • They are a small acceptance corpus, not a comprehensive prompt-injection benchmark.

# Fixtures, one scenario across all five profiles:
go run ./integration/intent-eval --suite template-selection \
  --case api-availability

# Your installed local model, full live corpus:
INTENT_MODEL_PROVIDER=ollama \
INTENT_MODEL_MODEL=your-installed-qwen-model \
go run ./integration/intent-eval --suite template-selection --live \
  --timeout 15m --runtime-version=your-ollama-version \
  --out intent-evaluation-template-selection.json

• The same command works with the native Anthropic/Gemini or other configured provider routes below. A single-case template-selection probe adds --case api-availability. Provider charges require explicit --live; CI uses fixtures only.
• Selection reports include the suite SHA-256, built-in catalog digest, worker skill version/digest, safe model route identity, call counts and named model/guardrail checks.
  • The suite digest also covers the embedded adversarial extension bytes. acceptance.status is fixture_only, partial, passed, failed or incomplete.
  • Only a complete live run of all nine scenarios with every check passing receives passed; a successful filtered live run remains partial.
  • Check mode as well: scripted protocol success is not real-model acceptance. requiredCases counts 45 provider/scenario pairs for fixtures or nine scenarios for live runs.
  • No prompts, tool arguments, questions, raw responses or endpoint URLs enter the report.

Evaluate a configured model

• Live evaluation requires --live and explicit INTENT_MODEL_PROVIDER and INTENT_MODEL_MODEL, even when the application could otherwise infer a provider from an API key. Set credentials using the provider's existing environment variable or your secret manager. For example, after setting ANTHROPIC_API_KEY:

INTENT_MODEL_PROVIDER=anthropic \
INTENT_MODEL_MODEL=your-anthropic-model-id \
go run ./integration/intent-eval --live --out intent-evaluation-anthropic.json

INTENT_MODEL_PROVIDER=anthropic \
INTENT_MODEL_MODEL=your-anthropic-model-id \
go run ./integration/intent-eval --suite workflow --live \
  --timeout 10m --out intent-evaluation-anthropic-workflow.json

• Anthropic and Gemini use their native APIs.
  • Gemini uses INTENT_MODEL_PROVIDER=gemini and GEMINI_API_KEY; other supported providers retain their credential variables.
  • There is no mandatory provider SDK or hosted OpenAI service.

For Qwen already served by Ollama:

INTENT_MODEL_PROVIDER=ollama \
INTENT_MODEL_MODEL=your-installed-qwen-model \
go run ./integration/intent-eval --live \
  --runtime-version=your-ollama-version \
  --out intent-evaluation-ollama.json

# Probe native tool calling before running the complete workflow corpus:
INTENT_MODEL_PROVIDER=ollama \
INTENT_MODEL_MODEL=your-installed-qwen-model \
go run ./integration/intent-eval --suite workflow \
  --case availability-reviewed-apply --live \
  --runtime-version=your-ollama-version \
  --out intent-evaluation-ollama-workflow-probe.json

Ollama defaults to http://127.0.0.1:11434/v1/chat/completions. For vLLM:

INTENT_MODEL_PROVIDER=vllm \
INTENT_MODEL_MODEL=your-served-model-id \
INTENT_MODEL_OUTPUT_MODE=json \
go run ./integration/intent-eval --suite workflow --live \
  --runtime-version=your-vllm-version \
  --out intent-evaluation-vllm.json

• vLLM defaults to http://127.0.0.1:8000/v1/chat/completions.
  • Use schema output if supported by your runtime.
  • Full workflows require native tool calling in addition to structured drafting.
  • A model that can emit draft JSON may still be unsuitable for agentic workflows.
  • The harness does not download models, start runtimes or discover versions.
• For another compatible endpoint, use INTENT_MODEL_PROVIDER=custom, an explicit model, INTENT_MODEL_ENDPOINT ending in /chat/completions, and optionally INTENT_MODEL_API_KEY_ENV naming your existing key variable. INTENT_MODEL_API_STYLE selects the supported protocol and INTENT_MODEL_OUTPUT_MODE selects schema or json. Non-loopback endpoints require HTTPS. Provider selection does not trigger fallback to another service.
• The live drafting suite runs sixteen semantic cases and excludes five scripted-response cases.
  • The live workflow suite runs thirteen cases and excludes the scripted unauthorized-tool response. --case selects one workflow or template-selection case and is rejected for the drafting suite.
  • Requests for separate cases start at least five seconds apart; production request timeouts and tool/token budgets still apply.
  • The default total timeout is five minutes, configurable up to fifteen minutes. Hosted calls may incur charges.
  • Cancellation stops subsequent calls; requests already accepted by a provider may still be billed.

Evidence, progress and recovery

• The authoring UI shows the selected source, metric, SLO-owned measurement digest (or explicit legacy binding revision), observation point, measurement population, success condition, discovery interval and assumptions.
  • Expand the fingerprint details to inspect source, discovery and binding identities.
  • Backtest evidence includes its sampled interval, resolution and definition provenance.
  • Source changes during compilation or backtesting invalidate the result.
  • A changed source fingerprint also invalidates cached measurement health until it is rechecked.
• Provider stream progress is persisted as a phase and bounded byte count under the worker lease.
  • The browser reads it through the existing conversation polling endpoint.
  • Raw model tokens, reasoning and private tool correlations are not exposed.
  • Progress updates do not change the conversation revision used for reviewed actions.
  • Cancellation fences late progress, and persisted progress survives a repository restart.

Run persistent acceptance tests with a disposable PostgreSQL database:

# Set TEST_DATABASE_URL privately before running these commands.
make test-integration

• This target rejects skipped required database tests.
  • Memory-backed contract results alone do not prove restart recovery.
  • Existing tests exercise repository reopen, lease recovery, cancellation, and recovery when resources commit but the conversation receipt update fails.
• Known configured model keys are rejected if echoed into drafting responses, tool calls or conversation inputs.
  • This is a narrow guard, not general secret detection or anonymization of arbitrary user text.
  • Provider error codes are allowlisted; raw provider errors do not enter the public transcript.
• Atomic transaction failure rolls back that transaction.
  • Existing definitions can be restored through an explicit, separately approved configuration rollback, with current revision/digest checks and an immutable transaction receipt.
  • Newly created resources have no previous definition and show a blocked rollback; resource retirement is separate and not implemented.
  • External rules follow the owner-configured automatic or manual source policy after the restored definition commits.
  • There is no automatic error-budget-triggered rollback, and restoring configuration cannot undo consumed error budget. See recommendation review, audit and rollback.

Reports and interpretation

• Reports use apiVersion: nines.dev/v1alpha1 and kind: IntentEvaluationReport.
  • They contain suite name and SHA-256, mode, timestamp, completion state, case/provider/model identifiers, observed statuses, fixed failure codes and elapsed times.
  • Workflow failures also include a safe workflowError code when available. --runtime-version is optional operator-supplied metadata.
  • Keep credentials and private text out of identifiers, runtime versions and paths.
• Raw responses, clarification text, endpoint URLs and credential values are omitted. Root-level intent-evaluation*.json reports are Git-ignored.
  • Output files are created with mode 0600 before requests begin; existing files and symlinks are refused.
  • Invalid configuration or cancellation can leave an incomplete report.
  • Exit status is nonzero for incomplete runs, failed checks or report-write errors.
  • Inspect complete, failed, selectedCase and skipped together.
  • A single-case pass is not full-suite acceptance.
  • A selection containing only skipped scripted cases returns a nonzero exit status.
• Record git rev-parse HEAD, whether the tree is modified, exact model/version, runtime, protocol/output settings and suite digest alongside each report.
  • A moving model alias is not a reproducible model version. Repeat live runs to assess variability.
  • The harness does not yet measure token usage, cost or performance distributions.
  • A passing run applies only to its exact model, serving configuration and corpus; repeat runs before enabling a model for users.

Template-selection findings, 25 October 2026

• An isolated local Ollama 0.40.11 runtime on Apple M1 ran the full nine-case template-selection corpus with qwen3:0.6b, model digest 7df6b6e09427a769808717c0a93cadc4ae99ed4eb8bf5ca557c90846becea435.
  • The runtime used an 8192-token context and the unchanged production worker limits (1024 output tokens per turn, six model calls and six tool calls per run). No hosted provider credentials were used.
  • The temporary runtime and model files were removed after testing; the existing playground was untouched.

• These results reject this exact small-model/runtime configuration for agentic template selection.
  • They do not certify another model or show resistance to the catalog injection: that case never reached catalog discovery.
  • Passing guardrails does not compensate for failed model decisions.
  • Native Anthropic, Gemini, larger local models and other serving configurations remain unverified by real-model runs for this corpus.
  • Configure a suitable model and run the full acceptance command above before enabling model-assisted authoring; retain human confirmation and reviewed apply regardless of the result.

Local model findings, 22 October 2026

• An isolated Ollama 0.40.8 runtime on Apple M1 was tested with qwen3:0.6b, model digest 7df6b6e09427a769808717c0a93cadc4ae99ed4eb8bf5ca557c90846becea435. These were synthetic runs against a modified working tree based on e9ed9efcafdbb7c71842e484496f151eaea619e3; they are not a release certification.

• The small Qwen model did not return usable native tool calls under this contract.
  • These failures remain failures; assertions were not relaxed to count clarification or missing tools as successful authoring.
  • No model-created SLO was applied by these failed workflow runs.
  • Results do not establish that larger Qwen models, another serving configuration or another provider behave the same way.
• Real Anthropic evaluation remains unverified because no credentials were available.
  • Fixture coverage for Anthropic is not a substitute for that run.
  • The intent hardening issue is closed for the current alpha scope with this caveat. Real-provider acceptance is tracked separately in #34; closing the hardening issue does not certify model quality or complete its original provider acceptance criterion. Intent workflows remain experimental.

System One assessments

• Jev is a separate typed-judgment adapter, not a drafting/tool-calling provider. See System One advice for tenant configuration, per-run consent, the synthetic live acceptance command and its calibration limits.

Local generative probe, 29 October 2026

A bounded synthetic drafting probe used cached Qwen3-0.6B through a temporary, unconstrained CPU Transformers adapter. Six cases passed, nine failed and the final case was cancelled at the 90-second limit; the run was incomplete. The separate template-selection probe could not exercise the model because that adapter did not implement tool calling. This is a result for that exact small model and experimental adapter, not an assessment of Qwen generally or acceptance of a supported production provider. No customer data was used.

Focused draft diagnostics

Use a single synthetic live case to investigate one drafting failure without running the entire corpus:

INTENT_MODEL_PROVIDER=ollama \
INTENT_MODEL_MODEL=qwen3:4b \
go run ./integration/intent-eval --suite draft --live --case availability \
  --out intent-draft-case.json

Set the provider, model, endpoint and credentials for your own environment; the command is illustrative. Only non-fixture-only case IDs from the synthetic draft corpus are accepted. The report identifies selectedCase; a passing filtered probe does not establish full-suite acceptance. Unknown and fixture-only case IDs are rejected before any provider call.

JSON output mode sends the output schema without appending an unrelated missing-target example. Exact numeric phrases, ambiguity checks, tenant scope and review gates remain enforced by deterministic code. Provider outputs are not retained in the normal report.

On 29 October 2026, after removing the extra JSON-mode example, a complete synthetic draft run against the same cached Qwen3-0.6B and unconstrained CPU adapter passed 9 of 16 live cases; five adversarial fixture-only cases were skipped as designed. Seven cases still failed. In focused before/after probes, availability and latency changed from clarification to draft, and missing target changed from unavailable to clarification. These results do not establish full model acceptance or tool-calling support. The model sometimes rewrote exact quantities or emitted a response outside the required schema; Nines continued to reject those outputs rather than silently changing the SLO.

A follow-up added support for one complete JSON Markdown fence in portable JSON mode. The fenced content still goes through strict JSON decoding, duplicate and unknown field rejection, literal quantity checks and human review. A captured synthetic joint-condition response was replayed through the adapter and service and remained a clarification. This was a deterministic replay, not a new live-model acceptance run; the preceding 9-of-16 result is unchanged.

Diagnosing draft failures

When a draft case has an unexpected status, the synthetic evaluation report includes a bounded draftDiagnostic object. It records whether the model call returned or was unavailable, its allowlisted outcome, whether the model requested clarification, and whether each extracted quantity appears literally in the synthetic input. Fixed validation-reason codes identify which deterministic checks requested clarification. They are diagnostic evidence, not proof that a literal phrase is semantically correct.

The report does not include input text, extracted values, clarification text, provider responses, endpoint URLs or credentials. unavailable combines provider transport failures and responses rejected by the adapter; it does not identify the cause. Diagnostics never change case pass criteria or authorize a change.