Agent, LLM and inference SLOs

Use the same event-weighted SLO engine for AI workloads as for conventional services.

Start with the SLI menu and templates, then connect the measurement to the relevant critical user journey.
A fast inference call or successful tool attempt does not establish that a user's task succeeded.
Try the AI workload playground to see real exported metrics, template-generated SLOs and automatic recording rules with controlled synthetic failures. No model account is required.

Choose the observation point

Objective	Eligible population	Good events	Important limit
Agent task completion	One terminal outcome per logical task	Declared user outcome achieved	Include exhausted retries, failures, cancellations and timeouts
Tool-call success	Every eligible attempt	Tool's domain contract satisfied	Attempts and tasks have different denominators
Evaluation pass rate	Selected evaluation cohort with terminal verdicts	Versioned rubric passes	Missing evaluations need a separate completion-coverage signal
Budget compliance	One budget verdict per logical task	Declared cost/token/step limits met	Instrument accounting explicitly; unknown cost cannot pass
LLM response success	Logical generation requests	Usable response under the application contract	Specify refusals, truncation, retries and cancellation
Inference availability	Serving requests at a declared boundary	Successful result	Observe rejected admission and failed requests at that boundary
Inference latency	Requests observed in the duration histogram	Duration at most the threshold	Account for requests that never complete separately
Time to first token	Streams reaching their first token	First-token delay at most the threshold	Conditional latency; pair with request availability

Use type: availability for any classified event-success ratio, including quality verdicts. The field is the engine's event-ratio measurement type; it does not assert that model quality equals infrastructure availability.

Outcome evaluation example

The complete agent evaluation example uses an application-owned counter:

measurement:
  type: availability
  metric: agent_evaluations_total
  labels:
    service: support-agent
    environment: production
    rubric: support-v1
    evaluator: rules-v1
  successLabels:
    verdict: pass
  countFrom: counter
  scrapeIntervalSeconds: 15
  observationPoint: Terminal verdict for every task in the selected evaluation cohort; unknown verdicts stay in the population. Evaluation completion coverage is measured separately.

Instrument each selected task once with verdict=pass, fail or unknown.
- For 97 passes, two failures and one unknown, attainment is 97 / 100 = 97%.
- The denominator is every matching verdict, not just passes.
- This intentionally counts an emitted unknown as not passing the declared evaluation contract; it does not claim the underlying answer is incorrect.
- If no verdict was emitted, the counter cannot see that task.
- Track selected versus completed evaluations separately, and treat missing cohort coverage as insufficient evidence.
- Zero observed traffic yields no attainment value.
Pin the rubric, evaluator and sampling policy.
- Prefer reproducible checks where they represent the intended outcome.
- Model judges require calibration against reviewed examples and checks for bias and drift; a judge's score is not independent proof of correctness.
- Do not let the proposing agent rewrite the rubric, exclude failures, select only favorable evaluations, approve itself or apply its own recommendation.
Bound labels such as service, environment, model family and evaluator version.
- Keep prompts, answers, user IDs, task IDs and unbounded tool arguments out of metric labels.
- Retain sensitive evaluation artifacts in your separately controlled evidence system, not the SLO manifest.

Latency and instrumentation

Inference time-to-first-token YAML illustrates a classic histogram family in seconds.
- Nines compiles its exact threshold bucket as good events and its count series as total events.
- The threshold must exist for every selected histogram population, alongside +Inf and _count; missing boundaries fail verification.
- Percentile gauges, summaries and native histograms are not accepted by this measurement compiler. See latency SLOs and the Prometheus histogram guidance.
Templates deliberately use illustrative metric names. OpenTelemetry GenAI conventions describe GenAI signals, but an instrument name does not guarantee the metric names, units or labels exported into your Prometheus/Mimir dataset.
- Inspect your actual telemetry connection and confirm its schema.
- Nines does not automatically instrument your agent or invoke a model judge.

Review and operate

Verify the measurement, backtest available history, inspect freshness, traffic and cohort coverage, then review the exact SLO change.
- A template's target is not a production recommendation.
- For committed SLOs, add burn-rate policies with suitable event minimums and an owner error-budget policy.
Agent suggestions use the existing approval, audit, observation and rollback workflow. Template rendering adds no write authority. SLO updates remain revision-fenced; rollback restores reviewed configuration, never consumed error budget.
Two evaluation activities remain distinct: testing Nines' drafting/preparation agent, and measuring your application's agent outcomes. The evaluation harness reports deterministic measurement checks separately from adapter fixtures and live model runs. A green fixture suite does not establish live-model quality or production AI safety.