Latency SLOs

A latency SLO counts requests that completed within a duration threshold. The example below means 99.9% of the selected API requests should take at most 300 ms over a rolling 28 days.

Define the objective

• The metric name is the histogram's base name, without _bucket, _count or _sum. The source must expose a classic histogram in seconds, with the exact threshold bucket for every matching population.

apiVersion: nines.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: request-latency
  project: production
spec:
  displayName: API request latency
  service: api
  source: primary
  target: 0.999
  windowSeconds: 2419200 # Rolling 28 days
  stepSeconds: 60
  measurement:
    type: latency
    metric: http_request_duration_seconds
    labels:
      service: api
      environment: production
    thresholdSeconds: 0.3
    scrapeIntervalSeconds: 15
    observationPoint: Completed API request duration at the server. Excludes requests never reaching this server.
  policies:
    - name: fast-burn
      severity: critical
      shortWindow: 5m
      longWindow: 1h
      burnRate: 14.4
      forSeconds: 120
      minCoverage: 1
    - name: sustained-burn
      severity: warning
      shortWindow: 30m
      longWindow: 6h
      burnRate: 6
      forSeconds: 300
      minCoverage: 1

• Download the complete latency SLO.

How the queries are derived

• measurement deterministically supplies the good and total populations. Explicit goodQuery and totalQuery are unnecessary and cannot be combined with this inline contract.

• The bucket is cumulative: le="0.3" counts requests with durations less than or equal to 0.3 seconds. Nines selects an existing bucket; it does not configure instrumentation or create bucket boundaries.

For each evaluation window, the basic calculation is:

sum(increase(http_request_duration_seconds_bucket{
  service="api", environment="production", le="0.3"
}[1h]))
/
sum(increase(http_request_duration_seconds_count{
  service="api", environment="production"
}[1h]))

• This illustrates the one-hour window.
  • Generated rules additionally align the completed cutoff, validate the populations and require nonzero traffic.
  • Increases are calculated per series before summing, so counter resets and instance identity are retained.
  • Nines evaluates each long and short window independently; it does not average overlapping window ratios into the 28-day result.
• stepSeconds: 60 controls evaluation resolution. It does not replace the rolling 28-day measurement window or the configured burn-alert windows.

Read the recording metrics

• For example, an observed SLI of 99.8% against a 99.9% target produces a 2x burn rate.
  • Filter recordings by tenant, source, SLO, revision and window.
  • Query results through the provider's query API, not its /metrics instrumentation endpoint.

Where p95 and p99 fit

• A percentile asks which duration covers a fraction of requests.
  • This SLO asks which fraction completed within a fixed duration.
  • Nines does not automatically create p95 or p99 recordings for this definition.

A supplementary diagnostic p99 query for a classic histogram is:

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{
    service="api", environment="production"
  }[5m]))
)

• That estimates a duration in seconds. Do not substitute it for the good-event count or average per-instance percentiles to calculate the error budget. See Prometheus histogram guidance.

Instrumentation and coverage requirements

• The current verifier requires _count, the +Inf bucket and the exact le="0.3" boundary for every matching population.
  • A missing boundary is rejected rather than replaced with a nearby bucket.
  • Native histograms and summary quantiles are currently unsupported by this measurement compiler.
• The selected population includes every completed request with the declared labels, regardless of response status.
  • If success also requires a particular status, model that requirement explicitly through an appropriate measurement/query contract.
  • Review the observation point, retries and requests that never reach this server.
• See measurements, rule deployment and PromQL validation.
• Integer histogram thresholds match the two equivalent canonical boundary labels, such as le="2" and le="2.0", through a generated bounded matcher.
  • This supports legacy text exposition and Prometheus normalization without interpolation.
  • Verification requires exactly one matching threshold bucket, _count and +Inf per observed population; duplicate equivalent buckets are rejected. Other population labels retain equality matchers.