NinesDocumentation
Search becomes available when JavaScript loads. All chapters are listed in the navigation.
GitHub
Browse documentation
View Markdown

Immutable audit archival

Nines can deliver each tenant's decision journal to an owner-configured Amazon S3 or S3-compatible bucket with Object Lock. It is disabled by default.

  • The writer uses COMPLIANCE retention and verifies the object bytes, version, encryption and retention readback before persisting a delivery receipt.
  • A successful upload response alone does not count as verified delivery.
  • This protects the delivered object version for its retention period.
    • It does not make the Nines database immutable or guarantee that an event reaches S3 during an outage. Pending delivery remains visible.
    • Database administrators can alter local records and receipts; independently inspect the S3 versions and preserve separate access controls.

Prepare the destination

  • Provision a dedicated bucket with versioning and Object Lock enabled.
    • Choose the account, region and retention period deliberately.
    • In S3 compliance mode, protected versions cannot be overwritten or deleted during retention, and retention cannot be shortened.
    • Nines does not create buckets, change bucket policy, delete objects, enable legal holds or request governance bypass.
  • Give the Nines server an IAM role scoped to the bucket and its tenant prefix. The bucket-level permission is s3:GetBucketObjectLockConfiguration.
    • Object permissions are s3:PutObject, s3:PutObjectRetention, s3:GetObject and s3:GetObjectRetention.
    • An independent verifier may also need s3:GetObjectVersion.
    • Do not grant this writer object deletion, bucket policy administration or retention bypass.
    • With a customer KMS key, also configure its key policy and the required kms:GenerateDataKey/kms:Decrypt permissions.
  • Use separate tenant prefixes or buckets and apply corresponding IAM boundaries.
    • Nines puts the tenant ID and a destination-policy digest below the configured prefix.
    • A shared process IAM identity can access every prefix permitted to it; Nines configuration does not replace IAM isolation.

Configure the tenant

Add auditArchive to the tenant in the server's TENANTS_FILE:

yaml
tenants:
  - id: production
    name: Production
    access:
      - tokenEnv: PRODUCTION_API_TOKEN
        role: owner
    source:
      kind: prometheus
      url: http://prometheus:9090
    auditArchive:
      bucket: example-nines-audit
      region: eu-west-2
      expectedBucketOwner: "123456789012"
      prefix: nines/audit
      retentionDays: 365
      # Optional: full KMS key ARN, not an alias.
      # kmsKeyID: arn:aws:kms:eu-west-2:123456789012:key/00000000-0000-0000-0000-000000000000
  • Replace the example bucket/account with your own. retentionDays is required and accepts 1–3650.
    • Credentials are obtained by the AWS SDK's default credential provider chain, including workload roles and standard AWS environment/profile configuration.
    • Keep access keys out of tenant YAML, manifests, CLI arguments, browser forms and model prompts. Prefer a workload IAM role.
  • Archive object destinations are taken from this server-owned configuration.
    • SDK S3 endpoint environment overrides do not select another archive destination.
    • Set provider: s3-compatible with an explicit endpoint for a deliberately configured non-AWS HTTPS gateway; HTTP is accepted only for loopback acceptance fixtures.
    • Compatibility tests do not certify that another service provides Amazon S3's retention guarantees.
  • Restart the server after configuring the tenant.
    • Persistent server deployments require PostgreSQL; configured archival is rejected with production -memory mode.
    • An explicit in-memory demo can exercise the API but loses its journal, queue and receipts when the process stops.

Open-source object storage

  • External storage remains optional.
    • The built-in journal and exports work without an object store.
    • Use provider: s3-compatible for an owner-configured non-AWS S3 endpoint.
    • This is a capability-based integration, not certification of every S3 implementation.
  • Ceph RGW is the documented open-source target.
    • Its maintained Tentacle release documents S3 compatibility, Object Lock and server-side encryption.
    • Ceph requires its own cluster administration; Nines does not install it.
    • The disposable container acceptance below has passed against Ceph 20.2.4 with OpenBao 2.5.0.
    • Your production storage, permissions and key-management setup still require independent acceptance.
yaml
auditArchive:
  provider: s3-compatible
  endpoint: https://rgw.example.com
  bucket: nines-audit
  region: us-east-1
  prefix: nines/audit
  retentionDays: 365
  • Set the region to the value configured for your gateway.
    • Use the HTTPS S3 API endpoint with a trusted certificate. Explicit endpoints use path-style bucket addressing.
    • HTTP is allowed only on loopback for acceptance fixtures.
    • Loopback in a container refers to that container. Do not disable TLS verification.
  • Omit expectedBucketOwner and kmsKeyID in this mode; these fields select AWS account ownership and an AWS KMS key.
    • The default s3 provider continues to require the 12-digit expected owner.
    • Configure SSE-S3 on the gateway itself, including its key-management backend.
    • Nines requests and verifies AES256; missing encryption does not produce a verified receipt.
  • Provision the bucket with versioning and Object Lock before configuring delivery.
    • Supply a dedicated gateway access key to the Nines server through AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY using your deployment's secret mechanism.
    • These names belong to the S3 credential protocol; the explicit endpoint selects your gateway.
    • Keep secrets out of tenant YAML, URLs, browser forms and model inputs.
    • Limit the credentials to the bucket/prefix and required S3 operations above.
    • A process-wide credential chain is shared by configured tenants; use separate deployments if independent storage credentials are required.
  • Before production use, run the provider acceptance check below with an isolated bucket/prefix and retentionDays: 1.
    • The gateway must enforce conditional object creation, checksum validation, versioning, encryption and COMPLIANCE retention. An S3-compatible label alone is insufficient.
    • No verification requirement is silently relaxed for non-AWS storage.
  • Object Lock protects versions through the storage API.
    • Storage administrators with disk, encryption-key or backup access remain outside that boundary.
    • Keep storage administration separate and retain independent backups.

Inspect delivery

Open Audit log → External archive or run:

sh
nines audit status --context production --agent
nines audit status --record RECORD_UUID --context production --agent
  • The same evidence is available through GET /api/audit/archive and GET /api/audit/archive?record=RECORD_UUID. Owners/editors can read it; agents and viewers cannot read tenant-wide archive status. The Go client exposes AuditArchiveStatus.
  • Disabled: no external destination is configured. Local journal retention and NDJSON exports still work.
  • Pending: a journal record has no verified receipt for the current destination policy.
  • Retrying: one or more records failed verification/delivery and will retry. Codes distinguish unavailable destinations, missing bucket locking, content mismatch and unverified retention.
  • Verified: the record was read back with its exact bytes, object version, encryption and compliance retention. Its receipt contains bucket/key/version, SHA-256, retain-until and verified-at times.
  • The portal reports verified objects still within retention separately from elapsed retention periods.
    • Status is the last verified readback, not continuous provider monitoring.
    • Use provider-side monitoring for later access, lifecycle operations and account-level changes.
    • For an empty journal, zero verified records is not proof of an external write.
  • The journal itself is the durable outbox.
    • Conversation updates and journal insertion commit together; a background worker claims undelivered records.
    • A claim expires after 90 seconds, remote work is bounded to 45 seconds, and failed deliveries back off from 10 seconds to one hour. Restarting preserves PostgreSQL work and receipts.
    • Conditional object creation plus readback recovers ambiguous upload responses without overwriting an existing object. A content mismatch never authorizes replacement.
  • Retention starts at the current delivery generation's first claim. Retries keep that retain-until value.
    • If a pending generation expires during a prolonged outage, a later claim creates a fresh retention generation under a new object key.
    • Previously verified records are not automatically renewed after their configured retention expires.
  • Changing provider, bucket, owner, region, prefix, retention, encryption or explicit endpoint creates a new destination-policy digest and independently queues retained journal records for that policy.
    • Previous objects and receipts remain; Nines does not delete or shorten their retention.
    • Credential rotation alone does not change the destination identity.
  • Archival is asynchronous.
    • An archive outage does not undo or automatically block an approved SLO change.
    • Inspect pending counts and errors, alert on the intent.audit_archive operation outcome, and define your organization's outage/approval policy.
    • Configuration rollback still cannot restore consumed error budget.

Disposable Ceph acceptance

Run from the repository root:

sh
make test-ceph
  • Requires Docker, Go and OpenSSL with req -addext support.
    • The runner uses digest-pinned official Ceph 20.2.4 and OpenBao 2.5.0 images.
    • It starts a single-node Ceph cluster backed by a 2 GiB container-local file and an ephemeral OpenBao transit engine for SSE-S3.
    • Ceph is capped at 1536 MiB/2 CPUs; OpenBao at 256 MiB/1 CPU.
    • Allow additional disk space for image downloads.
  • The runner generates credentials and a two-day fixture certificate, trusts that certificate explicitly, and publishes only RGW's TLS endpoint on a random host loopback port.
    • The key service listens only inside the fixture network namespace.
    • It uses no privileged container, host block devices, production credentials or existing playground configuration.
  • It creates synthetic locked and unprotected buckets, then checks:
  • A Nines archive upload and independent exact-version JSON/encryption readback.
  • Conditional retry returning the same checksum/key/version, with exactly one stored version.
  • Successful deletion of an unprotected encrypted control object.
  • Denial of protected-version deletion and COMPLIANCE retention shortening.
  • Nines rejecting an unlocked bucket as bucket_not_locked and unchanged retention after the denied mutations.
  • An exit trap removes only its uniquely named containers and temporary credentials/certificate directory, including on failure or interruption.
    • The synthetic retained data is destroyed with the container filesystem; this is an application-protocol test, not disk-level immutability, disaster recovery, cluster HA or production key-management certification. Downloaded image caches remain.
    • The fixture uses broad synthetic storage credentials and an ephemeral development key service, neither of which is a production deployment template.

Provider acceptance

  • Local tests cover signed SDK requests, checksum/content/version checks, conditional retry recovery, retention rejection, leases, PostgreSQL restart, tenant isolation and backoff. They do not establish a real AWS account's IAM, KMS, region or retention configuration.
  • An opt-in acceptance test can write one synthetic object retained for one day into an isolated, pre-provisioned Object Lock bucket/prefix.
    • It never runs from ambient AWS credentials alone.
    • Create a private YAML file containing the auditArchive fields above, using retentionDays: 1 and a dedicated acceptance prefix; configure credentials outside that file. Then explicitly run:
sh
TEST_AUDIT_S3_CONFIG=/private/path/acceptance.yaml \
TEST_AUDIT_ARCHIVE_CONFIRM=COMPLIANCE \
go test -v ./internal/auditarchive -run '^TestRealS3ObjectLockReadbackAndIdempotency$' -count=1
  • The test does not create a bucket or modify retention policy.
    • It verifies content/retention readback and that an exact retry recovers the same version.
    • It leaves the synthetic object retained for one day; cleanup cannot shorten compliance retention.
    • Independently inspect that version with an authorized verifier.
    • A passing S3-compatible test is separate from AWS acceptance.