Observability

Self-host ships a local-first observability stack — Prometheus (metrics) + Loki (logs) + Promtail (log shipper) + Grafana (visualization) + cAdvisor (container metrics). It's gated behind a compose profile so it's off by default. (Adds ~500 MB RAM

• ~400 MB disk; not what you want during day-1 evaluation.)

Cloud edition uses Sentry instead. Both stories cover the same ground: errors, traces, container health.

Enabling

docker compose --profile observability up -d

Five containers start:

Open http://localhost:3001. Default login admin/admin — rotate on first sign-in.

What's pre-provisioned

Both datasources are auto-attached. Open Explore in the Grafana sidebar:

• Prometheus — has cadvisor metrics out of the box. Try container_memory_usage_bytes{name=~"ccd-.+"} for live CCD service memory.
• Loki — has every container's stdout/stderr labeled by container, service, project. Try {service="web"} for the Next.js logs, {service="ai-services"} for the agent engine.

Dashboards to import

CCD-specific dashboards (agent latency, queue depth, LLM spend) land once the corresponding /metrics endpoints exist on web/worker/ ai-services. For now, three community dashboards cover the basics — Dashboards → Import:

Each just needs the Prometheus or Loki datasource selected from the dropdown during import.

CCD service /metrics

Prometheus is configured to scrape these endpoints:

• api-gateway:3001/metrics — Fastify + OTel pipeline (live)
• ai-services:5100/metrics — FastAPI Prometheus middleware (live)
• worker:3030/metrics — TBD
• web:3000/api/metrics — TBD

Endpoints that don't exist yet show as up == 0 in Prometheus → Status → Targets. That's the explicit TODO list as we wire service-level metrics.

Loki ingest limits

Default caps (set in observability/loki-config.yml under your install dir):

• 30-day retention
• 8 MB/s ingestion rate, 16 MB burst
• 10k streams per user
• 5k entries per query

Tune in the config if you hit them. The ingestion cap is the most likely to bite if a service starts logging at high frequency.

Disabling

docker compose --profile observability down

Stops the 5 observability containers. The main stack keeps running. Data (metrics, logs, dashboards) survives on named volumes (ccd-prometheus-data, ccd-loki-data, ccd-grafana-data) — when you --profile observability up -d again, it picks up where it left off.

To nuke observability data too:

docker compose --profile observability down -v

Scaling beyond single-node

This profile assumes a single host. If your log volume justifies clustering, swap Loki's common.storage.filesystem for S3/MinIO and run distributor/ingester/querier separately. That's an operator-tuned config, not something we ship as a default — see Grafana Loki's deployment modes.