Observability Stack Overview

Why observability is a first-class concern

Agent systems suffer from a particularly sneaky class of bugs: behaviorally correct, economically wrong. The LLM still returns sensible answers, tests still pass, but every step quietly misses cache, every compaction runs more aggressively than it should, every tool call burns 25% more of the token budget than expected. The human eye can’t catch any of this — only a dashboard can.

This chapter describes the stack Zapvol currently runs — two direct pushes into Grafana Cloud: pino → Loki for logs and @ai-sdk/otel → Tempo for traces, with no collector in between — why it’s shaped this way, when a collector finally earns its place, and the pitfalls to avoid. It is not a deployment manual; it is the mental model you need to operate this system.

The pipeline at a glance

Today there is no Alloy / collector hop: the pino-loki transport ships log lines to Loki on a background worker thread, and the OTLP exporter ships spans to Tempo on a background timer. Both are async and batched, so the export path never sits on a request. The diagram’s collector layer is the topology you graduate into later (see When a collector earns its place), not what runs today.

The pipeline SVG still depicts the older collector-based topology and is owed a redraw to match the current direct shape.

The combination — fully open source, standard protocols, a Grafana Cloud free tier, a clean path to scale up — still has the best overall value; the only change from the drawing is that the collector is deferred until a concrete signal demands it.

Why this stack — three alternative designs and why we didn’t choose them

Candidate 1: ELK (Elasticsearch + Logstash + Kibana)

Most mature, most feature-complete. Overkill for Zapvol’s scale:

Elasticsearch is a full-text indexer — every field gets an inverted index. In a log workload, 90% of fields are never queried; index cost is pure waste.
Logstash’s JRuby runtime uses 5-10× more memory than Alloy.
Kibana’s permission model is more complex than a single team needs.

Verdict: suited for “grep everything in prod” full-text scenarios. Zapvol’s log queries are all structured (filter by event, taskId, time range) — the ES strengths don’t apply.

Candidate 2: OpenTelemetry Collector + any backend

Most standardized. But Alloy is itself an OTel-Collector-based distribution, and the differences are:

Alloy ships the River config language + a visual debugging UI (:12345).
Alloy has first-class integration with the Grafana ecosystem.
OTel Collector is more generic but has a rougher config surface.

Verdict: Zapvol runs no collector today (both signals push direct). When one is eventually needed, and you’re not leaving the Grafana ecosystem, Alloy is a superset of OTel Collector; reach for the vanilla OTel Collector only if you need a non-Grafana backend (Datadog, Honeycomb).

Candidate 3: Commercial (Datadog / Honeycomb / Logz.io)

Best UI, best support. Price:

Datadog pricing is $1.27/GB ingest + $2.50/M indexed events.
A medium-sized agent task produces 5-10 info + debug logs per step. 20 steps × 100 tasks/day = 20k logs/day = 600k/month = a few hundred dollars.
The open-source alternative at the same volume costs < $10 (object storage + small VM).

Verdict: pick when money is loose. Not needed for an internal tool.

Zapvol’s existing infrastructure

pino config (`apps/server/src/lib/logger.ts`)

const pinoLogger = pino(
  { level: process.env.LOG_LEVEL || (isDev ? "debug" : "info") },
  isDev ? pretty({ colorize: true, translateTime: "HH:MM:ss" }) : undefined,
);

Dev mode uses pino-pretty (colored, readable); production emits JSONL — on stdout for docker logs, and pushed to Loki by the pino-loki transport directly, with no collector in between to consume it.

event-first schema

log.info("task.created", { taskId, userId });
log.error("stream.failed", { taskId, err }, "Stream failed");

event is always the first required parameter. This convention runs through @zapvol/backend, @zapvol/server, and @zapvol/desktop. Consequences:

event="task.created" precisely filters one class of events in Grafana.
All event names form an auditable event catalog.
New contributors are forced to name the thing they’re logging — instead of log.info("something happened").

AsyncLocalStorage injection

function mergeContext(event, data) {
  const ctx = RequestContext.get();
  if (ctx?.traceId) merged.traceId = ctx.traceId;
  if (ctx?.userId) merged.userId = ctx.userId;
  // ...
}

Every log line carries traceId and userId without an explicit parameter. During debugging, filter by traceId to retrieve all logs for one request — across services, across async boundaries.

Existing key events (excerpt)

Event name	Location	Level	Business meaning
`task.created` / `task.completed`	`routes/tasks.ts`	info	Task lifecycle
`stream.messages_prepared`	`agent-loop.ts`	info	Turn-start prefix size (input vs model messages)
`compaction.step_fired`	`agent-loop.ts`	info	In-loop compaction fired (only when `savedTokens > 0`)
`compaction.budget_measured`	`agent-loop.ts`	info	Measured overhead (instructions + tools) vs context window
`agent.created`	`agent-loop.ts`	info	ToolLoopAgent assembled
`stream.step_usage`	`agent-loop.ts` (`createOnStepEnd`)	debug	Per-step token usage — not shipped to Loki
`cache.breakpoints_placed`	`model.ts` (`markPrefixCacheBoundary`)	debug	Cache breakpoint positions (`placedAt`) — not shipped to Loki

Only info+ reaches Loki (see the pino config above). The two debug events carry the richest cache / per-step detail, but by design that detail lives in the admin task inspector (DB-backed) and — for aggregate trends — in OTel traces (Tempo, via TraceQL), not in Loki. See Observability Dashboards → three surfaces for which question goes where. When building a Loki dashboard, treat the info-level events as primary keys.

Three deployment modes

Mode A: Grafana Cloud free tier (recommended starting point)

Simplest, and what Zapvol runs. Create a Grafana Cloud account; nothing extra to deploy — the app process ships both signals itself. Cost: $0/month (50 GB logs + 50 GB traces + 10k metrics).

Logs — set the LOG_LOKI_* vars and pino’s built-in pino-loki transport (a background worker thread) pushes straight to Loki:

LOG_LOKI_ENABLED=true
LOG_LOKI_URL=https://logs-prod-XX.grafana.net
LOG_LOKI_USER=<loki-instance-id>
LOG_LOKI_TOKEN=<grafana-cloud-token>

Traces — set the OTEL_* vars and the OTLP exporter (BatchSpanProcessor, a background timer) pushes straight to Tempo. Server only — desktop registers no exporter:

OTEL_TRACES_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-<zone>.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(tempo-instance-id:token)>

The Loki credential and the OTLP/Tempo credential are different instance-id/token pairs — don’t reuse one for the other. No sidecar, no config.alloy: both gates default off and no-op until their vars are set, so the same image runs locally and in prod.

Mode B: Self-hosted single VM

One VM runs Loki + Grafana + Alloy. Data lives in local object storage (S3 / Cloudflare R2).

Cost: VM + object storage ≈ $10-30/month (depends on log volume).
Ops: you manage Loki retention, Grafana upgrades.
Fits: teams with existing VM infrastructure who don’t want logs leaving the network.

Mode C: Kubernetes

Alloy as a DaemonSet, one per node. Loki as a StatefulSet inside the cluster, or managed Loki.

Cost: depends on cluster size.
Ops: standard K8s operations.
Fits: teams already running K8s.

Strongly recommend Mode A as the starting point — zero infrastructure burden, migrate out if it stops fitting. Loki data can be exported via loki-migrate.

The three modes above describe production deployments. Local dev does not ship to Loki — by design, dev uses in-process pino-pretty terminal output. The worker-thread transport that ships to Loki fights the --inspect debug port, so Loki shipping is production-only. To watch cache / compaction behavior while developing, use the built-in admin runtime-context inspector rather than Grafana.

Label cardinality pitfalls (required reading)

Loki’s storage cost is almost entirely a function of label combination count (cardinality), not log volume. The rule:

Field	Label?	Reason
`event`	Yes	Finite enum (a few dozen values)
`module`	Yes	Finite enum
`level`	Yes	Five values
`taskId`	No	High cardinality (millions)
`userId`	No	Medium-high cardinality
`traceId`	No	One per request
Numeric fields (tokens, ratios, …)	No	Continuous

One bad label config can slow Loki down 100× and inflate storage 50×. Rules:

Label = “something I will sum by”; field = “something I will filter on for precise queries.”
Never use ID fields as labels.
Monitor loki_ingester_memory_streams for one week before promoting any new label to production.

At query time, | json parses fields. The difference:

# Label filter (fast)
{event="stream.step_finished"}

# Field filter (slower, but doesn't contribute to cardinality)
{event="stream.step_finished"} | json | taskId="abc-123"

The two compose. Correct pattern: filter with labels down to low millions, then narrow with fields.

When to escalate to Prometheus metrics

Logs are best for “why did this specific thing happen” (“why did this task miss cache?”). Metrics are best for long-term trends + alerting (“p95 cache hit ratio over the past 7 days”).

Escalation signals:

Signal	Action
A dashboard query routinely takes > 30s	Emit that metric via `prom-client`; query it from Prometheus
Need declarative alert rules (“ratio < 0.3 for 5 min”)	Prometheus Alertmanager
Log volume approaching Grafana Cloud free-tier limits	Downgrade debug events to metrics; keep info+ in Loki

Escalation path: add prom-client to the app and expose a /metrics endpoint (or remote_write). No collector is required today — Prometheus, or Grafana Cloud’s metrics endpoint, can scrape the app directly; if you later run Alloy, its prometheus.scrape / prometheus.remote_write can front it.

Distributed tracing with `@ai-sdk/otel` (live)

Logs answer “why did this specific thing happen”; they carry no timing. One agent run crosses task-orchestrator.ts, the 20+ step loop in the engine, tool calls into the sandbox, and BUA sessions over WebSocket — and “which step stalled, and for how long” is a question logs can only answer by sorting timestamps by hand. Distributed tracing answers it directly, and it is now on for the server.

@ai-sdk/otel emits spans following the OpenTelemetry GenAI semantic conventions — no custom instrumentation:

Emitted	Names
Spans	`invoke_agent {modelId}` · `chat {modelId}` · `execute_tool {toolName}`
Usage attributes	`gen_ai.usage.input_tokens` · `output_tokens` · `cache_read.input_tokens` · `cache_creation.input_tokens`
Request/response attrs	`gen_ai.request.model` · `request.temperature` · `response.finish_reasons` · `response.id`
Timing (span attrs)	`gen_ai.client.operation.duration` · `...time_to_first_chunk` (TTFO) · `...time_per_output_chunk` · `gen_ai.execute_tool.duration`

These are spans, not metrics. @ai-sdk/otel registers no meter — the timing figures ride as span attributes in Tempo, queried with TraceQL, not as Prometheus series. See Observability Dashboards for the panels this unlocks. Performance stats are also surfaced imperatively through the onLanguageModelCallEnd({ usage, performance }) callback: response time, total step time, tool execution time, time to first output, and output tokens per second.

How it’s wired

Registration is one call at boot, per process — the API server and the BullMQ worker both make it, because agent runs execute in the worker, not in the HTTP request:

// apps/server/src/lib/otel.ts — called first in index.ts AND worker.ts
registerTelemetry(new OpenTelemetry());

registerTelemetry stores the integration on globalThis, so it is enabled by default for every generateText / streamText / ToolLoopAgent call in the process — no per-call isEnabled flag (that was the AI SDK 5/6 experimental_telemetry shape; SDK 7 uses the telemetry option and defaults enabled once an integration is registered). Each call still carries a telemetry tag via the shared aiTelemetry(functionId) helper so spans group by functionId (agent.main, agent.subagent, title, compaction). @zapvol/backend stays telemetry-agnostic: the tag is inert until a process registers an integration, so desktop — which never registers one — emits nothing.

Prompt content is off by default — and how to turn it on when you must

The AI SDK defaults recordInputs / recordOutputs to on, and the OpenTelemetry integration exposes no global switch to force them off — so left alone, every span would carry the full system prompt, user messages, and model output. aiTelemetry() therefore sets both to false on every call. Spans carry only skeleton + metrics: step, duration, token usage, finish reason, error — never conversation content.

When a bug genuinely needs the prompt/output body, flip it for one process, temporarily:

OTEL_CAPTURE_CONTENT=true   # that process's spans now carry gen_ai.input.messages / output.messages

Turn it back off when done. Three costs to respect while it is on:

Sensitive-data egress — BUA drives employees’ logged-in sessions; prompts can contain private data you don’t want in Tempo.
Ingest cost — full bodies run tens of KB per span; the free-tier trace quota drains fast.
Memory — large spans sit in the BatchSpanProcessor queue until the next flush, pinning big strings.

For durable prompt/output inspection, use the admin task inspector (DB) — the product surface built for it — not the trace backend.

One trace id across logs and traces

Jumping from a slow trace in Tempo to its logs in Loki only works if both carry the same id in the same format. Two changes make that hold:

RequestContext.newTraceId() now returns a 32-hex string (was a UUID) — a valid Tempo / W3C trace id even when no span is active.
Background agent-run entry points (worker.ts job, schedule/nudge fires, browser entrypoint) run inside runWithTrace(), which starts an OTel span and sets RequestContext.traceId from spanContext().traceId.

The logger’s context merge is untouched — it still emits ctx.traceId on every line — but that value is now the real OTel id. So a job’s Loki logs and its Tempo spans share one id: copy the trace_id off a slow span, run {app="zapvol-server"} | json | traceId="<that id>" in Loki, and you have the whole run’s logs. HTTP request seams are left un-spanned on purpose (they mostly enqueue); they still get a 32-hex traceId for log attribution.

Wiring to Grafana Cloud (direct)

Grafana Cloud exposes a native OTLP gateway (https://otlp-gateway-<zone>.grafana.net/otlp, basic auth = base64(instanceID:token)) that routes traces → Tempo. The exporter points straight at it — no collector — matching Mode A and the direct pino-loki log path:

OTEL_TRACES_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-<zone>.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(instanceID:token)>

The gate mirrors LOG_LOKI_*: OTEL_TRACES_ENABLED=true and the endpoint present, else warn + no-op — never throw on a misconfigured env. The Tempo credential is a different instance-id/token from the Loki one.

When a collector earns its place (Alloy)

Direct export is the current answer because the export is async and off the request path, so a collector buys no latency. A collector — Alloy, an OTel-Collector superset — earns its extra container only when a concrete signal shows up:

Signal	What only a collector gives you
Tempo’s 50 GB trace quota under pressure from “20 steps × N tasks”	Tail sampling — keep a trace after seeing all its spans (errors / slow ones). A direct exporter head-samples, blind to outcome.
Grafana Cloud outages start dropping telemetry	Buffering + retry decoupled from the app — the app talks to localhost, the collector holds the backlog.
logs + traces + metrics + host metrics want one exit	A single egress with one credential set and centralized batching.

Until one of these bites, the extra process is pure ops surface. When it does, the logs migrate onto the same Alloy for symmetry.

Local debugging

Everything above is production shipping. To exercise it locally against your own Grafana Cloud — tracing runs in dev too, since the BatchSpanProcessor is main-thread and, unlike the Loki worker-thread transport, does not fight --inspect:

Put the three OTEL_* vars in apps/server/.env (build the header from base64 of instanceID:token).
Smoke the pipeline first — no agent, no Redis / DB needed:
```
pnpm --filter @zapvol/server exec tsx scripts/smoke-otel.ts
```
It emits one span and flushes; find it in Grafana → Explore (Tempo) by service.name="zapvol-smoke" within ~15 s. An OTEL_TRACES_ENABLED=true but … missing line means the env did not load.
End to end — run the server + the worker (agent runs happen in the worker), trigger a task, then query service.name="zapvol-worker" in Tempo to see the job:task.run → invoke_agent → chat/execute_tool tree. Copy its trace_id into the Loki query above to confirm logs and trace share one id.

To watch cache / compaction behavior while developing, the built-in admin runtime-context inspector is still the right surface — that detail is DB-backed, not in traces.

Node Runtime Health — detecting whether Node is degrading (memory / GC / event loop / handles) + the perf cost of each approach
Observability Dashboards — Current built-in core dashboards and how to read them
Context Compaction — How the compaction boundary shapes cache breakpoint design (origin of the cache.breakpoints_placed event)
prepareStep Semantics — Where stream.step_finished is emitted