Observability Dashboards

Where each cache / compaction / latency question is answered now — the admin task inspector (DB), Loki info-level events, and OTel traces in Tempo — with the query and read criteria for each surface

Design principle for panels

Not “add if it displays cleanly” but each panel answers one specific question. The narrower the question, the shorter the action path when something breaks.

Observability splits across three surfaces. Read the next section first: it decides which surface answers your question before you reach for a query.

Three surfaces — which answers what

SurfaceBest forData sourceSection below
Admin task inspectorPer-task deep dive: this task’s cache read/write tokens, compaction count + savings, per-step predicted-vs-actualDB stepUsages[] (persisted per step)Cache & compaction — per-task
Loki / LogQLCross-deployment aggregate over info-level events: lifecycle, compaction-fire rate, budget linepino → Loki (info+ only)Loki panels
OTel traces (Tempo)Latency + throughput + per-tool duration — carried as span attributes, queried with TraceQL@ai-sdk/otel → TempoLatency / throughput panels

The constraint that shapes all of this: pino ships only info+ to Loki (apps/server/src/lib/logger.ts drops debug to protect the free-tier quota). The two most cache-relevant events — stream.step_usage (per-step usage) and cache.breakpoints_placed — are emitted at debug, so they do not reach Loki in production. Their per-task detail lives in the admin inspector. To dashboard them in Loki, raise them to info first and accept the volume.

Cache & compaction — per-task (admin inspector)

Per-task cache and compaction detail lives in the admin task inspector, sourced from the DB stepUsages[] snapshot (agent-loop.ts createOnStepEnd), persisted every step: cacheReadTokens / cacheCreationTokens / predictedTokens / compactedTokens / durationMs. Open a task to read:

  • Is prompt cache saving money on this task? — per-step cacheReadTokens vs cacheCreationTokens. The aggregate read/write ratio across tasks is the TraceQL panel below.
  • Is compaction firing, and how much does it save? — per-step compactedTokens and predicted-vs-actual input.
  • Are cache breakpoints placed correctly? — cache.breakpoints_placed payload { messagesCount, stepIndex, placedAt, lastRole } (model.ts markPrefixCacheBoundary). placedAt is the set of marked message indices from computeInnerAnchors + the tail block. Read it in the inspector’s runtime-context tab.

Loki panels — info-level events that work today

These read events that are shipped to Loki (info+), so they run as-is in production.

Panel: Compaction fire rate + savings

Question — “How often is in-loop compaction firing, and how much is it saving?”

sum(rate({job="zapvol-server", event="compaction.step_fired"}[5m]))

Savings trend:

avg_over_time(
  {job="zapvol-server", event="compaction.step_fired"} | json | unwrap savedTokens [10m]
)

compaction.step_fired carries { taskId, step, savedTokens } (agent-loop.ts), emitted only when a step newly triggers reduction (savedTokens > 0) — so the rate is genuine fire frequency, not per-step replay.

Panel: Budget line vs context window

Question — “How close is measured overhead (instructions + tools) to the model’s context window?”

{job="zapvol-server", event="compaction.budget_measured"}
  | json
  | line_format "instr={{.instructionsTokens}} tools={{.toolsTokens}} window={{.contextWindow}}"

compaction.budget_measured carries { taskId, instructionsTokens, toolsTokens, contextWindow } (agent-loop.ts).

Panel: Lifecycle & prefix size

task.created / task.completed (routes/tasks.ts) for lifecycle rate; stream.messages_prepared { inputMessages, modelMessages } (agent-loop.ts) for turn-start prefix size; agent.created for assembly. All info+, all safe to dashboard.

Latency / throughput panels (TraceQL)

Everything above answers cache / compaction questions from logs or the DB inspector — none of it carries timing. stream.step_usage has token counts but no TTFO, no tokens/s, no per-step duration in the log line. That axis lives in the OTel spans.

@ai-sdk/otel emits spans, not metrics — it registers no meter (see Observability Stack Overview → distributed tracing). So the timing rides as span attributes in Tempo, and the panels below are TraceQL against the Tempo data source — not PromQL, not Loki. Tempo’s TraceQL-metrics functions (rate, quantile_over_time, sum_over_time) compute the aggregates on the fly from spans, so no separate metrics pipeline is needed.

Prerequisite: registerTelemetry(new OpenTelemetry()) at boot. Once registered, spans emit by default — there is no per-call flag to set (SDK 7 dropped experimental_telemetry: { isEnabled }). Until then the Tempo data source is empty and these panels read blank.

Panel: Time to first output (TTFO)

Question — “How long does the user wait before the first token streams?” The perceived-latency metric; a regression here is felt even when total throughput is fine. TTFO rides on the chat span:

{ name = "chat" } | quantile_over_time(span.gen_ai.client.operation.time_to_first_chunk, .95) by (resource.service.name)

Healthy range is model-dependent (read the p50/p95 spread, not an absolute). Alert on a step change aligned with a release or a provider incident, or p95 diverging from p50 by more than ~3× (queueing / rate-limit backoff).

Panel: Per-step duration

Question — “Is a step taking longer end to end?” The chat span’s own duration:

{ name = "chat" } | quantile_over_time(duration, .95) by (resource.service.name)

A rise here with stable TTFO points at slower mid-stream generation rather than queueing; cross-check against the output tokens per second figure from the onLanguageModelCallEnd callback.

Panel: Per-tool execution duration

Question — “Which tool is the long pole in a step?” execute_tool spans split by tool name surface the slow tool directly — the thing logs force you to reconstruct by sorting timestamps:

{ name =~ "execute_tool.*" } | quantile_over_time(duration, .95) by (span.gen_ai.tool.name)

Then drill in: from a slow task’s trace_id, open the invoke_agent root span and read its execute_tool / chat children on one timeline — the “step N stalled, where?” question logs can’t answer cheaply.

Panel: Cache economics (read / write token ratio)

Question — “Is cache-written content reused, or wasted?” The chat spans carry gen_ai.usage.cache_read.input_tokens and cache_creation.input_tokens. Aggregate each as its own TraceQL query:

{ name = "chat" } | sum_over_time(span.gen_ai.usage.cache_read.input_tokens)
{ name = "chat" } | sum_over_time(span.gen_ai.usage.cache_creation.input_tokens)

then divide the two in a Grafana math expression. R > 3 healthy (each write read 3+ times); R < 1 alert (writes exceed reads — cache strategy failing). For the per-task version of this ratio, the admin inspector’s stepUsages[] is authoritative — the TraceQL form is the cross-deployment aggregate.

Workflow for adding a new panel

Step 0: pick the surface first

Before writing any query, decide which of the three surfaces the question belongs to:

  • Per-task detail an operator drills into → admin inspector (DB). Don’t add a Loki event for it.
  • Cross-deployment aggregate / trend / alert → Loki (must be an info+ event) or OTel trace (TraceQL).
  • Latency / throughput → OTel trace (TraceQL) only; logs don’t carry timing.

A new Loki event must be logged at info or higher — log.debug(...) is dropped before Loki (see the shipping constraint above). High-volume per-step data intentionally stays at debug and belongs in the DB inspector or an OTel metric, not Loki.

Step 1: Add the event first, then the panel

Never treat Grafana as a code editor. Panels are views of data, not containers of business logic. First emit a structured event from Zapvol:

log.info("your_event.name", {
  taskId,
  /* low-cardinality fields */ kind: "...",
  /* numeric fields */ someMetric: 42,
});

Run it a few times. In Grafana Explore, confirm {event="your_event.name"} | json returns the expected shape.

Step 2: Define the four elements of a panel

Before creating the dashboard, write them down (at minimum in the PR description):

  • Question: what does this panel resolve
  • Query: LogQL / TraceQL prototype
  • Healthy range: written so someone without your context can still read “this number should be X”
  • Alert signals: at least 2 actionable investigation paths

If you can’t write these down, the question isn’t clear enough — don’t start drawing the panel.

Step 3: Commit the dashboard JSON to the repo

From Grafana Dashboard → Settings → JSON Model, copy the JSON into:

ops/grafana/dashboards/{domain}-{name}.json

In the same directory’s README.md, add import instructions and read criteria (or link to the corresponding section in this chapter).

Why commit JSON: a single Grafana instance loses data, you lose everything. Committed to the repo, a new engineer can spin up an equivalent Grafana locally in 15 minutes. It’s also a good code-review carrier — reviewers can diff query changes in a PR.

Step 4: If you need alerting

Use Grafana’s native Alert rules (declarative YAML — not the clicky UI). Commit the declarative rules:

ops/grafana/alerts/cache-hit-degradation.yaml

Alert rules should be extremely restrained. In an agent system, many things “could go wrong” but few “must be handled immediately”. Defaults:

  • for: 15m or longer — brief fluctuations don’t page.
  • Only alert on outcome metrics (cache hit ratio, error rate, latency tail), never on process metrics (breakpoint count, compaction frequency).
Was this page helpful?
Edit on GitHub