Local Development with Grafana Cloud

Why plug it in locally

Developers writing code want their logs streaming into Grafana Cloud already — to observe cache hit ratio, compaction frequency, and breakpoint placement distribution while editing, not to wait until production for the first look at those metrics. This matters most when changing cache-related code: you can run a local A/B and see the panel change immediately, instead of relying on “probably correct” intuition.

Constraints:

• Cannot be mandatory — devs not using Grafana Cloud must not be blocked; if env vars are unset, nothing should activate.
• Cannot slow the local feedback loop — the terminal’s pino-pretty colored output must stay; network failures must not stall pnpm dev:server.
• Cannot require Docker — dev machines do not always keep Docker running; asking for a separate Alloy container is too much.

Two paths

Path	Setup cost	Production parity	Recommended for
pino-loki transport direct (recommended)	10 min	Medium (skips the Alloy hop)	90% of daily dev
Local Alloy container	30 min + Docker	High (identical stack to prod)	Debugging Alloy config / validating label extraction

Default to pino-loki direct. The production-parity gap is concentrated in Alloy’s label extraction, batching, and retry policy — operational concerns that do not matter to local dev. Local dev cares about “can I see my log event on the dashboard?” Leave Alloy parity testing to staging / CI environments.

The pino-loki direct approach

Shape

pino (app)
  ├── transport 1: pino-pretty → stdout       (terminal, human-readable)
  └── transport 2: pino-loki  → Loki HTTP     (Grafana Cloud dashboards)
                                  opt-in, gated by env var

The two transports run in parallel and don’t interfere. pino’s worker_threads ensure the HTTP push never blocks the main event loop.

Step 1: Create a Grafana Cloud account

grafana.com/auth/sign-up — the free tier suffices: 50 GB logs + 10k metrics + 50 GB traces per month.

From the Stack management page, collect three pieces of information:

• Loki endpoint: https://logs-prod-XX.grafana.net (XX varies by region)
• User / instance ID: a numeric string
• API token: Security → API Tokens → create new, scope logs:write

Step 2: Local env configuration

Add to apps/server/.env.local (do not commit — the repo’s .gitignore already excludes .env.local):

# Switch: true enables Loki shipping; false or unset disables it.
# A dedicated switch (separate from credentials) lets you temporarily turn
# shipping OFF without clearing the URL/TOKEN — useful for sensitive debugging,
# saving free-tier quota, or working offline.
LOG_LOKI_ENABLED=true

# Required when the switch is true; ignored when the switch is false
LOG_LOKI_URL=https://logs-prod-XX.grafana.net
LOG_LOKI_USER=123456
LOG_LOKI_TOKEN=glc_eyJvIjoi...

# Multi-dev isolation: tag the log stream with a dev_user label for dashboard filtering
LOG_LOKI_LABELS_EXTRA=dev_user=jonathan

Three switch states:

LOG_LOKI_ENABLED	URL/TOKEN complete?	Behavior
true	Yes	Shipping active, normal operation
true	No	Startup console.warn about missing credentials; no shipping this process
false / unset	Don’t care	URL/TOKEN never read; silently skipped

Daily workflow: set URL/TOKEN once in .env.local and leave them; flip LOG_LOKI_ENABLED between true and false to toggle — no need to clear credentials when you want shipping off.

Purpose of the dev_user label: when multiple devs on the team have shipping enabled simultaneously, a panel query like {dev_user="jonathan"} lets you see only your own logs. The value must be low-cardinality (see Label Cardinality Pitfalls); use a fixed username or initials, never dynamic content.

Step 3: Install the dependency

pnpm --filter @zapvol/server add pino-loki

pino-loki is pino’s Loki transport — a worker-thread-based non-blocking HTTP pusher.

Step 4: Refactor the logger for multi-transport

Core diff in apps/server/src/lib/logger.ts (~20 lines):

import pino from "pino";

const isDev = process.env.NODE_ENV !== "production";

// Two-stage gate: shipping only activates if the switch is "true" AND credentials are complete
const lokiRequested = process.env.LOG_LOKI_ENABLED === "true";
const lokiReady = Boolean(process.env.LOG_LOKI_URL && process.env.LOG_LOKI_USER && process.env.LOG_LOKI_TOKEN);
const lokiEnabled = lokiRequested && lokiReady;

if (lokiRequested && !lokiReady) {
  // Intent to enable but credentials missing — explicit warning, don't ship this process.
  // Deliberately does NOT throw, to avoid blocking production startup on env misconfig.
  // eslint-disable-next-line no-console
  console.warn(
    "[logger] LOG_LOKI_ENABLED=true but LOG_LOKI_URL/USER/TOKEN incomplete; Loki shipping disabled for this process",
  );
}

const targets: pino.TransportTargetOptions[] = [
  // Terminal — dev = pretty with colors, prod = plain JSON
  isDev
    ? {
        target: "pino-pretty",
        level: "debug",
        options: { colorize: true, translateTime: "HH:MM:ss", ignore: "pid,hostname" },
      }
    : { target: "pino/file", level: "info", options: { destination: 1 } }, // stdout
];

if (lokiEnabled) {
  targets.push({
    target: "pino-loki",
    level: "info", // only ship info+, to conserve free-tier quota (see "Quota management" below)
    options: {
      host: process.env.LOG_LOKI_URL,
      basicAuth: {
        username: process.env.LOG_LOKI_USER,
        password: process.env.LOG_LOKI_TOKEN,
      },
      labels: {
        app: "zapvol-server",
        env: isDev ? "dev" : "prod",
        ...parseExtraLabels(process.env.LOG_LOKI_LABELS_EXTRA),
      },
      batching: true,
      interval: 5, // 5s batch to reduce API call volume
    },
  });
}

const pinoLogger = pino({ level: process.env.LOG_LEVEL || (isDev ? "debug" : "info") }, pino.transport({ targets }));

function parseExtraLabels(raw: string | undefined): Record<string, string> {
  if (!raw) return {};
  return Object.fromEntries(raw.split(",").map((kv) => kv.split("=").map((s) => s.trim())));
}

The wrapLogger / createLogger logic below this is unchanged — this refactor is purely additive and breaks no existing API.

Step 5: Verify

pnpm dev:server

Send a few requests to trigger logs. In Grafana Cloud → Explore → Loki datasource:

{app="zapvol-server", env="dev", dev_user="jonathan"}

Logs should appear within 5 seconds (interval: 5 is the batch cycle). If nothing after 30 seconds:

1. Check the pnpm dev:server terminal for pino-loki transport errors (typically token / URL issues)
2. Check the Grafana Cloud console for ingest error records
3. Confirm .env.local was loaded (Astro / Vite may need a restart)

Label cardinality rules for local

Same rules as production (full background in the main doc):

Field	Label?	Notes
app	Yes	Fixed: “zapvol-server”
env	Yes	Two values: “dev” / “prod”
dev_user	Yes	Team size; low cardinality
event	Yes (prod)	Extracted by Alloy pipeline; local pino-loki skips this; filter by field at query time
taskId	No	High cardinality (one per task)
userId	No	Medium-high cardinality
traceId	No	One per request
Any numeric field	No	Continuous values

pino-loki by default makes every item in its labels config a Loki label. Never put traceId or taskId into labels — it will burn the free-tier quota on day one and collapse Loki’s query performance.

Quota management

Free tier: 50 GB logs/month. Empirical estimates:

Scenario	Log volume
One typical agent task (20 steps), info+ logs	~150-300 KB
Single dev, 200 tasks/day	~60 MB
Single dev, month of active development	~1.8 GB
Whole team of 10 devs running simultaneously	~18 GB/month

One dev nowhere near the limit. But:

• LOG_LEVEL=debug for a full day can push 1-2 GB — cache.breakpoints_placed and similar per-step debug events add up
• Whole team on LOG_LEVEL=debug simultaneously could exceed the quota

Defensive policy:

1. pino-loki transport level fixed to info — debug goes only to terminal, never to Loki. The code above already does this.
2. Terminal LOG_LEVEL=debug only on demand — when investigating a specific cache miss, not as the default.
3. Configure billing alerts in the Grafana Cloud console — early warning on quota burn.

Secret handling

.env.local must be in .gitignore — the repo defaults already handle this; no extra config required.

Extra safeguard: add a pre-commit hook to scan for the glc_ey prefix (the fixed signature of a Grafana Cloud API token) to prevent accidental commits. Simplified shell hook:

# .git/hooks/pre-commit
if git diff --cached | grep -qE 'glc_ey[A-Za-z0-9_-]{20,}'; then
  echo "ERROR: Grafana Cloud token detected in staged changes. Abort."
  exit 1
fi

Graceful degradation on network failure

pino-loki has built-in retry. During a transient dev-machine network outage:

• pnpm dev:server continues running — the HTTP push fails async without blocking the main flow
• The terminal pino-pretty output remains visible — it’s fully independent of pino-loki
• Grafana Cloud will not receive the lost logs — logs produced during the outage are dropped locally

If you need “network failure, backfill later” behavior, upgrade to the local Alloy path (Alloy has a local WAL that buffers during outages). Rarely needed for daily dev work.

When to upgrade to local Alloy

Direct push is enough in most cases. These rare scenarios justify running an Alloy container locally:

Scenario	Why Alloy fits better
Debugging the production config.alloy	Locally iterate on label extraction and pipeline stages
Collecting stdout from multiple containers simultaneously	One Alloy handles many services
Verifying real label-cardinality impact	Alloy’s pipeline_stages can mock production filter logic
Need a local WAL for offline buffering	Direct push lacks this

Upgrade path:

1. Comment out LOG_LOKI_URL in .env.local (pino falls back to single-stdout output)
2. Launch an Alloy container with the config.alloy from the main doc
3. Use the same config.alloy file in prod — after local iteration, promote directly

Related chapters

• Observability Stack Overview — Design of the four-layer pipeline and three production deployment modes
• Observability Dashboards — The four core dashboards you’ll see once connected