Observability

Debugging

Finding out what a running application actually did.

There is no editor configuration in this repository — no .vscode/launch.json, no debugger profiles. Attaching a debugger is something your editor already knows how to do to a Node process, and pinning a configuration to one editor ages badly.

What the template does ship is the machinery for answering "what happened to this request", which is the question you usually have in production and cannot answer with a breakpoint.

Uncaught server errors

Each app's instrumentation.ts calls register() and exports onRequestError, which Next invokes for every uncaught server error — route handlers, server actions and the render itself. Each one is logged with its stack, method and route, and reported to PostHog through the injected error reporter.

Reporting is best-effort and never awaited into the request path. A slow collector must not add latency to a response that has already gone wrong.

So a server-side exception is in the container's stdout even when nothing in the code caught it. Look there before adding logging.

Correlating a request

@kreogen/observability ships two pieces for tying lines together, and register() already teaches the logger to read them — but nothing establishes a context for you. Until you wire it up, log lines carry only the fields you pass them.

resolveRequestId adopts an id from upstream where possible, in this order:

  1. traceparent — the W3C trace id, if a proxy or tracing sidecar set one
  2. x-request-id, x-correlation-id, x-amzn-trace-id
  3. A fresh UUID

Adopting rather than always generating is the point: it is what makes a log line here joinable with the load balancer's access log for the same request. A client-supplied id is sanitised and truncated to 128 characters first, since it would land in every line for that request and an unbounded value is a cheap way to fill a log budget.

runWithContext puts it in AsyncLocalStorage so nothing downstream has to thread it through its arguments, and enrichContext adds to it as the request proceeds — the id is assigned before anything knows who is calling, and the session resolves later:

import { enrichContext } from '@kreogen/observability/context';

enrichContext({ userId: user.id, organizationId: orgId });

Wire resolveRequestId and runWithContext in whichever layer sees every request, and echo the id back on the response. When a user reports an error, that header is the thing to ask them for.

Turning up the volume

debug lines are suppressed in production, where they are almost always the bulk of the volume and almost never the line anyone wants. LOG_LEVEL moves the floor without a code change:

LOG_LEVEL=debug

Values are debug, info, warn, error. Outside production the default is already debug.

Fields whose names look like secrets — password, token, apikey, authorization, cookie and others, matched case- and separator-insensitively — are redacted before output. That is a floor, not a guarantee: it catches the routine accident of logging a whole request body, and it cannot catch a secret arriving under a name nobody anticipated.

Health endpoints

Each app exposes two, and they answer different questions:

RouteQuestionTouches
/healthIs this process wedged?Nothing
/readyCan this instance serve traffic?Database, Redis

/health is a liveness probe and deliberately touches nothing. If it can answer, the event loop is running. Pointing a liveness probe at the database means a database outage restarts every container, which makes an incident strictly worse.

/ready is the one to read when debugging. It returns 200 with a per-check breakdown, or 503 naming what failed:

curl -s localhost:3002/ready | jq
{
  "status": "degraded",
  "checks": [
    { "name": "database", "ok": true, "durationMs": 4 },
    { "name": "redis", "ok": false, "durationMs": 3001, "error": "timed out after 3000ms" }
  ]
}

Each check is capped at three seconds, because hanging a readiness probe tells the orchestrator less than failing it does.

Local services

The everyday loop runs the apps on the host against infrastructure in docker compose:

bun run docker:up
ServiceWhereFor
Postgreslocalhost:5432The database
Redislocalhost:6379Rate limits, session storage
MinIOlocalhost:9001Object storage console
Mailpitlocalhost:8025Every email the auth flow sends

Mailpit is where verification, reset, magic-link and invitation messages land whenever Resend is unconfigured — which is the default. Sign-up needs it: with this container down and no RESEND_TOKEN, the send throws and the sign-up fails rather than silently skipping a step nobody could complete. See transactional email.

For the database itself, Prisma Studio is faster than a query:

bun dev --filter studio

Stripe webhooks need the Stripe CLI forwarding to the API. bun dev --filter api starts the listener automatically when the CLI is installed.

Failure modes worth recognising

These three account for most of the confusing hours, and none of them look like what they are.

Every page 404s immediately after signing in. Sessions carry activeOrganizationId, set by a session.create.before hook, and authenticated pages call notFound() when it is null. Break the hook and it reads like a routing bug. Check the member row exists for the user.

Sign-in redirects to sign-in, forever. /api/auth has to stay in the middleware's public routes. The app's matcher covers /(api|trpc)(.*), so treating the auth endpoints as protected redirects the sign-in request itself.

The build fails but typecheck passes. Something reachable from a client component imported a module marked server-only. The bundler traces dynamic imports too, so await import(...) does not get you out of it. The error names the module, not the import chain — work backwards from what imports it.

Something works locally and not in production

In rough order of likelihood:

  1. A missing environment variable. Validation runs at boot and names the variable, so check the container's first few log lines.
  2. A blocked origin. The Content Security Policy is enforced outside development and report-only within it, so a new third-party host works locally and is refused in production. See security headers.
  3. An unsubstituted placeholder. If a NEXT_PUBLIC_* value shows up in the browser as a .invalid hostname, the variable was added to the build but not to the entrypoint's list. See Docker.