Observability

Every Lunora function call produces telemetry across all three OpenTelemetry signals: an RPC event (one per query/mutation/action, carrying path, duration, ok/error, shard, and fan-out), log events (ctx.log), spans (ctx.trace), and metrics (ctx.metrics). A sink receives them.

Locally, none of it needs setup: the Studio groups error events into an Issues view, streams log lines in Logs, and renders ctx.trace waterfalls in Traces. To watch a deployed app you point a sink at an OpenTelemetry collector (your own, a vendor's, or the Lunora cloud), and the worker and any container ship the same telemetry over one protocol.

SinksCopy Anchor Link

A sink is passed to createWorker as the observability option. Lunora ships several; combine them with combineSinks.

import { analyticsEngineSink, combineSinks, otlpSink } from "@lunora/runtime";

export default createWorker({
    // …schema, functions…
    observability: combineSinks(
        otlpSink({ endpoint: env.LUNORA_OTLP_ENDPOINT, token: env.LUNORA_OTLP_TOKEN }),
        analyticsEngineSink({ dataset: env.ANALYTICS }),
    ),
});

sentrySink bundles no Sentry client and takes no dsn. It forwards to a capture callback you wire to your own client, so the runtime keeps no dependency on @sentry/*; constructing it without one throws:

sentrySink({ capture: (event) => Sentry.captureMessage(event.name) });

Sinks that carry RPC events accept onlyErrors to drop successful RPCs and export only failures. It defaults to false everywhere except sentrySink, which defaults to true — capturing every successful RPC would flood a Sentry project — so pass onlyErrors: false there to get all events. pipelineLogSink is log-only, so it has no such option; log events always pass through regardless. All four network/log sinks (otlpSink, webhookSink, pipelineLogSink, plus sentrySink when you wire its captureLog) forward ctx.log lines; webhookSink takes a transformLog redactor mirroring its RPC transform.

:::caution[Upgrading a webhookSink] webhookSink now ships ctx.log lines (message and structured fields, which may carry user input) in addition to RPC events; previously it shipped neither. Two consequences for an existing config:

• onlyErrors does not gate log lines (only RPC events); every ctx.log call egresses to the endpoint.
• The RPC transform redactor does not cover log lines, so add a transformLog (same fail-closed contract) if you scrub PII before it leaves the worker.

Set transformLog (or point the sink at a trusted endpoint) before upgrading if your handlers log user data. sentrySink (opt-in via captureLog) and otlpSink don't have this expansion. :::

otlpSinkCopy Anchor Link

otlpSink({
    endpoint: env.LUNORA_OTLP_ENDPOINT, // required — the collector base URL
    token: env.LUNORA_OTLP_TOKEN, // optional — sent as `Authorization: Bearer <token>`
    headers: { "x-lunora-deployment": env.LUNORA_DEPLOYMENT_ID }, // optional correlation headers
    serviceName: "my-app", // optional — the `service.name` resource attribute (default `"lunora"`)
    onlyErrors: false, // optional — export only error spans
});

Read endpoint/token from the environment so the platform can inject them at deploy time with no code change. Exports are fire-and-forget; on a Worker each is registered with waitUntil so it outlives the response.

Batching is on by default. Events are buffered and shipped as one POST per signal per flush — whichever of 512 buffered events, a 200 ms window, or the invocation boundary comes first. A Worker is capped at 50 (free) / 1000 (paid) subrequests per invocation, so a per-event POST is a correctness problem, not just an efficiency one. Pass batch: false to restore one POST per event, or batch: { maxItems, maxDelayMs } to retune the window.

Resource attributesCopy Anchor Link

A resource is the identity of the thing producing telemetry: the service, its version, where it runs. It rides on every span, log record, and metric, so a collector can group "all signals from v1.2.3 of the checkout API in Frankfurt" without you stamping it on each call.

otlpSink({
    endpoint: env.LUNORA_OTLP_ENDPOINT,
    serviceName: "checkout-api", // service.name
    serviceVersion: env.CF_VERSION_METADATA?.id, // service.version — the binding is `{ id, tag, timestamp }`, not a string
    serviceNamespace: "storefront", // service.namespace — when several services share a name
    deploymentEnvironment: "production", // deployment.environment
    resourceAttributes: { "service.instance.id": env.INSTANCE_ID }, // anything else
});

resourceAttributes is a free-form bag and wins over every convenience field above it, so it can also override one.

Auto-detection is opt-in with detectResources: true. The worker then reads service.version (from SERVICE_VERSION / CF_VERSION_METADATA — the version_metadata binding's tag, else its id — / VERCEL_GIT_COMMIT_SHA / GITHUB_SHA / COMMIT_SHA), deployment.environment (DEPLOYMENT_ENVIRONMENT / ENVIRONMENT / NODE_ENV), and cloud.provider / cloud.region (the Cloudflare colo the request landed in). A value it cannot determine is omitted rather than guessed, and anything you set explicitly always wins.

It is off by default because detection is a guess about your deployment: a stray NODE_ENV in a binding would otherwise silently label every span. Detection runs at most once per request, so turning it on costs nothing per log line or span.

:::note[Shard-side signals carry the explicit attributes only] ctx.log, ctx.trace, and ctx.metrics originate inside the shard, which has no env of its own, so host-detected attributes are not attached to them. Set the values explicitly if you need them to match across the worker span and the shard spans of the same trace. :::

pipelineLogSinkCopy Anchor Link

Persist every ctx.log line durably to a Cloudflare Pipeline → R2, which gives you a queryable log store (read back with R2 SQL) in your own account, with no cloud required. It is the durable counterpart to otlpSink (which streams to a collector). Only log lines are stored; RPC-span metrics belong in analyticsEngineSink.

import { pipelineLogSink } from "@lunora/runtime";

pipelineLogSink({ pipeline: env.LOG_PIPELINE });

Each record carries message, level, functionPath, fields, traceId, spanId, shardKey, userId, and ts. The send is registered with the request's waitUntil so it survives isolate teardown.

Reading the durable archive backCopy Anchor Link

pipelineLogSink only writes. To read those records back, with no cloud involved, use createPipelineLogReader, which builds a safe, keyset-paginated R2 SQL query over the Iceberg table the Pipeline lands in, or run the lunora logs --durable CLI command.

This needs operator setup; it is not code-only. The reader queries a table that only exists once you have wired the Pipeline to an R2 Data Catalog (Iceberg) table whose schema matches the written columns below, and provided read credentials. Until then the CLI fails closed with a clear message.

1. Create the destination table with a column per written field. The reader owns these names (its DEFAULT_LOG_COLUMNS), matching exactly what the sink writes:

If your table types fields as a string column (so R2 SQL can query it), turn on serializeFields so the sink stores it as a JSON string; the reader parses it back to an object on read:

pipelineLogSink({ pipeline: env.LOG_PIPELINE, serializeFields: true });

Renamed a column in your Iceberg schema? Pass a columnMap to realign the reader without touching the writer.

2. Provide the R2 SQL credentials as env vars (also used by ctx.r2sql):

• R2_SQL_ACCOUNT_ID: the Cloudflare account that owns the bucket/catalog
• R2_SQL_TOKEN: an API token scoped to R2 SQL read + R2 Data Catalog + R2 storage
• R2_SQL_BUCKET: the R2 bucket (warehouse) the catalog runs against

3. Read it back, from code:

import { createPipelineLogReader } from "@lunora/runtime";
import { createR2Sql } from "@lunora/bindings/r2sql";

const reader = createPipelineLogReader(createR2Sql({ accountId, apiToken, bucket }), { namespace: "default", table: "logs" });

const page = await reader.query({ minLevel: "warn", sinceTs: Date.now() - 3_600_000, limit: 200 });
// page.rows — newest first; page.nextCursor — pass as `cursor` for the next page

or from the CLI:

# newest 200 warn+ lines from the last hour, as a table
lunora logs --durable --table logs --namespace default --min-level warn --limit 200

# one JSON object per line, filtered to a function-path prefix (pipeable to jq)
lunora logs --durable --table logs --function-prefix "messages:" --ndjson

# resume the next page with the opaque cursor token the previous run printed
lunora logs --durable --table logs --cursor eyJ0cyI6MTczNzQ2MDAwMDAwMH0

The reader paginates by keyset on ts DESC, not OFFSET, so deep pages stay cheap over Iceberg. The predicate is inclusive (WHERE ts <= cursor) and the cursor carries a seen set of the boundary millisecond's already-returned rows, so ties are neither dropped nor duplicated — pass page.nextCursor (or the CLI's printed token) back whole. A bare ts is accepted for convenience, but without seen it re-emits every row sharing that millisecond. Filters (sinceTs/untilTs, level, minLevel, functionPathPrefix, traceId, shardKey, userId) are all inlined as escaped SQL literals, so user values can never inject.

…or from the Studio. The Studio Logs → Archive feed reads the archive back through an admin-gated route (the worker holds the R2 SQL credentials and runs the reader, so the token never reaches the browser).

For a Vite-first / scaffolded app there's nothing to wire in code: the generated worker entry already calls createWorker({ logArchive: … }) for you, resolved from env. Set one more variable alongside the R2_SQL_* credentials:

• LUNORA_LOG_ARCHIVE_TABLE: the Data Catalog table (e.g. logs)
• LUNORA_LOG_ARCHIVE_NAMESPACE: the Iceberg namespace, optional (e.g. default)

That's the opt-in. With the table unset, the Archive feed stays "not configured".

For a hand-written worker (or to pass a columnMap), set logArchive on createWorker directly:

import { createWorker } from "@lunora/runtime";

export default createWorker({
    // …your shardDO, adminToken, etc.
    logArchive: { namespace: "default", table: "logs" },
});

The Archive feed offers the same function-prefix / user / min-level filters and a Load more button that pages via nextCursor. Until logArchive and the R2_SQL_* env vars are set, it shows a "not configured" panel that links to the setup above, never an error. (The live Requests and Errors feeds keep reading the bounded in-DO log, which resets on hibernation; the Archive feed is the durable, unbounded history.)

Structured logging with ctx.logCopy Anchor Link

ctx.log spans the full OpenTelemetry severity ramp (trace, debug, info and its log alias, warn, error, fatal) and takes either console-style values or a structured message + fields object:

export const placeOrder = mutation.mutation(async ({ ctx, args }) => {
    // Console-style: any number of values, joined into the message.
    ctx.log.debug("placing order", args);

    // Structured: a message plus a fields object. The fields become
    // filterable/indexable log-record attributes.
    ctx.log.info("order placed", { orderId: order._id, total: order.total });

    // Bind context once with `.with(...)`; every line inherits it
    // (per-call fields win on a key clash).
    const log = ctx.log.with({ orderId: order._id });

    log.warn("inventory low", { sku });
    log.fatal("charge failed", { code: err.code });
});

The (string, object) shape is the structured form; every other shape is console-style. The rendered message and the structured fields reach the dev terminal, Workers Logs, the Studio Logs panel, and any sink; the raw positional args of a console-style call reach only the in-process onLog sink you control.

:::caution[Behavior change] A two-argument call whose second argument is a plain object, such as ctx.log.info("saved", user), is now the structured form: the message is "saved" and user becomes fields, instead of being rendered into the message as saved {…}. Calls with a non-object second argument, or three or more arguments, are unchanged. If you relied on the object being folded into the message text, pass it as a third argument (ctx.log.info("saved", "-", user)) or pre-render it. :::

Tracing sub-operations with ctx.traceCopy Anchor Link

Every dispatch is already one span, named after the function path. That tells you a request took 900ms; it doesn't tell you which part took 900ms. ctx.trace wraps a sub-operation so it becomes its own span nested under the request:

export const checkout = action.input({ cartId: v.id("carts") }).action(async ({ ctx, args }) => {
    const cart = await ctx.trace("cart.load", () => loadCart(args.cartId));

    // Attributes are structured like log fields, and become span attributes.
    const charge = await ctx.trace("stripe.charge", () => stripe.charges.create({ amount: cart.total }), { cartId: cart._id });

    // Nesting is explicit: the body receives a tracer bound to its own span,
    // and calling that is what makes a child.
    await ctx.trace("fulfil", async (trace) => {
        await Promise.all([trace("reserve.stock", () => reserve(cart)), trace("email.receipt", () => sendReceipt(charge))]);
    });
});

The body's value is returned unchanged, and a throw is recorded as an error span and then re-thrown: this is instrumentation, never flow control. Recording is best-effort, so a failing sink can't turn a working handler into a broken one.

:::note[Why the tracer is passed in, rather than nesting being implicit] It would read nicer if a bare ctx.trace inside another span's body were automatically its child. That can't be done correctly here: with Promise.all([trace("a", …), trace("b", …)]), b starts while a is still open, so an "innermost currently-open span" rule records b as a child of a rather than its sibling, and parallel fan-out is one of the main things a tracer is for. Telling "called inside a" apart from "called concurrently with a" needs AsyncLocalStorage, which Lunora's Durable Objects deliberately don't require. Passing the parent is correct in every case, and visible where it happens.

Calling ctx.trace inside a body instead of the passed tracer isn't an error; that span is just parented to the request rather than the enclosing span. :::

A span created inside a function invoked via ctx.runQuery / runMutation / runAction is attributed to the outer entrypoint's function path, since the composed call reuses its context. ctx.log follows the same rule.

Spans share the dispatch's trace id with its ctx.log lines and with any container the handler calls (the same traceparent is propagated), so one trace stitches together worker, shard, and container.

Joining a trace from upstreamCopy Anchor Link

Everything above happens inside one Lunora deployment: the worker mints a trace and propagates it down. If something in front of the worker already started a trace (an API gateway, a service mesh, another service), joining it needs one more decision, because the W3C traceparent header arrives from the caller.

By default Lunora ignores it and starts its own trace. Continue an upstream trace with trustInboundTraceContext:

export default createWorker({
    // …schema, functions…
    trustInboundTraceContext: true, // nothing untrusted can reach this worker
});

When the upstream is trusted, the dispatch adopts its trace id, parents its span under the upstream span, and carries tracestate onward, so the waterfall stitches end to end across services.

:::caution[Why this is off by default] traceparent is supplied by whoever called the worker. On a worker an untrusted client can reach directly, honouring it lets that client choose which trace its spans and ctx.log lines join, grafting entries into another tenant's waterfall in a shared collector. Because the head-sampling decision is derived from the trace id, it also lets that client choose its own sampling outcome.

true is the right answer when the worker is genuinely unreachable except through a front door you control. Check that it really is: a *.workers.dev route left enabled, or a hostname outside your Access policy, is a second front door with no gate on it. Use "mtls" when the check itself has to carry the proof: cf.tlsClientAuth is set by the Cloudflare edge and a caller cannot forge it, unlike a header.

Error traces are unaffected either way: a trace that produced an error is kept whole regardless of the sampling decision, and that decision is the worker's, never the caller's. :::

If an inbound traceparent is dropped while this option is unset, Lunora logs a one-time hint naming it, because a broken waterfall shouldn't be silent. Setting the option to false keeps the behaviour and silences the hint.

:::note[Cloudflare Traces and its incoming propagation policy] Cloudflare Traces adds a zone-level incoming propagation policy that decides whether the edge accepts a W3C traceparent from upstream. Lunora still treats the traceparent that reaches the worker as caller-controlled: Cloudflare does not document a forge-proof header the edge mints, so the worker cannot tell an edge-issued context from one a client sent. Keep trustInboundTraceContext off unless the zone policy guarantees that no client-supplied traceparent reaches the worker (for example, the zone strips or replaces it and the worker has no other route in). Adopting Cloudflare's native OpenTelemetry API is tracked in #939. :::

:::tip[Keep span names low-cardinality] Put the varying part in the attributes, not the name: ctx.trace("stripe.charge", …, { orderId }), never a name interpolated from the order id. A name built from an id makes every span its own group in a collector, which is exactly what attributes exist to avoid. :::

Locally, the Traces panel in the Studio renders recent waterfalls from an in-memory ring on the shard: recent activity on this instance, reset on hibernation. It is a development readout, not a trace store. For retention and cross-instance search, point otlpSink at a real collector, where each span is exported as an OTLP INTERNAL span carrying its parentSpanId.

Expanding a trace gives an elapsed-time ruler over the bars, a per-span detail block (span id, kind, the full attribute bag, the error, and any span.addEvent / span.recordException events), and the ctx.log lines the same dispatch emitted, joined by trace id. Errors only narrows the list to traces where something threw; the search box matches a span name or span id as well as the trace's own identifiers, so an id pasted from an error report finds the trace containing it. Both Logs views carry a Trace link on rows emitted inside a dispatch, opening the waterfall on the shard it came from.

:::caution[The Requests view's Trace link is best-effort] The durable request log records each dispatch's trace id, but it outlives the span ring it points at: the log survives hibernation and the ring does not. A Trace link on a row older than the current DO instance therefore lands on an empty Traces panel, because the trace has aged out locally. The id itself stays valid wherever otlpSink ships spans, which is where a deployed app's traces are retained; it is also emitted on the Logpush console event, so a SIEM can join a request to its spans. :::

Attributing a spike to a deployCopy Anchor Link

Bind Cloudflare's version metadata and the runtime stamps deploymentId / versionTag onto every durable request-log row and every emitted Workers-Logs event:

// wrangler.jsonc — every `lunora init` template ships this
"version_metadata": { "binding": "CF_VERSION_METADATA" },

That turns "latency doubled some time this afternoon" into a group-by rather than a graph-correlation exercise: filter the Requests view or the Workers Logs Query Builder on the deploy id and the before/after split is exact. It costs nothing when unused, and nothing reads it but the stamp.

Pair it with "observability": { "logs": { "invocation_logs": true } }, which the templates also ship — the Query Builder groups on the per-invocation summary line, and without it only explicit console events survive, leaving the group-by nothing to group.

From a Lunora log to Cloudflare TracesCopy Anchor Link

On Cloudflare, every request carries a Ray ID: the edge stamps it as the cf-ray header, and it is the key Cloudflare Traces, Workers Logs and Security Events are searched by. Lunora reads it at the Worker entry and carries it through the request:

The value is the bare 16-hex id, with the data-center suffix (-FRA) dropped. That suffix names a colo, not the request. cloudflare.ray_id is the same attribute Cloudflare's own Workers traces use, so one collector query on it finds both Lunora's spans and the platform's spans for a request.

To jump from a Lunora log line to the Cloudflare trace:

1. Copy rayId off the log line: the Workers Logs event, the onRpc event, or the cloudflare.ray_id attribute in your OTLP backend.
2. In your worker's observability view in the Cloudflare dashboard, filter traces (or logs) on cloudflare.ray_id equal to that value.

The reverse works too. A Ray ID taken from a Cloudflare error page or a support ticket finds the matching Lunora events by filtering on rayId / cloudflare.ray_id.

The Ray ID is informational only. It is never used for authorization or routing, it is not echoed onto client-visible responses, and a malformed value is dropped rather than logged. It is a Cloudflare-only signal: wrangler dev (Miniflare) does not set cf-ray, and neither does any other host, so locally every event simply has no rayId. Dispatches that no HTTP request started, such as alarms, cron jobs and subscription re-runs, have none either.

The same holds for work dispatched over a WebSocket: a subscription's first evaluation, stream queries, and the onConnect / onDisconnect lifecycle hooks. The upgrade request does carry a cf-ray, but the worker strips every x-lunora-* header from the upgrade before forwarding it to the shard (a security boundary, so a client cannot forge the server-minted ones). The Ray ID does not cross that hop, and the socket's later frames are not HTTP requests with a Ray ID of their own. Correlate socket-driven work by its traceId instead.

Wiring Lunora's spans into Cloudflare's native trace tree, rather than linking the two by Ray ID, is tracked in #939.

Generation spans (AI)Copy Anchor Link

An AI model call is a span with the OpenTelemetry gen_ai.* attributes on it: the model and, once the call returns, its token usage. Two paths emit them:

• @lunora/ai traces every defineRag embedding call as a generation span (gen_ai.operation.name: "embeddings", gen_ai.request.model) whenever the bound context carries ctx.trace. Wrap your own generateText/streamText calls in ctx.trace("…", fn, { "gen_ai.request.model": model }) for the same.
• @lunora/agent emits a generation span per model turn (and a span per tool call), with token usage, through its otlpTelemetry integration. Add it to defineAgent({ telemetry: { integrations: [...] } }) to opt in.

A generation span carries gen_ai.request.model and, from the model result, gen_ai.usage.input_tokens / gen_ai.usage.output_tokens; with the agent integration's recordInputs / recordOutputs it also carries the prompt / completion (off by default). The Lunora cloud reads these to show the model, token counts, and cost on the Traces waterfall.

Span durations on WorkersCopy Anchor Link

A 0 ms span means "no I/O happened here", not "this was fast".

The Workers runtime pins Date.now() to the time of the last I/O operation. It is a Spectre mitigation — a high-resolution timer is half of a cache-timing side channel — so it is a deliberate security property of the platform, not a bug, and it applies to performance.now() equally. Every duration Lunora records is Date.now() - startTs, so:

• A span wrapping pure computation reports 0, whether that computation took a microsecond or forty milliseconds.
• A parent span whose child performed no I/O shares the child's exact start and end, so the two tie on the waterfall.
• Durations across an await that actually does I/O — a ctx.db read, a ctx.fetch, a model call, a Durable Object hop — are real. Those are the spans worth reading, and in practice they are the ones that cost anything.

There is no workaround at the framework level: no clock available to a Worker advances without I/O. What you can do is make the spans you care about wrap I/O, and read a 0 ms CPU span as "unmeasurable" rather than "free".

This is invisible in local development. wrangler dev and Miniflare run workerd without the production mitigation, so locally the clock advances normally and a CPU-only span shows a plausible non-zero duration. The same span collapses to 0 ms in production. A timing profile that looks fine locally is not evidence that it will hold.

Database telemetry (ctx.db and ctx.sql)Copy Anchor Link

With a sink configured, ctx.db and the action-only ctx.sql (Hyperdrive) are instrumented automatically. One sink field, instrumentDatabase, sets the level for both:

export default createWorker({
    // …schema, functions…
    observability: { ...otlpSink({ endpoint: env.LUNORA_OTLP_ENDPOINT }), instrumentDatabase: "spans" },
});

The two surfaces keep separate tallies, because they are different cost classes — a ctx.db call is a local SQLite read inside the Durable Object, a ctx.sql call is a network round trip to your own Postgres/MySQL. Merging them would hide which one a slow request was bound on.

A ctx.sql span's db.operation.name is the statement's leading keyword only (SELECT, INSERT, UPDATE, …, or OTHER), parsed past leading comments and parentheses and nothing further. It also carries db.response.returned_rows. It never records the statement text or its parameters, and sets no db.collection.name — finding the table means parsing the statement, which is neither cheap nor safe. A failed call sets error.type (the error class, or the LunoraError code); outside development the span's error message is that type too, never the driver's message, because an external database's errors routinely quote the offending row.

fuseCloudflareTraces treats the two surfaces identically: per-call database spans go to your sink but are not re-emitted as Cloudflare custom spans (only ctx.trace spans are), while the summary counters ride the dispatch root span, whose attributes are mirrored onto Cloudflare's invocation span.

A fused ctx.trace span that fails gets an exception event (span.recordException) and an error span status (span.setStatus), both with the redacted message, so Cloudflare's trace view shows it as failed. A successful span's status stays unset, as OTel instrumentation leaves it.

Cloudflare's invocation span is marked failed in two cases: when an RPC dispatch answers with a 5xx, whether or not the handler recorded any telemetry, and whenever a trigger such as an alarm throws. An RPC that answers with a 4xx, such as a validation, auth or not-found error, leaves it unset. Both methods are feature-detected, so on a runtime without them the span simply carries the error attributes.

If you run your own Drizzle instance against Hyperdrive outside ctx.sql, Lunora does not see those queries. cloudflare-drizzle-tracing's instrumentDrizzle() wraps such an instance in Cloudflare-native spans — note it records query text and parameters, which ctx.sql deliberately does not.

Trace samplingCopy Anchor Link

At real traffic, exporting every trace is mostly paying to store the same successful request a million times. sampling keeps a fraction of them, mirroring Cloudflare Workers' head_sampling_rate:

export default createWorker({
    // …schema, functions…
    sampling: {
        headRate: 0.1, // keep ~10% of traces (default 1 — keep everything)
        alwaysSampleErrors: true, // …but never drop a trace that errored (default)
    },
});

It governs trace spans only: the per-dispatch SERVER span and the ctx.trace spans beneath it. ctx.log lines and ctx.metrics measurements are never sampled: logs are the thing you go looking for when something broke, and a sampled metric is a wrong metric.

:::tip[sampling vs. onlyErrors] Both reduce what you export, differently. sampling is deployment-wide and keeps a representative fraction of whole traces, so you retain a true picture of normal traffic at a lower bill. A sink's onlyErrors keeps no successful RPC spans at all, which is cheaper still, but you lose the baseline you would compare a regression against. Reach for sampling first; use onlyErrors on a sink whose only job is failures (a Sentry or alerting destination sitting alongside a sampled otlpSink). :::

sampling is Lunora's own export decision. Cloudflare's Workers Logs and Traces sample separately, through observability.head_sampling_rate and observability.traces.head_sampling_rate in wrangler.jsonc. Lunora writes head_sampling_rate: 1 by default and never lowers a rate you set. Workers Observability is billed from 2026-12-01 (Free caps ingestion at 0.5 GB/day; Paid includes 50 GB per billing cycle, then $0.25/GB), so lunora doctor reports an observability-full-sampling info finding while either rate is still 1.

Traces are kept or dropped wholeCopy Anchor Link

The decision is deterministic per trace, derived from the trace id rather than a coin flip. That matters because a trace spans processes: the worker, the shard, and any container all reach the same verdict for the same trace, so you never get half a waterfall: the worker span kept and its ctx.trace children missing, or the reverse.

The worker decides once and propagates it: the sampled bit rides the traceparent it forwards, so the shard drops the matching spans without re-deciding. That includes a batched call (client.batch(...), and the coalesced replay of queued offline writes): one verdict covers the whole batch and every sub-request carries it, so a batch is kept or dropped as one thing rather than each entry deciding for itself.

Errors are never sampled awayCopy Anchor Link

alwaysSampleErrors (default true) is the tail bias: a trace that produced an error is exported whole even when the head decision dropped it. The shard holds a sampled-out trace's spans until the dispatch settles, then flushes them if anything failed and discards them otherwise. The hold is capped at 500 ctx.trace spans per trace, oldest dropped — a loop that traces per item past that exports only its tail.

So headRate: 0.01 does not mean "1% of your errors". It means 1% of your successful traffic, plus every failure. That is usually the shape you want: aggressive sampling on the boring path, complete traces on the interesting one.

A container applies the same bias to its own spans, and is told the verdict rather than guessing it: the shard stamps this dispatch's toggle onto every ctx.containers.<name> request beside the traceparent, so createContainerTelemetry({ request }) picks up both. One honest limit — a container is a long-running process with no dispatch boundary to re-decide at, so it keeps the span that errored rather than holding and re-exporting that span's siblings.

// inside the container's request handler
const telemetry = createContainerTelemetry({ request });

Pass alwaysSampleErrors explicitly to override what the request carries. A one-shot container with no request to read falls back to LUNORA_SAMPLE_ERRORS in its env, then to the same default as every other tier.

:::note[headRate and the trace id] headRate >= 1 keeps everything and <= 0 drops every non-error trace; in between, the last 8 hex characters of the trace id (its low 32 bits) are mapped onto [0, 1) and compared against the rate — the same rightmost portion OpenTelemetry's TraceIdRatioBased sampler uses, because an inbound id is not necessarily uniform across all 128 bits (a 64-bit id left-pads with zeros; X-Ray puts epoch seconds in the leading digits).

Because the trace id decides, which id is used matters. When the inbound trace context is untrusted (the default), Lunora keys the decision on a freshly-minted server-side id, so a caller cannot choose its own sampling outcome by picking a trace id. When you have opted into trusting the upstream, the upstream's trace id is used, which is what keeps a distributed trace whole across services. :::

Application metrics with ctx.metricsCopy Anchor Link

The third signal, alongside logs and traces. A trace tells you what one request did; a metric tells you what a million requests did.

export const checkout = action.action(async ({ ctx }) => {
    const started = Date.now();

    // "How many" — summed over time.
    ctx.metrics.count("orders.placed", 1, { plan: user.plan });

    // "How many right now" — replaces the previous reading.
    ctx.metrics.gauge("cart.items", cart.items.length);

    // "What's the distribution" — percentiles, not just a mean.
    ctx.metrics.record("checkout.latency_ms", Date.now() - started);
});

Each becomes an OTLP metric at POST {endpoint}/v1/metrics: a monotonic Sum, a Gauge, or a Histogram.

:::caution[Keep attributes low-cardinality] Attributes are the metric's dimensions, and every distinct combination is a separate time series. { plan: user.plan } is a handful of series; { userId } is one per user, which is how a metrics bill gets out of hand. Identifiers belong on a log line or a span, which is what those are for. :::

Nothing is pre-aggregated: one call is one exported measurement, with counter and histogram values carrying delta temporality for the collector to aggregate. That keeps the sink model identical to logs and spans. Batching means a hot loop costs buffered events rather than one POST each, but the measurements themselves are still one per call — sum locally and record once at the end rather than calling per iteration.

:::caution[Durable per-minute history is opt-in] The Durable Object can also keep a per-minute rollup of every measurement in a reserved per-shard SQLite table, so the Studio can chart a 24h local trend without an external collector. It is off by default because it is not free: with it on, each ctx.metrics.* call becomes a durable SQLite write on the request path, a billed, rate-limited storage op that competes with your app's own data for the shard's write budget. Turn it on with metricHistory: true on the sink you pass to createShardDO (or an object, { maxSeries, retentionBuckets }, to tune the caps). The live, cross-instance path is onMetric → your collector, and is unaffected either way. :::

Metrics have no local buffer and no Studio panel: their value is in the aggregate over time, which an in-memory ring on a hibernating instance can't represent. consoleSink prints them in dev; for anything real, point otlpSink at a collector. (The Studio's Metrics page shows framework-level per-function metrics, which are collected separately and always on.)

Delivery metrics (notify)Copy Anchor Link

@lunora/notify counts every notification send onto ctx.metrics for you, with no code, as long as the send runs on a ctx that carries ctx.metrics (every handler does). Two low-cardinality series:

A single send emits one notify.send measurement; a broadcast aggregates its outcomes into one measurement per (provider, status) bucket (its value is the bucket's count), not one per recipient. Each ctx.metrics.count is a durable write, so a large fan-out must not pay it per subscription.

accepted means the push/chat provider accepted the message, not that it was delivered or opened. Web Push and FCM give no delivery/open receipts, so the status stops at the send attempt. A failed send also emits one ctx.log.warn line ("notify <channel> delivery failed") carrying the error and, for push, the subscription and user ids, trace-correlated to the enclosing action and durably archived by the log sink. Successful sends and prunes stay off the log; they live on the metric.

Container telemetryCopy Anchor Link

Code running inside a container can't use a worker sink, because it is a separate process. @lunora/container/otel gives it a zero-config exporter that speaks the exact same wire contract, so container spans and worker spans land in the same collector side by side.

import { createContainerTelemetry } from "@lunora/container/otel";

// Reads LUNORA_OTLP_ENDPOINT / LUNORA_OTLP_TOKEN from the container env.
const telemetry = createContainerTelemetry();

// Time a unit of work — records an ok span, or an error span if it throws.
const result = await telemetry.trace("transcode", () => transcode(job), { jobId: job.id });

telemetry.emitLog({ level: "info", message: "done", attributes: { jobId: job.id } });

// REQUIRED before the process exits. Spans and logs are batched, so nothing has
// necessarily been sent yet — a job that exits without this reports nothing at all.
await telemetry.flush();

Which trace a span lands on depends on whether the instance has an anchor: a traceparent option, the inbound request (its traceparent header), or LUNORA_TRACEPARENT in the env. An anchored instance puts every span under that trace and stamps it on every log, so one request or one job reads as one thing. An unanchored instance gives each span its own trace and leaves logs unstamped. So in a long-running server, create the telemetry per request from the inbound request rather than once per process:

app.post("/transcode", async (request) => {
    const telemetry = createContainerTelemetry({ request }); // one trace per request
    // …
});

A module-scope instance is right for a one-shot job started with LUNORA_TRACEPARENT, where the whole process is one unit of work.

With no endpoint resolvable the exporter is a silent no-op (telemetry.enabled === false): trace still runs your work, it just records nothing. The same code runs unchanged locally and in the cloud.

Each POST is bounded by timeoutMs (default 10s), so a hung collector aborts instead of pinning a send in flight and stalling flush(); failures are handed to the optional onError callback and never break the container.

It takes most of the same resource attributes as otlpSink (serviceVersion, deploymentEnvironment, resourceAttributes, and an opt-in detectResources), so container and worker telemetry can be grouped by the same service and version. There is no serviceNamespace option here — pass resourceAttributes: { "service.namespace": "…" } to set it:

createContainerTelemetry({
    serviceName: "transcoder",
    detectResources: true, // service.version, deployment.environment, host.name, k8s.pod.name, process.pid
});

Detection here reads the container's environment, so unlike the worker it can also resolve host.name, k8s.pod.name (only when actually running under Kubernetes), and process.pid. As on the worker, it is opt-in and anything you set explicitly wins, including the deploymentEnvironment and serviceVersion env fallbacks, which only apply when detectResources is on.

Thread the endpoint/token into the container the same way any other config reaches it, by declaring them on defineContainer:

defineContainer({
    name: "transcoder",
    // …
    env: { LUNORA_OTLP_ENDPOINT: env.LUNORA_OTLP_ENDPOINT },
    secrets: ["LUNORA_OTLP_TOKEN"],
    // The collector host must be reachable from the container egress allow-list.
    allowedHosts: ["collector.example.com"],
});

The wire contractCopy Anchor Link

Both the worker otlpSink and the container exporter conform to one contract, so any OTLP-compatible collector (and the Lunora cloud ingest) accepts either without special-casing.

Transport. OTLP over HTTP with JSON encoding (not protobuf). Two endpoints, derived from the configured base endpoint (trailing slashes are tolerated):

:::note[The Lunora cloud ingest accepts more] Lunora's own sinks emit JSON, gzipped once the body clears 1 KiB (which a batched flush essentially always does). The cloud ingest accepts that, and also standard OTLP protobuf (Content-Type: application/x-protobuf, what most Collectors default to) on all three endpoints, so any OpenTelemetry SDK or Collector can ship to it unchanged. A third-party collector or proxy behind the sink has to decode content-encoding: gzip, or set batch: false and keep bodies under the threshold. A capped batch comes back as OTLP partialSuccess. Its authorization bearer is a scoped ingest key, telemetry-only so a leaked token can't deploy, which the platform mints per org and injects into each tenant as LUNORA_OTLP_TOKEN at deploy time (alongside LUNORA_OTLP_ENDPOINT). :::

Headers.

The x-lunora-* headers are a convention the cloud ingest reads to route telemetry; pass them through the headers option. A collector that ignores them still accepts the payload.

Encoding. Bodies are standard OTLP ExportTraceServiceRequest / ExportLogsServiceRequest JSON. Per the OTLP/JSON spec:

• traceId is 16 random bytes as 32 lowercase hex chars; spanId is 8 bytes as 16 hex chars (the documented exception to proto3 JSON's base64 bytes).
• timeUnixNano fields are the nanoseconds since the epoch as a decimal string (Lunora works in millis, so this is the millisecond value followed by six zeros, and is exact).
• resource always carries service.name, plus any other resource attributes configured or detected; the instrumentation scope.name is @lunora/runtime (worker) or @lunora/container (container).
• Attribute values follow the OTLP AnyValue union: strings as stringValue, booleans as boolValue, integers as intValue (a decimal string), floats as doubleValue.

Span shape. One span per RPC event (worker) or per trace/emitSpan (container):

Worker spans carry these attributes:

The http.* / url.* / server.* / user_agent.* keys are the OpenTelemetry HTTP semantic conventions, so a collector's built-in HTTP dashboards work against Lunora spans with no mapping. http.route deliberately carries the function path rather than the transport path (/_lunora/rpc), because that is the route actually being invoked.

A span that errored also carries a standard OTel exception span event with exception.type and exception.message, the canonical error representation collectors look for, alongside the error.type attribute.

Worker spans additionally carry parentSpanId when the trace was joined from upstream, and flags with the W3C sampled bit.

Container spans carry whatever attributes you pass, plus error.type when the work throws, and the same exception event on failure.

Log record shape. body.stringValue is the message; severityText is the upper-cased level; severityNumber maps as:

Worker log records also carry lunora.function_path, and lunora.shard_key / lunora.user_id / cloudflare.ray_id when known. Structured fields (below) become additional log-record attributes, and each record carries its dispatch's trace_id / span_id so a line links back to its RPC span.

PrivacyCopy Anchor Link

Spans carry error.type and error messages, and log records carry the rendered message, and either can include user-supplied input. Point endpoint only at a collector you trust, and use onlyErrors to narrow what leaves the deployment.

url.path is the raw inbound path. On the RPC transport that is a constant (/_lunora/rpc), but on the opt-in public REST surface a path can carry record ids or user identifiers, which is worth knowing before pointing a sink at a third party. http.route is always the templated function path, so it is the safe one to group and alert on.

Writing your own sinkCopy Anchor Link

A sink receives (event, context). The context is deliberately narrow: a waitUntil for keeping a send alive past the response, and a resourceAttributes() thunk resolving the small allowlisted bag described above. It does not expose the Worker env or the raw Request, because every registered sink (including your own) is handed the same context, and a single debug console.log(context) would otherwise print every secret binding and the caller's Authorization and Cookie headers.