docs/observability.md

1# Observability and error reporting
2
3Every Worker here writes structured logs with `console.*`, and the runtime adds uncaught exceptions. Both land in [Workers Observability](https://developers.cloudflare.com/workers/observability/), which can export logs and traces to any OTLP endpoint from the Cloudflare dashboard. That is the default path for getting telemetry into an external system, and it needs no code in this repository.
4
5The Error Reporter is the escape hatch. Upstream `reportIssue()` calls cross an RPC boundary into a private Worker you own, so exception events can go to an exception collector that groups occurrences, tracks releases, and pages someone. Browser reporting is a third input, privacy-sensitive and off by default.
6
7```mermaid
8flowchart LR
9    W[Workshop Worker] -->|structured logs and uncaught errors| O[Workers Observability]
10    O --> L[Workers Logs]
11    O --> T[Automatic traces]
12    O -->|dashboard OTLP export| D[External telemetry system]
13    W -->|reportIssue RPC| R[Private Error Reporter]
14    R -->|your transport| X[Exception collector]
15    R -. bundled demo, structured log .-> O
16    B[Trusted browser UI] -. disabled by default .-> W
17```
18
19## Choose the signal
20
21| Signal | Best for | Produced by | Starter default |
22| --- | --- | --- | --- |
23| Structured application logs | Lifecycle, decisions, handled degradation, debugging | Producer Worker `console.*` calls | Enabled |
24| Uncaught exceptions | Crashes not handled by application code | Workers runtime | Enabled |
25| Explicit issue reports | Known failure sites that deserve cross-service triage | Upstream `reportIssue()` calls through RPC | Enabled for the Workshop backend |
26| Invocation logs | One runtime record per invocation | Workers runtime | Disabled to control volume |
27| Automatic traces | Timing and dependency paths across bindings/storage | Workers runtime | Disabled until sampling is chosen |
28| Browser reports | Trusted first-party UI exceptions | Same-origin browser endpoint | Disabled |
29| Source maps | Mapping generated stack frames to source | Wrangler/build pipeline | Destination-specific |
30
31Error reporting does not replace logs or traces. The default Reporter sees only explicit `reportIssue()` calls. It cannot catch an exception that terminates a Worker before a capture site, and it does not turn every `console.error()` into an issue.
32
33## Quick start
34
35The annotated [`deployment.jsonc`](../deployment.jsonc) enables the backend Reporter by default:
36
37```jsonc
38"workers": {
39  "errorReporter": { "name": "your-cloudflare-os-errors" }
40},
41"errorReporting": {
42  "enabled": true,
43  "environment": "production",
44  "release": null
45}
46```
47
48Deployment creates a private [`packages/error-reporter`](../packages/error-reporter/README.md) Worker and binds the Workshop to its `ErrorReporter` RPC entrypoint. No route, public hostname, credential, or vendor account is required.
49
50The bundled implementation writes each event as one structured `console.error()` object. That demonstrates the collection path and gives you a queryable destination on the first deploy. Send the events somewhere richer when you want grouping, release tracking, and alerting; see [Replace the destination](../packages/error-reporter/README.md#replace-the-destination).
51
52To read what the bundled implementation collects:
53
541. Open [Workers & Pages](https://dash.cloudflare.com/?to=/:account/workers-and-pages), select the configured Error Reporter Worker, then open **Observability**.
552. Filter structured fields with the Query Builder. Start with `event = error_report`, then narrow by `service`, `environment`, `release`, `failureSite`, `severity`, or `handled`.
563. Use `occurrenceId` for one event and correlation/attribute fields to reconnect it to producer logs.
574. Use [`wrangler tail`](https://developers.cloudflare.com/workers/observability/logs/real-time-logs/) for live reproduction.
58
59Set `release` to the deployed commit or release identifier when your pipeline has one. Keep it `null` rather than inventing a value when it does not.
60
61To turn explicit issue forwarding off, set `errorReporting.enabled` to `false`. The Reporter is not built or deployed, the service binding is omitted, and upstream `reportIssue()` calls become their designed no-op. Normal producer logs and uncaught exceptions continue.
62
63## Structured logs
64
65[Workers Logs](https://developers.cloudflare.com/workers/observability/logs/workers-logs/) indexes object fields passed to `console.log()`, `console.warn()`, and `console.error()`. Prefer one event object with a stable discriminator over interpolated prose:
66
67```ts
68console.error({
69  event: "sync.failed",
70  operation: "context.sync",
71  durationMs,
72  error,
73});
74```
75
76Use consistent event names and bounded scalar context. Log at `error` when action is required, `warn` for best-effort degradation, `info` for notable lifecycle events, and `debug` for noisy breadcrumbs. Do not log whole requests, responses, model prompts, credentials, or arbitrary user data.
77
78The Error Reporter intentionally flattens the normalized event and trusted producer props into one object. This makes high-value fields directly queryable and preserves the upstream event's occurrence ID, timestamp, exception, truncation marker, HTTP facts, correlation IDs, and attributes.
79
80## Sampling and traces
81
82The deployment controls apply consistently to the router, Workshop, Context, Scheduler, MCP connector, Error Reporter, and Fork Horse's core and gatekeeper Workers:
83
84```jsonc
85"observability": {
86  "enabled": true,
87  "headSamplingRate": 1,
88  "logs": { "invocationLogs": false },
89  "traces": { "enabled": false, "headSamplingRate": 0.1 }
90}
91```
92
93- `headSamplingRate` accepts `0` through `1` and controls eligible observability telemetry.
94- `logs.invocationLogs` adds runtime invocation records. Keep it off unless those records answer a question your structured application logs cannot.
95- `traces.enabled` turns on [automatic tracing](https://developers.cloudflare.com/workers/observability/traces/). Start with a measured sample such as `0.1`, inspect volume and usefulness, then tune it.
96- `traces.headSamplingRate` is independent so traces do not need the same volume as logs.
97
98Sampling can hide individual events. Use full sampling during an incident or initial evaluation when volume permits, but do not promise that a sampled telemetry pipeline records every request.
99
100## Source maps
101
102[Worker source maps](https://developers.cloudflare.com/workers/observability/source-maps/) let Cloudflare remap uncaught Worker stack traces after an invocation. They are not available inside the Worker at runtime, so a stack copied into an explicit Reporter event remains the stack available at capture time.
103
104Browser source maps need a destination-specific workflow. Upstream can generate hidden maps when `VITE_FRONTEND_ERROR_REPORTING=true`, but a production pipeline must upload them to the selected destination and ensure they are not served as public static assets. The starter does neither implicitly.
105
106Record a stable `release` when uploading maps so the destination can select the matching artifact. Test symbolication with a deliberate non-sensitive error before relying on it during an incident.
107
108## Browser reporting
109
110Browser reporting is intentionally disabled. Enabling it safely requires all of the following:
111
1121. Build trusted first-party surfaces with `VITE_FRONTEND_ERROR_REPORTING=true`.
1132. Bind `FRONTEND_ERROR_REPORTER` to a private Reporter entrypoint.
1143. Bind `FRONTEND_ERROR_RATE_LIMITER` with a unique Rate Limiting namespace and an intentional limit.
1154. Keep reports on the existing same-origin `POST /api/client-errors` path so the deployment's sign-in identity, origin checks, payload limits, and normalization remain in force.
1165. Upload hidden source maps to the chosen private destination, then remove them from static deployment assets.
1176. Define retention, access, and deletion policy for browser stacks and coarse browser facts.
118
119The backend treats browser input as untrusted diagnostic data. Gatekeeper/configurator frames report through their known parent window; do not add direct cross-origin reporting from Worker-hosted frames, and never collect errors from gadget-authored/user-authored code automatically.
120
121## External destinations
122
123Workers Observability exports on its own. Reach for the Error Reporter only when exception events need handling that a telemetry pipeline does not give them.
124
125| Need | Path |
126| --- | --- |
127| Logs and traces in an external observability system | [OpenTelemetry export](https://developers.cloudflare.com/workers/observability/exporting-opentelemetry-data/) from the Cloudflare dashboard |
128| An existing log pipeline | [Workers Logpush](https://developers.cloudflare.com/workers/observability/logs/logpush/) |
129| Processing events in code before they leave the account | A [Tail Worker](https://developers.cloudflare.com/workers/observability/logs/tail-workers/) |
130| Exception grouping, release tracking, and alerting | [Replace the transport](../packages/error-reporter/README.md#replace-the-destination) inside `packages/error-reporter` |
131
132The first three need no code in this repository, and Sentry, Honeycomb, and Grafana all accept OTLP. Replacing the Reporter transport is worth it when exception events need destination-specific grouping, release handling, or alerting.
133
134Error reporting is not an alerting policy. Configure alerts in the system that owns the retained signal, and alert on actionable rates or service-level symptoms rather than every occurrence.
135
136## Security and privacy
137
138Exception messages and stacks may contain sensitive data even when nobody intended to log it.
139
140- Never put secrets, prompts, tokens, cookies, authorization headers, request/response bodies, or raw user documents in thrown errors or report attributes.
141- Keep Reporter Workers private and reachable only by service binding.
142- Use the minimum retention and destination access needed for incident response.
143- Treat browser metadata, failure sites, and correlation values as diagnostics, never identity or authorization.
144- Review external export regions, subprocessors, deletion behavior, and access controls before enabling a destination.
145- Prefer identifiers over payloads. Look up protected business data in its system of record after authorization.
146
147The upstream contract bounds strings and scalar attribute counts and refuses to traverse arbitrary thrown objects. That limits accidental volume; it is not a substitute for keeping sensitive material out of errors.
148
149## Troubleshooting
150
151| Symptom | Check |
152| --- | --- |
153| No Error Reporter Worker | `errorReporting.enabled`, Worker name, and deploy output |
154| Reporter exists but has no events | Confirm the failure crossed an explicit upstream `reportIssue()` capture site; inspect producer logs for uncaught or differently handled failures |
155| `error_report.dispatch.failed` in producer debug logs | Reporter service name/entrypoint, deployment order, and Reporter Worker availability |
156| Reporter invocation succeeds but query is empty | Observability enablement, sampling, time range, selected Worker, and external export settings |
157| Wrong environment/release | Service-binding props in the generated Workshop config; redeploy all affected Workers together |
158| Stack does not map to source | Matching source map/release, map upload, generated filename, and whether the stack came from runtime or explicit reporting |
159| Browser endpoint returns no report | This is expected by default; both the reporter and rate-limiter bindings are required before it dispatches |
160
161## Production checklist
162
163- Give every deployment a stable environment label and every releasable build a release identifier.
164- Verify one synthetic, non-sensitive explicit report after deployment.
165- Confirm the Reporter has no public route.
166- Confirm logs/traces are retained or exported where responders actually look.
167- Add an actionable alert outside the Worker when the destination supports it.
168- Document sampling changes during incidents.
169- Test source-map symbolication before an incident.
170- Recheck privacy and retention whenever browser reporting or a third-party destination is enabled.
171