.agents/skills/cloudflare-os-operator/references/troubleshooting.md

1# Cloudflare OS Starter Troubleshooting
2
3Use this reference only after reading the current checkout. Commands, validation, and deployment order may have changed.
4
5## Evidence Rules
6
7Collect the smallest useful, sanitized evidence:
8
9- Exact failing command and first error.
10- Root commit, submodule gitlink, and checked-out submodule commit.
11- Whether the worktree changed before or during the failure.
12- Node, pnpm, project-pinned Wrangler versions.
13- Boolean presence of `CLOUDFLARE_API_TOKEN`, never its value.
14- Sanitized `wrangler whoami` identity and target account ID.
15- Which deploy stages completed and their deployment/version IDs.
16- Resource names/IDs only when the audience is authorized to see them.
17- Narrow log events with secrets, prompts, headers, bodies, JWTs, and user data removed.
18
19Do not paste full production configs or logs by default. Never run a command that prints the active authentication token.
20
21## First Classification
22
23```text
24Failure
25|- Before submodule checkout: provenance or source access
26|- During install/build/test: local setup or compatibility
27|- During generated-config validation: deployment.jsonc
28|- During Wrangler dry-run: generated configuration, build output, or local validation
29|- During live deploy: auth, permission, product, route, provisioning, or resource
30|- After one or more Workers deployed: partial deployment
31|- Browser cannot reach the Router: route, TLS, Access, or app auth
32|- Router answers but a path fails: its service bindings or the Worker behind that path
33|- App loads but capability fails: service binding, Gatekeeper, AI, or storage
34|- Data is absent: binding identity, sharing boundary, or migration; never provision a replacement first
35`- Reports/logs are absent: signal semantics, sampling, time range, or binding
36```
37
38## Submodule And Provenance
39
40| Symptom | Check | Safe response |
41| --- | --- | --- |
42| `cloudflare-os/package.json` is absent | `git submodule status`, parent gitlink, `.gitmodules` | Initialize only from the reviewed configured source. |
43| SSH permission denied | Whether `.gitmodules` points to a private host and operator is entitled | Stop. Use authorized access, or approve a public mirror only after proving the exact SHA exists. |
44| Host key prompt | Expected source host and organization instructions | Do not accept an unknown key merely to proceed. Verify out of band. |
45| Submodule begins with `-` in status | It is not initialized | Run the checkout's documented init after source review. |
46| Submodule begins with `+` | Checked-out SHA differs from gitlink | Stop. Restore the pinned commit or review an explicit upgrade. |
47| Commit unavailable on proposed mirror | Mirror provenance or incomplete history | Do not substitute a nearby commit or branch head. Escalate. |
48
49Never use `git submodule update --remote` for setup or recovery.
50
51## Toolchain And Install
52
53| Symptom | Check | Safe response |
54| --- | --- | --- |
55| Unsupported engine or syntax | Node major and repository `packageManager` | Install the documented Node/pnpm versions. Do not regenerate the lockfile with another manager. |
56| `pnpm` lockfile changes on install | pnpm version, checkout cleanliness, dependency provenance | Stop and review. A setup operation should not silently become a dependency update. |
57| Package missing under submodule | Separate root and `cloudflare-os` installs | Run both documented installs. |
58| Build fails after submodule move | Parent wrapper compatibility, base configs, APIs, dependencies | Treat it as an incomplete upgrade, not an ordinary build issue. |
59| `wrangler.prod.jsonc` remains | Active check/deploy process and original failure | Preserve the error, prove no active process, then remove only stale generated files. Never edit/deploy them. |
60| Concurrent writes to generated config | Multiple operations in one checkout | Stop both safely and use separate worktrees. |
61
62## `deployment.jsonc` Validation
63
64| Error | Likely cause | Correction |
65| --- | --- | --- |
66| Missing value or placeholder | An active config branch still has a placeholder | Replace only active required values. Dormant AI/Reporter placeholders may be intentionally ignored by current code; verify the script. |
67| Account ID invalid | Not the exact 32-character account ID | Obtain it from an authorized Cloudflare surface. Do not infer it. |
68| Worker names not unique | Two active Workers share an identity | Select stable non-conflicting names after remote inventory. |
69| Worker name format invalid | Uppercase, punctuation, leading/trailing hyphen, or excessive length | Use a lowercase Cloudflare-compatible service name. |
70| Route requires exactly one mode | Both/neither `customDomain` and `workersDev` selected on the Router | Choose one route based on the approved operation mode. No other Worker takes a route. |
71| Custom domain invalid | URL/path/uppercase supplied instead of hostname | Use the exact lowercase hostname only. |
72| `publicBaseUrl` required on a workersDev route | Nothing in the checkout knows the account's `workers.dev` subdomain, and `PUBLIC_BASE_URL` plus the Context sharing boundary both need an origin | Set it to the exact origin the Router will answer on, scheme included. |
73| `publicBaseUrl` does not match `customDomain` | Two different hostnames configured for one origin | Correct one. Leaving them disagreeing would scope Context data to a host the deployment does not answer on. Prefer `null`, which derives the origin from the domain. |
74| `publicBaseUrl` must be an HTTPS origin only | Path, query, trailing slash, or `http` | Supply the bare origin, e.g. `https://os.example.com`. |
75| `publicBaseUrl` is not a workers.dev origin, or names another Worker | A host other than `https://<router-name>.<subdomain>.workers.dev` on a `workersDev` route | Supply the origin Wrangler actually serves the Router on. Only the account subdomain is unverifiable here; the Worker label and the `workers.dev` suffix are not, and a wrong host would become both `PUBLIC_BASE_URL` and the Context boundary. |
76| `sharingDomain` must be null or a non-empty string | Empty string or a non-string pinned boundary | Use `null` to follow the public origin, or the exact literal the existing collections live under. |
77| Access issuer invalid | Path, query, non-HTTPS, or malformed team origin | Copy the Access team HTTPS origin, without path. |
78| Audience blank/padded | Wrong Access value or whitespace | Copy the exact AUD tag from the matching application. |
79| Admin email invalid | Non-email or mismatched identity representation | Use the exact verified email expected by the backend. |
80| Storage must be null/string | Empty or malformed resource identifier | Choose automatic provisioning (`null`) or a proven existing ID/name. |
81| AI gateway name required | `aiGateway.enabled` is true with no `name` | Supply a reviewed existing or new gateway name, or set `enabled: false` and accept that users must bring their own model keys. |
82| `aiGateway.workersAi` does nothing | An older config still carries the key | Delete it. Workers AI rides the same gateway route as every other provider, and leaving the key in place would deploy a Workshop with no models. List `"cloudflare"` in `providers` to keep Workers AI. |
83| Sampling outside 0..1 | Percentage entered as 10/100 rather than fraction | Set a value from 0 through 1 based on volume and incident needs. |
84
85## Wrangler Identity And Permissions
86
87| Symptom | Check | Safe response |
88| --- | --- | --- |
89| Not authenticated | Project-pinned `wrangler whoami` | Use interactive login or the approved credential mechanism. Do not paste tokens. |
90| Unexpected account | `CLOUDFLARE_API_TOKEN` presence, Wrangler profile, configured account ID | Stop before mutation. Select the intended credential/account explicitly. |
91| Permission denied | Exact API operation and product | Grant only the current documented minimum. Do not switch to broad administrator access as a diagnostic. |
92| Product/binding unavailable | Account entitlement for Workers, KV, R2, Browser Rendering, Dynamic Worker Loaders, or enabled AI | Enable/obtain the product or revise the approved design. |
93| Worker already exists | Remote owner and deployment history | Continue only if it is the known deployment identity; otherwise choose another name. |
94| Secret required during Workshop deploy | Whether `providers` includes `google` or `aiGateway.accountId` names another account | Only those two require a token. Either install it interactively on the exact Workshop identity, or revise the configuration back onto the binding transport. |
95| `secret put` targets the wrong account/Worker | Missing account selector or unproven Worker ownership | Stop. It already deploys a version; inventory the mutation, contain if needed, and use an account-bound approved correction. |
96
97## Routing, DNS, TLS, And Access
98
99### Custom-domain deploy fails
100
1011. Confirm the zone is active in the same account.
1022. Check for existing DNS records, custom domains, Worker routes, or Access apps on the hostname.
1033. Do not delete a collision until ownership and impact are approved.
1044. Inspect certificate/domain status before retrying; repeated deploys do not fix an ownership conflict.
105
106### Redirect loop
107
108Check, in order:
109
1101. Capture the sanitized HTTP status and `Location` chain without cookies, JWTs, or query secrets; identify whether Access or the Workshop emits the loop.
1112. Access application covers the exact hostname being opened.
1123. `access.issuer` is the matching team origin.
1134. `access.audience` is the exact AUD for that application.
1145. No proxy, alternate hostname, or stale deployment points at a different app.
1156. Browser cookies/session belong to the expected Access organization.
116
117Use an incognito session after config correction. Do not bypass Access to prove the Workshop works.
118
119### Access returns 403
120
121- If everyone is denied, test the application policy and IdP mapping.
122- If one intended user is denied, test that identity against the policy and exact email claim.
123- If Workshop loads but `/admin` is denied, compare the non-sensitive authenticated email claim to `access.admins`, then prove the intended admin list was deployed to the running Workshop; Access policy and admin authorization are separate.
124- If an unauthorized user enters, stop exposure and tighten Access before investigating application features.
125
126### Evaluation route is unexpectedly public
127
128`workersDev: true` creates a route; it does not configure Access. Protect the exact `workers.dev` hostname the Router answers on with its own Access application.
129
130`scripts/deploy.ts` writes `preview_urls: false` on all Workers other than the router worker, because a Preview URL is an unauthenticated path around the Access-protected origin. Verify that in the derived config rather than assuming it, check the current Wrangler docs for changed defaults, and never patch the generated config to alter one.
131
132If a Worker other than the Router is publicly reachable, that is an Access bypass, not a routing nicety: the derived config gives the other five `workers_dev: false` and no routes, so an unexpected route means something outside this deploy created it.
133
134## Storage And Missing Data
135
136Treat missing data as a possible wrong binding, not an invitation to create storage.
137
1381. Stop writes.
1392. Compare current Worker name and binding to the last-known-good deployment.
1403. Identify the bound KV namespace or R2 bucket in the target account.
1414. Determine whether automatic provisioning created a new empty resource.
1425. Confirm the old resource still exists and its ownership/data classification.
1436. Restore a known-good binding only after compatibility and approval.
1447. Verify reads and writes with non-sensitive test data.
145
146Never copy, merge, delete, or adopt production data automatically. A Worker rename combined with automatic provisioning is a prime suspect when the application appears empty.
147
148### Context collections are missing but the KV namespace is right
149
150Suspect the sharing boundary before the storage. The Workshop's `GATEKEEPER_CONTEXT` binding carries a `sharingDomain` prop, and collections are visible only within it. `context.sharingDomain: null` derives it from the public origin, so a changed hostname silently changes the boundary while every KV binding stays correct.
151
152Compare the deployed binding's prop against the value the collections were written under, and restore it by pinning that exact literal. Do not re-provision the namespace, and do not treat the collections as lost: nothing was deleted.
153
154### Schedules are missing
155
156Suspect the Worker identity, and nothing else. The Scheduler keeps everything in Durable Objects belonging to its own script identity, with no KV, R2 or sharing-boundary prop involved, so a changed `workers.scheduler.name` is the only ordinary way its schedules disappear: the new script starts with empty DO namespaces while the old one keeps running with the data, bound to nothing.
157
158Compare the deployed `GATEKEEPER_SCHEDULER` binding on the Workshop and the Router against the previously deployed Scheduler Worker name, and restore that name. Nothing was deleted; do not delete the old Worker until the binding is proven.
159
160### Automatic provisioning denied
161
162Verify the exact required permission and product availability. If reproducible production binding is needed after provisioning succeeds, inventory the resulting resource IDs/names and plan an explicit binding update. Do not treat temporary generated Wrangler files as the durable inventory.
163
164## AI
165
166| Symptom | Check | Safe response |
167| --- | --- | --- |
168| Deploy says required secret is missing | Secret name on exact Workshop identity | Human installs it interactively after account/name confirmation. Re-enter approval before redeploy. |
169| 401/403 during inference | Token account and current required scopes | Correct least privilege; do not print or replace the token in chat. |
170| Gateway not found | Gateway name/account and direct vs gateway mode | Correct configuration after ownership review. |
171| Provider authorization/billing error | Unified Billing/BYOK/provider setup | Complete provider setup or disable that provider. |
172| Model picker is empty | `aiGateway.enabled`, and whether the deployed Workshop actually carries `CF_AI_GATEWAY*` vars | Most often a migration from the hosted deploy left `aiGateway` disabled: `wrangler deploy` replaces `vars` wholesale, so the hosted values are gone and `getAiGatewayConfig()` returns null. Enable it and point `name` at the instance's own gateway. See `docs/migrate-from-hosted.md`. |
173| Deploy demands a token unexpectedly | `aiGateway.accountId` and whether `google` is in `providers` | Those are the only two configurations that need `CF_AI_GATEWAY_API_TOKEN`. `accountId: null` and no `google` uses the pre-authenticated `WORKERS_AI` binding instead. |
174| Unexpected logs contain prompts | Gateway/logging configuration and export | Stop sensitive traffic, disable collection where supported, review exposure and retention. |
175| Unexpected cost or rate limit | Billing account, provider, model, retries, alerts | Disable funded models if needed, preserve usage evidence, and require cost review. |
176
177Secret existence is not runtime verification. Use only an approved non-sensitive low-cost prompt for an end-to-end connectivity check. Verify scope, billing, prompt collection, retention, and log access separately in their authoritative control surfaces.
178
179## Custom Gatekeeper
180
181| Symptom or finding | Risk | Response |
182| --- | --- | --- |
183| Gatekeeper still matches starter example | Low-stakes shared text only | Keep disabled until operator chooses policy; optional smoke test may be approved. |
184| Example verifier used with tenant/confidential data | Collaborators may observe unauthorized data | Disable immediately and design fail-closed observer verification. |
185| External read is not authorized as an observation | Audit/security boundary broken | Disable capability, load `write-gatekeeper`, add authorization and tests. |
186| Side effect executes directly | Approval boundary bypassed | Disable writes; redesign around submit/apply/reject/simulation. |
187| OAuth/token added to vars or tracked files | Credential exposure | Stop, revoke/rotate, remove from current and historical exposure paths, then redesign secret storage. |
188| Auto-provisioned singleton gains broad authority | Ambient capability may reach every user/workspace under policy | Reassess provisioning mode, identity, capability granularity, and admin policy. |
189
190Do not patch a serious Gatekeeper issue during a production deploy. Disable it, preserve evidence, and handle it as a separate security-reviewed change.
191
192## Error Reporter And Logs
193
194| Symptom | Check | Interpretation or response |
195| --- | --- | --- |
196| No Reporter Worker | `errorReporting.enabled`, configured name, deploy stage output | It may be disabled or its deploy failed. |
197| Reporter has no events | Whether a known explicit `reportIssue()` capture occurred | Empty is expected without explicit reports. It is not a catch-all. |
198| `error_report.dispatch.failed` | Service name, `ErrorReporter` entrypoint, deployment order/availability | Correct the binding or Reporter deployment; reporting must remain best-effort. |
199| Invocation exists but query is empty | Selected Worker, time range, sampling, field filter, export | Narrow the query and confirm retention/sampling. |
200| Wrong environment/release | Workshop service-binding props in derived config | Correct config and redeploy affected Workers together. |
201| Stack is not symbolicated | Matching source map/release and whether stack is runtime or explicit | Follow destination-specific source-map handling; do not expose maps publicly. |
202| Browser endpoint sends nothing | Reporter and rate-limiter bindings, build flag | Expected by default. Do not enable ad hoc. |
203
204The Reporter must have no public route. Its events remain visible to authorized account log readers and configured exports, so private ingress does not remove privacy obligations.
205
206## Partial Deployment
207
208The baseline sequence is Error Reporter, Context, Scheduler, Custom Gatekeeper, Workshop, then Router. Re-read `scripts/deploy.ts` for the current sequence.
209
210The Router is last because it binds every other Worker, which shapes recovery: a failure before it leaves the previous Router version serving the public URL against a partly-updated set of backends, so the live site can look healthy while the stack is mixed. A failure *of* the Router leaves the site on its previous version with new backends behind it.
211
212When deployment stops:
213
2141. Do not rerun immediately.
2152. Record the failed stage and deployment/version IDs for every completed stage.
2163. Compare new downstream RPC/storage contracts with the still-running old Workshop.
2174. Check live errors without exposing user data.
2185. Choose one approved path:
219   - **Complete:** fix the blocker and deploy the remaining/full compatible stack.
220   - **Roll back changed Workers:** only when current bindings, migrations, and data remain compatible.
221   - **Contain:** restrict Access or disable an affected Gatekeeper while a safe plan is prepared.
2226. Re-run `pnpm check`, re-enter approval, and perform full verification.
223
224For breaking contracts, do not alternate old/new code under the same service names. Use parallel service identities and a controlled binding switch.
225
226## Upgrade Regression
227
228Likely causes after a gitlink update:
229
230- An upstream Wrangler base-config field is overwritten by `scripts/deploy.ts`.
231- Shared RPC or Gatekeeper interfaces changed.
232- Durable Object migrations/classes changed.
233- Storage schema changed.
234- Authentication or AI environment requirements changed.
235- Dependencies or build outputs are no longer wrapper-compatible.
236
237Compare old and new base configs and generated configs section by section, especially `vars`, `services`, KV, R2, assets, migrations, and required secrets. Do not solve incompatibility by copying old generated Wrangler files forward.
238
239## Rollback Decision Matrix
240
241| Change | Version rollback alone? | Required review |
242| --- | --- | --- |
243| Code-only, same bindings/contracts/storage | Possibly | Current Cloudflare rollback support and post-rollback smoke test. |
244| Service binding or entrypoint changed | Usually insufficient | Restore compatible configuration and coordinated service versions. |
245| Access/DNS changed | No | Restore separately and retest allowed/denied paths. |
246| Secret changed | No | Restore/rotate separately without exposing values. |
247| KV/R2 binding changed | No | Restore intended binding; verify data compatibility. |
248| Durable Object migration/class changed | Often blocked or unsafe | Specialist migration review; never promise rollback. |
249| Data mutated | No | Product-specific data recovery/forward repair. |
250| First deployment partially completed | No prior system state | Complete safely or perform separately approved teardown. |
251
252After any rollback, rerun the complete live verification. Do not call rollback successful because a command returned zero.
253
254## Escalation Packet
255
256After two identical failures, provide:
257
258```text
259Goal and operation mode:
260Root/submodule commits and source provenance:
261Target account/route (redacted as appropriate):
262Exact failing stage and first sanitized error:
263Stages already mutated and deployment IDs:
264Expected vs actual behavior:
265Checks performed and results:
266Single changes attempted:
267Current Access/data exposure:
268Rollback/containment state:
269Specific decision or expertise needed:
270```
271
272Do not include secrets, JWTs, full logs, prompts, user data, request/response bodies, or unredacted private configuration.
273