Troubleshooting

Run the self test first. It is faster than this page and it knows which hop failed. Come here to interpret what it told you.

This page is indexed by symptom. For the jobs that have a right order — proving the egress, verifying a credential before you flip a source live, rotating one — see Procedures.

Install-time

agent-runner and synth-seed show Init:0/1 for the first seconds of a fresh install with the bundled Postgres — expected, and it is the install holding them back on purpose. Both workloads run their schema migration at boot, and from 0.14.0 each carries an init container, wait-for-postgres, that polls the bundled instance until it accepts connections and only then lets the main container start. On 0.13.x and earlier there was no gate, and the same two workloads instead restarted once or twice while Postgres came up — if that is what you are looking at, the rest of this entry still describes it.

What a normal first minute looks like on 0.14.0 with the bundled Postgres:

NAME                       READY   STATUS      RESTARTS   AGE
prism-postgres-0           0/1     Running     0          10s
prism-agent-runner-...     0/1     Init:0/1    0          10s
prism-synth-seed-...       0/1     Init:0/1    0          10s

then Postgres turns 1/1, the two Init:0/1 pods go PodInitializing and Running within a couple of seconds of it, and synth-seed ends Completed. RESTARTS stays at 0 on every pod. Measured on 0.13.0, Postgres takes about 14 seconds from pod start to Ready on an unremarkable node, and the gate waits up to postgres.startupWaitSeconds (180 by default) for a slower one. A Postgres pod that stays 0/1 Running well past that with Readiness probe failed events is a different thing from a slow one, and the entry below on the two refused-connection errors says how to tell them apart. Init:0/1 can last longer than Postgres does, though: the init container runs from the Postgres image, the largest in the release, and on a multi-node cluster the runner's node and the seed's node each have to pull it before the wait starts. That pull is outside the bound, and a registry problem during it shows as Init:ImagePullBackOff on these two pods. Nothing else waits: the app, Postgres and the MCP servers each start once and do not depend on the gate. The init container's own log says what it saw:

kubectl logs -n <namespace> <pod> -c wait-for-postgres

postgres at prism-postgres:5432 is accepting connections after 12s

On OpenShift with the default postgres.bundled: true, agent-runner and synth-seed do not appear at all — no pod, and the install ends with the post-install hook timing out. The init container has to run as the Postgres image's own user (UID 999), and the restricted-v2 SCC refuses a UID outside the range it assigned the namespace, so the ReplicaSet and the Job are refused admission; kubectl describe replicaset and the Job's events carry the reason. It is the same root cause as the bundled Postgres not starting there: set postgres.bundled: false and point at a managed instance, as install step 1 says, and no gate is rendered.

If you run an external database (postgres.bundled=false), there is no gate and a refused connection will not converge — the database is not something this install is starting, so there is nothing to wait for, and a wait would only postpone the error below. The gate also could not run there: it uses the Postgres image, whose default user is root, so it has to pin runAsUser to that image's postgres user, and OpenShift's restricted-v2 SCC refuses a UID outside the range it assigns the namespace — the same reason the bundled Postgres itself cannot start on OpenShift, and the platform where an external database is the only option. Check that the instance is reachable from the namespace (firewall rule, private endpoint, network policy) and that postgres.external.host, .port, .database and the credential are right. Left alone, agent-runner crash-loops indefinitely and synth-seed exhausts its six retries and fails the post-install hook.

The two workloads report a refusal in different languages, so match the one you are looking at. In agent-runner's previous container you will find:

kubectl logs -n <namespace> <pod> --previous

Error: connect ECONNREFUSED <postgres-ip>:5432
    at async applySchema (/app/packages/db/src/migrate.ts:72:3)
    at async main (/app/apps/server/src/index.ts:55:3)

synth-seed is Python and says the same thing differently — a traceback ending:

ConnectionRefusedError: [Errno 111] Connect call failed ('<postgres-ip>', 5432)

When you see those two errors on 0.14.0 with the bundled Postgres on a fresh install, the gate gave up: Postgres did not accept connections within postgres.startupWaitSeconds, and the init container stood aside so the failure would be the familiar one above rather than a pod parked in Init:0/1 with no explanation. Its log ends with did not accept connections within 180s ... starting anyway. The question is then why the bundled Postgres is slow or absent, and the answer is on its pod, not on the two that are restarting: kubectl describe pod <release>-postgres-0 names an unbound volume, an image that will not pull, or a quota. It can also show a server that is up while the probe is what fails: Readiness probe failed: command timed out events on a pod whose own log already says listening on IPv4 address "0.0.0.0", port 5432 mean Postgres is running under a CPU cap too tight for pg_isready to answer inside postgres.readinessProbe.timeoutSeconds (5 by default; on 0.13.x the chart set no timeout and Kubernetes applied 1 second, and an install capped at 50m CPU never met it), so the pod is never Ready, the Service has no endpoints, and every client is refused by a ClusterIP in front of a server that would answer — raise the timeout or the cap. The restarts converge on their own once Postgres does listen, exactly as they did on 0.13.x. If Postgres is merely slow on your node — a first boot has to bind a volume, pull the image and run initdb — raise postgres.startupWaitSeconds and the churn goes away on the next install; above about four minutes raise helm install --timeout with it (5 minutes by default), because the synth-seed Job is a post-install hook and helm gives up on the hook before the gate would. The same two errors on an install that was running fine mean Postgres went away later — a restart, a node change, an upgrade rolling the StatefulSet — and the init container is no help there: init containers do not re-run when the main container restarts, so its log still reads is accepting connections, and the entry for a Postgres that will not start is further down.

On 0.13.x, expect one or two restarts of agent-runner and synth-seed, settling inside a minute. The count is not a fixed figure — it is however many attempts land before Postgres starts listening, so two installs of the same version on the same cluster can differ; a rehearsal install of 0.13.0-rc0 took two and three restarts and was equally healthy. STATUS may read Error or CrashLoopBackOff while the restart backoff runs. Once every pod reports Ready and the synth-seed Job reads Completed, the RESTARTS counts stop climbing and stay at the number they reached; a non-zero count on a settled 0.13.x install is a record of those first seconds, not something to clear.

When to start worrying, on any version: counts still climbing after a few minutes, or an error in --previous that is neither of the two above. A genuine fault gives a different error, and the entries below name them. If pods are Pending rather than restarting or initialising you are not looking at this at all; kubectl get events -n <namespace> will name a scheduling or quota reason instead.

Two reactions cost more than the problem, and both are common on a first install: raising a support case for an install that is in the middle of converging, and changing things underneath it — editing values, deleting pods, re-running helm upgrade. Neither makes it settle sooner, and the second restarts the clock.

agent-runner CrashLoopBackOff, logs show a vector extension error — your Postgres refused the CREATE EXTENSION IF NOT EXISTS vector the runner's boot migration runs, and the runner has no way around it. What to do depends on the provider: Azure flexible server refuses until VECTOR is in the azure.extensions server parameter, while Cloud SQL needs nothing and this error means something else — the connecting role lacking the privilege, or the wrong database. Install step 1a has the one psql line that tells you which.

Every chat answers "no answer from the agent", and the runner logs private outbound URLs are disabled — you are on 0.4.0 or 0.4.0-rc0. The runner refuses to fetch URLs that resolve to private addresses, and the MCP server addresses the chart generates are in-cluster services, so the very first turn fails while registering them. The self test says it precisely: Could not register MCP profile 'Agent Router Usage' (HTTP 500). Upgrading to 0.5.0 fixes it and needs no values change. If you cannot upgrade yet:

kubectl set env deploy/<release>-agent-runner \
  RUNNER_ALLOW_PRIVATE_OUTBOUND_URLS=1 -n <namespace>

That is a stopgap, and the reason to move it is legibility rather than durability. A kubectl set env variable does survive a routine helm upgrade — Helm preserves fields no chart version manages — but it does not survive helm uninstall and reinstall, helm upgrade --force, or anything else that recreates the Deployment, and a later chart version that renders the same variable overrides it silently. It is also in no values file, so nobody reviewing your configuration can see that it is load-bearing. From 0.5.0, anything of this kind belongs in agentRunner.extraEnv in your values file instead, where it is both visible and durable.

helm upgrade refuses with router.platformUrl is empty or router.proxyUrl is empty — working as intended, and cheaper than the alternative. Both Router URLs are required and neither has a sensible empty behaviour, so the chart refuses to render one rather than install it. router.platformUrl is your tenant's management-plane URL from your Router console, and it is required even under router.keyMode: byok, where no admin key is needed — the runner derives its Router hosts from it before it looks at any credential. router.proxyUrl is the data-plane gateway for the project in app.projectId, and it does not fall back to router.platformUrl. See install step 4.

On a chart before 0.6.0 there is no render error and the symptoms arrive later: an empty router.platformUrl is agent-runner CrashLoopBackOff logging TARE_PLATFORM_URL is not set, and an empty router.proxyUrl is a healthy install where every question fails, with the self test's inference endpoint line skipped as not configured. Set the value and helm upgrade.

helm upgrade refuses with app.projectId is empty — also working as intended, and the state it refuses is the quietest failure this chart had. The value is not required to be any particular thing: default is a real Router project and is the chart default, so an install that never touches this key is fine. Only "" is refused, and only because values.yaml sets the key explicitly, which makes an empty value a deliberate override rather than an omission. Set the Router project this install serves — the same project your inference key and router.proxyUrl belong to.

On a chart before 0.6.0 an empty app.projectId installed, and became default: the app's PROJECT_ID and the runner's allowedProjectIds both. The two agreed, every pod was healthy, the render was clean, and every question came back 403 from the gateway — because on a Router whose projects have their own gateways, default is the one project with no data plane behind it. It presents exactly like a bad credential, so the search goes to the key, the tenant, the Secret and the egress policy, and the wrong value is one that appears in no values file. If you are on an older chart and seeing that 403, check kubectl exec deploy/<release>-app -- printenv PROJECT_ID against the project your inference key was issued for before you touch anything else.

helm upgrade refuses with sources.github is "Live" — also working as intended, and usually one keystroke to fix. Source modes are matched exactly and in lower case: sources.github, sources.jira and sources.directory take synthetic or live, sources.spend takes synthetic, indexed or live, and indexed means nothing on the other three. The message names the key and the value as you typed it, so read that value against the list — Live, LIVE and a misspelling all land here, and so does an empty value, which names no mode at all. The check runs before anything is applied, because every component in the chart is handed the same four values and they have to agree.

On a chart before 0.7.0 a capitalised mode was refused by nothing at all, and the install was the quiet kind: the chart rendered no ingest for that source — no backfill, no schedule, not even the pre-flight check — while the app lowercased the same string and served the source as live, so the mirror was empty on install and stayed empty. Where it surfaced was the self test, as a red live but 0 rows line that reads as a data problem rather than as a typo. An empty or misspelt mode got as far as the synthetic seed job, which exited with SOURCE_JIRA must be one of 'synthetic' / 'live', got '' after the upgrade had applied everything. If a source is unaccountably empty on an older chart, check the mode as spelt — helm get values <release> -n <namespace> — before you go looking at credentials.

Every chat turn fails with Agent 'prism' already exists but is not reachable by 'prism-provisioner' — the install predates the change that gave Prism a fixed owner for the agent everybody shares, and the upgrade cannot migrate the row it left behind: the agent is owned by an identity the app can no longer act as, and its name is unique, so it can be neither used nor replaced. Nothing else is wrong and no further upgrade fixes it. It is a one-off repair against the runner database, with a script that ships in the chart tarball — Re-own the shared agent is the procedure, and its dry run confirms the diagnosis before anything is written.

Pods rejected with must specify limits.cpu — the namespace quota requires every container to declare CPU limits, and something is not declaring them. The chart sets them on every container, so this is an override that emptied a resources block — put the limits back. (Before 0.13.0 the bundled Postgres declared none of its own and this was the chart's fault, not yours; on that release the pod needs either a LimitRange in the namespace or an upgrade.)

If it is the bundled Postgres that will not start, and it was running until something restarted it, the cause is usually a quota added after Prism was installed. A ResourceQuota is enforced when a pod is created, not while it runs, so a datapond started before the quota keeps running and cannot be replaced — and the failure appears at a node drain or a rescheduled pod rather than at the upgrade that introduced it. Upgrading to 0.13.0 or later gives the pod its own limits; until then, a LimitRange in the namespace supplies a default and lets it be recreated.

Pods rejected with exceeded quota — a different failure with a similar shape, and the important thing is which limit it names:

Error creating: pods "prism-app-..." is forbidden: exceeded quota:
  requested: pods=1, used: pods=10, limited: pods=10

Error creating: pods "prism-synth-seed-..." is forbidden: exceeded quota:
  requested: limits.cpu=500m, used: limits.cpu=3750m, limited: limits.cpu=4

A ResourceQuota caps pods, CPU and memory independently, so raising the pod count can leave you refused a minute later on limits.cpu with a message that looks like the same problem. Jobs retry, so the CPU form repeats for the same Job and the pod count appears nowhere in it.

Size against the upgrade rather than the steady state — the synthetic-seed hook runs while the ingest jobs are still going, so the peak is roughly double. Compute, and the namespace quota you need carries the per-component figures, generated from the chart, and what to leave room for.

Image pull failures — global.imageRegistry must include the repository prefix the images were imported under, and your image pull secret must exist in the namespace.

Postgres connection failures that look like a DNS or host problem — check the password in runner-database-url is percent-encoded. An @ or / in the password produces a DSN that parses to the wrong host, and the error you get back describes the wrong host rather than the real fault.

agent-runner failing its /health probe, with no logs at all — this is almost never a probe or a networking fault. The runner dies inside its boot migration, before it writes anything to stdout, so the running pod's logs are empty and only the previous container has the evidence:

kubectl logs -n <namespace> deploy/<release>-agent-runner --previous

An UNABLE_TO_VERIFY_LEAF_SIGNATURE there means TLS to Postgres, not the probe. The runner is the one component that is not a libpq client, and it needs uselibpqcompat=true in runner-database-url to read sslmode the way every other component does. Without it, sslmode=require means verify the certificate chain to the runner alone — which fails against any Postgres whose certificate is not chained to a publicly-trusted root (Cloud SQL, or any private-CA instance), while the same value works everywhere else. Add the flag to the DSN and restart. If you build the Secret yourself (existingSecret), this is the step most likely to have been missed; see install step 3.

Do not reach for sslmode=no-verify to get past it. It is a node-postgres spelling that libpq does not accept, so it trades a broken runner for a broken everything-else: invalid sslmode value: "no-verify" from the app, the MCP servers, ingest and the seed job.

root certificate file "/root/.postgresql/root.crt" does not exist — the install is on sslmode=verify-ca or verify-full without a CA bundle the pods can read. Both client families need one and neither ships with it. Either supply the CA, or use sslmode=require — which still encrypts, but does not verify the server certificate.

prism-app CrashLoopBackOff with LOCAL_AUTH=1 but the install cannot sign anyone in — local sign-in is on and something it cannot work without is missing; the log line names every missing value (SMTP host, sender, base URL, super admins, Postgres). This is deliberate: a pod that came up anyway would serve a sign-in page whose emails go nowhere. Fix the named values and helm upgrade. The same message right after an install can also mean Postgres simply isn't up yet — the app fails fast and Kubernetes retries until it is.

Local sign-in

The magic link never arrives — three places to look, in order. (1) The self test's smtp relay line: it asks the relay whether it would deliver to each bootstrap admin, per address, without sending mail. (2) The app's app_events table (or its log): a refused request is recorded — an address not on the allowlist, a rate-limited address (max 3 links per address per 15 minutes), or a delivery failure (auth_email_failed, with the relay's error). The sign-in page deliberately says "check your email" in every one of those cases; the operator's records are where the truth is. (3) Spam. Corporate mail scanners that pre-fetch links are not on this list: opening a link shows a confirmation page, and only its "Continue to Prism" button (a form POST, which scanners never submit) spends the single-use token — so a scanned link still works when the person clicks it.

auth_email_failed (or the smtp relay line) reports a certificate verification error — the relay presented a TLS certificate the pod cannot verify, typically one issued by a private CA. Prism verifies the relay's certificate — chain and hostname — on both TLS modes, by default. The right fix is at the relay: a certificate from a CA the pod trusts, matching the hostname in smtp.host. The last resort is smtp.tlsVerify: false, which keeps TLS but trusts whoever answers — accept that knowingly or not at all. (A relay with no TLS is smtp.tls: none, not an unverified handshake.)

auth_email_failed reports The handshake operation timed out — a different fault from the one above, with a different fix. The relay answered and advertised STARTTLS, and then the TLS negotiation itself stalled until the 30-second timeout. Verification never got the chance to run, so smtp.tlsVerify: false will not touch it. Two causes account for nearly all of these: something in the path inspecting SMTP and mangling the negotiation, which enterprise firewalls do, or a port that does not terminate TLS the way its banner claims. Try implicit TLS — smtp.port: 465, smtp.tls: implicit — which is encrypted from the first byte and so offers no cleartext negotiation to interfere with. Expect the failure to be intermittent rather than absolute: the same relay can complete one handshake and stall the next. One colleague receiving their link is therefore not evidence that another's failure is about their address — nothing in the send path varies by recipient.

smtp.tls: none also clears the error, and is the wrong reflex unless the relay is in-cluster: it sends sign-in links, which are bearer credentials for the account, in the clear to anyone on the path. Reach for it only where that traffic never leaves the node network, and treat it as temporary anywhere else.

Locked out — no administrator can sign in — add an email to app.auth.superAdmins and helm upgrade: seeding is additive, so the next boot creates that admin without touching anything else. One sharp edge: the address must be one with no existing allowlist row — seeding never touches an existing row, so a listed address that was demoted in-app stays demoted (the skipped seed is logged as seed_skipped_existing, which is why an upgrade can report having added nobody). Use a fresh address. This needs cluster access, not a Prism login — that is what makes it break-glass. For a report while locked out, the self-test CLI (see self-test.md) runs via kubectl exec with no sign-in at all.

The model path

Chat errors, runner logs cannot obtain an inference key — the Secret has neither tare-inference-key nor tare-platform-api-key. Put your project-scoped inference key in tare-inference-key (install steps 1c and 3). Setting valet-api-key does not satisfy this and is not meant to — it is the Router management credential, read by different components for a different purpose (which credential buys what).

Chat returns 403 "not authorized for this data plane" — the inference key's project doesn't match the gateway. app.projectId, router.proxyUrl and tare-inference-key must all name the same project (install step 1d). Read the project the app is actually presenting rather than the one you meant to set — kubectl exec deploy/<release>-app -- printenv PROJECT_ID — and if it says default on a Router whose projects have their own gateways, that is the answer: before 0.6.0 an empty app.projectId became default silently. See the install-time entry above.

Every question fails, but everything else looks healthy — pods running, database connected, MCP servers answering. This is almost always the model: Prism is configured to call a model your gateway does not serve. The self test's model catalog line confirms it in one look. See model.md.

The self test says model catalog cannot tell whether the gateway answered — the agent runner image predates the catalogue reporting this check relies on. Upgrade to a matched set of images; the warning is the check declining to guess rather than a fault in your environment.

Data

A GitLab source shows rows up to one date and never advances, and the ingest log repeats transport error (ReadTimeout) six times — on 0.10.1 and earlier, the sync's read timeout was a fixed 30 seconds and GitLab answers a page it cannot finish at about 32. Upgrade to 0.10.2, which waits longer than GitLab does (ingest.engine.httpTimeoutSeconds, 120 by default) and keeps the page GitLab sends back with the diff sizes empty rather than giving up. On 0.10.2 the same line names the endpoint, how long it waited and the value to raise; a page that comes back with errors and nothing else is named by its cursor and the field GitLab timed out on, and the run goes on to the next group before it reports the failure. If the same cursor is named run after run, GitLab is answering that page with no merge requests at all; send us the line — the remedy is a smaller page, which is a change to the source's document. See How Prism reads GitLab.

One GitLab group never answers and the other groups stop filling too — before 0.10.2 the first group that could not be fetched ended the run, and the groups after it in the list were never walked. From 0.10.2 every group is walked, the one that could not be fetched is named in the failure with what it answered, and the others' rows are in the mirror. The run is still reported as failed, and a failed source is retried hourly rather than on its schedule: each retry re-walks the whole window for every group (the rows are refreshed, not duplicated) and waits about six minutes on a group that never answers. That continues until the group answers or is removed from the source's groups. Removing it is a coverage decision, not a filter (see The groups you list are a coverage decision).

A source shows live but 0 rows — synthetic data evicted, none ingested — the first live ingest truncates that source's synthetic rows before writing real ones. It truncated, then wrote nothing. Your credential or network path to that source is the place to look, not Prism. The ingest job's logs say which.

This state is now preventable, and the prevention is a helm upgrade you were going to run anyway: configure the credential and target, upgrade without flipping the source, and the self test reports whether that upstream is ready before the flip evicts anything. See Take a source live. Reaching this line means the flip happened first — the same assertions diagnose it after the fact, and a credential that authenticates cleanly and finds nothing is the usual cause.

On GitHub, read org_not_visible and org_visible_but_no_prs_found as token policy before you read them as network. A fine-grained personal access token has two ways to produce them that no firewall is involved in: the organisation requires approval for fine-grained tokens and nobody has approved this one — it authenticates, and keeps only the read-only access to public resources every such token carries — or the organisation blocks them outright. Both verdicts look identical to an egress rule that has not landed, so check the organisation's personal access token settings first; it is much the cheaper of the two to check. Note that this only reaches the verdict where the organisation's repositories are private throughout: where some are public, the same unapproved token sees those, the probe passes, and the symptom is the low figures below rather than a verdict. What the GitHub credential needs is the full list, for both token types.

GitHub figures are plausible but low, and nothing anywhere is red — the likeliest cause is a GitHub token that cannot see every repository: one scoped to Only select repositories rather than to all of them (re-scoping it to All repositories is the fix, and it needs no change of credential kind — see Widen a source's scope), or a fine-grained token awaiting an organisation owner's approval, which retains access to the organisation's public repositories and nothing else. There is no fault to find in either case: the walk succeeded, the pre-flight passed, and the repositories the token cannot reach were never fetched, so every org-wide total is a subtotal — and Prism labels it org-wide (every repo in the index), because the index is all it can see. Compare the repositories in the answer against the ones you expected, then compare the token's repository access — and its approval state — against the same list. Widening the token fixes it from the next sweep onwards, and only for the days that sweep covers: the newly-reachable repositories' history arrives only when those day buckets are walked again. Widen a source's scope is the procedure for both halves. Do not reach for the DELETE FROM ingest_state recipe further down this page to force it — it is written for a different problem, and it makes the next run a first run, which truncates the table and re-walks only ingest.backfillDays. The re-sweep on that page is non-destructive, and it is not worth trading years of history for a few repositories' worth of it.

GitHub reports unrefreshable_installation_token — the credential in github-token is a GitHub App installation access token, the ghs_… string GitHub hands back when you exchange a JWT. It works at this moment and it expires one hour after it was minted; Prism cannot refresh one it was simply handed, so the source would run, look healthy, and then start failing with bad_credential — which reads as somebody having revoked it. The pre-flight catches it deliberately rather than passing, because an hour-late failure with a misleading name is worse than a red check now.

The fix is to give Prism the App's identity and key instead of a token it minted for you, so it can mint its own: ingest.github.app.appId, ingest.github.app.installationId and secrets.githubAppPrivateKey. Widen a source's scope is the procedure. The verdict already tells you how many repositories that installation reaches, so you can see what you are about to gain.

GitHub reports bad_app_credential — the App is configured and cannot be used as configured, and the verdict's error says which of the three values is wrong. The three shapes it distinguishes, because they have different fixes:

• half-configured — one or two of the three set. Prism refuses rather than falling back to secrets.githubToken, which would ingest under a credential you did not choose;
• the private key does not match the App, or the pod's clock is more than a few minutes out — the JWT Prism signs is valid for nine minutes and a drifted clock puts every one of them outside its own window;
• no such installation — installationId is the installation's id, the number at the end of the App's Configure URL on your organisation, and not the App's id.

A source shows ingest stalled — last completed …, past the Nh budget — the rows are there and are being served, but nothing has swept that source for several of its own scheduled intervals. The ingest is the place to look, not the app — but since 0.15.0 a failing source does not make the ingest Job fail, so a green Job here does not mean the source is fine.

Start with the diagnostic bundle, not with kubectl. Its ingest block has an entry per source with last_failure.at and last_failure.error — when it failed and why, in full, for every source including the ones you registered yourself.

If you do go to the pod logs, search for the source's own name rather than the tick's done line. A failed scheduled source backs off for up to an hour while the tick runs every few minutes, so the done line you find will usually belong to a later, clean tick and read failed: []. The source's own line on those ticks says not_due — and carries the reason after it. Only the tick where it actually failed carries it on done, and successfulJobsHistoryLimit: 1 means that pod is replaced by the next tick's.

An uploaded file — a roster, or any kind: file source — has no retry backoff, so it is re-attempted every tick and stays in failed on every done line until the file is corrected. Searching for its name works either way, which is why that is the instruction.

kubectl logs -n <namespace> -l app.kubernetes.io/component=ingest-tick --tail=200 \
  | grep '<source name>'

kubectl get jobs -n <namespace> still finds a run stuck Running, and a failed Job now means the tick itself could not run, or a source failed and Prism could not write down why. The usual causes are a credential that has been revoked since go-live, an egress rule that changed, and a node or quota problem that stops the CronJob creating pods at all. Answers keep working throughout and stay labelled with the coverage they actually have — the figures are not wrong, they are just not moving. ingest.stallAfterIntervals is the multiple the budget is built from, and ingest.github.schedule / ingest.jira.schedule the interval it multiplies.

A GitHub ingest Job failed with the table is empty … there is a watermark — GitHub has ingested here before and the run has just ended with pull_requests holding no rows at all, so it refused to record itself as a successful sweep. Note the gate is on the table being empty, not on the run having fetched nothing: a quiet weekend fetches nothing, leaves the rows that are already there, and passes. Same causes as the stall above, caught one run earlier. It is deliberately not fatal on the very first run: a first ingest against a genuinely empty window is indistinguishable from a broken one, so the first run is allowed to succeed and the next firing is the one that fails. The Jira ingest has no equivalent gate; a Jira mirror that ends up empty shows up as live but 0 rows on the self test instead.

An ingest Job failed with DeadlineExceeded — the run hit ingest.runDeadlineSeconds (24 hours by default). Either it wedged, or you have asked for more history than fits in one run; see Limits for the sizing. Nothing is lost either way — the walk records each completed day and the next firing resumes from there. Failing rather than hanging is the point: the CronJob runs one at a time, so a run that never ends would block every later one indefinitely.

A source says abandoned after …s (ingest.engine.sourceDeadlineSeconds) so the sources behind it could run — it ran for longer than one source may in a run (ingest.engine.sourceDeadlineSeconds, two hours by default), so the run stopped it and went on to the next source. When the job's day was nearly spent the brackets say (what was left of ingest.runDeadlineSeconds) instead: less than the limit was left. The self test shows it as a recorded failure until a run completes. What the rest of the sentence says tells you whether to act:

• it resumes where it stopped — a mirror that moved forward. It is catching up on history a little each run. Leave it, or raise the value so it catches up in one.
• it made no progress, so its next run goes last with the rest of the day's budget — a snapshot or file (the stream it was copying keeps its previous copy; a stream it had finished keeps its new one), or a mirror whose resume point did not move. Its next run gets what is left of that run's time, once (or again, if what was left was less than the limit).
• stopped … with the day's whole remaining budget … does not fit in ingest.runDeadlineSeconds — even that was not enough, and it moved nothing forward (a mirror that did is told it resumes where it stopped). It is held to the limit on later runs, which then say did not fit in ingest.runDeadlineSeconds on an earlier run. The sentence names what the stream reads from: check that statement's plan with your DBA, or the source's rate limits. Another failure or a restart does not end the hold; it ends when a run completes, or when you change the source's manifest, its parameters (on the source or the chart values it reads them from, such as ingest.github.orgs) or its uploaded file, either ingest.engine.sourceDeadlineSeconds or ingest.runDeadlineSeconds or oracle.mode, or upgrade. Its next run then gets the rest of the day again, once.

Every GitHub ingest run starts the backfill again, and the table never fills — each Job ends in DeadlineExceeded, each one logs first live run: truncating …, and the row count returns to roughly the same number. Progress is recorded per completed day bucket, so a run killed before it finishes even one records nothing — and a run that recorded nothing is indistinguishable from a first run, which truncates and starts over. It takes a run deadline too short for one bucket at the configured request rate: a bucket costs at least 3 requests per org and up to 30 on a busy day, so at maxRequestsPerMinute: 1 it needs anywhere from half a minute to half an hour. Raise ingest.github.maxRequestsPerMinute, or raise ingest.runDeadlineSeconds back towards its 24h default — never the other way. Current builds say so in the log before it happens: a WARNING naming both numbers where a busy bucket will not fit, and a refusal to start at all (refusing to start, also naming both) where even an empty one cannot. An ingest that exits immediately with that message is this check, not a failure to reach GitHub.

The Jira ingest can loop the same way and says so the same way, in its own unit: it records progress per page of search results, and a page costs one request whatever it holds, so the whole floor is 3 requests for the run — deployment detection, the field registry, one page. Where that does not fit inside the deadline it refuses to start, naming JIRA_MAX_REQUESTS_PER_MINUTE and the deadline; raise ingest.jira.maxRequestsPerMinute or the deadline.

A source shows rows but upstream unverified — the data is there and the ingest is current, but nothing has confirmed the upstream is still reachable with a working credential. That is a real gap rather than a pedantic one: a token revoked five minutes ago leaves a row count identical to a healthy one, and the mirror only starts to look wrong days later.

The line names the probe that answers it. Either run it directly:

kubectl create job -n <namespace> probe-github \
  --from=cronjob/<release>-github-check

…or work out why the CronJob that does this every ingest.checkSchedule has not:

kubectl get cronjob,job -n <namespace> -l app.kubernetes.io/component=github-check

The usual causes are a source flipped live by editing a value without running helm upgrade (so the CronJob was never created), a missing credential secret, or an image the namespace cannot pull. The probe exits 0 whatever it finds, so a failed Job here is the datapond, not the upstream — an upstream fault comes back as a red upstream check failed: <fault> on the self test instead.

Two source lines never ask for a probe, and neither is a fault: directory has no prism-ingest job at all (the Agent Router is its upstream, and the AGENT ROUTER checks on the same page cover it), and a live spend source has no local index to probe.

A source reports no_token or no_credential, and you are certain the credential is in the Secret — then it is almost certainly under a different key than the one the chart reads. These are the three, and they are exact:

Source	Secret key
GitHub	github-token
GitHub, when the credential is an App	github-app-private-key
Jira	jira-token
Router spend and directory	valet-api-key

This only arises with existingSecret, where you build the Secret yourself — a chart-managed Secret always writes the right names. Check for a typo, an underscore for a hyphen, or a key you added to a different Secret than the one existingSecret names:

kubectl get secret <your secret> -n <namespace> -o jsonpath='{.data}' | tr ',' '\n'

The verdict in the diagnostic bundle names the key it looked for, so compare that against the output above rather than against your notes. Nothing is broken while this is showing — the credential is simply not being found, and the source keeps serving whatever it has.

helm upgrade came back clean but the source is still empty — expected for the first half-hour or more after a flip. helm returns once the backfill has started; the walk itself takes tens of minutes and scales with ingest.backfillDays and how many orgs or projects you named. Watch it with kubectl logs -n <namespace> -l app.kubernetes.io/component=github-ingest -f (or jira-ingest) — it prints a line per day of history. Still empty once that Job has finished is the credential/egress case above, not a slow backfill.

A source is still not refreshing after the upgrade that was meant to fix it — a source whose last run failed is held off for an hour before it is tried again, and until 0.13.0 that hour was measured from the failure alone. It therefore survived the helm upgrade you performed because it fixes that source, and for up to an hour afterwards nothing changed. The tick's own log said so — the last attempt failed 0:23:44 ago; retry after 1:00:00 — but that line is in a CronJob pod's log rather than on the admin page.

From 0.13.0 there are two ways through, and you should not need either very often:

• Ask for it. POST /api/admin/sources/<id>/ingest/run-now records a request that the next tick honours — within ingest.engine.tickMinutes (15 by default). There is no button for it on Admin → Context streams; it needs a super-administrator session, the same as the page, and the source's id is under registry.rows in GET /api/admin/sources. It overrides the retry backoff and the schedule, and it asks for one walk however many times you send it. The source's record (GET /api/admin/sources/<id>) says whether one is waiting: run_request_pending is true until a walk honours it. (run_requested_at is when it was asked for, and stays set after the walk has run, so read run_request_pending, not that.) To take a waiting request back, send DELETE /api/admin/sources/<id>/ingest/run-now — the /ingest/run-now path, never the source's own: DELETE /api/admin/sources/<id> deletes the source and its stored credential. A request is never discarded on your behalf, because nothing having run is not the same as your having changed your mind.
• It is usually unnecessary after an upgrade. An attempt is now stamped with the version that made it, so a failed source whose appliance has been upgraded since is tried at the next tick rather than waiting out the hour. This needs your chart version to have moved: an appliance running an unversioned chart (0.0.0) has nothing to compare and keeps the old behaviour.
• From 0.16.0, a values-only redeploy counts too, for the values that decide how a source connects. Correcting oracle.mode and running helm upgrade on the same chart version leaves the version where it was, so until 0.16.0 the failed source still waited out the hour. An attempt is now also stamped with a digest of those values, and a change ends the backoff at the next tick. Today the only such value is oracle.mode. For any other change, the run-now request above is the lever, and from 0.16.0 the tick's not-due line names it.

The tick says a run failed and gives no reason — from 0.16.0 the line says so in words: "no reason was recorded for it, which usually means the run was killed rather than failing". Both of the tick's failure paths write a reason to the source's row, so an attempt with none after it usually means neither ran to completion. The likeliest cause is that the process was killed: an out-of-memory kill, or a crash inside a native library. Less often, the write itself failed, which the tick's own done line reports under red. Look at the tick pod's exit reason, which the pod log does not carry:

kubectl get pods -n <namespace> -l app.kubernetes.io/component=ingest-tick
kubectl describe pod <the failed pod> -n <namespace>   # "Last State: Terminated", "Reason: OOMKilled"

A SIGTERM (the run passing ingest.runDeadlineSeconds, an eviction or a node drain) is caught from 0.16.0, from the moment the tick starts. If it arrives while a source is running, that source's reason says "the run was stopped by SIGTERM before it finished" instead, and no further source is started. If it arrives before any source starts, the tick only logs terminated and exits 143. When the deadline is the cause, the Job controller deletes the pod, so kubectl describe pod finds nothing. The recorded reason and kubectl describe job are where to look.

On an older release, the escape hatch is to clear the attempt stamp by hand. Clear the one key — not the row:

UPDATE ingest_state SET detail = detail - 'last_attempt_at'
 WHERE source = '<the source name>';

DELETE FROM ingest_state WHERE source = … also works and costs more than it looks: that row holds the resume point of any interrupted backfill, the backfill horizon and the record described next, so deleting it can send the next run back to the start of history and truncate what is there. Remove the key, not the row.

A source that has never finished a run keeps what it has copied, and will not start again from empty. The first live run of a source clears its tables — whatever was in them (sample data, an older configuration) must not mix with real rows — and records that it has done so, as live_since on that source's ingest_state row. Every later run reads that record: a source that cannot finish adds to what its own earlier runs copied instead of deleting it every hour. That is what stops one stream that always fails from wiping the streams beside it.

You will meet it in the ingest log as a line beginning "a previous run of this source already truncated …", on a source whose Last completed is still empty. It is not an error; it is the run declining to destroy its own earlier work.

If you genuinely want the next run to start from empty — you have changed what the source means, not merely how it is filtered, and the rows already copied are wrong rather than incomplete — remove that one key and let the schedule do the rest:

UPDATE ingest_state SET detail = detail - 'live_since'
 WHERE source = '<the source name>';

The next run then clears the tables and backfills, exactly as a first run does. Editing the source's manifest does this for you: a run whose document differs from the one the tables were taken over under takes them over again, because the rows were written by a different document.

Two ingest Jobs for one source, one of them Completed in seconds — not a fault. Only one run of a source may hold the datapond at a time; a second (usually the CronJob firing during a first backfill) logs another … ingest is already running and exits successfully rather than racing it.

Review latencies look impossibly fast, or a code-review bot appears as a person — a time to first review measured in minutes is usually an automated reviewer being counted as a colleague. Prism classifies a review as automated when GitHub reports its author as an app; releases before this one classified by account name instead, which missed every review app whose login does not end in [bot] (GitHub's own review bots among them). Upgrading fixes what is ingested from then on, and does not correct history: the flag is written when a review is stored. To restate the existing rows, clear the GitHub ingest bookkeeping and let the next run re-walk the history —

DELETE FROM ingest_state WHERE source = 'github';

— then wait for the next CronJob firing (or trigger one). That re-walk is a first run: it clears the table before it starts, exactly as the warning further up this page says, and the page said the opposite here until 0.14.0. Unless INGEST_TRUNCATE_ON_FIRST_RUN=0 is set, expect the table to be empty while it runs. It takes as long as the original backfill. Until it has run, review-latency figures and the reviewer matrix may still count review apps as people. Answers name the accounts they excluded as automated, so an empty exclusion list beside a two-minute median is the symptom to look for.

Every question fails instantly, and the self test's chat round trip says "no answer from the agent" — if the diagnostic bundle's runner_error for that check mentions toolConfig.tools, the Agent Router in front of this install is mistranslating the prompt-caching hints Prism attaches to its tool definitions. A Bedrock-backed gateway folds the cache point into the same toolConfig.tools[] element as the tool itself, and Bedrock refuses the pair:

400 The value at toolConfig.tools.11 can only set one of the following keys:
    toolSpec, systemTool, modelTool, or cachePoint.

Nothing else goes red — the model catalogue, the inference endpoint and every MCP line stay green, because the request only becomes invalid at the last hop. Set agentRunner.promptCaching=false and upgrade; the chat works again and the install gives up the caching discount until the Router is fixed. Report it: the translation is the Router's to correct, and the setting is meant to be temporary.

If you have set it before and this came back after an upgrade, check the bundle's config block: prompt_caching_configured says which way the setting went in on the release that is actually running. It reports the configured value, including anything you overrode in agentRunner.extraEnv, so on there means nobody turned it off in your values file — and the usual cause is a values file that was not passed to the upgrade. Two things it cannot see: a bare kubectl set env on the runner Deployment (move it into agentRunner.extraEnv, where it is both visible and durable), and the runner switching caching off by itself after a gateway rejects the hint, which it does from 0.9.0 and logs.

Everything works, and inference costs far more than expected — Prism re-sends a large, stable prefix on every turn (the instructions and the tool definitions), and asks the Router to cache it. Whether that hint takes effect depends on which Router API the request goes down, and the two do not agree:

Your Agent Router fronts	agentRunner.anthropicApi
AWS Bedrock, or Vertex / AWS Anthropic	completions (the default)
the Anthropic API directly	messages

Set the wrong one and nothing looks wrong. Every check on /selftest stays green, every answer is correct, and the only symptom is the bill — the cached part of each turn is worth roughly 9x, so on a long conversation this is most of what you pay. Confirm before changing it: in the Agent Router's own request log, a working install shows cache-read tokens on the second and later turns of a conversation; an install on the wrong path shows none, ever.

This is a Router-side gap rather than a Prism setting we would rather not have — the gateway carries the caching hint on every path where it translates between API shapes, and one path passes the request through untranslated instead. When that is fixed the default will be right everywhere and this parameter goes away. agentRunner.promptCaching must also be on (it is by default) for either path to cache at all. The bundle's config block reports both as they were configured — anthropic_api_configured and prompt_caching_configured — which tells you what this install was asked to do, but not whether the Router honoured it. Only the Router's request log answers that. invalid there means the value set is not one the runner recognises, so it was ignored and the default is in force.

The chat starts answering, then dies part-way — the idle timeout on something between the browser and Prism, not Prism. An answer that makes several tool calls runs for minutes with quiet gaps between events, and Prism sends no keepalive traffic to fill them. Raise the idle timeout on your ingress, gateway and any proxy in the path to at least 10 minutes (Limits). The answer itself says this happened — an amber band above it, carried into the exported PDF — and a cut arriving at about the same elapsed time every time is the timeout rather than bad luck. Nothing in the self test fires for it, and nothing server-side records it: every component is healthy, the answer was produced normally, and the stored copy is complete. What the reader saw is the only evidence, which is why the band exists.

Prism refuses a chart, or says it cannot answer something it clearly has the data for — check Limits before raising it. A chart with spend on one axis and delivery output on the other, a "productivity" heading over a count, and inferring who is on a team are refused by design, not by fault.

Every question on a stream's Verification tab says the stream is not loaded, while the stream itself is enabled and answering — the questions record which stream they ask about, and on a stream registered or renamed before 0.12.0 that record did not move when the name did. So each question asks about a name this installation no longer has, and reports that there is no such stream to ask. Nothing else is wrong: the stream is collecting data and answering in chat, and none of its figures are affected. Open its Manifest tab, set every query.source under policy.verification.questions to the stream's own name, and Save. Saving a manifest clears any verdicts already recorded — the questions come back as pending, which is the ordinary consequence of replacing a model, not of this fix. From 0.12.0 a rename carries the questions with it and a manifest whose questions name another stream is refused with the question named.

The chat quotes numbers that look wrong — check the SOURCES group. A source still in synthetic mode is answering from generated data, which is realistic enough to look real. This is the intended day-one state, but it surprises people who expected a flipped source to have taken effect.

A verification question says the proxy in front of Prism gave up

On Admin → Context streams › Verification, a question's card reads "No result: the proxy in front of Prism gave up on this request after about N s (HTTP 504)". A fast 502 or 503 reads differently — "could not reach Prism" — because that is a pod restarting or a service with no ready endpoint rather than a timeout, and raising a timeout would not touch it. The question was still running when a Route, ingress or load balancer between you and Prism stopped waiting for it. Nothing was recorded, and it says nothing about the source. On OpenShift the Route's default timeout is 30 seconds, and a question over a large copied source can take longer. Raise the timeout as in Installing Prism, step 1e — in app.ingress.annotations where the chart renders your Ingress, or a hand annotation is lost at the next helm upgrade — then press Run on that question again. Prism's own ceiling on one question is 45 seconds, so past that the card carries Prism's reason rather than a proxy's, and no Route timeout raises it. The same N on every attempt is the proxy's timeout; since 0.16.0 Run all questions sends one request per question, so only a question that on its own takes longer than the proxy allows is affected.

A source loads nothing, writes no error, and its job is still running

A context stream that copies your Oracle or Postgres starts, loads nothing, and its ingest job stays Running. The tick walks its sources one after another, and a new tick does not start while one is still running. Before 0.16.0 the tick therefore ran no other source while it waited. From 0.16.0 one source runs for at most ingest.engine.sourceDeadlineSeconds (two hours by default), or, after it was stopped without moving forward (for a snapshot or file: without completing), for the rest of the day's ingest. That happens once, and only when at least the limit was left; if that run is stopped again without moving forward, the source is held to the limit until a run completes or its settings or version change. Then it is stopped with an abandoned after … or stopped after … line and the sources behind it run.

From 0.16.0: wait for the bound, then read the line

From 0.16.0 most stalls end on their own with a line that says what stopped them. A line that names one stream can come while the run goes on to the next stream. Read the lines in the tick's log, newest Job last:

kubectl -n <namespace> get jobs --sort-by=.metadata.creationTimestamp \
  -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=ingest-tick
kubectl -n <namespace> logs job/<the job>

The log is kept only until the next tick finishes, or not at all once the Job has hit its deadline. The tick's failed line for the source, and last_failure.error in the diagnostic bundle's ingest block, carry that line, or a summary that counts the streams refused (N streams refused (…)).

While a stream is being copied, the log also has a line for it every minute, <stream>: still copying: N rows in M chunk(s) so far; Ts …, ending with what the copy has been doing for those seconds: opening a session on the source, waiting on the source, writing to Prism's database, or one of Prism's own pauses (pausing between requests to the source (pacing), backing off after a failed attempt). Rows that keep climbing mean a slow copy; seconds that keep climbing on the same end mean that end is where it waits.

How long to wait for the stream the copy is on:

• Setting up a Postgres session: 60 seconds for each of connect, session setup and begin, tried six times: about nine minutes.
• The statement: the manifest's statement_timeout, or without one ingest.engine.databaseFetchTimeoutSeconds (an hour by default) for each step. On Oracle, every round trip after connecting gets that bound (half the manifest's statement_timeout, if it states one), and a stalled fetch can take about twice as long to end: about two hours by default.
• Connecting to Oracle: no bound. Each session, once set up, writes oracle session opened in … mode to the tick's log. The copy connects again after a database error (…) line or WARNING: that fault closed the session …. If no new oracle session opened line has come for several minutes after one of those, or since the run began, it may be stuck connecting: go to No such line.

The line says	What it means
no answer to connect within 60s	Postgres: the server, or something between it and Prism, did not answer the connection
no answer to session setup within 60s	Postgres: the connection opened, and a session setting got no answer
no answer to begin within 60s	Postgres: starting the read-only transaction got no answer
giving up on <stream> after 6 attempts (the last: …)	Every attempt failed; the part in brackets is the last one's reason or code
the statement did not finish inside the configured statement_timeout	The manifest states a statement_timeout, and the statement was stopped by it. On Postgres a cancel by your DBA reads the same; on Oracle so does a connection closed late in a statement (DPY-4011)
… inside the engine's own …s bound on one step of the statement …	The manifest sets no statement_timeout, and one step of the statement returned nothing for the fetch bound. On Oracle a connection closed late in a statement (DPY-4011) reads the same
… inside a statement_timeout the source's role or cluster carries …	Postgres: 57014 with no statement_timeout in the manifest: a timeout set on the login or the cluster, or a cancel by your DBA
… inside an Oracle call timeout, or a connection closed before the session was set up …	Oracle, before the session was set up: DPY-4011 is both
… inside the Oracle call timeout while the session was being set up …	Oracle: a session setting did not finish inside the call timeout (DPY-4024)
the connection dropped (…) after N page(s) …	The connection closed after that stream had read pages. That stream keeps its previous copy for this run, and the streams after it are copied on a new connection
abandoned after …s (ingest.engine.sourceDeadlineSeconds) so the sources behind it could run (or its progress could not be read)	None of the bounds above ended it inside the per-source limit, so the tick stopped this source and ran the next; see A source says abandoned after … below
stopped after … with the day's whole remaining budget …	The same, on the one run a stalled source is given the rest of the day: the copy does not fit in a day, and it is held to the limit until a run completes or its settings or version change; see the same entry
the run was stopped by SIGTERM before it finished …	No bound fired: the tick's deadline (ingest.runDeadlineSeconds), an eviction, a node drain, or someone deleting the Job stopped it

For a Postgres source, If something is wrong on Copy a Postgres table or view has entries for several of these. A statement waiting on a lock on your server also ends with one of these lines. To see it, run the query on your server below while a run of the source is waiting on it. For an Oracle source, send us the line with the diagnostic bundle.

No such line, or before 0.16.0

Connecting to Oracle, a wait inside Prism's own Postgres, and a far end that keeps sending a trickle of data (on Oracle, on Postgres with no statement_timeout, or a copy that pages) have no bound of their own. From 0.16.0 the per-source limit still stops them, with the abandoned after … line; before 0.16.0 a copy with no statement_timeout could wait until the job's deadline. If no line has come after the wait above, then, while the job is still running, run these two queries and send us both outputs with the diagnostic bundle. No rows for the source's login on your server is itself an answer: no session is open.

On the Postgres Prism uses for its own memory. With the bundled Postgres:

kubectl exec -n <namespace> statefulset/<release>-postgres -- \
  psql -U postgres -d prism -c "
    select pid, client_addr, backend_start, state, wait_event_type,
           wait_event, xact_start, pg_blocking_pids(pid) as blocked_by,
           left(query, 60)
    from pg_stat_activity where datname = current_database()
    order by xact_start nulls last;"

With postgres.bundled: false, run the same select from any client that can reach your server, against postgres.external.database, as postgres.external.user.

While the copy waits on your server, its own session here is normally idle in transaction.

On your server, filtering on the login the source connects as (Parameters on the source's page shows it as db_user in the recipes). On Postgres, as a superuser or a role with pg_read_all_stats:

select pid, usename, datname, state, wait_event_type, wait_event,
       xact_start, now() - state_change as in_state_for,
       now() - query_start as age, left(query, 60)
from pg_stat_activity
where usename = '<the login>'
order by age desc nulls last;

On Oracle, as an account that can read gv$session:

select inst_id, sid, status, event, last_call_et, sql_id
from gv$session where username = upper('<the login>')
order by last_call_et desc;

To free the tick sooner, disable the source, save the Job's log, then delete the Job; its log goes with it. Find it through its running pod, since finished tick Jobs stay listed too:

kubectl -n <namespace> get pods \
  -l app.kubernetes.io/instance=<release>,app.kubernetes.io/component=ingest-tick \
  --field-selector=status.phase=Running -L job-name
kubectl -n <namespace> logs job/<the JOB-NAME it shows> > tick.log
kubectl -n <namespace> delete job <the JOB-NAME it shows>

When to send it to us

A screenshot of the self test page covers most of it. If we need more, the diagnostic bundle carries the evidence behind every line — see self-test.md for exactly what is and isn't in it.

Worth including if you have it: which step you were on, and whether it ever worked.