Rotate a credential

The one thing to know before you start: every credential reaches its container as an environment variable, read once at container start. Kubernetes does not restart a pod when a Secret it references changes, and the chart puts no checksum annotation on its Deployments. So a rotated Secret reaches a running pod never. Jobs and CronJobs are the exception — they get the current value on their next firing, because that is a new pod.

This is the failure mode where the rotation appears to work for hours and then does not survive the next unrelated restart, so treat the restart as part of the rotation rather than as cleanup.

Credential	Secret key	Read by	Restart
Inference key	tare-inference-key	agent runner	the runner
Admin platform key	tare-platform-api-key	agent runner	the runner
Router management key	valet-api-key	management MCP, spend sweep	the management MCP (the sweep is a Job)
GitHub token	github-token	GitHub Live MCP, ingest	the GitHub Live MCP, if enabled
Jira token	jira-token	ingest only	nothing — the next Job picks it up
SMTP password	smtp-password	the app	the app
App JWT secret	app-jwt-secret	the app and the runner	both, together
Postgres password	postgres-password	app, MCPs, ingest, bundled Postgres	everything that connects
Runner database URL	runner-database-url	agent runner	the runner
At-rest key	encryption-key	agent runner, ingest tick, the app, and the semantic MCP	all four — and see the warning below
Internal query secret	mcp-shared-secret	the app and all five query servers	all six, and see the note below

Not every credential is in that table. The table lists credentials held as Kubernetes Secret keys. A credential you add through the admin surface — a database connection to a warehouse, for example — is held in the appliance's own database instead, and it is rotated there rather than with kubectl: replace the value on the connection and the next read uses it, with no Secret to patch and no pod to restart. What proves such a value is the probe the admin surface runs as you enter it — not the self test in step 3, which says below exactly what it does and does not cover. Those credentials are protected by the encryption-key Secret, so rotating that key is a separate and much larger operation than rotating any one credential.

A query-through warehouse is rotated on the source, and only there. Its key pair used to be in both places at once — a Secret the semantic MCP signed in with, and a copy on the source row the probe used — with nothing checking that the two agreed, so replacing one and not the other left a source that either probed green and answered nothing or answered and failed its probe. That is gone. There is one copy, on the source: replace it in the admin surface, probe again, and the next question uses it. No Secret to patch, no pod to restart.

Rotating encryption-key is a re-entry, not a restart

encryption-key is the only row in that table that does not protect one thing. It is the key everything else is stored under:

• every per-user Agent Router key the runner has custody of, and
• every data-source credential entered through the admin surface.

Neither is re-encrypted when the Secret changes, and there is no second key: a new value does not re-key what the old one wrote, it makes it unreadable. So rotating this key is a planned operation with a re-entry step, not a rolling restart:

1. Note which sources hold a credential: GET /api/admin/sources (as a super admin) reports has_credential and credential_updated_at per source; the value itself is never shown, here or anywhere else. Admin → Context streams carries the same two facts on each source's Overview tab, so the list can be taken off the page rather than the API.

2. Change the Secret value and restart all three long-running consumers — the agent runner, the app, and the semantic query server. The ingest tick needs nothing: it is a CronJob and picks the new value up on its next firing.

   The semantic query server was missing from this list until 2026-09-04, and it is the one that matters most here: it is the container that opens a query-through source's stored credential, so a rotation that left it holding the old key would leave every warehouse source unreadable with the app and the runner both looking healthy. The table at the top of this page has listed four consumers since #741; this step listed two.

   Restart the runner BEFORE the app, and this is new in 0.11.0. The app writes the internal query secret onto each of the runner's query-server profiles, and the runner stores it encrypted with this key. It does that write once per app process, on the first conversation after the app starts. So an app restarted first writes the secret under the OLD key, records that it has done so, and when the runner then restarts it cannot read what was just written — every tool call fails, and the only sign is [mcp] profile … dropped from the toolset in the runner's log. Restarting the app a second time repairs it. Doing the runner first avoids it entirely.

3. Re-enter every source credential. On Admin → Context streams this is the source's Credential tab and its Save and test button; through the API it is POST /api/admin/sources/<id>/credential with the credential's fields (for example {"secret": "<token>"}) — the path segment is the row's id, the UUID GET /api/admin/sources reports, and a name there is a 404. Each entry is probed before it is stored, so a source that comes back green has been re-keyed and re-checked in one step. A source that is switched on will not accept a credential the probe did not pass — that is what keeps a connected source connected to something that answered — so if the probe cannot run at all, switch the source off, enter the credential, and switch it on again once it probes.

4. Users sign in and their Router key is minted again on first use.

Until step 3 is done, the scheduled ingest of any source whose credential was stored in the registry fails and says so — it refuses to fall back to an older environment credential, because authenticating as a possibly-different identity is worse than not running. The value is sources.registry.enabled only: an install that has never used the admin surface to enter a credential has nothing in step 3 to redo.

Step 1 — change the value. Which command depends on who owns the Secret:

# chart-managed Secret (secrets.* in your values)
helm upgrade <release> <chart> -n <namespace> -f <your values file> \
  --set secrets.githubToken=<new token>

# existingSecret — the chart renders no Secret, so patch yours directly
kubectl patch secret <your secret> -n <namespace> \
  -p '{"stringData":{"github-token":"<new token>"}}'

Step 2 — restart what reads it, per the table:

kubectl rollout restart deploy/<release>-agent-runner -n <namespace>
kubectl rollout status  deploy/<release>-agent-runner -n <namespace>

app-jwt-secret is the one with an ordering trap. The app signs the tokens the runner verifies, so while the two hold different secrets every call between them fails. Change the value once, then restart both and expect chat to fail in between — this is a short outage, not a rolling change, and doing it during a demo is a bad idea.

mcp-shared-secret has the same shape of trap and one extra step, which the appliance takes for you. Six workloads hold it — the app and the five internal query servers — and while any of them disagrees, the questions that route through the disagreeing server come back unanswered rather than wrong. Change the value once, then restart all six:

kubectl rollout restart deploy/<release>-app deploy/<release>-mcp-management \
  deploy/<release>-mcp-github deploy/<release>-mcp-jira -n <namespace>
# and, where they are enabled:
kubectl rollout restart deploy/<release>-mcp-semantic \
  deploy/<release>-mcp-github-live -n <namespace>

The extra step is the agent's own copy. The agent reaches these servers through the runner, which holds the secret on each of its query-server profiles rather than reading it from its own environment — so the runner is deliberately absent from the list above. The app pushes the new value onto those profiles on its first provisioning pass after it restarts, which is the first new conversation anyone opens. Nothing is owed from you; it is worth knowing because it explains why an in-flight conversation can still fail for a moment after every pod above is green, and why restarting the runner does not help.

There is no value of this secret that turns the check off. Blank is not "unauthenticated" — it is "refuse everything" — so a rotation that lands an empty value is an outage, not a security regression, which is the way round we wanted it.

Step 3 — prove it. Run the self test. What it exercises, credential by credential:

Credential	The line that would go red
Inference key	the agent runner line and the AGENT ROUTER group
App JWT secret	agent runner — runner rejected the app tokens
Internal query secret	each mcp * line — refused this app's shared secret (401)
Router management key	mcp agent router usage
Postgres password	datapond postgres
SMTP password	smtp relay, per super-admin address
GitHub token, Jira token	their SOURCES lines — with the two caveats below
Router spend	the spend SOURCES line, only under sources.spend=indexed

The mcp * row is the one worth knowing about: a rotation of mcp-shared-secret that reached some of the six workloads and not the others is exactly what those lines are written to name, and it is the failure this page's own warning above is about.

Under sources.spend=live there is no spend verdict to read. The line reports live · served by the Agent Router usage API (no local index) and the chart renders no check job for it at all; whether the Router answers is the AGENT ROUTER group's verdict, not this one's.

The SOURCES group is not a list of your streams. Its per-source lines are a fixed set written into the app — the GitHub mirror, the GitHub issue index, Jira and spend — not a reading of what you have connected. A stream you added on Admin → Context streams gets no line that tests its credential. With the stream registry on you will see source registry, designated route, recipe drift, identity crosswalk and identity attribution in the same group, and three of those print your designated stream by name — so the group does not look silent about it. Read what those lines actually say: they report the registry rollup and the last compiled crosswalk. None of them opens a connection to the stream, so none of them is evidence about a credential you have just replaced.

Two caveats on the GitHub and Jira lines, and both are about a verdict that is not current:

• It can be up to half an hour old. The app holds no GitHub or Jira credential of its own, so those lines show the verdict last recorded by the prism-ingest check CronJob — ingest.checkSchedule, every 30 minutes by default. A rotation done as a helm upgrade re-records it before the command returns, because that check also runs as a post-upgrade hook. A rotation done with kubectl patch secret does not, so a green line can still be describing the credential you have just replaced. Wait for the next firing.
• If you moved that stream's credential onto its row in the admin surface, the line is about the wrong credential entirely. The scheduled check reads the environment variable; the ingest, from the next tick, reads the row. So the line goes on reporting the Secret value while the ingest uses the one you entered. Rotate it where the ingest reads it, and prove it the way the next paragraph describes.

A credential you entered in the admin surface is proved when you enter it. The credential route probes the value before it stores it: a probe that fails writes nothing and leaves the old credential exactly where it was, so a rotation that would have broken the stream is refused at the moment you make it rather than discovered later. A stream that is already enabled will not take a credential this request has not probed at all — it refuses the store instead, so the enabled state never outlives the evidence behind it. The verdict is on the stream's row on Admin → Context streams. Two stores go through unproven, and both leave the stream switched off until something proves it: asking for the store without a probe, which only a disabled stream will accept, and a stream of a kind this build cannot probe — that one says so rather than passing, and keeps the value for custody.

An uploaded staff list has no credential to rotate at all. Nothing is fetched from it: it is fed, and what stands where a probe stands is the upload itself.

For a query-through warehouse the probe is the whole story, and it is the change worth knowing about if you rotated one before 0.11.0: the key pair used to sit in two places at once, and the probe now authenticates with the one copy there is, so a passing probe is evidence about the key you have just entered. It also checks the session it was given against the session it asked for, so a warehouse that no longer exists is a refusal rather than a pass. Asking a question the stream answers is still the last step, and still the only one that exercises the whole path.

The GitHub token may be the one with a clock on it. A fine-grained personal access token's lifetime is chosen when it is minted — GitHub allows a non-expiring one, but an organisation or enterprise policy can forbid that and can cap the maximum well under a year, so ask which you were given. Where it does expire, record the date and name who owns the rotation at go-live rather than at the first red line: the scheduled probe does report an expired credential, but it reports it on the day it stops working, and a token nobody owns is a token nobody renews.

Mint the replacement from the same list the first one came from — What the GitHub credential needs — and check each line rather than assuming: the same resource owner, All repositories rather than a subset, the same three repository permissions, and approved by an organisation owner where that organisation requires approval. A replacement narrower than what it replaces is the failure below, and the repository-subset version of it does not fail at all.

A replacement wider than what it replaces does not fail either, and it leaves a step in your data. Rotating a token onto a broader grant — the common case being a fine-grained token re-scoped from a hand-picked list to All repositories — is invisible to Prism: nothing errors, the self test stays green, and the newly reachable repositories start appearing in the sweeps that happen after the rotation. The days already walked keep the narrower scope for ever. No ordinary run revisits them, and neither of the two verdicts that normally mark an incomplete day can fire — capped_days describes a day that hit the per-query ceiling, unenriched_days a day whose enrichment could not be reached, and both are about rows the walk knew it missed. These are rows it never knew existed.

So a question spanning the rotation date answers low and confident, with no caveat. The repair is the same one-off re-sweep a widened grant needs — ingest.github.rewalkFrom, and ingest.jira.rewalkFrom for Jira — and the procedure, including how far back to set it and how to prove it landed, is Widen a source's scope. If the credential you are rotating to can see more than the one it replaces, do that page as well as this one.

A GitHub App has a key rather than a token, and the key does not expire. Where the credential is an App (when the credential cannot see the whole organisation), secrets.githubAppPrivateKey is the thing to rotate and there is no expiry date to diarise — which cuts both ways. Nothing will ever tell you it is old, and it is a stronger credential than any token it replaces: an installation token is scoped to one installation and lives an hour, while the private key mints tokens for every installation of that App, for as long as it exists. Treat it like the App's password, put a date in your own calendar, and prefer installing the App on the organisations you are measuring and no others — the key's reach is the App's reach, not this appliance's.

Rotating it is one helm upgrade and needs no coordination: generate a new private key on the App's settings page, upgrade with --set-file secrets.githubAppPrivateKey=./the-new.pem, confirm the pre-flight still reports "credential": "github_app" with the repository count you expect, then delete the old key at GitHub. Do it in that order: GitHub lets an App hold more than one key at a time precisely so the overlap can be arranged, and deleting first turns a rotation into an outage. Tokens already minted from the old key keep working until they expire, which is at most an hour.

If the new credential fails and the old one worked: send the self test screenshot plus which credential you changed. The most common cause by a distance is scope rather than the rotation — a replacement token issued with narrower grants than the one it replaces, which the pre-flight check will show as an org that is visible with no PRs findable.