Flip a source to the ingest engine

Prism reads each of your systems twice over in the course of its life: first through a legacy lane, a hand-written module per source, and — as each source is ported — through the ingest engine, one generic walker driven by a manifest document that describes the source rather than by code written for it. The engine is what every new source arrives on, and the two sources that have a legacy lane — GitHub and Jira — are being moved onto it one at a time. This page is how you move one, what to check afterwards, and how to put it back.

This is not the same flip as taking a source live. That one turns a source from synthetic data to your real system. This one changes which of two pieces of Prism does the reading, and the source stays live throughout.

The three modes, for one source

Mode	Who writes the real tables	What the self test shows
off	The legacy lane, alone. The engine does not run for this source.	The source's usual rows, freshness and stall verdict, and no parity fragment unless an earlier shadow run left one behind.
shadow	The legacy lane. The engine walks the same window beside it into <table>__shadow and writes a parity verdict.	The usual verdict, plus shadow parity: <source> … on the source registry line.
authoritative	The engine. The legacy lane's scheduled job runs the engine too, so there is exactly one writer.	The usual verdict, unchanged. The parity fragment stops moving.

shadow is a measurement mode and nothing else: the lane that serves your users is the one that always served them, and it is untouched. authoritative is the flip. The legacy module ships for one more release after a source flips, as the rollback.

The two values

ingest:
  engine:
    mode: shadow            # install-wide: off | shadow | authoritative
  github:
    engineMode: authoritative   # this source alone
  jira:
    engineMode: ""              # empty — inherits ingest.engine.mode

ingest.engine.mode is what a source with nothing of its own runs in. ingest.github.engineMode and ingest.jira.engineMode replace it for that source, in either direction: up, to promote one source ahead of the rest, or down to off, to hold one source back while the others go on. Empty is the default and inherits. Each is spelt exactly off, shadow or authoritative; anything else — Shadow, authoritive, an unquoted bare off, which YAML hands the chart as a boolean — fails the helm upgrade naming the key and what you typed, rather than quietly falling back.

ingest.engine.mode is the master switch. While it is off — the default — a per-source value of anything but off is refused, at helm upgrade and again at ingest start-up, naming both keys. In off the chart renders no ingest tick and no engine at all, so a per-source mode under it would be an instruction nothing could carry out. Arm the install first, then raise or lower one source.

Only GitHub and Jira have these keys, because they are the only two sources with a legacy lane to flip from. A source that arrived on the engine — GitLab, a file source, a warehouse recipe — has only ever had one reader.

If you are reading a pod spec rather than a values file, these are INGEST_ENGINE, INGEST_ENGINE_GITHUB and INGEST_ENGINE_JIRA. The chart renders both the install-wide and the per-source value raw and the ingest resolves the precedence, so what kubectl describe shows is what your values file says, not a computed answer that could disagree with it. You do not set these by hand on a chart install.

Both keys are in Parameters.

What the flip is gated on

GitHub — a parity streak you earn on your own install

In shadow, every engine run for a source compares its shadow tables against the lane's and records a verdict. The self test's source registry line carries the count:

PASS  source registry   … · shadow parity: github ok 3/3 ready

Read it as status, streak, required, earned. A passing verdict is ok — the two agree — or legacy_behind, where they differ only where the new engine is ahead of the mirror it would replace. That is evidence for the flip rather than against it, and it happens two ways:

• the engine holds a later version of a row, by the source's own change stamp, because the old mirror never came back for it;
• the engine holds a value the old mirror does not have at all, reported as engine_ahead with a count per column. GitHub's search serves at most 1,000 results for one query, and the old mirror simply stopped there when enriching a busy day; the engine narrows its window and fills in what was missed.

A diff is a real disagreement and resets the count to 0. Values the engine is missing, and values the two lanes disagree about, are always a diff — the engine being ahead is allowed, being behind never is.

ingest.engine.parityRuns (3 by default) is how many consecutive passing verdicts earn the word ready. A run counts only where the comparison window has moved since the last one counted, so several engine ticks between two legacy sweeps are one piece of evidence rather than several — at the default six-hourly GitHub schedule, three runs is about eighteen hours of evidence. prism-ingest check parity github re-compares on demand, prints the verdict, and never advances the count.

A verdict that carries a caveat — one lane's rows could not be windowed against the other's, most often because the legacy lane has not stamped a cursor yet — never reads ready, however long the streak is. It is not evidence, and the word is withheld rather than the number.

Wait for ready before you flip GitHub. If the count sits at 0/3 through verdict after verdict with nothing apparently wrong, read When a shadow parity verdict will not clear — there is one cause that does not clear by itself, and its remedy is ingest.github.rewalkFrom.

Jira — verified before the release, not on your install

Jira is gated differently, and deliberately. The engine's Jira port is verified before a release ships: the legacy lane and the engine walk the same seeded Jira and the same comparator that produces the verdict above diffs them column by column, in the release's own test suite. When that verification holds, the release note says so, and that is what ingest.jira.engineMode: authoritative is gated on.

Running the two lanes side by side against your Jira for weeks to re-establish the same thing would produce one of two outcomes: agreement, which the release already asserts, or a defect in a lane that is being retired. Neither is worth a second walk against a production Jira. So there is no streak to wait for on this source.

The machinery does not know that, and will not pretend otherwise: a Jira source left in shadow does accumulate a shadow parity: jira … fragment, and you are welcome to read it. It is simply not the gate. If you would rather your Jira never took the second walk at all, hold it out with ingest.jira.engineMode: "off" while the install is armed for GitHub.

Flip one source

Change the one value and upgrade:

ingest:
  engine:
    mode: shadow
  github:
    engineMode: authoritative

helm upgrade <release> <chart> -n <namespace> -f <your values file>

The value is rendered into that source's ingest container, and the container's text is hashed into the -ingest-now- Job name, so the upgrade starts a run of that source immediately rather than waiting for the next scheduled firing. For authoritative that immediate run is the engine's first run against your real tables. Treat it as a real run against a real upstream — but on a source the legacy lane has already ingested it is an incremental run: the engine reads the progress that lane recorded and continues from it, keeping every row. Budget a full backfill only where no run of that source has ever completed here.

Afterwards:

• The SOURCES lines are unchanged. The self test still reports github (and github_issues) by name, with the same row counts, the same freshness and the same stall budget against the same schedule. Nothing about the flip moves a source onto a different line or a different verdict — that is the point of it.

• The ingest log names the engine. The scheduled job opens with github engine (authoritative): request rate … against …; tables … instead of the lane's own opening line. prism-ingest github inside the pod does the same.

• The parity fragment stops moving. A verdict is written only by a shadow run, so once GitHub is authoritative nothing writes github__shadow any more. The fragment does not disappear — the row is still there — it simply freezes at whatever it last said, ready included. That is expected and is not a regression. If you need it live again, put the source back on shadow.

  prism-ingest check parity github re-records the verdict, not the count. The comparison is bounded above by the earlier of the two lanes' watermarks, which after the flip is the frozen shadow copy's own — so every row the engine has written or updated since the flip is withheld from both sides rather than counted as a difference, and the usual result is the verdict you already had with a fresh timestamp and the streak written back untouched (the row's reason reads unchanged: re-check, not an engine run). The one thing that does move it: a day the engine caps or cannot enrich after the flip is recorded against the live source and not against the frozen shadow copy, and that difference in the bookkeeping is compared — the verdict is then diff and the count resets to 0. Either way the number says nothing about a source that is already authoritative; read its SOURCES line for that. To earn a fresh streak, put the source back on shadow and let the two lanes run side by side again.

Flip the second source the same way, on its own upgrade, once you are satisfied with the first.

After the upgrade that carries this release

Every parity streak recorded by an earlier build is discarded. A streak now claims N consecutive passing runs of a particular configuration, and it is keyed on the manifest the engine walked plus the version of the comparator that judged it — so a count recorded before this release, under rules that let one sweep's evidence be counted several times, cannot be carried across.

Concretely: the number on the self test does not change at the moment you upgrade, because only a run rewrites it. The first engine run for a source after the upgrade discards the old count and starts a new one at 1/3 (or 0/3, if that first run does not produce a passing verdict, or has no settled window to compare over yet). An install that read ready yesterday will read a small number today. Nothing is wrong; the evidence has to be re-earned on the comparator that will actually judge the flip.

Re-earn the streak before you flip GitHub. If you had already flipped it on a previous release, this changes nothing — you are past the gate and no shadow run is being made.

Roll back

Put the source back to shadow — or delete the key, if the install-wide mode is where you want it:

ingest:
  github:
    engineMode: shadow

helm upgrade <release> <chart> -n <namespace> -f <your values file>

The same thing happens as on the way in: the container text changes, the -ingest-now- Job fires, and that run is the legacy lane's. From then on the lane serves and the engine goes back to walking beside it.

What is proven. The two lanes keep their bookkeeping in the same ingest_state row, in the same shape, and that is not an assumption — the parity comparator reads it on every shadow run. Be precise about what it checks, because it is not a field-by-field equality:

• capped and unenriched days are compared entry for entry over the window, and a day either lane records and the other does not is a diff, in either direction;
• the horizons are a containment check, not an equality one: the engine must not claim coverage from earlier than the legacy lane, except as deep as its own manifest declares;
• the resume point is reported when the two differ, and never counted as a diff — the two lanes are at different points in their walks by design;
• the watermark is not compared at all. The earlier of the two lanes' watermarks closes the window the rows are compared over: a key either lane has touched at or after it is withheld from both sides, so one lane having swept further than the other cannot read as a difference. It does not bound the two lists above — capped and unenriched days are compared from the horizon the two lanes share onwards, whatever the watermarks say.

So what is proven is that the row the engine writes is one the legacy lane can read and reason about — the same keys, the same shapes, the same meanings — and not that the two lanes' rows are identical.

What is not proven. A rollback after a long authoritative run has not been rehearsed end to end on a populated install. Assume the lane's next run is an ordinary incremental one, from the watermark the engine left. If you want certainty over the ground the engine covered rather than an assumption about it, re-sweep it:

helm upgrade <release> <chart> -n <namespace> -f <your values file> \
  --set ingest.github.rewalkFrom=<the date you flipped>

That costs a second pass over the window, so give it the room the original backfill took, and blank the value once the sweep has finished — see Widening the window later for what a re-sweep does and why it happens once per date.

What authoritative does not do

The GitHub COMMIT-AUTHOR harvest does not run. The harvest that derives an email ↔ GitHub-login join from commit metadata is a selection on the legacy lane's GraphQL query; the engine's manifest has no commit-author step. So while GitHub is authoritative — by ingest.engine.mode or by ingest.github.engineMode, whichever decided it — that harvest writes no identity_observations rows for this source, and ingest.github.identityEmailDomains has no effect on that step. It is not inert: the same setting narrows the profile sweep, next.

It is not the only writer any more. From 0.11 the identity sweep reads each organisation member's published profile address, and it runs on the ingest tick rather than on either lane — so it is indifferent to which mode GitHub is in, and identity_observations goes on filling under authoritative. The one thing it is not indifferent to is ingest.engine.mode: off, which renders no tick at all and therefore no sweep. See below. Pull requests, issues and reviews are unaffected. The ingest says so in its log on every run where you have named domains at all, and prism-ingest check github reports it as identity.harvest_runs_in_this_mode: false.

This is accepted, not a defect to wait out. Prism keeps no directory of its own any more: the people in your organisation come from the source you designate as your organizational context stream, and that is the designated route. The crosswalk that correlates people across systems still folds in any identity_observations it finds, so the consequence is precise rather than sweeping.

So it is a choice, and the page says which you get: flip GitHub and the commit-author evidence stops accruing; keep GitHub on shadow and it goes on. See How Prism reads GitHub for what that harvest collects and what it refuses.

From 0.11 this is a much smaller trade than it was, and holding a source back for it is almost certainly the wrong call. The identity sweep that reads each organisation member's published profile address is a step on the tick rather than on the legacy lane, so it runs whatever mode GitHub is in and it is on by default. An install running GitHub authoritative therefore still accrues email-to-login evidence; what it gives up is only the commit-author harvest, which is off until you name domains and which reads addresses out of git config rather than out of your own organisation's records. Before 0.11 this section was the reason to leave GitHub on shadow. It is no longer that reason.

Where the legacy lane goes

It stays. For one release after a source flips, the hand-written module is still in the image and is still what a rollback lands on — which is what makes the rollback above a values change rather than a downgrade. A later release removes it, and removes these mode values with it: at that point the engine is simply how Prism reads that source, and there is nothing to flip. The release note will say so before it happens.