What Prism is pulling, right now

http://<prism>/ingest-activity

The page to have open while a first backfill goes in, and the page to open when someone asks what is calling their GitHub or their Jira. It shows, per source: what we are pulling, from where, at what rate, and how the upstream is answering — live, refreshed every few seconds, without a shell.

It is the companion to the ceiling you set. ingest.github.maxRequestsPerMinute and ingest.jira.maxRequestsPerMinute decide how fast Prism may call; this shows how fast it actually is, against that number. See Take a source live for setting it and limits.md for what it costs.

What it will not tell you

It records no content, and cannot. Endpoint shapes, counts, outcome classes and timings — never a repository, a ticket, a person, a branch, or the text of a query. The endpoint column can only ever hold one of a fixed list of strings written into Prism's own source code; a path with an id in it is templated (GET /rest/api/2/issue/{key}/changelog) before anything is counted, and anything unrecognised is recorded as other. The downloaded file is the same data the page draws from — it additionally carries a few things the page does not print, all of them your own install's configuration: the ingest schedules as cron strings, and the exact timestamps behind the "2d ago" figures. Nothing in it comes from your GitHub or Jira, so it is safe to attach to a ticket or hand to a reviewer.

It is our record, not your audit log. It is an honest account of the calls Prism made. If you need to know what reached your instance, your instance is the authority; this is the side of the conversation we can answer for.

It counts the ingest runs. The reachability probe (prism-ingest check) calls each upstream a handful of times on its own schedule — see ingest.checkSchedule — and is reported on this page as a verdict per source rather than counted in the rates. It is a few calls every half hour and it is not what a rate complaint will ever be about, but it is not nothing, and the page says so rather than letting you find out.

Reading a source

Each source gets a card: the built-in ones, and every context stream you have registered and switched on under Admin → Context streams. A stream you have registered and not switched on gets no card here — it is pulling nothing yet, and the Context streams page is where a registration in progress is worked on.

GITHUB          RUNNING          mode: live

  4.2 requests/min
  measured over 5 min · ceiling 30/min

  [ ▁▃▅▅▆▅▅▅▄▅▅▅▅▆▅▅ ]
  60 min ago            412 request(s) in the window            now

  backfill              day 34 of 90
  last completed run    2d ago
  partitions short in last run   2 — core-banking, payments
  runs every            6h
  data swept through    2d ago
  upstream check        reachable (12m ago)
  refused by upstream   0 call(s)
  failed calls          0

The rate is the headline, in requests per minute — the unit any limit your platform team quotes will be in. It counts every attempt, including retries and calls your instance refused, because that is what your instance saw. A run being throttled cannot answer by calling faster, so this number stays under the ceiling even while Prism is backing off.

A dash rather than a number means nothing has been recorded in the last few minutes. That is different from zero and the page keeps them apart on purpose: "we have sent nothing" and "we cannot tell" are different answers.

The bar chart is the last hour, one bar per minute. Empty minutes are drawn empty, so a run that stopped forty minutes ago looks like one. A bar turns amber where the upstream refused calls in that minute.

The state is one of:

State	Means
RUNNING	calls have gone out in the last few minutes
IDLE	between scheduled runs — normal on a healthy install
LAST RUN FAILED	the last run finished without leaving the source serveable; the self test says why
NEVER COMPLETED	live, but no run has ever finished — a backfill that keeps dying looks idle between attempts, and this is how you tell
NOT LIVE	still serving synthetic data

RUNNING is read from the calls themselves, not from anything the job writes about itself. A GitHub backfill records its progress only as each day bucket completes, so a job two hours in has no other way of saying it is alive.

day 34 of 90 is the in-flight backfill's own count of completed day buckets, and it is the same figure Prism's answers use to describe their own coverage while the refill runs — so the page and the answers can never disagree. It disappears when the backfill finishes, at which point data swept through is the better reading.

runs every is the interval, not a next-firing time. Cron fires on the clock rather than on an offset from the last run, and Prism cannot see the CronJob, so a predicted "next run" would be wrong every time a run was late or manual. It is blank for a registered context stream: the interval comes from the built-in sources' own schedule settings, and inventing one for a stream that has none would be a claim the page cannot check.

partitions short in last run is the line to read when a source looks healthy and its figures look low. A source that walks per group — a GitLab group, a Jira project — can come back having fetched some of them and not others, and that run completes: the state stays IDLE, last completed run is recent, rows are there and the source keeps answering. This is the only line that says otherwise.

It names the groups, up to five and then "and N more" — the same names, capped the same way, that an answer over the source gives a reader. Whatever a group had already sent before it stopped is kept, so figures covering it are a floor rather than a total, and an absence of rows for it is missing data rather than a finding.

none means the last run fetched every group, which is worth seeing after you have fixed one. No line at all means the source keeps no such record — it has no groups to walk, or it has not run under a version that records this.

Answering "something is hammering us"

The three readings, in order:

1. Is it us? If every source reads IDLE with no bars in the last hour, it is not us. Screenshot the page.
2. How hard? The rate, against the ceiling. Quote both.
3. Are they already pushing back? refused by upstream counts calls your instance rejected — a rate limit, primary or secondary. A number climbing there means your instance is already telling Prism to slow down and Prism is already backing off, and the ceiling should come down.

To turn it down, lower the ceiling and upgrade — the change takes effect on the next run, and nothing needs re-ingesting:

helm upgrade <release> <chart> -n <namespace> -f <your values file> \
  --set ingest.github.maxRequestsPerMinute=5

A run already under way keeps the ceiling it started with. To bring that one down now, delete its Job; the next scheduled run resumes from the day bucket it had reached rather than starting over.

Who can see it

The same people who can run the self test: a super-admin under an identity front door, and anyone under single-operator identity. It is a browser page and a sign-in, so the people on a bridge call can read it without cluster access — which is the point of it existing.

How far back it goes

The page draws the last hour. The record behind it is kept for a fortnight and survives pod restarts, upgrades and node moves, because it lives in the datapond rather than in a log — a pod being gone is exactly the case where somebody is asking. It is a few thousand small rows a day and needs no attention.