How Prism reads GitLab (and what the token needs)

Written for whoever owns your GitLab instance and whoever issues the credential. Short version: your users never generate GitLab traffic, the background sync is a single serial rate-capped job, the token it needs is read-only, and the one setting that can silently under-read is the list of groups.

GitLab is read by the ingest engine from a shipped recipe, not by a GitLab-specific program: there is no separate job to schedule and no chart credential to set. You add it in Admin → Context streams, paste a token, and the ingest tick picks it up. That means it needs sources.registry.enabled and ingest.engine.mode set to shadow or authoritative; with the engine off, nothing runs it.

1. Users never touch GitLab

When someone asks Prism a question, the agent answers from a local read-only copy of the GitLab fields it needs, held in Prism's own Postgres. It does not call GitLab at query time — there is no live drill-down for this source.

2. The sync is one background job

The ingest tick refreshes that copy — by default every 6 hours per source. It runs one request in flight at a time: no parallelism, no fan-out, no burst.

3. It's incremental

After a one-time initial load (ingest.backfillDays, default 90) every subsequent run asks only for merge requests updated since the last run, using GitLab's own updatedAfter argument with an ascending updatedAt sort — so a merge request touched mid-walk moves towards pages Prism has not read yet rather than behind it.

4. What the token needs

A project or group access token, or a personal access token, with these scopes:

Scope	Why
read_api	the merge requests themselves, through GraphQL
read_user	the author's username on each merge request
read_repository	branch names, which are how a merge request is tied to a Jira issue

It travels in the PRIVATE-TOKEN header, and it is never written to a log. Prism performs no writes of any kind against GitLab.

Two things to check before you issue it:

• Its expiry. GitLab tokens expire, and when this one does the source stops refreshing and says so on the self test. Put the date in your calendar, and see Rotate a credential.
• What it can see. The walk covers exactly the groups you list (below), and a token that cannot see one of them reads it as empty rather than as an error — GitLab answers an invisible group with a 200 and a null, with no message attached.

5. The groups you list are a coverage decision

Prism walks one top-level group at a time, including its subgroups. You give it the list; on most instances that is a handful of paths.

Anything outside the listed groups is invisible, and nothing in an answer will say so. A token can typically see far more projects than the groups it belongs to, so this list decides the population every GitLab figure is computed over. It is worth checking once, and the check is cheap: ask GitLab itself how many merge requests were updated in a period, then ask Prism the same question. The first verification question the source ships does exactly that.

If the two disagree, the usual cause is a group missing from the list rather than anything wrong with the sync.

6. What Prism stores, and what it does not

One table, one row per merge request: the project path and the merge request's number within it, the author's username, its state, whether and when it merged, the branch names, the diff size (additions, deletions, files), and the title.

• No merge request content. No description, no comments, no diffs, no file contents.
• No issues. GitLab Issues are not read at all.
• No commit content, and no commit walk. Prism does not read commit messages, diffs or history, and it does not walk the commit stream. It does read the author address on the first few commits of each merge request, and only to learn who a username belongs to — see section 7. Nothing from a commit is stored in the merge-request table.

7. Names, and how a username becomes a person

Prism can always tell you what a username did. Whether it can tell you who that username is depends on two things below — and where it cannot, it says so rather than guessing.

A self-managed GitLab discloses a user's email address on the user API only to an administrator token, so the merge request itself carries a username and nothing else that identifies a person. There are two routes past that, and they are not equally good.

Route	What it needs from you	What it yields	Status
A — an administrator-scoped token	A service token whose account can read user email addresses	Every user resolves, including people who never open a merge request. One setting, no coverage figure to watch	Not built. Prism reads no GitLab user API today; issuing such a token would change nothing until it does (tracked on #808)
B — the commit-author harvest	Developers commit with a corporate address, GitLab matched that address to a GitLab account, and you have listed the domain in ingest.github.identityEmailDomains	Only the people who authored commits Prism saw, and only where GitLab made the match	This is what runs today

Route A is the better design and it is not available yet: nothing in Prism calls GET /api/v4/users or reads public_email, so an administrator-scoped credential issued today would sit unused. It is recorded here because it is where this is going and because a token like that takes weeks to approve — not as something to go and ask for on the strength of this page. Route B is what resolves people today, and the rest of this section is about it.

Route B, in detail

Prism already asks GitLab for each merge request over one GraphQL call. That same call returns the author address on the merge request's first few commits, so the evidence costs no extra request and no extra permission. An address is recorded only when all of these hold:

• the domain is one you listed in ingest.github.identityEmailDomains. This is empty by default, and while it is empty nothing at all is recorded — a commit's author address is whatever the committer typed into their own git config, so an install that has not opted in collects none of them. Subdomains are not implied: list contractor.acme.com separately if you want it. The setting is shared with GitHub deliberately: which addresses may enter this appliance's directory is one decision, not one per source;
• GitLab resolved that address to a GitLab account. Prism pairs the address with the account GitLab matched it to, never with whoever opened the merge request. Those are usually the same person, and "usually" is the problem — a guessed join produces a confident, wrong attribution, which is worse than no attribution at all;
• and it is not a no-reply address. These resolve to an account perfectly well, which is exactly why they are dropped: they join to nothing a person is known by anywhere else. Prism drops them and keeps looking down the commit list rather than stopping at the first address it finds.

What lands is evidence, not a decision: a table of "this address was seen against this username, this many times". An address seen against two usernames stays two rows — the disagreement is preserved rather than resolved. Nothing is renamed, nothing is merged, and no record you keep elsewhere is changed.

If you connected GitLab before 0.11.0, upgrading is enough

The harvest is declared in the GitLab recipe, and until 0.11.0 a source walked the copy of its recipe it was registered from — so a gitlab source connected under an earlier release never ran the harvest, however the allowlist was set, and nothing said so. Since 0.11.0 a source held under the recipe's own name is walked under the recipe this image ships: upgrade, set the allowlist, and the next walk harvests. There is nothing to re-apply on the Manifest tab (it will refuse, and say why). A source you renamed keeps its own copy of the recipe, and for that one you do re-apply the recipe on its Manifest tab first.

Then read the next section, because the walk that starts harvesting is forward-only — and two things measured on a test install after exactly this upgrade: with the allowlist still empty the harvest ran and kept nothing, every counter at zero, which looks the same as "no addresses exist"; and a re-sweep date is claimed by the next walk the schedule allows, so setting it does not by itself make the source due.

Turning it on does not reach the merge requests you already hold

This is the one to plan for on an existing install. Setting identityEmailDomains changes what the next walk records. It does not go back over merge requests already in the mirror — those were walked when the allowlist was empty and their commit addresses were refused at the time. The walk resumes from its watermark, so it re-reads a merge request only if that one has been updated since; an old merge request touched after the flip is picked up and harvested like any other, and one that has not been touched is not.

So on an install that has been mirroring GitLab for months, switching the allowlist on resolves the people who author or touch a merge request after the flip, and nobody else. Ask "which team merged the most last quarter" and the answer is built from whoever happened to be active this week.

To reach the history, set a re-sweep date at the same time:

# values.yaml
ingest:
  gitlab:
    rewalkFrom: "2026-06-01"    # YYYY-MM-DD, earlier than the mirror's watermark

It runs once per date, when the tick next finds the source due, and re-walks from that day forward — harvesting the addresses on every merge request it passes. Set it to the beginning of the period you want people resolved over, and expect the walk to take proportionally longer that once. docs/install/go-live.md carries the procedure.

A re-sweep that does not finish is not resumed, and says nothing. The date is claimed before the walk, and the source's watermark then takes precedence over any resume point — so a re-sweep interrupted part way through leaves the days it never reached unharvested, and the next tick carries on from the watermark as though the sweep had completed. Nothing retries it and nothing reports it as incomplete. This is worth planning for on GitLab specifically, because a walk dying part way is the failure that actually happens on a large self-managed instance: a slow nested field has killed this one before now.

So check it landed rather than assuming it did, and if it did not, name a different date to claim a fresh sweep — re-setting the same one is a no-op, which is what "once per date" means. The coverage and identity_evidence figures in the gitlab row of ingest_state are what tell you: coverage says how far back the mirror reaches, and a kept count that stopped climbing across runs says the harvest did not get where you sent it.

Leaving it unset is a supportable choice — coverage then builds up as people open merge requests — but it is a choice, and it is worth making deliberately rather than discovering at the first question about a quarter.

Expect to measure this, not to assume it

How much route B yields depends entirely on your instance, and one thing in particular is worth checking early: a merge request whose commits were mirrored in from another forge has no GitLab user behind them, so those commits resolve to nobody however good the address is. Every run reports the count under identity_evidence in the ingest_state row for the gitlab source, as five numbers — how many pairs were kept, and how many were dropped for each of: no address at all, a no-reply address, no GitLab account, and a domain you did not list. If no_account dominates, route B is not going to work on your instance and route A is the answer.

Where a username does not resolve, Prism reports GitLab activity on its own, grouped by username, and will not join it to a Jira assignee or a GitHub login. Counts, rankings, cycle time and merge-request size are unaffected — what you do not get is a person behind that username, or any team-shaped answer, since a team is a property of a person in your directory.

8. The link to Jira is the branch name

A GitLab merge request carries no declared link to a Jira issue, so Prism reads one out of the source branch name: an uppercase project key followed by a number, as in feature/ABC-1234-short-description. A branch with no such key records no Jira issue.

Two consequences worth stating plainly:

• This is a ceiling on cross-source questions. Whatever share of your branches carries a key is the share of merge requests that can ever be tied to Jira work. On the instance this recipe was measured against it was about 30%.
• A missing key is not untracked work. It means the branch did not name the issue. Prism reports it that way and never as a compliance figure.

Lower-case keys are not read, deliberately: matching case-insensitively turns an ordinary branch like hotfix/abc-12 into a confident reference to a Jira issue that does not exist.

9. Rate

The walk is capped at 9 requests per minute by default (GITLAB_MAX_REQUESTS_PER_MINUTE), which is what a large self-managed instance sustained comfortably while returning a hundred merge requests per request. If your instance publishes rate limits or your platform team would rather it went slower, lower it; there is no lower bound at which the sync stops working, only a longer first load.

A full first load of a quarter's merge requests is typically a few dozen requests. After that, each cycle is normally one or two.

9a. A page GitLab cannot finish

GitLab computes a merge request's diff sizes on the way out, and it gives the whole query about thirty seconds. One merge request that is expensive to diff — a rewritten branch, an unreachable merge base; the size of the diff is not the tell — can push a page of a hundred over that line. When it does, GitLab still answers, at about thirty-two seconds, with a page whose diff sizes are missing and an errors list saying Timeout on DiffStatsSummary.additions once per merge request on it.

Prism keeps that page. The merge requests on it are mirrored with their additions, deletions and changed-file counts empty, the walk goes on, and the run's log ends with a WARNING naming the field that was withheld and how many values. The same is recorded in ingest_state.detail.graphql_withheld for the last completed run only — the next completed run replaces it — so the log is the record to keep, and on a run that fails for another reason it is the only one. Nothing else about those merge requests is affected — but the sizes do not fill in by themselves, and re-walking the window does not fill them either. The merge request that is slow to diff is slow every time it is asked for, so the page that holds it fails the same way on every fetch. The remedy is in GitLab: repair or remove that merge request (the log names the page; fetching its merge requests one at a time with the API finds the one that takes tens of seconds), after which every page answers in time at any page size.

Two things have to be true for that to happen, and both are Prism's to get right rather than yours:

• Prism waits longer than GitLab does. The sync's HTTP read timeout is ingest.engine.httpTimeoutSeconds, 120 by default. It has to be longer than GitLab's own deadline, or Prism gives up two seconds before GitLab tells it what went wrong — which is what a log full of transport error (ReadTimeout) lines, six per page, means on a version before 0.10.2. Raise it if your instance is slow to answer in general; there is no reason to lower it.

• An answer that will not change is not retried. A page that comes back with errors and no merge requests at all is asked for once more, and if the second answer is the same the sync stops asking, says which page (its cursor and window) and which field, and moves on to the next group. A page that does not come back at all is asked for three times, not six. A group the token cannot read is different: GitLab says so in the answer, and the walk ends there as it always did — with no rows, and now with the reason in the log — rather than being treated as a page to retry.

• One group that will not answer no longer costs you the others. The sync has given up on such a group and walked the rest since 0.10.2. What 0.12.0 adds is the half that made those rows usable: the run now completes rather than being reported as failed, so the source shows a last-refreshed time instead of reading as one that has never run, it is not held off for an hour after every attempt, and it states its coverage. Before 0.12.0 the rows landed and nothing said so.

  What the sync will not do is pretend the coverage is whole, and this is the part to read before you rely on a figure:

  • Every figure over that group is a floor, at any point in its history. Whatever the group had already sent before it stopped is kept; the rest was never fetched. An answer over the source says so in words — a group that did not finish is reported as missing data rather than as a group with no merge requests — and the sync's own log names the group.

    The admin pages carry it, which is the place to look when you are not asking a question. The source's row under Admin → Context streams says how many of its groups did not finish the last refresh and names them, and so does its card on Ingestion activity. The names are the same ones an answer gives, capped the same way: the first five, then "and N more".

    The self test carries the count and not the names, deliberately, and that difference is not an oversight: The self test reports one entry per source under ingest, and a source that came back short says how many of its groups did not finish. That report is a file you may hand back to us, and a group path is your configuration rather than ours to publish — while the two admin pages are read by your own operators on your own appliance, where the names are what makes the fault actionable.

    It is a caveat about the refresh, not about recent dates. The sync walks by each merge request's own last-updated time, so an old merge request updated last week is fetched in last week's pass: a group that failed then can leave a figure short for a month a year ago. Do not read the caveat as applying only to the last few days.

  • No progress is recorded, so the next run repeats the whole window — for every group, not just the one that failed. That is what makes a group that was merely slow or briefly unreachable heal itself without anyone doing anything, and it is why a group that is permanently silent should be taken off the list rather than left to fail. Two costs, and the second is the one to plan around: the repeated window re-walks the groups that DO answer on every run, and until one run fetches every group the sync has no completed run to resume from — so each run empties the table and refills it, the way a first run does, and readers see a partial table while it does. A group that will not answer for days should come off the list until it will.

  • If NO group answers the run still fails, and the sync retries it on its backoff. Nothing was fetched, so there is no partial coverage to report.

10. What it looks like when it is working

On Admin → Context streams the source shows a last-refreshed time and a row count. On the self test, its line names the tables it filled and how fresh they are. If the token expires or a group stops answering, the freshness check fails there first — see The self test and Known failure modes.