nightjar

How an incident runs

Nightjar takes over after the alert, end to end on GitLab.

Flow: probe, rollback, incident, Duo fix, proof, risk gate, staging, production, postmortem
probe → rollback → incident → Duo fix → proof → risk gate → staging → production → postmortem.
  1. Probe. A scheduled probe checks real customer journeys in production. When one fails, the pipeline rolls back first.
  2. Incident. An incident issue is opened with the exact failing request, expected vs. actual response and the suspected code location. It is assigned to the GitLab Duo Developer Flow.
  3. Fix. AGENTS.md makes the agent write a regression test first, then the smallest fix.
  4. Proof. The new test must fail on the old code and pass on the new code, and no existing test may be changed.
  5. Gate. A risk gate decides how much autonomy the change gets.
  6. Release. No-traffic staging revision, smoke test, then exactly that revision is promoted. The pipeline writes a postmortem into the incident.

Proof: red before, green after

Risk gate comment on merge request 3: score 16, auto-merge allowed up to 30, regression test failed on base and passed with the fix
Gate comment in MR !3, score 16: it merges on its own.

The risk gate

The gate runs the tooling and policy of the target branch, so a merge request cannot loosen its own gate.

Postmortem written by the pipeline

Postmortem comment on incident 3: probe failed 15:55:31, incident opened 15:56:48, fix MR 15:59:13, merged 16:02:48, verified 16:09:16
Postmortem comment on incident #3.

Architecture

Architecture: GitLab runs the loop, Google Cloud Run in europe-west9 hosts the app, images in Artifact Registry, keyless via Workload Identity Federation
GitLab runs the loop, Google Cloud Run (europe-west9) hosts the app, keyless via Workload Identity Federation.

What this does not claim