Incidents — Overview
Incidents in CodeXray are automatic SLO violations. When an application stops meeting the reliability targets you've set for it — either availability or latency — CodeXray opens an incident, tracks it through severity changes, and closes it on its own once the application is healthy again. There's nothing to author, nothing to acknowledge, and nothing to close manually — the system is the source of truth.
- Alerts answer "has this metric crossed a threshold right now?" — one rule, one condition, one fire.
- Incidents answer "is this application still meeting its reliability promise?" — application-scoped, lifecycle-tracked, computed from SLO burn rate.
The two systems share the same underlying SLO evaluation but feed into independent pipelines, so a critical incident and a critical alert can fire for the same root cause and be routed to different channels.
What an Incident Is
Every incident has the same five facts:
| Field | What it tells you |
|---|---|
| Application | The service the incident belongs to — one incident is always tied to exactly one application |
| Severity | Critical · Warning · Resolved — driven entirely by SLO burn rate |
| Opened at | When the SLO first started burning fast enough to violate the objective |
| Resolved at | When the application's burn rate returned to healthy. Empty while the incident is still open |
| Key | A short unique ID — what shows up in notifications and URLs |
An incident only exists when an SLO is breached — it's not created by a rule, it can't be opened by a user, and it doesn't represent a "ticket" or a "post-mortem". It's a state record.
Incidents have no acknowledge, no suppress, no manual create, and no manual close in this version. Severity and resolution are computed from the underlying SLO. If you want manual workflow on top of incidents — assignment, acknowledgment, post-mortems — route them out to your incident-management tool via a Webhook or PagerDuty channel.
Two Kinds of SLO Drive Incidents
CodeXray evaluates two SLO families per application. Either one going into burn opens an incident.
| SLO | What it measures | What you configure |
|---|---|---|
| Availability SLO | The fraction of requests to this application that succeed (don't error) | objective_percentage — e.g. 99.9% |
| Latency SLO | The fraction of requests that complete faster than a target latency | objective_percentage + objective_bucket — e.g. 99% of requests under 5s |
Both are configured per application in the application's Overview view. The defaults are sensible production values; you only need to tune them if your service has a different reliability bar (an internal batch job is not held to the same target as a customer-facing API).
CodeXray computes the burn rate against the configured objective on a rolling window. When the burn rate is fast enough that the application is on track to blow its error budget, an incident opens — Warning for moderate burn, Critical for fast burn.
Severity and Lifecycle
The lifecycle is intentionally simple — open, severity may change in flight, resolved.
Three things to know:
- The same incident can change severity — a Warning can escalate to Critical without a new incident being opened. The transition is tracked on the same record.
- An incident resolves when the SLO returns to healthy — the resolver clears the
Resolved attimestamp; no human action is required. - One incident at a time per application — while an incident is open for a given application, new SLO breaches roll into that same incident as severity changes, not as new records. Resolution closes the cycle; a fresh burn opens a brand new incident.
Where Incidents Appear in the UI
Incidents are surfaced in three places — each one tuned for a different question.
1. The Incidents module — "what's burning right now?"
A dedicated top-level page that lists every open incident in the project. The view leads with three counters — Critical, Warning, and Resolved (the last is hidden by default; flip Show resolved to surface it). Each row shows the incident ID, application, opened-at, duration, current Availability and Latency compliance, the share of requests affected, and the percentage of the application's error budget burned. Filter by application search, namespace, and category. Walkthrough: see the list view section.
This is the page on-call lives in. Sort by Consumed Error Budget to find the incidents doing the most real damage to the SLO promise.
2. The incident detail view — "why is it burning?"
Clicking any incident in the list opens the detail view, which is built to answer the next question. It pulls together:
- A SLO compliance table showing both objectives in plain English (e.g. "99% of requests should not fail") with the live compliance value — a glance tells you whether you have an availability or a latency problem. Each row has an inline pencil that opens the SLO Config dialog for the application.
- A latency heatmap of request volume across the incident window, bucketed by latency band — at a glance, whether the burn is steady or bursty
- Requests & Errors and Latency charts scoped to the incident window
- A Traces tab — sampled traces from inside the incident window, available from both eBPF and OpenTelemetry SDK sources so you can pivot straight from "the burn started at 11:43" to a slow request that actually represents the failure mode
For the full walkthrough of both tabs, see Overview tab and Traces tab.
3. Application views — "is the service I'm looking at healthy?"
When you open an application's Overview, three things tie back to incidents:
- An inline severity badge next to the application name — red for Critical, yellow for Warning, grey for recently-resolved
- Bell + triangle counters in the top-right header — open alert count and open incident count, both clickable to jump back into the respective module scoped to that application
- An SLO Config button — opens the per-application SLO objectives dialog
The badge alone is the cheapest possible signal that the service you're about to investigate has a known SLO problem. See the application overview walkthrough for the full picture.
Notification Routing
Incidents use the same notification-channel system as Alerts and Deployments — channels are configured once per project and referenced from any event source. Each channel has three independent toggles:
- Alerts — rule-based firings
- Incidents — this section
- Deployments — deployment events
Flip Incidents on for any channel that should hear about SLO breaches. Channel behaviour depends on the integration:
| Channel | On open | On severity change | On resolve |
|---|---|---|---|
| Slack / Teams / Webhook | Sends an open message with app, severity, opened-at, SLO details | Sends a follow-up severity-change message | Sends a resolve message |
| PagerDuty / OpsGenie | Critical → new page; Warning → low-urgency notification | Escalates Warning → Critical, de-escalates Critical → Warning | Resolves the page |
A common pattern is to send alerts to a wide team Slack channel (lots of fires, mostly informational) and incidents to a tighter on-call channel + PagerDuty (rare, but always actionable). Configure two separate channels with the right toggles and you get this split for free.
For the channel-edit modal layout, see the Alerts walkthrough — the modal is the same one used for alert routing, with the Incidents toggle controlling whether SLO incident events flow through.
Configuring SLOs
SLOs are per-application. The same SLO Config dialog can be reached from two places — whichever is closer to where you are:
- From the application — open Applications → <your app> and click the SLO Config button in the top-right of the Overview tab
- From any incident for that application — click the pencil icon on the SLO row in the incident detail's compliance table
Both routes save against the application, so a change made from one is visible from the other.
| Field | Applies to | Example |
|---|---|---|
| Availability objective | Availability SLO | 99% |
| Latency objective | Latency SLO | 99% |
| Latency bucket | Latency SLO | 5s — the threshold latency target |
The defaults work for most production services. The two situations that justify changing them:
- Internal-only services can usually drop their availability objective (a backfill job doesn't need 99.99%)
- Latency-sensitive paths (checkout, payment, search) often want a tighter latency bucket than the project default
Once you save the dialog, the SLO evaluator picks up the new objective on its next cycle — there's no restart, and no separate apply step. Note that the new objective applies to future evaluation only; it does not retroactively re-score an already-open incident.
How Evaluation Works
The SLO evaluator runs on a fixed cycle for every application in the project. On each cycle:
- It computes the current burn rate for both Availability and Latency SLOs.
- It looks up the application's current incident state — open with some severity, or no open incident.
- It writes the transition — opening a new incident, updating severity in place, or marking the existing one resolved.
- It enqueues notifications — every channel with
for_incidentsenabled receives the transition.
The state-transition log is what you see in the Incidents history; the latest-state cache is what powers the application-card badges.
Incidents vs Alerts — Choosing What to Route Where
Both systems can fire for the same underlying SLO breach. A practical mapping:
| Signal | Best as an Alert | Best as an Incident |
|---|---|---|
| "This API just crossed 5% error rate for 1 minute" | ✓ rule on slo_burn_rate_fast or a custom PromQL | |
| "Our checkout service is burning its monthly error budget" | ✓ SLO-driven incident | |
| "CPU on payments is at 90%" | ✓ rule on cpu_container | |
| "The payments service has been unreliable for 45 minutes" | ✓ incident, with severity that escalated mid-flight |
If you're tempted to write a rule that looks like an incident (long-running, application-scoped, with severity escalation), check that you have an Availability or Latency SLO configured for that application first — you probably want it as an incident.
Next Steps
- UI Walkthrough — hands-on tour of the list view, the incident detail (Overview + Traces tabs), and the SLO Config entry points
- Alerts — Overview — the rule-based companion to incidents, sharing the same SLO evaluation and the same channel routing
- Alerts — UI Walkthrough — the notification-channel modal lives here; the
Incidentstoggle on each channel controls incident routing - Architecture — how SLO evaluation, incidents, and notifications fit into the full CodeXray pipeline