Alerts — Overview

Alerts in CodeXray turn the same telemetry that powers your dashboards — metrics, logs, traces, and SLOs — into actionable signals delivered to the channels your team already lives in. There's nothing extra to instrument: every rule evaluates against data CodeXray is already collecting, and every fire links back to the application, log pattern, or trace that triggered it.

One alerting surface for everything

Most platforms split alerting across "infrastructure alerts", "log alerts", and "SLO alerts" — each with its own UI and quirks. CodeXray keeps all three in a single rules engine, with one notification model and one history view. Whether you're watching CPU, an error log pattern, or an SLO burn rate, the workflow is the same.

What You Get Out of the Box

Drop CodeXray on a cluster and you immediately have 25 built-in inspection rules — production-tested defaults that cover the things teams almost always want to be paged on. Each one wraps a CodeXray inspection (the same checks that power the application health view), so the rule and the dashboard always agree about what "unhealthy" means.

Category	Built-in Inspection	Inspection ID
CPU	Container CPU Utilization	`cpu_container`
	Node CPU Utilization	`cpu_node`
	Container CPU Throttling	`cpu_throttle`
Memory	Memory Leak	`memory_leak`
	Out of Memory Kill	`memory_oom`
	Memory Pressure	`memory_pressure`
	Container RSS Memory High	`memory_rss_high`
Availability	Instance Availability Low	`instance_availability`
	Container Restarts	`instance_restarts`
	Deployment Stuck	`deployment_status`
Network	TCP Connection Failures	`network_connectivity`
	Container Network Latency High	`network_rtt`
	Cross-Namespace Network Latency High	`network_rtt_cross_cluster`
	External Network Latency High	`network_rtt_external`
Storage	Container Disk I/O Delay High	`storage_io_latency`
	Disk I/O Load High	`storage_io_load`
	Container Disk Space Low	`storage_space`
Logs	Log Errors	`log_errors`
JVM	JVM Availability	`jvm_availability`
	JVM GC Pause Time High	`jvm_gc_pause`
	JVM Safepoint Time High	`jvm_safepoint`
.NET	.NET Runtime Availability	`dotnet_availability`
SLO	SLO burn rate (fast)	`slo_burn_rate_fast`
	SLO burn rate (slow)	`slo_burn_rate_slow`
	SLO burn rate (daily)	`slo_burn_rate_daily`

Each built-in rule is enabled by default and can be tuned without writing PromQL — adjust severity, for duration, scope to specific applications, attach notification channels, or disable it entirely.

Three Ways to Create a Custom Rule

When the built-ins aren't enough, custom rules cover the rest. CodeXray supports three creation modes — pick the one that matches what you're trying to detect. For teams provisioning rules programmatically, the bulk API accepts up to 500 rules per request.

1. Wrap a Built-in Inspection

The fastest path. Pick any inspection from the table above, give it a name, and override only what you need (threshold, duration, scope). Behind the scenes the rule shares its evaluation logic with the matching inspection, so it stays consistent with the rest of the platform as inspections evolve.

Use this when you want the same check as a built-in, but with different bounds — e.g. a stricter restart threshold for your payments service than the cluster default.

2. Plain PromQL

Drop in any PromQL expression, choose a comparison operator, and set a threshold. Every metric CodeXray collects — node, cluster, application, database, JVM, .NET, custom — is queryable.

avg by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])) * 100

Use this when your team already speaks PromQL or you need a derived metric (rates, percentiles, joins) that no single inspection covers.

3. Log Pattern

Fire when a log pattern matches with a configurable frequency. You provide a substring to match, the log levels to watch (error, warn, …), and a minimum count over the evaluation window.

Because CodeXray clusters logs into patterns at ingest time, log-pattern alerts don't fire on every matching line — they fire on the pattern crossing your count threshold, which is what you actually care about.

Use this when the signal lives in the logs, not the metrics — startup errors, vendor-API failures, dropped messages.

UI-only mode

Log-pattern rules can only be created from the UI. The bulk API (see below) supports the other two modes — wrap-a-builtin and plain PromQL — but not log patterns. If you're scripting rule deployment, plan to either author log-pattern rules manually in the Rules tab or accept that they're outside your config-as-code workflow.

Which mode should I pick?

Wrap a built-in when the check already exists. PromQL for anything metric-derived. Log Pattern when the signal only appears in stdout/stderr. You can mix modes in the same project — they all share the same evaluation engine, severity model, and notification routing.

Anatomy of a Rule

Every rule — built-in or custom — exposes the same set of knobs. The vocabulary is consistent across the UI and the bulk API.

Field	What it does
Name	Display name shown in the UI and notifications. Unique within the project, max 30 chars.
Severity	`warning` or `critical`. Drives routing, colour, and the badge in the alerts table.
`for` duration	How long the condition must hold before the rule fires. Format: `30s`, `5m`, `2h`. `0s` fires immediately.
`keep_firing_for`	How long the alert stays open after the condition recovers. Suppresses flap-y notifications.
`repeat_interval`	While an alert keeps firing, CodeXray re-sends a reminder every interval. Built-in rules use `5m`; custom rules start at `0s` (no reminder) until you set one.
Application scope	Which applications the rule targets — all, a category, or a specific list. Empty = every matching application.
Enabled	Pause/resume without deleting the rule.
Summary template	The one-line text rendered into the notification body. Supports the tokens `{value}`, `{count}`, and `{bytes}`, filled with the live numbers when the rule fires.
RCA on fire	(Custom rules) When enabled, CodeXray generates an AI root-cause summary for the fire, shown in the alert detail — see below.

AI-Powered RCA on Fire

Flip RCA on fire on a custom rule and every time it fires, CodeXray's analysis engine produces a plain-English root-cause summary for that specific incident. (The shipped built-in rules don't trigger RCA — wrap a built-in as a custom rule if you want RCA on that check.) It correlates the firing condition against:

recent log patterns and severity shifts on the affected application
deployment events on or near the fire window
trace error spikes for upstream and downstream services
CPU / memory / network deviations on the same instances

The summary lands in the alert detail view in the Alerts module, so the on-call engineer has a starting hypothesis before they open the dashboard. It's generated asynchronously the moment the rule fires — the notification goes out straight away, and the RCA write-up appears on the alert a few moments later.

Keep it for the alerts that wake people up

RCA generation is genuinely useful but isn't free. The recommended pattern is to enable it on critical rules — the ones that page someone — and leave it off for low-severity informational rules.

Notification Channels

A notification channel is a named group of integrations, configured once per project and referenced by ID from any rule. Within one channel you switch on any mix of the four integrations an on-call rotation needs:

Integration	What you provide	Best for
Slack	Bot token + default channel	Day-to-day team awareness; warning-level rules
Microsoft Teams	An incoming-webhook URL for the target channel	Posting alerts into a Teams channel your team already watches
Webhook	URL + optional custom headers	Routing into your own incident systems, in-house bots, or other vendors
PagerDuty	Integration key	Critical paging straight to the on-call schedule

Because integrations live inside a channel, one channel can fan out to several at once — every critical alert can post to #oncall in Slack, drop a card in Teams, and page PagerDuty simultaneously. Each integration is toggled independently for Alerts, Incidents, and Deployments.

How a firing alert notifies

On fire, CodeXray sends one notification to each integration (with Alerts on) in the rule's attached channel.
While it keeps firing, a reminder goes out every repeat_interval. Slack keeps reminders in the alert's own thread and PagerDuty folds them into the single open incident, so you don't get spammed; Teams and Webhook receive each reminder as a fresh message.
On resolve, a final notification closes it out — Slack threads it under the original message, PagerDuty resolves the incident.

An empty channel list means "notify no one"

A rule with no channels attached deliberately notifies nobody — it's the explicit way to say "track this, but don't page anyone." If you want a rule to reach someone, attach at least one channel to it.

How Evaluation Works

Built-in inspection rules and the inspection-wrapping custom rules share their evaluation logic with the inspection itself — there's one source of truth per check. Plain PromQL and log-pattern rules evaluate on the same cadence as the rest, against the same backing store, so a rule never disagrees with the chart on the application page.

Managing Rules at Scale — The Bulk API

For teams managing dozens of services, CodeXray exposes a bulk-create endpoint for custom rules.

Bulk endpoint capabilities and limits

Up to 500 rules per request, 4 MB body cap, whichever comes first.
Only two of the three creation modes are supported via this endpoint — wrap a built-in inspection (set type to an inspection ID) and plain PromQL (set type to ""). Log-pattern rules must be created in the UI.
Each rule is validated independently — a typo in one entry doesn't reject the batch.

POST /api/{tenant}/project/{project}/alerts/rules/custom/bulk
Content-Type: application/json

A minimal request creating one PromQL rule and one inspection-wrapping rule:

{
  "rules": [
    {
      "name": "High CPU on payments",
      "query": "avg by (instance)(rate(node_cpu_seconds_total{mode!=\"idle\"}[5m])) * 100",
      "operator": ">",
      "threshold": 85.0,
      "severity": "critical",
      "for_duration": "5m",
      "keep_firing_for": "10m",
      "application_ids": ["codexray:Deployment:payments"],
      "notification_channels": ["11111111-1111-1111-1111-111111111111"],
      "rca_enable": true,
      "type": ""
    },
    {
      "name": "Container restarts (web)",
      "type": "instance_restarts",
      "severity": "warning",
      "for_duration": "0s",
      "keep_firing_for": "0s",
      "repeat_interval": "1h"
    }
  ]
}

The response is per-rule, so a typo in one entry doesn't reject the batch:

{
  "created": 1,
  "failed": 1,
  "results": [
    { "index": 0, "name": "High CPU on payments", "status": "created", "id": "<uuid>" },
    { "index": 1, "name": "Container restarts (web)", "status": "failed", "error": "unknown builtin rule type: typo_id" }
  ]
}

Validation rules to remember:

name ≤ 30 chars, unique per project
Durations match <positive int>(s|m|h) — no decimals, no compound forms
For PromQL rules: query, operator, threshold, severity, for_duration, keep_firing_for are all required
For inspection-wrapping rules: only type, name, for_duration, keep_firing_for are required — query / operator / threshold are ignored if sent
notification_channels UUIDs must already exist in the project

Treat rules as code

Because every rule round-trips cleanly through this endpoint, the recommended pattern for production environments is to keep your rules in a Git repo and apply them with a short CI step. Diffing alert configuration becomes a normal pull-request review.

Next Steps

UI Walkthrough — hands-on tour of Rules, Channels, the rule-creation modal, and the fired-alert detail drawer
Architecture — where the alert engine sits in the full CodeXray pipeline
Log Monitoring — pattern extraction, which powers log-pattern alerts

What You Get Out of the Box​

Three Ways to Create a Custom Rule​

1. Wrap a Built-in Inspection​

2. Plain PromQL​

3. Log Pattern​

Anatomy of a Rule​

AI-Powered RCA on Fire​

Notification Channels​

How a firing alert notifies​

How Evaluation Works​

Managing Rules at Scale — The Bulk API​

Next Steps​