Skip to main content

Cluster Agent Metrics

The CodeXray Cluster Agent collects cluster-scope signals that aren't visible from a single node — workload inventory, pod lifecycle events, resource metrics from the Kubernetes Metrics API, and (optionally) scraped metrics from databases you've annotated for discovery.

Where these metrics come from

Everything on this page is emitted by the Cluster Agent based on what it observes in the Kubernetes API plus what it scrapes from annotated Pods. The agent makes read-only get/list/watch calls to the cluster — nothing is modified.

Workload Inventory​

These metrics describe the state of objects in the cluster.

kube_pod_info​

  • Description: Meta information about each Pod
  • Type: Gauge
  • Labels: namespace, pod, host_ip, pod_ip, node, uid, owner_name, owner_kind

kube_pod_status_phase​

  • Description: Pod lifecycle phase (one-hot across phase labels)
  • Type: Gauge
  • Labels: namespace, pod, phase (Pending | Running | Succeeded | Failed | Unknown)

kube_pod_status_ready​

  • Description: Whether the Pod is ready to serve traffic
  • Type: Gauge
  • Labels: namespace, pod, condition

kube_pod_container_info​

  • Description: Per-container metadata
  • Type: Gauge
  • Labels: namespace, pod, container, image, image_id

kube_pod_container_status_*​

  • Description: Container state — restarts_total, ready, waiting_reason, terminated_reason
  • Type: Counter / Gauge (varies)
  • Labels: namespace, pod, container, plus reason-specific labels

Workload objects​

The Cluster Agent watches these Kubernetes object kinds and emits equivalent metrics for each:

ObjectMetric prefix
Deploymentkube_deployment_* — spec_replicas, status_replicas, status_replicas_available, status_replicas_unavailable
StatefulSetkube_statefulset_* — replicas, status_replicas, status_replicas_ready
DaemonSetkube_daemonset_* — status_number_ready, status_number_unavailable, status_desired_number_scheduled
ReplicaSetkube_replicaset_* — spec_replicas, status_replicas
Jobkube_job_* — status_succeeded, status_failed, status_active
CronJobkube_cronjob_* — status_last_schedule_time, spec_suspend

Each has namespace and the object name (deployment, statefulset, etc.) as labels, plus object-specific additional labels.

Service & Endpoint Topology​

kube_service_info​

  • Description: Service metadata
  • Type: Gauge
  • Labels: namespace, service, cluster_ip, type, load_balancer_ip

kube_endpoint_info​

  • Description: Service endpoints (the actual Pod IPs backing a Service)
  • Type: Gauge
  • Labels: namespace, endpoint, ip, port, port_name

kube_ingress_info​

  • Description: Ingress routes — host/path → service mapping
  • Type: Gauge
  • Labels: namespace, ingress, host, path, service, service_port

Node State​

kube_node_info​

  • Description: Node metadata
  • Type: Gauge
  • Labels: node, kernel_version, os_image, kubelet_version, container_runtime_version, provider_id

kube_node_status_condition​

  • Description: Node condition state — one-hot across Ready, MemoryPressure, DiskPressure, PIDPressure, NetworkUnavailable
  • Type: Gauge
  • Labels: node, condition, status (true | false | unknown)

kube_node_status_capacity / kube_node_status_allocatable​

  • Description: Node capacity and allocatable resources
  • Type: Gauge
  • Labels: node, resource (cpu | memory | pods | ephemeral-storage)

Resource Metrics (via metrics.k8s.io)​

If your cluster has the Metrics Server installed, the Cluster Agent pulls live resource usage.

kube_pod_container_resource_usage_cpu_cores​

  • Description: Live CPU usage per container, in cores
  • Type: Gauge
  • Labels: namespace, pod, container

kube_pod_container_resource_usage_memory_bytes​

  • Description: Live memory usage per container, in bytes
  • Type: Gauge
  • Labels: namespace, pod, container

kube_node_resource_usage_*​

  • Description: Node-level CPU / memory usage as reported by the Metrics API
  • Type: Gauge
  • Labels: node
Requires Metrics Server

These metrics are only available if metrics.k8s.io is registered in your cluster. Install the Kubernetes Metrics Server if you don't have it yet.

Kubernetes Events​

When COLLECT_KUBERNETES_EVENTS=true (the default), the Cluster Agent ingests native Kubernetes events as structured telemetry.

Events are not exposed as metrics — they flow to the CodeXray backend as events correlated with the workloads they reference. You'll see them in the Incidents stream and on the affected service's page. Common event reasons surfaced:

  • BackOff, CrashLoopBackOff, OOMKilled
  • FailedScheduling, Preempting, NodeNotReady
  • Unhealthy, ProbeWarning
  • ImagePullBackOff, ErrImagePull
  • FailedMount, FailedAttachVolume

Database Scraping (Annotation-Discovered)​

The Cluster Agent can additionally scrape metrics from databases running in your cluster when Pods are annotated. Enabled per-database:

PostgreSQL​

  • Annotation: codexray.io/postgres-scrape: "true"
  • Port annotation: codexray.io/postgres-scrape-port (default 5432)
  • Credentials annotations (either plain or Secret-based):
    • codexray.io/postgres-scrape-credentials-username / -password
    • or codexray.io/postgres-scrape-credentials-secret-name + -secret-username-key + -secret-password-key
  • SSL: codexray.io/postgres-scrape-param-sslmode

Emits standard Postgres exporter metrics: pg_stat_database_*, pg_stat_user_tables_*, pg_stat_bgwriter_*, pg_stat_replication_*, pg_locks_count.

MySQL​

  • Annotation: codexray.io/mysql-scrape: "true"
  • Port: codexray.io/mysql-scrape-port (default 3306)
  • Credentials follow the same pattern as Postgres
  • TLS: codexray.io/mysql-scrape-param-tls

Emits: mysql_global_status_*, mysql_global_variables_*, mysql_slave_status_*, mysql_info_schema_innodb_*.

Redis​

  • Annotation: codexray.io/redis-scrape: "true"
  • Port: codexray.io/redis-scrape-port (default 6379)
  • Credentials follow the same pattern

Emits: redis_connected_clients, redis_memory_used_bytes, redis_commands_total, redis_keyspace_*, redis_replication_*.

Profile Scrape Targets​

When Pods are annotated for profile scraping, the Cluster Agent pulls pprof data from them — not as metrics, but as profile telemetry.

  • Annotations: codexray.com/profile-scrape: "true" + codexray.com/profile-port: "<port>" (with pyroscope.io/scrape / pyroscope.io/port as fallbacks)
  • Profile types collected: CPU, Memory, Blocking, Mutex
  • See Profiling Overview for the full pipeline

Cluster Agent Health​

The agent itself exposes operational metrics on its /metrics endpoint (for Prometheus scraping):

MetricTypePurpose
codexray_cluster_agent_scrape_duration_secondsHistogramHow long each scrape cycle takes
codexray_cluster_agent_scrape_errors_totalCounterFailures when contacting the Kubernetes API or scraped targets
codexray_cluster_agent_targets_discoveredGaugeNumber of targets the agent is actively monitoring
codexray_cluster_agent_events_processed_totalCounterNumber of Kubernetes events ingested

Next Steps​