Skip to main content

Node Agent Metrics

This page describes metrics gathered by the CodeXray Node Agent.

Each container metric has the container_id label. This is a compound identifier whose format varies between container types — e.g., /docker/upbeat_borg, k8s/namespace-1/pod-2/container-3, or /system.slice/nginx.service.

CPU​

container_resources_cpu_limit_cores​

  • Description: CPU limit of the container
  • Type: Gauge
  • Source: CPU cgroup, the cpu.cfs_quota_us and cpu.cfs_period_us files

container_resources_cpu_usage_seconds_total​

  • Description: Total CPU time consumed by the container
  • Type: Counter
  • Source: CPU accounting cgroup, the cpuacct.usage file

container_resources_cpu_throttled_seconds_total​

  • Description: Total time duration the container has been throttled for
  • Type: Counter
  • Source: CPU cgroup, the cpu.stat file

container_resources_cpu_delay_seconds_total​

  • Description: Total time duration the container has been waiting for a CPU (while being runnable)
  • Type: Counter
  • Source: Delay accounting

container_resources_cpu_pressure_waiting_seconds_total​

  • Description: Total time in seconds that the container was delayed due to CPU pressure
  • Type: Counter
  • Source: PSI (Pressure Stall Information)
  • Labels: kind

Memory​

container_resources_memory_limit_bytes​

  • Description: Memory limit of the container
  • Type: Gauge
  • Source: Memory cgroup, file memory.limit_in_bytes

container_resources_memory_rss_bytes​

  • Description: Amount of physical memory used by the container (doesn't include page cache)
  • Type: Gauge
  • Source: Memory cgroup, file memory.stats

container_resources_memory_cache_bytes​

  • Description: Amount of page cache memory allocated by the container
  • Type: Gauge
  • Source: Memory cgroup, file memory.stats

container_oom_kills_total​

  • Description: Total number of times the container has been terminated by the OOM killer
  • Type: Counter
  • Source: eBPF: tracepoint/oom/mark_victim

container_resources_memory_pressure_waiting_seconds_total​

  • Description: Total time in seconds that the container was delayed due to memory pressure
  • Type: Counter
  • Source: PSI (Pressure Stall Information)
  • Labels: kind

Disk​

container_resources_disk_delay_seconds_total​

  • Description: Total time duration the container has been waiting for I/Os to complete
  • Type: Counter
  • Source: Delay accounting

container_resources_io_pressure_waiting_seconds_total​

  • Description: Total time in seconds that the container was delayed due to I/O pressure
  • Type: Counter
  • Source: PSI (Pressure Stall Information)
  • Labels: kind

container_resources_disk_size_bytes​

  • Description: Total capacity of the volume
  • Type: Gauge
  • Source: statfs()
  • Labels:
    • mount_point — path in the mount namespace of the container
    • device — device name, e.g., vda, nvme1n1
    • volume — Persistent Volume (Kubernetes only)

container_resources_disk_used_bytes​

  • Description: Used capacity of the volume
  • Type: Gauge
  • Source: statfs()
  • Labels: mount_point, device, volume

container_resources_disk_(reads|writes)_total​

  • Description: Total number of reads or writes completed successfully by the container
  • Type: Counter
  • Source: Blkio cgroup, the blkio.throttle.io_serviced file
  • Labels: mount_point, device, volume

container_resources_disk_(read|written)_bytes_total​

  • Description: Total number of bytes read from the disk or written to the disk by the container
  • Type: Counter
  • Source: Blkio cgroup, the blkio.throttle.io_service_bytes file
  • Labels: mount_point, device, volume

Network​

container_net_tcp_listen_info​

  • Description: A TCP listen address of the container
  • Type: Gauge
  • Source: eBPF: tracepoint/sock/inet_sock_set_state, /proc/<pid>/net/tcp, /proc/<pid>/net/tcp6
  • Labels: listen_addr (ip:port), proxy

container_net_tcp_successful_connects_total​

  • Description: Total number of successful TCP connection attempts
  • Type: Counter
  • Source: eBPF: tracepoint/sock/inet_sock_set_state
  • Labels: destination, actual_destination
About destination vs actual_destination

The IP and port of the connection's destination. For example, a container might be establishing a connection to port 80 of a Kubernetes Service IP (e.g., 10.96.1.1). This destination address may be translated by iptables to the actual Pod IP (e.g., 10.40.1.5). In this case, the actual_destination would be 10.40.1.5:80.

container_net_tcp_retransmits_total​

  • Description: Total number of retransmitted TCP segments. Collected only for outbound TCP connections.
  • Type: Counter
  • Source: eBPF: tracepoint/tcp/tcp_retransmit_skb
  • Labels: destination, actual_destination

container_net_tcp_failed_connects_total​

  • Description: Total number of failed TCP connects to a particular endpoint. The agent takes into account only TCP failures, so this metric doesn't reflect DNS errors.
  • Type: Counter
  • Source: eBPF: tracepoint/sock/inet_sock_set_state
  • Labels: destination

container_net_tcp_active_connections​

  • Description: Number of active outbound connections between the container and a particular endpoint
  • Type: Gauge
  • Source: eBPF: tracepoint/sock/inet_sock_set_state
  • Labels: destination, actual_destination

container_net_tcp_connection_time_seconds_total​

  • Description: Time spent on TCP connections
  • Type: Counter
  • Source: eBPF: tracepoint/sock/inet_sock_set_state
  • Labels: destination, actual_destination

container_net_latency_seconds​

  • Description: Round-trip time between the container and a remote IP
  • Type: Gauge
  • Source: ICMP round-trip time to IPs the container is currently talking to
  • Labels: destination_ip

container_net_tcp_bytes_(sent|received)_total​

  • Description: Total number of bytes sent/received from the peer
  • Type: Counter
  • Source: eBPF
  • Labels: destination, actual_destination

Application Layer Protocol Metrics​

Each protocol below is auto-detected by the eBPF tracer; no instrumentation required.

HTTP​

  • container_http_requests_total — Counter — destination, actual_destination, status
  • container_http_requests_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

PostgreSQL​

  • container_postgres_queries_total — Counter — destination, actual_destination, status
  • container_postgres_queries_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

Redis​

  • container_redis_queries_total — Counter — destination, actual_destination, status
  • container_redis_queries_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

Memcached​

  • container_memcached_queries_total — Counter — destination, actual_destination, status
  • container_memcached_queries_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

MySQL​

  • container_mysql_queries_total — Counter — destination, actual_destination, status
  • container_mysql_queries_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

MongoDB​

  • container_mongo_queries_total — Counter — destination, actual_destination, status
  • container_mongo_queries_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

Kafka​

  • container_kafka_requests_total — Counter — destination, actual_destination, status
  • container_kafka_requests_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

Cassandra​

  • container_cassandra_queries_total — Counter — destination, actual_destination, status
  • container_cassandra_queries_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

RabbitMQ​

  • container_rabbitmq_messages_total — Counter — destination, actual_destination, status, method

NATS​

  • container_nats_messages_total — Counter — destination, actual_destination, method

DUBBO​

  • container_dubbo_requests_total — Counter — destination, actual_destination, status
  • container_dubbo_requests_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

DNS​

  • container_dns_requests_total — Counter — domain, request_type, status
  • container_dns_requests_duration_seconds_total — Counter (histogram) — le

ClickHouse​

  • container_clickhouse_queries_total — Counter — destination, actual_destination, status
  • container_clickhouse_queries_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

ZooKeeper​

  • container_zookeeper_requests_total — Counter — destination, actual_destination, status
  • container_zookeeper_requests_duration_seconds_total — Counter (histogram) — destination, actual_destination, le

ip_to_fqdn​

  • Description: Mapping IP addresses to FQDNs based on DNS requests initiated by containers
  • Type: Gauge
  • Source: eBPF DNS request tracking
  • Labels: ip, fqdn

GPU​

container_resources_gpu_usage_percent​

  • Description: Percent of GPU compute resources used by the container
  • Type: Gauge
  • Source: NVIDIA Management Library (NVML)
  • Labels: gpu_uuid

container_resources_gpu_memory_usage_percent​

  • Description: Percent of GPU memory used by the container
  • Type: Gauge
  • Source: NVIDIA Management Library (NVML)
  • Labels: gpu_uuid

JVM​

Each JVM metric has the jvm label which refers to the main class or path to the .jar file.

MetricTypeDescription
container_jvm_infoGaugeMeta information about the JVM — labels jvm, java_version
container_jvm_heap_size_bytesGaugeTotal heap size in bytes
container_jvm_heap_used_bytesGaugeUsed heap size in bytes
container_jvm_heap_max_size_bytesGaugeMaximum heap size in bytes (-Xmx)
container_jvm_gc_time_secondsCounterTime spent in the given JVM garbage collector — label gc
container_jvm_safepoint_time_secondsCounterTime stopped for safepoint operations
container_jvm_safepoint_sync_time_secondsCounterTime spent getting to safepoints
container_jvm_profiling_statusGaugeWhether async-profiler is enabled (1/0)
container_jvm_alloc_bytes_totalCounterTotal bytes allocated — requires --enable-java-async-profiler
container_jvm_alloc_objects_totalCounterTotal objects allocated — requires --enable-java-async-profiler
container_jvm_lock_contentions_totalCounterMonitor enter events — requires --enable-java-async-profiler
container_jvm_lock_time_seconds_totalCounterTime waiting for locks — requires --enable-java-async-profiler

Source for heap, GC, safepoint metrics: hsperfdata

Go Runtime​

container_go_alloc_bytes_total​

  • Description: Total bytes allocated by a Go application
  • Type: Counter
  • Source: Go runtime memory profiling data (runtime.mbuckets)

container_go_alloc_objects_total​

  • Description: Total objects allocated by a Go application
  • Type: Counter
  • Source: Go runtime memory profiling data (runtime.mbuckets)

Node.js Runtime​

container_nodejs_event_loop_blocked_time_seconds_total​

  • Description: Total time the Node.js event loop spent blocked
  • Type: Counter
  • Source: eBPF uprobes

Python Runtime​

container_python_thread_lock_wait_time_seconds​

  • Description: Time spent waiting acquiring GIL in seconds
  • Type: Counter
  • Source: eBPF uprobes

.NET Runtime​

Each .NET runtime metric has the application label, allowing distinction between multiple applications within the same container.

MetricTypeDescription
container_dotnet_infoGaugeMeta information about the CLR — labels application, runtime_version
container_dotnet_memory_allocated_bytes_totalCounterBytes allocated
container_dotnet_exceptions_totalCounterExceptions that have occurred
container_dotnet_memory_heap_size_bytesGaugeHeap size — label generation
container_dotnet_gc_count_totalCounterGC occurrences — label generation
container_dotnet_heap_fragmentation_percentGaugeHeap fragmentation
container_dotnet_monitor_lock_contentions_totalGaugeMonitor lock contentions
container_dotnet_thread_pool_completed_items_totalCounterThreadPool work items completed
container_dotnet_thread_pool_queue_lengthGaugeThreadPool queue length
container_dotnet_thread_pool_sizeGaugeCurrent thread pool size

Source for all .NET metrics: .NET diagnostic port

Logs​

container_log_messages_total​

  • Description: The number of messages grouped by automatically extracted repeated patterns
  • Type: Counter
  • Source: The container's log. Supported logging methods:
    • stdout/stderr — captured by Dockerd (JSON file driver) or Containerd (CRI)
    • Journald
    • /var/log/*
  • Labels:
    • source — journald, stdout/stderr, or path to the file in the /var/log directory
    • level — unknown | debug | info | warning | error | critical
    • pattern_hash — the ID of the automatically extracted repeated pattern
    • sample — a sample message of the group

Other Container Metrics​

container_info​

  • Description: Meta information about the container
  • Type: Gauge
  • Source: dockerd, containerd
  • Labels: image

container_restarts_total​

  • Description: Number of times the container has been restarted
  • Type: Counter
  • Source: eBPF: tracepoint/task/task_newtask, tracepoint/sched/sched_process_exit

container_application_type​

  • Description: Type of application running in the container (e.g., memcached, postgres, mysql)
  • Type: Gauge
  • Source: /proc/<pid>/cmdline of processes running within the container
  • Labels: application_type

Node Metrics​

CPU​

  • node_resources_cpu_usage_seconds_total — Counter — Amount of CPU time spent in each mode (user | nice | system | idle | iowait | irq | softirq | steal). Source: /proc/stat
  • node_resources_cpu_logical_cores — Counter — Number of logical CPU cores. Source: /proc/stat

Memory​

  • node_resources_memory_total_bytes — Gauge — Total amount of physical memory
  • node_resources_memory_free_bytes — Gauge — Amount of unassigned memory
  • node_resources_memory_available_bytes — Gauge — An estimate of how much memory is available for allocations, without swapping. Roughly the sum of free memory and reclaimable page cache.
  • node_resources_memory_cached_bytes — Gauge — Amount of memory used as page cache

Source for all memory metrics: /proc/meminfo

Disk I/O​

  • node_resources_disk_(reads|writes)_total — Counter — device label
  • node_resources_disk_(read|written)_bytes_total — Counter — device label
  • node_resources_disk_(read|write)_time_seconds_total — Counter — includes queue wait; divide by IOPS for avg latency
  • node_resources_disk_io_time_seconds_total — Counter — service time only, not queue wait. 100% utilization if derivative equals the sample window.

Source: /proc/diskstats

Reference IOPS by storage type
TypeMax IOPS
Amazon EBS sc1250
Amazon EBS st1500
Amazon EBS gp2/gp316,000
Amazon EBS io1/io264,000
Amazon EBS io2 Block Express256,000
HDD200
SATA SSD100,000
NVMe SSD10,000,000

Reference throughput:

TypeMax throughput
Amazon EBS sc1250 MB/s
Amazon EBS st1500 MB/s
Amazon EBS gp2250 MB/s
Amazon EBS gp31,000 MB/s
Amazon EBS io1/io21,000 MB/s
Amazon EBS io2 Block Express4,000 MB/s
SATA600 MB/s
SAS1,200 MB/s
NVMe4,000 MB/s

Reference average I/O latency:

TypeAvg latency
Amazon EBS gp2/gp3/io1/io2single-digit ms
Amazon EBS io2 Block Expresssub-ms
HDD2–4 ms
NVMe SSD0.1–0.3 ms

Network​

  • node_net_received_(bytes|packets)_total — Counter — interface
  • node_net_transmitted_(bytes|packets)_total — Counter — interface
  • node_net_interface_up — Gauge — interface (0 = down, 1 = up)
  • node_net_interface_ip — Gauge — interface, ip

GPU​

  • node_gpu_info — Gauge — gpu_uuid, name
  • node_resources_gpu_memory_total_bytes — Gauge — gpu_uuid
  • node_resources_gpu_memory_used_bytes — Gauge — gpu_uuid
  • node_resources_gpu_memory_utilization_percent_avg / _peak — Gauge — gpu_uuid
  • node_resources_gpu_utilization_percent_avg / _peak — Gauge — gpu_uuid
  • node_resources_gpu_temperature_celsius — Gauge — gpu_uuid
  • node_resources_gpu_power_usage_watts — Gauge — gpu_uuid

Node metadata​

  • node_uptime_seconds — Gauge — Uptime of the node
  • node_info — Gauge — hostname, kernel_version, agent_version
  • node_cloud_info — Gauge — Cloud-provider metadata; auto-detected for AWS, GCP, Azure, Hetzner, Scaleway, DigitalOcean, Alibaba (can be set manually via --provider, --region, --availability-zone flags)
    • Labels: provider, account_id, instance_id, instance_type, instance_life_cycle, region, availability_zone, availability_zone_id (AWS only), local_ipv4, public_ipv4

Next Steps​