Skip to main content

eBPF-based Tracing

Fully instrumenting every service with OpenTelemetry is the gold standard — but it's not always immediately possible. Legacy services, third-party components, vendor binaries, and services you simply haven't gotten to yet leave blind spots.

The CodeXray Node Agent fills those blind spots. Using eBPF, it captures outbound network traffic from your containers at the kernel layer and exports it as OpenTelemetry spans — with zero code changes, no SDK, no restart required.

When to use it
  • You need traces today for a service you can't (or won't) instrument
  • You want immediate visibility into database calls, especially during an incident
  • You're rolling out OTel gradually and need coverage for services still in the backlog
  • You run legacy binaries, third-party services, or closed-source components

How It Works​

The CodeXray Node Agent runs on every host (DaemonSet on Kubernetes, systemd service on VMs). Using eBPF, it attaches to kernel hooks that fire on every TCP connect(), send(), and recv() — so it sees every outbound request a container makes, without needing hooks in the application itself.

For each connection, the agent:

  1. Identifies the protocol — HTTP, gRPC, Postgres wire protocol, MySQL, MongoDB, Redis, Memcached, RabbitMQ, Kafka
  2. Extracts the relevant fields — HTTP method + path + status, SQL statement fingerprint, Redis command, etc.
  3. Measures latency — time between request start and response
  4. Emits an OTel span — with the captured attributes, service name from the container, and standard OTel span conventions

These spans arrive in CodeXray looking indistinguishable from SDK-generated spans — they appear in the Traces tab, populate the service map, and show up in Database Insights.

What eBPF Tracing Captures​

The Node Agent's eBPF tracer auto-decodes these protocols out of the box:

ProtocolCaptured signals
HTTP / HTTPSMethod, path, status code, latency
gRPCService, method, status, latency
PostgreSQLQuery fingerprint, latency, error status
MySQLQuery fingerprint, latency, error status
MongoDBCommand, collection, latency
RedisCommand (GET, SET, HGET, etc.), key pattern, latency
MemcachedCommand, key pattern, latency
RabbitMQExchange, routing key, latency
KafkaTopic, partition, latency

Each becomes one or more OTel spans in the recipient's trace view.

What It Does Not Capture​

eBPF tracing is observational — it sees what goes over the wire, not what happens inside your process. This means it can't see:

  • Internal function calls — only network boundaries produce spans
  • Custom business attributes — things only your app's code knows (user ID, feature flag, tenant tier)
  • Logic that stays in-process — CPU-bound computation, synchronous caches, in-memory queues
  • The caller side of the trace — eBPF sees outbound requests from a container, not the inbound handler's internal work
  • Encrypted payloads without TLS termination — the agent sees protocol metadata but not decrypted application-layer content when TLS is end-to-end in-process

For those, you'll want the OpenTelemetry SDK in the service's language. The two are additive — eBPF gives you breadth, SDK gives you depth.

Best of both worlds

Most production CodeXray deployments run both. eBPF gives you immediate coverage across the entire fleet; SDK instrumentation layered on top adds rich detail for services where it matters most.

Enabling eBPF Tracing​

eBPF tracing is on by default when the CodeXray Node Agent is installed. No additional configuration is required.

To install the Node Agent (if you haven't already):

  • Kubernetes — one manifest creates the DaemonSet, ClusterRole, and namespace. See the Node Agent installation guide.
  • Linux VMs — one install.sh script registers the systemd service.
  • Docker — one docker run command with host networking.

The agent needs Linux kernel 4.16 or newer and elevated capabilities (CAP_BPF, CAP_SYS_ADMIN, CAP_NET_ADMIN). The DaemonSet manifest handles permissions on Kubernetes automatically.

Disabling eBPF tracing​

If you only want metrics and logs from the Node Agent (not traces), set:

DISABLE_L7_TRACING=true

on the agent. Everything else (TCP maps, logs, delay accounting, OOM events, profiling) continues to work.

Sampling​

At high traffic, eBPF traces can produce a lot of spans. Tune the sampling rate with:

TRACES_SAMPLING=0.1   # keep 10% of traces

Values are in the 0.0–1.0 range. The default is 1.0 (keep everything).

Verifying It's Working​

After the Node Agent is running on at least one host:

  1. Open any service in CodeXray → switch to the Traces tab
  2. Within a minute or two, traces should appear in the scatter chart and the trace list — even for services that have no OTel SDK integration
  3. Open one trace — the spans carry attributes like http.method, http.url, db.statement, db.system, etc., all populated by the eBPF tracer

A good quick test is a service that only makes database calls — the Traces tab will fill with DB spans without you writing a single line of tracing code.

When to Upgrade to SDK Instrumentation​

eBPF tracing is a strong foundation. Consider layering OpenTelemetry SDK on top of it when:

  • You need custom attributes specific to your business (user tier, request region, feature flag state)
  • You want to trace in-process logic (an expensive computation, a cache miss path)
  • You want trace context propagation across hops your team owns — so trace IDs match end-to-end
  • The service is business-critical and deserves the richest visibility

The upgrade is additive, not destructive. eBPF spans and SDK spans coexist in the same trace view.

Next Steps​