Skip to main content
clawker’s control plane runs netlogger — a userspace pipeline that drains an eBPF ringbuf populated by the cgroup/connect/sendmsg/sock_create programs and emits one OTLP log record per egress decision. netlogger starts as part of CP’s own boot sequence, independent of whether the firewall subsystem (Envoy + CoreDNS) is enabled — it degrades to a no-op when its OTLP export chain is unavailable, most commonly when no OTLP collector endpoint is configured. Every connect, sendmsg, and socket-create call from a managed agent container produces a record carrying the kernel’s verdict (allowed / denied / bypassed), the container’s attribution (agent, project, container_id), the destination 4-tuple, and the resolved domain when DNS context is available. The headline use is bypass-mode forensic coverage. The firewall’s bypass switch (clawker firewall bypass <duration> --agent <name>) intentionally short-circuits enforcement so the operator can perform supervised exploration without rule churn. Before netlogger, bypassed traffic flowed without leaving an enforcement record — the so-called “forensic black hole” of bypass mode. netlogger emits an action=bypassed record at the same decision points that would have emitted allowed/denied, so an audit trail exists for every bypass window without changing enforcement semantics.

Record Shape

Each record is an OTel log emitted on the trusted infra OTLP lane with:
  • service.name = ebpf-egress (distinct from clawkercp so retention + volume profile are independent)
  • event.name = ebpf.egress.connect / ebpf.egress.sendmsg / ebpf.egress.sock_create (per-emit-site so dashboards can filter by record kind without inspecting flag bits)
  • body = "ebpf egress"
  • severity = INFO
Attributes carried on each record (most are unconditional — empty strings and zero numbers ship verbatim; dst_ip / dst_port / dst_host are omitted when their source value is absent so operators can partition via _exists_:attributes.<key> in OS Discover): The BPF event struct is fixed-size with explicit padding (48 bytes; layout asserted at compile time). Operators filter and aggregate at dashboard/query time — the emitter never decides which fields are “interesting” for a given verdict, except for the sock_create carve-out above where BPF carries no destination.

Where Records Land

Records flow:
The index is preconfigured by the clawker-opensearch-bootstrap one-shot service every time clawker monitor up runs. The retention policy (default 7 days, throwaway-stack semantics) auto-attaches via the same ISM policy that covers the other clawker indices. Cross-index queries against clawkercp,clawker-envoy,clawker-coredns,clawker-ebpf-egress work out of the box — ingest_source is stamped on every record for filtering.

Per-Connection Bytes and Duration

Not in this stream. netlogger records the decision — the moment the kernel approved, denied, or bypassed an outbound connection. Byte counts and durations belong to the L7 proxy lifecycle, not the decision point. For action=allowed records, the matching Envoy access log carries bytes_sent, bytes_received, duration_ms. Operators pivot from a netlogger record to the corresponding Envoy record by 5-tuple at query time. For action=denied records there are no bytes to record — no traffic flowed. For action=bypassed records, only the netlogger record exists — Envoy and CoreDNS enforcement are skipped under bypass by design. Sock_ops-based per-connection byte tracking inside BPF is not on this stream’s roadmap. It would double the BPF surface area, leave UDP/connectionless flows without an analogous signal, and overlap with Envoy’s access-log emission for the cases where it matters.

Domain Resolution

dst_host is populated for every record whose destination IP came from a dnsbpf-resolved A record under a firewall-allowed zone. The translation is control-plane-driven: the control plane allocates a sticky route identity per firewall destination, the BPF dns_cache map stores {identity, expire_ts} keyed by IPv4, and netlogger reads the identity → destination table directly from the allocator on a 5-second refresh tick. dnsbpf writes the same CP-allocated identity (delivered per zone via its Corefile directive), so the two sides agree by construction — attribution is a direct read of the allocation, not a hash inversion. dst_host will be empty when:
  • The destination IP was reached without DNS resolution (direct-IP connect).
  • The IP was resolved through a path other than the managed CoreDNS (e.g., /etc/hosts entry inside the agent container).
  • A rule was removed and netlogger hasn’t yet refreshed (worst case: 5 seconds of stale records on the previously-allowed domain).
Identities are allocated — not derived — so two destinations can never share one (Cilium’s identity-allocation pattern). An identity stays bound to its destination for as long as the destination remains configured, surviving rule churn and control-plane restarts.

Reliability

netlogger is engineered to fail open with respect to the firewall — enforcement runs whether or not netlogger is healthy.
  • BPF token-bucket rate limiter keyed by cgroup_id (burst 64, refill 64 tokens/100ms ⇒ ~640 records/sec/cgroup ceiling). A misbehaving container cannot monopolize the ringbuf; throttled events are counted in ratelimit_drops, keyed by the noisy cgroup.
  • Kernel-fault drop counter (events_drops, PERCPU_ARRAY) bumps when bpf_ringbuf_reserve returns NULL on a full buffer — distinct from rate-limit drops so the operator response is different (ringbuf size vs. noisy-agent triage).
  • Userspace queue between the ringbuf reader and the processor is bounded with drop-newest semantics; the reader never blocks on the consumer. Drops are counted in clawker_netlogger_queue_dropped_total (Prom counter declared; scrape exposure is not wired).
  • Circuit breaker wraps the OTLP exporter: three consecutive Export() failures permanently trip the breaker for the rest of the CP lifetime. Records drop on the floor afterward; the BatchProcessor queue drains via the SDK’s own drop-oldest path. No background reconnect — telemetry availability is binary per-CP-lifetime by design. Operator response: restart CP after fixing the collector.
  • Preflight TLS dial runs at CP boot with a 20-second deadline against the configured OTLP endpoint. Failure degrades netlogger to a no-op for the rest of the CP lifetime and emits event=netlogger_unavailable (warn for “no endpoint configured”, error for actual failures like cert problems or unreachable collector). The firewall, AdminService, agent dispatch, and registry are unaffected — netlogger’s failure is contained.
If netlogger is degraded, the kernel records continue to land in the ringbuf and are dropped on the floor — the BPF programs themselves are unaffected, no enforcement decision is missed, and the BPF token-bucket prevents unbounded buildup of pinned ringbuf records.

Trust Lane

netlogger emits on the trusted infra lane — the same OTLP/gRPC + mTLS path the CP zerolog bridge, the Envoy access logger, and the CoreDNS otel plugin use. Identity reuse:
  • Cert: per-handshake ephemeral leaf minted by otelcerts.Service (LoadTLSConfig("netlogger")), chained through the infra intermediate CA — not the CLI root.
  • Endpoint: the collector’s otlp/infra receiver at otel-collector:4319 (not the unauthenticated host CLI receiver). The control plane, Envoy, and CoreDNS reach it over an internal telemetry network that agents never join. The receiver is not published on the host.
  • The OTLP endpoint must be https:// (or bare host:port). A plaintext http:// endpoint is rejected at boot — pushing infra telemetry over plaintext would smuggle records onto the unauthenticated host CLI receiver, defeating the trust-lane separation.
The control plane also forwards agent telemetry to the collector on this lane (its own otlp/agents receiver, ingest_source=cp_gateway). Agent processes export to a loopback OTLP/HTTP endpoint served by clawkerd inside the container; clawkerd carries each batch to the control plane over its mTLS leaf, and the control plane overwrites the project, agent, and container_id resource attributes with the identity it verified against the agent registry. When the monitoring stack is down the control plane drops those batches after three failed forwards and probes once a minute, so telemetry resumes when the stack comes up. Agent containers cannot forge service.name=ebpf-egress records onto the trusted index — they don’t hold a leaf chained through the infra intermediate, so the receiver’s TLS handshake fails the chain check. The strict-directive promise (every field on every record, no discretion) only delivers if the records actually originate from the CP — the mTLS boundary is what makes that promise enforceable.

Configuration

netlogger inherits its endpoint from the standard OTEL_EXPORTER_OTLP_ENDPOINT resolution path used by the CP zerolog bridge — when clawker monitor up is running, the CP boot sequence wires the collector’s infra receiver automatically. No netlogger-specific knobs ship; the BatchProcessor sizing, retry cap (10s vs. SDK default 1 min), and circuit-breaker threshold are CP-level constants. To point netlogger at a custom collector (for a centralized SIEM, or to bypass the local stack entirely), override OTEL_EXPORTER_OTLP_ENDPOINT in the CP container’s environment and ensure the receiver presents a server cert chained through the infra intermediate. Plaintext endpoints are rejected by design — see Trust Lane above.

Current Limitations

  • Prom counters not scraped — the clawker_netlogger_* counters are declared but not wired into a /metrics endpoint. The structured CP log surface is the operational signal for throughput, queue drops, parse errors, and OTLP export success/error. Kernel-side drop counters (events_drops, ratelimit_drops) live on the firewall/eBPF subsystem surface — they are subsystem health, not security telemetry, and intentionally don’t ride on the netlogger OTel stream.

See Also

  • Firewall — the source of every record on this stream (decision points, bypass mechanics, rule lifecycle).
  • Monitoring — the OpenSearch + OpenSearch Dashboards + Prometheus stack that hosts the clawker-ebpf-egress index.