eBPF authoring guide (engine/ebpf)

This document explains how to add a new eBPF program to the Dagger engine and wire it so it can be loaded at engine start. It is written for future LLMs and humans to follow step-by-step.

Scope and expectations

• Programs live under engine/ebpf/<name>/ with a BPF C file, Go tracer, and generated bpf2go outputs.
• Each tracer implements the Go interface in engine/ebpf/common.go:
  • Run(ctx) blocks until context cancel; read events and log.
  • Close() releases links, maps, ringbuf, etc.
• Programs are enabled via env vars DAGGER_EBPF_PROG_=y and registered in cmd/engine/main.go.
• Defaults are only enabled when extra-debug/trace logging is on.

Directory layout (follow existing patterns)

Use either filetracer/ or ovltracer/ as a template.

engine/ebpf/bpf/
  common.h                    # shared helpers + standard BPF includes
  vmlinux.h                   # CO-RE BTF header (checked in once)
engine/ebpf/<prog>/
  tracer.go                    # Go wrapper implementing ebpf.Tracer
  bpf/<prog>.bpf.c              # eBPF C source (//go:build ignore)
  <prog>_x86_bpfel.go           # generated by bpf2go
  <prog>_arm64_bpfel.go         # generated by bpf2go
  <prog>_x86_bpfel.o            # generated by bpf2go at build time (not committed)
  <prog>_arm64_bpfel.o          # generated by bpf2go at build time (not committed)

Step-by-step: adding a new tracer

1. Create package skeleton under engine/ebpf/<prog>/ with tracer.go and bpf/<prog>.bpf.c.
2. Write the BPF C program (see “BPF C guidelines” below). Prefer #include "common.h" for shared helpers.
3. Add go:generate bpf2go lines in tracer.go (see existing packages).
4. Generate bindings: run go generate in the package (or at repo root). This creates *_bpfel.go and .o files.
5. Implement Go tracer to:
   • rlimit.RemoveMemlock()
   • check /sys/kernel/btf/vmlinux exists
   • loadObjects() and handle ebpf.VerifierError
   • attach links (tracepoints/kprobes) with clear required/optional logic
   • open ringbuf reader
   • return a Tracer that can Run and Close
6. Wire into engine: add to ebpfPrograms in cmd/engine/main.go and optionally add to defaultEbpfPrograms.

Wiring into engine start

• Registration happens in cmd/engine/main.go:
  • Add {name: "<name>", new: <pkg>.New() } to ebpfPrograms.
• Env var name is derived automatically:
  • Name is uppercased and non‑alphanumeric chars become _.
  • Example: name: "ovl_inuse" → DAGGER_EBPF_PROG_OVL_INUSE.

BPF C guidelines (deep gotchas included)

• Stack limit is tiny (~512 bytes). Avoid large stack buffers:
  • Use per‑CPU arrays for temporary scratch (see ovltracer’s tmp_mount_args).
  • Keep local structs small; move large data into maps.
• CO‑RE + BTF:
  • Use #include "vmlinux.h" and BPF_CORE_READ for kernel struct reads.
  • The runtime requires /sys/kernel/btf/vmlinux (checked in Go before loading).
• Tracepoint vs kprobe:
  • Tracepoints are more stable; kprobes may fail if symbols are missing or renamed.
  • For kprobes, treat non‑essential probes as optional and log a debug message instead of failing (see ovltracer).
  • Symbols may have suffixes like .isra.0; resolve via /proc/kallsyms if needed.
• User vs kernel memory:
  • Use bpf_probe_read_user_str for user pointers (syscall args).
  • Use bpf_probe_read_kernel_str / BPF_CORE_READ for kernel memory.
• Event structs must match Go exactly:
  • Use fixed-width integer types (__u32, __u64, etc.).
  • Fixed-size arrays only; no pointers.
  • Keep field order identical between C and Go.
• Ring buffer limits:
  • bpf_ringbuf_reserve can return NULL; handle it gracefully.
  • Keep event sizes reasonable to reduce drops.
• Context maps and cleanup:
  • If you store data at entry (e.g., sys_enter_*), delete it on exit to avoid leaks.
  • If a kretprobe is optional, also guard map cleanup for the missing side.
• Loops must be bounded:
  • Use #pragma unroll for short loops; avoid dynamic loops (verifier dislikes them).
• Strings:
  • Always null‑terminate and bound copies; ringbuf events should not carry unterminated strings.

Go tracer guidelines

• Memlock: call rlimit.RemoveMemlock() before loading.
• BTF: check for /sys/kernel/btf/vmlinux and return a clear error if missing.
• Verifier errors: wrap *ebpf.VerifierError with %+v for verbose diagnostics.
• Attachment patterns:
  • Use link.Tracepoint("syscalls", "sys_enter_*", ...) for tracepoints.
  • Use link.Kprobe / link.Kretprobe for kernel functions.
  • Keep required vs optional probes explicit.
• Run loop:
  • On ctx.Done(), close the ringbuf reader to unblock Read().
  • Decode event bytes with binary.Read(..., binary.LittleEndian, &event).
• Close():
  • Close reader, then links, then objects.
  • Aggregate errors (errors.Join).

bpf2go generation

• Add //go:generate lines in tracer.go, e.g.:
  • go run github.com/cilium/ebpf/cmd/bpf2go -cc clang -cflags "-O2 -g -Wall -Werror -D__TARGET_ARCH_x86 -I../bpf" -target amd64 <name> ./bpf/<name>.bpf.c
  • go run ... -D__TARGET_ARCH_arm64 -I../bpf -target arm64 ...
• The generated .go files are checked in; the .o files are build artifacts and are not committed.

Quick checklist before shipping

• C structs and Go structs match exactly (sizes + field order).
• No large stack allocations in BPF program.
• All maps have bounded sizes; cleanup paths remove entries.
• BTF check is present and error messages are clear.
• New() returns errors for hard failures, logs for optional probes.
• Program registered in cmd/engine/main.go with correct name.
• Env var name is documented (or easy to infer from name).
• go generate run and new .go files committed (no .o files in git).

Common failure modes and fixes

• Verifier error about stack size: move large buffers into a map or per‑CPU array.
• Missing BTF: kernel lacks CONFIG_DEBUG_INFO_BTF; fail gracefully.
• Kprobe attach failures: symbol not present; make probes optional or resolve symbol suffixes.
• Ringbuf read hangs on shutdown: make sure Run closes the reader on ctx.Done().
• Garbage map entries: always delete per‑pid context maps on exit.