How Linux tracing systems fit together
Summary
Julia Evans sorts strace, perf, ftrace, kprobes and BPF by which events they observe and what overhead they cause. Linux has several tracing subsystems that emerged independently. kprobes set dynamic probe points in the kernel.
Ideas
- strace logs a process's system calls.
- perf counts hardware events and creates sampling profiles.
- ftrace traces functions and events in the kernel.
- BPF runs verified measurement programs at selected hooks.
Insights
- The best tracing tool follows from the question and the probe point needed.
- More visibility causes different run-time costs and data volumes.
- Several tools can explain the same symptom from different layers.
Facts
- uprobes allow corresponding probe points in user space programs.
Recommendations
- First define the event, the target process and the frequency you want.
- Measure the observation overhead before longer recordings in production.
References
Links to the original source and the Web Archive open in a new tab.