A 20-year-old Linux kernel bug
Summary
Robert O’Callahan describes how rr exposed a decades-old kernel edge case and how reproducible execution made the root cause analysis possible. The bug had been in the Linux kernel for about twenty years. rr records program execution and replays it deterministically.
Ideas
- Deterministic replay turns rare bugs into repeatable experiments.
- The bug stayed hidden for a long time because of an unusual sequence of events.
- Kernel history explains when and why problematic behaviour arose.
- A small reproduction case separates application, debugger and kernel.
Insights
- Debugging tools unintentionally test the platform they are built on.
- Rare timing bugs become normal logic bugs through controlled replay.
- Long-lived code can contain undiscovered bugs even though it runs every day.
Facts
- The investigation led to a kernel fix.
Recommendations
- Keep minimal reproduction cases together with version data.
- Only search version histories after narrowing down the faulty behaviour.
References
Links to the original source and the Web Archive open in a new tab.