Booting Linux 293,000 times
Summary
Richard W. M. Jones finds a rare Linux boot regression through automated repetition and statistical Git bisecting. Jones booted Linux 292,612 times within 21 hours.
Ideas
- A bug with a probability of 1 in 1,000 easily gets lost in ordinary tests.
- A parallel harness repeats boots and detects the characteristic hang.
- Every supposedly good bisect step needs thousands of successful runs.
- Flaky tests make binary search slow and statistically asymmetric.
- Reproduction before and after the suspect commit confirms the cause.
Insights
- Rare bugs require test budgets that match their probability of occurrence.
- The absence of a bug only proves stability reliably with a growing sample.
- Automated evaluation turns endurance into reproducible engineering.
Facts
- The test harness used up to eight parallel threads.
- A regression in the printk time function caused the hang in Linux 6.4.
Recommendations
- Fully automate start, timeout, log collection and evaluation of results.
- For rare bugs, use several repetitions per Git bisect step.
References
Links to the original source and the Web Archive open in a new tab.