bk99.de entertain the web since 1997

Booting Linux 293,000 times

Summary

Richard W. M. Jones finds a rare Linux boot regression through automated repetition and statistical Git bisecting. Jones booted Linux 292,612 times within 21 hours.

Ideas

  • A bug with a probability of 1 in 1,000 easily gets lost in ordinary tests.
  • A parallel harness repeats boots and detects the characteristic hang.
  • Every supposedly good bisect step needs thousands of successful runs.
  • Flaky tests make binary search slow and statistically asymmetric.
  • Reproduction before and after the suspect commit confirms the cause.

Insights

  • Rare bugs require test budgets that match their probability of occurrence.
  • The absence of a bug only proves stability reliably with a growing sample.
  • Automated evaluation turns endurance into reproducible engineering.

Facts

  • The test harness used up to eight parallel threads.
  • A regression in the printk time function caused the hang in Linux 6.4.

Recommendations

  • Fully automate start, timeout, log collection and evaluation of results.
  • For rare bugs, use several repetitions per Git bisect step.

References

Read the original article

Search the Web Archive