Go, containers and the Linux scheduler
Summary
River Phillips shows how unrecognised container CPU limits worsen Go garbage collection and application latency. The test container was given a quota of four CPUs. Adjusting GOMAXPROCS shortened the GC cycle from about 2.5 to under one millisecond.
Ideas
- Cgroup quotas allocate CPU time rather than exclusive processor cores.
- At the time the Go runtime derived parallelism from the visible host CPUs.
- Too many runtime threads use up their container quota at the same time.
- Stop-the-world phases then wait for threads throttled by the kernel.
- GOMAXPROCS can adapt runtime parallelism to the CPU time actually allowed.
Insights
- Two schedulers make decisions independently and together cause unexpected latencies.
- Resource limits only work reliably once runtime environments understand them.
- Average CPU consumption hides short-term throttling during coordinated phases.
Facts
- automaxprocs automatically determines suitable values from cgroup information.
Recommendations
- Match GOMAXPROCS to container quotas or use automatic detection.
- Also analyse latency spikes with
runtime/traceandgo tool trace.
References
Links to the original source and the Web Archive open in a new tab.