Cloudflare hits the limits of the Linux network stack
Summary
Hackaday describes Cloudflare’s attempt to redirect TCP connections for soft unicast and why a simple local proxy ultimately proved more robust. Cloudflare uses anycast extensively for its services. TCP_REPAIR is normally used to migrate connections.
Ideas
- Soft unicast distributes redundantly announced services across several processes.
- Conntrack and socket assignment make it hard to rewrite packets for existing connections.
- TCP_REPAIR can reconstruct the complete socket state.
- A local proxy terminates TCP and forwards internally.
Insights
- Technically elegant shortcuts can create more edge cases than measurable benefit.
- Kernel mechanisms have assumptions that unusual routing models bring to light.
- The maintainable solution is often deliberately simpler than the fastest possible one.
Facts
- In the end, the team chose a local proxy.
Recommendations
- Compare optimisations with a simple reference architecture.
- Test connection states during switchover, loss and reordering.
References
Links to the original source and the Web Archive open in a new tab.