bk99.de entertain the web since 1997

Cloudflare hits the limits of the Linux network stack

Summary

Hackaday describes Cloudflare’s attempt to redirect TCP connections for soft unicast and why a simple local proxy ultimately proved more robust. Cloudflare uses anycast extensively for its services. TCP_REPAIR is normally used to migrate connections.

Ideas

  • Soft unicast distributes redundantly announced services across several processes.
  • Conntrack and socket assignment make it hard to rewrite packets for existing connections.
  • TCP_REPAIR can reconstruct the complete socket state.
  • A local proxy terminates TCP and forwards internally.

Insights

  • Technically elegant shortcuts can create more edge cases than measurable benefit.
  • Kernel mechanisms have assumptions that unusual routing models bring to light.
  • The maintainable solution is often deliberately simpler than the fastest possible one.

Facts

  • In the end, the team chose a local proxy.

Recommendations

  • Compare optimisations with a simple reference architecture.
  • Test connection states during switchover, loss and reordering.

References

Read the original article

Search the Web Archive