Building a Multi- Site Kubernetes Cluster with BGP Anycast

Summary

The restlessness#I had a perfectly stable single-site Kubernetes cluster in the Netherlands. It worked, it was reliable, and by any reasonable measure, it was enough. But I have learned over the years that when things become too stable, too predictable, I start looking for the next challenge; this pattern has followed me throughout my career, moving on whenever circumstances headed toward a stale that was beyond my control to influence.The initial justification was disaster recovery — what happens if the house burns down? All that work, gone. But if I am being honest with myself, DR was merely the acceptable excuse; the real motivation was the desire to understand how the internet actually works at the routing level, to own that knowledge rather than abstracting it away behind managed services.Hence, the project scope expanded: not just a second site, but my own Autonomous System Number, my own IP space, actual BGP peering with upstream providers and an internet exchange. The kind of infrastructure I had only ever seen from the outside during my years in enterprise environments.The architecture#AS214304 is my own Autonomous System Number, and 2a0c:9a40:8e20::/48 is the IPv6 prefix registered via RIPE NCC (sponsored by iFog GmbH, who have been excellent for hobbyist allocations).The infrastructure spans 20 nodes across four European countries:Netherlands & Greece — on-premises clusters behind Cisco ASAsNorway & Switzerland — cloud edge nodes running strongSwanConnecting it all: 39 FRR BGP sessions, 4 Cilium BGP sessions, and 12 IPsec tunnels forming a full mesh between sites. The encrypted tunnel topology links Cisco ASA on the on-prem side with strongSwan on the cloud edge nodes.Traffic flow: IPv6 anycast reaches whichever edge node is closest, then traverses the IPsec mesh to the Kubernetes clusters running Cilium with native BGP control plane for dynamic pod routing. NAT64 at the edge enables IPv6 ingress to the IPv4 core, while dual-stack anycast with SNI-based r...