In today’s AI and machine learning ecosystem, computational efficiency is not a luxury—it’s a necessity. Yet many companies discover that their costly cloud infrastructure investments do not translate into real performance. A recurring scenario in Kubernetes with Kubeflow and Cilium perfectly illustrates how two correctly configured systems can create a silent and expensive failure. At Q2BSTUDIO, as a company specialized in custom software development, we have seen this pattern across multiple clients deploying clusters for distributed training. The problem is simple to describe: more than half of the GPUs remain idle, training never starts, but all health indicators show green. No errors in logs, no crashes, no OOMKills. Only silence. And a bill that keeps growing.
To understand it, imagine a concert hall with every seat occupied. The musicians are in place, instruments tuned, lights on. But the conductor has been taken to the wrong wing of the building, and the fire doors, doing their safety job perfectly, have sealed that area. The result is a full hall and absolute silence. That is exactly what happens in a GPU cluster when the training coordinator (the conductor) lands in one availability zone and the workers (the musicians) land in another, and Cilium’s network policies block communication between them. The failure is not in any individual component, but in the absence of a single map that shows both pod placement and locked doors simultaneously.
The root cause lies in the fact that Kubernetes scheduler, by design, is topology-agnostic. It places pods based on available resources (CPU, memory, GPUs) without considering which availability zone they land in. Kubeflow inherits that assumption. On the other hand, Cilium is topology-aware: operators use CiliumNetworkPolicy to draw zone boundaries for blast-radius isolation or to protect a dedicated GPU pool. That is good security hygiene, but Cilium cannot reschedule a pod that has already been placed. When the coordinator ends up in one zone and the workers in another, the network silently blocks the connection. No one holds the combined map of placement and network rules at the same time.
This same failure manifests in three different ways in production environments. The first is a hard block: the network policy denies the connection outright. Workers cannot reach the coordinator and training never starts. It is noticed in seconds. The second is more subtle: there is no explicit block, but gradient synchronization (NCCL AllReduce) must traverse cross-zone latency, reducing throughput by 30–60% without any visible error. This is noticed in hours. The third is economic: traffic crosses availability zones and appears on the bill as inter-AZ data transfer. It is only discovered days later, when reviewing the invoice. At Q2BSTUDIO, we have helped clients detect these cases using BI and Power BI tools to monitor GPU utilization and per-zone costs, providing a consolidated view that would otherwise go unnoticed.
The solution is surprisingly simple. There is no need to touch Cilium (the doors remain locked because the security policy was never the problem), nor to patch Kubernetes or Kubeflow. We just need to give the scheduler the missing information: keep the entire training group together, in a zone whose network path is open. In practice, this is achieved with a few lines of native Kubernetes YAML in the workload spec: nodeAffinity to pin the group to the GPU zone, topologySpreadConstraints to colocate the coordinator and workers, and a toleration so the coordinator can land on GPU-tainted nodes. Once all communicating pods share the same zone, traffic becomes intra-zone: the hard block, latency, and cross-AZ cost all disappear. One fix, three symptoms eliminated.
If you don’t want to hard-code a zone name, you can use podAffinity with a zone topology key, so workers land in the same zone as the coordinator. Same idea, but portable across clusters. In our lab tests, GPU utilization jumped from about 40% to about 85% the moment the group was colocated. This type of optimization is part of the services we offer at Q2BSTUDIO, both in the cloud AWS/Azure domain and in the integration of AI agents and cybersecurity, where understanding network topology is critical to avoid compromising performance or security.
The takeaway is clear: your container network interface (CNI) has opinions about topology that your scheduler cannot see. Topology-aware network policies are silent to the Kubernetes scheduler, and that silence is where the failure lives. The fix is Kubernetes-native: no CNI changes, no framework patches, just topology spread constraints and affinity in the workload spec. Instrumentation beforehand is key: GPU utilization and pod-zone metrics in Prometheus and Grafana expose this problem in seconds; without them it can take days. This pattern generalizes to any combination of topology-aware CNI and distributed ML framework, not just Cilium and Kubeflow.
For those who want to experiment directly, there is a reproducible lab that includes a kind cluster with GPU and CPU zones, the Cilium policy, Prometheus recording rules, a Grafana dashboard, and before/after demo scripts. It can be found in the repository https://github.com/ram2valar/kubeflow-cilium-lab and a talk recording at KubeCon + CloudNativeCon India 2026 at https://www.youtube.com/watch?v=BG9XGouyM9c . At Q2BSTUDIO, we believe preventing these silent failures is one of the most cost-effective investments in any modern cloud infrastructure.
