Integrating the Overlays of Apstra and CN2 to Connect K8s with Bare Metal and SRIOV

In a previous blog, we covered how Juniper CN2 for Kubernetes improves networking at waypoints with seamless, open-standards-based routing north-south—in and out—of clusters. Now, with the release of CN2 version 23.1, Juniper is launching the integration of CN2 and Apstra.

Why are our data center and telco cloud customers so enthusiastic about the CN2 and Apstra integration? Kubernetes overlay networking and data center switching fabrics are traditionally operated and managed as independent layers of the infrastructure stack. Connecting virtual networks and workloads across and between these layers requires a mash-up of disparate tools and operations teams who often need to manually stitch these networks and workloads together taking the ‘agility’ out of the cloud.

Now integrated, CN2 and Apstra together automate, secure and simplify networking east-west between Kubernetes Single Root I/O Virtualization (SRIOV)-based workloads, normal Kubernetes workloads and any other workload in the physical data center fabric next to the cluster, including bare-metal servers, Virtual Machines (VMs) and even containers outside the cluster. Connecting virtual networks and workloads across these layers is now Kubernetes-automated, dynamic, and hands-free for the netops team that manages Apstra and the physical fabric.

Dovetailing the Overlays of Kubernetes and Apstra-managed Fabrics

Apstra is well known for its intent-based, multi-vendor management of data center switching fabrics. The most commonly deployed templates set up a cloud-scale EVPN-VXLAN fabric. Although we typically think of the switching fabric as the “underlay” in comparison to up-stack CN2, the fabric VXLANs are leaf- or edge-routed overlays just like CN2’s overlays based in the server nodes’ vRouter. These overlays can come together for east-west routing with EVPN as the control plane and with VXLAN as a data-plane encapsulation. Now, we have automated this dovetailing of overlays.

New capabilities are optionally installed into Kubernetes along with CN2 to connect to Apstra, using its API. With these latest features, CN2 can create physical fabric-based virtual networks via Apstra that extend into the CN2 domain, and it can bind server-facing leaf ports into such virtual networks. It can also reference existing virtual networks in Apstra to be extended into CN2.

The overlay virtual networks of CN2 and Apstra could always come together through manual configuration of DC gateway routers that are typically used for north-south overlay termination and origination, but with this new integration, we have automated it through the fabric switches too, improving traffic flow and eliminating manual toil, which is especially useful when such plumbing can be rather complex and dynamic for SRIOV-based workloads.

Use Cases for Enterprise and Telco Cloud

A simple use case that applies inside just about any environment is the interconnection of legacy bare-metal, VM or physical networking appliances into the modern Kubernetes world. With CN2 powering the overlay Kubernetes networking, it can now extend the reach of a single subnet (intra-virtual network style) or route across subnets (inter-virtual network style) into one or many existing Apstra virtual networks, uniting networking from Kubernetes containers to legacy endpoints. In the case of extending a subnet, CN2’s IP address management can be set up to allocate from a range that doesn’t overlap with the legacy workloads.

Where this automated connectivity ends up saving a lot of time is in the networking of Kubernetes SRIOV-based workloads. Data Plane Development Kit (DPDK) high-performance packet processing is already well supported by the CN2 vRouter, but SRIOV can take performance to essentially line rate, which may or may not be possible with DPDK, depending on the resources allocated to packet processing. In any case, SRIOV is more resource efficient because the server compute is not doing any packet processing. The downside is the server compute can’t add any value. In CN2, the server-based packet forwarding is done by the vRouter component, but SRIOV-based workloads have networking that bypasses the vRouter and goes straight to the network interface card (NIC).

To run alongside CN2, there is a standard SRIOV CNI plug-in for Kubernetes, but its main job is merely to set up workloads to bind to the correct NIC and correct port of the node to which the workload is pinned. More importantly, SRIOV networking is primitive, whereas, for bare-metal servers, the physical switching fabric needs to provide the VLAN/VXLAN isolation and interconnectivity. With CN2’s new add-on, it watches the Kubernetes API for “NetworkAttachmentDefinition” objects and SRIOV-bound pods whereby it completely automates all the fabric networking through Apstra. CN2 topologies, with hub-and-spoke or mesh building blocks, can now stitch together Apstra and CN2 virtual networks.

In enterprise and high-performance cloud platforms, SRIOV is used for its virtues of low latency and high throughput for machine learning applications that use Remote Device Memory Access (RDMA) technologies like Converged Ethernet (RoCE) over the network. In Telco Cloud, many 5G waypoint components also need (or are only qualified with) SRIOV connectivity. In both cases, SRIOV workloads exist with, if not also interact with, regular workloads or at the very least the Kubernetes controllers. The CN2 model of being able to mix different data planes to best treat each workload’s needs is advantageous and unique with this new automation. Furthermore, across the SRIOV workloads, it’s common that they are not all interacting in a full mesh, so the ability to use several fabric virtual networks to partition the workloads is also more secure, especially for 5G workloads with single pods having multiple interfaces that need to exist in different networks.

How it Works

Without the automation from this new CN2 capability, it’s extremely impractical to use SRIOV. The complexity besetting operators is already high just to set up SRIOV node profiles and pod pinning, but some manage to struggle through networking as well with whatever fabric they have. Of course, we believe in the value that the full CN2 packet processing provides, but when SRIOV is needed, now it can be managed with much less pain, more reliability and as fast as you can spin up workloads. CN2 and Apstra shoulder all the work, and it’s nicely observed through Kubernetes tooling, CN2’s Graphical User Interface (GUI) and Apstra’s GUI.

This miniseries playlist of short videos helps to explain and demonstrate the new features we’re launching in our release. Our lead engineer for the integration has also posted more details about the new Juniper Elevate community for CN2.

Judge For Yourself

CN2’s free trial is a good place to start playing with CN2 and your choice of Kubernetes, and Juniper also has cloud labs to try Apstra. However, to see this new CN2-Apstra integration in action, you’ll need a real fabric setup and servers with SRIOV-capable NICs. If you’re interested in seeing more, visit us at KubeCon in Amsterdam. If after reading and watching, you’re ready to jump in right away, ask your local Juniper account team for an on-site PoC, where we can help get you set up with this winning combination.