The 400G era is well underway and Juniper Networks is leading the charge. Customers worldwide have deployed our switch and router platforms that support 400G Ethernet. Part of our ongoing work includes delivering 400G optics modules to customers that meet industry standards for performance and interoperability. These standards are developed by the networking industry as a whole and Juniper is playing an influential role. Partnering on optics solutions with Juniper infrastructure provides pre-integration to ensure high quality that ultimately saves time and smoothes the deployment and testing process for our customers. Pluggable optics is a complex world – at Juniper, we aim to eliminate as much of this complexity as possible.
The New Alphabet of the Optics World
Engineering teams at Juniper have developed a broad set of 400G pluggable optics that support an extensive range of use cases for customers, including 500m and 2km single-mode fiber intra-data center interconnects. Juniper’s current 400G optics are based on PAM4 modulation technology that has been standardized in the IEEE 802.3 with some additional optical specifications provided by the 100G Lambda MSA. This was a major transition within the optics technology industry roadmap where past generations of 100G, 40G, 10G and lower-rate optics were based exclusively on NRZ on-off keying (modulation). The transition to PAM4 required not only new laser modulators, digital signal encoding/decoding processors (DSPs) and receiver device designs for optical transceivers, but it also required a completely new set of performance parameters, test methodologies and new lab equipment for test and measurement.
During this time, Juniper worked closely with our pluggable optics vendors to provide guidance on technology/product roadmap alignment, functional specifications, performance requirements and early prototype evaluations as they developed their 400G optics technologies and products for volume manufacturing. We’re now seeing the fruits of this labor stemming from the IEEE Std. 802.3bs-2017, with the industry offering a wide array of 400G optics. Table 1 below provides a list of standards-based 400G optics known to the author at this time at various stages of maturity.
Some of the 400G pluggable optics support more than one rate of Ethernet. Of course, there is support for 400G Ethernet but also, in particular, 4 times 100G Ethernet. In some cases, even 2 times 200G Ethernet can be supported in a common module. These breakout ports of lower speeds are fully independent and can run on separate time domains. This required a new generation of single laser wavelength 100G optics based on the PAM4 technology. These “single lambda” optics have resulted in a significantly lower cost for 100G interconnects, as the number of lasers and modulators are reduced by 4x compared with the legacy 100G optics that use 4x25Gb/s NRZ architecture. These PAM4 optics, QSFP-100G-DR/FR1/LR1, are now being developed and readied for qualification testing.
The IEEE P802.3db task force is developing a standard for 100GBASE-SR1 over multimode fiber that will fill in the shortest reach with a 100G lambda optic. Table 2 below provides a summary of the 100G lambda optics for 100G Ethernet. The DSP deployed in these modules implements the conversion between single wavelength 100Gbps PAM4 optical modulation to 4x25Gb/s NRZ electrical modulation across the module-to-host interface.
The QDD-400G-FR4, -LR4-6, -LR4-10 and -LR8 optics listed in Table 1 support only 400G Ethernet where the 4x 100G-λ (FR4, LR4) or 8x50G-λ (LR8) wavelengths are optically multiplexed/de-multiplexed onto duplex single-mode fiber. The QDD-400G-DR4 supports both 400GBASE-DR4 onto parallel SMF with 8 fibers (4 Tx plus 4 Rx) via the MPO-12 receptacle connector and breakout to 4 x 100GBASE-DR, which can interconnect using duplex fiber links to QSFP-100G-DR. The QDD-4x100G-FR1 module, also known as 400G-4xFR1 or 400G-DR4+, supports 4 times 100G Ethernet breakout, which can interconnect to QSFP-100G-FR1 and in 400G Ethernet mode can interoperate with QDD-400G-DR4 at 500 meters. Similarly, the QDD-4x100G-LR1 module, also known as 400G-4xLR1 or 400G-DR4++, supports 4 times 100G Ethernet breakout, which can interconnect to QSFP-100G-LR1 and, in 400G Ethernet mode, can also interoperate with QDD-400G-DR4 at 500 meters. The breakout ports also interoperate at the shortest mutual reach, e.g., the 4x100G-DR breakout ports interoperate with QSFP-100G-FR1 or QSFP-100G-LR1 up to 500m. However, the 100G Ethernet breakout ports are not backward compatible with the legacy 100G Ethernet using 4x25G NRZ optics, e.g., QSFP-100G-LR does not interoperate with QSFP-100G-LR4.
QSFP Double D (QSFP-DD)
PAM4 technology was not the only innovation introduced in the 400G optics era. A new mechanical form factor needed to be developed to support 400G using an 8x50G electrical interface between the pluggable optics module and host system. This new form factor, called QSFP-DD, DD meaning “double density,” was developed by the QSFP-DD MSA industry consortium, of which Juniper is a founding member. QSFP-DD is a significant departure from the 4-lane electrical interface modules of the past, such as QSFP+ and QSFP28. It also requires a larger power supply and improved thermal management capabilities for the commensurate higher dissipated power, stemming from the power requirements of the PAM4 DSP, which amount to roughly a 3x increase compared to QSFP28 used today for 100G pluggable optics.
Can’t Miss with CMIS
Other innovations introduced by 400G optics technology are embodied in the Common Management Interface Specification (CMIS). CMIS was developed by the Quad Small Form Factor Pluggable Double Density Multi Source Agreement (QSFP-DD MSA) to support the system host to QSFP-DD module interoperability. It’s called “common” because other 8-lane form factors, such as OSFP and the on-board optics modules defined by the Consortium for On-Board Optics (COBO), have adopted it. The solution is also in use with 4-lane QSFP56 modules. A two-lane variant of CMIS is in use by the SFP-DD MSA.
A particularly novel customer-facing aspect of CMIS is that it advertises which Ethernet rates and reaches it supports. The module basically advertises which row of Table 1 it supports. Not all modules support breakout. Some vendors choose to support 400G Ethernet only. The module in those cases does not advertise the support of lower rates of Ethernet for breakout. It’s because of this support for multiple applications in a single module that the use of a module name is not necessary from a software configuration perspective. Module names are merely for inventory management and marketing. Table 1 gives alternative module names according to two additional schemes (besides scheme A spearheaded and used by Juniper).
An important but less customer-facing attribute of CMIS is that it also defines that the module advertise which host-to-module electrical interfaces it supports. The host selects the desired electrical interface and the module configures itself to support the interface in compliant manner for host-to-module interoperation. For electrical interfaces using 50G PAM4 electrical lanes, there is no need for the host to configure in a piecemeal fashion any SI parameters. The technology relies upon the adaptive equalization of the module for the received electrical signal from the host for the optical Tx path and similarly the adaptive equalization of the host for the electrical signal received from the module for the optical Rx path. Other static SI parameters are set automatically by the QSFP-DD module in compliance with the IEEE 400GAUI-8 chip-to-module specification for 400G Ethernet applications and IEEE 100GAUI-2 chip-to-module specification for 100G Ethernet applications. This is, of course, enabled also by having a host design that is compliant to the IEEE standards, as well. This approach greatly simplifies the integration and qualification of modules across Juniper’s systems since no specialized SI parameter programming of the module is required and certainly not in a per-vendor fashion either.
The host only programs parameters in which the module cannot know the proper value, e.g., when the module is a QSFP-DD and uses two times the CAUI-4 chip-to-module electrical interface for which programmed equalization is employed and adaptive equalization is only an option. The host programs the equalization setting (the CTLE gain value) for the module to match the known characteristics of the host traces reaching the module from the host ASIC or re-timer.
CMIS 3.0 is the first publication of the specification suitable for product. CMIS 4.0 introduces optional features. The facility for firmware upgrade is the most customer-facing feature. The QSFP-DD hardware specification 5.0 provides a dedicated pin for an ePPS input signal to further support the PTP application with better accuracy using the latency reporting features of CMIS 4.0.
Looking Ahead
These first-generation 400G pluggable optics are now in production and we are rolling them out for deployment in customers’ labs around the globe for qualification and production-network readiness. Check with your Juniper representative to learn about optics currently available for purchase. Optical pluggables is a complex industry in itself. At Juniper, we work hard to navigate this complexity – so the optics just work.
In the meantime, next-gen 400G pluggable designs are already underway by vendors, including 400ZR, which will enable 80km distances, which includes data center interconnect within a metro area. 2021 promises to be an exciting year for industry 400G deployments and the optics that make them possible.
