RoCE is an end-to-end network design. Buying a switch with high-speed Ethernet ports does not automatically create a validated RoCE fabric. The NIC, switch, cable, optical module, operating system, driver, and QoS configuration must work together.
- How server access, leaf, spine, and storage roles affect the BOM
- ConnectX adapter selection and switch capability checks
- The congestion-control requirements for a predictable RoCE fabric
- How to match DAC, AOC, optics, distance, speed, and coding
- Drivers, firmware, NOS, management, power, and rack requirements
- BOM examples, procurement information, and frequently asked questions
- The minimum information needed to build a compatible RoCE BOM
What a RoCE Network Actually Needs
A production RoCE network normally includes RDMA-capable server adapters, RoCE-aware switches, compatible optics or cables, host drivers, switch QoS configuration, and a separate management path.
| Category | Required Item | Main Selection Criteria |
|---|---|---|
| Compute | GPU or server nodes | PCIe slots, GPU topology, operating system |
| RDMA connectivity | ConnectX or another supported RDMA NIC | Port speed, port count, PCIe generation, firmware |
| Network fabric | Spectrum or another RoCE-capable switch | PFC, ECN, buffers, QoS, and breakout support |
| Interconnects | DAC, AOC, or optical transceivers | Speed, distance, connector, FEC, and coding |
| Host software | RDMA drivers and utilities | Operating system, driver version, and NIC firmware |
| Switch software | Cumulus Linux, Onyx, SONiC, or another validated NOS | Hardware support, feature support, and contract |
| Infrastructure | Management, rack, PSU, and cooling equipment | Power redundancy, depth, airflow, and cable clearance |
The exact BOM depends on whether the network is being built for GPU-to-GPU communication, distributed storage, HPC, database traffic, or a mixed data center fabric.
Send your server count, NIC model, target speed, cable distance, and switch requirements.
Start with the Network Role
Before selecting a switch, determine whether it will operate as a server-facing leaf, high-density leaf, spine, storage fabric switch, dedicated GPU cluster switch, or mixed enterprise and RoCE switch.
A small server rack may need many 25GbE connections and only a few 100GbE uplinks. A GPU fabric may require mostly 100GbE, 200GbE, or 400GbE connections. These are different BOMs even when both are described as RoCE networks.
Examples of Spectrum Platforms
A compact 25GbE access design may use the MSN2010-CB2RC, which provides native SFP28 access ports together with QSFP28 uplinks.
A QSFP28-based leaf design may use the MSN2100-CB2F, where breakout planning becomes important if 25GbE server connections are required.
A higher-density fabric may use the MSN2700-CS2R, with a larger number of QSFP28 ports for 40/50/100GbE links or supported breakout configurations.
These are platform examples, not universal recommendations. The final choice depends on port count, oversubscription, rack design, optics, and the target NOS.
RDMA NICs and RoCE-Capable Switches
RDMA NIC Selection
The server adapter is one of the most important components in a RoCE deployment. A suitable RDMA NIC should be checked for RoCEv2 support, port speed, port count, PCIe generation, firmware, driver compatibility, PFC, ECN, and any GPU Direct or storage-fabric requirements.
NVIDIA ConnectX adapters are frequently used in RoCE environments, but the exact ConnectX generation and OPN still matter. A dual-port 25GbE SFP28 adapter creates a different cable and switch-port requirement from a single-port 400GbE OSFP adapter.
- NIC model and OPN
- Port count and port speed
- Connector type
- PCIe slot requirement
- Firmware version
- RDMA driver package
- GPU Direct or storage integration requirements
RoCE-Capable Switch Requirements
A switch used for RoCE should be evaluated by its traffic-management capabilities, not only its maximum port speed.
- Priority Flow Control
- Explicit Congestion Notification
- Lossless or loss-aware queue profiles
- Sufficient packet buffering
- DSCP or 802.1p traffic classification
- Congestion notification handling
- Required FEC modes
- Port breakout support
- Validated NOS and software version
A general-purpose Ethernet switch may forward RoCE packets, but that does not mean it provides a validated or predictable RoCE fabric.
PFC, ECN, Buffers, and QoS
Priority Flow Control
PFC pauses selected traffic priorities rather than stopping the entire link. This allows RoCE traffic to receive lossless treatment while other traffic classes continue using the link.
For a lossless RoCE profile, PFC must be designed consistently across the host NIC, leaf switches, spine switches, storage endpoints, and remote server adapters in the traffic path.
- Assign RoCE traffic to a defined priority or traffic class.
- Configure the same priority mapping on the host and switches.
- Do not enable PFC on every priority without a traffic-class design.
- Monitor pause frames and queue behavior after deployment.
Incorrect PFC configuration may result in PFC pause storms, head-of-line blocking, congestion spreading, and unstable application latency.
Explicit Congestion Notification
ECN provides end-to-end congestion signaling for RoCEv2. The switch marks packets when queues reach the configured threshold, and the receiving host participates in congestion response.
- RoCE traffic classification
- ECN-capable host NICs
- Switch marking thresholds
- Host NIC congestion response
- CNP handling
- Consistent DSCP or priority mapping
Buffer and ASIC Differences
RoCE configuration is not automatically portable between switch ASIC generations. A validated profile should be matched to the exact switch ASIC, NOS version, and deployment topology.
NVIDIA’s Ethernet Storage Fabrics guidance covers RoCE-related buffer, queue, PFC, and ECN considerations for Spectrum-based deployments. The configuration should be tested before applying it to production.
Cables, Optics, Breakout, and FEC
DAC and AOC
DAC is commonly used for short in-rack or adjacent-rack connections. AOC may be suitable when the distance is longer than a practical DAC run but does not require separate optical modules.
Check the speed, connector, length, bend radius, FEC, power draw, and switch and NIC support for every DAC or AOC.
Optical Transceivers
For longer links, the BOM should specify the form factor, speed, wavelength, single-mode or multimode fiber, connector type, reach, FEC requirement, temperature range, DOM support, and switch and NIC compatibility.
Breakout Cables
Breakout changes the port-count calculation. One 100GbE QSFP28 port may become four 25GbE connections, while other platforms may support 200GbE or 400GbE breakout modes.
The BOM must identify the parent port and each breakout lane. Do not count a breakout port as four usable interfaces until the exact switch, cable, NOS, and port mode have been verified.
FEC and Link Coding
Two devices may have the same nominal speed and connector but still fail to establish a stable link because of FEC or coding differences.
- Supported FEC modes
- Default FEC behavior
- Manual or automatic FEC configuration
- NIC-to-switch FEC compatibility
- Breakout-lane FEC requirements
- Optic-specific coding requirements
FEC should be part of the link worksheet rather than an issue discovered during installation.
Host Software and Infrastructure
Host Software
- Operating system
- RDMA driver package
- NIC firmware
- RDMA utilities
- PFC and ECN configuration
- DSCP or priority mapping
- GPU Direct or storage-related drivers
Switch Software
- NOS name and version
- Switch ASIC support
- RoCE profile
- PFC and ECN configuration method
- Buffer profile
- Breakout syntax
- Upgrade and rollback procedure
- Software support contract
Management and Rack Infrastructure
A production RoCE network should include an independent management path whenever possible. This allows engineers to reach the switch and servers during data-plane congestion or QoS troubleshooting.
- Out-of-band management switch
- Console access
- Management VLAN or management network
- Monitoring system
- NTP and DNS
- Configuration backup
- Rack power distribution
- Redundant power feeds
- Airflow-compatible switch and PSU variants
- Cable-management accessories
- Spare optics and cables
RoCE BOM and Quote Request Checklist
Example BOM by Deployment Type
| Deployment | Host Connectivity | Switch Role | Typical Interconnect | Main Planning Issue |
|---|---|---|---|---|
| 25GbE server rack | 25GbE SFP28 RDMA NICs | Leaf or top-of-rack | SFP28 DAC, AOC, or optics | Native SFP28 port count and 100GbE uplinks |
| 100GbE GPU fabric | 100GbE QSFP28 NICs | Leaf-spine fabric | 100GbE DAC, AOC, or optics | PFC, ECN, oversubscription, and buffers |
| 200GbE or 400GbE AI fabric | 200/400GbE RDMA adapters | High-speed leaf or spine | QSFP56, QSFP112, or OSFP cables | FEC, breakout, power, and platform support |
| RoCE storage network | RDMA NICs or storage adapters | Dedicated or converged fabric | Speed and distance dependent | Traffic separation and congestion control |
| Mixed enterprise and RoCE network | RDMA NICs and standard Ethernet hosts | Shared data center fabric | Mixed optical and copper links | Queue isolation and PFC scope |
Information to Include in a RoCE Quote Request
- Number of servers or GPU nodes
- RDMA NIC model and quantity
- Required host-side port speed
- Required switch port speed
- Number of leaf and spine switches
- Desired oversubscription ratio
- RoCEv1 or RoCEv2 requirement
- GPUDirect, storage, or HPC application
- Rack-to-rack distances
- DAC, AOC, or optical preference
- Fiber type and connector requirements
- FEC requirements
- PFC and ECN requirements
- NOS preference
- Rack depth, airflow, and power requirements
- Required spare optics and cables
Frequently Asked Questions
Final Takeaway
A complete RoCE network BOM normally includes RDMA-capable server or GPU NICs, RoCE-aware switches, DACs, AOCs or optical transceivers, host drivers, a supported switch NOS, PFC and ECN configuration, management networking, and rack, power, cooling, and cable-management components.
The safest way to build the BOM is to start from the application and traffic pattern, then match the NIC, switch, cable, optics, software, and support model as one system.
Send the server count, RDMA NIC model, target speed, switch preference, cable distance, and RoCE requirements before finalizing the equipment list.