Selecting networking hardware for an AI or HPC cluster requires more than comparing switch port speeds. The correct design depends on the network protocol, GPU node count, fabric topology, NIC configuration, cable type, PCIe generation, congestion-control features, and rack environment.
Share your GPU nodes, fabric, rail design, NICs, and rack requirements for a configuration review.
Start with the Network Fabric
AI and HPC clusters commonly use InfiniBand, Ethernet with RoCE, or conventional Ethernet for different network layers. The choice should be made before selecting a switch model.
| Fabric | Typical Use | Main Planning Concern |
|---|---|---|
| InfiniBand | Large-scale GPU training and latency-sensitive HPC | NDR/XDR compatibility, Subnet Manager, cables, and topology |
| Ethernet/RoCE | AI clusters using Ethernet infrastructure | PFC, ECN, QoS, congestion control, and lossless behavior |
| Conventional Ethernet | Management, storage, frontend, and smaller compute networks | Port density, uplink capacity, and upgrade path |
InfiniBand and Ethernet/RoCE should not be treated as interchangeable simply because both may use 400G or 800G interfaces. A 400G NDR InfiniBand adapter is not automatically compatible with a 400GbE switch.
The protocol mode, firmware, connector, transceiver, cable, and operating system must be checked together. For QM9700 and QM9790 deployments, consult the official NVIDIA QM97xx hardware documentation before finalizing the fabric.
For Ethernet or RoCE designs, platforms from NVIDIA Spectrum, Cisco Nexus, Arista, and Juniper may all be considered. Their operating systems, congestion-management features, automation tools, telemetry, and validated designs are different, so the comparison should focus on the complete architecture.
Size the Network by Nodes and Rails
Do not begin with the number of switches available in inventory. Begin with the number of network connections required by the servers.
Required fabric ports = GPU servers × NIC ports per server × number of network planes + uplink ports + spare capacity
A single-rail design may use one backend connection per server. A dual-rail design uses two independent network planes and therefore requires additional NIC ports, switch ports, cables, and fabric-management planning.
A 64-port switch does not necessarily provide 64 available server-facing ports. Some ports may be reserved for spine links, inter-switch links, storage connections, redundancy, maintenance, and future expansion.
In a two-tier fat-tree or leaf-spine architecture, the final switch count depends on the uplink ratio and the desired oversubscription. The same number of GPU servers can require different switch quantities under single-rail, dual-rail, and multi-plane designs.
Single-Rail Design
Single-rail networking may be suitable for smaller clusters, moderate-scale inference, or projects that prioritize a lower initial hardware cost. Its limitation is that one adapter, cable, or fabric failure can affect the server’s backend connectivity.
Dual-Rail Design
Dual-rail networking is more appropriate when training jobs depend heavily on east-west traffic, when the project requires higher availability, or when the software stack is designed around separate network planes.
Compare Hardware by Network Role
Different brands can be considered, but they should be compared by network role and fabric capability rather than by brand name alone.
| Network Role | Example Platforms | What to Compare |
|---|---|---|
| NDR InfiniBand fabric | NVIDIA Quantum-2 QM9700/QM9790 | Port mode, management model, Subnet Manager, airflow, and fabric software |
| 400G/800G Ethernet fabric | NVIDIA Spectrum, Cisco Nexus 9000, Arista 7060X/7060X6, Juniper QFX | RoCEv2, PFC, ECN, buffers, telemetry, and breakout options |
| 100G/200G server access | Cisco Nexus, Arista 7050/7060, Juniper QFX | NIC compatibility, uplink ratio, port density, and migration path |
| Storage or frontend network | Ethernet platforms from multiple vendors | Storage protocol, routing, redundancy, and bandwidth isolation |
A switch designed for an AI backend fabric may not be the best choice for management or storage traffic. In many deployments, the backend, frontend, storage, and management networks should be planned as separate logical or physical layers.
If you are comparing Quantum-2 options, the NVIDIA Quantum-2 QM9700 Series provides a useful starting point for reviewing related switch platforms.
Match the Switch to the NIC
The server adapter must be selected together with the switch, fabric protocol, cable, and host platform.
| Adapter | Primary Role | Suitable Planning Scenario | Important Check |
|---|---|---|---|
| MCX515A-CCAT | Single-port 100GbE QSFP28 | 100G Ethernet or RoCE endpoint | Not a default NDR InfiniBand endpoint |
| MCX75310AAS-NEAT | Single-port 400G OSFP adapter | 400G Ethernet or NDR InfiniBand endpoint | PCIe generation, firmware, cable, and operating mode |
| 900-9X81Q-00CN-ST0 | Dual-port ConnectX-8 adapter | Dual-rail or high-bandwidth 400G designs | QSFP112, host PCIe, cable, and protocol mode |
| 900-9X81E-00EX-ST0 | ConnectX-8 800G/XDR-oriented adapter | Next-generation 800G or XDR designs | Do not assume direct compatibility with an NDR QM97xx fabric |
A higher port speed does not automatically make an adapter a suitable replacement. 100GbE is not the same as 100G InfiniBand, and 400GbE is not the same as 400G NDR.
OSFP, QSFP28, QSFP56, and QSFP112 are also not interchangeable without checking the platform documentation, cable specification, lane mapping, and supported breakout mode.
Managed and Unmanaged InfiniBand Switches
A managed platform such as MQM9700-NS2F may be considered when the switch needs to participate in fabric management and operational monitoring.
MQM9790 variants should be evaluated when external fabric management is part of the design. The NS2F and NS2R variants also represent different airflow directions, so the choice should follow the rack cooling plan rather than price alone.
Select Hardware by Deployment Scenario
400G NDR InfiniBand Cluster
A typical NDR design may include QM9700 or QM9790 switch platforms, 400G NDR-compatible server adapters, OSFP DAC or AOC cables, a defined Subnet Manager strategy, and a leaf-spine or fat-tree topology.
The main risk is mixing an NDR fabric with Ethernet-only adapters or cables that do not support the selected InfiniBand mode.
100G Ethernet or RoCE Cluster
A 100G Ethernet/RoCE design may use MCX515A-CCAT server adapters, 100G Ethernet switches, PFC and ECN configuration, and RoCE-aware QoS.
This path may be easier to integrate with an existing Ethernet data center, but congestion control and lossless behavior require careful engineering.
High-Density 400G Cluster
A high-density 400G cluster may use MCX75310AAS-NEAT, dual-port ConnectX-8 adapters, 400G Ethernet or NDR switching, and dual-rail connectivity.
The server PCIe generation, OSFP or QSFP112 cable, firmware version, and network protocol must be confirmed before the adapter is added to the BOM.
800G or XDR-Oriented Cluster
An 800G design should be planned as a separate network generation. The 900-9X81E-00EX-ST0 may be considered for an 800G or XDR-oriented architecture, but it should not automatically be added to an MQM9700 or MQM9790 NDR bill of materials.
Confirm the switch generation, fabric protocol, adapter mode, cable and optical module, firmware, PCIe support, and management software before combining these components.
Apply the Inventory to a Project BOM
The following quantities and prices are provided as a project inventory snapshot for planning. They should not be treated as fixed public pricing or as a complete cluster BOM.
| SKU | Quantity | Reference Unit Price | Possible Role |
|---|---|---|---|
| MQM9700-NS2F | 20 | US$29,412 | Managed NDR InfiniBand fabric |
| MQM9790-NS2F | 10 | US$22,200 | Unmanaged NDR fabric, P2C airflow |
| MQM9790-NS2R | 25 | US$22,945 | Unmanaged NDR fabric, C2P airflow |
| MCX515A-CCAT | 200 | US$307 | 100G Ethernet/RoCE endpoint |
| MCX75310AAS-NEAT | 268 | US$1,300 | 400G Ethernet/NDR endpoint |
| 900-9X81Q-00CN-ST0 | 653 | US$1,680 | Dual-port ConnectX-8 option |
| 900-9X81E-00EX-ST0 | 980 | US$1,680 | 800G/XDR-oriented option |
The quantities should be mapped to a specific project design before quotation. Switch quantity cannot be evaluated without the number of GPU nodes, rail architecture, uplink ratio, and expansion target.
Reference pricing, availability, lead time, and allocation should be reconfirmed before purchasing.
Compatibility Checklist and FAQs
Information to Include in a Project BOM
- Number of GPU servers and GPUs per server
- Single-rail or dual-rail architecture
- InfiniBand, Ethernet, or RoCE fabric
- NIC port count and port speed
- Switch port count and usable server-facing ports
- OSFP, QSFP28, QSFP56, or QSFP112 interface requirements
- DAC, AOC, or optical transceiver requirements
- Server PCIe generation and slot width
- Switch and NIC firmware versions
- FEC, PFC, ECN, and QoS requirements
- Subnet Manager or external fabric-management requirements
- Rack airflow, power, cooling, and expansion requirements
Frequently Asked Questions
Information to Include in a Quote Request
- GPU server quantity and GPU model
- Single-rail or dual-rail requirement
- Preferred InfiniBand, Ethernet, or RoCE fabric
- Required switch and NIC speed
- Rack airflow and power requirements
- Required cables, optics, and delivery region
For a configuration-specific review, you can send your AI/HPC networking requirements, including the GPU node count, NIC configuration, fabric type, rail design, and target delivery date.
Final Takeaway
The best AI/HPC networking decision is not based on the lowest switch price or the highest advertised port speed.
A reliable design connects four decisions: the fabric protocol, the cluster topology, switch and NIC compatibility, and the cable, firmware, power, and cooling requirements.