The NVIDIA SN5610 switch is designed for high-density Ethernet fabrics, including AI and data center networks that need to connect 800G, 400G, 200G, and 100G devices in the same platform. This guide explains how the breakout modes work, how to configure them with Cumulus Linux, and what to check when a split interface or link does not come up.
SN5610 800G Breakout Options
The SN5610 supports 64 OSFP 800GbE interfaces. Depending on the required endpoint speed, the physical ports can be used natively or split into multiple logical interfaces.
| Physical mode | Logical ports per OSFP | Maximum on SN5610 | Main limitation |
|---|---|---|---|
| 800G | 1×800G | 64 ports | Native port operation |
| 400G | 2×400G | 128 ports | Requires a supported two-way breakout design |
| 200G | 4×200G | 256 ports | Requires a supported four-way breakout design |
| 100G | 8×100G | 256 ports | Uses 32 OSFP source ports and requires the adjacent port to be disabled |
The 100G mode needs special attention. It is not a simple conversion of all 64 physical OSFP ports into eight logical ports each. The adjacent-port restriction reduces the number of physical source ports available for this configuration.
The SN5610 supports both PAM4 and NRZ encoding. Lower-speed operation may use NRZ encoding, while the exact lane behavior depends on the port mode, transceiver, breakout cable, and peer device. Confirm the current platform matrix before finalizing the port map.
For native 800G connections, review the required form factor and optical reach before selecting an 800G OSFP transceiver. A transceiver that matches the advertised data rate may still be unsuitable for the selected breakout topology or remote interface.
What to Confirm Before Configuration
Breakout planning should begin with the endpoint requirements. Do not select the cable first and try to fit the network design around it.
Confirm the required port shape
- Required number of 800G, 400G, 200G, and 100G links
- Speed supported by the remote switch, NIC, server, or accelerator
- OSFP, QSFP112, QSFP56, or QSFP28 interface at each end
- DAC, AOC, optical transceiver, or fiber-patch requirements
- Required FEC mode and auto-negotiation behavior
- Cumulus Linux version running on the SN5610
Record the physical-to-logical port map
For every breakout source port, record the physical OSFP number, resulting logical interface names, connected device, cable or optic model, speed, FEC, and MTU. This is especially important for 8×100G designs because the adjacent physical port may be unavailable.
A configuration record should also identify whether the link is used for a compute-facing connection, leaf-to-spine uplink, storage network, management path, or inter-switch link. The same breakout mode can be suitable for one role and inefficient for another.
Send your port counts, Cumulus version, and cable requirements.
Configure Breakout Ports in Cumulus Linux
Cumulus Linux 5.4 and later uses breakout notation such as 1x, 2x, 4x, and 8x. The breakout mode is configured separately from the speed of the resulting logical interfaces.
NVUE example: 800G to 2×400G
The following example uses swp1 as the physical source port. The actual interface name must match the SN5610 port map.
NVUE example: 800G to 4×200G
NVUE example: 800G to 8×100G
The resulting interfaces are normally named from swp1s0 through swp1s7. Confirm the adjacent-port state before using this configuration in production.
ports.conf method
With the Linux configuration method, define the breakout shape in /etc/cumulus/ports.conf. The following is an illustrative 8×100G configuration. Replace the port numbers with the correct values for the planned SN5610 port map.
The port speed is configured separately in NVUE or in /etc/network/interfaces. After editing the file, validate the syntax and reload the switching service according to the installed Cumulus Linux version and production-change procedure.
On a production fabric, confirm whether the reload procedure causes a temporary traffic impact in the selected configuration mode. Do not assume that the same change procedure applies to every Cumulus Linux release or ACL mode.
Check lane settings when connecting to another platform
On some 8-lane QSFP-DD or OSFP ports, Cumulus Linux uses a default lane allocation for a 4x breakout. If the peer platform uses a different lane width, both ends may need an explicit lane configuration. Check the current port attributes before changing the speed alone.
OSFP, Breakout Cable, and QSA Limits
A breakout cable must match the source port, target port, lane count, encoding, speed, reach, and FEC requirements. Connector shape alone is not sufficient evidence of compatibility.
For procurement, compare the required topology with the available NVIDIA Mellanox breakout and DAC cables. Confirm whether the selected item is an 800G-to-2×400G, 800G-to-4×200G, or 800G-to-8×100G assembly before ordering.
Check the cable and optic combination
- Source connector and target connector
- Electrical lane mapping
- Supported data rate on every logical interface
- DAC, AOC, or optical transceiver type
- Fiber type, connector, and maximum reach
- FEC requirement for the cable or optic
- Support status on the installed Cumulus Linux release
Do not assume generic QSA support
Generic QSA guidance for placing an SFP transceiver in a QSFP port does not automatically apply to an SN5610 OSFP port. NVIDIA documents QSA behavior for supported QSFP interfaces, but that should not be treated as a blanket guarantee for OSFP-to-SFP operation.
Before using an adapter, confirm the exact adapter model, OSFP port mode, SFP or SFP28 optic, Cumulus Linux version, peer interface, and supported speed. A QSA configuration should not be treated as a breakout configuration unless the platform documentation explicitly supports that design.
Match FEC at both ends
FEC behavior depends on the cable type, optical module, lane speed, and remote device. A link may remain down when the two ends use incompatible FEC settings, or it may show increasing errors after coming up.
Plan SN5610 for an AI Leaf-Spine Fabric
In an AI fabric, breakout planning should include compute-facing links, leaf-to-spine uplinks, redundancy, oversubscription, and future expansion. A higher logical port count does not automatically produce a better design.
| Deployment requirement | Mode to evaluate | Planning focus |
|---|---|---|
| High-bandwidth spine or core links | 800G or 2×400G | Optics, FEC, redundancy, and link reach |
| Leaf-to-spine uplinks | 400G or 200G | Uplink count, oversubscription, and failure domains |
| 200G accelerator or NIC connections | 4×200G breakout | Endpoint connector, cable length, and lane mapping |
| 100G server or legacy connections | 8×100G breakout | Adjacent-port restriction and logical-port allocation |
| Mixed-speed migration | 800G, 400G, 200G, and 100G | Reserved ports, optics, FEC, and documented port map |
For each leaf and spine, document the physical OSFP port, breakout mode, logical interface, endpoint, cable or optic model, FEC, MTU, and expected failover path. This makes later replacement and troubleshooting much faster.
NVIDIA reference architectures provide examples of 400G-class connections in AI infrastructure, but the correct topology still depends on the accelerator platform, NIC speed, fabric size, redundancy model, and oversubscription target. Use the NVIDIA AI Factory reference architecture as a design reference rather than a universal port plan.
Keep MTU consistent across the path
Cumulus Linux commonly uses a 9216-byte MTU on data-plane interfaces, but the effective path MTU depends on every physical interface, bond, bridge, VLAN, VXLAN, and peer device. A breakout change does not by itself require a new MTU, but AI and overlay networks must use an end-to-end MTU plan.
Troubleshoot Missing Ports and Link Failures
If the physical OSFP module is detected but the expected split interfaces do not appear, check the breakout mode, adjacent-port restriction, split-interface speed, lane configuration, and cable or optic support before changing the MTU.
| Symptom | Likely cause | First checks |
|---|---|---|
| Split interfaces do not appear | Breakout mode was not applied or the port map is incorrect | nv show interface, cat /etc/cumulus/ports.conf |
| The adjacent port is down | 8×100G breakout restriction | Check the source port and adjacent-port allocation |
| The optic is detected but the link is down | Wrong speed, cable, lane mapping, or FEC | nv show platform transceiver, ethtool |
| The link comes up and then flaps | FEC, cable quality, or peer negotiation issue | Compare FEC and error counters on both ends |
| Large packets fail | MTU mismatch or overlay overhead | Check physical, bond, bridge, VLAN, VXLAN, and peer MTU |
Verify the breakout and transceiver state
Check the link speed and FEC
Compare the operational and applied values for the split interface. If the speed is correct but the link remains down, compare the FEC mode and lane count with the peer device.
Check MTU only after the physical link is correct
Use the following commands to confirm the configured and operational MTU. If the physical link, speed, FEC, and transceiver are correct, compare the MTU across the complete forwarding path.
Frequently Asked Questions
Final Takeaway
The SN5610 can operate as a native 800G switch or as a high-density 400G, 200G, or 100G platform. The main design detail to remember is that the 8×100G mode uses 32 OSFP source ports and requires the adjacent-port restriction to be included in the port plan.
A reliable deployment should document the breakout mode, logical interface names, cable or optic model, lane configuration, speed, FEC, and MTU together. If you need help validating a port map or matching the required optics and cables, send your SN5610 requirements for review.