The main mistake in AI cluster sizing is to start with the GPU count and immediately divide it by switch ports. A reliable design must first identify the number of GPU servers, the number of network links per server, the required fabric layers, and the separate storage and management networks. This guide explains how to estimate switch, optic, DAC/AOC, and breakout requirements for 400G and 800G AI Ethernet fabrics.
For a broader architecture view, see our AI GPU cluster networking solutions covering Ethernet fabrics, high-speed optics, DAC/AOC, and multi-rack deployment planning.
- Compute, storage, management, and service networks
- Leaf, spine, and Super-Spine sizing examples
- Port density, breakout, reach, and expansion considerations
- Optics, DAC/AOC, breakout, and spare planning
- How reference architecture counts should be interpreted
- Steps for turning GPU requirements into a BOM
- Lossless Ethernet and quotation requirements
- Common AI GPU cluster sizing questions
- The practical rule for AI cluster switch sizing
Start With GPU Servers, Not Only GPU Count
The first sizing mistake is to start with the number of GPUs and immediately divide by switch ports. The correct starting point is the number of GPU servers and the number of high-speed network links used by each server.
Use the following basic calculation:
GPU server count = Total GPU count ÷ GPUs per server
A cluster with 1,024 GPUs could use 128 servers with 8 GPUs per server, or 256 servers with 4 GPUs per server. These designs do not create the same number of network links. The server configuration, network adapter count, port speed, and rail design determine the access-side port requirement.
A server with multiple 400G or 800G connections can consume several leaf ports even when the total server count remains unchanged.
What Networks Should Be Included in the BOM?
A complete AI cluster BOM should not only count the network used for GPU-to-GPU communication. A production cluster may include several separate or partially converged networks.
| Network | Main Function | Planning Question |
|---|---|---|
| East-West compute fabric | GPU collective communication and distributed training | How many high-speed links connect each GPU server? |
| Storage network | Dataset access, checkpoints, and parallel storage | Is storage traffic isolated from GPU traffic? |
| Management network | Provisioning, orchestration, monitoring, and control | Which nodes and devices require management access? |
| Out-of-band network | BMC, console, and infrastructure management | Are server and switch management ports separated? |
| PDU and service network | Power control, console access, and operational services | Are power and support systems included in the BOM? |
Cisco’s reference architecture separates the East-West compute network from the North-South storage and management network. It also describes separate switch management, BlueField management, and miscellaneous service networks.
This is why a BOM that only lists GPU-facing leaf switches is usually incomplete. For broader compute, storage, and network planning, see the AI cluster network, storage, and compute planning guide.
Send your GPU count, server configuration, link speed, storage needs, and growth plan.
How Many Compute Switches Are Needed?
Cisco’s reference architecture provides an example of a three-tier East-West compute network using high-density Cisco 8000 platforms. The architecture uses Leaf switches, Spine switches, Super-Spine switches, 800G switch-to-switch links, and 400G server-to-leaf connections.
For larger AI fabrics, compare the available Cisco 8000 Series switches by port density, uplink speed, supported breakout modes, and exact hardware PID.
| GPU Nodes | GPUs | Leaf Switches | Spine Switches | Super-Spine Switches |
|---|---|---|---|---|
| 128 | 1,024 | 16 | 16 | 8 |
| 256 | 2,048 | 32 | 32 | 16 |
| 512 | 4,096 | 64 | 64 | 32 |
| 1,024 | 8,192 | 128 | 128 | 64 |
| 2,048 | 16,384 | 256 | 256 | 128 |
| 4,096 | 32,768 | 512 | 512 | 256 |
These figures describe the compute fabric in a specific Cisco reference architecture. They should not be copied directly into a smaller enterprise cluster. A different server NIC layout, port density, or topology can produce a different result.
400G or 800G: Which Should You Choose?
The choice between 400G and 800G depends on cluster scale, server interface design, switch radix, and expansion requirements.
| Consideration | 400G Design | 800G Design |
|---|---|---|
| Typical role | GPU server links, storage, and medium-to-large fabrics | High-density leaf, spine, and large-scale AI fabrics |
| Port density | Easier to deploy in many current server and switch designs | Higher bandwidth per physical port |
| Breakout use | 400G to 4 × 100G on compatible platforms | 800G to 2 × 400G on compatible platforms |
| Procurement complexity | More familiar optic and cabling options | Requires strict OSFP/QSFP-DD and breakout validation |
| Expansion model | Suitable for phased deployment | Useful when high-density growth is planned from the beginning |
A 400G deployment is not automatically smaller or cheaper. If every server requires multiple 400G ports, the total number of server-facing links can still be substantial.
An 800G platform may reduce the number of physical switch ports in some spine or leaf roles, but the final result depends on how the ports are broken out and how much bandwidth each server requires.
How to Count Transceivers and Cables
Every physical link should be counted from both ends. For each connection, confirm the port speed, lane configuration, optical module or cable type, fiber distance, breakout design, and FEC requirements.
| Link Type | Device A | Device B | Items to Count |
|---|---|---|---|
| GPU server to leaf | GPU NIC | Compute leaf | One module or cable assembly per endpoint |
| Leaf to spine | Leaf switch | Spine switch | Two switch-side endpoints per link |
| Spine to Super-Spine | Spine switch | Super-Spine | 400G/800G links according to the topology |
| Storage node to storage leaf | Storage NIC | Storage leaf | Storage-side optics or DAC/AOC |
| Management and BMC | Server management port | Management switch | Management cables and port adapters |
For a compatible 400G optical connection, a product such as QDD-400G-DR4-S may be considered after checking the switch PID, fiber type, reach, coding, and supported FEC.
For very short in-rack connections, a passive cable such as QDD-400-CU1M may be considered after checking port compatibility and supported cable length.
A 400G-to-400G connection, a 400G-to-4 × 100G breakout, and an 800G-to-2 × 400G breakout do not use the same BOM structure. The port configuration and every resulting endpoint must be included.
For the selection tradeoffs between copper DAC, fiber AOC, and optical modules, see the 400G DAC, AOC, and optical link guide.
Cisco Reference BOM Examples
Cisco’s published BOM table gives an example of how a complete reference architecture grows from 4K to 32K GPUs.
| GPUs | HGX Server Nodes | 800G Switches | 400G Switches | Low-Speed Management Switches |
|---|---|---|---|---|
| 4,096 | 512 | 184 | 12 | 36 |
| 8,192 | 1,024 | 378 | 26 | 72 |
| 16,384 | 2,048 | 754 | 50 | 144 |
| 32,768 | 4,096 | 1,506 | 98 | 288 |
These are minimum BOM values for Cisco’s defined architecture, not a generic estimate for all AI deployments. The reference design uses specific Cisco and NVIDIA hardware, server configurations, link speeds, and topology assumptions.
The table also shows why the full cluster BOM includes more than the main compute leaf and spine switches. Storage, management, and lower-speed infrastructure switches can contribute a significant number of devices.
Cisco’s current 8000 Cloud Reference Architecture should be used to verify the scope and assumptions behind these example counts.
A Practical Sizing Workflow
Define the Compute Unit
- Record the total GPU count.
- Confirm the number of GPUs per server.
- Calculate the number of GPU servers.
- Identify the NIC model and port count per server.
- Record the target port speed.
- Confirm the number of independent GPU network rails.
Define the Fabric
- Choose a two-tier or three-tier topology.
- Confirm leaf port density.
- Confirm spine radix and uplink requirements.
- Define the oversubscription target.
- Document the redundancy model.
- Include the planned growth target.
Define the Non-Compute Networks
- Storage leaf switches
- Storage spine or uplink requirements
- Management switches
- BMC and out-of-band access
- Switch management
- PDU and console connectivity
Define the Physical Interconnects
- 400G and 800G optics
- 100G or 200G breakout links
- DAC and AOC lengths
- Single-mode or multimode fiber
- FEC and lane requirements
- Spare modules and cables
Validate the Platform
- Exact switch PID
- Supported network operating system
- Port breakout support
- Optic and cable compatibility
- Buffer and congestion-management features
- PFC, ECN, and MTU support
- Power and airflow requirements
- Software and support entitlement
RoCE and Procurement Checks
If the GPU fabric uses RoCEv2, switch sizing alone is not enough. The design should also verify PFC behavior, ECN marking, queue and buffer profiles, MTU consistency, priority mapping, congestion control, FEC compatibility, NIC and driver support, failure recovery, monitoring, and telemetry.
A fabric can have enough raw bandwidth and still perform poorly if congestion control and queue behavior are not aligned across servers, switches, and optics. Use the AI Cluster Lossless Networking with RoCEv2 guide for the detailed PFC, ECN, buffer, and GPUDirect planning considerations.
A sizing or quotation request should include:
- Total GPU count
- GPU model and GPUs per server
- Number of GPU servers
- NIC model and port count
- 400G or 800G target speed
- Storage system and interface speed
- Management node count
- Target Leaf-Spine topology
- Required redundancy
- Fiber distance
- DAC/AOC length
- Breakout requirements
- Planned expansion size
- Optics and cable spares
- Power and airflow restrictions
Frequently Asked Questions
Final Takeaway
AI GPU cluster switch sizing begins with the server and network topology, not with a simple GPU-to-switch ratio.
For a reliable 400G or 800G BOM, calculate GPU servers, GPU-facing links, Compute Leaf and Spine layers, Super-Spine requirements, storage and management networks, optical modules, breakout assemblies, DAC/AOC and fiber cables, redundant links, and spares.
Cisco’s 8000 Cloud Reference Architecture provides useful examples from 1K to 32K GPUs, but those figures apply to a defined Cisco and NVIDIA design. For a different GPU server, switch platform, NOS, or topology, the BOM must be recalculated.
If you are preparing a cluster purchase, send the GPU count, server configuration, target link speed, storage requirements, and expansion plan for a switch, optics, and cabling review.















































































































































