Data Center Fabrics

Stage count - is number of devices I need to pass through in order to get from source to the destination.

Clos fabric is a non-planar graph - there is no way to draw all those connections on a flat plane without crossings.

5-stage fabrics type:

• Fat tree
• Benes
• Butterfly

Over-subscription - is the ratio between total bandwidth offered to workloads and to total available fabric bandwidth.

Total Edge port count = spine ports * over-subscription * break-out ration.

Example 1 - three stage Clos using 64×100 spines and leafs with over-subscription 2:1 can provide 4096 edge ports.

Example 2 - 5 stage butterfly using 64×100 fabrics with over-subscription 2:1 can provide 65536 edge ports.

It's possible to add 7th stage, which often used for DCI.

Number indicates number of fibers: SR8 has 8 fibers, FR4 has 4 fibers etc…

In general, you don't want deep buffers in the fabric, because most applications prefer to drop traffic rather than queue it.

Bufferbloat occurs when excessively large switch/router buffers absorb traffic bursts instead of dropping packets, hiding congestion signals from TCP (or other transport protocols). The result:

1. Latency spikes — packets sit in deep queues for milliseconds to tens of milliseconds instead of being delivered or dropped promptly.
2. Reduced throughput — TCP's congestion control reacts too late because loss signals are delayed.
3. Jitter — unpredictable queue depths cause variable RTTs.

There are three main designs:

1. iBGP underlay and eBGP overlay
   1. Spines and Super-spines are in iBGP domain
   2. Each leaf has its unique ASN assigned
   3. Leaves establish multi-hop eBGP with each other
2. eBGP underlay and iBGP overlay (Juniper way)
   1. All leaves share the same ASN
   2. Spines and super-spines are connected using eBGP
   3. Leaves are connected using iBGP
3. eBGP for both

BGP convergence challenges:

1. Path hunting
2. Over-flooding

Solutions:

1. Valley-free routing - make sure leaves are non-transit routers. One of the solutions is to apply outbound AS-path filter allowing only ^$

BGP Tuning options:

1. MRAI (Min route advertisement interval)