====== Data Center Fabrics ======

Stage count - is number of devices I need to pass through in order to get from source to the destination.

Clos fabric is a non-planar graph - there is no way to draw all those connections on a flat plane without crossings.


5-stage fabrics type:
  * Fat tree
  * Benes
  * Butterfly


Over-subscription - is the ratio between total bandwidth offered to workloads and to total available fabric bandwidth.

Total Edge port count = spine ports * over-subscription * break-out ration.

Example 1 - three stage Clos using 64x100 spines and leafs with over-subscription 2:1 can provide 4096 edge ports.\\

Example 2 - 5 stage butterfly using 64x100 fabrics with over-subscription 2:1 can provide 65536 edge ports.

It's possible to add 7th stage, which often used for DCI. 


==== Optics ====

^Name^Meaning^Distance^Notes^
|SR|Short Range|Up to 100m|Multimode Fiber|
|DR|Data Range/Rate|Up to 500m| SMF or MMF|
|FR|Far Range|Up to 2km| SM |
|LR|Long Range|Up to 10km| SM |
|ER|Extended Range|Up to 40km| SM |
|ZR| Zero Dispersion/Coherent| Over 80 km| SM|

Number indicates number of fibers: SR8 has 8 fibers, FR4 has 4 fibers etc...

==== QoS ====

In general, you don't want deep buffers in the fabric, because most applications prefer to drop traffic rather than queue it.

Bufferbloat occurs when excessively large switch/router buffers absorb traffic bursts instead of dropping packets, hiding congestion signals from TCP (or other transport protocols). The result:

  - Latency spikes — packets sit in deep queues for milliseconds to tens of milliseconds instead of being delivered or dropped promptly.
  - Reduced throughput — TCP's congestion control reacts too late because loss signals are delayed.
  - Jitter — unpredictable queue depths cause variable RTTs.


==== ROUTING ====

There are three main designs:
  - iBGP underlay and eBGP overlay
    - Spines and Super-spines are in iBGP domain
    - Each leaf has its unique ASN assigned
    - Leaves establish multi-hop eBGP with each other
  - eBGP underlay and iBGP overlay (Juniper way)
    - All leaves share the same ASN
    - Spines and super-spines are connected using eBGP
    - Leaves are connected using iBGP
  - eBGP for both 


BGP convergence challenges:
  - Path hunting 
  - Over-flooding

Solutions:
  - Valley-free routing - make sure leaves are non-transit routers. One of the solutions is to apply outbound AS-path filter allowing only ''^$''

BGP Tuning options:
  - MRAI (Min route advertisement interval)