Differences

This shows you the differences between two versions of the page.

--- networking:clos [2026/03/27 11:13] – v1ctor
+++ networking:clos [2026/04/07 13:26] (current) – [ROUTING] v1ctor
@@ Line 9: / Line 9: @@
   * Fat tree
   * Benes
   * Butterfly
+Over-subscription - is the ratio between total bandwidth offered to workloads and to total available fabric bandwidth.
+Total Edge port count = spine ports * over-subscription * break-out ration.
+Example 1 - three stage Clos using 64x100 spines and leafs with over-subscription 2:1 can provide 4096 edge ports.\\
+Example 2 - 5 stage butterfly using 64x100 fabrics with over-subscription 2:1 can provide 65536 edge ports.
+It's possible to add 7th stage, which often used for DCI.
+==== Optics ====
+^Name^Meaning^Distance^Notes^
+|SR|Short Range|Up to 100m|Multimode Fiber|
+|DR|Data Range/Rate|Up to 500m| SMF or MMF|
+|FR|Far Range|Up to 2km| SM |
+|LR|Long Range|Up to 10km| SM |
+|ER|Extended Range|Up to 40km| SM |
+|ZR| Zero Dispersion/Coherent| Over 80 km| SM|
+Number indicates number of fibers: SR8 has 8 fibers, FR4 has 4 fibers etc...
+==== QoS ====
+In general, you don't want deep buffers in the fabric, because most applications prefer to drop traffic rather than queue it.
+Bufferbloat occurs when excessively large switch/router buffers absorb traffic bursts instead of dropping packets, hiding congestion signals from TCP (or other transport protocols). The result:
+  - Latency spikes — packets sit in deep queues for milliseconds to tens of milliseconds instead of being delivered or dropped promptly.
+  - Reduced throughput — TCP's congestion control reacts too late because loss signals are delayed.
+  - Jitter — unpredictable queue depths cause variable RTTs.
+==== ROUTING ====
+There are three main designs:
+  - iBGP underlay and eBGP overlay
+    - Spines and Super-spines are in iBGP domain
+    - Each leaf has its unique ASN assigned
+    - Leaves establish multi-hop eBGP with each other
+  - eBGP underlay and iBGP overlay (Juniper way)
+    - All leaves share the same ASN
+    - Spines and super-spines are connected using eBGP
+    - Leaves are connected using iBGP
+  - eBGP for both
+BGP convergence challenges:
+  - Path hunting
+  - Over-flooding
+Solutions:
+  - Valley-free routing - make sure leaves are non-transit routers. One of the solutions is to apply outbound AS-path filter allowing only ''^$''
+BGP Tuning options:
+  - MRAI (Min route advertisement interval)