Data Center Networking Guide

This is an original planning guide. It is not a vendor manual, an official standard, or legal advice. Example addresses use documentation ranges. Review every command before you paste it.

A data-centre network is a fabric plus a policy

Leaf-spine is a way to give every leaf the same number of hops to every other leaf, through more than one spine, so one spine failure is a reduction in bandwidth and not an outage. It is not a product name. You can build a poor leaf-spine with no redundancy in the attachments, and you can build a fine smaller network that is not leaf-spine at all. Choose it when east-west traffic and a predictable failure mode are the requirements.

Oversubscription is downlink bandwidth divided by uplink bandwidth. It is a capacity fact. A leaf with more server bandwidth than spine bandwidth will drop traffic when the servers actually speak. Many designs accept that, because servers do not all speak at once. Write the ratio you accept. The capacity guide is how you defend it. A ratio you cannot say out loud is a ratio you have not chosen.

Routing in the fabric is usually a small, fast protocol with equal paths, or a design that uses the same idea under another name. Every link should be a routed link with a prefix you planned, not a giant VLAN stretched across the room. A stretched VLAN feels simple and fails as one broadcast domain. Keep the VLAN at the leaf, at the edge of the server, if the server needs one at all.

This page does not design your storage network for you and it does not quote a vendor fabric guide. The calculators linked here estimate ports and ratios from numbers you type. They do not log into a switch. The ticket still needs the failure you are protecting against.

Three sketches and their ratios

2 spines and 4 leaves, with each leaf using 2 uplinks because it connects to every spine, and 4 downlinks toward servers in this sketch, is an oversubscription of 4:2 if every downlink and every uplink is the same speed. If the uplinks are faster, recompute. The point of the sketch is that you can say the ratio. A leaf that connects to only one spine is not a leaf-spine fabric. It is a star, and the loss of that spine is the loss of the leaf. Every leaf should see every spine. A server that needs two attachments should see two leaves, not two ports on one leaf, if the question was leaf failure. Write which failure you are buying protection from. Port count is not the same question.

4 spines and 8 leaves, with each leaf using 4 uplinks because it connects to every spine, and 8 downlinks toward servers in this sketch, is an oversubscription of 8:4 if every downlink and every uplink is the same speed. If the uplinks are faster, recompute. The point of the sketch is that you can say the ratio. A leaf that connects to only one spine is not a leaf-spine fabric. It is a star, and the loss of that spine is the loss of the leaf. Every leaf should see every spine. A server that needs two attachments should see two leaves, not two ports on one leaf, if the question was leaf failure. Write which failure you are buying protection from. Port count is not the same question.

4 spines and 16 leaves, with each leaf using 4 uplinks because it connects to every spine, and 4 downlinks toward servers in this sketch, is an oversubscription of 4:4 if every downlink and every uplink is the same speed. If the uplinks are faster, recompute. The point of the sketch is that you can say the ratio. A leaf that connects to only one spine is not a leaf-spine fabric. It is a star, and the loss of that spine is the loss of the leaf. Every leaf should see every spine. A server that needs two attachments should see two leaves, not two ports on one leaf, if the question was leaf failure. Write which failure you are buying protection from. Port count is not the same question.

Addressing the fabric

Give each leaf-to-spine link a /31 from a block you reserved for the fabric, in a lab using documentation space only as a drawing. Point-to-point addressing keeps the table honest. A /24 per link wastes space and usually means someone reused a user VLAN plan inside the fabric. The fabric is not a user VLAN.

Loopbacks for the switches come from a separate prefix, one /32 each. Those are the addresses protocols and management should prefer. Management should not depend on a server-facing VLAN. If the servers are on fire, you still want the leaves.

Servers get addressing from the leaf they attach to, or from an overlay if you have chosen one. An overlay is another control plane. Do not add it because a diagram had a purple cloud. Add it when you need mobility or scale the underlay cannot give, and staff it. An overlay without an owner is a second network nobody troubleshoots.

Gateways live where you decided. Anycast gateways on leaves are a common pattern so a server's gateway is local. They require the leaves to agree on the address and on the way they advertise it. Write that agreement down. Two leaves that both answer and do not agree about the prefix will blackhole one direction and pass the other.

Changing a fabric without surprising the servers

Drain a leaf before you work on it. Move the servers or accept the hit, but do not bounce a leaf that still has the only attachment of a storage node. Dual attachments exist so you can drain. If the server has one cable, the fabric's redundancy does not include that server. Say so.

A spine change should be visible as a drop in bandwidth, not as a loss of reachability. Test that in a maintenance window by shutting one spine link and watching the paths move. If reachability dies, the leaf was not dual-attached in the way you thought. Fix the attachment before you call the fabric done.

Upgrades one leaf at a time, then one spine at a time, with the protocol stable between them. A blast upgrade of every spine is a reboot of the data centre. Automation can do the blast. It should not. The automation guide's rule about the first small batch applies twice here.

Document the ratio, the failure you bought, the addressing blocks, and the drain procedure on one page. A fabric diagram without the drain procedure is a poster. The on-call will not derive the procedure from the poster at 2 a.m.

Before the first server moves in

Prove two leaves can talk through either spine. Prove a leaf survives a spine failure. Prove a server with two attachments survives a leaf failure. Prove monitoring sees a link loss as a link loss and not as a silent drop. Then move one unimportant server and watch it for a day.

The first production server is a test, even if the project plan calls it production. Keep its owner on the call. A fabric that passed with test VMs and fails with the real NIC driver is a common story. You want that story on the first server, not on the fiftieth.

Capacity comes next. The ratio you accepted is a promise about simultaneous talkers. If the first application is a backup that does speak at line rate, the ratio was a fantasy. Revisit it before the second rack lands.

Write the spine count, the leaf count, and the oversubscription.
Write the failure a dual attachment is supposed to survive.
Address the links as point-to-point and the switches as loopbacks.
Keep server VLANs off the spine.
Drain and upgrade one device at a time.
Test spine loss and leaf loss before the important servers arrive.
Put the drain steps on the same page as the diagram.

Mistakes that look like a fabric and are not

A leaf with a single spine attachment.

A VLAN stretched across every leaf because it was easy.

An oversubscription ratio nobody can state.

Management that depends on a server VLAN.

An overlay with no owner.

Upgrading every spine in one job.

Calling a single-homed server protected because the fabric has two spines.

A diagram with no drain procedure.

Fabric checks before a server is called protected

Confirm this before you call the change done: Each leaf connects to every spine. The oversubscription ratio is written and uses the real link speeds. Point-to-point links are /31 or another length you chose on purpose. Loopbacks are the management and protocol addresses. A spine failure was tested and reachability held. A leaf failure was tested for a dual-attached server. Single-attached servers are listed as single-attached. The drain steps were used once. Monitoring saw the test failures. No documentation prefix from the lab remains in the production fabric. The overlay, if it exists, has an owner. Upgrades are one device at a time. Server VLANs do not cross the spines. The ratio uses real speeds, not a count of cables. A single-homed storage node is listed before the window. The first server had its owner on the call. Monitoring alerted on the spine test, not only on a ping from a laptop. And the drain procedure was followed, not invented during the test. If one of those fails, stop.

Tools for this guide

These tools run in the browser on this site. They do not log into your machines.

Guides in this cluster

These supporting pages belong to this pillar.

Companion: Cloud: Cloud Networking Guide.