Data Center Networking Guide

This is an original planning guide. It is not a vendor manual, an official standard, or legal advice. Example addresses use documentation ranges. Review every command before you paste it.

A data-centre network is a fabric plus a policy

Leaf-spine is a way to give every leaf the same number of hops to every other leaf, through more than one spine, so one spine failure is a reduction in bandwidth and not an outage. It is not a product name. You can build a poor leaf-spine with no redundancy in the attachments, and you can build a fine smaller network that is not leaf-spine at all. Choose it when east-west traffic and a predictable failure mode are the requirements.

Oversubscription is downlink bandwidth divided by uplink bandwidth. It is a capacity fact. A leaf with more server bandwidth than spine bandwidth will drop traffic when the servers actually speak. Many designs accept that, because servers do not all speak at once. Write the ratio you accept. The capacity guide is how you defend it. A ratio you cannot say out loud is a ratio you have not chosen.

Routing in the fabric is usually a small, fast protocol with equal paths, or a design that uses the same idea under another name. Every link should be a routed link with a prefix you planned, not a giant VLAN stretched across the room. A stretched VLAN feels simple and fails as one broadcast domain. Keep the VLAN at the leaf, at the edge of the server, if the server needs one at all.

This page does not design your storage network for you and it does not quote a vendor fabric guide. The calculators linked here estimate ports and ratios from numbers you type. They do not log into a switch. The ticket still needs the failure you are protecting against.

Three sketches and their ratios

2 spines and 4 leaves, with each leaf using 2 uplinks because it connects to every spine, and 4 downlinks toward servers in this sketch, is an oversubscription of 4:2 if every downlink and every uplink is the same speed. If the uplinks are faster, recompute. The point of the sketch is that you can say the ratio. A leaf that connects to only one spine is not a leaf-spine fabric. It is a star, and the loss of that spine is the loss of the leaf. Every leaf should see every spine. A server that needs two attachments should see two leaves, not two ports on one leaf, if the question was leaf failure. Write which failure you are buying protection from. Port count is not the same question.

4 spines and 8 leaves, with each leaf using 4 uplinks because it connects to every spine, and 8 downlinks toward servers in this sketch, is an oversubscription of 8:4 if every downlink and every uplink is the same speed. If the uplinks are faster, recompute. The point of the sketch is that you can say the ratio. A leaf that connects to only one spine is not a leaf-spine fabric. It is a star, and the loss of that spine is the loss of the leaf. Every leaf should see every spine. A server that needs two attachments should see two leaves, not two ports on one leaf, if the question was leaf failure. Write which failure you are buying protection from. Port count is not the same question.

4 spines and 16 leaves, with each leaf using 4 uplinks because it connects to every spine, and 4 downlinks toward servers in this sketch, is an oversubscription of 4:4 if every downlink and every uplink is the same speed. If the uplinks are faster, recompute. The point of the sketch is that you can say the ratio. A leaf that connects to only one spine is not a leaf-spine fabric. It is a star, and the loss of that spine is the loss of the leaf. Every leaf should see every spine. A server that needs two attachments should see two leaves, not two ports on one leaf, if the question was leaf failure. Write which failure you are buying protection from. Port count is not the same question.

Addressing the fabric

Give each leaf-to-spine link a /31 from a block you reserved for the fabric, in a lab using documentation space only as a drawing. Point-to-point addressing keeps the table honest. A /24 per link wastes space and usually means someone reused a user VLAN plan inside the fabric. The fabric is not a user VLAN.

Loopbacks for the switches come from a separate prefix, one /32 each. Those are the addresses protocols and management should prefer. Management should not depend on a server-facing VLAN. If the servers are on fire, you still want the leaves.

Servers get addressing from the leaf they attach to, or from an overlay if you have chosen one. An overlay is another control plane. Do not add it because a diagram had a purple cloud. Add it when you need mobility or scale the underlay cannot give, and staff it. An overlay without an owner is a second network nobody troubleshoots.

Gateways live where you decided. Anycast gateways on leaves are a common pattern so a server's gateway is local. They require the leaves to agree on the address and on the way they advertise it. Write that agreement down. Two leaves that both answer and do not agree about the prefix will blackhole one direction and pass the other.

Changing a fabric without surprising the servers

Drain a leaf before you work on it. Move the servers or accept the hit, but do not bounce a leaf that still has the only attachment of a storage node. Dual attachments exist so you can drain. If the server has one cable, the fabric's redundancy does not include that server. Say so.

A spine change should be visible as a drop in bandwidth, not as a loss of reachability. Test that in a maintenance window by shutting one spine link and watching the paths move. If reachability dies, the leaf was not dual-attached in the way you thought. Fix the attachment before you call the fabric done.

Upgrades one leaf at a time, then one spine at a time, with the protocol stable between them. A blast upgrade of every spine is a reboot of the data centre. Automation can do the blast. It should not. The automation guide's rule about the first small batch applies twice here.

Document the ratio, the failure you bought, the addressing blocks, and the drain procedure on one page. A fabric diagram without the drain procedure is a poster. The on-call will not derive the procedure from the poster at 2 a.m.

Before the first server moves in

Prove two leaves can talk through either spine. Prove a leaf survives a spine failure. Prove a server with two attachments survives a leaf failure. Prove monitoring sees a link loss as a link loss and not as a silent drop. Then move one unimportant server and watch it for a day.

The first production server is a test, even if the project plan calls it production. Keep its owner on the call. A fabric that passed with test VMs and fails with the real NIC driver is a common story. You want that story on the first server, not on the fiftieth.

Capacity comes next. The ratio you accepted is a promise about simultaneous talkers. If the first application is a backup that does speak at line rate, the ratio was a fantasy. Revisit it before the second rack lands.

Write the spine count, the leaf count, and the oversubscription.
Write the failure a dual attachment is supposed to survive.
Address the links as point-to-point and the switches as loopbacks.
Keep server VLANs off the spine.
Drain and upgrade one device at a time.
Test spine loss and leaf loss before the important servers arrive.
Put the drain steps on the same page as the diagram.

Mistakes that look like a fabric and are not

A leaf with a single spine attachment.

A VLAN stretched across every leaf because it was easy.

An oversubscription ratio nobody can state.

Management that depends on a server VLAN.

An overlay with no owner.

Upgrading every spine in one job.

Calling a single-homed server protected because the fabric has two spines.

A diagram with no drain procedure.

Fabric checks before a server is called protected

Check 1 in Fabric checks before a server is called protected. Each leaf connects to every spine. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 2 in Fabric checks before a server is called protected. The oversubscription ratio is written and uses the real link speeds. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 3 in Fabric checks before a server is called protected. Point-to-point links are /31 or another length you chose on purpose. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 4 in Fabric checks before a server is called protected. Loopbacks are the management and protocol addresses. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 5 in Fabric checks before a server is called protected. A spine failure was tested and reachability held. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 6 in Fabric checks before a server is called protected. A leaf failure was tested for a dual-attached server. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 7 in Fabric checks before a server is called protected. Single-attached servers are listed as single-attached. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 8 in Fabric checks before a server is called protected. The drain steps were used once. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 9 in Fabric checks before a server is called protected. Monitoring saw the test failures. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 10 in Fabric checks before a server is called protected. No documentation prefix from the lab remains in the production fabric. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 11 in Fabric checks before a server is called protected. The overlay, if it exists, has an owner. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 12 in Fabric checks before a server is called protected. Upgrades are one device at a time. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 13 in Fabric checks before a server is called protected. Server VLANs do not cross the spines. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 14 in Fabric checks before a server is called protected. The ratio uses real speeds, not a count of cables. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 15 in Fabric checks before a server is called protected. A single-homed storage node is listed before the window. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 16 in Fabric checks before a server is called protected. The first server had its owner on the call. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 17 in Fabric checks before a server is called protected. Monitoring alerted on the spine test, not only on a ping from a laptop. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Check 18 in Fabric checks before a server is called protected. The drain procedure was followed, not invented during the test. Record the device name, the result, and the time on the same line. If the result does not match this check, stop and fix the ticket before you add another change. The rollback for this check is the previous value you wrote down, not a reboot. A reboot hides whether the check was the one that mattered. When the check passes, move to the next one. When it fails, the incident is this check.

Tools for this guide

These tools run in the browser on this site. They do not log into your machines.

Guides in this cluster

These supporting pages belong to this pillar. A published page is linked. The others are named until they are published.

The content plan also names a companion in another desk: Cloud: Cloud Networking Guide.