Larger MTU (e.g. jumbo frames) can improve throughput in data center networks. Ensure the full path supports it; otherwise you get fragmentation or black holes. Test with ping and trace; document MTU in runbooks.
What is MTU
- MTU: Maximum Transmission Unit — largest packet size the link or path accepts. Ethernet default is 1500 bytes. Jumbo frames are typically 9000 bytes. Larger MTU means fewer packets per unit of data and less header overhead; can improve throughput for bulk transfer.
- Path MTU: The minimum MTU along the path from source to destination. If you send a packet larger than the path MTU, it may be fragmented (split) or dropped (black hole). End-to-end path must support the same MTU for jumbo to work.
- Where it helps: Inside a data center (server-to-server, server-to-storage) where you control switches and NICs. Often not end-to-end to the internet (internet path is 1500 or less).
Enabling jumbo frames
- NIC and switch: Set MTU 9000 (or provider's value) on the server interface and on every switch in the path. Misconfiguration on one hop causes fragmentation or drops.
- Verify path: Use
ping -M do -s 8972(9000 - 28 byte IP+ICMP header) to test. If you get replies, path supports 9000. If you get no reply or "fragmentation needed," path MTU is smaller. Trace path and fix or reduce MTU. - Document: Record MTU per environment and link in runbooks. New servers or network changes must match. Wrong MTU is a common cause of "mystery" packet loss or slowness.
Fragmentation and black holes
- Fragmentation: When a packet exceeds path MTU, some networks fragment it (split into smaller pieces). Fragmentation adds overhead and can hurt performance; some firewalls or NAT drop fragmented packets. Avoid relying on fragmentation; instead use path MTU discovery (PMTUD) or set MTU to the path minimum.
- Black hole: If a link in the path has smaller MTU and does not send "fragmentation needed" ICMP (e.g. blocked by firewall), large packets are dropped and never reach the destination. Symptoms: small packets work, large (e.g. bulk transfer) fail. Fix: lower MTU or fix PMTUD/ICMP.
Summary
Larger MTU (e.g. jumbo frames) can improve throughput in data center networks. Ensure the full path supports it; otherwise you get fragmentation or black holes. Test with ping and trace; document MTU in runbooks.
