Explore & discover

Helpful Links

Path MTU Discovery on the U-Verse RG


Path MTU Discovery on the U-Verse RG

I broke this discussion out into a separate thread from where the discussion originally started since this is a separate issue.


The discussion originated with a discussion of using a VPN router behind the RG, in the DMZ, and doing IPSec/ESP through the RG and 1) Whether that will work properly or at all, and 2) what problems might arise in this configuration.


I posted the following in that thread:


@SomeJoe7777 wrote:

I can also confirm that IPSec/ESP and IPSec/AH both work properly through the RG when the VPN router is set in the DMZ.  However, there is a caveat: The RG does not properly send ICMP packet-too-big responses back to the VPN router, which breaks PMTU discovery.  You will have to set the IP MTU manually on your IPSec tunnel to avoid fragmentation.  I used an MTU of 1350, and it was working fine.  This was on a Cisco 2811 with a tunnel to a Cisco 2851.



Another user asked the following question:


@ka9q wrote:

Thanks for that information about path MTU discovery. But why would the RG have to generate too-large messages? Is the MTU anything less than 1500?


OK, well the issue is the way PMTU Discovery works.  Firs, remember that MTU = Maximum Transmission Unit, the maximum frame size that a network can handle due to its architecture.  Note that MTU is generally a layer 2 limitation.


In the old days, different networks had different MTU sizes that they could handle, and there was no way for the end stations to know what networks their packets might traverse over, and what the MTU was of each of these networks.  If a station sent a 1500 byte IP packet, but there was a link between the two stations that had a 576 byte limit, the IP packet would be fragmented into three layer 2 frames to be transported across that network, and would then get put back together.


There are several problems with fragmentation.  First, it uses a lot of CPU/memory resources on the routers that have to fragment and reassemble.  On the receiving router, fragments have to be held in memory until all the pieces have been received before they can be reassembled.  Second, if the link is dropping a small percentage of frames, it drastically increases the likelihood that a given IP packet will have to be retransmitted by TCP, because if one fragment is lost, all the fragments must be resent.


To avoid fragmentation, you used to have to be able to see all the network segments (or know their MTU), and manually adjust your MTU on your computer downward until the packet sizes your computer was sending out were small enough so that they wouldn't get fragmented.


Starting with Windows 95, a new mechanism to control IP fragmentation was introduced called path MTU discovery, or PMTUD.  What the end station does is simply send all IP packets out with the DF (Don't Fragment) bit in the IP header set.  If this packet hits a router where the network it needs to forward the packet to has an MTU smaller than the packet size (which would require fragmentation), the router is supposed to drop the packet, and send an ICMP "Packet Too Big" message back to the computer.


PMTUD takes advantage of that, because if it receives an ICMP Packet-Too-Big message, it lowers the IP datagram size and sends it out again.  Eventually the packet will make it through all the way to the target station without being fragmented, and the computer just remembers the correct IP datagram size for that TCP session.


MTU issues show up a lot in IPSec implementations because of the repeated encapsulations for all the carrier protocols.  For example, on an IPSec/ESP link coupled with GRE, a source IP packet is encapsulated in GRE, which is then encapsulated in IPSec/ESP.  Each encapsulation reduces the available MTU to the computer.


The issue with the U-Verse RG is that it seems to ignore the DF bit, and instead of dropping it and sending ICMP Packet-Too-Big, it just goes ahead and fragments the packet anyway.  This obviously breaks PMTUD, requiring you to manually set it on your router.


For a Cisco IPSec/ESP with GRE tunnel, the manually set MTU should be 1438, but I typically set it a little lower to avoid any unexpected issues, say to 1400.


The best document I've ever seen that explains these issues is Resolve IP Fragmentation, MTU, MSS, and PMTUD Issues with GRE and IPSEC on Cisco's site.


Message 1 of 19

Re: Path MTU Discovery on the U-Verse RG

Ow! My brain hurts!


I guess all that address translation can work (if Cisco says it can, they're probably right) but what a mess! Isn't it easier to use globally unique addresses that never have to be translated? You never have to worry about "NAT unfriendly" application protocols that exchange IP addresses, you can use new transport protocols other than UDP and TCP without any prior arrangement, and so on.


I'd like to think that the pain level associated with all this NAT stuff will eventually get so intense that people will simply decide that they've had enough and start using IPv6. But I've often underestimated the endurance of the average human, especially those who simply don't know that there exist better ways.


Message 16 of 19

Re: Path MTU Discovery on the U-Verse RG

@ka9q wrote:


I'd like to think that the pain level associated with all this NAT stuff will eventually get so intense that people will simply decide that they've had enough and start using IPv6. But I've often underestimated the endurance of the average human, especially those who simply don't know that there exist better ways.


I used to believe that as well, but I'm not so sure now.  A few things have happened in the last 10 years or so since 1998 when NAT became popular.  Remember the WebRamp? :smileyhappy:


  1. Nowdays we use fewer protocols that have problems with NAT.  Not much use of FTP anymore since a web application that supports upload/download is pretty easy to do.  Who uses H.323 applications anymore with all the features of it built into IM clients?
  2. NAT helper code in the routers has gotten really sophisticated -- even most of the VPN flavors work properly over NAT now.
  3. Configuring NAT on a Cisco, while not straightforward, is familiar to anyone who configures Ciscos, and extremely flexible.  You can do multiple different kinds of NAT between all the different networks connected to the router simultaneously.


I configure NAT on Ciscos from time to time, and it's not that much of a problem, and works just fine.  My company does a world conference every year at a resort hotel, and when we go there, I hook up a Cisco to do internet access and a GRE/IPSec VPN tunnel back to the headquarters.  I use NAT on that connection and can run the VPN, VOIP, Internet, streaming video, and things like SQL replication and Terminal Server sessions all at the same time, and I only need 1 static IP from the hotel. :smileyhappy:


Consumer routers can't do that, and frequently have problems in their NAT implementations.  But a seasoned product like a Cisco 2811 running IOS 12.4 is far more powerful and flexible than that.


Message 17 of 19

Re: Path MTU Discovery on the U-Verse RG

[quote]Nowdays we use fewer protocols that have problems with NAT.[/quote]

And why is that? I'd say it's because a lot of new and novel protocols that couldn't run over NATs have died on the vine before anyone was able to benefit from them.


NAT has a huge opportunity cost. It's not that we haven't managed to work around its limitations for most of the things we do every day, but isn't the Internet supposed to encourage new and innovative applications that weren't thought of by the original designers of the Internet protocols?


Funny you should mention VoIP. A while ago I bought a pile of VoIP phones and adapters, with the intent of learning about the protocols by doing and with a vague plan to set up a private telephone network for my family. It was easy enough to get them talking to each other without NATs in the way, but I eventually gave up trying to make them work properly through NATs. So the pile sits unused in a closet. (It doesn't really help that everybody now has cheap flat-rate long distance plans that make a private VoIP family network somewhat less nifty than it would have been.)


I've been working with TCP/IP since about 1984. I am one of its first implementers. If *I* can't figure out how to make this stuff work, then something is seriously wrong somewhere.


SIP is a lot more complicated than it ought to be, but at least it works in a fairly straightforward fashion when every phone has a public address. I just couldn't figure out how to get it to work through a Linux-based NAT; whatever application helper existed at that time simply wasn't up to it. That's why I'm so surprised with the VoIP guys; you'd think that they would be the ones clamoring the loudest for IPv6, yet few VoIP devices even support it.


IMHO, the easiest way to retire NAT is to encourage the commodity "router" vendors (D-Link, Netgear, Linksys) to implement 6to4 tunneling. Apple already does it in their newer Airport base stations. You just check a box and voila - assuming the base station has a public IPv4 address, and you haven't disabled IPv6 on your computers, IPv6 addresses automatically appear on their interfaces. As it was back in the good old days, it is once again possible for the average computer -- not a server in a co-lo rack, but a computer in someone's home -- to communicate directly, as client or server, using any port or set of ports, or even any transport protocol they choose, with any other similarly equipped computer, without any address translation or port forwarding hacks. It Just Works. How much easier can it get than that?


Message 18 of 19

Re: Path MTU Discovery on the U-Verse RG

I've always had the Christopher Columbus MTU (1492), never changes. :smileysad:
Message 19 of 19
Share this topic
Share this topic
Additional Support