Explore & discover

Helpful Links

Impacted by the government shutdown? AT&T can help.  Learn more

Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

Teacher

Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

I have a Pace 5268AC and have my own router behind it using DMZplus. The firmwware is versions are:

 

Hardware Version260-2173300
Software Version10.5.3.527171-att

 

I can't get any devices to sync time using NTP. This includes laptops, desktops, Raspberry Pis and other devices. This is causing me no end of issues as with devices so far out of sync SSL certs fail checks, services on Linux boxes hang etc.

 

I have searched everywhere and found references to others having issues but haven't found any soltuions. I have tried factory resetting the 5268AC and tried adjusting the "Strict UDP Session Control" setting in the Advanced firewall tab.

 

I find it hard to accept that no gigapower users are allowed to sync time, am I missing something? Is this a bug in the 5268AC? Any help would be very much appreciated.

 

Thanks!

6,535 Views
Message 1 of 67
Contributor

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

nettexan,

 

Thanks for posting the issue and solution. I had been struggling with the exact same issue and did not know how to fix it.  I have a Ubiquiti USG behind a PACE 5268AC in DMZPlus mode and can NOT get the USG to upgrade.  I've had similar issues with some of the other Ubiquiti devices behind the PACE but they ultimately all upgraded.  The USG will not.

 

Your recommendation of having the USG mask the traffic before it leaves the networks seems like a perfect plan.  However, I'm a total newbie and don't know how to do that. Any chance you can walk me through it?

 

Seems nutty for AT&T to broadly block the traffic.... 

Message 31 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

Good news @onlinedolhare, your issues are linked and by fixing one you will likely fix both!

 

The reason your USG won't update is because it can't sync time and so the SSL connection to the Ubiquiti servers errors out (time has to be semi correct for SSL to work).

 

The fix you are looking for is here. You'll need to create a config.gateway.json file on your UniFi controller. I am assuming you are on a USG and not a USG Pro 4, if so when you paste in the config from the fix make sure you adjust the "outbound-interface" option to whatever your WAN port is (pretty sure it is eth0 on the USG). Then trigger a reprovision of the USG in the controller and you should be able to sync time and upgrade your USG. If this is s bit beyond your experience level I would suggest chatting with Ubiquiti via their online chat or hitting up their forums, folks there are pretty friendly.

 

Your other (easier?) option is to call AT&T and have them replace your Pace with a gateway that doesn't have this strange issue, such as the BGW210.

Message 32 of 67
Contributor

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@nettexan that is super helpful but unfortunately I will not get into a manual configuration, I will end up messing it up for sure! I was hoping there was a GUI way to mask the port and fix the issue.  I'll either have to ask (or pay) somebody to do it or maybe I can ask for a different router but AT&T told me before the PACE one (which I hate!) was the only one they would provide. Thanks again for the help!

Message 33 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

No worries @onlinedolhare, totally understand your concern.


I would try calling AT&T back again, or even calling the tech that installed for you if you still have their contact info. Raise heck with them, talk to a supervisor etc. Smiley Happy

 

I wouldn't recommend paying someone, just risky IMO and what if you need to make an adjustment in the future? The USG firmware is rapidly evolving and the ability to use it to do outbound masquerading is on the roadmap for the future from what I understand. Just no idea when. :/

 

Best of luck!

Message 34 of 67
ACE - Expert

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

AT&T distinctly says that it blocks UDP with a source port of 123 in U-verse Internet traffic.  Unless your NTP traffic is inside a tunnel or has been through PAT, it will be blocked in the AT&T network, not just at the Gateway by a certain model of gateway.  NTP works fine for me because it's all going through PAT.

 

See http://about.att.com/sites/broadband/network

 

(It doesn't go into detail here, but I've seen source port blocking specified elsewhere).

 

FOOTNOTE: Why it suddenly stops working when you turn on DMZplus mode is that you disabled NAT/PAT for the device when you turned the mode on.

 

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 35 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

Hey @JefferMC,

 

I am aware of the page you linked to. I am not sure what to tell you except that the BGW210 absolutely does not block UDP with a source of 123 with a router behind it in ip-passthrough, while the PACE does (although it is called DMZ+ on the PACE). I have tested this first hand and am currently running in this configuration (BGW210 in ip-passthrough) and source UDP traffic on 123 is not blocked and all works well.

 

The PACE blocking this traffic only when in DMZ+ has been confirmed by others earlier in this thread as well, it passes the traffic if you are directly connected. So while the stated policy of AT&T is that they block source UDP on 123 it is certainly done by the gateway device and only in certain configs on certain models.

 

This is extremely poor practice as this breaks all sorts of IoT devices causing real issues for folks, again look earlier in the thread I think we even had someone from the NTP project comment on this.

Message 36 of 67
ACE - Expert

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?


@nettexan wrote:

 

This is extremely poor practice as this breaks all sorts of IoT devices causing real issues for folks, again look earlier in the thread I think we even had someone from the NTP project comment on this.


It should only affect an IoT device if it is the directly-connected DMZplus device.  Any IoT device passing through a router designated as a DMZplus device would not be affected (assuming that the router, as it must, performs PAT on the traffic from the IoT device).

 

As far as I can tell, the device being discussed at the end of this thread is the DMZplus target device.

 

AT&T is doing this blocking due to the exploitation of the protocol by nefarious elements.  They're not alone.  

 

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 37 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

Yes the device being discussed (Ubiquiti USG) is the DMZplus device, however as that particular device doesn't do PAT by default all devices behind it will also not be able to sync time. Your assumption that all routers that would be the DMZplus device perform PAT is a poor assumption... many do not as is clear by the number of folks who have had issues here.

 

The NTP reflector attack was valid years ago, it isn't an issue today (at least as far as I have seen). AT&T actually is, from what I can tell, the only ISP filtering like this. If there are other examples I would be interested to know what they are, however after searching extensively on this issue when I first encountered it I can tell you none of the other national ISP competitors do this.

 

While this is a good discussion I think we are at an impasse and need to agree to disagree. IMO this is a very bad thing to do to customers, many of whom are going to want to run their own router as a DMZplus target (for better WiFi etc) and won't have the technical depth to even understand why all of a sudden most of their devices start showing incorrect time and in many cases stop working entirely. As this is not an issue on the non PACE gateways they are at a minimum inconsistently applying this policy which again contributes to customers being confused and upset.

Message 38 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@JefferMC

Just to add to the conversation, I can confirm the same behavior on the PACE in DMZ+ mode. On the PACE in DMZ+ mode, packets with source port 123 are blocked. With the NVG599 in IP-Passthrough mode, the packets are not blocked. Some routers will mangle the packet to change the source port. For example, pfSense does this. It assigns a random source port on outgoing packets. However, many other routers do not do this by default and thus all NTP request packets die on the PACE in DMZ+ mode. For example, the router mentioned earlier in this thread Ubiquiti Edgerouter does not and you have to tell it to mangle the packet and rewrite the source port from 123 to something else.

In fact, per IETF RFC 958, the PACE device in DMZ+ mode is breaking the NTP specification (symmetric mode, which many IoT devices use):

      The mode assumed by a peer can be determined by inspection of the
      UDP Source Port and Destination Port fields (see Appendix A).  If
      both of these fields contain the NTP service-port number 123, the
      peer is operating in symmetric mode.  If they are different and
      the Destination Port field contains 123, this is a client request
      and the receiver is expected to reply in the manner described
      above.  If they are different and the Source Port field contains
      123, this is a server reply to a previously sent client request.



https://tools.ietf.org/html/rfc958

Message 39 of 67
ACE - Expert

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

IMHO, the NTP spec is flawed in that regard (i.e. mandating a source port, instead of allowing an ephemeral one), which made it a candidate for exploitation for DDoS.  PAT normally comes along with NAT and gets you around the AT&T filter.

PAT has its own RFCs (for UDP recommendations WRT NAT, please see RFC 4787).  And the AT&T Gateway is compliant with this as far as I know.  However, when you turn on DMZ+ mode, the AT&T Gateway would not be doing what you call "mangl[ing] the packet to change the source port" (and I call PAT, because that's what most people call it), because it would pass through what the DMZ+ mode device handed it, but instead your device behind the Gateway would be doing the "mangling" a/k/a PAT.

 

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 40 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@JefferMC

 

I use packet mangling as that's Linux's iptables term for it, which is how I cut my teeth on NAT and firewalls.

 

Finally, I would like to point out that the Motorola NVG599 (switched to this after PACE) in IP-Passthrough does not block source port 123/udp. As other posters pointed out, this is quite clearly a PACE DMZ+ mode only problem, but you keep saying it's blocked on the network. We are asserting, in our experience with other AT&T equipment, this is not true.

 

Anyway, it sure would be great to have some acknowledgement over this issue. 40 messages in this thread and some workarounds, but still most fingers pointing to an issue within the PACE equipment. If not for my own benefit (I'm fine with the NVG599), the benefit of others who are still battling this issue and not understanding why their NTP is not working.

Message 41 of 67

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

I'm fascinated by the intellectual wrangling, kinda like a bonus Big Bang episode...so for the non-network engineer if I have Port Forwarding configurations capabilities on my ASUS router is there a configuration I can setup which will "mangle" the NTP request allowing them to pass through the Pace 5268AC in DMZplus mode? I'm not sure if PAT (port address translation) and ASUS' port forwarding are the same thing or not. Since we are talking about numerous consumer grade routers out here it would be nice to have a practical solution. It's been very frustrated that this needed feature is being blocked on my network. I use a manual work around with a VPN but that is a conscious sync where I'd like a passive solution. This thread discussion lighting up my inbox is like picking a scab, it's been years and still haven't seen a workable solution for the masses.

Message 42 of 67
ACE - Expert

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

Just so I'm clear:

You are sending UDP protocol packet, with a source port of 123, directly from the device that is the target of IP Passthrough from the NVG 599 (not from a device on the other side of that target device), through the NVG 599, and you can confirm that it is reaching the NTP server with a source port of 123?

 

 

 

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 43 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

This is extremely poor practice as this breaks all sorts of IoT devices causing real issues for folks, again look earlier in the thread I think we even had someone from the NTP project comment on this.

I'm not from the NTP project.  I have maintained servers in the NTP Pool for more than a decade and have a decent understanding of the protocol.  As I said, this completely breaks the reference implementation of NTP and is a bone-headed move on AT&T's part.  I discovered this issue when I switched to uVerse VDSL from Cox and realized my NTP server was no longer able to communicate with its peers.

 

Your assumption that all routers that would be the DMZplus device perform PAT is a poor assumption

Linux based NAT routers (including the Ubiquiti under discussion) never mangle port mappings unless:

 

  1. Specifically told to do so via the --random declaration to the appropriate iptables rule(s).

  2. Port mangling is necessary to uniquely identify connections, in the case of two or more NAT hosts speaking to the same IP endpoint with the same ports.

    This scenario should be pretty rare these days but would have happened more frequently in bygone eras.  For example, Windows XP sequentially used 1024 to 5000 for source/ephemeral ports, so if two XP hosts came online at roughly the same time and established a connection to the same IP through NAT.....

IMHO, the NTP spec is flawed in that regard (i.e. mandating a source port, instead of allowing an ephemeral one)

NTP is a two way protocol.  Servers can sync each other running in symmetric mode, with either side initiating communication.  Clients are not mandated to use 123 as a source port but blocking that port as AT&T does denies one the ability to use symmetric mode, as well as clients (the reference implementation!) that use 123 even when not in symmetric mode.

 

which made it a candidate for exploitation for DDoS

The DDoS was a reflection/amplification attack that was only possible with servers that were poorly configured.  The "bug" -- if you can call it that; is setting your password to "password" a bug or your own stupidity? -- has been fixed for years now.  I can't think of another ISP that does this.  None of the ones I've ever used did at least.  Not Verizon, Time Warner, or Cox.  I used to keep my home server in the pool on those ISPs, it was an easy way to monitor the stablity of my connection, since the pool monitor will quickly detect the effects of jitter or a downed connection.

 

PAT normally comes along with NAT 

Nope.  Mandating PAT can break a lot of things.  UDP hole punching is at the very least complicated by PAT -- sometimes it's made impossible -- and is used by mainstream applications like Facetime.  Facetime will still work if UDP hole punching fails, but the call then has to be routed through Apple, rather than peer-to-peer, and quality suffers.

 

Message 44 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

I'm fascinated by the intellectual wrangling, kinda like a bonus Big Bang episode...so for the non-network engineer if I have Port Forwarding configurations capabilities on my ASUS router is there a configuration I can setup which will "mangle" the NTP request allowing them to pass through the Pace 5268AC in DMZplus mode? I'm not sure if PAT (port address translation) and ASUS' port forwarding are the same thing or not.

I can't directly answer your question.  If it's based on Linux and if you can get access to the command line (two very big ifs) some variation on the iptables nat rule I posted earlier would solve your problem.

 

FWIW, I just purchased a EdgeRouter X for myself.  My understanding is that Ubiquiti provides a very powerful CLI and direct access to the iptables rules for those inclined to mess with them.  The X can be had for less than $60 on Amazon and is light-years ahead of any consumer grade router.  Does't come with wireless, it's a pure router, not a consumer gateway, so you'd need a separate AP, but I've always been a proponent of that anyway.  All-in-one devices invariably come with compromises..... Smiley Sad

Message 45 of 67
Share this topic
Share this topic
Announcements...
Are you having trouble logging in? Is your email password not working? Let us show you how to Reset your Email Password using myAT&T!
Additional Support