Explore & discover

Helpful Links

Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

Announcements...
Are you having trouble logging in? Is your email password not working? Let us show you how to Reset your Email Password using myAT&T!
Teacher

Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

I have a Pace 5268AC and have my own router behind it using DMZplus. The firmwware is versions are:

 

Hardware Version260-2173300
Software Version10.5.3.527171-att

 

I can't get any devices to sync time using NTP. This includes laptops, desktops, Raspberry Pis and other devices. This is causing me no end of issues as with devices so far out of sync SSL certs fail checks, services on Linux boxes hang etc.

 

I have searched everywhere and found references to others having issues but haven't found any soltuions. I have tried factory resetting the 5268AC and tried adjusting the "Strict UDP Session Control" setting in the Advanced firewall tab.

 

I find it hard to accept that no gigapower users are allowed to sync time, am I missing something? Is this a bug in the 5268AC? Any help would be very much appreciated.

 

Thanks!

6,857 Views
Message 1 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@JefferMC

 

Here's a tcpdump of the external interface of my router (IP-Passthrough on NVG599). The first series I told my router to use source port 33333, the next, I let the ntp client use source port 123. As you can see, all successful:

 

[2.4.2-RELEASE][root@xxx]/root: tcpdump -i em0 -n port 123
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes

----Force mapped source UDP source port to 33333----
18:07:49.285999 IP 45.xx.xx.xx.33333 > 132.163.96.3.123: NTPv4, Client, length 48
18:07:49.336420 IP 132.163.96.3.123 > 45.xx.xx.xx.33333: NTPv3, Server, length 48
18:07:51.285903 IP 45.xx.xx.xx.33333 > 132.163.96.3.123: NTPv4, Client, length 48
18:07:51.335337 IP 132.163.96.3.123 > 45.xx.xx.xx.33333: NTPv3, Server, length 48
18:07:53.286066 IP 45.xx.xx.xx.33333 > 132.163.96.3.123: NTPv4, Client, length 48
18:07:53.335534 IP 132.163.96.3.123 > 45.xx.xx.xx.33333: NTPv3, Server, length 48
18:07:55.286106 IP 45.xx.xx.xx.33333 > 132.163.96.3.123: NTPv4, Client, length 48
18:07:55.335697 IP 132.163.96.3.123 > 45.xx.xx.xx.33333: NTPv3, Server, length 48
18:08:01.597189 IP 45.xx.xx.xx.33333 > 69.164.202.202.123: NTPv4, Client, length 48
18:08:01.633008 IP 69.164.202.202.123 > 45.xx.xx.xx.33333: NTPv4, Server, length 48

----Static mapped source UDP source port to 123----
18:08:29.380840 IP 45.xx.xx.xx.123 > 128.138.141.172.123: NTPv4, Client, length 48
18:08:29.430894 IP 128.138.141.172.123 > 45.xx.xx.xx.123: NTPv3, Server, length 48
18:08:30.596804 IP 45.xx.xx.xx.123 > 208.75.89.4.123: NTPv4, Client, length 48
18:08:30.664541 IP 208.75.89.4.123 > 45.xx.xx.xx.123: NTPv4, Server, length 48
18:08:31.380741 IP 45.xx.xx.xx.123 > 128.138.141.172.123: NTPv4, Client, length 48
18:08:31.432940 IP 128.138.141.172.123 > 45.xx.xx.xx.123: NTPv3, Server, length 48
18:08:33.380747 IP 45.xx.xx.xx.123 > 128.138.141.172.123: NTPv4, Client, length 48
18:08:33.429984 IP 128.138.141.172.123 > 45.xx.xx.xx.123: NTPv3, Server, length 48
18:08:35.380942 IP 45.xx.xx.xx.123 > 128.138.141.172.123: NTPv4, Client, length 48
18:08:35.430785 IP 128.138.141.172.123 > 45.xx.xx.xx.123: NTPv3, Server, length 48
18:08:38.608643 IP 45.xx.xx.xx.123 > 45.76.244.193.123: NTPv4, Client, length 48
18:08:38.684915 IP 45.76.244.193.123 > 45.xx.xx.xx.123: NTPv4, Server, length 48

 

Message 46 of 67
ACE - Expert

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?


@Javerial wrote:

Linux based NAT routers (including the Ubiquiti under discussion) never mangle port mappings unless:

 

  1. Specifically told to do so via the --random declaration to the appropriate iptables rule(s).

  2. Port mangling is necessary to uniquely identify connections, in the case of two or more NAT hosts speaking to the same IP endpoint with the same ports.

    This scenario should be pretty rare these days but would have happened more frequently in bygone eras.  For example, Windows XP sequentially used 1024 to 5000 for source/ephemeral ports, so if two XP hosts came online at roughly the same time and established a connection to the same IP through NAT.....

 

PAT normally comes along with NAT 

Nope.  Mandating PAT can break a lot of things.  UDP hole punching is at the very least complicated by PAT -- sometimes it's made impossible -- and is used by mainstream applications like Facetime.  Facetime will still work if UDP hole punching fails, but the call then has to be routed through Apple, rather than peer-to-peer, and quality suffers.

 


Your two statements conflict.  You just gave an example of when PAT happens in Linux.  I gave the applicable RFC for UDP NAT/PAT, which addresses PAT usage.  

 


@Javerial wrote:

IMHO, the NTP spec is flawed in that regard (i.e. mandating a source port, instead of allowing an ephemeral one)

NTP is a two way protocol.  Servers can sync each other running in symmetric mode, with either side initiating communication.  Clients are not mandated to use 123 as a source port but blocking that port as AT&T does denies one the ability to use symmetric mode, as well as clients (the reference implementation!) that use 123 even when not in symmetric mode.

 

Having it be bi-directional does not require that the sender use its listening port as its source port outbound.  Nothing wrong with using a WKP for listening (123) and a empheral port for sending. 

 


@Javerial wrote:

which made it a candidate for exploitation for DDoS

The DDoS was a reflection/amplification attack that was only possible with servers that were poorly configured.  

 


Most DDoS attacks are the result of poor configuration.

Award for Community Excellence 2019 Achiever*
*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 47 of 67
Tutor

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

I would like to point you back to post number 18 in this thread - yes, 30 posts ago in the ridiculous thread.

 

I had an NVG589 in DMZ+ (or ip-pass-through), with all my own equipment behind it (router, switch, WiFi access points), all working fine.  I upgraded to Gigapower and AT&T switched the NVG589 for the Pace 5268AC - after this, my router could no longer receive NTP and all my logging times became inaccurate.  This is also when other devices started complaining about NTP issues.

 

I'm starting to think the Pace 5268AC device is just a piece of junk.  I'm a contributor to another thread about how you cannot fully turn off the radios on the Pace device: (https://forums.att.com/t5/AT-T-Fiber-Equipment/POSSIBLE-VULNERABILITY-Pace-5268AC-Leaves-Wifi-Radio-...

The same sort of thing as this thread, lots of theories and input, even from AT&T, still no action or fixes.

 

On another note, I lose telephone service during power outages with this Pace 5268AC.  My phones are connected to the RJ11 port on the Pace 5268AC.  My phone system is on a UPS.  I have the Pace 5268AC on a UPS, The AT&T ONT in my garage, it has a working backup battery in it.  On top of that, I have tried having the ONT also plugged into a UPS device.  And yet, if I lose power I still lose phone service.  How on earth does the Pace device even know I've lost power since everything is on UPS??    But then, Pace sells their own internal backup battery for the Pace 5268AC, so I finally broke down and bought this overpriced battery that I should not need... and now I don't lose phone during a power outage.

 

The Pace 5268AC is full of problems... and regardless of what AT&T claims they are blocking on their network, I and several others have given information showing other devices that AT&T supplies do not have these issues.

 

So, those of you who have mentioned the NVG599 or the BGW210, do you have AT&T Fiber Gigapower service?  If one of those devices will still give me Gigapower service I'll be on the phone with AT&T in a heartbeat.

 

Thanks!

Message 48 of 67
ACE - Expert

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?


@stealth2108 wrote:

@JefferMC

 

Here's a tcpdump of the external interface of my router (IP-Passthrough on NVG599). The first series I told my router to use source port 33333, the next, I let the ntp client use source port 123. As you can see, all successful:

 

...

----Force mapped source UDP source port to 33333----
18:07:49.285999 IP 45.xx.xx.xx.33333 > 132.163.96.3.123: NTPv4, Client, length 48
18:07:49.336420 IP 132.163.96.3.123 > 45.xx.xx.xx.33333: NTPv3, Server, length 48
...

----Static mapped source UDP source port to 123----
18:08:29.380840 IP 45.xx.xx.xx.123 > 128.138.141.172.123: NTPv4, Client, length 48
18:08:29.430894 IP 128.138.141.172.123 > 45.xx.xx.xx.123: NTPv3, Server, length 48
...

 


That's pretty clear and compelling.  And you have residential (vs. Business-class) service?

 

Award for Community Excellence 2019 Achiever*
*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 49 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

Having it be bi-directional does not require that the sender use its listening port as its source port outbound.  Nothing wrong with using a WKP for listening (123) and a empheral port for sending. 

Yes it does.  It is a server-server relationship, not client-server.  Either side can send the first packet in an exchange.  In established relationships packets are sent sparingly, every 1,024 seconds with the reference implementation's default configuration.  What NAT device do you know of that keeps a UDP entry in the table when no traffic has been exchanged for that length of time?

 

You are asking for a total redesign of one of the oldest and most important protocols on the Internet -- NTP goes back to 1985 -- because of a non-issue that was fixed years ago.

 

I repeat, this was a stupid move on AT&T's part.  If you want my guess, this was the genius idea of some off-shore engineer that learned everything he knows from the Cisco and/or Microsoft study guides, saw an article about an amplification attack, and decided to block NTP without knowing anything about the protocol or the attack.

 

 

Message 50 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

Seems that the 5268AC doesn't block it on IPv6.  Disabled my mangle rule on a hunch:

 

pi@gw:~ $ sudo tcpdump -i eth0.15 -n udp port 123 and ip6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0.15, link-type EN10MB (Ethernet), capture size 262144 bytes
22:06:00.049342 IP6 2600:1702:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx.123 > 2001:4978:1:2:20e:feff:fe01:ac0.123: NTPv4, Client, length 48
22:06:00.105279 IP6 2001:4978:1:2:20e:feff:fe01:ac0.123 > 2600:1702:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx.123: NTPv4, Server, length 48

(SIde note: I have native IPv6 now!  No more 6rd!  Must've happened in the last few days; only noticed because of the new prefix assignment when I looked at these captures.)

 

So there's a potential workaround for folks with IPv6.  Just pick IPv6 enabled peers for your NTP devices.  About to play with the advanced firewall rules on the 5268AC and see if I can add a rule to override the 5268AC's blocking behavior on IPv4.  Knowing that this is done at the GW rather than AT&T's edge router(s)......

 

(Second side note: Clearly not a major security issue if they're blocking it in CPE rather than at the network edge.  Kinda reinforces my belief that this was the brain dead idea of an "engineer" that barely deserved the title.....)

 

Message 51 of 67
Tutor

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

hi there, I saw, there is some solution available to solve this problem. I dont have pfsense  router, but it seems my router ( have mikrotik, it is very flexible, by the way Smiley Happy ) , can you please help me to setup my NAT in order to sync time using NTP?

nat_rule_1.JPGnat_rule_2.JPGnat_rule_3.JPGnat_rule_4.JPG

i appreciate your help, thanks

Message 52 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@JefferMC

 

You are correct. This is residential GigaPower 1000/1000. Had same service with the PACE in DMZ+ and got them to switch for the NVG599.

Message 53 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@9999masks

 

What options do you have under Action tab>Action besides Passthrough? I would think you could rewrite the packet here potentially with a different source port.

Message 54 of 67
Tutor

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@stealth2108

Thanks for reply.

I have:

accept

add dst to address list

add src to address list

dst-nat

Jump

Log

Masquerade 

Netmap

Passthrough 

Redirect

Return

Same

Src-nat

 

 

Message 55 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@9999masks

 

Without knowing exactly what happens when you select those, I can't advise. When you change the selection in that dropdown, do you get additional input boxes?

 

What you're looking for is somewhere to rewrite the SRC or source port.

Message 56 of 67
Tutor

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@stealth2108

for dst-nat to address to ports

for jump  to jump target

for masquerade to ports

for netmap to address, to ports

for redirect to ports

for same to address, to ports

for src-nat to address to ports

rest of options just have logging option

 

thanks in advance

 

 

 

 

 

Message 57 of 67
Teacher

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@9999masks

 

I spent about 30 minutes reading through Mikrotik 's documentation and unfortunately I don't see a way to do this through the GUI. There should be a way to do it from the terminal, but I am not versed enough in their syntax to help you.

 

Sorry I couldn't help. Perhaps you can do as I did and get rid of the PACE device and demand another device that is not "broken" and does true IP-Passthrough. I'm using the Motorola/Arris NVG599 on Gigapower with no issues.

 

Good luck!

Message 58 of 67
Tutor

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

@stealth2108

thank you so much for your time. I guess I will request them to change my router, even though i dont think they will do.. 

Message 59 of 67
Contributor

Re: Pace 5268AC in DMZplus blocks UDP 123 (NTP)?

to set this up on Mikrotik,  you need

 

chain  srcnat

protocol udp

src port (NOT dst port)  123

action masquerade

to-ports  10000-20000

Message 60 of 67
Share this topic
Share this topic
Additional Support