Explore & discover

Helpful Links

Increasingly terrible latency over the past 1-2 months

Teacher

Increasingly terrible latency over the past 1-2 months

I've had no problems since getting Uverse installed March of '17 until now. Beginning in mid December (somewhere around the 12-14th by my recollection), I started experiencing latency spikes in online multiplayer games. It was just an annoyance at first, small spikes of a couple hundred ms every half hour or so. Without going into all of the details in between, as it stands today I get spikes of several seconds (not ms) every 15-30 seconds. This makes online multiplayer gaming a literal impossibility. I'm just going to lay out all of the information that I have gathered so far, there's quite a lot.

 

First, this is on fiber, not DSL. The model/router is a Pace 5268AC. Yes, I'm on a wireless connection, no a wired connection is not a possibility, however I've tested with wired connections and the problems are the same. It is not a problem with wireless signal strength or quality. More on that in the screenshots below. I've tested this on five separate devices in from my desktop, to my laptop, to phones and tablets. All show identical results with high latency spikes and packet loss in the same places.

 

Here come a number of screenshots of various diagnostics I've performed.

 

3Jg8vvUinSSIDer capture showing there is zero interference or traffic from other networks in the area. I also have a very strong signal to my modem.

RoybelGStatistics from the modem captured just today. Still perfectly acceptible signals to the modem. This is after a recent reboot of the modem. Uptime: 2 days, 8 hours. Please note, the increase in noise is because the modem reset me to "default" channels in the 5GHz band and I was overlapping the other two networks near me. I've fixed that since this screenshow and am currently showing -40db again.

KRTsDHNMore modem stats. Taken at the same time as previous.

AnahenVWinMTR screenshot taken just now. The second hop is the problem spot for high latency, hops 4 and 5 always show high packet loss. I don't know if it's purposeful dropping of ICMP packets due to high traffic or actual loss.

qiR1yJyJust a quick WinMTR with hostnames resolved.

f7h8SRBFinally a Wireshark capture filtering only Duplicate ACKs and TCP Retransmissions on the route shown in the WinMTRs. You can see an RTO of over 2 seconds which is common for most of these Retransmissions.

 

A final thing to note, I'm in Cleveland. We had a very warm and mild November/first couple weeks of December. The problems started just about the same time "winter" finally hit us in mid December and the temps dropped and so did the snow. This could be purely coincidental, but it's definitely worth mentioning. I'll gladly provide any additional information needed to fix this.

 

EDIT: I forgot to mention, my bandwidth seems unaffected for the most part. Speedtests using various services (speedtest.comfast.com and speedof.me for example).

 

EDIT2: I also realize that I didn't mention that this problem exists 24/7. There is no specific time of day where it is worse or better.

1,474 Views
Message 1 of 16
Community Support

Re: Increasingly terrible latency over the past 1-2 months

Hi @Darxide23,

 

I apologize about the issues you are having with your service. Thank you for providing all that information. It is always hard to identify latency issues. For testing purposes, have you tried connecting through a VPN to see if you are running into similar issues?

 

David, AT&T Community Specialist

AT&T Customer Care

Need help with an account specific question?  Post a new question here on the forums by clicking the "Ask a Question" button.
For additional support, please visit us at our AT&T services hub.
Follow us on: Twitter @ATTCares and @DIRECTVService

*I am an AT&T employee, and the postings on this site are my own and don't necessarily represent AT&T's position, strategies or opinions.
Message 2 of 16
Teacher

Re: Increasingly terrible latency over the past 1-2 months

I hadn't even thought of that, but here's the results.

Fer7dk5
Looks a lot more normal through my VPN service. It is a paid VPN, by the way. I don't usually consider it for gaming because it's inherently slower than a healthy connection through any ISP.

Message 3 of 16
Teacher

Re: Increasingly terrible latency over the past 1-2 months

This is still an issue. I have a bandwidth limit on my VPN, so I can't keep wasting it on this. If I have to upgrade my VPN plan because of this I guess I'll send AT&T the bill?

Message 4 of 16
Teacher

Re: Increasingly terrible latency over the past 1-2 months

Bumping. Still an issue.

Message 5 of 16
Scholar

Re: Increasingly terrible latency over the past 1-2 months

@Darxide23

 

I have DSL and have the same problem, seems to be peering issues with AT&T partners, obviously they don't care about customer satisfaction since they haven't answered you. Sorry to say. 

Message 6 of 16
Teacher

Re: Increasingly terrible latency over the past 1-2 months

Bumping. Still an issue.

Message 7 of 16
Teacher

Re: Increasingly terrible latency over the past 1-2 months

 Bumping. Still an issue.

Message 8 of 16
Highlighted
ACE - Expert

Re: Increasingly terrible latency over the past 1-2 months

Looking at your very first WinMTR:

Capture-2018-02-05.PNGYour winMTR

I want to mention two things, one of which you mentioned (hops 4/5 high loss) and one you didn't (W/C latency on hop 2).

Hop 4/5 loss: Most AT&T Residential internet customers (U-verse VDSL2 & Fiber) will tell you that this is normal.  These just drop a large number of TTL-expired ICMP packets without spending time in the main CPU generating the error response.  By the fact that nodes further down the line respond more normally, you can infer that they pass ICMP packets that aren't TTL-expired.

Worst Case latency on hop 2.  This is your first hop router inside AT&T.  It's interesting because it has to be passing your ICMP traffic faster than it's generating an error reply (which is expected).  On copper, this isn't typically noticable.  I'm not sure about fiber.

 

Any chance you could get a winMTR report more focused on the period when you have a latency event?  An alternative is a bunch of PING -t on a variety of nodes on the path towards your server, check all of them to see which ones show something interesting at the time of increased latency (dropped packet, long latency).  Might let us draw a map to where it's occurring.

 

 

 

Award for Community Excellence 2019 Achiever*
*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 9 of 16
Teacher

Re: Increasingly terrible latency over the past 1-2 months

As I did mentioned, there are no "latency events". This is a 24/7 latency event. Any given moment, my WinMTR looks just like the one I posted originally.

 

Edit: Just for emphasis, I took this right now after I posted. I didn't have to wait for an "event". I wasn't doing anything where I could notice any latency. (I was just reading some blogs, not gaming).

06sVP3A

 

Something you also mentioned about pinging the nodes individually, if I do a ping to the second hop (the one where the high latency spikes are appearing which I did mention as the main focus of my post) then there are no latency issues to that node when pinging individually. So I ran WinMTR on that node for a while. No problems. Then I ran WinMTR to the third node and the second node then showed it's horrible WC latency with lots of spikes. I don't know if that means anything. See image below to see what I mean.

 

4sqLGZ6

Message 10 of 16
ACE - Expert

Re: Increasingly terrible latency over the past 1-2 months


@Darxide23 wrote:

As I did mentioned, there are no "latency events". This is a 24/7 latency event. Any given moment, my WinMTR looks just like the one I posted originally.

 

Edit: Just for emphasis, I took this right now after I posted. I didn't have to wait for an "event". I wasn't doing anything where I could notice any latency. (I was just reading some blogs, not gaming).

[img]

 

Something you also mentioned about pinging the nodes individually, if I do a ping to the second hop (the one where the high latency spikes are appearing which I did mention as the main focus of my post) then there are no latency issues to that node when pinging individually. So I ran WinMTR on that node for a while. No problems. Then I ran WinMTR to the third node and the second node then showed it's horrible WC latency with lots of spikes. I don't know if that means anything. See image below to see what I mean.

 [img]

 


Two things:

First image: Please indicate what your issue here is.  Your average latency is 40 ms to your target server.  You appear to be in Ohio, the game server appears to be in New Jersey.  What latency do you expect?  If you want to say "packet loss," notice that the packet loss appears to be in the New York Telia network.  You could complain to Telia... if they'll listen.

 

Second image: There is a difference in processing between being the destination of a PING and being the second hop on a PING when the original TTL of the packet was 2.  In the first case, the router creates the standard reply.  In the second case, the router has to generate the exception reply for expired TTL.  ICMP Exception reports are deprioritized in AT&T residential edge routers.  This "terrible" response means nothing, because it passed 100% of the packets destined for hop 3 with a TTL=3 and did so such that the latency was 3 ms, a great job.  So, as long as you're not asking hop 2 to report errors, it does a fine job.

 

Award for Community Excellence 2019 Achiever*
*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 11 of 16
Teacher

Re: Increasingly terrible latency over the past 1-2 months

First image: Please indicate what your issue here is.  Your average latency is 40 ms to your target server.  You appear to be in Ohio, the game server appears to be in New Jersey.  What latency do you expect?  If you want to say "packet loss," notice that the packet loss appears to be in the New York Telia network.  You could complain to Telia... if they'll listen.

The latency I expect is ~40ms. This isn't what I'm seeing in-game. Also, if the packet loss was as high as indicated by hops 4 & 5 then I wouldn't be able to move around in-game at all with that many packets being dropped, this is why I'm pretty sure that (at least some) of those lost packets are just ICMP requests being dropped purposefully. Wireshark doesn't seem to indicate nearly that many lost packets as it should for over 50% packet loss (and upwards of 85% at times according to WinMTR), just a lot of packets arriving out of order. I've also expanded my filter in Wireshark and have noticed many Suprious Retransmission entries which just screams latency issues and not lost packets. If the problems I'm experiencing are truly coming from lost TCP packets at hops 4 & 5 then my complaining to the owner will do no good. I would be beneath their notice.

 

Another reason why I think this problem is not due to dropped TCP packets is that from asking around to other players I have not found anybody else who lives in the Great Lakes region that has this kind of problem. This is isolated to myself and potentially anyone else in my neighborhood doing online gaming, but knowing the demographics of my immediate area I would not at all be surprised if I'm the only one. Of course, that's neither here nor there.

 

At any rate, why not just tell me what it is you need me to do and I'll do it. What info do you need? What tests can I run? I will also see if there's any way I can get out of my neighborhood and try to connect to the game and run some tests, perhaps through a coffee shop wifi or something. I'd have to borrow a laptop, so I'm not sure how that will turn out if at all.

Message 12 of 16
Contributor

Re: Increasingly terrible latency over the past 1-2 months

For what it's worth, this customer isn't alone.  Many AT&T customers seem to be routed through Telia and seeing lag spikes there. Just search this forum for "telia" and you'll see a lot.

 

I'm a new AT&T customer who recently left Comcast, and while the bandwidth is great, the latency is AWFUL in comparison. I was getting 16ms pings in games of Rocket League very reliably before switching to AT&T, where they're now consistently 45ms or so at best, but often 70-100ms. This is just a few weeks apart - and otherwise with all of the same network hardware, ethernet cables, etc - I didn't move, I just canceled Comcast and started AT&T.

 

I understand you're saying the problem is with Telia -- but I'm not a paying Telia customer, and AT&T is (likely?) peering through some sort of relationship/agreement with Telia -- so it'd sure be appreciated if AT&T could be an advocate for its customers and share this sort of information with Telia. They have no financial obligation or reason to care what some random AT&T customers say to them.  They seem to be affecting a large number of your customers if you consider how few folks are technically savvy enough to know what traceroute is and there's a fair number of threads in this forum starting last fall.

 

Here's my own traceroute to yahoo.com just as an experiment, for example:

 

1 192.168.1.1 (192.168.1.1) 1.332 ms 0.990 ms 0.814 ms
2 192.168.1.254 (192.168.1.254) 1.177 ms 1.282 ms 1.420 ms
3 108.232.84.1 (108.232.84.1) 5.131 ms 3.137 ms 6.215 ms
4 71.151.17.114 (71.151.17.114) 3.244 ms 3.036 ms 2.698 ms
5 71.145.65.236 (71.145.65.236) 3.597 ms 3.372 ms 3.710 ms
6 12.83.43.89 (12.83.43.89) 6.387 ms
12.83.43.93 (12.83.43.93) 2.928 ms 5.602 ms
7 cgcil403igs.ip.att.net (12.122.133.33) 6.323 ms 4.009 ms 3.903 ms
8 chi-b21-link.telia.net (213.248.87.253) 3.873 ms 4.123 ms 3.723 ms
9 sea-b2-link.telia.net (62.115.117.48) 48.297 ms 48.459 ms 72.583 ms
10 yahoo-ic-328471-sea-b2.c.telia.net (62.115.36.197) 48.643 ms 48.688 ms
yahoo-ic-328472-sea-b2.c.telia.net (62.115.61.122) 46.095 ms
11 ae-7.pat2.gqb.yahoo.com (216.115.101.109) 52.300 ms
ae-5.pat1.gqb.yahoo.com (216.115.101.111) 50.567 ms 50.169 ms
12 et-0-0-0.msr1.gq1.yahoo.com (66.196.67.97) 49.896 ms
et-19-1-0.msr2.gq1.yahoo.com (66.196.67.111) 49.657 ms
et-0-0-0.msr2.gq1.yahoo.com (66.196.67.109) 52.626 ms
13 et-0-0-0.clr2-a-gdc.gq1.yahoo.com (67.195.37.73) 53.239 ms
et-19-1-0.clr2-a-gdc.gq1.yahoo.com (67.195.37.99) 52.829 ms
et-1-1-0.clr2-a-gdc.gq1.yahoo.com (67.195.37.75) 52.602 ms
14 et-16-6.bas1-2-flk.gq1.yahoo.com (98.137.120.6) 53.187 ms
et-18-6.bas1-2-flk.gq1.yahoo.com (98.137.120.25) 54.986 ms
et-16-6.bas2-2-flk.gq1.yahoo.com (98.137.120.14) 52.905 ms
15 media-router-fp2.prod1.media.vip.gq1.yahoo.com (98.137.246.8) 49.368 ms 50.172 ms 49.615 ms

Message 13 of 16
ACE - Expert

Re: Increasingly terrible latency over the past 1-2 months


@honestbleeps wrote:

For what it's worth, this customer isn't alone.  Many AT&T customers seem to be routed through Telia and seeing lag spikes there. Just search this forum for "telia" and you'll see a lot.

...

Here's my own traceroute to yahoo.com just as an experiment, for example:

 

...
7 cgcil403igs.ip.att.net (12.122.133.33) 6.323 ms 4.009 ms 3.903 ms
8 chi-b21-link.telia.net (213.248.87.253) 3.873 ms 4.123 ms 3.723 ms
9 sea-b2-link.telia.net (62.115.117.48) 48.297 ms 48.459 ms 72.583 ms
10 yahoo-ic-328471-sea-b2.c.telia.net (62.115.36.197) 48.643 ms 48.688 ms
yahoo-ic-328472-sea-b2.c.telia.net (62.115.61.122) 46.095 ms
...
15 media-router-fp2.prod1.media.vip.gq1.yahoo.com (98.137.246.8) 49.368 ms 50.172 ms 49.615 ms


In your example, Telia is carrying your traffic from Chi(ago) to Sea(ttle).  Seeing an increase in latency of 40ms for 2000 mile link is a little more than I would expect, but not an order of magnitude more.  Perhaps you are being routed to different servers based on a bad geolocation of your IP address?

 

Award for Community Excellence 2019 Achiever*
*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 14 of 16
Contributor

Re: Increasingly terrible latency over the past 1-2 months

I'm honestly not sure how it's decided that I'm getting routed through to seattle - but I know my latency with AT&T Gigabit fiber is far worse than my latency was with 100mbps Comcast, and this is one example of that.

 

It may entirely be on Telia's end, I just don't think Telia gives a hoot about one guy who pays AT&T for service instead of Telia, you know?

 

That said, since switching from Comcast to AT&T, everything's Geolocating me in Cicero IL, not Chicago IL.  I'm in Chicago. That said, Cicero isn't anywhere near far enough (it's a suburb) to justify routing me the wrong way geographically, I wouldn't think?

 

I haven't actually taken the time to figure out rocket league server IP addresses because I play it on Xbox, not PC, so it's not trivial to dig up that info. That said, I'll do some more research and confirm whether or not it's always Telia where the latency spikes up. Even if it is, I just don't see what reasonable course of action there is for me besides "send a pointless email to Telia that they will definitely never care about because I'm not their customer" :-(

Message 15 of 16
Share this topic
Share this topic
Announcements...
Are you having trouble logging in? Is your email password not working? Let us show you how to Reset your Email Password using myAT&T!
Additional Support