Tutor
•
49 Messages
AT&T Fiber: Slow Internet (Latency / Packet Loss) started on or right after 6/30
Has anyone else seen random slowness / latency / packet loss that started in the last week? I've had AT&T GigaPower for about 7 years across 2 homes (yeah lucked out). I guess I've been lucky, but I've never had an issue like this on their fiber 1G plan. However, something changed, which I believe to be external to my home that's caused random slowness on web browsing, video streaming, and VOIP services. Most of the speed tests results are "OK" somewhere in the 820 - 980 range across several speed test sites.
I'm in IT and so speak the language and have done the basics for sure. Restarted my ONT, Gateway, and Router. I've done this several times over the first few days. I called AT&T support and they said they saw some issues with the line from their side and scheduled a tech to come out. The tech came out 2 days ago and tested the fiber link and he said it was good and gave me a new BGW320-505 modem and left. I have tested connectivity behind my router and attached directly to the back of the BGW320-505 gateway with no difference. Also with the new router, it supports SFP so they removed my ONT and it's direct into the GBW320-505 gateway now.
I have a block of static ips on a /29 network which you'll see in the screenshot. I downloaded ping plotter and have been running that for the last few days from a client on my network behind my router. In short, it believes the issue is provider bandwidth saturation on their network on the 5th and 6th hops 12.242.113.21 and 32.142.143.102. I did a whois on both of these IPs and they're owned by AT&T.
And yes I know this data isn't ironclad as ICMP is deprioritized, but it's the best I got at the moment to show that LAN network is fine and the WAN side is not.
I have more data, this is just a quick screen grab from this morning, but I have data showing the 6th hop as an issue as well.
My router behind the BGW320-505 is a pfsense 1U rackmounted server with the following specs:
| Intel(R) Atom(TM) CPU C2758 @ 2.40GHz
8 CPUs: 1 package(s) x 8 core(s)
8103MB memory |
The utilization is always super low (less than 5%).
adac-solutions
Tutor
•
49 Messages
2 years ago
@JefferMC @ATTHelp
This screenshot is from Pingplotter. I'm now collecting data via Prometheus and graphing it via Grafana for longer historical data. I'm also adding more data points. What I can say, and you'll just have to believe be for the moment is that when 1.1.1.1 was all green on 7/8/2022 my internet was "back to normal" as it's been for the past several years. Where there are large volumes of packet loss (red bars) or high latency "black line graph" is when I've also experienced the worse internet experience from surfing websites, streaming, VoIP, gaming, and downloads across multiple devices: wired, wireless, mobile, etc.
0
0
JefferMC
ACE - Expert
•
35.6K Messages
2 years ago
This looks like the same problem experienced elsewhere for a couple of days, or several weeks, before being resolved. I've asked that you be looked into again (I see ATTHelp offered early, but that didn't go far).
What's strange is that I am using some of the same routers, but not having the same issues.
0
0
adac-solutions
Tutor
•
49 Messages
2 years ago
Thanks @JefferMC , last time that ATTHelp engaged was late at night and the timing was off. I couldn't get back to pick it up the next day.
As stated previously I'm collecting all this out of Pingplotter now, though Pingplotter is still running. So we'll have some comparisons.
It's much appreciated for all the help and dialog. I'm hoping to run this to ground soon.
0
0
ATTHelp
Community Support
•
232.4K Messages
2 years ago
We're here to help get your internet back up to speed, adac-solutions.
First of all, thank you JefferMC for all the helpful troubleshooting. This helps us better determine what to look for when we dig deeper into this.
In the meantime, we'd like to invite you to a Direct Message so we can look into your service concerns.
Keep an eye out for a notification at the top right-hand corner of the forums, next to the bell icon, for our Direct Message.
Donovan, AT&T Community Specialist
0
0
adac-solutions
Tutor
•
49 Messages
2 years ago
Thanks @ATTHelp . I'll keep a look out for it. My neighbor is also having the same issue I'm having and it started the same time. We're in the same subdivision, but on different streets. There service was great on 7/8 as I stated above, but went back to bad after 7/8. I just checked in with them and they said Monday was so bad they had to cancel their video meetings; they WFH. I just verified with my neighbor that they called AT&T yesterday about their internet slowness and issues.
0
0
adac-solutions
Tutor
•
49 Messages
2 years ago
@ATTHelp I haven't seen a DM request yet. Today is a good day if you have time. Thanks.
0
0
JefferMC
ACE - Expert
•
35.6K Messages
2 years ago
FTR, mentions to @ATTHelp do not work like they do for "normal" community accounts, i.e. it doesn't send a notification like me mentioning @adac-solutions does for you. However, hopefully your reply to this thread will trigger some sort of notification to ATTHelp's system.
0
adac-solutions
Tutor
•
49 Messages
2 years ago
I'm still out here and still having issues, no contact from AT&T yet as suggested. I'm patiently waiting.
In the meantime I've setup an ec2 instance and setup ICMP ping support and have added that to my list. I've not seen any issues there from my house to us-east-1. It's a steady 22ms roundtrip.
Cloudwatch is all over the place. As it turns out a co-worker of mine is using AT&T fiber as well and has same last couple of hops as me in his traceroute. He's collecting some traceroute and ping data that I'll add here. The on address from hop 7 that is consistently slow is not slow for him in the time he's be testing. It's 172.71.80.4. Where I'm getting 500ms+ he's getting like 6ms.
I can tell that AT&T is using ECMP from hop 6 (32.142.133.106) to hop 7.
My connectivity is okay sometimes and the experience I'm getting now is a slight android like or drop of VoIP services, Streaming services going from great to crap. Streaming service websites not loading tile images, loading spinners. Websites randomly stalling out for every from sites like AWS cloudfront, Google Drive, Google search results taking 30+ seconds to return randomly etc. Things I use on a regular basis.
From my co-worker:
❯ traceroute -q5 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
1 dsldevice (192.168.1.254) 3.369 ms 1.733 ms 1.962 ms 2.060 ms 1.944 ms
2 162-201-76-1.lightspeed.tukrga.sbcglobal.net (162.201.76.1) 3.210 ms 2.877 ms 2.944 ms 1.974 ms 2.013 ms
3 76.239.207.228 (76.239.207.228) 3.243 ms 4.882 ms 4.007 ms 3.948 ms 4.917 ms
4 12.242.113.21 (12.242.113.21) 5.289 ms 4.674 ms 4.010 ms 4.951 ms 4.022 ms
5 32.142.143.102 (32.142.143.102) 18.539 ms 4.735 ms 4.880 ms 6.080 ms 4.984 ms
6 172.71.28.4 (172.71.28.4) 8.640 ms
172.71.80.4 (172.71.80.4) 5.390 ms
172.71.24.4 (172.71.24.4) 5.413 ms
172.71.28.4 (172.71.28.4) 5.898 ms
108.162.235.12 (108.162.235.12) 5.542 ms
7 one.one.one.one (1.1.1.1) 4.073 ms 3.897 ms 4.006 ms 4.493 ms 4.471 ms
❯ traceroute -q5 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
1 dsldevice (192.168.1.254) 3.755 ms 1.773 ms 1.994 ms 2.078 ms 1.826 ms
2 162-201-76-1.lightspeed.tukrga.sbcglobal.net (162.201.76.1) 3.244 ms 2.860 ms 1.991 ms 2.996 ms 2.019 ms
3 76.239.207.228 (76.239.207.228) 4.188 ms 4.333 ms 3.988 ms 4.707 ms 5.015 ms
4 12.242.113.21 (12.242.113.21) 4.294 ms 4.610 ms 3.953 ms 4.051 ms 3.923 ms
5 32.142.143.102 (32.142.143.102) 5.491 ms 4.606 ms 4.954 ms 5.176 ms 6.908 ms
6 172.71.28.4 (172.71.28.4) 5.195 ms
172.71.20.4 (172.71.20.4) 4.980 ms
172.71.80.4 (172.71.80.4) 6.046 ms
108.162.235.12 (108.162.235.12) 4.926 ms
172.71.28.4 (172.71.28.4) 14.117 ms
7 one.one.one.one (1.1.1.1) 5.079 ms 4.051 ms 4.868 ms 4.001 ms 3.868 ms
❯ ping 172.71.80.4
PING 172.71.80.4 (172.71.80.4): 56 data bytes
64 bytes from 172.71.80.4: icmp_seq=0 ttl=59 time=11.381 ms
64 bytes from 172.71.80.4: icmp_seq=1 ttl=59 time=11.997 ms
64 bytes from 172.71.80.4: icmp_seq=2 ttl=59 time=12.277 ms
64 bytes from 172.71.80.4: icmp_seq=3 ttl=59 time=11.124 ms
64 bytes from 172.71.80.4: icmp_seq=4 ttl=59 time=5.692 ms
64 bytes from 172.71.80.4: icmp_seq=5 ttl=59 time=4.967 ms
64 bytes from 172.71.80.4: icmp_seq=6 ttl=59 time=12.271 ms
64 bytes from 172.71.80.4: icmp_seq=7 ttl=59 time=5.859 ms
64 bytes from 172.71.80.4: icmp_seq=8 ttl=59 time=15.357 ms
64 bytes from 172.71.80.4: icmp_seq=9 ttl=59 time=11.123 ms
64 bytes from 172.71.80.4: icmp_seq=10 ttl=59 time=11.515 ms
64 bytes from 172.71.80.4: icmp_seq=11 ttl=59 time=12.586 ms
64 bytes from 172.71.80.4: icmp_seq=12 ttl=59 time=6.141 ms
64 bytes from 172.71.80.4: icmp_seq=13 ttl=59 time=7.643 ms
64 bytes from 172.71.80.4: icmp_seq=14 ttl=59 time=14.894 ms
64 bytes from 172.71.80.4: icmp_seq=15 ttl=59 time=10.498 ms
64 bytes from 172.71.80.4: icmp_seq=16 ttl=59 time=12.205 ms
64 bytes from 172.71.80.4: icmp_seq=17 ttl=59 time=8.484 ms
64 bytes from 172.71.80.4: icmp_seq=18 ttl=59 time=14.916 ms
64 bytes from 172.71.80.4: icmp_seq=19 ttl=59 time=20.214 ms
64 bytes from 172.71.80.4: icmp_seq=20 ttl=59 time=4.545 ms
64 bytes from 172.71.80.4: icmp_seq=21 ttl=59 time=12.010 ms
64 bytes from 172.71.80.4: icmp_seq=22 ttl=59 time=6.752 ms
64 bytes from 172.71.80.4: icmp_seq=23 ttl=59 time=5.546 ms
^C
--- 172.71.80.4 ping statistics ---
24 packets transmitted, 24 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.545/10.417/20.214/3.861 m
0
0
JefferMC
ACE - Expert
•
35.6K Messages
2 years ago
I'm still wondering if that's AT&T doing that or Cloudflare. AT&T could be sending the traffic to one IP, but when Cloudflare receives it, it may be handled by multiple hosts that report errors using different IPs.
Have you tried using google or someone else for DNS services?
(edited)
0
0
adac-solutions
Tutor
•
49 Messages
2 years ago
@JefferMC I switched to 8.8.8.8 and 8.8.4.4 when we talked about Cloudflare last week. So I've been using 8.8.8.8 for primary dns for the last week.
Also, I'm not overly concerned with 1.1.1.1 not have good performance. It's just the closest thing I have to a consistent reduction in service that's consistent with my internet issues. Just so we're all on the same page. Every time I found slowness I try to write a test to monitor for it to happen again. I've yet to create one for streaming services or voip, but I've been looking into it.
I've wondered the same thing too. Is it Cloudflare or AT&T. I've decided it's AT&T since it's direct hops. They're probably using routing protocols between edges to determine which IP to route to from AT&T -> Cloudflare.
It's also strange that my neighbor an I are having issues, but my co-worker isn't. But I know there's ways to look at this an AT&T Network Specialist could probably explain or know that this all makes more sense to them. I'm not a hardcore network guy, it's not my IT specialty.
(edited)
0
0