AT&T Fiber: Slow Internet (Latency / Packet Loss) started on or right after 6/30

I wish I could say "aha!"  But, er, well, nope.  Can't tell what's going on with all the non-1.1.1.1 traffic for all the huge latency on 1.1.1.1.  These are just ICMP probes right at the IPv4 1.1.1.1, right?

I am in the southern Alabama area near Mobile with the AT&T fiber 1Gbps plan. Beginning yesterday 8/3 around 6pm CT, I started experiencing high latency issues that continued into the night past 11pm before going back to normal. All was fine today until roughly 6pm came around and the same exact issue began cropping up again. Latency to google.com goes from a rock solid 20ms to between 200ms and 1000ms for the duration of the issues. Very little packet loss, just very high latency. Bandwidth is about 500 Mbps down and 900 Up.

Yesterday and today, I performed the typical troubleshooting steps: restarted my BGW-210, restarted my ONT box, restarted all my devices, etc. Nothing helps. Tests have been done on two separate devices connected via ethernet as well as two separate devices connected via WiFi. I am a developer with extensive IT experience and can tell you that the issue is not with my equipment or devices (but good luck getting a tech support rep to believe that).

Using WinMTR, I believe I have identified which AT&T hop is causing the issues.

In the screenshots below, you can see two separate traces of my connection to google.com and cloudflare.com. In both cases, my latency skyrockets between the 99.176.76.138 hop and the next, all AT&T owned IP addresses.

How can I get this information to someone at AT&T with the authority to do anything about it? Support has been painfully unhelpful as usual unfortunately and these issues are crippling my internet for nearly half the day.

Oh geez.  @Splurge5346   Please create your own thread and post what you posted here.  This one is already quite complicated and full of data and adding yours into the mix is only going to make things more confusing. 

I setup a new firewall Unifi Dream Machine Pro (UDMP) last night, gave it one of my static IPs, setup a new VLAN on my 48-port switch, moved a couple of clients to that VLAN and... the AT&T network still sucked.  When I first plugged it in I was on the Dynamic Public IP and the latency ping to 1.1.1.1 was averaging around 500ms.  

Once I switched to my static IP address and gateway IP it was around 6ms.  Somethings felt better, but other things that worked fine while things that had worked "okay" on my original setup before and during this Saga started having issues.  I had to switch back to PfSense to work today.

I need @ATTHelp to get this right.  I'm not using this as a threat, but I feel like I've gone over and beyond here and not getting much in return, and I'm going to have to start looking at using another service.

@jefferMC

In response to 
> I wish I could say "aha!"  But, er, well, nope.  Can't tell what's going on with all the non-1.1.1.1 traffic for all the huge latency on 1.1.1.1.  These are just ICMP probes right at the IPv4 1.1.1.1, right?

I'm my history on this threat I've provided other non ICMP examples of my network having issues from downloads, streaming.  I stuck with ICMP and 1.1.1.1 because it was the most consistent experience I was having that was different from other AT&T customers on the same routes and service.

According to the HOA FB page for my subdivision someone cut an AT&T line down the street, lots of people in my subdivision are without internet right now.  Based on the metrics that I collect all the time now.  I started getting packet loss on AT&T(Pi) Static IP direct connected at around 2:30am EDT last night that's persisted into today.

AT&T is scheduled Wednesday to come out and inspect my outside line and possibly replace it.  I really believe the issue is outside my house.  One thing that @ATTHelp mentioned that was interesting is that having your AT&T gateway plugged into a power strip can cause power fluctuations that manifest as connectivity issues.  I asked if it's plugged into a UPS would that be the same case.  All of my network devices are on UPS to survive power outages and brownouts that occasionally happen due to storms.

Here's the graph showing the new packet

loss:

If someone cut your fiber, you should be down; 100% packet loss, infinite latency.  If they didn't cut your fiber, I don't know why you're seeing any effects, unless they somehow managed to damage your fiber in addition to cutting the other.

ATTHelp loves to say that thing about power strips.  Yes, some not-so-clean power strips could cause issues ON DSL LINES.  Most power strips are fine even then. UPSs should also be fine.  FTR, I use a UPS to feed my ONT via PoE.  I used the same (and other) UPS on my DSL circuit.

If you are feeling cooperative, move to an outlet, demonstrate that the issue continues, then move back to the UPS.

FYI, for what it is worth, I see this issue as well at times (here in California) where things slow WAY down, and latency goes through the roof. I agree, my T-Mobile 5G phone is faster with lower latency used as a hotspot and MUCH more reliable.

We also get occasional short, full drops of internet connectivity (usually indicated by a home automation hub status light going red, confirmed with Alexa and Google devices just dropping off line, or streams just hanging) which lasts from 30 seconds to several minutes, and things return to "normal" with no intervention. We often see this several times a day.  The modem's lights stay green when this happens, but data simply stops flowing anywhere. Connecting locally to the modem shows no issues. But trying to even ping outside the local network(like google.com) comes back with "unreachable" for short periods. 

Do not know if this is related, but I have a replacement modem but saw this behavior on the previous modem too. 

@JefferMC Yes, I'm aware that if they cut my line I'd have 100% packet loss.  I'm not saying they cut my line.  What I don't know is how their infrastructure is laid out for routing and if the change could have caused an issue that my probes picked up.  TL;DR it seems more than just a coincidence to me that I see packet loss where I haven't for days the same time as damage.  Or that fact that AT&T called and said they "noticed" something on my line and asked if my network was fixed yesterday in order to "close" the AT&T guy from coming out tomorrow.  In the works of "Among Us" it seems "sus".

To indulge AT&T I moved the AT&T gateway from to the wall outlet, but no difference... I guess unless magically at 2:30AM EDT it started causing a problem with packetloss now that it's not on a UPS.

So another development today 8/10 at 2:30AM EDT all packet loss and abnormal latency disappeared on both my static ip RasberryPI directly connected to AT&T box and my PfSense FW that was there since the beginning.  I woke up to my internet feeling "normal" again.  The problems started on 6/30 and at least for the moment seems resolved as of 8/10.  So we're clear I didn't have a tech come to my house at 2:30AM this morning.  So the problem was clearly outside my home.

My assumption is that when someone cut the fiber in the subdivision and they had to fix it they guys certifying the repair probably saw my problem and fixed it at the same time.

My grafana / prometheus monitoring started on 7/26.  I was using Ping plotter before then and I only have screenshots so I can't show the full 6/30 - 8/10 graph.

Here's the last 15 days, the last 12h, the last 6h, and the last hour.

Any chance I'll get some acknowledgement from @ATTHelp what the issue was?

Any chance I'll get some acknowledgement from@ATTHelpwhat the issue was?

Thanks.  I needed a good laugh.