Pre-order the New Samsung Galaxy Z Flip4 & Galaxy Z Fold4 and, for a limited time, get twice the storage and a free case!
Get superfast AT&T Fiber internet
V

New Member

 • 

8 Messages

Sun, Dec 6, 2020 6:51 AM

Weeks-long service degradation affecting many customers in SF Bay area

So I'd been using U-Verse fiber for about a year now with no real complaints (apart from the price relative to e.g. PAXIO, but good luck getting that in an apartment complex south of Emeryville)...but in mid/late November I noticed that some sites were starting to have difficulty loading, especially over TLS. The issue was intermittent so I didn't take much note at first, but as it dragged on and got worse I started looking into it.

According to some server logs I have on a remote VPS I sync with, the issue started on Nov 15 when TLS connections would somewhat frequently have issues establishing the connection. As I dug deeper, I discovered that there was packet corruption between my home and my VPS, with symptoms such as a single bit in the DH key exchange getting flipped, invalidating the whole handshake. As I poked at the issue more and more it became apparently that the issue only occurred on certain routes, meaning that it wasn't just affecting me, but would also affect anyone hitting that route. I dug deeper and discovered that everyone I knew who also had AT&T Fiber in the area has been having this same issue. I called AT&T and reported that some sites, like Twitter's CDN (pbs.twimg.com) and Wikipedia (dyna.wikimedia.org) were having this issue too. They sent out a tech who...did nothing. Showed up, determined the issue wasn't coming from inside my apartment, and left without even informing me they'd shown up. So I hoped it would get better soon.

Anyway that was a week ago and I've been being tossed around from people on the phone who sympathize but can't do much to techs and supervisors who run tests within AT&T's network and say it's fine so the problem must be on my end, without even listening to me saying the problem is when traffic LEAVES or ENTERS their network (AS7018) to other networks (e.g. AS2914 is particularly impacted, but it doesn't seem to be the only one). They keep saying things about speed tests, which are irrelevant, and that they can connect to me fine, which is also irrelevant. They won't listen no matter who I try to talk to, and I'm running out of people I can contact.

One tech insisted on replacing my RG with an identical one (but it was brand new!!!) and of course this did nothing other than give me an hour of completely broken service until it got back up to the same heavily-broken state it was in before. This specific tech was especially bad, refused to even listen to anything I said and was very concerned about my Wi-Fi coverage (I usually use Ethernet), and that having another port used on my switch would "drain" my bandwidth (which doesn't even mean anything). He didn't seem to even comprehend the issues I was having at all, making things substantially worse for an hour or so in the process by screwing around with my network. I've never been so glad for a tech to leave without having even fixed anything, and he had the audacity to ask for a good review even after failing to establish basic communication with me. Sorry Raul V, you're not getting a good review.

I've posted about it on Twitter a few times and other people have chimed in and said the issue is affecting them too. In fact, more are more people are finding my posts and replying that they have the same issues. I keep telling them to also contact AT&T but most of them either don't or put in least effort contact and give up after having trouble reaching anyone who's even affiliated with U-Verse tech support. It's been a slow trickle since I don't have much visibility, but I have had at least a dozen other people reach out and corroborated that they're experiencing the same issue. How can I actually get AT&T to listen so their now 3 weeks of severely impacted service for what appears to be everyone in the SF Bay Area on their service will stop having this issue?

I'm familiar with people having this sort of next-level nightmare dealing with Comcast tech support, but I switched to AT&T in part so I wouldn't have to deal with that, and yet this ongoing debacle far exceeds any problem I ever had when I've used Comcast. I'm sorely tempted to switch back even though my speeds would be worse (upstream matters too!), my and my bill would be higher if I paid to remove the data cap (that I would exceed regularly just by doing backups). AT&T just doesn't seem to even care to acknowledge there's an issue here, much less actually look into it. How is this acceptable service? It's impacted my ability to do my job from home, my ability to stream using sites like Hulu, my ability to do video chats during lockdown, and a lot more, and I've been banging my head against it for three weeks while AT&T refuses to lift a finger unless it's to audaciously point the blame back at me, the person who is paying them to provide a service that I assumed they would care to keep at least somewhat maintained, and had had no issues for a year until something randomly broke for a bunch of people at once. They can't even seem to figure out there is a genuine partial service degradation somewhere in the area. I am sorely disappointed by the lack of professionalism in how this has been handled.

browndk26

ACE - Professor

 • 

4.7K Messages

2年前

First level support doesn’t have access to or understand the type problems you have. It’s probably a routing or peering issue on the web after att’s network. Either beyond att control or not a big enough problem to them. It may clear itself up over time. 

FCC complaint is an option. 

New Member

 • 

5 Messages

2年前

Yes! This is exactly the same issue I've been experiencing for weeks now on ATT fiber in the Bay Area.

In particular, I've been seeing high packet drop exiting ATT's network in Palo Alto. I have 20% packet drop into AS2914:

--- 129.250.2.2 ping statistics ---
378 packets transmitted, 304 packets received, 19.6% packet loss
round-trip min/avg/max/stddev = 7.312/13.904/171.418/18.949 ms

I'm not sure how to get ATT's attention?

(I also noticed this specifically with Twitter and Wikipedia.)

(edited)

New Member

 • 

8 Messages

2年前

@browndk26 I understand that first level support won't get it. The problem is that I was unable to get them to route me to someone who does understand enough to connect with the people who actually have the ability to fix it. Instead I was stuck with some contacts who told me "well my supervisor and my supervisor's supervisor say the techs say the issue is on your end," which solves nothing and leaves me stuck. They don't give me a route to escalate, and that is a serious issue when it's a problem that's beyond their scripts. Further, given I have at least a dozen, probably closer to two dozen at this point, individuals who have chimed in here and elsewhere confirming they also have the issue, it's pretty clear that it's a widespread issue that AT&T either does not have proper diagnostics to have detected or they just don't care. Both of those options are very concerning, especially for a large scale ISP.

I filed an FCC complaint earlier today, as a few people have suggested and I'd been waffling on for a few days.

(edited)

browndk26

ACE - Professor

 • 

4.7K Messages

2年前

Like it or not, it won’t get escalated. A couple dozen people with a problem is not a problem to att. 

New Member

 • 

8 Messages

2年前

"A couple dozen people" is how many people have reached out with this limited visibility. I'm sure the actual number is at LEAST an order of magnitude higher, if not 2.

New Member

 • 

2 Messages

2年前

Running into the exact same issues as vpfau here in the East Bay.  I started noticing issues around Dec 1, with the problem becoming significantly worse around Dec 4

dave006

Scholar

 • 

3.1K Messages

2年前

It's impacted my ability to do my job from home

Then you should not be on a residential connection. Consider paying for business service and you will have an SLA and access to network support.

How responsive has NTT been to your queries concerning a possible issue with their network AS2914 since it is their network peering to AS7018?

Dave

New Member

 • 

8 Messages

2年前

Find me a business ISP that services my apartment, has upstream that is at least 100 Mbit, convince my company to subsidize it, and then we'll talk.

I talked to someone at NTT who asked me to collect data and relay it to noc at us.ntt.net, but I've been preoccupied enough that I haven't gotten to it yet.

New Member

 • 

5 Messages

2年前

So far NTT has been unresponsive for me, but I don't believe this is an issue with NTT. As demonstrated by the script shared in this Reddit thread (which attempts to query example.com), the failure occurs outside of NTT's network also.

It looks to me like packet corruption at 12.242.117.22, which is the last hop in ATT's network.

dave006

Scholar

 • 

3.1K Messages

2年前

Find me a business ISP that services my apartment, has upstream that is at least 100 Mbit, convince my company to subsidize it, and then we'll talk.

@vpfau the easy answer is AT&T Business Fiber.

The subsidy payment is between you and the company you work for. How important or valuable to the business is it that you are able to WFH?

Just a reminder this is a customer to customer forum and not a formal support tool.

Dave

New Member

 • 

8 Messages

2年前

Once the issue is resolved AT&T Business Fiber might be viable. Until then, it wouldn't fix issues other than maybe letting me talk to better support.

And yes, I know this isn't a formal support tool. I've exhausted the formal support tools by now and am mostly trying to give the issue increased visibility so other customers who are having the same issue know it's not just them, and to encourage them to reach out to support as well so AT&T might get a clue.

New Member

 • 

5 Messages

2年前

This is now one of the top posts on Hacker News. Some users have shown how packets are corrupted in predictable ways.

New Member

 • 

8 Messages

2年前

People have started to report that this has been fixed. I can't say for sure yet but I have not noticed any issues since I started seeing the reports, so we might have finally gotten AT&T's attention. Finally.

New Member

 • 

5 Messages

2年前

It's fixed for me too.

Need help?
New to AT&T Community?
New to the AT&T Community? Start by visiting the Community How-To.
New to the AT&T Community?
Visit the Community How-To.