Fixed Wireless High Packet Loss

@Snowydays They've said they will no longer send techs out to check on this problem because they cannot fix it.

Expect them to look out the window and say, "Yep, tower's still standing there," and then tell you they've done some advanced diagnostics and to reboot your gateway to receive the fix.

My connection shows significantly less packet loss and less jumps in latency. I would not say "normal" but it is enough to notice.

I do not have fixed wireless. I have gigafiber but the presentation of packet loss has been identical. That points to the (Edited per community guidelines) issue at ATT core routing functions.

If others running ping monitoring tools can they check and comment.

[eth0] is my WISP, [eth1] is AT&T. 8.8.8.8 has always manifested early loss for me and is much better today. Fingers crossed that 16:00 EST brings more of the same for my other endpoints. And then the question becomes "will it last?" -- sure would be nice if we'd get a post-mortem.

it was working good up until 4pm and now it is acting silly, have you guys ran a test?

still 40% lost for me, not really seeing much change.

Edit: actually worse for me currently, also have a question, how many of you recieved communication regarding yesterdays supposed fix? I have an open fcc complaint and haven’t heard a peep from At&t regarding the supposed fix in progress.

Same for me, though it was pretty good last night. I'm showing serious packet loss tonight though. Sometimes I flat out error out on the 100% packet loss towers. And I error out at google.

Today has probably been the best day for me in the last few weeks. I've been able to game with very little disconnects. Usually at this time today, it's really bad. But I'll keep using it and see if anything comes up and the packet loss starts coming back heavily.

ICMP as diagnostic notwithstanding, somebody changed something -- my pinger for google dns is vastly improved (same route). Everything else looks like it's going to be more of the same ol' thing.

@dkresge We can't message each other here directly/privately?

Thanks for the info about grafana! Super cool project, and the related info about "TSDB"s!

Found: https://github.com/lomik/go-carbon

Neato, hoping to work on a tutorial and some tools to help other people here run automated tests using iperf, dns, curl, and ICMP on timers. (one container for instance w/ grafana and go-graphite, another for testing iperf)

Do you think asking users to install docker, download a container, run the container, and then edit the config, is acceptable? Or is that asking too much?

@unleadedgass This is a really interesting update on the situation. Can I ask how you're getting updates from the FCC on this? I also sent in a report for the same thing and they have yet to send me even 1 e-mail about the progress.

However, if that is true, it's good to see that the pressure is being put on them to solve it. 14 more days after nearly 2 months of a crippled service is a bit annoying, but I suppose all we can do is wait.

I have definitely noticed a SLIGHT improvement to the service in the last couple of days, but the problems are certainly still there from what I've seen on my end. I'd like to thank everyone in this thread for coming together to solve our mutual suffering.

@unleadedgass If you ask me, the FCC should make these kind of things public record if they acknowledge there is a problem.

If you get a ham radio license, the FCC makes your name and address publcily available for anyone to find but there's no accountability for this kind of stuff.

@unleadedgass Wait, so they have 30 days to respond with a fix, but they were given 14 additional days meaning 30+14 (44) or they plan to have the fix done in 14 days? It's a little unclear so hopefully you can clarify for me. 14 vs 44 from the day they were served is a big difference.

@espespesp Given who is over the FCC, I would not be surprised if they are given 14 day extensions until the end of time. I have absolutely no faith in either the FCC or AT&T.

Even if this is fixed tomorrow, it's still been going on for 2+ months. Sorry if I come across as a broken record about this.

@malleable_malus It's not a broken record at all. I completely agree with you that it's ridiculous AT&T hasn't already gotten this under control on their own without having to involve the FCC. 2 months (or even 1 month) is far too long for a national "outage" as they put it.

@unleadedgass Thanks for that. I wasn't aware they already served them on January 29th for this. I feel like it was just this month they actually started admitting there was a real issue, but perhaps I'm wrong there. I'll be keeping my eye on this thread for updates since I'm not really getting any feedback from the FCC myself.

I am thinking the filing with FCC will turn out to be an important push here.

I am still thinking at the core of these issues is ATT prioritizing packets / traffic shaping. The packet manipulation likely went off rails and a fix must be developed and propogate throughout thier infrastructure which can take days / weeks.

My HouTx fiber (no wireless) but have same issues. For past 24-48 hours things have improved but not normal. I am still getting hit hard 7pm-11pm. I am seeing about 1 instance per hour of PL that hits my BOVPN including between midnight and 6am a nadir for network traffic.

As of 4.08am this morning Level3 dns went down and is still down. It is possible a separate issue but my pings from my office ISP (not ATT) are clean and responsive so at best this is an issue with an ATT nexus. Adds to pattern over past 2 months of random loss of connectivity to 3rd party DNS when such DNS is otherwise available. If you ask the ATT support they will tell you to use ATT dns.

@dennisharrison Hi! Glad you're enjoying the rabbit hole :-)

I like your idea of a simple drop-in diagnostics and monitoring package. My only reservation would be its potential to derail this forum into a docker support discussion.

That said, I wonder if centralizing the collector and dashboard on a VPS would allow for a more cohesive picture while also minimizing the number of "moving parts" at the edges.

( e.g. iperf client/telegraf pinger runs on client nodes; iperf server/influx runs on VPS)

Unfortunately, I don't have much time to contribute - and, well, I'm a tad bit "edited per community guidelines" that it's devolved to the point where we're doing AT&T's job for them. But feel free to ping me at <attforums-mail at kresge.com> if you need someone to bounce an idea off of.

suggest organizing in dslreports forum or something similar. Once that setup then post steps to locate here so those open to this join that seperate discussion.

@lottytx taking steps backwards. Worse today

Once it hit about 8PM Est, my connection has went downhill from there. But I managed to go all day without any packet loss, or anything, so that's a start at least.

5PM in Michigan on a Wed Night. Was experiencing 50% Packet Loss and frequent disconnects from the Network. For over an Hour the internet was almost totally unusable. This has been the Norm since late December. No improvement in sight

Just wanted to update....I haven't been home long enough the last couple of days to really give my connection a good workout, as life has been pretty busy, however I did have some time shortly after midnight to see if I had any improvement.

Sadly, I did not, which is actually kinda odd because at that hour, things are usually "ok", nothing great mind you, but better than afternoons/evenings at least.

I was getting consistent 25-40% packet loss using both https://packetlosstest.com/ as well as the built in test on the Xbox. Even had a few 75-90% loss results as well from both. I'm hoping I can check things out today during the times its usually at its worse.

My brother downstate from me told me yesterday that he thought things were somewhat better during the late afternoon hours , but not by much, and became an anger fest about 9pm, lol. He's in the Northville area (MI).

While its "nice" to have this finally acknowledged officially tho, I honestly have no faith in att to really fix it properly. I honestly think they were hoping us "simple rurals" wouldn't notice. HA! I don't mean to be a downer or anything, just with the way I personally have been treated when talking to them about this, as well as their denial of any issues until the FCC got on 'em, well I just don't trust them. I wont be surprised one bit if they beg for another extension.

I will happily love to be proven wrong tho!

Just wanted to chime in that, I too, saw a definite improvement in service quality from 3PM - 11PM but absolutely not back to normal. Tentatively encouraged that we're moving in the right direction.

So It’s 2:30pm right now and I just ran I speed test getting 10 Mbps download and 15 upload with 1% packetloss. My Download speeds have always been significantly higher than my upload (usually around 40 Mbps) idk if they could be working on the service right now or what but Download speed has took a hit. Will try to test router and see if that helps out

As of this am, fixed wireless is working like it use to. Maybe they have the problem fixed.

Still bad as ever here.

Have been having the same problem in Arkansas for weeks. Sometimes upload starts out about 1.5 or 2 Mbps and then drops to practically nothing. I have a Sprint hotspot I was just about to cancel. Glad I didn't. The fixed wireless was great the first year or so. Since yesterday I've had periods where the upload stayed fairly steady but it's quite bad again tonight.

It’s 8:30 rn and the packetloss is high once again. I seen some improvements today but nope still the same stuff..

We'll see if tomorrow is a fluke.

Chat this morning "checked the team" and said it would be next week and they would let me know when it was fixed with an email. And that it wouldn't happen again. lol

If you look at the forum leaderboard, several of us in this thread are in the top 10-20 amongst all the actual employees. That's how long this problem has been going on and how much we've had to post about it. lol

https://forums.att.com/leaderboard?dateFrom=LAST_30_DAYS&page=1&sortOrder=DESC

I received and answered the email. Results:

ping 491 DL 3.97 Mbps UL .10 Mbps

UL packet loss 7.4% DL packet loss 31.2% Total 36.2% late packets 63.8%

Central Arkansas a little over a mile from the tower

@ATTHelpForums BRO WELCOME TO THE PARTY

@ATTHelpForums Thanks for the reply although not a fan of your choice of wording here, you realize that our delay is on your end right? And as others have said it has nothing to do with my browser, I see it on windows, ios, android. It is present in file downloads and uploads as well as gaming.

Also you are not fixing it for us, you are fixing it because it’s yours, you are selling it and it’s broken.

I’m concerned you don't have a grasp of the entirety of the problem, therefore it should not be captioned as an “official solution” as noted in your post.

and scope of impact is FAR in excess of " delay when their browser is loading" ... the understatement of the year ... read and it is clear you do not need a browser to be impacted .. .. how many users complained they cannot do their gaming .. that is just tip of the "NOT" browsing iceberg

And not just browsing and gaming, I use it for work (and pay the commercial rate) and couldn't even download a 40mb python library necessary for code that was being deployed the next morning. I had to load up my equipment and go to someone else place at 930pm just to get connectivity. It's not just a delayed loading of a browser...it is, at times, not even functional.

@ATTHelpForums A delay with their browser loading? The title of the thread explicitly says packet loss.

With hundreds of responses to this thread about consistent issues (packet loss, latency, interruptions, inaccessibility), AT&T support still can't accurately communicate the problems.

The lack of a solution makes a lot more sense now...

@ATTHelpForums Glad to see someone at ATT is aware of the problem. Just glad to see someone admit there's a problem without saying reboot the equipment over and over. Still it's aggravating to pay for service that goes belly up every evening.

@ATTHelpForums Ayyyyy my IP address is now allocating me in Redmon Washington...Im in Houston lol Packet loss...can't stream...can't game...thanks for violating me with the monthly bill! Of course they are "working on it" this has been since late December

02/27/2020 ~06:14 CST almost nonexistent upload capability

From AT&T's speed test:

Your Test Results (Tested on 2/27/2020, 6:14:00 AM)

Download Speed HELP 61 Mbps (7625 KB/sec transfer rate)

Upload Speed HELP 0.01 Mbps (1.3 KB/sec transfer rate)

Latency HELP 38 ms

@ATTHelpForums @ATTHelpFeedback I understand you are aware of the issue, but this does not give us a timeline of when to expect the issue to big fixed. Some of us that depend on the internet cannot sit around and wait for a year for you to fix the issue. Can you please update us with some type of timeline on when the issue will be resolved. Everyone in my neighborhood is canceling their service with you and moving to another company in our area. We do not want to switch, but without any timeframe for a fix it is hard to stay.

@ATTCommunityTeam I did not get an email, but I'll gladly take one, do the test, and reply with the results.

@ATTCommunityTeam I replied to to the e-mail but figured I'd post my results here as well for others to view:

1. Speed: Upload: 2.95 Mbps, Download: 8.13 Mbps

2. Packet Loss: Upload: 12.1%, Download: 19.6%. TOTAL: 28.8%

3. Mississippi

@ATTCommunityTeam If you want accurate results then you should be asking for tests from DSLreports that includes bufferbloat. It shows the problem much clearer than speedtest.net.

Bufferbloat while under load is a big contributor to the issue, and only DSLreports seems to measure this in their test.

@unleadedgass I'm not sure how you got speeds that low, since my speeds seem normal levels when testing but it shows the massive latency issue going on at the same time. That's why I don't use speedtest.net as a source of information anymore. DSLreports seems to be the most accurate, and it's endorsed by AT&T themselves (as you can see on the right on the test page).

They are also the only ones that measure bufferbloat DURING the test rather than the beginning/end.

@espespesp on the DSLReport test site, which connection type do we use for this fixed wi-fi? That connection type is not listed.

@dusty1usa I find it doesn't really matter much as long as you're not picking Satellite. I usually pick 4G but get the same results picking "Cable". There is no Fixed Wireless type as you mentioned.

Thanks. I'll give it a try.

I did get an email this afternoon asking me to conduct a speed test and then a packet loss test. I did both and replied to them with the results.

I also told them that for the last two days my connection has been almost normal in the afternoons. Which it has been. I did power cycle the antenna box day before yesterday.

They replied to my reply already thanking me for the info on the tests and said that they hope I continue to see improvement.

Ping - 89ms Down - 26.76Mbps Up - 3.55Mbps

Total packet loss - 14.8% Latency - 172.57 Jitter - 68.79ms

To the people who received discounts for complaining, how easy was it?

I called and the support are aware of the situation now. I asked for a complainer discount (I should try finesse next-time).

They went on about a credit and that they'd make a note of my complaint for a future credit when I call back when the Fixed Wireless Internet is fixed.

Sounds plausible I suppose.

Per email request:

Time: 8:50 pm central

Location: North-central Alabama

PING ms

54

DOWNLOAD Mbps

9.93

UPLOAD Mbps

18.59

Progress

Sent

148

/ 149

99%

Time

0:10

/ 0:10

100%

Received

126

/ 149

85%

Results

Upload Packet Loss

🠥 4.7%

(7 / 148)

Total Packet Loss

🠥🠧 11.5%

(17 / 148)

Download Packet Loss

🠧 7.1%

(10 / 140)

Late Packets

⏱ 3.4%

(5 / 148)

Average Latency: 115.55ms

Average Jitter: 19.74ms

I got the email too, when I ran the tests I got less than 2Mbps down

Download Mbps

1.69

Upload Mbps

15.24

From just a few minutes ago. Northern Michigan

Edit to show single connection speedtest result. Forgot to do it that way, lol

speedtest picked georgia for your server?

@BadInnernet Yeah...it always picks somewhere in the south. Usually Kentucky, but sometimes Georgia, Nashville, or Alabama. Whenever I let stuff use my location while on fixed wireless, it always thinks I'm from one of those states.