Fixed Wireless High Packet Loss

We had fixed wireless installed last week (northern MS). We are having the same issue as you are all seeing with your setups. I keep watching this thread hoping one of you will get some kind of explanation or (*cough*) a solution. I really didn't want to go through all of the trouble with support if all they are going to run a speed test and say everything is good. Since we have a 14 day trial period, I guess we will give this a few more days and see if anyone has any luck. If not, we will tell them to come get their equipment.

@john9999 When it works, it's great.

Try unplugging the power to the antenna box (not the router, the other one) only and reconnecting. This somehow made mine work last night.

1:30pm central - 4.72 down / 23.77 up.

6:30 pm central - 16.2 down / 21.43 up.

@bamashooter have you tried anything that anyone has recommended or just getting speed test readings?

So far since the random disconnect last night around 7:30PM central, I have maintained under 2% packetloss to most locations and up to 3% for some others. Haven't done any heavy upload traffic yet to really test it out - will be doing that when we get back there later today.

If you're having high packetloss still can you try powering off just the ONT (antenna) for 2 minutes and then turning it back on? Wait up to 5 minutes after powering it back on to check your packetloss again. Make sure to leave everything else powered up and online like normal, only power cycle the ONT (antenna) and leave it off for 2 minutes before turning it back on. Please let me know what you run into?

@dennisharrison this has been my exact experience so far. Unplugging antenna power seems to reset something, just not sure what.

@twright33 It resets the open ports for your NAT pooled IP. There is/was (?? still trying to figure out if it's been fixed) a timeout (and possibly MTU, more testing was required to say for certain) mismatch on the equipment 3 hops in for most of us (so it seems to me as of now).

Take all my information with a grain of salt, because I haven't done enough packet introspection and route testing to have high confidence in my assumptions yet.

@unleadedgass I have a feeling that something with a lot of UDP packets is going to cause this issue faster than some other types of usage. Would be curious as to how fast you hit what % packetloss with and without WOW after the ONT reset.

ive decided to make a account after looking at this thread for a week i have fixed wireless aswell to game on and i havent been able to for over a month i ive 200 yards away from a tower and i get 80 down 20 up and 50 ping and weekdays 5-12pm its unplayable kicked from everygame and weekends its unusable at any time. i called for a tech to come and when he got here he said everyones having problems and that theres nothing he can do and that when i searched and found this thread showing me were all having the same issues thats when i filed a FCC complaint for a ATT executive to called me the next day act like he didnt know about any of the problems i said were happening united states wide said a fixed wireless team would contact in 2 days and never did but today a tower technician took down the cellphones and fixed wireless internet to handle a fix on the tower cause of my complaint i decided to drive down to the tower and ask him about all this stuff for him to tell me he doesnt think there will be a fix for a year or 2 cause of how hard there pushing to roll 5g out saying he thinks were having these troubles is cause theres to many people on fixed wireless and there turning downsare fixed wireless speeds cause there trying to roll out 5g

the tower tech told me to just keep comlaining to the FCC and to not stop till they do something orther than that he doesnt think there is a fix

@Spunts95 being as ATT was paid $420,000,000 from the Connect American program to create this service, I doubt this is going to go unnoticed and not be fixed for awhile.

The service is not working as intended. They aren't going to just let it not work for a year.

Oh I 100% agree with you I’m just going off what the tower tech told me,I think the problem should be fixed already but I don’t think ATT cares and they know about the problem. We have to ride the FCC till they demand ATT does something

@Spunts95 My tech told me that is was caused from PlayStation and XBox users with their "high speed gaming". After that, I don't think the techs know much about what's going on.

Well, back to an absolute crap show now! Resetting the ONT doesn't do a darn thing for me (though I did pick up the same IP again, will try leaving it off for longer and see if that makes some sort of difference if we can get a new IP).

FRUSTRATING - but I won't be able to really dig into this further without spending some time with a device to replace the Pace which may not be until this weekend depending on the weather and the construction crew out here.

where are you making these complaints to the FCC ?

I have approx 30% packet loss thins evening and just spoke with Rameses, one of the social media folks who reached out to me aboug a different thread I started.

1008 bytes from 157.240.18.35: icmp_seq=0 ttl=49 time=83.262 ms

1008 bytes from 157.240.18.35: icmp_seq=1 ttl=49 time=60.694 ms

Request timeout for icmp_seq 2

1008 bytes from 157.240.18.35: icmp_seq=3 ttl=49 time=82.914 ms

1008 bytes from 157.240.18.35: icmp_seq=4 ttl=49 time=65.696 ms

1008 bytes from 157.240.18.35: icmp_seq=5 ttl=49 time=90.327 ms

1008 bytes from 157.240.18.35: icmp_seq=6 ttl=49 time=180.740 ms

1008 bytes from 157.240.18.35: icmp_seq=7 ttl=49 time=119.594 ms

1008 bytes from 157.240.18.35: icmp_seq=8 ttl=49 time=173.461 ms

1008 bytes from 157.240.18.35: icmp_seq=9 ttl=49 time=60.898 ms

1008 bytes from 157.240.18.35: icmp_seq=10 ttl=49 time=87.812 ms

1008 bytes from 157.240.18.35: icmp_seq=11 ttl=49 time=57.788 ms

1008 bytes from 157.240.18.35: icmp_seq=12 ttl=49 time=71.912 ms

1008 bytes from 157.240.18.35: icmp_seq=13 ttl=49 time=68.587 ms

1008 bytes from 157.240.18.35: icmp_seq=14 ttl=49 time=142.023 ms

Request timeout for icmp_seq 15

1008 bytes from 157.240.18.35: icmp_seq=16 ttl=49 time=199.252 ms

Request timeout for icmp_seq 17

1008 bytes from 157.240.18.35: icmp_seq=18 ttl=49 time=295.171 ms

1008 bytes from 157.240.18.35: icmp_seq=19 ttl=49 time=276.380 ms

1008 bytes from 157.240.18.35: icmp_seq=20 ttl=49 time=321.226 ms

Request timeout for icmp_seq 21

Request timeout for icmp_seq 22

1008 bytes from 157.240.18.35: icmp_seq=23 ttl=49 time=62.064 ms

1008 bytes from 157.240.18.35: icmp_seq=24 ttl=49 time=79.492 ms

1008 bytes from 157.240.18.35: icmp_seq=25 ttl=49 time=69.529 ms

^C

--- fbcdn.net ping statistics ---

27 packets transmitted, 21 packets received, 22.2% packet loss

round-trip min/avg/max/stddev = 57.788/126.134/321.226/81.395 ms

New IP address, ~24% packetloss across 5 targets.

0 packets transmitted, 21 packets received, 30.0% packet loss

round-trip min/avg/max/stddev = 57.368/102.477/239.787/55.446 ms

@twright33 No. Other than disconnect / connect router/modem and antenna, I don't know how to do any of that stuff others are doing. This morning @ 7am:

23 down / 24 up.

OK, so the only reason resetting the ONT/Modem/Antenna does anything is that it can give you a new IP address that isn't (for lack of a better term as of yet) "poisoned". If you get the same IP - you will see no change in performance (I think).

Can you guys play around with this a bit once your connection gets crappy again today and see if better performance is related to a new IP?

For example I'm at 10.91.219.59 with a /29 subnet so that's 4 hosts encapsulated by 2 addresses and they have the gateway as 10.91.219.60.

Throughout the course of today my packetloss will get worse and worse the more I send/upload.

Once that happens, no matter peak time or not (based upon previous days testing - so it's just assumptions still), I should be able to cycle the ONT/Modem/Antenna until it gives me a different IP (2 minutes of downtime seems to do it) and if that IP hasn't been used by someone else in X hours (I think? Possibly since last reset of whatever equipment is handing them out as well?) then the packetloss will drop back down to single digits during peak and sub 1% off peak.

Weird, right?

I did the reset on the antenna as well last night and I can confirm the 2 minutes to change your ip. I’ve just been using the ping tool on cmd the last couple days and pinging google 100x to get an accurate packet loss %. Before I reset the antenna I was 43% loss. After resetting and getting a new IP I was 6% and I ran it in 15 min intervals for an hour until I was back up to 30%. I’m going to try resetting and doing tracert this evening when I get home and see what I come up with. I’ve been just using google as my control for my tests on cmd and ping plotter.

I've only reset my antenna once 48 hours ago and still have less than 10% packet loss after running pingplotter for a couple of hours.

It's not fixed, but it's a bandaid.

Still no word back from my FCC complaint or technician visit.

@twright33 Are you back down to less than 10% or did it never go above 10% during that 48 hour period?

Does the packet loss you do have start at att server still?

@dennisharrison I know it went above (some spikes), but the PL % is an average. There will be some 10 minute intervals with higher packet loss than others, but I haven't had any disconnects during that time while online gaming. Just some buffering with streaming apps.

@KP45697 I had a similar experience last night, had to get a new IP again after the packetloss shot back up over an hour - but this one has been fine since. Which is why I'm thinking it needs to be one that hasn't been used in X hours or since reset of state table for whatever equipment handles this routing.

I'd suggest hitting more than google, I'm pinging akamai, 8.8.8.8, pingplotter.com, and twitter.com as well. Some will be much lower packetloss than others at times.

Ping Plotter from this morning

@twright33 OK, that's different than my experience. I can force the packetloss issue by opening more connections. Using this connection for work (stupid, I know) but I'm desperate to not drive into the city and rent an office just to sit at a computer all day ;p

How many devices do you have using the connection during a typical 24 hour period? We're at around 12 here between phones, tablets, set top boxes, consoles, laptops, and whatever like those google assistants.

Currently getting a setup to log all this in a more complete way. pcaps and whatnot on a device I can control the interface to the ONT and shape local traffic to test some assumptions.

In the meantime I'm headed to pick up a postpaid sim from an MVNO to put into a cradlepoint and see if it works better than the fixed wireless at different times.

@dennisharrison We have two phones, a tablet, two smart tvs, and my gaming PC connected on this network.

Are you running Windows 10? Try resetting your network settings so you know for sure there is nothing PC related making the issue worse.

Okay I will add those as well when I get home this evening and run another 12 hour test between peak and overnight. The last one I ran on ping plotter I started at 4:30pm and ran until 12:30am and it clear as day starts spiking at 6 and by 7 it is dropping packets consistently enough that using the internet for any real time applications is impossible. But by 11pm all the packet loss was gone and I continued to let it run over night and by the next morning I hadn’t experienced any loss at all. This would’ve all been on the same ip. My concern is still the possibility that all traffic for fixed wireless could be being sent through the same servers and they don’t have enough of them. My theory is maybe they did before the end/beginning of the year but when they began work to upgrade towers for new gen wireless, ATT decided to just slam everyone on the wireless network through a select few servers. Which would explain how all of a sudden this is for the people that have had service for a year or better. I may be completely wrong and I will continue to pump the representatives for any info I can get. But I guess that would be worst case because then we have no choice but to wait for the gen 2 fixed wireless.

@twright33 I have a couple machines here - two of them run wintendos10 :) I've just been doing this so far with the windows 10 laptop from the couch in the evenings.

Once I have the pfSense box setup in place of the Pace and a few hours to setup some pcap logging and scripts based upon iperf results with a server I have for troubleshooting like this - we will have a lot more useful info on what is actually going on. I used to do this sort of thing for a living (medium sized networking for wireless ISPs).

I wish we could get some insight from someone internal to the network topology for fixed wireless and how it differs from other internet products from AT&T on the implementation perspective.

Thought I'd post the results I see with/without my router connected to the vpn.

These numbers are taken from https://us-looking-glass.battle.net

Apparently I cannot post the results of MTR or Traceroutes, the info was just wiped out when I submitted.... Real classy (I obfuscated any of my IPs)

Anyway, after the tech left the PACE powered off for 20 minutes, when I run the test without the VPN, I am currently not seeing any dropped packets. Still takes 5 more hops to resolve and a higher overall trip time, than when I have the VPN activated (NordVPN UDP profile on an Asus RT-AC86U

I will run this on my pc when I get home. The tech that they sent out twice to my place (first time was before I realized this problem is nationwide) told me basically the same thing as the yours did. He had no idea how to actually diagnose the issues we are having. Really he only knew how to replace equipment and run speed tests.

@connectionless_in_bfe If your packetloss gets too high you won't be able to keep the VPN up most likely, which has me curious as to if you just tunnel all your traffic through a VPN if it prevents the steady packetloss climb I'm seeing. Will give this a shot - thanks!

I've been through many setups and configurations over the years, I've been through exactly all of the same issues. The fixed wireless was great for the first 3-4 months, it's obvious now that AT&T is just overselling the service to the point that it's not really any better than having a cellular hot spot. (Arguably tethering to my Verizon service on my phone is much better, at least for my individual needs, but then you're not really getting enough data allowance)

Frankly, I'm just disgusted with the obvious abuses and exploitation that these monopoly services get away with, and now they're getting more subsidies to do it. I'd rather see that money go to new competition to balance the playing field, because there really aren't any other viable options for people in situations like ours.

@twright33 There may be some areas which are oversold, but not to the point of causing these types of packetloss issues. This is a configuration issue in just a few parts of our routes. I mean, I'm sure AT&T would rather just have our money instead of having to provide a service for it - but they (fortunately) have to provide the service :)

It's raining HEAVILY here right now and I've got less than 1% packetloss, and the internet is in excellent condition. So odd, but at least now I can totally rule out antenna pointing, power levels, RSRQ/RSRP/SINR/etc.

That's also very true. There's no way in December when everyone started having issues the service was all of a sudden oversold. Exactly as you have said, this is some type of configuration issue. Their lack of acknowledgment on this forum about the issue leads me to believe they 100% know about the problem but just don't know when or how to fix it.

When I called yesterday to check the status of my escalation there was no update but they took this month's internet charge off of my ATT bill. No questions asked.

Everything seems good (speed-wise). 31 down and 24 up. 8:30-9:00 am.

@bamashooter test packet loss easy here - https://packetlosstest.com/

@twright33 Thanks, I'll run it now.

@twright33 Thanks for that link..... this also confirms 36.2% packet loss. Speed test means nothing when you are dropping packets like that... Impossible to play an online game like this.

@KP45697 Keep trucking!

My neighbor is an AT&T tech and we're opening a ticket every day.

Everyone here should keep calling them and chatting/emailing them every day too. I am blown away that they aren't sending out company memos about what they're doing with the connections though - because at this point we know they know.

Nice! I’m in a pretty small town so most of the techs are local I just don’t know any of them personally. I wish I did to get some more insight on how it works but I’m not so sure they even know what happens beyond the towers

@KP45697 Just got off the phone with a rep w/ the president's office @ AT&T. They're trying to nail down which group to assign this too and are actively engaged in getting a response/answer. Their protocol is to respond either when something changes or 2 business days if there are no changes/updates otherwise.

I provided the link to this forum post and will update here withy any extra info.

Good job everyone - we got them to pay attention and I'm betting they'll get it fixed!

@unleadedgass Call them. They reimbursed me a full month and gave me a $10/mo discount for a year

Awesome!

Having the techs assigned is a winner - they'll be able to get this sorted out in no time.

just got my 2nd call after about a week regarding my FCC complaint talked to a manager at ATT executive office asking if the tower technician that came out a few days ago fixed the issue I told him it didn’t cause i still get packet loss and told him we’re all dealing with packet loss and that’s the problem not my speeds. Praying for a fix soon

@twright33 I think this is just a line they have given us to tide us over for a month or two.

@unleadedgass South Central Indiana as well. (Though, I'm sure if we were to throw darts at a map blindfolded, that location would probably have an issue too! ;)

Per packetlosstest.com //////// No idea what it means though I hope it helps the cause.

Results

Upload Packet Loss

🠥 5.4%

(8 / 149)

Total Packet Loss

🠥🠧 16.1%

(24 / 149)

Download Packet Loss

🠧 11.3%

(16 / 141)

Late Packets

⏱ 4.0%

(6 / 149)

Average Latency: 127.64ms

Average Jitter: 31.16ms

@malleable_malus so your router firmware had reset to pre issues time and everything was working fine?

@KP45697 That is certainly what it seems like.

You can even go back and find my post when I said how odd it was when I got e-mailed about my settings being changed on my router.

Found my post:

malleable_malus 2 weeks ago

I just got an e-mail saying a change was made to my gateway. I'm wondering what this is about.

That's when my problems went away. I checked my settings and they were all normal except that my firmware had been rolled back.

Well, here I am now and the problem has returned and my firmware is back to 11.5

When I was on 11.4, my connection was great (simultaneous streaming video, downloading files, and playing online games), even in prime use time. I am not exaggerating.

Now I'm back on 11.5 and I can't even load webpages.

I'm not saying the firmware is the only cause of this but it's the only thing I can personally observe.

I'm tempted to manually downgrade the firmware myself but that is probably not a great idea.

@malleable_malus Question, have you run ping plotter or WinMTR? Most of us are getting packet loss at the 4th or 5th hop and are fine on our home to the network.

@malleable_malus that’s pretty interesting. I don’t know why your firmware would’ve reset to a previous version unless att changed it. If that were the case they should just change all of the fixed wireless pace routers back to 11.4. I’m only skeptical of that because when running ping plotter I don’t have any packet loss from the home portal. Only once it reaches att servers. But who knows at this point. I’m just glad it seems like they have at least acknowledged that there is a problem with the service. I’m going to call billing today and see if I can get some kind of compensation for dealing with this mess.