01-14-2011 8:29 AM
This happened to me on Wednesday night. I thought I'd share this story because it illustrates both the sensitivity of U-Verse VDSL to sources of interference and also illustrates an ad-hoc troubleshooting procedure that may help find these things.
I'm watching TV on Wednesday night, and about 8:30 I get a picture freeze. Not entirely uncommon, I see it a few times a month. Usually comes back within 10-15 seconds. Not so in this case. I wait 2-3 minutes, picture is still frozen. So I think there might be some sort of outage. I go to the computer and try to bring this forum up ... no Internet either.
So I browse the forum on my smartphone. No mention of any outage. So I bring up my 2Wire RG's web interface. No VDSL connection ... go look at the RG, blinking red broadband light. Uh oh. Not good.
I watch the RG for a few more minutes, and the line is going up and down. It'll go blinking green, then solid green, then get a solid service light and be up for 10-15 seconds. But then it loses it again and goes back to a blinking red broadband light.
I log into the RG and look at the C53 page for DSL diagnostics ( http://192.168.1.254/xslt?PAGE=C_5_3 ), there's a huge number of retrains listed there. Normally, my line connects with a Max Rate of about 49,000 kbps. The first retrain initially showed a max rate of 43,000, and each subsequent retrain is going down. I have about 10 retrains listed, and the max rate is down to 37,000 on the most recent one.
I look at the bitloading during a brief period where the line was up and there's a large notch/hole in the graph in the 100 kHz - 300 kHz range, which is way down low in the spectrum. This is some sort of interference, and because the max rate is going down with each retrain, that means the interference is getting worse.
So now I have to try to track down this interference, because at this point I have no TV and no Internet. I go outside and look around and see if there's anyone doing anything odd, like running a generator or something. Don't see anything. I figure the interference must be coming from somewhere inside my house but I have no idea where.
The first thing I try is to see if there's power line interference that's causing it. I have the RG on a UPS, so I unplug the UPS from the wall and let the RG run on battery power. This isolates the RG from any power problems. No dice - line keeps going down, and at this point, it really isn't coming up. The broadband light is cycling between flashing red and flashing green, but I'm not even briefly getting the service light anymore.
So I think maybe the interference could be coming through Ethernet connections. I disconnect the Ethernet connections to the RG, no change. Only thing left is the VDSL line itself, and I have a home run on that directly to the NID. The only other thing in that path is the alarm system, I even go so far as to shut down the alarm system and remove power from it just to test, but it didn't make any difference.
So I figure there must be some other component in the house that's doing it. Fine -- go outside to my circuit breaker box and start turning stuff off. A/C, Furnace, Water Heater, Dryer, Washer, Dishwasher/Garbage Disposer, all major stuff. I leave only 4 breakers turned on, they're the ones for outlets & lights.
Go back inside -- no change. Line is still down.
OK, go back outside, shut off the last 4 breakers. Now I have no power in the house, but the RG is still up on the UPS.
Line comes up in about 20 seconds and stays up.
So I think Wow, I have it isolated. Turn all breakers back on except the 4 for lighting and outlets. Go in and check the RG, line is still up. So I start going back outside and turning the last 4 breakers on one at a time and checking the RG between each one. Living room - no problem. Garage/bathrooms - no problem. Bedrooms 2 & 3 - no problem. Turn on the master bedroom - line goes down. Turn the breaker back off, line comes back up.
OK, so now I have to divide up the master bedroom. I figure the likely source is the TV/STB/Slingbox/Popcorn Hour area, so I unplug the whole power strip there and turn the breaker back on. Line stays up. Cool, so at this point it's isolated in that mini-entertainment center.
I unplug everything from that power strip and plug the strip by itself back in. I wait 3-5 minutes just to make sure, but the line stays up. I then start plugging devices back in one at a time and waiting 3-5 minutes between each one to see what happens.
Line goes down about 2 minutes after I plug one of the Slingboxes in.
I go to unplug it and the power supply brick of the Slingbox is way hotter than normal. I figure the power supply has gone south. Line comes back up by the way.
So I go check the RG's parameters -- totally normal. 49,000 kbps max rate, smooth bitloading graph. I have found the culprit.
This troubleshooting procedure took me about 2 hours to go through. The amazing thing is what this illustrates:
• The Slingbox has absolutely no physical connection to the RG or any of its wiring. It's on a different power circuit, it isolated from the rest of the network by a switch, and any other cables only connect to the TV. The interference here was completely via RF, and was being received by the telephone wiring acting as an antenna.
• VDSL and the 2Wire RGs are highly sensitive to interference from a variety of sources, including sources that are too weak to affect anything else. But even these weak sources are enough to cause VDSL to fail.
• Troubleshooting this type of interference is time-consuming and beyond the scope of AT&T premises technicians. Interference of this type will require the homeowner to troubleshoot.
Solved by: Go to Solution.
01-14-2011 9:01 AM
Excellent rundown on your fix, thanks!
Since the brick appears to have been radiating, you might have been able to use an AM Radio to chase the noise too.
I have used devices like the SystemWare Power Line Antenna in conjunction with oscilloscope and / or spectrum analyzers to characterize the electrical noise (knowing what it looks like often helps to figure out what's creating it).The PLA basically removes everything that looks like normal AC power and sends the remaining noise & glitch (up to 50MHz) to a high or low power output (high-O'scope, low - SA).
The other "best friend" for chasing this kind of noise is a Radar Engineers EMI/RFI sniffer (I use a 242), for indoor work, something like a 245 is really all you'd need though.
Radar Engineers: http://www.radarengineers.com/rfitvi.htm
I don't suppose I could sweet-talk you into sending me the offending brick? I'll sweep it and post the pics if you do. If you're interested, let me know and we can work out the details.
01-14-2011 10:06 AM
Awesome Joe! I am an AM (Mediumwave) broadcast band DX'er and 30+ years ago when I was in high school, I had horrible interference in certain spots of the AM band. I could not even listen to some semi-local stations. I did what you did and isolated the house first, confirmed it was internal and then did a breaker at a time and once that was isolated, went plug by plug and stumbled upon a "touch-on" lamp that I had gotten for my mother. Someone apparently had went one touch beyond "off" and it was on a very dim setting (could not even tell that the light was on, unless it was totally dark). Obviously the dimmer / step switch was a horrific offender. That lamp never saw the light of day, again, so to speak!
Now back to Joe's post - how does the average, non-tech person know what to do, if their U-Verse setup runs into something like SJ's (maybe the PS in a printer goes bad)?
01-14-2011 10:28 AM
No offense, but you went way beyond what 99pct of homeowners can or should do. Hopefully AT&T will see your comments and take them into consideration, because no homeowner should have to do what you did. AT&T should bulletproof their system so this doesn't happen (i.e. interference of any kind).
01-14-2011 11:15 AM
Thanks for putting that up, Joe. Amazing how complex things have become. To a non-tech dufus like me, it seems impossible to understand, but it's clear that a little logic and persistence will pay off.
- edited 01-14-2011 11:56 AM
That's the thing ... how many power bricks or other innocuous electronic devices are in the average house these days? 20? 30? More?
And if just one of them fails in a certain way, VDSL goes bye-bye.
Maybe the premises techs should be able to get ahold of and use an RF signal finder like Scott was talking about. Maybe this kind of interference could be found in a short period of time with one of those, and techs could then actually do this kind of troubleshooting.
Scott, the RF locators you referred to look like great units but I'm not sure they would have worked here. The specs list the lowest frequency that those will pick up at 500 kHz, but this interference was well beneath that, down near 100-200 kHz (from looking at the notch/hole on the bitloading graph). I will PM you regarding the brick, though.
01-14-2011 12:03 PM
SJ I was reading to see if you ever rebooted the rg, but due to your vast knowledge of how this system works it wasn't necessary. I would never have thought of shutting down circuit breakers to find the source of the freezing. Thanks for yet another area to troubleshoot when services get squirrelly.
01-14-2011 12:12 PM
150khz is right in the center of RFI produced by many common incandescent light dimmers. I went through my home a couple of years ago and replaced about a half dozen of these cacophonous dimmers with newer (low-RF) ones and noise on my phone line and AM radio in my workshop went away. I'm not positive, but I think it also contributed to a reduced error count on my RG as well.
01-14-2011 1:25 PM
RC, I actually did reboot the RG after all was said and done to clear any lingering effects and clear the retrains table. But the line was up before I rebooted, so it probably would have been OK.
BeeBee, I actually only have 2 clocks to reset. My main alarm clock in the bedroom has a built-in battery backup, so no reset needed there. No VCRs anywhere. Only two clocks that need a reset are stove & microwave, and those take 30 seconds.
Most of the time was waiting the necessary 3-5 minutes after a change to be sure that the line would either stay up or not. If you go too fast with flipping breakers or plugging stuff in you might not be sure which item caused the line status to change.
01-15-2011 5:20 AM
I would of never figured that one out!!!! Good job man. I would of been contacting Tier 2 with a flurry lol. It amazes me how sensitive VDSL is and yet my drop from the ped to my NID is unshielded. Thank God I have had no probs like this "knocks on wood".
01-17-2011 5:33 AM
I bet that power supply has a bad capacitor in it. They're used to filter RF from the AC line. I had an old cordless phone years ago that had 60 cycle hum in it nearly all the time. Opened it up and found a capacitor that had popped (the tops of them usually bulge when they go). I replaced it and never had another problem with that phone again.
01-17-2011 7:24 AM
Well, the 100-300 kHz interference doesn't exist on the AC line, so a bad capacitor that has failed in its filtering function wouldn't cause this.
This type of power brick is transformerless, which means it probably has a switching power supply in it. Switching power supplies perform voltage conversion using pulse width modulation of the current, and then a capacitor-inductor network to average the voltage to a new lower value. My hunch is that the inductor may have opened. An open inductor would cease the filtering/average function and instead act as an antenna. With a 200 kHz switching frequency, the inductor would begin broadcasting.
Visit these related resourcesView Internet Speed Help!