Reply
Posted Jan 14, 2011
8:29:19 AM
Tracking Down Line Interference

This happened to me on Wednesday night.  I thought I'd share this story because it illustrates both the sensitivity of U-Verse VDSL to sources of interference and also illustrates an ad-hoc troubleshooting procedure that may help find these things.

 

I'm watching TV on Wednesday night, and about 8:30 I get a picture freeze.  Not entirely uncommon, I see it a few times a month.  Usually comes back within 10-15 seconds.  Not so in this case.  I wait 2-3 minutes, picture is still frozen.  So I think there might be some sort of outage.  I go to the computer and try to bring this forum up ... no Internet either. :smileysurprised:

 

So I browse the forum on my smartphone.  No mention of any outage.  So I bring up my 2Wire RG's web interface.  No VDSL connection ... go look at the RG, blinking red broadband light.  Uh oh.  Not good.

 

I watch the RG for a few more minutes, and the line is going up and down.  It'll go blinking green, then solid green, then get a solid service light and be up for 10-15 seconds.  But then it loses it again and goes back to a blinking red broadband light.

 

I log into the RG and look at the C53 page for DSL diagnostics ( http://192.168.1.254/xslt?PAGE=C_5_3 ), there's a huge number of retrains listed there.  Normally, my line connects with a Max Rate of about 49,000 kbps.  The first retrain initially showed a max rate of 43,000, and each subsequent retrain is going down.  I have about 10 retrains listed, and the max rate is down to 37,000 on the most recent one.

 

I look at the bitloading during a brief period where the line was up and there's a large notch/hole in the graph in the 100 kHz - 300 kHz range, which is way down low in the spectrum.  This is some sort of interference, and because the max rate is going down with each retrain, that means the interference is getting worse.

 

So now I have to try to track down this interference, because at this point I have no TV and no Internet.  I go outside and look around and see if there's anyone doing anything odd, like running a generator or something.  Don't see anything.  I figure the interference must be coming from somewhere inside my house but I have no idea where.

 

The first thing I try is to see if there's power line interference that's causing it.  I have the RG on a UPS, so I unplug the UPS from the wall and let the RG run on battery power.  This isolates the RG from any power problems.  No dice - line keeps going down, and at this point, it really isn't coming up.  The broadband light is cycling between flashing red and flashing green, but I'm not even briefly getting the service light anymore.

 

So I think maybe the interference could be coming through Ethernet connections.  I disconnect the Ethernet connections to the RG, no change.  Only thing left is the VDSL line itself, and I have a home run on that directly to the NID.  The only other thing in that path is the alarm system, I even go so far as to shut down the alarm system and remove power from it just to test, but it didn't make any difference.

 

So I figure there must be some other component in the house that's doing it.  Fine -- go outside to my circuit breaker box and start turning stuff off.  A/C, Furnace, Water Heater, Dryer, Washer, Dishwasher/Garbage Disposer, all major stuff.  I leave only 4 breakers turned on, they're the ones for outlets & lights.

 

Go back inside -- no change.  Line is still down.

 

OK, go back outside, shut off the last 4 breakers.  Now I have no power in the house, but the RG is still up on the UPS.

 

Line comes up in about 20 seconds and stays up.

 

So I think Wow, I have it isolated.  Turn all breakers back on except the 4 for lighting and outlets.  Go in and check the RG, line is still up.  So I start going back outside and turning the last 4 breakers on one at a time and checking the RG between each one.  Living room - no problem.  Garage/bathrooms - no problem.  Bedrooms 2 & 3 - no problem.  Turn on the master bedroom - line goes down. :smileysurprised:  Turn the breaker back off, line comes back up.

 

OK, so now I have to divide up the master bedroom.  I figure the likely source is the TV/STB/Slingbox/Popcorn Hour area, so I unplug the whole power strip there and turn the breaker back on.  Line stays up.  Cool, so at this point it's isolated in that mini-entertainment center.

 

I unplug everything from that power strip and plug the strip by itself back in.  I wait 3-5 minutes just to make sure, but the line stays up.  I then start plugging devices back in one at a time and waiting 3-5 minutes between each one to see what happens.

 

Line goes down about 2 minutes after I plug one of the Slingboxes in.

 

I go to unplug it and the power supply brick of the Slingbox is way hotter than normal.  I figure the power supply has gone south.  Line comes back up by the way.

 

So I go check the RG's parameters -- totally normal. 49,000 kbps max rate, smooth bitloading graph.  I have found the culprit.

 

 

This troubleshooting procedure took me about 2 hours to go through.  The amazing thing is what this illustrates:

 

• The Slingbox has absolutely no physical connection to the RG or any of its wiring.  It's on a different power circuit, it isolated from the rest of the network by a switch, and any other cables only connect to the TV.  The interference here was completely via RF, and was being received by the telephone wiring acting as an antenna.

 

• VDSL and the 2Wire RGs are highly sensitive to interference from a variety of sources, including sources that are too weak to affect anything else.  But even these weak sources are enough to cause VDSL to fail.

 

• Troubleshooting this type of interference is time-consuming and beyond the scope of AT&T premises technicians.  Interference of this type will require the homeowner to troubleshoot.

 

 

0
(0)
  • Rate this reply
Solved
Jan 14, 2011 9:01:41 AM
0
(0)
Employee

Excellent rundown on your fix, thanks!

 

Since the brick appears to have been radiating, you might have been able to use an AM Radio to chase the noise too.

 

I have used devices like the SystemWare Power Line Antenna in conjunction with oscilloscope and / or spectrum analyzers to characterize the electrical noise (knowing what it looks like often helps to figure out what's creating it).The PLA basically removes everything that looks like normal AC power and sends the remaining noise & glitch (up to 50MHz) to a high or low power output (high-O'scope, low - SA).

 

The other "best friend" for chasing this kind of noise is a Radar Engineers EMI/RFI sniffer (I use a 242), for indoor work, something like a 245 is really all you'd need though.

 

SystemWare PLA: http://www.caci.com/systemware/systemware-products/power-line-antenna.html

 

Radar Engineers:  http://www.radarengineers.com/rfitvi.htm

 

I don't suppose I could sweet-talk you into sending me the offending brick? I'll sweep it and post the pics if you do. If you're interested, let me know and we can work out the details.

 

 

*I am an AT&T employee and the postings on this site are my own and don’t necessarily represent AT&T’s position, strategies or opinions.
Accepted Solution

Tracking Down Line Interference

12,085 views
54 replies
(0) Me too
(0) Me too
Post reply
Replies
(54)
0
(0)
  • Rate this reply
Solved
Jan 14, 2011 9:01:41 AM
0
(0)
Employee

Excellent rundown on your fix, thanks!

 

Since the brick appears to have been radiating, you might have been able to use an AM Radio to chase the noise too.

 

I have used devices like the SystemWare Power Line Antenna in conjunction with oscilloscope and / or spectrum analyzers to characterize the electrical noise (knowing what it looks like often helps to figure out what's creating it).The PLA basically removes everything that looks like normal AC power and sends the remaining noise & glitch (up to 50MHz) to a high or low power output (high-O'scope, low - SA).

 

The other "best friend" for chasing this kind of noise is a Radar Engineers EMI/RFI sniffer (I use a 242), for indoor work, something like a 245 is really all you'd need though.

 

SystemWare PLA: http://www.caci.com/systemware/systemware-products/power-line-antenna.html

 

Radar Engineers:  http://www.radarengineers.com/rfitvi.htm

 

I don't suppose I could sweet-talk you into sending me the offending brick? I'll sweep it and post the pics if you do. If you're interested, let me know and we can work out the details.

 

 

*I am an AT&T employee and the postings on this site are my own and don’t necessarily represent AT&T’s position, strategies or opinions.

Re: Tracking Down Line Interference

2 of 55 (12,069 Views)
Solution
0
(0)
  • Rate this reply
Jan 14, 2011 10:06:23 AM
0
(0)
ACE - Master

Awesome Joe!  I am an AM (Mediumwave) broadcast band DX'er and 30+ years ago when I was in high school, I had horrible interference in certain spots of the AM band.  I could not even listen to some semi-local stations.  I did what you did and isolated the house first, confirmed it was internal and then did a breaker at a time and once that was isolated, went plug by plug and stumbled upon a "touch-on" lamp that I had gotten for my mother.  Someone apparently had went one touch beyond "off" and it was on a very dim setting (could not even tell that the light was on, unless it was totally dark).  Obviously the dimmer / step switch was a horrific offender.  That lamp never saw the light of day, again, so to speak!

 

Now back to Joe's post - how does the average, non-tech person know what to do, if their U-Verse setup runs into something like SJ's (maybe the PS in a printer goes bad)?

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.

Re: Tracking Down Line Interference

3 of 55 (12,044 Views)
0
(0)
  • Rate this reply
Jan 14, 2011 10:28:39 AM
0
(0)
Scholar

SomeJoe,

 

No offense, but you went way beyond what 99pct of homeowners can or should do.  Hopefully AT&T will see your comments and take them into consideration, because no homeowner should have to do what you did.  AT&T should bulletproof their system so this doesn't happen (i.e. interference of any kind).

Re: Tracking Down Line Interference

4 of 55 (12,031 Views)
0
(0)
  • Rate this reply
Jan 14, 2011 11:15:02 AM
0
(0)
Guru

Thanks for putting that up, Joe.  Amazing how complex things have become.  To a non-tech dufus like me, it seems impossible to understand, but it's clear that a little logic and persistence will pay off.

Re: Tracking Down Line Interference

5 of 55 (12,008 Views)
0
(0)
  • Rate this reply
Jan 14, 2011 11:46:34 AM
0
(0)
Expert
Edited by SomeJoe7777 on Jan 14, 2011 at 11:56:48 AM

That's the thing ... how many power bricks or other innocuous electronic devices are in the average house these days?  20?  30?  More?

 

And if just one of them fails in a certain way, VDSL goes bye-bye.

 

Maybe the premises techs should be able to get ahold of and use an RF signal finder like Scott was talking about.  Maybe this kind of interference could be found in a short period of time with one of those, and techs could then actually do this kind of troubleshooting.

 

 

Scott, the RF locators you referred to look like great units but I'm not sure they would have worked here.  The specs list the lowest frequency that those will pick up at 500 kHz, but this interference was well beneath that, down near 100-200 kHz (from looking at the notch/hole on the bitloading graph).  I will PM you regarding the brick, though.

 

Re: Tracking Down Line Interference

[ Edited ]
6 of 55 (11,989 Views)
0
(0)
  • Rate this reply
Jan 14, 2011 12:03:20 PM
0
(0)
ACE - Expert

SJ I was reading to see if you ever rebooted the rg, but due to your vast knowledge of how this system works it wasn't necessary.  I would never have thought of shutting down circuit breakers to find the source of the freezing.  Thanks for yet another area to troubleshoot when services get squirrelly.

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.

Re: Tracking Down Line Interference

7 of 55 (11,974 Views)
0
(0)
  • Rate this reply
Jan 14, 2011 12:12:17 PM
0
(0)
Guru

150khz is right in the center of RFI produced by many common incandescent light dimmers. I went through my home a couple of years ago and replaced about a half dozen of these cacophonous dimmers with newer (low-RF) ones and noise on my phone line and AM radio in my workshop went away. I'm not positive, but I think it also contributed to a reduced error count on my RG as well.

Re: Tracking Down Line Interference

8 of 55 (11,965 Views)
0
(0)
  • Rate this reply
Jan 14, 2011 12:50:13 PM
0
(0)
ACE - Expert
SJ...Did this 2 hours include resetting all of your digital clocks, stove clock, etc.??

:smileyvery-happy:

 


SomeJoe7777 wrote:

 

This troubleshooting procedure took me about 2 hours to go through. 

"If you find this post helpful and it solved your issue please mark it as a solution.  This will help other forum members locate it and will also let everyone know that it corrected your problem. If they have the same issue they will know how to solve theirs"

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.

Re: Tracking Down Line Interference

9 of 55 (11,944 Views)
0
(0)
  • Rate this reply
Jan 14, 2011 1:25:25 PM
0
(0)
Expert

RC, I actually did reboot the RG after all was said and done to clear any lingering effects and clear the retrains table.  But the line was up before I rebooted, so it probably would have been OK.

 

BeeBee, I actually only have 2 clocks to reset.  My main alarm clock in the bedroom has a built-in battery backup, so no reset needed there.  No VCRs anywhere.  Only two clocks that need a reset are stove & microwave, and those take 30 seconds.

 

Most of the time was waiting the necessary 3-5 minutes after a change to be sure that the line would either stay up or not.  If you go too fast with flipping breakers or plugging stuff in you might not be sure which item caused the line status to change.

 

Re: Tracking Down Line Interference

10 of 55 (11,922 Views)
0
(0)
  • Rate this reply
Jan 14, 2011 6:58:53 PM
0
(0)
Employee

SJ it amazes me how you get 48 hrs of activity on a slow day in a 24 hr day.

*I am an AT&T employee and the postings on this site are my own and don’t necessarily represent AT&T’s position, strategies or opinions.

Re: Tracking Down Line Interference

11 of 55 (9,528 Views)
0
(0)
  • Rate this reply
Jan 15, 2011 5:20:20 AM
0
(0)
Mentor

I would of never figured that one out!!!!  Good job man.  I would of been contacting Tier 2 with a flurry lol.  It amazes me how sensitive VDSL is and yet my drop from the ped to my NID is unshielded.  Thank God I have had no probs like this "knocks on wood".

Re: Tracking Down Line Interference

12 of 55 (9,492 Views)
0
(0)
  • Rate this reply
Jan 16, 2011 5:38:40 PM
0
(0)
Expert

Excellent!!!  Are you for hire?  j/k

Re: Tracking Down Line Interference

13 of 55 (9,444 Views)
0
(0)
  • Rate this reply
Jan 17, 2011 5:33:02 AM
0
(0)
Scholar

I bet that power supply has a bad capacitor in it. They're used to filter RF from the AC line. I had an old cordless phone years ago that had 60 cycle hum in it nearly all the time. Opened it up and found a capacitor that had popped (the tops of them usually bulge when they go). I replaced it and never had another problem with that phone again.

 

 

Re: Tracking Down Line Interference

14 of 55 (9,423 Views)
0
(0)
  • Rate this reply
Jan 17, 2011 7:24:01 AM
0
(0)
Expert

Well, the 100-300 kHz interference doesn't exist on the AC line, so a bad capacitor that has failed in its filtering function wouldn't cause this.

 

This type of power brick is transformerless, which means it probably has a switching power supply in it.  Switching power supplies perform voltage conversion using pulse width modulation of the current, and then a capacitor-inductor network to average the voltage to a new lower value.  My hunch is that the inductor may have opened.  An open inductor would cease the filtering/average function and instead act as an antenna.  With a 200 kHz switching frequency, the inductor would begin broadcasting.

 

Re: Tracking Down Line Interference

15 of 55 (9,401 Views)
0
(0)
  • Rate this reply
Jan 18, 2011 11:37:55 AM
0
(0)
ACE - Master

Hi SJ.  Glad that you got this tracked down!  Many of us seem to be plagued by more intermittent RG reboots or retrains, where TV and the internet will freeze for a few minutes and the RG will flash red for about 5 seconds.  No rhyme nor reason to when it happens.  Could be 6 times per day or twice per week.

 

Tier 2 noted from my stats that all of my issues causing the RG to go red are upstream, meaning that they are coming from within the house.  Any idea what to check?  Belkin battery backup?  I do not have U-Voice but the installer hooked one up anyway.  He noted that might be the issue.  there is one coax run (upstairs bedroom) that I think is suspect.  Could that be it?

 

I also have noticed an increase in this behavior by subscribers across the country.  Could a parameter / threshold have changed in U-Verse systems that have made the RG more sensitive to RF or signal levels?  Thanks!

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.

Re: Tracking Down Line Interference

16 of 55 (9,336 Views)
0
(0)
  • Rate this reply
Jan 18, 2011 4:57:27 PM
0
(0)
Expert

I'd like to know that, too!  Just had a 35 second outgage here of all services, and no longer just late night. :smileysad:

 

I had an APC battery backup in the kitchen that died, and it was removed over 6 mos. ago.  CyberPower backup is used for the RG and it was ATT supplied.  Wonder if that is causing a problem here, battery going weak?  We do get a lot of power outages in this area of Houston.

Re: Tracking Down Line Interference

17 of 55 (9,313 Views)
0
(0)
  • Rate this reply
Jan 18, 2011 7:57:26 PM
0
(0)
Scholar

I got a multiband radio that I use for a poor mans spectrum analyzer. It has a long wave band that tunes from 150KHz to 500KHz. Then the AM band covers 500KHz to 1700KHz then short wave bands up to 30MHz.

 

It's also a fun toy. I got a Radio Shack DX-398 which is the same thing as the Sangean ATS909.

 

http://www.sangean.com/products/products_main.asp?pid=11&pan=9&um=3 

 

It's pretty old. They seem to hold up pretty well. Mine has an issue with the speaker but I use head phones anyway so it's not an issue.

 

If you use the RG spectrum to find a hole, then you can use the radio to zero in on a device or area in that frequency range. With the radio alone, it can be tough since just about anything electronic makes quite a bit of noise. You can get some sense of how much power is being emitted from the range.

 

The main thing at my place down at the lower end is WTB AM at 1100KHz. It knocks out about 2 or 3 bands but that's not too bad and its consistent. I'm about 6 miles from the antenna but I get them all over the place while trouble shooting on a good day. LOL

 

Larry

 

 

Re: Tracking Down Line Interference

18 of 55 (9,299 Views)
0
(0)
  • Rate this reply
Jan 19, 2011 8:48:33 AM
0
(0)
ACE - Master

Yes the DX-398 / ATS909 sounds like a cool radio.  A DX'er in Canada uses one to DX FM signals via meteor scatter using the built in RDS display 9since Meteor scatter signals in some cases only last for 3-5 seconds).  

 

At any rate - I have never thought of using a multiband radio to track down RFI.  Cool idea.

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.

Re: Tracking Down Line Interference

19 of 55 (9,271 Views)
0
(0)
  • Rate this reply
Jan 19, 2011 10:54:49 AM
0
(0)
Employee
Edited by ScottMac on Jan 19, 2011 at 10:55:25 AM

 


dhascall wrote:

Yes the DX-398 / ATS909 sounds like a cool radio.  A DX'er in Canada uses one to DX FM signals via meteor scatter using the built in RDS display 9since Meteor scatter signals in some cases only last for 3-5 seconds).  

 

At any rate - I have never thought of using a multiband radio to track down RFI.  Cool idea.


 

The trick is to hold the radio in front of you (such that your body blocks the rear of the radio). Most of those radios use a ferrite bar / loopstick antenna that is more sensitive in the front/back direction than from the sides. With the radio in front of you, signals from the front will tend to be the strongest.

 

By walking around and "aiming" the radio you (usually) see the direction the noise is coming from. This works best for impulse type noise (like bad motors, weak flourescent baluns, arcing aquarium heaters / thermostats, bad streetlights, etc) regardless of the actual frequency, because impulse noise tends to be broad spectrum by it's nature.

 

*I am an AT&T employee and the postings on this site are my own and don’t necessarily represent AT&T’s position, strategies or opinions.

Re: Tracking Down Line Interference

[ Edited ]
20 of 55 (9,246 Views)
0
(0)
  • Rate this reply
Jan 29, 2011 2:31:22 PM
0
(0)
Teacher

Hi - How does one reset the bit loading graph?

 

Thanks!

Re: Tracking Down Line Interference

21 of 55 (9,057 Views)
0
(0)
  • Rate this reply
Jan 29, 2011 5:14:01 PM
0
(0)
Expert

 


horntoad wrote:

Hi - How does one reset the bit loading graph?

 

Thanks!


 

You mean after you've removed (or believe you've removed) a source of interference?

 

The only way is to reboot the RG.  The RG seems to adapt to interference only negatively.  In other words, if interference appears, the RG will reduce or eliminate the bitload on those frequencies to avoid the interference.  But, when the interference disappears, the RG will not reclaim those lost bits until the next reboot.

 

Re: Tracking Down Line Interference

22 of 55 (9,035 Views)
0
(0)
  • Rate this reply
Feb 3, 2011 12:24:49 PM
0
(0)
Teacher

Thanks SomeJoe7777! It's been a frustrating investigation. I just rebooted my RG. On the bitloading graph, I have two small holes - one at 230 and one at 576. Surreounding these are numbers 5,6, and 7.

 

In your humble opinion, do you think these small holes would make the RG retrain? No consistent times for the retrains, in fact it happens in the middle of the night most frequently - 12AM, 3:30AM, 6AM. All computers are shutdown, no TV's running.

 

If these holes are the cause, I will eventually get some shielded cat 5e and connect from the NID inside - and ground the shielding at both ends.

 

Thank You -

 

 

 

 

Re: Tracking Down Line Interference

23 of 55 (8,975 Views)
0
(0)
  • Rate this reply
Feb 3, 2011 12:25:53 PM
0
(0)
Expert

Post the screenshot of the bitloading graph and I'll take a look.

 

Re: Tracking Down Line Interference

24 of 55 (8,973 Views)
0
(0)
  • Rate this reply
Feb 3, 2011 12:53:21 PM
0
(0)
Teacher

uverse scr1.JPGThanks, please see below...

 

 

Re: Tracking Down Line Interference

25 of 55 (8,958 Views)
0
(0)
  • Rate this reply
Feb 3, 2011 12:56:02 PM
0
(0)
Teacher

Sorry, one more screen...

 

 

scr2 uverse.JPG

Re: Tracking Down Line Interference

26 of 55 (8,963 Views)
0
(0)
  • Rate this reply
Feb 3, 2011 8:03:43 PM
0
(0)
Expert

Sorry, the bitloading graph that's available on the C_2_5 page of the RG is difficult to read because it is shown very wide.

 

Please download U-Verse Realtime, install it, and post screenshots Bitloading tab from there.

 

Re: Tracking Down Line Interference

27 of 55 (8,909 Views)
0
(0)
  • Rate this reply
Feb 4, 2011 6:44:26 AM
0
(0)
Teacher

Thanks for the tool. Screen shot should be below. I appreciate your interest and assistance!

 

Bitloading-2011-02-04-08-42-12.png

Re: Tracking Down Line Interference

28 of 55 (8,887 Views)
0
(0)
  • Rate this reply
Feb 5, 2011 6:58:21 AM
0
(0)
Teacher

Hi -

 

I installed the tool on my laptop as well, worked once without issue. Since then it displays message cannot open channel db file. Tried uninstalling, reinstalling - same issue.

 

Thanks

Re: Tracking Down Line Interference

29 of 55 (8,826 Views)
0
(0)
  • Rate this reply
Feb 5, 2011 7:29:03 AM
0
(0)
Expert

1. Uninstall the program.

 

2. Delete the following folder:

 

(Windows XP): C:\Documents and Settings\<username>\Local Settings\Application Data\_UVerseRealtime

 

(Windows Vista/7): C:\Users\<username>\AppData\Local\_UVerseRealtime

 

3. Reinstall the program.

 

Re: Tracking Down Line Interference

30 of 55 (8,813 Views)
Share this post
Share this post