Reply
Posted Feb 14, 2014
5:49:31 AM
IPv6 Tunnels broken yet again (this time on the NVG589 modem)

So AT&T has done it again it would seem - After loosing my ipv6 tunnel to hurricane electric via the 2wire firmware mess, It would seem now the NVG589 modems are affected.  I've not yet been able to tell if its the firmware or just AT&T internally dropping protocol 41 packets not bound to their rd endpoint, but either way it would seem that any ipv6 tunnels not sanctioned by AT&T are down for me.  Has anyone else seen this, and have a clue if theres any workaround?  At this point, even in the light of the potential comcast merger, I think Time Warner would be a better solution than this.

IPv6 Tunnels broken yet again (this time on the NVG589 modem)

2,388 views
27 replies
(0) Me too
(0) Me too
Post reply
Replies
(27)
0
(0)
  • Rate this reply
View profile
Feb 14, 2014 4:36:49 PM
0
(0)
Tutor
Edited by tdd37 on Feb 14, 2014 at 5:18:02 PM

I also have the NVG589 and can confirm they're filtering protocol 41 (6in4) traffic as of around midnight last night.  I worked with a Hurricane Electric engineer today to figure out what's going on.

 

 

I add the following rule to the top of my firewall rules:

 

    -A INPUT -i eth1 -p 41 -j ACCEPT

 
Where eth1 is the U-Verse connection.  Did an ifdown/ifup he-ipv6, then started pinging HE's side of the tunnel while monitoring eth1 for protocol 41 traffic.  I see no responses from HE's side:

 

    root@tiferet:/var/log# tcpdump -eni eth1 ip proto 41
    14:30:13.021629 00:14:d1:b0:b8:9d > 38:6b:bb:3a:7b:f0, ethertype IPv4
        (0x0800), length 138: MY.IPv4.ADD.RESS > 184.105.253.10:
        MY:HE:IPv6:ADD:RESS::2 > HEs:END:0F:THE:TUNNEL::1: ICMP6, echo request,
        seq 1, length 64
    14:30:14.030289 00:14:d1:b0:b8:9d > 38:6b:bb:3a:7b:f0, ethertype IPv4
        (0x0800), length 138: MY.IPv4.ADD.RESS > 184.105.253.10:
        MY:HE:IPv6:ADD:RESS::2 > HEs:END:0F:THE:TUNNEL::1: ICMP6, echo request,
        seq 2, length 64
    etc.

 

And of course the counter for the firewall rule doesn't increment:

    root@tiferet:~# iptables -L -n -v | grep 41
        0     0 ACCEPT     41   --  eth1   *       0.0.0.0/0    0.0.0.0/0

 

And I do have reachability to the tunnel endpoint:

    root@tiferet:~# ping 184.105.253.10
    PING 184.105.253.10 (184.105.253.10) 56(84) bytes of data.
    64 bytes from 184.105.253.10: icmp_req=1 ttl=51 time=59.6 ms
    64 bytes from 184.105.253.10: icmp_req=2 ttl=51 time=59.7 ms
    etc.

 



Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

[ Edited ]
2 of 28 (2,350 Views)
0
(0)
  • Rate this reply
View profile
Feb 14, 2014 4:57:41 PM
0
(0)
Expert
Edited by SomeJoe7777 on Feb 14, 2014 at 4:59:48 PM

I do not have an NVG589, so I can't confirm the issue. I do have a 3800HGV-B which stopped passing protocol 41 several months ago.

I worked around the issue by utilizing an IPv6 connection that I have at work. I formed an encrypted IPv4 IPSec tunnel between myself and work, and then assigned myself an IPv6 subnet at home. (Work has a /56 IPv6 prefix, so I took one of the 256 available networks for myself). I then formed an IPv6 tunnel between myself and work, and this traffic runs inside the encrypted IPv4 tunnel.

Now, my IPv4 packets carrying my IPv6 are not type 41 (IPv6 in IPv4), but instead are type 50 (ESP). The U-Verse routers are not blocking type 50 since it's used for many VPN implementations.

This increased my latency on IPv6 by about 30 msec (15 msec to and from work). But it works.  The UVRealtime website is available on IPv6 through this setup, at http://www.uvrealtime.com (dual stack) or specifically on IPv6 at http://ipv6.uvrealtime.com .

I wonder if someone could approach HE-Net and see if they're willing do do something similar with GRE (IP type 47). You could then do IPv6 inside IPv4 inside GRE and the U-Verse routers wouldn't block it. Type 47 is also used in many VPN implementations.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

[ Edited ]
3 of 28 (2,347 Views)
0
(0)
  • Rate this reply
View profile
Feb 14, 2014 7:37:10 PM
0
(0)
Tutor

SomeJoe7777 wrote:

I wonder if someone could approach HE-Net and see if they're willing do do something similar with GRE (IP type 47).


Done.  Here's what they say:

 

"We'd tried GRE (for PPTP) in the past, but the increased overhead of doing that vs vanilla 6in4 was surprisingly high.  Not in actual bandwidth, but system load.  The increase in interrupts and CPU load kicking it back through the IP stack was just too great at our scale, unless we wanted to operate ~10% the number of tunnels per server."

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

4 of 28 (2,319 Views)
0
(0)
  • Rate this reply
View profile
Feb 14, 2014 8:42:37 PM
0
(0)
Teacher

I'm seeing the same thing. When testing with ICMPv6 echo requests and tcpdump on the source and target systems, I see the request make it to the target and the target send the echo reply, but that response never makes it back to the source system. So it looks like the filtering is only inbound.

 

My RG hasn't rebooted in 18 days and my tunnel didn't die until 1 AM CST, so I'm guessing the filtering is taking place upstream from the RG within AT&T's network.

 

I'm losing patience with these guys.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

5 of 28 (2,308 Views)
0
(0)
  • Rate this reply
View profile
Feb 16, 2014 8:33:56 AM
0
(0)
Contributor

Confirmed in Columbus, Ohio. My Tunnel died this weekend as well.

I found that nothing had been rebooted in over 2 weeks, was troubleshooting my end of the connection and found that it's definitely upstream.

 

This is awful, as I've been through this before with my original 2Wire modem, and then after fighting up the chain got them to replace it with the Motorola, which has been great until now.

 

It may be time to ditch AT&T since they can't seem to let their customers use their connections for what they want/need. Very sad.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

6 of 28 (2,213 Views)
0
(0)
  • Rate this reply
View profile
Feb 16, 2014 1:20:19 PM
0
(0)
Teacher

Yeah, I only had the NV589 for 18 days before they broke it again.

 

I just got off the phone a few minutes ago signing up with the other provider in my area.  Promotional pricing is $30/mo for 50 mbit, native IPv6, installed this coming Saturday.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

7 of 28 (2,197 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 4:07:49 AM
0
(0)
Mentor

So I've been on the phone with AT&T which normally doesn't help at all, but in this case I've found a supervisor that is actually trying to be very helpful.  Shes calling me every few days while she attempts to figure out who in network operations to contact to make a query about this.  I'd like to help her as much as we can, since she's really going out of her way here.  FWIWwe checked and confirmed that its not a firmware update to the RG (like the hvg3800-B was), as the NVG589 hasn't had its firmware updated, so that suggests a definate network operations change.  In an amusing  twist I received a call from tech360.att.com a few months ago indicating malicious traffic from my site that suggested a compromise (which there was), but during the search for the cause they were more than happy in their response to provide me with network traces of teh offending traffic, which suggests they have the capability to do said traces.  I'm wondering if contacting them might lead to some traction here.  I'll let you know how that goes, and if you are so inclined, pelase do the same

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

8 of 28 (2,084 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 6:58:47 AM
0
(0)
Employee
IPv6 protocol 41 is currently disabled on all RGs, there is another post above (Somejoe7777) about trying to use a different protocol
*I am an AT&T employee and the postings on this site are my own and don’t necessarily represent AT&T’s position, strategies or opinions.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

9 of 28 (2,070 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 7:16:38 AM
0
(0)
Mentor

That doesn't seem to be accurate. Tthe NVG hasn't had a firmware update in months (according to your support organization), and all of our private tunnels were working up until  around feb 12th.  Given that, and the fact that AT&T's ipv6 is working to my rg, which uses Rapid Deployment (on the same protocol identifier), it would seem that something internal to AT&T's network is dropping any protocol 41 traffic destined to anything other than your own RD gateway, whcih has broken us all.  Can you please investigate why this was done, and indicate if/when you plan to correct this oversight?

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

10 of 28 (2,069 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 8:00:51 AM
0
(0)
Tutor

my thoughts wrote:
IPv6 protocol 41 is currently disabled on all RGs, there is another post above (Somejoe7777) about trying to use a different protocol

There's no setting I can see that does that on the NVG589, no firmware update was pushed out on the night our tunnels stopped working (the uptime on my gateway was 60+ days), and the gateway is very obviously doing something over protocol 41 over the WAN right now (Rapid Deployment, as neilhorman mentioned) as evidenced by this being in its NAT sessions table:

 

IP Family:  ipv4
Protocol Number:  41
Lifetime:  599
Source Address:  192.168.254.254
Destination Address:  12.83.49.81

When I attempt to bring up my own 6in4 tunnel to Hurricane Electric, I see it appear in the gateway's NAT sessions table as well:

 

IP Family:  ipv4
Protocol Number:  41
Source Address:  MY.PUBLIC.IPv4.ADDRESS
Destination Address:  TUNNEL.ENDPOINT.IPV4.ADDRESS

...but traffic over that tunnel does not traverse the WAN and back.  The HE network engineer and I sniffed traffic on both sides to confirm.

 

All evidence suggests that protocol 41 is being filtered further inside AT&T's network.

 

The HE network engineer also offered this:

 

"There was a long running thread on NANOG about AT&T's 6in4 tunnel breakage, as well as on the U-verse support forums.  It may be that with their 6rd deployment moving along, they're either intentionally or erroneously also grabbing other 6in4 traffic.  The NANOG thread does have someone who did manage to eventually get through to [an AT&T network engineer], but it took them a while of escalations and mentions of FCC complaints to get there."

 

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

11 of 28 (2,064 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 8:23:17 AM
0
(0)
Mentor

tdd37, thanks for the info, for the sake of posterity, can you post links to the NANOG and AT&T forum threads here, so others can find them and read up on the subject?  Thanks!

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

12 of 28 (2,057 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 8:39:09 AM
0
(0)
ACE - Master

The U-verse forum thread to which they are referring is probably this one:

 

http://forums.att.com/t5/Features-and-How-To/UVerse-and-IPv6-Tunneling-with-3800-HGV-B/td-p/3511251/

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

13 of 28 (2,052 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 9:40:42 AM
0
(0)
Employee
Perhaps did not stand clearly
" IPv6 protocol 41 is currently disabled on all RGs, there is another post above (Somejoe7777) about trying to use a different protocol"

Meaning no RG currently has access to protocol 41, not necessarily a software update to RG, limited understanding is security issue, possible breach exits to ATT network, all 41 not allowed, no ETA known.

For general reading pleasure...
http://www.ietf.org/rfc/rfc4942.txt
*I am an AT&T employee and the postings on this site are my own and don’t necessarily represent AT&T’s position, strategies or opinions.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

14 of 28 (2,045 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 9:46:46 AM
0
(0)
Employee
http://www.att.com/esupport/article.jsp?sid=KB409112&cv=812
Does AT&T favor certain Internet applications by blocking, throttling or modifying particular protocols on its broadband Internet access service?

No, AT&T does not favor certain Internet applications by blocking, throttling or modifying particular protocols, protocol ports, or protocol fields in ways not prescribed by the protocol standards. However, in response to a specific security threat against our network or our customers, AT&T may occasionally need to limit the flow of traffic from certain locations or take other appropriate actions. For example, while extending IPv6 capabilities through a software update made to U-verse gateway devices, we discovered an issue with the software code that temporarily prevents us from continuing our IPv6 deployment for those devices and may disrupt third party IPv6 tunneling capabilities for some U-verse customers until we are able to update the software. Our IPv4 routing is unaffected and the deployment of IPv6 functionality will resume as soon as the corrective measures have been deployed. In addition, we prevent the use of certain ports on our wired and Wi-Fi broadband services to protect our customers and network against malicious activity. For more information about the network practices and performance characteristics of the AT&T network, visit our page.
*I am an AT&T employee and the postings on this site are my own and don’t necessarily represent AT&T’s position, strategies or opinions.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

15 of 28 (2,043 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 10:03:50 AM
0
(0)
Tutor

my thoughts wrote:
Meaning no RG currently has access to protocol 41, ...

This is still not true.  Our 6in4 tunnels were working fine until Friday February 14th at around 1am.  My Hurricane Electric tunnel had been up and working fine for the 6+ months I'd had U-verse service.

 

In addition, as I already explained, the gateway is using protocol 41 right now for AT&T's Rapid Deployment, confirmed by the contents of the NAT sessions table.

 

It was apparently known before that 2Wire gateways weren't allowing protocol 41 traffic through (I never had a 2Wire gateway, unlike others in this thread who switched to the NVG589 just for this purpose), but this is now clearly being filtered further inside AT&T's network, not at the gateway.

 


my thoughts wrote:
For example, while extending IPv6 capabilities through a software update made to U-verse gateway devices, we discovered an issue with the software code that temporarily prevents us from continuing our IPv6 deployment for those devices and may disrupt third party IPv6 tunneling capabilities for some U-verse customers until we are able to update the software. Our IPv4 routing is unaffected and the deployment of IPv6 functionality will resume as soon as the corrective measures have been deployed.


Um.  Huh?  This seems to say that you know you broke it, and you're going to stop moving forward until you fix it.  ...?

 

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

16 of 28 (2,035 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 10:11:08 AM
0
(0)
Mentor

agree with tdd, while this might be the best reason you can come up with, its still not acceptable. The security concerns outlined in the referenced RFC are:

 

a) vague (i.e. not actionable)

b) affect endpoints only, not intermediate transport networks

c) are not in any way timely (i.e. that rfc was written 7 years ago), but our tunnels only stopped working last week.

 

 

So, to summarize your statement, you found some flimsy RFC citing vague security concerns regarding tunnels, that anyone who has the wherewithall to support their own tunnel already knows about, and used it as an excuse to disable private tunnels entirely, while convieniently excempting your own 6rd tunnels from those same security concerns.  You've broken our networks in the guise of saving us from ourselves, without providing us any additional security.  Thats unacceptable.  Please provide us with a contact point to lodge an official complaint regarding this behavior.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

17 of 28 (2,032 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 1:06:38 PM
0
(0)
Teacher

When someone from AT&T called me about my FCC complaint regarding this, they used the same "security vulnerability" excuse. Specifically, they said that by using an IPv6 tunnel we're being allocated a static IPv6 address by a third party that allows access back into our home network, bypassing any security controls AT&T has in place. It's obviously a weak argument, since that's the entire point of PPTP, GRE, OpenVPN, IPSEC, or any other VPN technology that isn't also being banned.

 

Someone on dslreports.com said that since people are filing FCC complaints about this issue, AT&T has to respond with a "valid" reason for breaking our tunnels. "Security" is likely just the fastest way to get there.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

18 of 28 (2,009 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 1:13:41 PM
0
(0)
Mentor

Can you post a link with directions to file such a complaint?

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

19 of 28 (2,008 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 1:25:08 PM
0
(0)
Teacher

http://www.fcc.gov/complaints

 

I filed my complaint before the Verizon ruling that heralded the beginning of the end of the open Internet (I am so dramatic!), but I filled out the online form under Broadband, Billing/Service/Availability.

 

A large portion of the form is dedicated to billing disputes, so I left those blank and filled in item (5) with some text about how AT&T's behavior is a clear violation of the "no blocking" open Internet rules, as set forth here:

 

http://www.fcc.gov/openinternet

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

20 of 28 (2,001 Views)
0
(0)
  • Rate this reply
View profile
Feb 18, 2014 2:20:36 PM
0
(0)
Mentor

Done, thank you, I've filed an FCC complaint there, I'll keep everyone apprised of how that turns out.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

21 of 28 (1,986 Views)
0
(0)
  • Rate this reply
View profile
Feb 19, 2014 8:02:51 AM
0
(0)
Contributor
Edited by bsdbum on Feb 19, 2014 at 8:08:19 AM

So the story goes like this...Last year on my home network I lost IPv6 Internet connectivity due to a firmware upgraded to my Motorola 3800 RG (ISP supplied router) which was filtering Protocol 41. In order to continue service with my ISP I upgraded my connection to a new 45Mbit "Power Tier" which came with a new Motorola NVG589 RG that no longer filtered Protocol 41. Great! Since the upgrade (about 3-4 months) I have been operating 2 IPv6 tunnels dual homed to SixXS and HE and have not had anymore interuption until Feburay 14th when once again my IPv6 access was disabled.  This time its not a firmware upgrade of my RG but rather a filter of some sort on the upstream provider (AT&T Uverse).  My only choice now is to change my ISP or wait out my FCC complaint to be resolved I wonder which will happen first...CAN YOU HEAR ME NOW AT&T!?!?

 

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

[ Edited ]
22 of 28 (1,957 Views)
0
(0)
  • Rate this reply
View profile
Apr 25, 2014 4:11:51 PM
0
(0)
Contributor

My story is exactly the same as just about everybody elses. I upgraded to the "power" service (for more money) when it became available in my neighborhood last week. I got the shiny new NVG589 that I promptly put into cascade mode and sent my static /29 to a Cisco 2901 on my home network. Net result? It's faster (when line 1 and line 2 will bond properly, which is iffy), but still no gif tunnel to OCCAID for my IPv6 tunnel. I am not going to subject myself to the pain of dealing with AT&T for anything more technical than a billing problem, so I have gone straight to the FCC complaint site as well. We'll see what happens.

Meanwhile, the "security" excuse is terminally laughable.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

23 of 28 (1,244 Views)
0
(0)
  • Rate this reply
View profile
Apr 27, 2014 1:49:28 PM
0
(0)
Mentor

I've since given up.  AT&T is just useless here.  They have no interest in fixing this.  I've dropped my service with them and filed an FCC complaint.  I'm using Time Warner Cables 50 Mb/s service now for the time being.  Its shared media, so latency jitter is a bit higher than AT&T was, but nothing is blocked which is great.  In a year I'll let go of TWC as Google fiber is coming to my area.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

24 of 28 (1,223 Views)
0
(0)
  • Rate this reply
View profile
May 4, 2014 2:19:08 AM
0
(0)
Contributor

Update: Confirmed with OCCAID that the gif tunnel is up and they can see my ping6 echos. The ping6 replies are not making it back to me however, which suggests the 6to4 is being either filtered or intercepted in the AT&T network in only one direction. 

 

Does anybody from AT&T want to step up to acknowledge this and provide info on a remedy? You people have a lot of hacked off customers out here. 

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

25 of 28 (1,150 Views)
0
(0)
  • Rate this reply
View profile
May 26, 2014 7:44:02 PM
0
(0)
Teacher

Those who are more knowledgeable than myself may step in and comment / correct me but, if the ATTs network were being stressed by supporting U-verse TV – perhaps due to bandwidth limitations? – would shutting down IPv6 traffic buy them some time to upgrade or ovoid the cost of upgrading equipment?

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

26 of 28 (1,005 Views)
0
(0)
  • Rate this reply
View profile
May 27, 2014 3:42:39 AM
0
(0)
Mentor

Thats a fair question, but the answer is no, since they didn't shutdown ipv6 traffic, they only shutdown ipv6 traffic being routed by ipv6 tunnels that they didn't own.  The ipv6 service that is natively offered by AT&T still works just fine.  That fact also makes their argument about blocking non-AT&T ipv6 tunnels due to security risks patently false.

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

27 of 28 (981 Views)
0
(0)
  • Rate this reply
View profile
May 31, 2014 2:43:55 AM
0
(0)
Teacher
Thanks - didn't expect a reply :smileyhappy:

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

28 of 28 (944 Views)
Share this post
Share this post