IPv6 Tunnels broken yet again (this time on the NVG589 modem)

Mentor

IPv6 Tunnels broken yet again (this time on the NVG589 modem)

So AT&T has done it again it would seem - After loosing my ipv6 tunnel to hurricane electric via the 2wire firmware mess, It would seem now the NVG589 modems are affected.  I've not yet been able to tell if its the firmware or just AT&T internally dropping protocol 41 packets not bound to their rd endpoint, but either way it would seem that any ipv6 tunnels not sanctioned by AT&T are down for me.  Has anyone else seen this, and have a clue if theres any workaround?  At this point, even in the light of the potential comcast merger, I think Time Warner would be a better solution than this.

Message 1 of 32 (4,896 Views)
Tutor

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

[ Edited ]

I also have the NVG589 and can confirm they're filtering protocol 41 (6in4) traffic as of around midnight last night.  I worked with a Hurricane Electric engineer today to figure out what's going on.

 

 

I add the following rule to the top of my firewall rules:

 

    -A INPUT -i eth1 -p 41 -j ACCEPT

 
Where eth1 is the U-Verse connection.  Did an ifdown/ifup he-ipv6, then started pinging HE's side of the tunnel while monitoring eth1 for protocol 41 traffic.  I see no responses from HE's side:

 

    root@tiferet:/var/log# tcpdump -eni eth1 ip proto 41
    14:30:13.021629 00:14:d1:b0:b8:9d > 38:6b:bb:3a:7b:f0, ethertype IPv4
        (0x0800), length 138: MY.IPv4.ADD.RESS > 184.105.253.10:
        MY:HE:IPv6:ADD:RESS::2 > HEs:END:0F:THE:TUNNEL::1: ICMP6, echo request,
        seq 1, length 64
    14:30:14.030289 00:14:d1:b0:b8:9d > 38:6b:bb:3a:7b:f0, ethertype IPv4
        (0x0800), length 138: MY.IPv4.ADD.RESS > 184.105.253.10:
        MY:HE:IPv6:ADD:RESS::2 > HEs:END:0F:THE:TUNNEL::1: ICMP6, echo request,
        seq 2, length 64
    etc.

 

And of course the counter for the firewall rule doesn't increment:

    root@tiferet:~# iptables -L -n -v | grep 41
        0     0 ACCEPT     41   --  eth1   *       0.0.0.0/0    0.0.0.0/0

 

And I do have reachability to the tunnel endpoint:

    root@tiferet:~# ping 184.105.253.10
    PING 184.105.253.10 (184.105.253.10) 56(84) bytes of data.
    64 bytes from 184.105.253.10: icmp_req=1 ttl=51 time=59.6 ms
    64 bytes from 184.105.253.10: icmp_req=2 ttl=51 time=59.7 ms
    etc.

 



Message 2 of 32 (4,746 Views)
Expert

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

[ Edited ]

I do not have an NVG589, so I can't confirm the issue. I do have a 3800HGV-B which stopped passing protocol 41 several months ago.

I worked around the issue by utilizing an IPv6 connection that I have at work. I formed an encrypted IPv4 IPSec tunnel between myself and work, and then assigned myself an IPv6 subnet at home. (Work has a /56 IPv6 prefix, so I took one of the 256 available networks for myself). I then formed an IPv6 tunnel between myself and work, and this traffic runs inside the encrypted IPv4 tunnel.

Now, my IPv4 packets carrying my IPv6 are not type 41 (IPv6 in IPv4), but instead are type 50 (ESP). The U-Verse routers are not blocking type 50 since it's used for many VPN implementations.

This increased my latency on IPv6 by about 30 msec (15 msec to and from work). But it works.  The UVRealtime website is available on IPv6 through this setup, at http://www.uvrealtime.com (dual stack) or specifically on IPv6 at http://ipv6.uvrealtime.com .

I wonder if someone could approach HE-Net and see if they're willing do do something similar with GRE (IP type 47). You could then do IPv6 inside IPv4 inside GRE and the U-Verse routers wouldn't block it. Type 47 is also used in many VPN implementations.

Message 3 of 32 (4,745 Views)
Tutor

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)


SomeJoe7777 wrote:

I wonder if someone could approach HE-Net and see if they're willing do do something similar with GRE (IP type 47).


Done.  Here's what they say:

 

"We'd tried GRE (for PPTP) in the past, but the increased overhead of doing that vs vanilla 6in4 was surprisingly high.  Not in actual bandwidth, but system load.  The increase in interrupts and CPU load kicking it back through the IP stack was just too great at our scale, unless we wanted to operate ~10% the number of tunnels per server."

Message 4 of 32 (4,717 Views)
Teacher

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

I'm seeing the same thing. When testing with ICMPv6 echo requests and tcpdump on the source and target systems, I see the request make it to the target and the target send the echo reply, but that response never makes it back to the source system. So it looks like the filtering is only inbound.

 

My RG hasn't rebooted in 18 days and my tunnel didn't die until 1 AM CST, so I'm guessing the filtering is taking place upstream from the RG within AT&T's network.

 

I'm losing patience with these guys.

Message 5 of 32 (4,706 Views)
Contributor

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

Confirmed in Columbus, Ohio. My Tunnel died this weekend as well.

I found that nothing had been rebooted in over 2 weeks, was troubleshooting my end of the connection and found that it's definitely upstream.

 

This is awful, as I've been through this before with my original 2Wire modem, and then after fighting up the chain got them to replace it with the Motorola, which has been great until now.

 

It may be time to ditch AT&T since they can't seem to let their customers use their connections for what they want/need. Very sad.

Message 6 of 32 (4,611 Views)
Teacher

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

Yeah, I only had the NV589 for 18 days before they broke it again.

 

I just got off the phone a few minutes ago signing up with the other provider in my area.  Promotional pricing is $30/mo for 50 mbit, native IPv6, installed this coming Saturday.

Message 7 of 32 (4,595 Views)
Mentor

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

So I've been on the phone with AT&T which normally doesn't help at all, but in this case I've found a supervisor that is actually trying to be very helpful.  Shes calling me every few days while she attempts to figure out who in network operations to contact to make a query about this.  I'd like to help her as much as we can, since she's really going out of her way here.  FWIWwe checked and confirmed that its not a firmware update to the RG (like the hvg3800-B was), as the NVG589 hasn't had its firmware updated, so that suggests a definate network operations change.  In an amusing  twist I received a call from tech360.att.com a few months ago indicating malicious traffic from my site that suggested a compromise (which there was), but during the search for the cause they were more than happy in their response to provide me with network traces of teh offending traffic, which suggests they have the capability to do said traces.  I'm wondering if contacting them might lead to some traction here.  I'll let you know how that goes, and if you are so inclined, pelase do the same

Message 8 of 32 (4,482 Views)
Employee

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

IPv6 protocol 41 is currently disabled on all RGs, there is another post above (Somejoe7777) about trying to use a different protocol
Employee Contributor*
*I am an AT&T employee and the postings on this site are my own and don't necessarily represent AT&T's position, strategies or opinions.
Message 9 of 32 (4,468 Views)
Mentor

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

That doesn't seem to be accurate. Tthe NVG hasn't had a firmware update in months (according to your support organization), and all of our private tunnels were working up until  around feb 12th.  Given that, and the fact that AT&T's ipv6 is working to my rg, which uses Rapid Deployment (on the same protocol identifier), it would seem that something internal to AT&T's network is dropping any protocol 41 traffic destined to anything other than your own RD gateway, whcih has broken us all.  Can you please investigate why this was done, and indicate if/when you plan to correct this oversight?

Message 10 of 32 (4,467 Views)
Tutor

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)


my thoughts wrote:
IPv6 protocol 41 is currently disabled on all RGs, there is another post above (Somejoe7777) about trying to use a different protocol

There's no setting I can see that does that on the NVG589, no firmware update was pushed out on the night our tunnels stopped working (the uptime on my gateway was 60+ days), and the gateway is very obviously doing something over protocol 41 over the WAN right now (Rapid Deployment, as neilhorman mentioned) as evidenced by this being in its NAT sessions table:

 

IP Family:  ipv4
Protocol Number:  41
Lifetime:  599
Source Address:  192.168.254.254
Destination Address:  12.83.49.81

When I attempt to bring up my own 6in4 tunnel to Hurricane Electric, I see it appear in the gateway's NAT sessions table as well:

 

IP Family:  ipv4
Protocol Number:  41
Source Address:  MY.PUBLIC.IPv4.ADDRESS
Destination Address:  TUNNEL.ENDPOINT.IPV4.ADDRESS

...but traffic over that tunnel does not traverse the WAN and back.  The HE network engineer and I sniffed traffic on both sides to confirm.

 

All evidence suggests that protocol 41 is being filtered further inside AT&T's network.

 

The HE network engineer also offered this:

 

"There was a long running thread on NANOG about AT&T's 6in4 tunnel breakage, as well as on the U-verse support forums.  It may be that with their 6rd deployment moving along, they're either intentionally or erroneously also grabbing other 6in4 traffic.  The NANOG thread does have someone who did manage to eventually get through to [an AT&T network engineer], but it took them a while of escalations and mentions of FCC complaints to get there."

 

Message 11 of 32 (4,462 Views)
Mentor

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

tdd37, thanks for the info, for the sake of posterity, can you post links to the NANOG and AT&T forum threads here, so others can find them and read up on the subject?  Thanks!

Message 12 of 32 (4,455 Views)
ACE - Expert

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

The U-verse forum thread to which they are referring is probably this one:

 

http://forums.att.com/t5/Features-and-How-To/UVerse-and-IPv6-Tunneling-with-3800-HGV-B/td-p/3511251/

*The views and opinions expressed on this forum are purely my own. Any product claim, statistic, quote, or other representation about a product or service should be verified with the manufacturer, provider, or party.
Message 13 of 32 (4,450 Views)
Employee

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

Perhaps did not stand clearly
" IPv6 protocol 41 is currently disabled on all RGs, there is another post above (Somejoe7777) about trying to use a different protocol"

Meaning no RG currently has access to protocol 41, not necessarily a software update to RG, limited understanding is security issue, possible breach exits to ATT network, all 41 not allowed, no ETA known.

For general reading pleasure...
http://www.ietf.org/rfc/rfc4942.txt
Employee Contributor*
*I am an AT&T employee and the postings on this site are my own and don't necessarily represent AT&T's position, strategies or opinions.
Message 14 of 32 (4,443 Views)
Employee

Re: IPv6 Tunnels broken yet again (this time on the NVG589 modem)

http://www.att.com/esupport/article.jsp?sid=KB409112&cv=812
Does AT&T favor certain Internet applications by blocking, throttling or modifying particular protocols on its broadband Internet access service?

No, AT&T does not favor certain Internet applications by blocking, throttling or modifying particular protocols, protocol ports, or protocol fields in ways not prescribed by the protocol standards. However, in response to a specific security threat against our network or our customers, AT&T may occasionally need to limit the flow of traffic from certain locations or take other appropriate actions. For example, while extending IPv6 capabilities through a software update made to U-verse gateway devices, we discovered an issue with the software code that temporarily prevents us from continuing our IPv6 deployment for those devices and may disrupt third party IPv6 tunneling capabilities for some U-verse customers until we are able to update the software. Our IPv4 routing is unaffected and the deployment of IPv6 functionality will resume as soon as the corrective measures have been deployed. In addition, we prevent the use of certain ports on our wired and Wi-Fi broadband services to protect our customers and network against malicious activity. For more information about the network practices and performance characteristics of the AT&T network, visit our page.
Employee Contributor*
*I am an AT&T employee and the postings on this site are my own and don't necessarily represent AT&T's position, strategies or opinions.
Message 15 of 32 (4,441 Views)