02-14-2014 5:49 AM
So AT&T has done it again it would seem - After loosing my ipv6 tunnel to hurricane electric via the 2wire firmware mess, It would seem now the NVG589 modems are affected. I've not yet been able to tell if its the firmware or just AT&T internally dropping protocol 41 packets not bound to their rd endpoint, but either way it would seem that any ipv6 tunnels not sanctioned by AT&T are down for me. Has anyone else seen this, and have a clue if theres any workaround? At this point, even in the light of the potential comcast merger, I think Time Warner would be a better solution than this.
02-14-2014 4:36 PM - edited 02-14-2014 5:18 PM
I also have the NVG589 and can confirm they're filtering protocol 41 (6in4) traffic as of around midnight last night. I worked with a Hurricane Electric engineer today to figure out what's going on.
I add the following rule to the top of my firewall rules:
-A INPUT -i eth1 -p 41 -j ACCEPT
Where eth1 is the U-Verse connection. Did an ifdown/ifup he-ipv6, then started pinging HE's side of the tunnel while monitoring eth1 for protocol 41 traffic. I see no responses from HE's side:
root@tiferet:/var/log# tcpdump -eni eth1 ip proto 41 14:30:13.021629 00:14:d1:b0:b8:9d > 38:6b:bb:3a:7b:f0, ethertype IPv4 (0x0800), length 138: MY.IPv4.ADD.RESS > 22.214.171.124: MY:HE:IPv6:ADD:RESS::2 > HEs:END:0F:THE:TUNNEL::1: ICMP6, echo request, seq 1, length 64 14:30:14.030289 00:14:d1:b0:b8:9d > 38:6b:bb:3a:7b:f0, ethertype IPv4 (0x0800), length 138: MY.IPv4.ADD.RESS > 126.96.36.199: MY:HE:IPv6:ADD:RESS::2 > HEs:END:0F:THE:TUNNEL::1: ICMP6, echo request, seq 2, length 64 etc.
And of course the counter for the firewall rule doesn't increment:
root@tiferet:~# iptables -L -n -v | grep 41 0 0 ACCEPT 41 -- eth1 * 0.0.0.0/0 0.0.0.0/0
And I do have reachability to the tunnel endpoint:
root@tiferet:~# ping 188.8.131.52 PING 184.108.40.206 (220.127.116.11) 56(84) bytes of data. 64 bytes from 18.104.22.168: icmp_req=1 ttl=51 time=59.6 ms 64 bytes from 22.214.171.124: icmp_req=2 ttl=51 time=59.7 ms etc.
02-14-2014 4:57 PM - edited 02-14-2014 4:59 PM
I do not have an NVG589, so I can't confirm the issue. I do have a 3800HGV-B which stopped passing protocol 41 several months ago.
I worked around the issue by utilizing an IPv6 connection that I have at work. I formed an encrypted IPv4 IPSec tunnel between myself and work, and then assigned myself an IPv6 subnet at home. (Work has a /56 IPv6 prefix, so I took one of the 256 available networks for myself). I then formed an IPv6 tunnel between myself and work, and this traffic runs inside the encrypted IPv4 tunnel.
Now, my IPv4 packets carrying my IPv6 are not type 41 (IPv6 in IPv4), but instead are type 50 (ESP). The U-Verse routers are not blocking type 50 since it's used for many VPN implementations.
This increased my latency on IPv6 by about 30 msec (15 msec to and from work). But it works. The UVRealtime website is available on IPv6 through this setup, at http://www.uvrealtime.com (dual stack) or specifically on IPv6 at http://ipv6.uvrealtime.com .
I wonder if someone could approach HE-Net and see if they're willing do do something similar with GRE (IP type 47). You could then do IPv6 inside IPv4 inside GRE and the U-Verse routers wouldn't block it. Type 47 is also used in many VPN implementations.
02-14-2014 7:37 PM
I wonder if someone could approach HE-Net and see if they're willing do do something similar with GRE (IP type 47).
Done. Here's what they say:
"We'd tried GRE (for PPTP) in the past, but the increased overhead of doing that vs vanilla 6in4 was surprisingly high. Not in actual bandwidth, but system load. The increase in interrupts and CPU load kicking it back through the IP stack was just too great at our scale, unless we wanted to operate ~10% the number of tunnels per server."
02-14-2014 8:42 PM
I'm seeing the same thing. When testing with ICMPv6 echo requests and tcpdump on the source and target systems, I see the request make it to the target and the target send the echo reply, but that response never makes it back to the source system. So it looks like the filtering is only inbound.
My RG hasn't rebooted in 18 days and my tunnel didn't die until 1 AM CST, so I'm guessing the filtering is taking place upstream from the RG within AT&T's network.
I'm losing patience with these guys.
02-16-2014 8:33 AM
Confirmed in Columbus, Ohio. My Tunnel died this weekend as well.
I found that nothing had been rebooted in over 2 weeks, was troubleshooting my end of the connection and found that it's definitely upstream.
This is awful, as I've been through this before with my original 2Wire modem, and then after fighting up the chain got them to replace it with the Motorola, which has been great until now.
It may be time to ditch AT&T since they can't seem to let their customers use their connections for what they want/need. Very sad.
02-16-2014 1:20 PM
Yeah, I only had the NV589 for 18 days before they broke it again.
I just got off the phone a few minutes ago signing up with the other provider in my area. Promotional pricing is $30/mo for 50 mbit, native IPv6, installed this coming Saturday.
02-18-2014 4:07 AM
So I've been on the phone with AT&T which normally doesn't help at all, but in this case I've found a supervisor that is actually trying to be very helpful. Shes calling me every few days while she attempts to figure out who in network operations to contact to make a query about this. I'd like to help her as much as we can, since she's really going out of her way here. FWIWwe checked and confirmed that its not a firmware update to the RG (like the hvg3800-B was), as the NVG589 hasn't had its firmware updated, so that suggests a definate network operations change. In an amusing twist I received a call from tech360.att.com a few months ago indicating malicious traffic from my site that suggested a compromise (which there was), but during the search for the cause they were more than happy in their response to provide me with network traces of teh offending traffic, which suggests they have the capability to do said traces. I'm wondering if contacting them might lead to some traction here. I'll let you know how that goes, and if you are so inclined, pelase do the same
02-18-2014 6:58 AM
02-18-2014 7:16 AM
That doesn't seem to be accurate. Tthe NVG hasn't had a firmware update in months (according to your support organization), and all of our private tunnels were working up until around feb 12th. Given that, and the fact that AT&T's ipv6 is working to my rg, which uses Rapid Deployment (on the same protocol identifier), it would seem that something internal to AT&T's network is dropping any protocol 41 traffic destined to anything other than your own RD gateway, whcih has broken us all. Can you please investigate why this was done, and indicate if/when you plan to correct this oversight?
02-18-2014 8:00 AM
my thoughts wrote:
IPv6 protocol 41 is currently disabled on all RGs, there is another post above (Somejoe7777) about trying to use a different protocol
There's no setting I can see that does that on the NVG589, no firmware update was pushed out on the night our tunnels stopped working (the uptime on my gateway was 60+ days), and the gateway is very obviously doing something over protocol 41 over the WAN right now (Rapid Deployment, as neilhorman mentioned) as evidenced by this being in its NAT sessions table:
IP Family: ipv4
Protocol Number: 41
Source Address: 192.168.254.254
Destination Address: 126.96.36.199
When I attempt to bring up my own 6in4 tunnel to Hurricane Electric, I see it appear in the gateway's NAT sessions table as well:
IP Family: ipv4
Protocol Number: 41
Source Address: MY.PUBLIC.IPv4.ADDRESS
Destination Address: TUNNEL.ENDPOINT.IPV4.ADDRESS
...but traffic over that tunnel does not traverse the WAN and back. The HE network engineer and I sniffed traffic on both sides to confirm.
All evidence suggests that protocol 41 is being filtered further inside AT&T's network.
The HE network engineer also offered this:
"There was a long running thread on NANOG about AT&T's 6in4 tunnel breakage, as well as on the U-verse support forums. It may be that with their 6rd deployment moving along, they're either intentionally or erroneously also grabbing other 6in4 traffic. The NANOG thread does have someone who did manage to eventually get through to [an AT&T network engineer], but it took them a while of escalations and mentions of FCC complaints to get there."
02-18-2014 8:23 AM
tdd37, thanks for the info, for the sake of posterity, can you post links to the NANOG and AT&T forum threads here, so others can find them and read up on the subject? Thanks!
02-18-2014 8:39 AM
The U-verse forum thread to which they are referring is probably this one:
02-18-2014 9:40 AM
02-18-2014 9:46 AM
Welcome to the internet boards! Check out our troubleshooting articles below and don’t forget to search the forums - your question may have been answered already!
Service acting up? Click here to troubleshoot now!
Do you have questions about Internet and Email Security? Have them answered on Wednesday, April 26th at our Knowledge Sharing Session: Hack Attack!