3G proxy (wnsnet.attws.com) strips HTTP response headers

[jreichenberg]

When accessing one of our websites through an AT&T 3G connection, it goes through a proxy XXXX.wnsnet.attws.com (where "XXXX" varies). For some HTTP responses, response headers are stripped. In our case, the Content-Encoding:gzip header is stripped. Since it is a gzip compressed response but does not have the Content-Encoding header when it gets to the web browser, the web browser doesn't know what to do (asks if the user wants to download the file rather than rendering it as HTML).

This appears to have started in our case a week or two ago.

Since this happens intermittently, it is diffiicult to track down. We have identified one clue: in responses that have headers stripped, we are sending a Set-Cookie header that has an "=" sign in the value. I'm imagining this confuses the proxy (just a theory).

Until we can get some guidance from AT&T or others on this issue our users are in the dark, confused, and angry at us (and AT&T)

[Roadshark]

Thanks for your response.  Customer service wasn't too responsive to me, don't know the lingo but I know when something is broken.

[Roadshark]

I have had 4 seperate calls for a total of about 5 hours time and gone to ATT store to demonstrate the situation there to no avail.  The only response I can get is that it is a problem with the website I'm trying to access.  The fact that I had no problem in the past with these sites and that the problem arose with multiple unrelated sites at the same time does not impress them either.  I do know of one site that appears to have determined what the problem is and modified their website to work with ATT 3g/4g.  That site is Hotwire.com.  Their support people may be able to tell you what they changed in order to make their site compatible with ATT.  I give up.  As a user I don't have the time or desire to learn the technology to a level that I can further argue my point with ATT support personnel.

[Roadshark]

Now seeing same effect on certain CNET.com and Walmart.com sites.  According to ATT it's not their problem though.

[jreichenberg]

Yes, Hotwire.com simply changed the order of their HTTP response headers. In some cases, some headers are still being stripped off, but this now causes much more subtle (or no) side-effects for users than before (when the all-important Content-Encoding:gzip response header was last, and thus stripped when AT&Ts bug is triggered).

AT&T does not have a bug tracking system to log this serious bug, and a call to their "Data Support" team was not helpful.

[matchdotcom]

We are having this problem at Match.com as well with our mobile website.  Changing the order in the headers solved the issue.  We do have a ticket open with AT&T and we are waiting on a response.  We were able to isolate the problem to a specific set of IP's within the AT&T network (on 3G/4G).

[ewave]

Can you share a little more on how you ordered your HTTP response headers in order to get around AT&T issue? We're having the same issue logging into our site on AT&T 3G network. Thanks.

[livejournal]

We're also having problems with this happening over at LiveJournal.com, and the problem is isolated to people on AT&T's 3G/4G network. 

[livejournal]

More specifically, in case anyone else stumbles across this, we've verified that when a request gets sent through AT&T's network,  cookies are being re-written to strip all spaces out of them, making them invalid.

[bivald]

I think I am seeing a similar issue here, problem is - I'm in Europe and can't really debug using the AT&T 3G proxy. Any help would be truly appreciated.

The site is http://www.rodebjer.com/en/artiklar/new-arrivals/index.html

If you visit it on wifi, it looks good we a list of dresses. If you visit it on AT&T 3g you only see a large "coming soon" image and nothing more. Same if you disable javascript, i.e I don't think it's a js/local issue.

Any ideas?

[rama_pac]

Our site is also experiencing problems because of this. It looks like they change the http headers causing problems with the way we manage session IDs.

AT&T should really respond to this as a lot of sites are experiencing problems only on their network.

[spkane]

We run about 100 sites for many technical standards organizations, including USB, Wifi, SNIA, and many others.

We have run into a very similar problem, and our investigation has lead us to this conclusion:

AT&T has recently implemented a proxy or similar technology on their LTE (4G) mobile network that is stripping the whitespace from Cookie headers and as a result is breaking rfc6265 ( http://tools.ietf.org/html/rfc6265#section-4.2.1 ), which states that:

the user agent will send a Cookie header that conforms to the following grammar:
cookie-header = "Cookie:" OWS cookie-string OWS
cookie-string = cookie-pair *( ";" SP cookie-pair )

By stripping out the whitespace, the cookie is no longer valid, and tools that utilize these cookies like the Apache web server's mod_auth fail to properly read them, since the cookies are now mal-formed and do not folow the technical standard properly.

We have only seen this on devices conencted to AT&T LTE (4G) network, so it is possible that it is a bit different then the initial issue posted, but likely related.

[tman204]

Hello,

I have also been experiencing problems since AT&T has started deploying [new] proxies on their 4G LTE networks.  This behavior has recently started in the Minneapolis/St. Paul area as of this weekend (2/10/2013).  The problem is that AT&T's proxies are indeed stripping the white space in the http request to the origin server.  For example:

cookie1=foo; cookie2=bar; cookie3="abc"; cookie4="xyz"

becomes:

cookie1=foo;cookie2=bar;cookie3="abc";cookie4="xyz"

which is WRONG - see RFC6265 as already pointed out.

AT&T - it is one thing to capture and record this data, but if you do so, at least reassemble it properly before sending it to our webservers.  Many of us have enough to deal with on a daily basis, and, would prefer not to have to debug your network(s) as well.

Thank you,

[tman204]

For those of you using mod_auth_tkt (ver 2.1.0), here is a quick patch that allows us to work around AT&Ts broken proxies.  It inspects the first 8 bytes of the remote ip, and, if they match, ignores to lack of whitespace caused by AT&Ts proxies. 

*** WARNING ***

THIS PATCH COMES WITH ABOSOLUTLY NO WARRANTY, EXPRESSED OR IMPLIED.  I, NOR MY EMPLOYER, WILL BE HELD RESPONSIBLE FOR THE USE OR MISUSE OF THIS PATCH.  USE AT YOUR OWN RISK!

*** WARNING ***

Apply to mod_auth_tkt.c ver 2.1.0:

591c591
< cookie_match(void *result, const char *key, const char *cookie)
---
> cookie_match(void *result, const char *key, const char *cookie, request_rec *r)
610a611,623
>
>   const char *remote_ip = (cr->r->connection->remote_ip);
>   char matchIP[] = "198.228.228.111";
>   unsigned long ip;
>   struct in_addr ia;
>
>   /* Convert remote_ip to unsigned long */
>   if (inet_aton(remote_ip, &ia) == 0) {
>     return (NULL);
>   }
>   ip = ntohl(ia.s_addr);
>
>
614,615c627,632
<         value++;
<         continue;
---
>       if (strncmp(remote_ip,matchIP,7) == 0) {
>          ap_log_rerror(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS, cr->r,
>           "Invalid cookie_match from ATT, ip %s", remote_ip);
>       } else {
>           value++;
>           continue;
616a634
>      }

--HTH

[tman204]

Here is a more concise version that can fed into "patch."

*** WARNING ***

THIS PATCH COMES WITH ABOSOLUTLY NO WARRANTY, EXPRESSED OR IMPLIED.  I, NOR MY EMPLOYER, WILL BE HELD RESPONSIBLE FOR THE USE OR MISUSE OF THIS PATCH.  USE AT YOUR OWN RISK!

*** WARNING ***

*** ../../tmp/mod_auth_tkt-2.1.0/src/mod_auth_tkt.c     2009-07-10 02:46:51.000000000 -0500
--- mod_auth_tkt.c      2013-02-12 22:29:47.000000000 -0600
***************
*** 608,619 ****
      cookie_name[strlen(cr->cookie_name) + 1] = '\0';

      value = (char*) cookie;
      while ((value = strstr(value, cookie_name))) {
        /* cookie_name must be preceded by a space or be at the very beginning */
        if (value > cookie && *(value-1) != ' ') {
!         value++;
!         continue;
        }
        /* Cookie includes our cookie_name - copy (first) value into cookiebuf */
        value += strlen(cookie_name);
        cookiebuf = apr_pstrdup(cr->r->pool, value);
--- 608,627 ----
      cookie_name[strlen(cr->cookie_name) + 1] = '\0';

      value = (char*) cookie;
+
+     char matchIP[] = "198.228.228.111";
+
      while ((value = strstr(value, cookie_name))) {
        /* cookie_name must be preceded by a space or be at the very beginning */
        if (value > cookie && *(value-1) != ' ') {
!       if (strncmp((cr->r->connection->remote_ip),matchIP,7) == 0) {
!          ap_log_rerror(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS, cr->r,
!           "Invalid cookie_match from ATT, ip %s", (cr->r->connection->remote_ip));
!       } else {
!           value++;
!           continue;
        }
+      }
        /* Cookie includes our cookie_name - copy (first) value into cookiebuf */
        value += strlen(cookie_name);
        cookiebuf = apr_pstrdup(cr->r->pool, value);

[jamileh]

Sorry you're having trouble. I need a bit more information from you, if you don't mind. 

Will you please check your private messages (upper right, flashing blue envelope)

[mightybs]

I'm experiencing this same issue with my customer's websites.  I'm currently in the process of data mining to be able to provide more information.  We've unfortunately been experiencing this for quite a while as well.

So far I've identified these proxies:

HTTP_VIA: HTTP/1.1 akrmspsrvz9ts107.wnsnet.attws.com

HTTP_VIA: HTTP/1.1 akrmspsrvz9ts323.wnsnet.attws.com

Being used by these Client IP addresses:

166.137.100.16
166.137.100.37
166.137.100.42

166.137.100.44

166.137.100.49

Accessing my port 80 websites in my 12.106.187.0/25 subnet.

[mightybs]

I wanted to add that this is platform independent.  I've been able to test different Android and iPhone models using the AT&T wireless network with the same results.

[spkane]

I also have some verification from within AT&T that this is a known issue that they are actually working on. I was expecting a call from an AT&T engineer 2 days ago, which unfortunetly, didn't actually occur (and I hope is simply becuase they are busy fixing the problem), but we shall see.

[matchdotcom]

Good luck.  We went through 3 levels of support with multiple tickets and phone calls.  Unless you live in Minneapolis/St. Paul and you can give them an exact test case where they can reproduce the problem they will be unable to fix it.  We provided all of our logs, example cookies, and the exact IP addresses and subnets that the issue was happening on and still got nowhere.  They were diligent about  trying to reproduce it with us on several of our phones (we did a live test on Android and iPhone) but since we were unable to reproduce it at that exact moment on their network they closed the ticket.  We tried to explain that we can see it happening for only a specific set of customers in a specific region of the country but that didn't seem to help.  I beleive at the time they suspected a patch on one of their proxies but never got past that.

In the end we just put up a patch on our Mobile Web servers and our main web farm that fixed the issue on our side.  

[mightybs]

Can you elaborate on your fix so that others can benefit?  As much as I want AT&T to fix their issue, I'm also game for doing something on my end to fix it.  I can always disable if I think they ever have the problem fixed.

[tman204]

We first observed the problem in MD, followed by GA, then TX, and most recently MN (where I am).  When it finally arrived in my area, I was able to tcpdump on my webservers and capture the issue.  Our problem was our users were prompted over and over to login to our site.  This was because mod_auth_tkt (http://www.openfusion.com.au/labs/mod_auth_tkt/) correctly expects a " " [space] after the ";" delimiter used in the cookie string.  If this space is missing, the cookie is rendered invalid, the user is no longer authenticated, and, prompted to login again.  Wash, rinse, repeat.

RFC6265, specifies the following under requirements in section 5.4.4:

      2.  If there is an unprocessed cookie in the cookie-list, output
           the characters %x3B and %x20 ("; ").

The crux of the issue is the %x20 is stripped when it traverses *some* of ATT's proxies.

We also patched our webservers (with the patch posted above and with greatly expanded ATT IP space).  It is a complete hack, yes, but, I'd rather break the RFC for a interim fix instead of not at all.  The nice thing, is, when ATT fixes the issue, the patch above renders itself useless by nature of the nested "if" - it also does nothing to users outside of ATT IP space.  There is a more aggressive patch for mod_auth_tkt here https://groups.google.com/forum/?fromgroups=#!topic/linux.debian.bugs.dist/lKC22GDfAXI that doesn't require the implementer to know the src IP space.  I did not write it.

I have worked with ATT, provided packet captures, reproduced the issue with them, and even gave them an account on our site to allow them to test at will.  The ball is in their court; I expect now that there is visibility on this issue, it will be addressed, as it is affecting thousands of sites and hundreds of thousands of people.

[rama_pac]

Jamileh,

Could you please provide us an update on this? 

Thanks

Rama.

[mightybs]

My tests have so far confirmed that indeed the AT&T proxy is removing the space from the HTTP_COOKIE header as part of the separator as others have noted, so depending on what's in that header you may or may not experience the issue.  I'm looking into where we can insert some code to try and repair this on our end even though it is not our problem.  A fix I'll readily remove when it looks like AT&T has fixed this horrible bug.  I'll also post what I can about the fix on our end.  We don't use the mod_auth_tkt module, so I can't use that patch, but I'll use the idea behind it to insert it where I can in my system.

[spkane]

AT&T provided our customer a "won't fix" response on the issue of their proxies stripping white space from cookies. Hopefully, this will change, or they will provide a reasonable explination of why they are doing this, since they are literraly breaking the intenet for their customers.

[mightybs]

I've posted a comment about our fix, and the code written to work around this issue with a Ruby on Rails site, where a Rails fix of Ruby libraries followed the RFC strictly, and we've modified the parsing to work around this issue.  Feel free to pose any questions, but feel free to use the code:

http://snipplr.com/view/70030/rails-fix-for-cookie-munging-att-proxies/

[jamileh]

bivald ,

I received your PM and will have a manger contact you soon. 

Regards,

Ray on behald of Jam

Community Manager  

[rama_pac]

Jamileh,

Could you please give us an update on this?

Thanks

[harryspar]

No, that's how they work, whenever there's a real issue they go to private messages and then you never find out the end of the story. It's up to the OP to come back and update.