AT&T fiber residential gateway limitations

I am just sharing this here because I have learned hard lessons that might be useful to others. This topic is relevant to those who

1. Have AT&T fiber and must use their residential gateway (RGW). I am talking about the Arris, Pace, etc. sitting in your closet connecting to either the ONT device (where your optical line connects) or directly to the fiberoptic cable itself in some models. 

2. Want to use their own firewalls / routers / mesh stuff behind the RGW

3. Do more with their internet connection than the typical Netflix-watching user (eg. peer-to-peer, crypto mining / staking, etc.)

Thousands of AT&T users have asked this question: How do I bypass AT&T RGW and use own my gear for all networking instead?  Why?  Because AT&T gear have limited networking capabilities AND they come with certain (intentional) restrictions in their firmware. AT&T, of course, makes money on equipment leases, and being the telco that they are by heart, they want to regulate customers' use of their network. Whatever the reason, it is what it is. 

Many people who are way smarter than me found ways to bypass the RGW. These methods involve Linux/Unix/NetGraph modifications made to various above-consumer-grade network devices such as Ubiquiti, PFSense, OPNSense, etc. I do not have the skills to make such modifications even though I use OPNSense in my own house and I am sure I am not the only noob.

The next best thing is to use what AT&T offers out of the box: A feature they call "IP Passthrough", "DMZ+" or whatever else. What does it do?

1. It passes the WAN public IP address of the RGW to your own router/firewall

2. It forwards all inbound traffic to your own firewall so you can manage it there 

How is that different than a true bridge mode where the front-line device does not interfere at all other than securing the initial connection to the line?

1. RGW still reviews your incoming packets (ingress both from LAN and WAN) and applies certain rules. This is true even if you completely disable their firewall or packet filters; it is hard-coded in their firmware. For instance, they will not permit repetitive packets from the same IP; I see hundreds of blocked packets with the notation "FLOOD limit 25pps burst 50" and "Invalid IP Packet" in my AT&T logs. Well, sometimes you legitimately need repeated packets in peer-to-peer networks so your RGW kills it. 

2. Even though it forwards all inbound WAN traffic to your device, it still maintains a NAT table (i.e. a table that tracks where each packet is coming from and going to with port and protocol details).  So, let it, why is this a problem? It is not if all you do is to watch TV. But if you are doing any peer-to-peer work where you need to accept connections from thousands of external IPs, this NAT table becomes a problem because it only allows a maximum number of 8192 entries in it and it starts behaving erratically as you get closer 6,000+ NAT records. You lose connections.      

Point being, RGW looks and filters no matter what you do.

I think I figured out a way to keep my NAT table under control. It may or may not work for everyone but I wanted to share. 

RGW's NAT table fills up because when you allow inbound WAN packets through, your own downstream firewall evaluates them to either allow or drop, which is standard firewall functionality. When it drops the packet, it does so silently. So, the upstream RGW has  already opened a connection in its NAT table and is waiting for the connection to either expire or get closed. When your firewall drops it, RGW doesn't know about that so it waits until the connection expires. Well, if you get 1,000 inbound connections per minute, it will fill up and die before they expire.  To solve this, I set up my downstream OPNSense WAN firewall rules (OPNSense WAN is connected to RGW's LAN) so that instead of using "Drop", I used "Reject". The difference is that Drop is silent while Reject sends back a rejection message. When the RGW sees the Reject, it closes the connection in the NAT table.       

My problems are by no means resolved after repetitive calls to technical support. I am going to move to AT&T Business with the hopes that the equipment that comes with that service is not as controlling. This is what I am told by AT&T Support anyway and will see how it goes. In the interim, hope this helps others.