Is the MicroCell Service Down?

I do indeed have a static IP address, so that's good. I've sent an email detailing the ports and another settings required for the box to function. Hopefully they haven't changed anything, like policies regarding VOIP devices. If necessary, I'll move the box back to after the router, set up a reserved address and port forwarding for 4500 and 500. But, if they've done something on their end of the wire, that's bad, and they've locked down the modem so I can't tweak anything there. I await their response in the morning.

Locking down the modem could be an issue if it is defaulted to perform NAT duties. It if does, and you go back to the nominal setup (modem -> router -> MicroCell) make sure the the router is not performing NAT as well becuase you'll end up with a double NAT and the MicroCell won't work.

Yep... When they switched out the old modem (which was simply a bridge and required my router to have a fixed IP address on the WAN) for the new modem (which includes DHCP service) I found myself in a double-NAT situation. Everything still worked, and I had also reserved an IP address for the MicroCell, but I eventually fixed it so that all addresses came from the modem.

I don't have an issue with placing the MicroCell between the modem and the router. We have a fairly low-bandwidth connection and, with the MicroCell connected after the router, it would go offline if  any heavy uploading was taking place. I'm guessing that having it ahead of the router allows it some amount of flow control.

I haven't heard back from the ISP yet, but they won't even be in the office for another few minutes...

OK! Here's the response I received from my ISP:

We've had a bunch of customers that suddenly had their AT&T MicroCell cut off late last week. These have been across multiple modem brands, and we haven't made any changes in our network that I'm aware of. We also note that customers with similar (identical?) devices from other cell carriers are NOT reporting a problem.

AT&T tech support has been quick to blame us for isolated MicroCell problems in the past, even when it's obvious from the lights flashing on the outside of the device that it's a GPS sync issue. We have also tried to work with their tech support on behalf of our customers and have been run around or simply hung up on.

We would be more than happy to have one here for testing, but unfortunately AT&T will only allow the specific devices to work in specific locations, which is corroborated w/ the on-board GPS. I had one customer who said he thought he could get AT&T to agree to relocate his MicroCell in our office. I don't know how helpful it might be but I'm willing to try if he can get back to us.

As of yesterday I had one savvy customer that had already done all of the port forwarding things in their standard trouble shooting guide as you mentioned, and still it didn't work, so he was getting the problem escalated with AT&T tech support. Supposedly the errors an AT&T tech was seeing from his end had something to do with power. ??? I also got wind that AT&T might be doing tower work trying to get 4G working in our area... which is a good thing, but I'm highly suspicious that they might have messed up a bunch of their devices in the process (MicroCell devices supposedly have to have at least a little tiny cell phone tower signal to work).

At this point I'm waiting to see what my one customer finds out via his escalated tech support discussion. Last night the final word was that they'd asked him to restart it or they were doing another location process that was going to take an hour or two. Hopefully I'll hear more later today from him.

If I can come up with any other angles from which I can investigate this I'll certainly be on it as soon as possible. In the meantime it appears we're stuck in the unfortunate and frustrating position of waiting for AT&T to troubleshoot the issue with our customers. This is currently my highest priority open support investigation, and you are now on my list of people to contact as soon as I hear of any progress, but if you discover anything before then PLEASE let me know.

You are also ALWAYS welcome to contact me with requests for updates. I know many people's cell reception is next to nothing in their home without this device working, and most of us count on our cell phones these days more than ever.

I have an ADSL connection that's supposed to be up to 768kb, but rarely exceeds 500kb, and the QoS is poor, typically around 30%. However, the MicroCell has worked very well for over a year now, as long as there's not a lot of upstream traffic. When I monitor the data flow during a call, the MicroCell uses very little bandwidth. It's fine for one phone, but I doubt that more than one call could be placed at a time.

The GPS signal is good and we seem to be in the database, so I don't see that as a problem. I have not replaced the power supply. My ISP tells me that they've detected a sudden, significant drop in data volume going to 12.230.0.0/16 from all across the state, which may be another indicator of a firmware update gone wrong.

I just received the following information:

There seems to be traffic [from your modem] going to AT&T, but no response to the UDP port 4500 traffic. No surprise, but it looks like the VPN is failing for some reason; like the MicroCell is happy with the VPN but the AT&T side isn't. Either that or the VPN is not properly set up. The MicroCell is sending traffic to 4500 but getting nothing in response.

This seems consistent with the flashing 3G light and no connection being established.

We have 2 Microcells in our office building side by side. One is flashing 3G lights and one is working fine. This also began about 1 week ago, all attempts to power cycle have not worked either. We sit below a tower, but are told that our building doesn't get service....in fact the whole block doesn't get service. Very frustrating for the employees and our customers who can't reach us.

AT&T - please fix this issue.

---

@CharlieSchorner wrote:

I have an ADSL connection that's supposed to be up to 768kb, but rarely exceeds 500kb, and the QoS is poor, typically around 30%. However, the MicroCell has worked very well for over a year now, as long as there's not a lot of upstream traffic. When I monitor the data flow during a call, the MicroCell uses very little bandwidth. It's fine for one phone, but I doubt that more than one call could be placed at a time.

 

The GPS signal is good and we seem to be in the database, so I don't see that as a problem. I have not replaced the power supply. My ISP tells me that they've detected a sudden, significant drop in data volume going to 12.230.0.0/16 from all across the state, which may be another indicator of a firmware update gone wrong.

---

You need to have at least a  1.5 MB upload speed (sustained) and my guess is that with the upgrade by AT&T for 4G you just don't have enough upload to handle it.

Given the bandwidth measurements I've made during phone conversations I'd be amazed that the upstream requirement could be that heavy for voice data. I know that the 3G requirements were for four simultaneous calls. Also, the voice requirements shouldn't change because the data technology has changed. As I only have a 3G capable phone I wouldn't be utilizing the new technology anyway, and that's for in-use bandwidth. I'm not experiencing dropped calls or poor voice quality. The box won't even connect in the first place. And, if all that weren't enough, all of the other users across Washington state, even in places where they have solid 5MB service, have also mysteriously lost their connections at the same time, without warning. Are you telling me that you've pushed out an update that kills all of the boxes with no warning to make a 3G radio somehow magically 4G capable? That seems highly unlikely, but, if that's the case, the timing couldn't be better as our contracts have expired, our phones are over two years old and other carriers actually have functional cell towers that can be used at our home. I realize that you're exploring all possible angles, but your argument becomes less convincing as more people chime in with this problem. I understand that several other people who have been affected by this outage have managed to get the problem escalated to a level above mere speculation and some actual investigative work is underway. Although I appreciate your willingness to respond and inquire, I think that continuing to insist that there are problems with my network, equipment, setup and connection, and not make the vaguest attempt to inquire about possible problems on you end of the wire is no longer helpful. We've exhausted the standard troubleshooting script and have not arrived at a solution. I hope to hear that some actual investigative work is underway soon.