Adding a SIP phone

To use SIP phones, you need a VOIP provider who will support the SIP phones of your choice.  Currently you apparently have AT&T Voice, which provides its own ATA adapter built into the Gateway (and in your case, apparently some sort of additional external adapter).  You may find that the AT&T Voice service will interfere with your other VOIP provider.

For the other office, we don't plan to keep the other provider (Comcast) but to completely replace it. But we'd like to continue using the SIP phones we already have.

For the office just upgraded, the question is how to add additional phones, to add SIP in addition to the existing POTS lines. I haven't found anything in the online menus that suggests a way to do this, like perhaps ordering another number or being able to answer the existing numbers using a SIP phone. 

I'd like to get some idea of what's involved before spending a long stretch on hold to talk to a salesperson. 

You keep using the word "POTS."  I don't think that word means what you think it means. ;-)

The preferred term for a phone with an RJ11/RJ14 cord that plugs into something else would be an "analog phone."  They can be plugged into the jack on the back of a Gateway, an VOIP ATA or the wall jack of an actually POTS-connected home.  In your case, you have AT&T Phone VOIP service which provides several lines accessible by analog phones.  POTS means the analog line comes into your home in analog form.  Now, having beat the terminology horse to death...

As far as I know, AT&T Phone only supports analog devices plugged into their provided/imbedded ATAs, not independent SIP phones.  I would talk to my friendly business telephony salesman at AT&T and see what he says, and ask for him to follow up anything he says with links to a website or something in writing describing it all.

ATA being analog telephone adapter.  

The gateway itself supports SIP, or at least doesn’t overtly block it.  I also know what a polycom phone is but am not aware of the prerequisites required to use it.  What you’re getting with Att Internet is an Ethernet connection fronted by an Arris router.  

If you’re running a business, why not deal with the part of Att that deals with this area instead of a basic consumer grade service?

FTR, Polycom is a manufacturer of phone equipment.  Polycom phones could be analog or digital.  A Polycom SIP phone would be a digital phone that directly supports the VOIP SIP protocol without needing an ATA.  It also cannot use an ATA because it won't have an analog port.

@JefferMC 

I've only ever seen them used in large corporate environments, so I don't know but imagine some additional infrastructure might be needed for them to function, a server perhaps which a gateway wouldn't likely have.

There are two Polycom SIP phones sitting in my church right now.  They barely get used because everyone prefers to just use their cell phone.  The SIP passes through the Internet modem/router to the phone just like it would to an ATA connected to an analog phone.  Our VOIP provider requires no on-site equipment.  We chose them because they were one of the ones that didn't want to put a separate box in the building.

So standard ethernet connection?

Yes.