If your SIM card has the 3G logo then yes you would need a new SIM card.
When you buy a device in an AT&T corporate, they do provide a new sim for your device if its needed.
If you buy an unlocked device, depending on the device and the SIM card it needs, you may or may not need a replacement sim. Lets say you have HTC One X which uses a microSIM and wanted to use an iPhone 5. You need a new SIM because the iPhone 5 uses a nanoSIM which is much smaller. Depends on the device to determine the need simply!