Could it be my headset or the USB port?

Well by the sounds of things its you headset.if it was hardware problems in your computer the headset wouldn't be reading. A quick test just plug something into your USB port and if it works than it is defiantly your headset. Although... I game allot on my PC and I find I have to unplug my speakers to hear anything in my headset. But that is because I am using an audio card with all three plugged in.(Speaker,Mic,Headphone).Also another suggestion for you could be to adjust the levels on the device through speaker properties.