Is the audio recorded as multi-channel? Is it PCM or Dolby? Try importing clips from the camera that have stereo audio. If it turns out that multi-channel audio is the problem, load the clip into Audition (if you have it) and process it there. If you don't have Audition you could try the open-source program Audacity, but I don't know how well or even if it supports multi-channel audio.
I did more research and found out that it is a known problem using premiere 6. The easiest work around is to extract the audio using a third party program and import it seperately. Fortunately I only have 10 30min clip to go through so it only took 4-5 hours to get all the audios out.
CS6 works just fine with Panasonic media. You should not have to do anything but import it.