Well, you're not alone. If you're using a standard windows soundcard (essentially, ANY Soundblaster, SBLive or compatible, Crystal Audio, ESS, Soundmax, or even some other cards from Turtle Beach) the remedy is in changing the input/output settings in your Windows Mixer. Here's how...
In any version of Windows up to and including XP (*except Vista, apparently...) that you have:
- Double-click on the yellow (or grey) speaker icon located on the lower right hand corner of your screen.
- Select Options>Properties.
- In the "Adjust Volume for" area, select 'Recording'.
- This will show you all the devices you can record from, in the "Show the following volume controls" window.
- Make sure all the devices have a check mark in them.
- Click on OK.
- At the bottom of each recording device, there is a check box with "Select" to the right of it. Use this to select the recording device that you want - if you're recording from Line In, choose LINE. If you're recording from Mic In, choose MIC, etc.
- Make sure the recording Volume is at least half way up.
- Minimize the "Recording Control" window.
- You are ready to record.
Now, if you still have bleed, it's possible that you've either selected something like 'WAVE' for your input device, or perhaps 'Mixed Output' or 'What You Hear'. The latter settings essentially lump ALL audio fed through the soundcard into Adobe Audition. The 'Wave' setting will do exactly what is described above - cause the playback track to be recorded with the overdub track.
If you're using a more pro-level card, chances are it has it's own mixer panel (and thus, is not controlled by the Windows Mixer).
Some cards that have their own mixers: Midiman/M-Audio (Delta series, Audiophile, etc), Echo (Layla, Mona, Gina, etc), Aardvark, Lynx, Frontier Designs, MOTU (828, 2408, etc)...
Adapted from an original post by Audition M.D.
*Ask on the forum. This was changed after a user comment. Also if you have problems with Windows 7, ask a forum question too. When we get a definitive answer to the W7 issues, they'll be added here.

