Yeah, probably you need to hack the Classes Root HKEY. Looks seriously messed up.... I'm sure there is some smart registry tool as well. Still, I'd look if not anotehr program uses the AEP file extension. such conflicts usualyl come down to that...
You were on the right path Mylenium, I managed to fix the problem.
So here's some tips for anyone who upgrades to CS5.5 from a previous version like I did, in this case CS3.
When I uninstalled CS3 I didn't quite do a full uninstall of all programs as there were some things I wanted to keep, like soundbooth etc. Now, I was pretty sure I completely uninstalled AE CS3 but it seems that some of the folders and files lingered behind during the uninstall process.
When I uninstalled CS5.5, it did warn me that there was a minor issue with the installation that a reboot would fix. I did this and everything was working perfectly, until I of course tried to open an After Effects project by clicking on the folder, then the project.
As you saw, the program wasn't allowed to be selected as the default and for some reason it wouldn't stick when using browse. SO! I rebooted my computer into Safe Mode and went into the Adobe Folder that had the old After Effects CS3. I completely deleted that folder and all of its contents and ran CCcleaner next to go through the registry. It found various conflicts with After Effects including invalid keys, could not open with CS5.5 by default, etc. Not exactly in those words, but clear enough to state that there was a conflict with the default program launcher.
After fixing the issues with CCcleaner, I went back into normal Windows. Right clicked on the icon properties, selected browwse, support files, AfterFX, and the problem was solved!
Was very boneheaded of me not to do a complete wipe of the installation, but after Adobe had gotten rid of audition initially, I was afraid of losing programs that I had use for.
Anyway, if anyone else runs into this problem just make sure to completely delete the old installation and run a registry cleaner to fix the issues.