Actually I cannot offer specific advice, but I would try the usual trouble shooting routines for both programs (trashing prefs by keeping command-alt-shift while starting Photoshop after making sure all customized presets like Actions, Patterns, Brushes etc. have been saved and making a note of the Preferences you’ve changed, 3rd party plug-ins deactivation) and the OS (repairing permissions, purging PRAM, running cron-scripts, cleaning caches, font validation, etc.).
Unfortunately none of that fixed the issue.
I got this problem when I installed Adob master collection Over CS 5.
I had children book story storis which created in Adobe illustrator CS 5 and I created art boards in Adobe photoshop cs 5.
After installting CS 6 i tried to edit the smart objects and it was not opeinging the smart object file in Illustrator.
I did some tests.
1) Right click on smart object layer and click export contents. Save on desktopp or anywhere you want. (I saved on desktop)
2) I had this file on my desktop (Vector Smart Object10.ai) but it didn't show any icon, which program should open this type of file.
3) Double click on the file and it pop up a extension window 'Click on change and select adobe illustrator' and press OK.
4) Go back to your photoshop file and right click on smart object layer and click dit content. now this time it opens up assets in illustrator.