The help in Fb4 is now delivered via Adobe Community Help Client (chc) rather than the eclipse system, but it should be taking you to the correct page within the documentation. It does for me.
Couple of things worth checking:
1. chc version : The version that shipped with Fb4 is 184.108.40.2060. If you had a previous version of chc installed, there's a chance that the Fb installer didn't update it. So you might consider to manually uninstall then reinstall the chc app from the labs site which has a newer version that than shipped with Fb4. See Mark Nicholson's response here : http://forums.adobe.com/message/2283431#2283431
2. From experience, the help files need to have downloaded. Even if you set chc to display in a browser rather than the chc app itself, it will only actually do this once the help files have been downloaded to your system. This should occur the first time chc is run. You can check their status via preferences > Local content
Paul's second point seems to be true - "open in browser" doesn't work if the CHC haven't downloaded the help files previously. So Shift+F2 doesnt' work for me at all.