Try with installing browser fix pack on 5.4. What is the java script error you are getting?
EDIT: @Sham HC: Are you able to provide me a link to a particular package with browser fix pack? I've got a dozen of "hotfix" and "featurepack" packages for CQ 5.4 already installed, but I'm not sure if these packages included browser fixes. Thank you.
I managed to find and install the package (I think that's the right one) from:
...but it didn't help. The tree navigation in BrowseDialog still doesn't work in IE 10 and there are no JS errors in my console. Description of the package I installed says about some fixes for IE 9, but not for newer versions.