Up until I installed Adobe Acrobat 8 Professional onto my Windows 8 32bit, I had no problems viewing previews of pdf in windows explorer, just as you can view images and word docs etc.
After installing it, I am met with an error message about there being an error with the PDF Preview Handler. I have not been able to find a solution to this error and it seems to have been caused exclusively by installing Adobe Acrobat 8
Much of this is an issue with the Microsoft Operating Systems changing how preview images within files are handled.
Have you changed the Windows Explorer to show the "Preview Pane"? Be careful because when this option is used and you open the file from Windows Explorer the file is locked and you cannot save the file back over itself.