On Windows, for a small expenditure, yes - in Explorer and File - Open and Save dialogs (which are really hosted by Explorer).
And you can get previews as well as thumbnails.
I use and love the FastPictureViewer Codec Pack: http://www.fastpictureviewer.com/codecs/
It also gives you access to raw file thumbnails.
I've run this software for years, and it's perfectly stable, and gets updates (e.g., to handle new raw formats) fairly often. Only thing I haven't been able to talk them into doing is to create thumbnails for .PSB files (yet).