I think what's happening is that LR is storing a thumbnail in the exported .jpg, and that thumbnail doesn't contain a watermark. You can verify that by extracting the thumbnail:
exiftool -b -ThumbnailImage myfile.jpg > thumbnail.jpg
(using the free Exiftool). Windows 8.1 File Explorer displays the thumbnail stored in the .jpg if it has one, otherwise it computes a thumbnail from the whole image. You mentioned "brand new Windows 8.1 operating systems" -- did you upgrade from Windows 7? I don't have Windows 7 installed anymore, but I'm guessing that Windows 7 Explorer always computed the thumbnail to be displayed (slower) rather than using any thumbnail stored inside the .jpg (faster).
You could file a bug report in the official Adobe feedback forum: Recently active topics in Photoshop Family about Photoshop Lightroom. It's a bug that the thumbnail stored in the image doesn't contain a watermark.
But as a practical matter, I'm afraid there's "nothing you can do about it" except wait for Adobe to fix this (before the next ice age, one hopes) or use a workaround. You could use Bridge rather than Explorer (blech). You could use the Run Any Command plugin to define a post-process action for your exports that uses Exiftool to delete the thumbnail from the exported .jpgs.
I submitted a bug report to the official Adobe feedback forum: Lightroom: Watermark omitted from embedded thumbnails of exported JPEGs. Please add your vote and opinion to make it more likely Adobe will prioritize a fix.
The bug report also contains a simpler workaround using Limit File Size To.