Does turning off »Use Graphics Processor« in the Performance Preferences (Photoshop > Preferences > Performance > Graphic Processor Settings) and restarting Photoshop have any bearing on the issue?
It does to seem likely but one can try …
Naturally the issue should not happen on its own but could the dialog have been repositioned to be mostly offscreen?
Is a second monitor involved?
What exactly does that do, turning off "use graphics processor" in the performance preferences? The info panel mentions certain features and interface enhancements, but doesn't mention what those are.
Yes, there is a second monitor involved, but Photoshop (and all my apps) are always on the main monitor (the other one is almost always off). I also use macOS's Spaces feature, which occasionally has caused problems in the past (though not with Photoshop that I can recall). That said, I never move that pop-up panel's location at all, and I'm a careful computer user.
The reason I didn't mention those things above in my post is that they simply seem like red herrings. I'll be using Photoshop just as normal, double-clicking on a layer style to edit it in the pop-up window that has always appeared in the past, and it will work (after the last preference reset) just as it always has, up until, at some random point within the same session (I could just have double-clicked less than a minute ago on the layer style and edited it as usual), so at this point, suddenly clicking again makes Photoshop act as though a pop-up control panel window is open (so you can't use other tools, and I have to hit 'Esc' or 'Return' to exit out)... but there's just no pop-up window there anymore, and won't be until the next preference reset at this point. Isn't that strange? I've used Photoshop for over 20 years, and I don't remember when Layer Styles were introduced, but I've never seen behavior like this in Photoshop before!
When the dialog-window is open but invisible does turning on the second monitor make any difference?
What exactly does that do, turning off "use graphics processor" in the performance preferences?
It turns off Photoshop’s GPU-usage – which means certain features like the Rotate View Tool or certain Filter acceleration will become invalid.
You are quite right! It took a few days for the bug to appear again since my last Preference reset, but it just appeared last night.
When I looked, I found the layer styles dialog window hiding on my second monitor, quite unexpectedly. As I mentioned, this is within my usual workflow, so it might have appeared in the appropriate spot seconds before it then leaps out of view onto the second monitor. How strange! This has been an issue only since the latest Photoshop update (19.1.1) I believe (I say I believe because there are times I use Photoshop less in a month, but it is a very recent bug, because I'm a heavy Layer Styles user).
Thanks so much for pointing out what I really should have checked!