On PC chances are you are not parenting your dialog correctly - it must be parented off the main Photoshop application windiow.
Sorry, this is on Mac.
My UI does not have a parent window because it is a separate application. One issue I see is I'm pretty much hanging PS while my application is running. Is there a certain way I can spin PS's event loop while it's running? Maybe PS needs a trip through its event loop to adjust the window priorities?
(though I'd expect a completely separate application would not end up BETWEEN the PS document windows and the background "application frame")
I'm not too clear as to what exactly you are trying to achieve. If you run, say, an automation plugin's GUI, it should be modal, in which case there is no advantage to have a separate process. If you are trying to create something semi-modal where the user can interact with Photoshop AND with your GUI simultaneously, the only official supported way to do it is to use CSXS and have your GUI in an extension panel. Everything else (i.e. some sort of fake semi-modal dialogs ran by a separate thread or another process) is a hack, they are explicitly unsupported by Adobe and will require tinkering with Photoshop's internal event look, explicit maintenance of Z-order, substituion of windows callbacks etc. While its doable, maintaining this sort of semi-modality is not a trivial task, it will likely to break from one release to another and you will be more or less on your own with it, since there is no simple solution for it.
Just something to consider.
Yep, well considered and unfortunately there is no way around it. Our UI toolkit collides with PS when we link directly with it, so I have to shell out to a separate process to do the work. It's been working well for years like this - it's just an annoyance that it doesn't work well in this one particular instance (tabbed windows turned off, and application frame turned on).
There are other third-party plugins out there that do exactly what we do - I was just curious if there was a known workaround that's public knowledge.
Look at the SetWindowGroup function with the second parameter set to GetWindowGroupOfClass(kMovableModalWindowClass).
I was able to work around this by periodically polling the NSApplication event loop with a type of PopupModal.