I don't think you should care. The option you do care about is this one:
That is part of PIActionParameters *actionParameters
That is part of your PSActionsPlugInMessage that comes into your automation plugin entrypoint.
This parameter has three options:
a) always display the dialog,
b) only display the dialog if you don't like the parameters you are given,
c) never display a dialog (either return an error or proceed with best options but don't stop with a dialog)
From the file menu, user selected, you should get option a (always). From the actions panel you should get option a or b. From scripting you can get all three options.
There is also a property in PIProperties.h called propPlayInProgress but the above information should be what you need to do the correct thing with dialogs.