You can do this with a little bit of script on your check box's Click event (in FormCalc in this example):
if ($ == 1) then
DateTimeField1.format.picture = "DD/MMM/YYYY"
DateTimeField1.format.picture = "MM/DD/YYYY"
The trick is to know that in XFA, what Designer refers to as the "Display" picture is the <format> node which contains a <picture> which, in turn, contains the "display" picture; or, the picture used to
format the field's value for display purposes.
I've attached a sample form which demonstrates this.
In the CheckBox's change event, add code that sets the field's format.picture.value:
if (this.rawValue == 1)
// Box is checked; use one format
DateTimeField1.format.picture.value = "YYYY-MM-DD";
// else use the other format
DateTimeField1.format.picture.value = "DD-MM-YYYY";