@trevorbutler - There is a known bug, being worked on in 4.2, that will cause the Ant task to hang if a test runner is unresponsive. We're implementing a heartbeat process to prevent this from occurring, but in 4.1 the timeout only applies to how long to wait for a connection from the test runner SWF (whether executed from FP or ADL). I noticed the documentation is a bit loose on this so I will update this to reflect how 4.1 behaves. Ideally, the timeout should apply to the timeout until a connection is recieved from the player and the length of time to wait for each test until failing the test run. HTH.
Ah... OK, I was thinking the timeout was more like the JUnit ant task timeout when running in forked mode, i.e.the entire test run has that amount of time to complete (see http://ant.apache.org/manual/Tasks/junit.html). I know I was making an assumption there, but it seemed logical to me since the Flexunit test runner is essentially a forked process...
So, you are saying that the timeout in the FU 4.1 ant task specifies the time to wait for a connection from the SWF running in the player? I see this stated in the Ant task doc on the wiki now - Doh! You also say that the timeout also applies to each individual test? So, it sounds like the behavior is similar to the JUnit ant task running in "perTest" forkmode.
Out of curiosity, is there any way to specify an overall timeout for the entire test suite run, by chance?
Anyway, I figured out why adl.exe was hanging. Evidently an error occurred in the tests which caused a pop-up window to be spawned, and this halted test execution. When I clicked "Dismiss All", the execution continued. Of course, in a CI environment, there is no one there to click the button to dismiss the pop-up! So this is kind of a show-stopper. Is there any way to configure the player (Flash or AIR) to suppress pop-ups, or at least not to stop execution when they occur?
1 person found this helpful
The heartbeat will allow us to catch that sort of thing and stop it. If you are testing with Flash Player 10.1 and later, the release version of 4.1 will also prevent this, but only for those player versions.
In the short term though, if you are seeing a popup making it all the way to the console, that means that there is an improperly coded test somewhere. Due to the way the Flash Player call stack works, if there is an error in an asynchronous process, and the developer of that test did not use the FlexUnit async APIs but rather decided to go out on their own, this can happen.
If all of the FlexUnit Async APIs are in use, we can catch that before it ever makes it the console, so, all of that sums up to:
1) Likely a problem in how that test was written, which should be addressed regardless
2) If using FP10.1, upcoming 4.1 release will still catch the improperly coded test using a new feature only available in the latest flash player
3) heartbeat feature will allow us to catch and at least fail these tests in any version, but that is a 4.2 target
1 person found this helpful
@Trevor - I probably should updated 4.2 to have a timeout for the entire test run as the JUnit Ant task does, good catch; I'll add this to my todo list. As far as the wiki, I updated it after posting on the forum, so that was just me getting stuff done.
As far as the debug player popping open its error window, I'm not sure if there is a setting in the mm.cfg that will restrict it, but typically if an error can be thrown I try to wrap a try/catch around the tests with an associated failure assertion. If you're using 10.1, you could also try using the global error handler in your tests runner to stop the test run. I'd think the framework could help to manage this, so this may be a better question for Mike.
Thanks for your responses.
I agree that the tests are at fault here. Each pop-up window contained a traceback indicating the Class where the issue was occurring. I also agree that fixing the tests is the right way to proceed, and I have notified the developer of this project about the failures.
My question about suppressing the pop-up windows was really a desire to allow our CI system to gracefully proceed and run the remaining tests, thereby reporting all possible results for the run, as much as possible.
So, I'll be eagerly looking forward to 4.2 it looks like. Though hopefully this particular project will run OK once the errors causing the pop-ups are resolved.
Also do check to see if running on Flash Player 10.1 is possible. That would mean you have a good solution next week.
Since this test uses some of the AIR apis to read/write benchmarks to the disk, I do not believe we can use the Flash player to run them. Note my first post in this thread mentioning that I am using adl.exe, i.e. player=air.
Please correct me if I am wrong...
Got it. Then the thing to find out is if you can run on AIR 2.0, where the same behavior is available in our next bits.