3 Replies Latest reply on Feb 20, 2011 9:32 AM by mlabriola

    Test is failing when run on batch. but the same test is getting passed when it been ran alone?

    veeru_vicky

      Hi,

       

      Problem 1:

       

      I have a problem when running Flex unit in batch. I mean if am running a bunch of test cases few test cases were failing in variably.

      The stack trace shows "Timeout occurs before the expected event" but the same test is getting passed when i run it alone...

       

      Am not sure why this behaviour is happening??

       

      If i increase the timeout of the failed test doesnt show me any change in the results...keeps on failin?

       

      Please help me out to resolve the issue.

       

      Problem 2:

      the generated FlexUnitApplication is throwing some compilation issues every time when I delete some test case. My assumption while running the test runner every time the application file gets genrated by itself.

      Some times it does not reflect the changes...say if i delete one test inside a test case, the results keep on coming when i run it...

       

       

      Please provide me some solution for the same...

        • 1. Re: Test is failing when run on batch. but the same test is getting passed when it been ran alone?
          mlabriola Level 4

          regarding the first issue, you will need to post some code for help. most likely you have an asynchronous event  which is causing a failure. when run alone, the test is marked complete before the problem emmerges. when run as a group, there is more time. i cant know for sure though unless you can give us something to work with.

           

          second issue. unfortunately not much i can do to help here. the problem yyou are describing is one with flash builder. flexunit is the framework that runs the tests, but adobe has written the user interface and associated files that make it work in flash builder. we do not have access to that code.

           

          i would encourage you to file a flash builder bug at bugs.adobe.com

           

          mike

          • 2. Re: Test is failing when run on batch. but the same test is getting passed when it been ran alone?
            veeru_vicky Level 1

            Thanks for reply Mike.

             

            Actually we are doing unit testing for a Pure MVC related application.

            Our expectation is to test the whole flow say for an example.

             

            If a mediator is being registered it will do some actions while registering and after that it will return a result for assertion.

             

            So what we typically do is we will initialize our application in setup()  [Before] and add it to the UIImpresonator and in our [test] method we will do the method call for the use case.

             

            The reason why we are initializing the app is because the test will grab some data which will be loaded as part of the application.

             

            So i will do a proceedOnAsyncEvent and wait for the creation complete of the application in my set up.

             

             

            In my Test i will be posting a server call and wait for the response by using sequencewaiter by adding a step to sequencerunner. Few cases were not special cases like this but still they are failing when run on batch.

             

            Unfortunately i am not supposed to post the code in forums.

            I know its tough for u guys to provide me solution without code but still lemme know if there is any kind of issues already exists.

             

            Bcoz if a test unit runs and gets passed individually then it has to get thru when run on batch. Thanks a lot for ur replies.

             

             

            Thanks,

            Veeru

            • 3. Re: Test is failing when run on batch. but the same test is getting passed when it been ran alone?
              mlabriola Level 4

              I am not sure how much I am going to be able to help. There are a few premises here that are not 100% correct.

               

              First, you say "Bcoz if a test unit runs and gets passed individually then it has to get thru when run on batch."

               

              Couple of problems. A properly written unit test is one that runs in isolation. If a test running in isolation works perfectly and doesn't have any unaccounted for side effects then I agree with your statement. The problem is that the testing you are trying to do here doesn't seem to meet any of those conditions.

               

              Unit tests are about isolation of a unit of code for testing. You are instantiating an entire PureMVC app here complete with service calls. That is so far from isolation the statement above can no longer be applied. These are not unit tests. They are somewhere between an integration test and a functional test.

               

              In flash player, asynchronous events are broadcast from the top of the call stack. Meaning that when an async event is dispatched, if you look at the call stack you will basically see the player code, maybe a actionScript method or two and then wherever the error was caught. When an error occurs, it goes up the call stack until someone handles it. In any async event, this means that neither FlexUnit nor much of your code is in the callstack for that error. This is why we have to use APIs like the Sequences or Async methods to put a shim between the player and the code that could thrown an error.

               

              When you write a unit test for a single method with all external dependencies removed, you can be assured that the potential sources of an error are relatively confined and that you can therefore account for and check them. There are over 100,000 events thrown in the startup of a Flex app each with their own timing, etc.

               

              So, when you run one of these tests alone, FlexUnit builds your app, runs  test and the moment that test is considered complete, it is marked as such. If an error comes in after the fact, FlexUnit is no longer running and will have no idea. The only way we can tell FlexUnit to wait longer is through the various Async apis.

               

              Now, run this same test with 50 after it. If an error occurs at somepoint later, perhaps FlexUnit will still be running. In fact, without knowing if the code in question properly garbage collects, etc. there could be dozens of instances in memory. So, instead of being in a state of isolation where you can now identify the cause of the failure, failures can occure due to timing, memory allocation, unexpected event timing, interaction of various code units that may or may not be intended to work at the same time, preserved state of the application that is unexpected, etc.

               

              So, no. Without isolated tests, we can't say that if it runs once it will run just fine with 50 tests. There is no way to make that assertion. I would strongly encourage you to try to break this down and test the code units individually. If you can't do that, then you are going to need to do some debugging to see where the problem is occuring. If you can post code, I can try to help you, but right now, this is more or less like saying, 'What is wrong with this application; when we can't see the code or even know what the application does.

               

              If you want some advice on breaking this problem down or trying to make more unit style tests out of it, let me know. I am happy to help, I just don't see a way to do so right now.

               

              Mike