15 Replies Latest reply on Nov 11, 2010 12:53 PM by brian.thomas

    External Data for Parameterized Tests

    brian.thomas Adobe Employee

      I'm trying to take advantage of the awesome new JUnit-style external data feature of Parameterized tests. Unfortunately I'm running into some trouble: Parameterized.createTest() is getting called before my ExternalDependencyLoader class calls notifyResult() on my ExternalDependencyToken. My parameters have yet to be retrieved so my test is labeled as (Missing Params) and then complains about the constructor having an incorrect number of arguments.

       

      Am I stumbling on a bug here?

       

      Thanks!

      Brian

       

      The trimmed down source for my dependency loader, in case it helps:

       

      public class TestGraphicsLoader implements IExternalDependencyLoader
      {     
           private var _token:ExternalDependencyToken;
           private var _service:HTTPService;
           
           public function TestGraphicsLoader()
           {
                _token = new ExternalDependencyToken;
           }
           
           private function handleResult( evt:ResultEvent ):void
           {
                _service.removeEventListener(ResultEvent.RESULT, handleResult);
                var files:XML = new XML(evt.message.body);
                
                var dataSet:Array = [];
                for each ( var file:XML in files.children() )
                {
                     // Build my data set here ...
                }
                
                _token.notifyResult(dataSet);
           }
           
           public function retrieveDependency(testClass:Class):ExternalDependencyToken
           {
                _service = new HTTPService();
                _service.url = "http://myservice/service.php";
                _service.addEventListener(ResultEvent.RESULT, handleResult);
                _service.send();
      
                return _token;
           }
      }
      

       

      And related snippets from the test...

      [Parameters(loader="assetRetriever")]
      public static var urlSet:Array;
                
      public static var assetRetriever:TestGraphicsLoader = new TestGraphicsLoader();
      
      public function ImportGraphicsTest( param:Object )
      {
      

        • 1. Re: External Data for Parameterized Tests
          mlabriola Level 4

          Brian,

           

          Do you know what version of the flexunit lib you are linking against? I took your code pretty much exactly, but listened to the fault of the HTTPService instead as I obviously don't have that one to hit, faked some data and everything worked for me.Perhaps you are hitting a bug that was fixed in a later version but never reported.

           

          I am including the code here for you to look at, but is there a chance something else is muddying the waters, for instance the parsing logic in the result handler throwing an error or something similar? Here are my minimal changes to yours to make it run fine on my machine:

           

           

          [RunWith("org.flexunit.runners.Parameterized")]

          public class SimpleParamTest

          {

            [Parameters(loader="assetRetriever")]

            public static var urlSet:Array;

           

            public static var assetRetriever:TestGraphicsLoader = new TestGraphicsLoader();

           

           

            [Test]

            public function myTest():void {

           

            }

           

            public function SimpleParamTest( param:Object )

            {

            }

          }

           

           

           

           

          AND

           

           

          public class TestGraphicsLoader implements IExternalDependencyLoader

          {

            private var _token:ExternalDependencyToken;

            private var _service:HTTPService;

           

            public function TestGraphicsLoader()

            {

              _token = new ExternalDependencyToken;

            }

           

            private function handleResult( evt:FaultEvent ):void

            {

              _service.removeEventListener(FaultEvent.FAULT, handleResult);

              var files:XML = new XML(evt.message.body);

           

              var dataSet:Array = [[1],[3],[4]];

              for each ( var file:XML in files.children() )

              {

                // Build my data set here ...

              }

           

              _token.notifyResult(dataSet);

            }

           

            public function retrieveDependency(testClass:Class):ExternalDependencyToken

            {

            _service = new HTTPService();

            _service.url = "http://myservice/service.php";

            _service.addEventListener(FaultEvent.FAULT, handleResult);

            _service.send();

           

            return _token;

            }

          }

           

           

           

           

           

          One other thing to note, if you use the beta 2 or later bits for 4.1 (I recommend RC1 of course) you can actually make this even simpler using the IExternalDependencyData interface. Your loader class needs one more method get data():Array:

           

          ..

           

          public function get data():Array {

            return _data; //return local copy

          }

           

          private function handleResult( evt:FaultEvent ):void

          {

            _service.removeEventListener(FaultEvent.FAULT, handleResult);

            var files:XML = new XML(evt.message.body);

           

            var dataSet:Array = [[1],[3],[4]];

           

            _data = dataSet; //keep a local copy

           

          _token.notifyResult(dataSet);

          }

          ..

           

           

           

           

          But then your test class just needs to decorator the loader instead:

           

           

          [Parameters(loader="assetRetriever")]

           

          public static var assetRetriever:TestGraphicsLoader = new TestGraphicsLoader();

           

          Let me know if you would like me to drop the working code on dropbox or somewhere else so you can run what I am seeing..  but I believe the code you posted is spot on.
          HTH,
          Mike
          • 2. Re: External Data for Parameterized Tests
            brian.thomas Adobe Employee

            Hey Mike,

             

            I'm using the RC1 bits. The IExternalDependencyData method is cool - I gave that a try but am running into the same problem.

             

            The data() getter is being called before I call notifyResult() on the token. I'm not noticing any big differences between what you're doing and what I'm doing. I'll keep playing with it and see if I can give you an isolated case outside of my app.

             

            Thanks!

            Brian

            • 3. Re: External Data for Parameterized Tests
              mlabriola Level 4

              Thanks Brian. I am using this pretty extensively in projects but obviously want to ensure we cover any situation when it fails so your help is appreciated.

               

              Let me know if I can be of any further assistance,

              Mike

              • 4. Re: External Data for Parameterized Tests
                mlabriola Level 4

                Brian,

                 

                One final question... it does actually call retrieveDependency() right? Just wondering as if it never reached that method it would be one thing, but to make it there and not wait is just so weird. Sorry, it is just stuck in my head now.

                 

                Mike

                • 5. Re: External Data for Parameterized Tests
                  brian.thomas Adobe Employee

                  Yes - it does call retrieveDependency(). If you have any tips on where to drop breakpoints to figure out why it thinks it can go ahead with retrieving the data before the token is called that might be helpful as I try to debug this.

                   

                  Thanks!

                  Brian

                  • 6. Re: External Data for Parameterized Tests
                    mlabriola Level 4

                    Brian,

                     

                    If you do get a repeatable case, please let me know. I am racking my brain trying to figure out how you could make this happen after reviewing the code... here is the basic flow if you want to set breakpoints:

                     

                    When a new runner supporting external dependencies is instantiated(IExternalDependencyRunner), it creates an instance of an ExternalDependencyResolver. When instructed to resolveDependencies(), that resolver introspects the class and looks for any external dependencies. As each is found, it calls the retrieveDependency() method. The token returned from that method is stored inside of a Dictionary in the ExternalDependencyResolver and adds the resolver adds itself as a responder to the token waiting for the notifyResult() or notifyFault() to be called. When the result method is called, the dependencyResolved() method of the ExternalDependencyResolver is called.

                     

                    This resolve is responsible for two things important pieces related to startup. #1 Keep a count of outstanding dependencies. #2 dispatch and event once all outstanding dependencies are resolved.

                     

                    Regardless of how you run your tests, the code eventually gets to runRunner() in the FlexUnitCore. By the time you get here, all of your runners have been instantiated, however, it should be impossible that your external dependencies have actually been resolve due to the single threaded behavior of Flash Player. This therefore becomes my first question, are you somehow loading your data synchronously wherein, the timing is different than I imagined?

                     

                    Here the FlexUnitCore provides a new dependency watcher (instance of ExternalRunnerDependencyWatcher) to the top level runner. It in turn passes it down to each other runner that understand external dependencies (IExternalDependencyRunner), so for example, the Suite passes it to its children.

                     

                    When a runner, such as the Parameterized one, receives the watcher, it calls watchDependencyResolver() on the watcher, passing it a local instance of a dependency resolver discussed in the 2nd paragraph above( IExternalDependencyResolver). This watcher then waits for the events broadcast from the dependency resolvers. When all dependencies in a given runner have resolved, it calls ExternalRunnerDependencyWatcher.handleRunnerReady(). Once all dependencies in all runners are resolved, it calls sendReadyNotification(), which eventually calls the verifyRunnerCanBegin() method in the FlexUnit core. When everything is good across all runners (and listeners which are unrelated here but also async) then the runner begins execution.

                     

                    So, the real thing you need to know is when beginRunnerExecution() in the FlexUnitCore happens in relation to your external dependency loader calling notifyResult(). If it happens before, then we need to know how it got there. The two ways would be in runRunner(), if the core does not believe there are any external dependencies, it will be called immediately. Or from verifyRunnerCanBegin() it could be called because it believes the resolution is complete.

                     

                    HTH somewhat,

                    Mike

                    • 7. Re: External Data for Parameterized Tests
                      mlabriola Level 4

                      Brian,

                       

                      I managed to reproduce this today. I have no clue what is going on but while fixing something else I managed to find a way to reproduce.

                       

                      stay tuned. Now that I can reproduce, a fix will be forthcoming.

                       

                      Mike

                      • 8. Re: External Data for Parameterized Tests
                        brian.thomas Adobe Employee

                        Somehow this good news had gotten sucked away into an email filter void. Thanks for following up on this Mike! Is there a fix sitting around on GitHub somewhere I could try out?

                         

                        Thanks!

                        Brian

                        • 9. Re: External Data for Parameterized Tests
                          brian.thomas Adobe Employee

                          My new theory here is that it has something to do with our custom filtering mechanism. The data() method on my dependency loader is being called synchronously from the run() call on FlexUnitCore:

                           

                          com.adobe.testing.utils.dependencyLoaders::TestGraphicsLoader/get data

                          org.flexunit.runners::Parameterized/getParametersList

                          org.flexunit.runners::Parameterized/buildRunners

                          org.flexunit.runners::Parameterized/get children

                          org.flexunit.runners::ParentRunner/getFilteredChildren

                          org.flexunit.runners::ParentRunner/get description

                          org.flexunit.runners::Suite/describeChild

                          org.flexunit.runners::ParentRunner/shouldRun

                          org.flexunit.runners::ParentRunner/filter

                          org.flexunit.runner.manipulation.filters::AbstractFilter/apply

                          org.flexunit.internals.requests::FilterRequest/get iRunner

                          org.flexunit.runner::FlexUnitCore/runRequest

                          org.flexunit.runner::FlexUnitCore/runClasses

                          Function/http://adobe.com/AS3/2006/builtin::apply [no source]

                          org.flexunit.runner::FlexUnitCore/run

                           

                          This is before I've called notifyResult() and before beginRunnerExecution() takes off. I'll start digging through our filter to see if I can avoid this.
                          Brian
                          • 10. Re: External Data for Parameterized Tests
                            mlabriola Level 4

                            Interesting. Give me a real top-level overview of your filter and I will try something here as well.

                            • 11. Re: External Data for Parameterized Tests
                              brian.thomas Adobe Employee

                              Our filter is pretty straight forward. It extends AbstractFilter and uses the description passed into shouldRun() to look for some metadata annotations and decide whether or not to run it based on some (FlexUnit-unrelated) criteria elsewhere in the app.

                              • 12. Re: External Data for Parameterized Tests
                                mlabriola Level 4

                                Yep, you found the root cause. It asks for the description before it is ready.

                                 

                                Okay, I am on it. Thanks for your patience and input on this one. I will figure out how to solve.

                                 

                                Mike

                                • 13. Re: External Data for Parameterized Tests
                                  brian.thomas Adobe Employee

                                  Excellent - much appreciated!

                                   

                                  Brian

                                  • 14. Re: External Data for Parameterized Tests
                                    mlabriola Level 4

                                    Okay, here is the deal. Parameterized isn't IFilterable in the current implementation meaning that a filter should not be able to be applied to it. In fact, if you try to make a request of just one Parameterized class all is well as the filter is not applied.

                                     

                                    However, there is a bug in ParentRunner. So, when a Parameterized test is in a Suite, the filter is applied even if it is not IFilterable. Since parameterized inherits from a class that has the correct methods (even though it does not implement) it attempted to apply it instead of crashing.. and that's how we manage to get to your situation.

                                     

                                    I was never able to replicate it because I never placed the parameterized test in a Suite AND applied a filter.

                                     

                                    So, step 1, I fixed the bug in Suite. Step 2, I am allowing parameterized to be filterable. This stuff will be in my 4.1 github branch tonight.

                                     

                                    Mike

                                    • 15. Re: External Data for Parameterized Tests
                                      brian.thomas Adobe Employee

                                      Excellent - many thanks for getting to this so quickly! Can you point me to where your 4.1 branch is?