3 Replies Latest reply on Dec 14, 2009 10:15 AM by jwbates

    Concurrent soap requests

    jwbates

      I'm having a heck of a time here, and I really have no idea what to do.

       

      Backstory:

       

      I have a proprietary server application that needs managing. Up until this point, while the server was under development, I was opening a port and connecting an AIR application using XMLSocket. That was working fine, up to the limitations of the simplistic exchange protocol we had in place, but as we moved towards final product, we rewrote that protocol, closed off the port to everybody but the local host, and wrote a python server to communicate with that port, with the idea that the python code would be able to be shared with 1) command-line utilities on the box and 2) a python-based web server that would expose a more robust API via SOAP. So now we're running CherryPy and soaplib, and I'm busily trying to rewrite the AIR app and bring it up to production quality.

       

      It looked so easy on paper. And it even seemed pretty easy at the start. Pull down the "Import wsdl...", point it at my server, and wham-bang, tons and tons of code for blabbering back and forth. (It's worth noting at this point that the performance demands of the application are very low, and we shouldn't be doing more than a request or two a second.) And with a little bit of wrangling, that seemed to be what was happening.

       

      But then I moved to actually trying to implement the application. Here's the basics of how it's supposed to work: one object polls for data from the server. There are several different types of data that it's pulling down, so it has to issue four requests to get the full set. I set a Timer to go off every four seconds to issue the requests.

       

      Problem the first: when I had the code issue all four requests in sequence, the system would eventually halt (error #2032) at the same point (62 requests in, although that included my authentication requests). Also, it didn't seem to be sending all the requests; at least, my pending queue was filling up. So I figured that some internal queue was filling up, and requests were only getting popped off one at a time.

       

      Try it this way: now my timer pops every second, but issues only one request at a time. OK, that worked. It'd poll indefinitely.

       

      Oh, but now, problem the second: if I hold down the mouse button, say, to resize the screen, the requests start piling up again. What's more, and this is the funky thing: if I shut down the entire application and start it up again, now my initial authentication checks fail the first time through. Why? Because they are getting RESPONSES FROM THE LAST SET OF REQUESTS. Argle-flargle-klabble!

       

      In fact, now that I start checking the data, everything is just plain screwed-up. I'm consistently getting callbacks fired on AsyncTokens that are totally unrelated to the data in the response. I.e., I have four resources I'm polling: call them AM, CM, PM, and VM, and four RPCs, list_AM, list_CM, list_PM, and list_VM. I call into the CAWebService for an AsyncToken for, say, list_AM. I call token.AddEventListener("result", handleResult). When handleResult fires on that token, everything looks right: the token contains the right event, the right response type, and the correct ID (set by me when I get the token). But when I actually look at the data in the token.response, well, it contains the response to a list_VM call.

       

      I'm lost. I can't figure out what to do. I've reloaded the WSDL dozens of times. I've checked the web server over and over: whenever the requests are issued one at a time via test applications, the data comes back perfectly. I could rewrite the soap API to include a unique token, as I did in the original XMLSocket implementation, but that still doesn't solve the problem of requests getting dropped.

       

      What can possibly be going on? What kinds of things can I do to figure out what's going on?

       

      And most importantly, can anybody tell me a magic word to use to fix my problem? ("Oh, yeah, you needs to set the AbstractWebService.imadummy flag to true. That'll reconfigure the wsdl abstraction to generate the fluxCapacitor flag, thus generating the 1.21 gigawatts of energy necessary to fry your brain so that you can stop worrying about this stuff and finally get some sleep.")

        • 1. Re: Concurrent soap requests
          paul.williams Level 4

          A few thoughts on this architecture:

           

          1) Initiating a remote operation every second without any regard for the success or failure of previous operations is not a robust way to poll a server. Send one request, wait for a result or fault. Then you can optionally wait for a set period of time (a polling interval). Then send the next request.

           

          2) Your Timer isn't guaranteed to fire every second. If the framerate of the flash player falls due to increased client-side processing (such as redisplaying the screen during a resize operation), then the accuracy of the Timer will fall. This will affect the regularity of your polled requests and may cause them to pile-up.

           

          4) If your client needs to do something as regularly as every second then your server should be designed to make this as simple as possible. So why not amalgamate all four operations into a single operation that returns all the information your client needs?

           

          I can't comment on the other issues you mention without seeing more code.

          1 person found this helpful
          • 2. Re: Concurrent soap requests
            JeffryHouser Level 4

            I don't have a magic answer to give you, but a few things come to mind:

             

            Isn't 1-2 requests a second a staggeringly huge number?  That'd be 120 request per minute, or 7200 in an hour.  If I understand correctly, that is from a single user.  Is everything self contained in the desktop or is your server supporting multiple users?

             

            Instead of firing off requests on a timer, should you fire them off the next request in the 'return' sequence?  I would expect that AIR / Flex / a browser / and your OS's TCPIP stack would have limits to the number of concurrent remote requests it will send.

            1 person found this helpful
            • 3. Re: Concurrent soap requests
              jwbates Level 1

              Thanks for your replies!


              On Dec 14, 2009, at 8:20 AM, JeffryHouser wrote:

              I don't have a magic answer to give you, but a few things come to mind:

              Isn't 1-2 requests a second a staggeringly huge number?  That'd be 120 request per minute, or 7200 in an hour.  If I understand correctly, that is from a single user.  Is everything self contained in the desktop or is your server supporting multiple users?
              Not in this context, no. This UI is a control mechanism for a server which is handling tens of thousands of requests per second per instance, and aggregation of multiple server instances, local and remote, occurs via the internal clustering protocol. We don't anticipate seeing more than one or two local administrators on the system concurrently, and even then only a few times per week. The polling overhead is noise compared to even just the (gossipy) heartbeating within the cluster.
              Instead of firing off requests on a timer, should you fire them off the next request in the 'return' sequence?  I would expect that AIR / Flex / a browser / and your OS's TCPIP stack would have limits to the number of concurrent remote requests it will send.
              Yeah. Duh. Good point.
              On Dec 14, 2009, at 8:13 AM, paul.williams wrote:

              A few thoughts on this architecture:

              1) Initiating a remote operation every second without any regard for the success or failure of previous operations is not a robust way to poll a server. Send one request, wait for a result or fault. Then you can optionally wait for a set period of time (a polling interval). Then send the next request.
              I reiterate: yeah. Duh. Good point.
              2) Your Timer isn't guaranteed to fire every second. If the framerate of the flash player falls due to increased client-side processing (such as redisplaying the screen during a resize operation), then the accuracy of the Timer will fall. This will affect the regularity of your polled requests and may cause them to pile-up.
              Made this change and the pending queue problem went away. So. Once again. Yeah. Duh. Good point.

              4) If your client needs to do something as regularly as every second then your server should be designed to make this as simple as possible. So why not amalgamate all four operations into a single operation that returns all the information your client needs?
              Well, now, that's an interesting discussion. If I were serving thousands, hundreds, or even dozens of users, I'd definitely agree with you, but...
              A central conceit of soap and other web-service protocols is that they make issuing network calls as easy as making a local RPC. You can thus map your API's functionality directly from server to client, maintaining the original abstraction, so to speak.
              Aggregating the four operations is logical: in fact, that's the way the original protocol worked. But it introduces a new abstraction: I have to have a bundler, and unbundler, and a new RPC to retrieve the bundle. That's a maintenance cost: probably not too much, but it means that any time I add a new retrieval module, I have to touch that bundler. Furthermore, I have to make sure that everybody who might add a retrieval module would also have to understand the bundler, or find somebody who does.
              In fact, the original protocol was much more aggressive about minimizing RPC overhead, resulting in the problem that any engineer who wanted to modify any aspect of the front-end interaction wound up having to understand enough about the protocol to become a protocol engineer, or found a (usually inefficient) way to modify their own requirements to fit into the existing (sloppy) RPC structures.
              I'm not talking about rocket science, of course, but engineering under a deadline means that the less interdependency you have, the better.
              The benefit that we've seen by moving to soap is that those interdependencies went away. Our engineers are perfectly happy designing system calls, and that's exactly what our external interface now looks like. The result is that the manageability of the system has increased, as more of the functionality is exposed, and the maintainability has increased, as the engineers are now directly responsible for their own features.
              Keeping the benefits of that abstraction in mind, and the fact that it's unlikely that our relatively moderate network overhead will even register, those four operations that I'm doing are from four completely independent subsystems. Aggregating them into a single request would look like, from the perspective of our current abstraction, writing a new system call to retrieve disk usage, process information, network statistics, and an audit trail. Sure, that system call would save on context switches within one application, but what sense would it make?

              I can't comment on the other issues you mention without seeing more code.
              I found one .imadummy flag: the CherryPy web server starts up by default with only one thread. It didn't occur to me to check that until I'd written out the problem. I'm still worried about the mixed-up responses, but as long as they happen only infrequently, I can deal with them.
              Thanks for your help!
              John