22 Replies Latest reply: Sep 10, 2014 12:25 AM by BKBK RSS

    throttled CFHTTP

    shudini1

      i recently upgraded from cf9 to cf10. running single instance on a brand new ubuntu server 12.04 with Apache. same settings as my other ubuntu servers as far as i can tell. Suddenly, all my cfhttp tags are being throttled with about 5 second breaks being pushed into each one. Each additional request made pushes the time taken up 5 seconds. i've compared every setting i could think of to the cf9 servers but nothing worked.

       

      Here's from the http.log:

      HTTP request completed {Status Code=200 ,Time taken=28525 ms}
      Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-18 
      HTTP request completed {Status Code=200 ,Time taken=22752 ms}
      Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-23 
      HTTP request completed {Status Code=200 ,Time taken=17549 ms}
      Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-17 
      HTTP request completed {Status Code=200 ,Time taken=12320 ms}
      Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-22 
      HTTP request completed {Status Code=200 ,Time taken=7137 ms}
      Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-25 
      HTTP request completed {Status Code=200 ,Time taken=4302 ms}
      Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-23 
      Starting HTTP request {URL='http://www.google.com', method='GET'}
      Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-17 
      Starting HTTP request {URL='http://www.google.com', method='GET'}
      Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-22 
      Starting HTTP request {URL='http://www.google.com', method='GET'}
      Sep 10, 2012  5:13 PM  Information  ajp-bio-8012-exec-19 
      Starting HTTP request {URL='http://www.google.com', method='GET'}

      Any help would be greatly appreciated.

       

      Thanks!

        • 1. Re: throttled CFHTTP
          psu_josh

          I am also seeing this. Tcpdump shows that it isn't the web server slowing things down. It does appear to be in the cfhttp tag. Did you find an answer to this?

          • 2. Re: throttled CFHTTP
            shudini1 Community Member

            unfortunately I never figured this out. In the meantime I downgraded to CF 9 until I either figure it out or there's an update that fixes the problem. Maybe it should be submitted as a bug request...

            • 3. Re: throttled CFHTTP
              12Robots Community Member

              Maybe?

              • 4. Re: throttled CFHTTP
                shudini1 Community Member

                by all means, submit it. would be much appreciated!

                • 5. Re: throttled CFHTTP
                  KrusiKarlsson Community Member

                  Were you able to resolve this?

                   

                  It's definitely a bug as far as we can tell.

                  • 6. Re: throttled CFHTTP
                    shudini1 Community Member

                    unfortunately I have not resolved this. I'm glad to know other people are having the issue though so at least I know it's not related to my own specific server configuration.

                     

                    I reported it to bugbase. If people vote it up there maybe we can get a fix going so I can upgrade finally.

                     

                    https://bugbase.adobe.com/index.cfm?event=bug&id=3366102

                    • 7. Re: throttled CFHTTP
                      psu_josh Community Member

                      Voted, and added a brief note on my OS being RHEL

                      • 8. Re: throttled CFHTTP
                        BKBK MVP

                        I have added the following note to the bug report. This may sound crazy, but I've seen weirder things in ColdFusion. It just might be that what you observe is expected behaviour, assuming single-threaded processing.

                         

                        Say, you have 10 cfhttp tags in a loop. When the loop starts, ColdFusion first processes a number of cfhttp start-tags, perhaps up to all 10 of them, possibly without strictly respecting order. Since the processing is single-threaded, ColdFusion executes the tag bodies in series, one after the other. This would explain why the successive execution times are cumulative.

                         

                        See Ben Nadel's blog on using cfthread with cfhttp. There he solves the problem by using multiple, asynchronous threads, like this:

                         

                        <cfloop index="intGet" from="1" to="10">

                        <!--- Start a new thread for this CFHttp call. --->

                        <cfthread action="run" name="objGet#intGet#">

                        <cfhttp method="GET" url="some_url" result="THREAD.Get#intGet#" />

                        </cfthread>

                        </cfloop>

                         

                        <!---

                        Now, we have to wait for all of concurrent threads to be joined before we can use the CFHttp results.

                        --->

                        <cfloop index="intGet" from="1" to="10">

                        <cfthread action="join" name="objGet#intGet#" />

                        </cfloop>

                        • 9. Re: throttled CFHTTP
                          shudini1 Community Member

                          the only thing is that the exact same code runs perfectly fine in a single-threaded process on CF9. it was only after upgrading that we noticed this. I'd rather not have to go into all of the code and modify it just because something changed on the CF side.

                           

                          Additionally, while in the log example I posted I was using a single threaded, single request to initiate multiple CFHTTP calls, this problem actually manifested on our site when multiple users began accessing some of our pages that make use of CFHTTP.

                           

                          However, just so no one can accuse me of not willing to try to find a workaround, I took Ben's code from this page: http://www.bennadel.com/blog/749-Learning-ColdFusion-8-CFThread-Part-II-Parallel-Threads.h tm and tested on a CF9 and CF10 configuration.

                          .

                          Here are the results from the servers:

                           

                          CF9: We Got 1000 Results in 0.72 seconds using CFHttp and CFThread

                          CF10: We Got 1000 Results in 45.02 seconds using CFHttp and CFThread

                           

                          So it doesn't really seem like parallel threads would help out here.

                          • 10. Re: throttled CFHTTP
                            BKBK MVP

                            45 to 0.7 is indeed a big ratio. That does not necessarily rule out the effect of single versus parallel threads. My guess is that some other factor in CF 10 is forcing single-threading.

                             

                            To start with, perhaps these settings in the Administrator:

                             

                            "Maximum number of simultaneous Template requests

                            The number of CFML page requests that can be processed concurrently. Use this setting to increase overall system performance for heavy load applications. Requests beyond the specified limit are queued.

                             

                            Maximum number of threads available for CFTHREAD

                            The maximum number of threads created by CFTHREAD that will be run concurrently. Threads created by CFTHREAD in excess of this are queued."

                            • 11. Re: throttled CFHTTP
                              shudini1 Community Member

                              I have a number of other pages that make use of parallel threads that have no issues. I've tuned the settings so that there are more than adequate numbers of threads available.

                               

                              Even if it was forcing single threaded processing this doesn't make any sense since the CFHTTP is not taking 5 seconds to respond. If I limit the page to a single CFHTTP request I get:

                               

                              Single request: Results in 1.16

                               

                              If threads were the issue, 10 requests should take no more than 15 seconds.

                               

                              For some reason, making two or more subsequent requests throttles the second and on with an extended delay.

                              • 12. Re: throttled CFHTTP
                                BKBK MVP

                                shudini1 wrote:

                                 

                                However, just so no one can accuse me of not willing to try to find a workaround, I took Ben's code from this page: http://www.bennadel.com/blog/749-Learning-ColdFusion-8-CFThread-Part-I I-Parallel-Threads.htm and tested on a CF9 and CF10 configuration.

                                .

                                Here are the results from the servers:

                                 

                                CF9: We Got 1000 Results in 0.72 seconds using CFHttp and CFThread

                                CF10: We Got 1000 Results in 45.02 seconds using CFHttp and CFThread

                                 

                                So it doesn't really seem like parallel threads would help out here.

                                Something is definitely wrong. My results are:

                                 

                                We Got 1000 Results in 4.95 seconds using standard CFHttp

                                We Got 1000 Results in 0.89 seconds using CFHttp and CFThread

                                 

                                They look like the kind of results one would expect. I am on CF 10.0.4.283281 (Update 4 of CF 10). Would the updates have patched something?

                                • 13. Re: throttled CFHTTP
                                  shudini1 Community Member

                                  I updated this morning to 10,0,4,283281 but the results have not changed:

                                   

                                  Single request: Results in 1.14

                                  We Got 1000 Results in 49.28 seconds using CFHttp and CFThread

                                   

                                  Here's the code I'm using to test:

                                   

                                  <cfset strBaseURL = ("http://www.google.com/search?q=Vin+Diesel&num=100&start=) />

                                  <!--- Get the starting time. --->

                                  <cfset intStartTime = GetTickCount() />

                                  <cfhttp method="GET" url="#strBaseURL#"/>

                                  <cfoutput>

                                            Single request: Results in

                                            #NumberFormat(((GetTickCount() - intStartTime) / 1000),",.00")#

                                  </cfoutput>

                                  <cfset intStartTime = GetTickCount() />

                                  <cfloop index="intGet" from="1" to="10" step="1">

                                            <!--- Start a new thread for this CFHttp call. --->

                                            <cfthread action="run" name="objGet#intGet#">

                                                 <cfhttp method="GET" url="#strBaseURL##((intGet - 1) * 100)#"

                                                      useragent="#CGI.http_user_agent#"

                                                      result="THREAD.Get#intGet#" />

                                            </cfthread>

                                  </cfloop>

                                   

                                  <cfloop index="intGet" from="1" to="10" step="1">

                                            <cfthread action="join" name="objGet#intGet#" />

                                  </cfloop>

                                   

                                  <cfoutput>

                                            <!--- Output retrieval times. --->

                                            <p>We Got 1000 Results in

                                                  #NumberFormat(((GetTickCount() - intStartTime) / 1000),",.00")#

                                                 seconds using CFHttp and CFThread</p>

                                  </cfoutput>

                                  (side point: i don't think google is actually returning the results as they do not permit scraping this way, but it demonstrates the point regardless that the requests are being throttled)

                                  • 14. Re: throttled CFHTTP
                                    shudini1 Community Member

                                    forgot to post the logs for the above code (running the updated CF):

                                     

                                    DateTimeSeverityThreadIDApplication Name
                                    Nov 19, 2012  9:40 AM  Information  cfthread-29 
                                    HTTP request completed {Status Code=200 ,Time taken=48882 ms}
                                    Nov 19, 2012  9:40 AM  Information  cfthread-30 
                                    HTTP request completed {Status Code=200 ,Time taken=43850 ms}
                                    Nov 19, 2012  9:40 AM  Information  cfthread-15 
                                    HTTP request completed {Status Code=200 ,Time taken=38868 ms}
                                    Nov 19, 2012  9:40 AM  Information  cfthread-31 
                                    HTTP request completed {Status Code=200 ,Time taken=33949 ms}
                                    Nov 19, 2012  9:40 AM  Information  cfthread-35 
                                    HTTP request completed {Status Code=200 ,Time taken=28826 ms}
                                    Nov 19, 2012  9:39 AM  Information  cfthread-33 
                                    HTTP request completed {Status Code=200 ,Time taken=23890 ms}
                                    Nov 19, 2012  9:39 AM  Information  cfthread-34 
                                    HTTP request completed {Status Code=200 ,Time taken=18835 ms}
                                    Nov 19, 2012  9:39 AM  Information  cfthread-36 
                                    HTTP request completed {Status Code=200 ,Time taken=13827 ms}
                                    Nov 19, 2012  9:39 AM  Information  cfthread-32 
                                    HTTP request completed {Status Code=200 ,Time taken=8878 ms}
                                    Nov 19, 2012  9:39 AM  Information  cfthread-28 
                                    HTTP request completed {Status Code=200 ,Time taken=3877 ms}
                                    • 15. Re: throttled CFHTTP
                                      Adam Cameron. Community Member

                                      It's not much help, but I ran a few tests and I don't see what you're seeing on my CF10 install: all CFHTTP requests are being fired immediately, and the length of time they take to complete is roughly the same per request. IE: I'm not seeing anything I would not expect to see if the thing was working properly.

                                       

                                      My CF10 install is a bog-standard one, patched to 10.0.4.  CF10 Developer, running on Windows 7 Home 64-bit.

                                       

                                      --

                                      Adam

                                      • 16. Re: throttled CFHTTP
                                        psu_josh Community Member

                                        So this to me helps narrow down the problem. You are saying it runs fine for you on windows. So far the only people who have reported issues are on Linux.

                                        • 17. Re: throttled CFHTTP
                                          psu_josh Community Member

                                          I can also report that 10,0,5,283319 doesn't solve this issue for me.

                                          • 18. Re: throttled CFHTTP
                                            BKBK MVP

                                            psu_josh wrote:

                                             

                                            I can also report that 10,0,5,283319 doesn't solve this issue for me.

                                            Not that this would matter. 10.0.5 is a Windows security update. As you say, the bottle-neck is likely a Linux issue. So what is it on Linux systems that is queueing up the threads?

                                            • 19. Re: throttled CFHTTP
                                              rupesh_kumar Adobe Employee

                                              We investigated this and found out that this does happen on some Linux machines but not all. The reason it happens is because Apache HttpClient library creates a SecureRandom number for every http call which can be slow on new Unix boxes as there are not enough entropy generated on those machines.

                                               

                                              There are few possible ways to solve this and I would recommend either of these two.

                                              1. 1. Set this system property to your JVM – if you are using standalone CF installation, you would set it in jvm.config.

                                              “-Djava.security.egd=file:/dev/./urandom”

                                              1. 2. In $JAVA_HOME/jre/lib/security/java.security file, change the value of securerandom.source to file:/dev/./urandom

                                               

                                              You can also refer to this excellent post by Shilpi from the ColdFusion team which talks about this issue

                                              http://www.shilpikhariwal.com/2012/04/random-number-generation-in-unix.html

                                              • 20. Re: throttled CFHTTP
                                                shudini1 Community Member

                                                Using option #1 from above seems to have fixed the issue for me. I added it to the JVM arguments in CF admin.

                                                 

                                                New results:

                                                 

                                                Single request: Results in 1.96

                                                We Got 1000 Results in 0.93 seconds using CFHttp and CFThread

                                                 

                                                Thank you VERY much!

                                                • 21. Re: throttled CFHTTP
                                                  cfmcoder

                                                  Hi Shudin,

                                                  I tried above solution doesn't work. I am on CentOS 6. What OS you are testing this?

                                                   

                                                  Thanks.

                                                  • 22. Re: throttled CFHTTP
                                                    BKBK MVP

                                                    @cfmcoder

                                                    You should start your own thread, as this one was last active almost 2 years ago. People, and Coldfusion, have moved on.