Skip navigation
D@yzW0rk
Currently Being Moderated

IIS Crashes with JRun Connector faulted?

May 8, 2012 10:03 AM

Tags: #debugging #iis_7.5 #crash_dump #jrun_connector

About our environment:

Win Server 2008 r2 64 bit, 8GB Ram

 

Our web server experienced a series of undetectable speed bumps for events that should have zero negative impact.  One such hiccup was the update from 9.0 to 9.0.1 on March 29, 2012 which broke the CF Web Admin page.  I'm not the web server admin so my ability to test for a fix was limited.  Searching for a solution online was futile.  The error given was:

Diagnostics Element ESAPIUTILS is undefined in a Java object of type class [Ljava.lang.String;. <br>The error occurred on line 55.

 

Eventually we requested a single incident request from Adobe tech support.  The first day the support technician visited our site there was little more than just simple prodding.  The technician claimed he needed to generate a VM to mimic our setup, which I still question how that was intended to be conducted.  The VM creation was never fulfilled and did nothing but delay repairs due to scheduling confilicts with Adobe, our admins availability, and the weekend.  Eventually the technician was able to work with us again and the CF web admin was fixed.  However, before the fix was put into place the technician asked twice, once each call, if we applied any hotfixes.  The answer was no both times asked on two separate days.  The technician appeared to believe we applied a Hotfix due to some existence of a file somewhere in some directory that appears to me, without being said, was the indicator of his belief that a hotfix was applied which it was not.

 

The technician suspected the source of the problem could be:

  • A hotfix applied previously, not the case.
  • Permissions
  • Corrupt file(s)

 

In the end applying a hotfix appears to be what eventually fixed our problem.

This was the processes approached during the technicians fix:

\ColdFusion9\runtime\servers\coldfusion\SERVER-INF\jrun.xml

some value was set to true to allow local web server to run.

http://helpx.adobe.com/coldfusion/kb/cumulative-hot-fix-2-coldfusion-1 .html

http://helpx.adobe.com/coldfusion/kb/cumulative-hot-fix-2-coldfusion-1 .html

http://helpx.adobe.com/content/help/en/coldfusion/kb/coldfusion-securi ty-hotfix.html#main-pars_heading_9

http://helpx.adobe.com/coldfusion/kb/coldfusion-administrator-fails-pe rmission-denied.html

 

Shortly after the fix over a weekend I was informed that our site was down.  It appeared Coldfusion was running still and IIS had something wrong since I was getting error 503.  I tested by swapping out an index.html on an unaffected site which gave back the output desired.  I learned later that logs also indicated that CF had not been restarted or failed.  It turned out that the IIS default application pool had stopped.  Starting this again in the IIS admin console alieivated the immediate problem.  However, we continued to get error 503 now and then indicating that an application pool has stopped.

 

DebugDiag was put into play to gather crash dumps from IIS to try and troubleshoot.  When we had gathered about 5 crash dumps and each analysis had mentioned jrun_iis6_wildcard.dll in the results I suspected that something to do with the recent fix was the cause.  However, when I contacted the Adobe technician on April 18th they claimed that the fix put into place could in no way affect the jrun_iis6_wildcard.dll.  When I brought up that I had crash dumps from IIS with indication that  the jrun connector dll was at fault he said that I would have to get in contact with Microsoft to analyze the crash dump to verify that.  I'm in the process of starting a support request with Microsoft for that, but I thought this would be useful to post if anyone else has experienced this.

 

I have noticed since the fix also that I'm seeing error 503 in the http.log file now and then.  I've also noticed that our site is far more sensative to bots visiting which I intend to address shortly to allieviate resource usage.

 

WinDebug analysis; tail lines of an IIS crash dump.

STACK_COMMAND:  ~8s; .ecxr ; kb

 

SYMBOL_STACK_INDEX:  7

 

SYMBOL_NAME:  jrun_iis6_wildcard!TerminateExtension+425

 

FOLLOWUP_NAME:  MachineOwner

 

MODULE_NAME: jrun_iis6_wildcard

 

IMAGE_NAME:  jrun_iis6_wildcard.dll

 

DEBUG_FLR_IMAGE_TIMESTAMP:  4a980f56

 

FAILURE_BUCKET_ID:  INVALID_POINTER_WRITE_c0000005_jrun_iis6_wildcard.dll!TerminateExtens ion

 

BUCKET_ID:  X64_APPLICATION_FAULT_INVALID_POINTER_WRITE_jrun_iis6_wildcard!Termin ateExtension+425

 

So my question is there any suspicion that the JRun connector is faulted or is this simply caused by excessive traffic exceeding some limits of the CF connector or do I need to focus on IIS application pool process?

 
Replies
  • Currently Being Moderated
    May 8, 2012 5:47 PM   in reply to D@yzW0rk

    Seems a lot going on there. Some other thing to know.

    Is CF9 64 or 32 bit? Eg:

    CFadmin > System Information > section Java VM Name = Java HotSpot(TM) 64-Bit Server VM   

    Is CF9 Standard or Enterprise licence?

    If CF9 Standard; when you run CFSTAT do you get Req Q'ed and Reg TO'ed values other than zero?

    If CF9 Enterprise; does CF Monitor show anything interesting? Eg:

    Requests with errors, Requests that time out, Requests slower than 20 seconds and JVM Memory (used/max).

    Something else that may help. Are there any warning errors in \ColdFusion9\runtime\logs

    coldfusion-event.log and coldfusion-out.log ?

    Why bother with CFSTAT, CF Mon and runtime logs? Sometimes the CF IIS connector errors after some other threshold has been reached elsewhere in CF.

    HTH, Carl.

     

    
     
    |
    Mark as:
  • Currently Being Moderated
    May 9, 2012 7:22 PM   in reply to D@yzW0rk

    CFSTAT is a command utility via:

    CMD prompt

    CD \ColdFusion9\bin

    cfstat 5

     

     

     

    refer:

    http://help.adobe.com/en_US/ColdFusion/9.0/Admin/WSc3ff6d0ea7785946117 2e0811cbf3638e6-7fe0.html#WS461F2C7B-9331-43eb-BB55-92BE901135C1

     

     

     

     

    JConsol and other tools from Java Dev Kit like Jvisualvm can be good tools to use to observe CF JVM activity.

     

     

     

    You have changed garbage collector and some memory settings. Since the server has 8gb perhaps setting min and max heap to 1024m might still be on the small side, perhaps min 1024m and max 4096m might be better tho Jconsol usage graph might show better choice of values.

     

     

     

    What do you have set for min and max Permanent generation size since MaxPermSize=192m (ie default CF install value) can be small?

     

     

     

     

    coldfusion-out\d\d\d.logs showing this is normal nothing to worry:

    warning Unable to open <driveLetter>:\ColdFusion9\runtime/lib/license.properties

     

     

    Regards, Carl.

     

    
     
    |
    Mark as:
  • Currently Being Moderated
    May 11, 2012 8:28 PM   in reply to D@yzW0rk

    Req TO'ed - 1 per minute sounds a lot to me. Does CFadmin > Server Settings > Settings > Timeout Requests after ( seconds) meet your requirements?

     

    Adobe supports CF8 and CF9 with JDK 1.6.0_24 so that should work. Your JVM.CONFIG points java.home= to that it seems but is that a JRE install or JDK install (JDK will include JRE)?

     

    To use Jconsole you  are going to need to apply some java.args changes. Are these applied?

     

    -Dcom.sun.management.jmxremote
    -Dcom.sun.management.jmxremote.port=<port number>
    -Dcom.sun.management.jmxremote.authenticate=false

     

    Since you have MaxPermSize = 512m perhaps you do well to specify an initial setting for perm gen eg -XX:PermSize=224m

     

    HTH, Carl.

     

     

     

    
     
    |
    Mark as:
  • Currently Being Moderated
    May 12, 2012 6:23 PM   in reply to D@yzW0rk

    @d@yz,

     

    There’s too much going on for me to be positive, but from what you’ve said I would suspect that things broke for you (if you really did not apply any hotfixes) when you ran the updater. It asks where the CFIDE is, which is to be updated. You could have accepted the default, or changed it to some value. Do you recall?

     

    More important, do you know what CFIDE is pointed to for the web site in IIS that you use to view the CFAdmin? Also, you refer to the jrun.xml and Adobe wondering about the “allow local web server to run”, that’s referring to the internal web server for CF (such as using port 8500, by default). Have you (or they) tried to access the CF Admin that way? And if so, what directory does that live in? 

     

    Here’s the thing: people sometimes have multiple different copies, and if they’re not careful when applying updaters (or hotfixes), they can apply one or the other to the wrong one. Again, you may then see a difference depending on what directory you updated and what directory is used for the CFIDE for the web server you’re using, and what web server you’re using to view the Admin.

     

    Resolving things can be difficult because there are so many moving parts, and different people may have had a hand in updating things on your end. Even if you did it yourself, you may not accurately remember exactly what you did. Worse, there’s no easy way to assess what a “correct” CFIDE should look like for your current level of configuration, since different updates will change different files in the directory.

     

    And then when you apply hotfixes, there are often other directories you are told to update (web-inf/lib and more), which can also then be troublesome if you update the wrong one (easy to do when you have multiple instances). Finally, another gotcha I see often is that when people extract the zips offered in the hofixes, they may extract then mistakenly to the wrong level in the destination directory. (But you said you have not applied hotfixes, so I realize this last paragraph doesn’t apply to you.)

     

    Finally,  I’ll point out for you and other readers that I blogged about this issue with some different details:

     

    http://www.carehart.org/blog/client/index.cfm/2011/10/21/why_chfs_may_ break

     

    Hope some of that helps.

     

    /charlie

     
    |
    Mark as:
  • Currently Being Moderated
    May 15, 2012 12:44 PM   in reply to D@yzW0rk

    No, the CAR would not have had any connection with the previous hotfixes. I’ll just say some step had to have been a mistake along the way. No way to tell now, this far removed.

     

    I’ll just caution everyone to be VERY careful applying updates, with respect to what CFIDEs you have (that you may not even think of), and which you do update. And I’ll suggest (as before) that if things break, there is almost always going to be an explanation in one of the steps followed, for the reasons I outline.

     

    Sorry that I can’t offer more for your specific challenge, though. Glad you’re up and working again.

     

     

     

    /charlie

     
    |
    Mark as:
  • Currently Being Moderated
    May 23, 2012 6:26 PM   in reply to D@yzW0rk

    I had a quick look at the details of jrun_iis6_wildcard.dll on 3 systems running CF9. The 32 bit CF Developer, 64 bit CF Standard and 64 bit Enterprise multiserver each had different jrun_iis6_wildcard.dll details. Not sure if any of that is helpful. Let me know if want more information.

    
     
    |
    Mark as:
  • Currently Being Moderated
    May 28, 2012 4:31 PM   in reply to D@yzW0rk

    No, the IIS connector has nothing to do with JDBC. What I suspect you’re seeing is that (for whatever reason) requests stop coming in (so no JDBC activity happens). Certainly if IIS stops letting requests in (which could happen due to problems with the IIS connector), then you’d see what you see. I’ll suggest you also look at the Windows Event Viewer to see if there are any errors there, especially with respect to the IIS Application Pool(s) you may have setup for use by the site(s) connecting to CF.

     

    So as for the differences you’re observing, are you doing any manual tweaks to either file contents or file version with respect to the web server connectors for CF? And either way, if you think things are somehow not “right”, have you tried just removing and re-adding the web server configuration (using the CF web server config tool)? Sometimes, that’s all you need to do to sort things out. Sometimes, also, it can help to remove and replace either the IIS app pool or IIS site (or reinstall IIS itself).

     

    No, I’m not saying you “should have to do this”, nor that it’s right (if it fixes things) that you should have had to do it. I’m just proposing some things to consider, since on the forums here we have so little to go on and only brief moments to communicate thoughts.

     

    I’ll just say that if you continue to struggle with things not feeling quite right, I (and others) can be available to help for paid assistance (done remotely, without providing “remote access”, looking “over your shoulder” using Adobe Connect or the like), with no minimum time requirement, and (for me at least) with a satisfaction guarantee that you don’t have to pay for time you don’t feel is valuable.

     

    Finally, I’m not at all surprised to hear that spiders and bots (and error handling) are at root of your problems. I find that when I help people they are indeed the root cause of many problems, though the problems may present themselves in many different ways, which is why it’s not often obvious how to connect the dots (in email discussions like this). It often becomes far more apparent in a shared desktop session where I’m looking at things with someone. Your call though.  Just pointing out an option.

     

    Also, as in the case of the past 12 days, sometimes I’m just unavailable on the forums due to travel, or sometimes just for being busy helping folks, so a direct engagement is also a way to get help sooner, as well as more directly. But as you can see, I Re: IIS Crashes with JRun Connector faulted? do try to help here however we may, so this suggested alternative for help is not meant as an ultimatum, nor a carrot/stick. Just an option to consider.

     

    /charlie arehart

     

    charlie@carehart.org

     

    Providing fast, remote, on-demand troubleshooting services for CF (and CFBuilder)

     

    More at http://www.carehart.org/consulting

     

    See also http://www.cf911.com for more on CF troubleshooting resources

     
    |
    Mark as:
  • Currently Being Moderated
    Jun 29, 2012 8:07 AM   in reply to D@yzW0rk

    Congrats on both count, and thanks for the kind regards.

     

     

     

    /charlie

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points