IIS Crashes with JRun Connector faulted?

[Guest]

About our environment:

Win Server 2008 r2 64 bit, 8GB Ram

Our web server experienced a series of undetectable speed bumps for events that should have zero negative impact.  One such hiccup was the update from 9.0 to 9.0.1 on March 29, 2012 which broke the CF Web Admin page.  I'm not the web server admin so my ability to test for a fix was limited.  Searching for a solution online was futile.  The error given was:

Diagnostics

Element ESAPIUTILS is undefined in a Java object of type class [Ljava.lang.String;. <br>The error occurred on line 55.

Eventually we requested a single incident request from Adobe tech support.  The first day the support technician visited our site there was little more than just simple prodding.  The technician claimed he needed to generate a VM to mimic our setup, which I still question how that was intended to be conducted.  The VM creation was never fulfilled and did nothing but delay repairs due to scheduling confilicts with Adobe, our admins availability, and the weekend.  Eventually the technician was able to work with us again and the CF web admin was fixed.  However, before the fix was put into place the technician asked twice, once each call, if we applied any hotfixes.  The answer was no both times asked on two separate days.  The technician appeared to believe we applied a Hotfix due to some existence of a file somewhere in some directory that appears to me, without being said, was the indicator of his belief that a hotfix was applied which it was not.

The technician suspected the source of the problem could be:

• A hotfix applied previously, not the case.
• Permissions
• Corrupt file(s)

In the end applying a hotfix appears to be what eventually fixed our problem.

This was the processes approached during the technicians fix:

\ColdFusion9\runtime\servers\coldfusion\SERVER-INF\jrun.xml

some value was set to true to allow local web server to run.

http://helpx.adobe.com/coldfusion/kb/cumulative-hot-fix-2-coldfusion-1.html

http://helpx.adobe.com/coldfusion/kb/cumulative-hot-fix-2-coldfusion-1.html

http://helpx.adobe.com/content/help/en/coldfusion/kb/coldfusion-security-hotfix.html#main-pars_headi...

http://helpx.adobe.com/coldfusion/kb/coldfusion-administrator-fails-permission-denied.html

Shortly after the fix over a weekend I was informed that our site was down.  It appeared Coldfusion was running still and IIS had something wrong since I was getting error 503.  I tested by swapping out an index.html on an unaffected site which gave back the output desired.  I learned later that logs also indicated that CF had not been restarted or failed.  It turned out that the IIS default application pool had stopped.  Starting this again in the IIS admin console alieivated the immediate problem.  However, we continued to get error 503 now and then indicating that an application pool has stopped.

DebugDiag was put into play to gather crash dumps from IIS to try and troubleshoot.  When we had gathered about 5 crash dumps and each analysis had mentioned jrun_iis6_wildcard.dll in the results I suspected that something to do with the recent fix was the cause.  However, when I contacted the Adobe technician on April 18th they claimed that the fix put into place could in no way affect the jrun_iis6_wildcard.dll.  When I brought up that I had crash dumps from IIS with indication that  the jrun connector dll was at fault he said that I would have to get in contact with Microsoft to analyze the crash dump to verify that.  I'm in the process of starting a support request with Microsoft for that, but I thought this would be useful to post if anyone else has experienced this.

I have noticed since the fix also that I'm seeing error 503 in the http.log file now and then.  I've also noticed that our site is far more sensative to bots visiting which I intend to address shortly to allieviate resource usage.

WinDebug analysis; tail lines of an IIS crash dump.

STACK_COMMAND:  ~8s; .ecxr ; kb

SYMBOL_STACK_INDEX:  7

SYMBOL_NAME:  jrun_iis6_wildcard!TerminateExtension+425

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: jrun_iis6_wildcard

IMAGE_NAME:  jrun_iis6_wildcard.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4a980f56

FAILURE_BUCKET_ID:  INVALID_POINTER_WRITE_c0000005_jrun_iis6_wildcard.dll!TerminateExtension

BUCKET_ID:  X64_APPLICATION_FAULT_INVALID_POINTER_WRITE_jrun_iis6_wildcard!TerminateExtension+425

So my question is there any suspicion that the JRun connector is faulted or is this simply caused by excessive traffic exceeding some limits of the CF connector or do I need to focus on IIS application pool process?

[carl type3]

Seems a lot going on there. Some other thing to know.

Is CF9 64 or 32 bit? Eg:

CFadmin > System Information > section Java VM Name = Java HotSpot(TM) 64-Bit Server VM   

Is CF9 Standard or Enterprise licence?

If CF9 Standard; when you run CFSTAT do you get Req Q'ed and Reg TO'ed values other than zero?

If CF9 Enterprise; does CF Monitor show anything interesting? Eg:

Requests with errors, Requests that time out, Requests slower than 20 seconds and JVM Memory (used/max).

Something else that may help. Are there any warning errors in \ColdFusion9\runtime\logs

coldfusion-event.log and coldfusion-out.log ?

Why bother with CFSTAT, CF Mon and runtime logs? Sometimes the CF IIS connector errors after some other threshold has been reached elsewhere in CF.

HTH, Carl.

[Guest]

CF9.0.1 installation = 64bit

Java VM Name    Java HotSpot(TM) 64-Bit Server VM

Standard license

Is CFSTAT a command line utility?  I'm not familiar with that.

I haven't yet begun viewing the server from JConsole to get memory heap usage stats.  I'll be doing that at some point next week.  I do know we had out of memory issues when the mentioned below issue with incorrect cfhttp was being used in one of our processes.  I assisted the garbage collector to be a little less latent by changing the algorightm from default to: -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC and also changing the min/max heap values to both 1024MB.  Before that the server achieved many GC overhead limit exceeded errors.  All caused by bot traffic when entering the directory that made improper use of cfhttp.

There was also a lesson to be learned from improper use of <cfcache timespan=> In all this.  I found that our site had made habbit of using now() within the <cfquery> tag which negated the cache attempt because every query was unique thus disabling the cache approach.  With enough traffic the database was overloaded and resulted in a surge of timeouts accompanied by other problems.  I believe the Garbage Collector stopped the world indefinately during these events resulting a forced service restart to restore functionality.

We used to get a huge number of request timeouts for cfhttp, but that was mainly due to abuse and incorrect use of cfhttp for "if file exists" methods.  That was corrected to check for file using the I/O filesystem methods.

Not a huge number of warnings, but yes about 14 recorded in server.log for the month of May, 2012 so far.  All had this similar message: 04/20 07:00:40 Warning [jrpp-147] - Thread: jrpp-147, processing template: <template>, completed in 30 seconds, exceeding the 30 second warning limit

Weird errors found in coldfusion-out\d\d\d.logs:

04/19 08:16:14 warning Unable to open <driveLetter>:\ColdFusion9\runtime/lib/license.properties

04/24 07:23:48 warning Unable to open <driveLetter>:\ColdFusion9\runtime/lib/license.properties

[carl type3]

CFSTAT is a command utility via:

CMD prompt

CD \ColdFusion9\bin

cfstat 5

refer:

http://help.adobe.com/en_US/ColdFusion/9.0/Admin/WSc3ff6d0ea77859461172e0811cbf3638e6-7fe0.html#WS46...

JConsol and other tools from Java Dev Kit like Jvisualvm can be good tools to use to observe CF JVM activity.

You have changed garbage collector and some memory settings. Since the server has 8gb perhaps setting min and max heap to 1024m might still be on the small side, perhaps min 1024m and max 4096m might be better tho Jconsol usage graph might show better choice of values.

What do you have set for min and max Permanent generation size since MaxPermSize=192m (ie default CF install value) can be small?

coldfusion-out\d\d\d.logs showing this is normal nothing to worry:

warning Unable to open <driveLetter>:\ColdFusion9\runtime/lib/license.properties

Regards, Carl.

[Guest]

Thanks for taking the time to look this over and respond Carl.

I don't have "Enable Performance Monitoring" Enabled as the document specifys, but I do have "Enable CFSTAT" checked.  I hope that suffices.

I gathered some cfstat data.  Monitored the output for about 2 minutes and saw a trend:

Reqs Q'd = 0

Reqs To 'ed = 2

Looking at taskmgr I see 3.96GB Physical memory in use, roughly 49% running idle.  I'll see if I can get JConsole or something like that to get an overhead view of resoure usage from the JVM perspective.  Maybe increasing to 4096m max might be a bit too high or is the JVM manager intelligent enough not to overstep the boundry if not enough memory exists to allocate to the heap?

I bumped up to -XX:MaxPermSize=512m at the same time the GC algorithm was modded.  That may not be adequate either.

Update:

Let it be known that JDK 1.6.0_24 was installed on this machine.  I'm wondering now if there is a problem with that installation after seeing this error.

Attempting to run JConsole resulted in this error:

"could not find the main class sun.tools.jconsole.jconsole. program will exit"

I then ran java -version and got the expected result:

java version "1.6.0_24"

I then tried running javac -version and got a similar error to the jconsole error:

Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/tools/javac/Main

.........

the program will now exit.

I know I specified to the server admin to preserve the old \Coldfusion9\runtime\jre directory so we created a new \Coldfusion9\runtime\jre1.6.0_24 directory for that version to be installed to.  The reason was in response to this: http://www.carehart.org/blog/client/index.cfm/2011/10/28/CF911-Have-you-updated-your-ColdFusion-JVM-...

Probably a bad classpath, but I'm not the server admin so can't view those variables or correct them.  I noticed that the CF admin sees this as Java Home: \ColdFusion9\runtime\jre1.6.0_24\jre whereas a default install sees \ColdFusion9\runtime\jre as Java Home.  This looks correct for directory structure.  Noticeable difference with abundance of LICENSE files in the default.

[carl type3]

Req TO'ed - 1 per minute sounds a lot to me. Does CFadmin > Server Settings > Settings > Timeout Requests after ( seconds) meet your requirements?

Adobe supports CF8 and CF9 with JDK 1.6.0_24 so that should work. Your JVM.CONFIG points java.home= to that it seems but is that a JRE install or JDK install (JDK will include JRE)?

To use Jconsole you  are going to need to apply some java.args changes. Are these applied?

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=<port number>
-Dcom.sun.management.jmxremote.authenticate=false

Since you have MaxPermSize = 512m perhaps you do well to specify an initial setting for perm gen eg -XX:PermSize=224m

HTH, Carl.

[Guest]

@carl_type3 CF admni timeout setting is 30 sec.  We try to bump that up for pages that we expect to take more time and cache them since the queries do take a bit of time to process.  I believe the timeout is adequate for over 90% of the site otherwise.  Does CFSTAT get cleared ever X days, hours, minutes, or app restart?  The numbers may represent our previous problem with the overabundance of cfhttp requests which was corrected.

That was a JDK install for u24.  As for JConsole, I'm more comfortable restricting it to local box vs remote access.  Thanks for the tip to set a -XX:PermSize value.  I'll be sure to add that.

[Charlie Arehart]

@d@yz,

There’s too much going on for me to be positive, but from what you’ve said I would suspect that things broke for you (if you really did not apply any hotfixes) when you ran the updater. It asks where the CFIDE is, which is to be updated. You could have accepted the default, or changed it to some value. Do you recall?

More important, do you know what CFIDE is pointed to for the web site in IIS that you use to view the CFAdmin? Also, you refer to the jrun.xml and Adobe wondering about the “allow local web server to run”, that’s referring to the internal web server for CF (such as using port 8500, by default). Have you (or they) tried to access the CF Admin that way? And if so, what directory does that live in?

Here’s the thing: people sometimes have multiple different copies, and if they’re not careful when applying updaters (or hotfixes), they can apply one or the other to the wrong one. Again, you may then see a difference depending on what directory you updated and what directory is used for the CFIDE for the web server you’re using, and what web server you’re using to view the Admin.

Resolving things can be difficult because there are so many moving parts, and different people may have had a hand in updating things on your end. Even if you did it yourself, you may not accurately remember exactly what you did. Worse, there’s no easy way to assess what a “correct” CFIDE should look like for your current level of configuration, since different updates will change different files in the directory.

And then when you apply hotfixes, there are often other directories you are told to update (web-inf/lib and more), which can also then be troublesome if you update the wrong one (easy to do when you have multiple instances). Finally, another gotcha I see often is that when people extract the zips offered in the hofixes, they may extract then mistakenly to the wrong level in the destination directory. (But you said you have not applied hotfixes, so I realize this last paragraph doesn’t apply to you.)

Finally, I’ll point out for you and other readers that I blogged about this issue with some different details:

http://www.carehart.org/blog/client/index.cfm/2011/10/21/why_chfs_may_break

Hope some of that helps.

/charlie

[Guest]

@Charlie I wasn't present when the update was applied, but I was sure to ask the server admin when we unable to login to the CF web admin if the update was directed to the correct drive.  I was informed that it was and I also saw the update installation log some time later to verify that all the paths looked correct.  I do know that the server admin installs applications a non-OS drive, lets say X: just for discussion consistancy.

Before the Adobe tech assisted we only had one CFIDE directory available and it is nested under the root of the site.  The Adobe tech had me copy the then crippled CFIDE directory to the X:\Coldfusion9\wwwroot\CFIDE\ and test once it was enabled on port 8500, but it too failed.  I believe after it was working again he had me copy the CFIDE from \wwwroot\  to the site root, but I can't remember for sure.

I looked over your blog on why chfs may break before calling Adobe to see if we could apply any of your wisdom to our situation to recover, but nothing really looked as thought it applied.  Since the CF admin wasn't available I couldn't verify if CF was properly mapped to the CFIDE directory in the root of the site.  I expected it should have since there was no other CFIDE directory in existence.  At this time after the Adobe tech had me add another CFIDE directory in the X:\Coldfusion9\wwwroot\ path I see now that CF admin shows in mapping that it finds CFIDE in X:\Coldfusion9\wwwroot\CFIDE\

At this time we have one security hotfix that was applied to correct our broken CF web admin portal as suggested to add by the Adobe tech during the call.

I have only one suspicion as to what could have been the cause.  This is only speculation and I have no proof and haven't uncovered anything to support it either.  The migration of our site from CF8 to CF9 went as follows:

• Install CF9 64 bit to new vanilla server 2008 R2 box
• Physical to Virtual copy of existing CF8 32 bit Server 2003 instance.
• Change Virtual CF8 licensing to Developer
• Export CFIDE settings .car file from Virtual CF8
• Hardened CF9 CF admin settings per recommendations from Pete Freitag's PDF article and a few other suggestions.
• Migration of files from Physical CF8 box to CF9 box.
• Ran code analyzer against existing cfm and cfcs and corrected any errors.
• Imported CF8 exported .car file
• Applied  Java JDK u24 update to address vulnerability.
• Update to 9.0.1

At that time the CF admin portal was noted broken.  My suspicion is that the .car file had paths or evidence of hotfixes perhaps applied to the old CF8 box that were not present on the CF9 box.  Is that possible?

[Charlie Arehart]

No, the CAR would not have had any connection with the previous hotfixes. I’ll just say some step had to have been a mistake along the way. No way to tell now, this far removed.

I’ll just caution everyone to be VERY careful applying updates, with respect to what CFIDEs you have (that you may not even think of), and which you do update. And I’ll suggest (as before) that if things break, there is almost always going to be an explanation in one of the steps followed, for the reasons I outline.

Sorry that I can’t offer more for your specific challenge, though. Glad you’re up and working again.

/charlie

[Guest]

@Charlie

Up and working yes, but with a severe limp.  The server functions rather well most of the time.  Then there are instances when the server is producing error 503 pages.

One such error I knew of the exact minute the server was noted to be down.  I got to searching logs for the timestamp.  I found these:

05/17 10:39:41 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@3a57f10d]) JRunPRoxyServer.invokeRunnable:

java.lang.NullPointerException

    at jrun.servlet.ServletEngineService$DispatchKey.<init>(ServletEngineService.java:457)

    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:539)

    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)

    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)

    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

05/17 10:41:32 info dispatch: host is null

05/17 10:41:32 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@3a57f10d]) JRunPRoxyServer.invokeRunnable:

java.lang.NullPointerException

    at jrun.servlet.ServletEngineService$DispatchKey.<init>(ServletEngineService.java:457)

    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:539)

    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)

    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)

    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

05/17 10:41:32 info dispatch: host is null

05/17 10:41:32 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@3a57f10d]) JRunPRoxyServer.invokeRunnable:

java.lang.NullPointerException

    at jrun.servlet.ServletEngineService$DispatchKey.<init>(ServletEngineService.java:457)

    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:539)

    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)

    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)

    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

05/17 10:41:32 info dispatch: host is null

05/17 10:41:32 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@3a57f10d]) JRunPRoxyServer.invokeRunnable:

java.lang.NullPointerException

    at jrun.servlet.ServletEngineService$DispatchKey.<init>(ServletEngineService.java:457)

    at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:539)

    at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

    at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)

    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)

    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

Coincidently there have been 42109 recorded instances of java.lang.NullPointerException found in the logs since the server was brought online about a year ago.  It was only in March I believe that the outside world firewall had rules applied to make the server available to the public.  The first noted error was innocently from the scheduler:

"Error","scheduler-1","03/26/12","11:10:41",,""

java.lang.NullPointerException

I did some Grep analysis and consolidated the logs to produce number of instances per unique day:

       1  03/26 java.lang.NullPointerException

   19974  04/18 java.lang.NullPointerException

    1384  04/19 java.lang.NullPointerException

       2  04/20 java.lang.NullPointerException

      10  04/22 java.lang.NullPointerException

       2  04/23 java.lang.NullPointerException

       2  04/26 java.lang.NullPointerException

       2  04/30 java.lang.NullPointerException

       2  05/03 java.lang.NullPointerException

       2  05/04 java.lang.NullPointerException

       2  05/07 java.lang.NullPointerException

       2  05/09 java.lang.NullPointerException

       6  05/10 java.lang.NullPointerException

       4  05/13 java.lang.NullPointerException

       6  05/15 java.lang.NullPointerException

       8  05/17 java.lang.NullPointerException

The 04/18 instance was the weekend after the 4/11 fix was applied.  That was the first time the server was reported offline in the 503 error page fashion.

I searched up that error and found something very similar to symptoms of this.  http://forums.adobe.com/thread/776671  I searched for any application.cfm or application.cfc that contains mention of clientmanagement.  I found that our root site has an application.cfm, but I don't believe the syntax is correct.  This looks wrong to me; note that I'm not the author of this:

<cf_inputFilter

scopes = "URL,COOKIE,FORM"

chars = "<,>,',|,\,?,;,%,+" tags="ALL" clientmanagement="No" setclientcookies="No">

I just noticed now after I pasted that there are what appear to be tabs and not spaces present there.  One right after <cf_inputFilter and one right after FORM".  I wonder if that's also causing some syntax issues.

I believe it should look like:

<cfapplication clientManagement="No" setClientCookies="No">

<cf_inputFilter scopes = "URL,COOKIE,FORM" chars = "<,>,',|,\,?,;,%,+" tags="ALL">

Yes, I know the cf_inputFilter is very old and it needs to be replaced.

I'm really curious what the default value of clientManagement and setClientCookies are without fixing the code so I took a look and found this:

WriteDump(application);

produced:

Variable APPLICATON is undefined.

Ok, so it appears the application.cfm is not valid.  I don't know how to get the setting for clientManagement output so dumped:

WriteDump(client);

and got

struct [empty]

So I'm not sure how to verify if clientManagement was being defaulted to "No" or not with the syntactically incorrect application.cfm of doom standing its ground.  After I applied the fix suggested above I got this for a dump of application settings:

struct
clientManagement	NO
clientstorage	undefined
datasource	undefined
loginstorage	cookie
name	[empty string]
scriptprotect	undefined
serverSideFormValidation	YES
sessionmanagement	NO
setclientcookies	NO
setdomaincookies	NO

So now at least I know clientManagement is disabled with certainty.  I don't know if that concludes this particular issue, but I've been at this almost all day and must be back at this in 6 or so hours.  My "today" goal is to enable JConsole for getting an idea of whats going on under the hood for resources.

[Guest]

I've discovered there is a difference in jrun_iis6_wildcard.dll running on the production machine and the version that was found on a development workstation.  The dev workstation version was extracted from wsconfig.jar with 7zip to expose its contents.

The version on the left was the extracted one from wsconfig.jar.  The one on the right is the production machine version.  Both are 9.0.1 where 2 major differences are my dev workstation lacks an IIS instance and the production machine had a SHF applied as mentioned above.  This is the differences discovered.

Also, after realizing how susceptable the server was handling floods of bot traffic I enabled a 3 requests per second from an IP address to throttle down on resources handed out to inorganics.  I'm adding to that to block traffic from known offenders in the future, but for now its helping.

[carl type3]

I had a quick look at the details of jrun_iis6_wildcard.dll on 3 systems running CF9. The 32 bit CF Developer, 64 bit CF Standard and 64 bit Enterprise multiserver each had different jrun_iis6_wildcard.dll details. Not sure if any of that is helpful. Let me know if want more information.

[Guest]

Late afternoon yesterday I applied some critical exception handling for errors that I'd noticed showing up in the logs.

Message     Invalid data '' for CFSQLTYPE CF_SQL_INTEGER.

Type     coldfusion.runtime.CfErrorWrapper

Most of those errors were generated by bots.  Ever since the fix the server instability seems to have tamed.  With monitoring from Fusion Reactor I noticed that there were periods of time where http requests just stopped arriving in ColdFusions realm.  While that may not have been the case, that is what the graph revealed.  The same went for JDBC requests.  Visually it looked like ColdFusion was either starved from the flow of requests or itself was locked up its own world dealing with the surge of unhandled exceptions.

This is what the graph looked like when a few of these errors were noted:

JDBC

Http requests

After handling the exceptions these flatlines stopped occuring.  This led me to believe that perhaps the jrun_iis6_wildcard.dll was perhaps still involved in this.  Does it have any play in JDBC handling of requests?

Also, I installed the exact same SHF to my development workstation and again compared version info on the jrun_iis6_wildcard.dll and found that the differences are still exactly the same.  I believe I failed to mention that although I'm comparing 64bit to 64bit the platforms are not the same.  Win7 64 compared to Server 2008 R2 64. Should I be alarmed about that?  I assume its not advised to swap out the .dll and see how it behaves.

[Charlie Arehart]

No, the IIS connector has nothing to do with JDBC. What I suspect you’re seeing is that (for whatever reason) requests stop coming in (so no JDBC activity happens). Certainly if IIS stops letting requests in (which could happen due to problems with the IIS connector), then you’d see what you see. I’ll suggest you also look at the Windows Event Viewer to see if there are any errors there, especially with respect to the IIS Application Pool(s) you may have setup for use by the site(s) connecting to CF.

So as for the differences you’re observing, are you doing any manual tweaks to either file contents or file version with respect to the web server connectors for CF? And either way, if you think things are somehow not “right”, have you tried just removing and re-adding the web server configuration (using the CF web server config tool)? Sometimes, that’s all you need to do to sort things out. Sometimes, also, it can help to remove and replace either the IIS app pool or IIS site (or reinstall IIS itself).

No, I’m not saying you “should have to do this”, nor that it’s right (if it fixes things) that you should have had to do it. I’m just proposing some things to consider, since on the forums here we have so little to go on and only brief moments to communicate thoughts.

I’ll just say that if you continue to struggle with things not feeling quite right, I (and others) can be available to help for paid assistance (done remotely, without providing “remote access”, looking “over your shoulder” using Adobe Connect or the like), with no minimum time requirement, and (for me at least) with a satisfaction guarantee that you don’t have to pay for time you don’t feel is valuable.

Finally, I’m not at all surprised to hear that spiders and bots (and error handling) are at root of your problems. I find that when I help people they are indeed the root cause of many problems, though the problems may present themselves in many different ways, which is why it’s not often obvious how to connect the dots (in email discussions like this). It often becomes far more apparent in a shared desktop session where I’m looking at things with someone. Your call though. Just pointing out an option.

Also, as in the case of the past 12 days, sometimes I’m just unavailable on the forums due to travel, or sometimes just for being busy helping folks, so a direct engagement is also a way to get help sooner, as well as more directly. But as you can see, I do try to help here however we may, so this suggested alternative for help is not meant as an ultimatum, nor a carrot/stick. Just an option to consider.

/charlie arehart

charlie@carehart.org

Providing fast, remote, on-demand troubleshooting services for CF (and CFBuilder)

More at http://www.carehart.org/consulting

See also http://www.cf911.com for more on CF troubleshooting resources

[Guest]

@Charlie

I do appreciate your wisdom and realize you are a busy man with plenty to do in the now and on the upcoming horizon of things to do.  I understand you have a business to run and free support isn't really an option so your replies to my forum posts is a blessing.  That being said I try as hard as I can to be efficient at self-repair whenever possible due to varios factors involved when requesting assistance from external help.  Some of those factors include:

• $ availability
• Self-repair is doable most times.
• Outsourcing help sometimes introduces more problems or security risks.
• Employer thinks the cause is the new guy so better fix it and prove them wrong.

Emphasis on the last point.  The vibe I'm getting is that the only thing that changed is the person that was added to the web office so it must be his fault.  It's a matter of pride and prooving self == not guilty.

Yes, there are EventViewer logs.  Typically this is almost the same for every crash for output.  However, there are cases where events are in the Application log and not in the System log.

Application log:

3:24:36

The description for Event ID 1 from source DbgHost cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

C:\LogPath\w3wp__DefaultAppPool__PID__6344__Date__06_03_2012__Time_03_24_35AM__427__Second_Chance_Exception_C0000005.dmp

3:25:24

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

Faulting module name: ntdll.dll, version: 6.1.7601.17725, time stamp: 0x4ec4aa8e

Exception code: 0xc0000005

Fault offset: 0x0000000000018e3d

Faulting process id: 0x15d0

Faulting application start time: 0x01cd422c79dd5343

Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

Faulting module path: C:\Windows\SYSTEM32\ntdll.dll

Report Id: b8627b5c-ae1f-11e1-8265-842b2b0c92c3

Followed by 2 Windows error reporting events:

Fault bucket , type 0

Event Name: APPCRASH

Response: Not available

Cab Id: 0

Problem signature:

P1: w3wp.exe

P2: 7.5.7601.17514

P3: 4ce7afa2

P4: ntdll.dll

P5: 6.1.7601.17725

P6: 4ec4aa8e

P7: c0000005

P8: 0000000000018e3d

P9:

P10:

Attached files:

These files may be available here:

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_d922589e287529aea3e1d5d65bc492fbc1523e_2b37a82a

Analysis symbol:

Rechecking for solution: 0

Report Id: aa516698-ad55-11e1-8265-842b2b0c92c3

Report Status: 4

Crash #2

3:25:25

Crash #3

3:25:26

Crash #4

3:25:30

System log:

3:24:36

The description for Event ID 5011 from source Microsoft-Windows-WAS cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

DefaultAppPool

6344

3:25:25 same message with exception of number at bottom under DefaultAppPool

3:25:26 same message with exception of number at bottom DefaultAppPool

3:25:24

The Windows Error Reporting Service service entered the running state.

3:25:26 similar entry as before, with exception of different Event ID and different number at bottom under DefaultAppPool

3:25:30 similar entry as before, with exception of different Event ID and different number at bottom under DefaultAppPool

3:25:30 similar entry as before, with 2 exceptions: 1) Listed as error and not warning 2)different Event ID and different number at bottom under DefaultAppPool

The problem is not at the moment an emergency so I believe handling it on a self-repair level is adequate.  I'm not an IIS guru and only recently was granted access to the web server remote login to parse the logs and review settings of IIS.  May I ask what is the procedure for resetting the IIS connector?

There have been no manual changes to any files other than the visit from the Adobe tech on a support call.  I have suspected the connector could be re-installed, but haven't suggested it because of lack of process and knowledge with IIS.  If that doesn't resolve crashes then IIS app pools are also on the list for re-configure and/or reinstall.

You're at the top of my list of help for hire should I fail in correcting this web server's problem.  Thanks thusfar.

The war on bots isn't anything new.  It was just never addressed or even a focus point until I showed up in the office.  I'm seriously looking into whitelisting, but need some time to verify it won't  block desired traffic.  I'm amassing some tools for load testing, UA spoofing, and modifying some existing python tools to conduct testing.  I'm also implementing Porticullis to cut down on XSS attacks and related traffic.  I've been looking at the full blown ESAPI project for decoding URLs among other things, but realize its not ready for the circus quite yet despite Adobe's inclusion in the SHF.  Lastly, I'm looking to block bots that violate the 3 http requests per second and generate tons of 404 traffic because they typically coincide with times the web server is behaving sickly.

[Guest · Accepted solution]

An update to the progress of this problem.  It looks like the jrun_iis6_wildcard.dll version changed when I ran the WSConfig.exe utility.  I coudn't find an official document on how to re-install the jrun connector.  I only found documentation on how to initially install it.  I do realize that IIS 6 Metabase Compatability is not required to be disabled for CF 9.0.1, but I saw no reason not to remove it so it was removed as part of the Jrun connector re-install process.  I decided to follow this process to conduct the work since it contained both the missing uninstall as well as the install steps.  http://www.codecurry.com/2010/07/installing-coldfusion-901-update.html

The resulting differences in the jrun_iis6_wildcard.dll comparing prior to jrun connector installation and after can be seen here:

I'm really hoping that this will stop the intermittent outages and undesirable IIS application pools stops that have been frequently occuring.  If I don't encounter any more problems I'll report back in a week with a claim that the issue is resolved, otherwise I'll report back with any related troubles.

[Guest]

I know I said I would reply in a week.  Having an early appearing newborn kinda took me by surprise   He's in good health regardless.

I just wanted to post that I feel the problem and behavior that led me to make this post has been resolved to the best of my knowledge.  I'm no longer spending my day trying to follow for signs of trends and taking corrective actions based on downtime.  The result is less stress and more productivity.  I was also able to get a license purchase approval for Fusion Reactor so that was good.  It's a welcomed tool for maintaining the health and performance of the CF web server.

Thanks to Charlie and Carl for their time assisting in this forum with guidance, tips, and potential solutions.

[Charlie Arehart]

Congrats on both count, and thanks for the kind regards.

/charlie