-
1. Re: CF 9.0 Schedule Tasks didn't execute.
So you're saying that the schedule.log indicates that it DID run, but you're relying on some other evidence that suggests they did not? What is that other evidence?
Also, are you using the feature in the scheduled task to have it log the output of running the request? What did that say? And have you viewed your web server logs to confirm whether the request ran (at the time the log says it did), and if so, can you see the status code for the request (an option that can be enabled in the logging for most web servers).
Finally, if the above don't help, what happens if you set up a CFML page to do a CHFTTP of the same request? If you view the cfhttp.filecontent that is returned as a result, what does it say? It's a useful diagnostic, because scheduled tasks run using the same underlying mechanism as CFHTTP (which can be different than your running the request yourself in a browser).
/charlie arehart
Providing CF and CFBuilder troubleshooting services
-
2. Re: CF 9.0 Schedule Tasks didn't execute.
carehart wrote:
So you're saying that the schedule.log indicates that it DID run, but you're relying on some other evidence that suggests they did not? What is that other evidence?
No, I was saying that the schedule log showed the LAST time they all ran, they all ran up to Tuesday (Oct 26) Morning at 5:00am. All the scheduled tasks ran before then and properly set the next instance for each task. But these next instances (the first being scheduled for 6:30pm) never happened. No entries in the logs, no exceptions or errors just silence. None of the evening tasks that executed on Monday and set up to run Tuesday evening nor the morning tasks that ran Tuesday morning and set up to run Wednesday morning happened.
We restarted the services at noon yesterday and all the tasks showed up in the log "Activating" and they did execute last night and this morning. But for some reason after the restart yesterday at noon, all the web server connectors (or something about that level) stopped working about 1/2 hour later. The ColdFusion administrators worked just fine, but all CFM web pages requested via the IIS web server returned blank pages, again no errors in any log I thought to look at. Just a very long spinning in the browser then a blank page.
We restarted the services again about 1pm and so far no problems. But it is uncomfortable not having any idea why things seem to be fundemntally failing, but having no idea or hint as to what is causing problems. Let alone some idea how to mitigate the problems.
-
3. Re: CF 9.0 Schedule Tasks didn't execute.
So you're saying it was not just scheduled task but indeed any requests to the CF server (via IIS) that failed, whereas your requests to the CF Admin worked. I'm curious: does the URL by which you were accessing the Admin use a port (like 8500 or 8300 or something like that)? If so, then it sounds like your problem was in the connector between IIS and CF.
In that you're wanting to better understand what happened and why, you may want to check out any of several diagnostic resources which may offer more insight:
- the windows event log (which shows info related to any IIS errors. You may have had an app pool failure, for instance.)
- the CF runtime/jrun logs (\runtime]logs or \logs) (which may be voluminous but often do have insight into what's amiss when CF seems unresponsive).
You've not indicated (I don't think) if this is CF Standard or Enterprise. If the latter, you could also have fired up the CF Server Monitor where, if you enabled "Start monitoring", it would show (at the time of the error) what requests (if any) were running. If the former, you could use the commercial tools FusionReactor or SeeFusion, which also can show what requests are running (and FusionReactor logs each request by default).
Your scenario makes it seem that perhaps no requests would have been running at the time of your error, but it could also have been that all threads were running for a long time, thus blocking new requests. Using any of these tools can give you more clear insight, helping to remove the "discomfort" you describe at feeling like it's just a black box not giving you clarity on what's going on.
You're not alone in that feeling, and its these logs and tools that can help break that. I help people every day in discovering and leveraging such tools to help resolve these very kind of problems, so it really does work, when you know where to look and how to connect the dots (or can leverage the help of others who can). So that this doesn't seem just a sales pitch for my own services, I'll note that I provide a list of such troubleshooting consultants on a page I keep: http://www.cf411.com/#cfassist.
Hope the info above helps.
/charlie arehart
Providing CF and CFBuilder troubleshooting services
-
4. Re: CF 9.0 Schedule Tasks didn't execute.
carehart wrote:
So you're saying it was not just scheduled task but indeed any requests to the CF server (via IIS) that failed,
At different times.
First the schedule tasks stopped responding, but normal user requests where working fine for two days. Then at noon yesterday we restarted the ColdFusion services to kick start the schedule tasks. That seemed to work as all the tasks showed "Activating" entries in the log.
BUT shortly after that something -- likely in the connectors -- started going wrong and all CFM requests where returning blank output. But not right away. All tests done immediately after the restarts responded correctly. No schedule tasks occurred during the window before we again restarted the ColdFusion services. After which all the tasks again showed "Activating" entries in the log. The ColdFusion applications have been responding normally since that second restart. All the schedule tasks of last evening and this morning also worked correctly.
But we have no idea why there where problems in the first place. We are on ColdFusion 9.0 (no updater yet) Enterpise, Multi-Home running on a Windows 2003 server. Yes the ColdFusion administrators are accessed with the built in web-server using port numbers like 8301, 8302, etc.
Something I have forgotten to mention is that the ColdFusion services do not stop properly on this box. When you go to the Windows Services panal and tell a CF Server to stop, windows reports that it is stopped nearly instantly. Not the minute or three I am used to seeing on windows services stops for ColdFusion servers. And we can't restart them do to an error saying that a port is 'already in use', the port being the jConsole port we have configured in our JVM.config files.
Using Process Explorer, we can see that the 'JRun' processes are still running. The 'JrunSVC' process that is the normal parent of the Jrun process is gone, but the JRun processes are still there. Killing them will then allow Windows to start new ColdFusion servers.
Thanks for you insights.
Ian
-
5. Re: CF 9.0 Schedule Tasks didn't execute.
Ian, I'm afraid I can't offer any more on the scheduled tasks other than to say to dig into those jrun logs like I mentioned earlier, or using tools to monitor what requests are/are not running at the time of the problem (if indeed it’s more than just the scheduled tasks that can't run at the time). I appreciate that it's a frustrating problem and you'd like to know more. I just don't sense there's much more that anyone can offer without more diagnostics.
But as for the instance not being able to stop/start as expected, that is indeed caused by your having enabled the RMI args in the JVM.config (which you'll have needed to do in order to use jconsole as you mentioned). If you don't really need that often, you may find it wise to avoid that problem (of shutting down) by commenting out those settings until you really do need to enable jconsole or similar tools.
(Really, since you have Enterprise, you can get most of what you likely need from the CF Server Monitor. The jvm memory graph, for instance, does NOT require enabling any of the "start" buttons at the top of the monitor--"start monitoring", "start profiling", nor even "start memory tracking".)
Finally, I notice that you referred to your CF configuration as "multi-home". Technically, it's "Multiserver" (as opposed to the other options offered at installation of CF, which are Server and EAR/WAR.) The phrase multi-home instead applies (typically) to web server configuration, where you have multiple web sites responding on a given web server. You could of course be doing that as well, but just to be clear (in case it may help you or others), when you refer to the CF config itself, you'd want to use the term Multiserver (though most people don't even use that, and they call it more generically "multiple instances".)
Hope that's helpful.
/charlie
-
6. Re: CF 9.0 Schedule Tasks didn't execute.
Hi,
In relation to Jconsol I find Jconsol RMI settings in JVM.CONFIG do not allow the ColdFusion Admin service to stop. Using Jconsol sometimes is handy to see how your memory buffers "heap","non heap" and the cost of collections are going on the JVM. Jconsol probably has very little overhead to run however if you need to stop the service you are going to have to kill the JRun.EXE task.
HTH, Carl.