I have an application running on ColdFusion 8 standard (single instance of jrun)/Windows 2008/IIS that has begun to show some really strange errors. They began when a distributed denial of service (DDoS) attack was launched against one of the sites that runs the application. The load was putting about 40 requests per second on the server. The latency of my application and that load would have easily meant ColdFusion not being able to service the requests based on the hardware and tuning I had in place. I extected to see errors about timeouts or memory consumption, but that is not what happened.
Instead, at first, my logs were filled with a ton of errors about a custom tag not being found. These errors were not just coming from the site being attacked, but from every site that uses the application (every site on the server uses the application). A unique use case requied the application to be hosted on the Apache web server, which is also running on the same machine, linked to the same instance of ColdFusion. Applications running under Apache saw the same errors. So I blocked traffic at the firewall and immediately the site returned to normal working order.
The application uses custom tag mappings in the Application.cfc file, which is important because there are several versions of the application with different custom tags, though those tags are named the same. Fortunately, my hosting provider was able to set up a dedicated server for me by the end of the day and I transferred the site being attacked to that server. At the same time I set the default custom tag path in the CF administrator to use the same location of this applications custom tag path. Upon doing that and reopening the firewall, the custom tag errors went away while under load. Unfortunately that led to other errors about variables in my application scope.
In the Application.cfc file, I parse an XML file during onApplicationStart that I store in the application scope. At the beginning of onRequestStart, I copy that application variable while in a readonly cflock into the request scope. I use duplicate() to assure that nothing happens to the original during the request. Frustratingly, once the custom tag error was no longer occurring, a new CF error sprung up on the duplicate() call, saying that the XML content had an illegal character within it. Close the firewall and suddenly it works fine, reopen it and the error continues.
My provider has tuned ColdFusion for me as usual and has it working as well as the entire server very well. This application has run very well as well up till this point.
So, question is, what does heavy load on a server do to ColdFusion that makes one technique - using the cf administrator custom tag paths - better than another - using application specific custom tag paths. What would load do to variables cached in the application scope, or other scopes for that matter? The answer, of course, is supposed to be nothing; but that is not what is happening.
If anyone has heard or dealt with similar issues before, or has any recommendations at all, your support would be greatly appreciated.