We are seeing a problem in an online registration application and I am looking for any insight I can get to address it.
The relevant system specs are as follows:
Two Win2003 servers, each with 2GB memory and each running CF 188.8.131.52559 Standard edition. Sitting in front of them is an F5 BigIP load balancer. Once directed to a server, a user stays on that server, via 'sticky sessions' in the F5 and session management by CF. CF on each server is configured for 8 simultaneous requests and uses Java 1.4.2_09 with max memory setting of 512MB.
CF on each server talks to a single SQL 2000 database on a separate box. I don't have the memory specs for that box, but I know that it's a beast. SQL is configured for unlimited connections.
The problem is this: The registration application in question opens at a given time of day and is immediately hit with a huge number of attempted registrations. Only 4500 registrations can be accepted before registration must close. The last time this application was made available, more than 3000 registrations were booked in the first hour, and the entire application closed in slightly more than 2 hours.
Each potential registrant must check whether registration is closed before he/she can start to register. Each does this by initially querying the database for a count of already submitted registrations. If that count is less than a predefined maximum value (4500, held in an application variable), we set a session variable to '1' for that registrant. The session variable allows the registrant to proceed to the registration form, fill it out and submit it. There is no additional checking just before the registration gets submitted -- if you're good to go when you start, you get to register even though the count might be above 4500 by the time you submit. (We usually end up only slightly over 4500.)
The problem is this: display response time for the application is terrible. Almost immediately upon opening, it begins to take a great deal of time to display the registration form. What's worse, certain select lists on that form, which are built via <cfselect> using application variables (examples: country, state) sometimes do not get correctly displayed.
Clearly CF is overworked, and I need to know where the bottleneck is. Are the individual session variables quickly maxing out memory? (Session-variable timeout is set at 15 min.) If so, would adding more memory per server and/or more servers, and/or increasing the JVM memory setting, help? Is CF queueing up a huge number of requests to SQL and is that what's slowing everything down? If so, will increasing the max number of simultaneous requests beyond 8 help -- and if so, how high can I go? (I've read elsewhere that setting this number is somewhat of a black art, dependent not only on total server memory but also on JVM memory and on the nature of the application.)
Any suggestions will be greatly appreciated.