Accidental Session Leakage under load

[Girl_DIP]

The problem appears to only manifest itself when traffic peaks. Without giving figures for commercial reasons but we are talking significant amounts. To give you an idea, we are running a load-balanced cluster of 4 windows server at the moment and will have to upsize that shortly. We are not sharing sessions across the cluster and are using WLBS NLB Windows Load Balancing in single-instance mode with sticky sessions.

The ID is the jsessionid, generated by CF.

We have already locked-down all session variable access.

As I see it there are only a few possibilities remaining:

1) The load-balancing is not keeping a user on one server and the sessionid is not unique across servers.

2) IIS is pushing cached content to the user (we put a cfheader fix for this possibility into application.cfm but it did not work!)

3) CF is leaking under stress


If you can think of any other possibilities then please let me know. All ideas appreciated.

I am coming to the conclusion that the only way I can overcome this is to roll our own session solution using DB access and GUID/Encrypted GUID tokens that correspond to values in the user's record. Basically, double-checking every request and, if there is a discrepancy, pushing the user to the login screen and destroying the session.

All thoughts more than welcome

[Newsgroup_User]

You haven't REALLY described what it is that's actually occuring to make
you think there's a problem.

And what version of CF are you running?

--
Adam

[BKBK]

Adam said:
You haven't REALLY described what it is that's actually occuring to make
you think there's a problem.
A common mistake I, too, was once guilty of. Then I realized the troopers here usually don't read the topic's title or summary.

Girl DIP said:
double-checking every request and, if there is a discrepancy, pushing the user to the login screen and destroying the session.
Better to locate and solve the problem than to lock down the session some more. Your users will soon get the feeling they're being hemmed in while the fair is going on outside.

These problems are more often caused by code than by the underlying technology. I would therefore assume sticky sessions are working and keeping the user to one server. Check the login code. Also check how you attribute roles and profiles to users.

[Girl_DIP]

We are using CFMX 7 Enterprise
Windows 2003 Server Standard x64

We have noticed other people have had this same problem.
Any further thoughts?

[BKBK]

... possibilities remaining:

1) The load-balancing is not keeping a user on one server and the sessionid is not unique across servers.

What does a typical jsessionid look like when the problem occurs?

[Grizzly9279]

Our production environment currently consists of a cluster of web servers (in our DMZ), that have JRun connectors installed that point to a multi-server CF cluster in a more protected subnet in our network.

We use Windows NLB for web servers (w/ IP affinity), but use ColdFusion's built-in clustering for the app servers (w/ J2EE session IDs, and sticky sessions).

We've not had any problems, and we do handle a fairly significant, and growing load of concurrent users.

I'm a little confused on if, or how your environment differs than ours. Are you using NLB for your APP servers? If so, I find that a little odd. Are you not using CF-clustering at all? Also, where are your web servers? Is each of your app servers also playing the web server role as well?

Using NLB for your app servers seems like a flawed approach to me. Can you clarify one some off these specifics a little more? There's must be an obvious explanation in here somewhere - I just feel like we're all going to need more info before we can get anywhere on this.

[Newsgroup_User]

> We have noticed other people have had this same problem.
> Any further thoughts?

Yes. My thoughts (still) are that I think you should describe the
manifestation of the problem more fully. You've not really given us much
to go on.

Just saying "it's not working" is not much help.

- does it happen that when a new visitor arrives at the site, they get
someone else's session instead of a new one?
- how is a user profile stored, and how is it fetched? What constitutes "a
different user"? It sounds like there is a login/authentication process
going on. Is that retrieving the correct user profile?
- does it happen half-way through a session?
- is their previous / correct session still there as well, which suggests
their session ID is changing some how?
- It's easy enough to find out if the sessions are hopping servers. Are
they?
- it's also easy enough to demonstrate whether IIS is returning cached
content. Is it?
- are you doing anything "clever" with session variables anywhere which
might not be working quite the way you think it is?
- what happens if you switch off J2EE sessions and just use CF ones
(temporarily)?
- what load balancing mechanism are you using?
- does it only happen with sessions on one of those servers, or all of
them?
- Does CF SEEM to be under load? What CPU usage is it hitting "at peak
times"? What RAM usage?
- what JVM RAM settings have you got in your JVM configs?
- what do your logs have to say for themselves?

Not related to your problem, but:
- why are you using Windows / IIS on a busy site?

--
Adam

[Newsgroup_User]

More things to think about:
- does it happen with ANY user at peak times, or only a subset of users?
- if they latter, what's the difference between users that DO switch
profiles, and those that don't?
- do the sessions get set wrong as soon as session management is being
enabled (ie: as soon as the site's <cfapplication> tag gets hit each
request, the user has the wrong session, or does it go wrong half-way
through a request?)
- Are you using <cfapplication> or application variables in
Application.cfc?

--
Adam