Troubleshooting CF8 + CentOS5 lockups

Thank you for the reply.

Yes, I agree that the garbage collection limit seems to be the issue. I have a ton of these errors in the web1 server logs, however very few in the logs on the other four servers.

My JVM args are:

-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib

This is the same as the other four web nodes.  I currently have Min heap size set to NULL and Max heap size set to 512MB.  I have tried upping this to min 512 and max 2048 with no luck.  Again, the sefault min NULL and max 512 is the same as the other servers.

As far as enabling logging, a quich google search leads me to believe it's as simple as adding a few arguments to the jvm.config such as:

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps  -XX:+PrintHeapAtGC -verbose:gc -Xloggc:PROD_GC.log

If this is not correct, I would very much appreciate a little help on enabling logging and what to look for.

Thanks you again for the help,

-Tony

OK, after adding the debug to the jvm.config, i only had to wait a few minutes before it locked up again. I let it run for a few minutes before finally killing the CF process.

Here is the bottom of the output log (the beginning was only more of the same):

{Heap before GC invocations=447 (full 411):
PSYoungGen      total 102336K, used 58240K [0xe9820000, 0xf42c0000, 0xf42c0000)
  eden space 58240K, 100% used [0xe9820000,0xed100000,0xed100000)
  from space 44096K, 0% used [0xf17b0000,0xf17b0000,0xf42c0000)
  to   space 58240K, 0% used [0xed100000,0xed100000,0xf09e0000)
PSOldGen        total 349568K, used 349568K [0xd42c0000, 0xe9820000, 0xe9820000)
  object space 349568K, 100% used [0xd42c0000,0xe9820000,0xe9820000)
PSPermGen       total 49664K, used 49624K [0xc82c0000, 0xcb340000, 0xd42c0000)
  object space 49664K, 99% used [0xc82c0000,0xcb3363d8,0xcb340000)
1692.556: [Full GC [PSYoungGen: 58240K->58240K(102336K)] [PSOldGen: 349568K->349568K(349568K)] 407808K->407808K(451904K) [PSPermGen: 49624K->49624K(49664K)], 2.0786770 secs] [Times: user=2.08 sys=0.00, real=2.08 secs]
Heap after GC invocations=447 (full 411):
PSYoungGen      total 102336K, used 58240K [0xe9820000, 0xf42c0000, 0xf42c0000)
  eden space 58240K, 100% used [0xe9820000,0xed100000,0xed100000)
  from space 44096K, 0% used [0xf17b0000,0xf17b0000,0xf42c0000)
  to   space 58240K, 0% used [0xed100000,0xed100000,0xf09e0000)
PSOldGen        total 349568K, used 349568K [0xd42c0000, 0xe9820000, 0xe9820000)
  object space 349568K, 100% used [0xd42c0000,0xe9820000,0xe9820000)
PSPermGen       total 49664K, used 49624K [0xc82c0000, 0xcb340000, 0xd42c0000)
  object space 49664K, 99% used [0xc82c0000,0xcb3363d8,0xcb340000)
}
{Heap before GC invocations=448 (full 412):
PSYoungGen      total 102336K, used 58240K [0xe9820000, 0xf42c0000, 0xf42c0000)
  eden space 58240K, 100% used [0xe9820000,0xed100000,0xed100000)
  from space 44096K, 0% used [0xf17b0000,0xf17b0000,0xf42c0000)
  to   space 58240K, 0% used [0xed100000,0xed100000,0xf09e0000)
PSOldGen        total 349568K, used 349568K [0xd42c0000, 0xe9820000, 0xe9820000)
  object space 349568K, 100% used [0xd42c0000,0xe9820000,0xe9820000)
PSPermGen       total 49664K, used 49624K [0xc82c0000, 0xcb340000, 0xd42c0000)
  object space 49664K, 99% used [0xc82c0000,0xcb3363d8,0xcb340000)
1694.635: [Full GC [PSYoungGen: 58240K->58239K(102336K)] [PSOldGen: 349568K->349568K(349568K)] 407808K->407807K(451904K) [PSPermGen: 49624K->49624K(49664K)], 2.0870480 secs] [Times: user=2.08 sys=0.00, real=2.09 secs]
Heap after GC invocations=448 (full 412):
PSYoungGen      total 102336K, used 58239K [0xe9820000, 0xf42c0000, 0xf42c0000)
  eden space 58240K, 99% used [0xe9820000,0xed0ffc08,0xed100000)
  from space 44096K, 0% used [0xf17b0000,0xf17b0000,0xf42c0000)
  to   space 58240K, 0% used [0xed100000,0xed100000,0xf09e0000)
PSOldGen        total 349568K, used 349568K [0xd42c0000, 0xe9820000, 0xe9820000)
  object space 349568K, 100% used [0xd42c0000,0xe9820000,0xe9820000)
PSPermGen       total 49664K, used 49624K [0xc82c0000, 0xcb340000, 0xd42c0000)
  object space 49664K, 99% used [0xc82c0000,0xcb3363d8,0xcb340000)
}
{Heap before GC invocations=449 (full 413):
PSYoungGen      total 102336K, used 58240K [0xe9820000, 0xf42c0000, 0xf42c0000)
  eden space 58240K, 100% used [0xe9820000,0xed100000,0xed100000)
  from space 44096K, 0% used [0xf17b0000,0xf17b0000,0xf42c0000)
  to   space 58240K, 0% used [0xed100000,0xed100000,0xf09e0000)
PSOldGen        total 349568K, used 349568K [0xd42c0000, 0xe9820000, 0xe9820000)
  object space 349568K, 100% used [0xd42c0000,0xe9820000,0xe9820000)
PSPermGen       total 49664K, used 49624K [0xc82c0000, 0xcb340000, 0xd42c0000)
  object space 49664K, 99% used [0xc82c0000,0xcb3363d8,0xcb340000)
1696.723: [Full GC [PSYoungGen: 58240K->58240K(102336K)] [PSOldGen: 349568K->349568K(349568K)] 407808K->407808K(451904K) [PSPermGen: 49624K->49624K(49664K)], 2.0798610 secs] [Times: user=2.08 sys=0.00, real=2.08 secs]
Heap after GC invocations=449 (full 413):
PSYoungGen      total 102336K, used 58240K [0xe9820000, 0xf42c0000, 0xf42c0000)
  eden space 58240K, 100% used [0xe9820000,0xed100000,0xed100000)
  from space 44096K, 0% used [0xf17b0000,0xf17b0000,0xf42c0000)
  to   space 58240K, 0% used [0xed100000,0xed100000,0xf09e0000)
PSOldGen        total 349568K, used 349568K [0xd42c0000, 0xe9820000, 0xe9820000)
  object space 349568K, 100% used [0xd42c0000,0xe9820000,0xe9820000)
PSPermGen       total 49664K, used 49624K [0xc82c0000, 0xcb340000, 0xd42c0000)
  object space 49664K, 99% used [0xc82c0000,0xcb3363d8,0xcb340000)
}
{Heap before GC invocations=450 (full 414):
PSYoungGen      total 102336K, used 58240K [0xe9820000, 0xf42c0000, 0xf42c0000)
  eden space 58240K, 100% used [0xe9820000,0xed100000,0xed100000)
  from space 44096K, 0% used [0xf17b0000,0xf17b0000,0xf42c0000)
  to   space 58240K, 0% used [0xed100000,0xed100000,0xf09e0000)
PSOldGen        total 349568K, used 349568K [0xd42c0000, 0xe9820000, 0xe9820000)
  object space 349568K, 100% used [0xd42c0000,0xe9820000,0xe9820000)
PSPermGen       total 49664K, used 49624K [0xc82c0000, 0xcb340000, 0xd42c0000)
  object space 49664K, 99% used [0xc82c0000,0xcb3363d8,0xcb340000)
1698.804: [Full GCtail: PROD_GC.log: file truncated
{Heap before GC invocations=1 (full 0):
PSYoungGen      total 19136K, used 16448K [0xe97c0000, 0xead10000, 0xf4260000)
  eden space 16448K, 100% used [0xe97c0000,0xea7d0000,0xea7d0000)
  from space 2688K, 0% used [0xeaa70000,0xeaa70000,0xead10000)
  to   space 2688K, 0% used [0xea7d0000,0xea7d0000,0xeaa70000)
PSOldGen        total 43712K, used 0K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 0% used [0xd4260000,0xd4260000,0xd6d10000)
PSPermGen       total 16384K, used 6849K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 41% used [0xc8260000,0xc8910540,0xc9260000)
1.180: [GC [PSYoungGen: 16448K->1679K(19136K)] 16448K->1679K(62848K), 0.0082790 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
Heap after GC invocations=1 (full 0):
PSYoungGen      total 19136K, used 1679K [0xe97c0000, 0xead10000, 0xf4260000)
  eden space 16448K, 0% used [0xe97c0000,0xe97c0000,0xea7d0000)
  from space 2688K, 62% used [0xea7d0000,0xea973f48,0xeaa70000)
  to   space 2688K, 0% used [0xeaa70000,0xeaa70000,0xead10000)
PSOldGen        total 43712K, used 0K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 0% used [0xd4260000,0xd4260000,0xd6d10000)
PSPermGen       total 16384K, used 6849K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 41% used [0xc8260000,0xc8910540,0xc9260000)
}
{Heap before GC invocations=2 (full 1):
PSYoungGen      total 19136K, used 1679K [0xe97c0000, 0xead10000, 0xf4260000)
  eden space 16448K, 0% used [0xe97c0000,0xe97c0000,0xea7d0000)
  from space 2688K, 62% used [0xea7d0000,0xea973f48,0xeaa70000)
  to   space 2688K, 0% used [0xeaa70000,0xeaa70000,0xead10000)
PSOldGen        total 43712K, used 0K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 0% used [0xd4260000,0xd4260000,0xd6d10000)
PSPermGen       total 16384K, used 6849K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 41% used [0xc8260000,0xc8910540,0xc9260000)
1.189: [Full GC (System) [PSYoungGen: 1679K->0K(19136K)] [PSOldGen: 0K->1591K(43712K)] 1679K->1591K(62848K) [PSPermGen: 6849K->6849K(16384K)], 0.0554590 secs] [Times: user=0.06 sys=0.00, real=0.05 secs]
Heap after GC invocations=2 (full 1):
PSYoungGen      total 19136K, used 0K [0xe97c0000, 0xead10000, 0xf4260000)
  eden space 16448K, 0% used [0xe97c0000,0xe97c0000,0xea7d0000)
  from space 2688K, 0% used [0xea7d0000,0xea7d0000,0xeaa70000)
  to   space 2688K, 0% used [0xeaa70000,0xeaa70000,0xead10000)
PSOldGen        total 43712K, used 1591K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 3% used [0xd4260000,0xd43edd20,0xd6d10000)
PSPermGen       total 16384K, used 6849K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 41% used [0xc8260000,0xc8910540,0xc9260000)
}
{Heap before GC invocations=3 (full 1):
PSYoungGen      total 19136K, used 16448K [0xe97c0000, 0xead10000, 0xf4260000)
  eden space 16448K, 100% used [0xe97c0000,0xea7d0000,0xea7d0000)
  from space 2688K, 0% used [0xea7d0000,0xea7d0000,0xeaa70000)
  to   space 2688K, 0% used [0xeaa70000,0xeaa70000,0xead10000)
PSOldGen        total 43712K, used 1591K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 3% used [0xd4260000,0xd43edd20,0xd6d10000)
PSPermGen       total 16384K, used 11673K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 71% used [0xc8260000,0xc8dc6630,0xc9260000)
2.239: [GC [PSYoungGen: 16448K->1398K(19136K)] 18039K->2989K(62848K), 0.0088530 secs] [Times: user=0.05 sys=0.01, real=0.01 secs]
Heap after GC invocations=3 (full 1):
PSYoungGen      total 19136K, used 1398K [0xe97c0000, 0xead10000, 0xf4260000)
  eden space 16448K, 0% used [0xe97c0000,0xe97c0000,0xea7d0000)
  from space 2688K, 52% used [0xeaa70000,0xeabcd898,0xead10000)
  to   space 2688K, 0% used [0xea7d0000,0xea7d0000,0xeaa70000)
PSOldGen        total 43712K, used 1591K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 3% used [0xd4260000,0xd43edd20,0xd6d10000)
PSPermGen       total 16384K, used 11673K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 71% used [0xc8260000,0xc8dc6630,0xc9260000)
}
{Heap before GC invocations=4 (full 1):
PSYoungGen      total 19136K, used 17846K [0xe97c0000, 0xead10000, 0xf4260000)
  eden space 16448K, 100% used [0xe97c0000,0xea7d0000,0xea7d0000)
  from space 2688K, 52% used [0xeaa70000,0xeabcd898,0xead10000)
  to   space 2688K, 0% used [0xea7d0000,0xea7d0000,0xeaa70000)
PSOldGen        total 43712K, used 1591K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 3% used [0xd4260000,0xd43edd20,0xd6d10000)
PSPermGen       total 16384K, used 13056K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 79% used [0xc8260000,0xc8f203a8,0xc9260000)
2.946: [GC [PSYoungGen: 17846K->2685K(19136K)] 19437K->4440K(62848K), 0.0088250 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
Heap after GC invocations=4 (full 1):
PSYoungGen      total 19136K, used 2685K [0xe97c0000, 0xebd20000, 0xf4260000)
  eden space 16448K, 0% used [0xe97c0000,0xe97c0000,0xea7d0000)
  from space 2688K, 99% used [0xea7d0000,0xeaa6f470,0xeaa70000)
  to   space 2688K, 0% used [0xeba80000,0xeba80000,0xebd20000)
PSOldGen        total 43712K, used 1755K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 4% used [0xd4260000,0xd4416d20,0xd6d10000)
PSPermGen       total 16384K, used 13056K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 79% used [0xc8260000,0xc8f203a8,0xc9260000)
}
{Heap before GC invocations=5 (full 1):
PSYoungGen      total 19136K, used 19133K [0xe97c0000, 0xebd20000, 0xf4260000)
  eden space 16448K, 100% used [0xe97c0000,0xea7d0000,0xea7d0000)
  from space 2688K, 99% used [0xea7d0000,0xeaa6f470,0xeaa70000)
  to   space 2688K, 0% used [0xeba80000,0xeba80000,0xebd20000)
PSOldGen        total 43712K, used 1755K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 4% used [0xd4260000,0xd4416d20,0xd6d10000)
PSPermGen       total 16384K, used 15249K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 93% used [0xc8260000,0xc9144468,0xc9260000)
4.071: [GC [PSYoungGen: 19133K->2680K(35584K)] 20888K->6500K(79296K), 0.0161350 secs] [Times: user=0.06 sys=0.01, real=0.02 secs]
Heap after GC invocations=5 (full 1):
PSYoungGen      total 35584K, used 2680K [0xe97c0000, 0xebd20000, 0xf4260000)
  eden space 32896K, 0% used [0xe97c0000,0xe97c0000,0xeb7e0000)
  from space 2688K, 99% used [0xeba80000,0xebd1e160,0xebd20000)
  to   space 2688K, 0% used [0xeb7e0000,0xeb7e0000,0xeba80000)
PSOldGen        total 43712K, used 3819K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 8% used [0xd4260000,0xd461afa8,0xd6d10000)
PSPermGen       total 16384K, used 15249K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 93% used [0xc8260000,0xc9144468,0xc9260000)
}
{Heap before GC invocations=6 (full 1):
PSYoungGen      total 35584K, used 35576K [0xe97c0000, 0xebd20000, 0xf4260000)
  eden space 32896K, 100% used [0xe97c0000,0xeb7e0000,0xeb7e0000)
  from space 2688K, 99% used [0xeba80000,0xebd1e160,0xebd20000)
  to   space 2688K, 0% used [0xeb7e0000,0xeb7e0000,0xeba80000)
PSOldGen        total 43712K, used 3819K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 8% used [0xd4260000,0xd461afa8,0xd6d10000)
PSPermGen       total 16384K, used 16234K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 99% used [0xc8260000,0xc923aac0,0xc9260000)
4.853: [GC [PSYoungGen: 35576K->2686K(35584K)] 39396K->6708K(79296K), 0.0106890 secs] [Times: user=0.06 sys=0.01, real=0.02 secs]
Heap after GC invocations=6 (full 1):
PSYoungGen      total 35584K, used 2686K [0xe97c0000, 0xedd40000, 0xf4260000)
  eden space 32896K, 0% used [0xe97c0000,0xe97c0000,0xeb7e0000)
  from space 2688K, 99% used [0xeb7e0000,0xeba7f8f8,0xeba80000)
  to   space 2688K, 0% used [0xedaa0000,0xedaa0000,0xedd40000)
PSOldGen        total 43712K, used 4022K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 9% used [0xd4260000,0xd464da10,0xd6d10000)
PSPermGen       total 16384K, used 16234K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 99% used [0xc8260000,0xc923aac0,0xc9260000)
}
{Heap before GC invocations=7 (full 1):
PSYoungGen      total 35584K, used 35582K [0xe97c0000, 0xedd40000, 0xf4260000)
  eden space 32896K, 100% used [0xe97c0000,0xeb7e0000,0xeb7e0000)
  from space 2688K, 99% used [0xeb7e0000,0xeba7f8f8,0xeba80000)
  to   space 2688K, 0% used [0xedaa0000,0xedaa0000,0xedd40000)
PSOldGen        total 43712K, used 4022K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 9% used [0xd4260000,0xd464da10,0xd6d10000)
PSPermGen       total 16384K, used 16240K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 99% used [0xc8260000,0xc923c320,0xc9260000)
5.192: [GC [PSYoungGen: 35582K->2678K(66752K)] 39604K->6720K(110464K), 0.0075670 secs] [Times: user=0.04 sys=0.01, real=0.01 secs]
Heap after GC invocations=7 (full 1):
PSYoungGen      total 66752K, used 2678K [0xe97c0000, 0xee0a0000, 0xf4260000)
  eden space 64064K, 0% used [0xe97c0000,0xe97c0000,0xed650000)
  from space 2688K, 99% used [0xedaa0000,0xedd3d948,0xedd40000)
  to   space 4416K, 0% used [0xed650000,0xed650000,0xedaa0000)
PSOldGen        total 43712K, used 4042K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 9% used [0xd4260000,0xd4652a10,0xd6d10000)
PSPermGen       total 16384K, used 16240K [0xc8260000, 0xc9260000, 0xd4260000)
  object space 16384K, 99% used [0xc8260000,0xc923c320,0xc9260000)
}
{Heap before GC invocations=8 (full 1):
PSYoungGen      total 66752K, used 66742K [0xe97c0000, 0xee0a0000, 0xf4260000)
  eden space 64064K, 100% used [0xe97c0000,0xed650000,0xed650000)
  from space 2688K, 99% used [0xedaa0000,0xedd3d948,0xedd40000)
  to   space 4416K, 0% used [0xed650000,0xed650000,0xedaa0000)
PSOldGen        total 43712K, used 4042K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 9% used [0xd4260000,0xd4652a10,0xd6d10000)
PSPermGen       total 18304K, used 18263K [0xc8260000, 0xc9440000, 0xd4260000)
  object space 18304K, 99% used [0xc8260000,0xc9435e10,0xc9440000)
6.353: [GC [PSYoungGen: 66742K->3464K(68480K)] 70784K->7506K(112192K), 0.0369970 secs] [Times: user=0.26 sys=0.02, real=0.04 secs]
Heap after GC invocations=8 (full 1):
PSYoungGen      total 68480K, used 3464K [0xe97c0000, 0xf1dc0000, 0xf4260000)
  eden space 64064K, 0% used [0xe97c0000,0xe97c0000,0xed650000)
  from space 4416K, 78% used [0xed650000,0xed9b2110,0xedaa0000)
  to   space 4544K, 0% used [0xf1950000,0xf1950000,0xf1dc0000)
PSOldGen        total 43712K, used 4042K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 9% used [0xd4260000,0xd4652a10,0xd6d10000)
PSPermGen       total 18304K, used 18263K [0xc8260000, 0xc9440000, 0xd4260000)
  object space 18304K, 99% used [0xc8260000,0xc9435e10,0xc9440000)
}
{Heap before GC invocations=9 (full 1):
PSYoungGen      total 68480K, used 67528K [0xe97c0000, 0xf1dc0000, 0xf4260000)
  eden space 64064K, 100% used [0xe97c0000,0xed650000,0xed650000)
  from space 4416K, 78% used [0xed650000,0xed9b2110,0xedaa0000)
  to   space 4544K, 0% used [0xf1950000,0xf1950000,0xf1dc0000)
PSOldGen        total 43712K, used 4042K [0xd4260000, 0xd6d10000, 0xe97c0000)
  object space 43712K, 9% used [0xd4260000,0xd4652a10,0xd6d10000)
PSPermGen       total 19200K, used 19149K [0xc8260000, 0xc9520000, 0xd4260000)
  object space 19200K, 99% used [0xc8260000,0xc95134c0,0xc9520000)
9.165: [GC [PSYoungGen: 67528K->4009K(132352K)] 71570K->12937K(176064K), 0.0203050 secs]

Not sure what I should be looking for here.  Sadly I didnt offload the file before CF restarted and the log reset.  Ill remember to do that next time it hangs.

OK, got a couple more logs here, but I wont bother taking up the space posting them.  More of exactly the same.  In a half hour, the jvm performed 410 GCs, 8 of which were full GCs.  And that was while it was running cleanly with seemingly very few hits.  When CF is hung up, it seems like it may or may not perform more frequent GCs, but a lot more of them are full GCs.  I have since removed this node from the load balancer, and the GC's appear to stop.

To me, this appears to be a wide spread issue with essentially ALL aspects of our web application, not just one smoking gun.  We saw and issue quite similar to this when we had relocated our codebase to another location, but left the original location in teh apache confis, and using a symbolic link.  CF immediately threw a temper tantrum as it did not like to follow the sym link at all, especially in trying to find CFCs.

Sadly, we have made no recent changes to any codebase location, nor the apache config, and again, this seems to only affect this one node out of five.

Thank you for the info.  The presentation was quite informitive. I made the suggestion changes and after a few minutes, the server locked up.  It seems to be isolated to our "sourceone" application.  Here are the logs:

Application log:

"Error","jrpp-4","02/16/11","10:06:01","SO","Java heap space The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 39 "
"Error","jrpp-13","02/16/11","10:06:01","SO","Java heap space The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 214 "

Exception Log:

"Error","jrpp-4","02/16/11","10:06:01","SO","Java heap space The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 39 "
java.lang.OutOfMemoryError: Java heap space
        at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
        at java.lang.StringCoding.decode(Unknown Source)
        at java.lang.String.<init>(Unknown Source)
        at java.lang.String.<init>(Unknown Source)
        at java.io.UnixFileSystem.list(Native Method)
        at java.io.File.list(Unknown Source)
        at java.io.File.listFiles(Unknown Source)
        at coldfusion.compiler.CustomTagTable.scanTags(CustomTagTable.java:54)
        at coldfusion.compiler.CustomTagTable.scanTags(CustomTagTable.java:62)
        at coldfusion.compiler.CustomTagTable.scanTags(CustomTagTable.java:48)
        at coldfusion.compiler.CustomTagTable.<init>(CustomTagTable.java:34)
        at coldfusion.runtime.RuntimeServiceImpl.resolveTemplateName(RuntimeServiceImpl.java:593)
        at coldfusion.tagext.lang.ModuleTag.setName(ModuleTag.java:84)
        at coldfusion.tagext.lang.CustomTag.setName(CustomTag.java:34)
        at cfdsp_edit2ecfm45629791._factor5(/san/html/sourceone/admin/dsp_edit.cfm:39)
        at cfdsp_edit2ecfm45629791._factor14(/san/html/sourceone/admin/dsp_edit.cfm:37)
        at cfdsp_edit2ecfm45629791.runPage(/san/html/sourceone/admin/dsp_edit.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:483)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_Switch2ecfm1533486892._factor41(/san/html/sourceone/admin/fbx_Switch.cfm:346)
        at cffbx_Switch2ecfm1533486892._factor109(/san/html/sourceone/admin/fbx_Switch.cfm:340)
        at cffbx_Switch2ecfm1533486892.runPage(/san/html/sourceone/admin/fbx_Switch.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:483)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor4(/san/html/sourceone/fbx_fusebox30_CF50_n ix.cfm:242)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor5(/san/html/sourceone/fbx_fusebox30_CF50_n ix.cfm:1)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667.runPage(/san/html/sourceone/fbx_fusebox30_CF50_ni x.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:483)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
"Error","jrpp-13","02/16/11","10:06:01","SO","Java heap space The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 214 "
java.lang.OutOfMemoryError: Java heap space
        at coldfusion.runtime.CfJspPage.initialize(CfJspPage.java:115)
        at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:175)
        at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:422)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor0(/san/html/sourceone/fbx_fusebox30_CF50_n ix.cfm:214)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor4(/san/html/sourceone/fbx_fusebox30_CF50_n ix.cfm:199)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor5(/san/html/sourceone/fbx_fusebox30_CF50_n ix.cfm:1)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667.runPage(/san/html/sourceone/fbx_fusebox30_CF50_ni x.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:483)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1792679210.runPage(/san/html/sourceone/index.cfm:11)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:483)
        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:288)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:74)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.CfmServlet.service(CfmServlet.java:198)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 )
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at jrun.servlet.FilterChain.service(FilterChain.java:101)

The interesting part of the GC log:

{Heap before GC invocations=128 (full 2):
PSYoungGen      total 125760K, used 101968K [0xe9790000, 0xf4030000, 0xf4230000)
  eden space 81152K, 100% used [0xe9790000,0xee6d0000,0xee6d0000)
  from space 44608K, 46% used [0xee6d0000,0xefb24010,0xf1260000)
  to   space 43200K, 0% used [0xf1600000,0xf1600000,0xf4030000)
PSOldGen        total 333120K, used 265279K [0xd4230000, 0xe8780000, 0xe9790000)
  object space 333120K, 79% used [0xd4230000,0xe453fd10,0xe8780000)
PSPermGen       total 131072K, used 46072K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 35% used [0xc8230000,0xcaf2e330,0xd0230000)
953.746: [GC [PSYoungGen: 101968K->24816K(129408K)] 367247K->310323K(462528K), 1.0616140 secs] [Times: user=8.41 sys=0.05, real=1.06 secs]
Heap after GC invocations=128 (full 2):
PSYoungGen      total 129408K, used 24816K [0xe9790000, 0xf40f0000, 0xf4230000)
  eden space 86208K, 0% used [0xe9790000,0xe9790000,0xeebc0000)
  from space 43200K, 57% used [0xf1600000,0xf2e3c000,0xf4030000)
  to   space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
PSOldGen        total 333120K, used 285507K [0xd4230000, 0xe8780000, 0xe9790000)
  object space 333120K, 85% used [0xd4230000,0xe5900d10,0xe8780000)
PSPermGen       total 131072K, used 46072K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 35% used [0xc8230000,0xcaf2e330,0xd0230000)
}
{Heap before GC invocations=129 (full 2):
PSYoungGen      total 129408K, used 111024K [0xe9790000, 0xf40f0000, 0xf4230000)
  eden space 86208K, 100% used [0xe9790000,0xeebc0000,0xeebc0000)
  from space 43200K, 57% used [0xf1600000,0xf2e3c000,0xf4030000)
  to   space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
PSOldGen        total 333120K, used 285507K [0xd4230000, 0xe8780000, 0xe9790000)
  object space 333120K, 85% used [0xd4230000,0xe5900d10,0xe8780000)
PSPermGen       total 131072K, used 46077K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 35% used [0xc8230000,0xcaf2f4f0,0xd0230000)
955.698: [GC [PSYoungGen: 111024K->22952K(129472K)] 396531K->333355K(462592K), 1.1595700 secs] [Times: user=9.20 sys=0.05, real=1.16 secs]
Heap after GC invocations=129 (full 2):
PSYoungGen      total 129472K, used 22952K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 0% used [0xe9790000,0xe9790000,0xeebc0000)
  from space 43264K, 53% used [0xeebc0000,0xf022a020,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 333120K, used 310403K [0xd4230000, 0xe8780000, 0xe9790000)
  object space 333120K, 93% used [0xd4230000,0xe7150d10,0xe8780000)
PSPermGen       total 131072K, used 46077K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 35% used [0xc8230000,0xcaf2f4f0,0xd0230000)
}
{Heap before GC invocations=130 (full 3):
PSYoungGen      total 129472K, used 22952K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 0% used [0xe9790000,0xe9790000,0xeebc0000)
  from space 43264K, 53% used [0xeebc0000,0xf022a020,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 333120K, used 310403K [0xd4230000, 0xe8780000, 0xe9790000)
  object space 333120K, 93% used [0xd4230000,0xe7150d10,0xe8780000)
PSPermGen       total 131072K, used 46077K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 35% used [0xc8230000,0xcaf2f4f0,0xd0230000)
956.857: [Full GC [PSYoungGen: 22952K->115K(129472K)] [PSOldGen: 310403K->333119K(349568K)] 333355K->333235K(479040K) [PSPermGen: 46077K->46077K(131072K)], 1.7985720 secs] [Times: user=1.78 sys=0.0
3, real=1.80 secs]
Heap after GC invocations=130 (full 3):
PSYoungGen      total 129472K, used 115K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 0% used [0xe9790000,0xe97acd70,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 333119K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 95% used [0xd4230000,0xe877ffd8,0xe9790000)
PSPermGen       total 131072K, used 46077K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 35% used [0xc8230000,0xcaf2f4f0,0xd0230000)
}

{Heap before GC invocations=132 (full 5):
PSYoungGen      total 129472K, used 86208K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 100% used [0xe9790000,0xeebc0000,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349466K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe9776848,0xe9790000)
PSPermGen       total 131072K, used 45686K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaecd8a8,0xd0230000)
963.273: [Full GC [PSYoungGen: 86208K->24566K(129472K)] [PSOldGen: 349466K->349567K(349568K)] 435674K->374134K(479040K) [PSPermGen: 45686K->45686K(131072K)], 2.0615770 secs] [Times: user=2.06 sys=0.00, real=2.06 secs]
Heap after GC invocations=132 (full 5):
PSYoungGen      total 129472K, used 24566K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 28% used [0xe9790000,0xeaf8da90,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349567K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe978fff0,0xe9790000)
PSPermGen       total 131072K, used 45686K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaecd8a8,0xd0230000)
}
{Heap before GC invocations=133 (full 6):
PSYoungGen      total 129472K, used 86208K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 100% used [0xe9790000,0xeebc0000,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349567K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe978fff0,0xe9790000)
PSPermGen       total 131072K, used 45706K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaed2b60,0xd0230000)
965.953: [Full GC [PSYoungGen: 86208K->45172K(129472K)] [PSOldGen: 349567K->349567K(349568K)] 435775K->394740K(479040K) [PSPermGen: 45706K->45706K(131072K)], 2.1236450 secs] [Times: user=2.12 sys=0.00, real=2.12 secs]
Heap after GC invocations=133 (full 6):
PSYoungGen      total 129472K, used 45172K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 52% used [0xe9790000,0xec3ad250,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349567K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe978fff0,0xe9790000)
PSPermGen       total 131072K, used 45706K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaed2b60,0xd0230000)
}

{Heap before GC invocations=134 (full 7):
PSYoungGen      total 129472K, used 86208K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 100% used [0xe9790000,0xeebc0000,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349567K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe978fff0,0xe9790000)
PSPermGen       total 131072K, used 45725K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaed77f8,0xd0230000)
968.449: [Full GC [PSYoungGen: 86208K->54960K(129472K)] [PSOldGen: 349567K->349567K(349568K)] 435775K->404528K(479040K) [PSPermGen: 45725K->45725K(131072K)], 2.1485350 secs] [Times: user=2.15 sys=0.00, real=2.15 secs]
Heap after GC invocations=134 (full 7):
PSYoungGen      total 129472K, used 54960K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 63% used [0xe9790000,0xecd3c210,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349567K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe978fff0,0xe9790000)
PSPermGen       total 131072K, used 45725K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaed77f8,0xd0230000)
}
{Heap before GC invocations=135 (full 8):
PSYoungGen      total 129472K, used 86208K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 100% used [0xe9790000,0xeebc0000,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349567K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe978fff0,0xe9790000)
PSPermGen       total 131072K, used 45737K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaeda478,0xd0230000)
970.860: [Full GC [PSYoungGen: 86208K->57327K(129472K)] [PSOldGen: 349567K->349567K(349568K)] 435775K->406895K(479040K) [PSPermGen: 45737K->45678K(131072K)], 2.5495570 secs] [Times: user=2.55 sys=0.00, real=2.55 secs]
Heap after GC invocations=135 (full 8):
PSYoungGen      total 129472K, used 57327K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 66% used [0xe9790000,0xecf8bf40,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349567K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe978fff0,0xe9790000)
PSPermGen       total 131072K, used 45678K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaecbb90,0xd0230000)
}
{Heap before GC invocations=136 (full 9):
PSYoungGen      total 129472K, used 86208K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 100% used [0xe9790000,0xeebc0000,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349567K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe978fff0,0xe9790000)
PSPermGen       total 131072K, used 45689K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaece6f8,0xd0230000)
973.642: [Full GC [PSYoungGen: 86208K->67532K(129472K)] [PSOldGen: 349567K->349567K(349568K)] 435775K->417100K(479040K) [PSPermGen: 45689K->45689K(131072K)], 2.1439540 secs] [Times: user=2.14 sys=0.00, real=2.14 secs]
Heap after GC invocations=136 (full 9):
PSYoungGen      total 129472K, used 67532K [0xe9790000, 0xf40a0000, 0xf4230000)
  eden space 86208K, 78% used [0xe9790000,0xed983140,0xeebc0000)
  from space 43264K, 0% used [0xeebc0000,0xeebc0000,0xf1600000)
  to   space 42432K, 0% used [0xf1730000,0xf1730000,0xf40a0000)
PSOldGen        total 349568K, used 349567K [0xd4230000, 0xe9790000, 0xe9790000)
  object space 349568K, 99% used [0xd4230000,0xe978fff0,0xe9790000)
PSPermGen       total 131072K, used 45689K [0xc8230000, 0xd0230000, 0xd4230000)
  object space 131072K, 34% used [0xc8230000,0xcaece6f8,0xd0230000)
}

It looks like something is causing CF to use an absurd amount of memory, and these are the symptons of its death roll.  I dont believe that this is a problem requiring simple jvm.config tuning.  I think something is very broken with the CF server, I just dont know how to trace it properly.  This application has no such problems on any of the other web servers.

Thank you again...

After a few more lockups and logs which didnt turn up any additional information, I manually cleared out all the files in the wwwroot-tmp and cfclasses directories.  The lockups persisted.  Then I went through CF administrator and recreated all the mappings, datasources and olther settings, just in case they had somehow become corrupt.  Still the server woudl lock up from time to time.

I have finally had enough trying to fix this thing.  I uninstalled CF, then reinstalled CF, configured, and FINALLY all is happy and right with the world - so far...

Not out of the woods yet, it seems.  Had another very quick lockup that resolved itself after a couple of seconds.  Looks like it was the same issue.  Im now logging web2 as a control to see how close the other servers are to locking up as well.

Charlie, I agree with you about the death throws.  I think adjusting adjusting JVM memory variables is merely treating the symptoms and not getting to the actual root cause of the issues.  This whole thing smells strangely familliar of a problem we had a few years ago, where coldfusion was having difficult searching for CFCs due to the codebase being a symbiolic link.  Once we pointed CF and Apache configs to the actual codebase instead of the link, things hummed along nicely until recently.  Unfortunately the portion of our application which is having issues right now, does not have any symbolic links referenced in code, apache config or in CF paths / mappings.  The only other thing I can think of is that I have too broad path specified in ther custom tag paths.  There are a few upload and image directories which have thousands if not millions of files.  Im thinking that maybe CF is scanning through these directories looking for CFCs. Perhaps we are just now hitting some practical limitation in the OS on this particular node first.  I will try and narrow down the scope of the custom tag paths to something a little more specific, and see if that gives some relief.

As far as other errors happening before the above posted logs, I am quite sure there are none.  The first errors seem to be the GC OOM errors.  Nothing really precedes them in the logs.

Thank you both for helping me through all this...

-Tony

Thanks again Carl,

I have tried everything from the default 512 MB all the way up to min 1024 / max 3072.  The larger memory seems to reduce how quickly the server becomes unresponsive, but it does not prevent it.  Once it hangs, it seems to hang for good.  Once in a while it will come back after a few minutes, but usually it just hangs at 100% cpu till I restart the service.

As for now, I have left all the JVM settings at the default since the coldfusion reinstall.

I have narrowed down the CFC paths, and so far so good this morning.  After a full day or two passes, Ill know with some certainty if this was the cause or not.

Thanks again.

Charlie, Thanks for the help.

Yes, I have CFSTAT enabled on all the five webservers.  When the problematic server starts to hang up, I see only one or two processes running, but then once we start feeing the fruitless full GC's take place, the queue builds up because the server has become unresponsive.

I have now installed the server stat tool in my user dir, and the output seems pretty useful.  Here are the outputs.  Web1 is the problematic server.  Web2 has been running the longest, and has the highest session count, although it could also have been hit by a spider. Web3, web4 and web5 were restarted on tuesday for OS patches, so their session count is fairly low as well.

Web1

Applications

Sessions

Total Sessions on Server: 380

System

Memory Info: Used 78.8 MB Free: 36.6 MB Total: 115 MB Limit: 493  MB

Processors: 8

Server Name: web1

Instance Name:  coldfusion

Applications

Sessions

Total Sessions on Server: 380

System

Memory Info: Used 78.8 MB Free: 36.6 MB Total: 115 MB Limit: 493  MB

Processors: 8

Server Name: web1.absbook.com

Instance Name:  coldfusion

Web2

Applications

Sessions

Total Sessions on Server: 3175

System

Memory Info: Used 130 MB Free: 44.8 MB Total: 174 MB Limit: 493  MB

Processors: 8

Server Name: web2

Instance Name:  coldfusion

Web3

Applications

Sessions

Total Sessions on Server: 150

System

Memory Info: Used 89.5 MB Free: 36.4 MB Total: 126 MB Limit: 493  MB

Processors: 8

Server Name: web3

Instance Name:  coldfusion

Web4

Applications

Sessions

Total Sessions on Server: 187

System

Memory Info: Used 94.5 MB Free: 19.5 MB Total: 114 MB Limit: 493  MB
Processors: 8
Server Name: web4
Instance Name: coldfusion

Web5

Applications

Sessions

Total Sessions on Server: 875

System

Memory Info: Used 114 MB Free: 24.2 MB Total: 138 MB Limit: 493  MB
Processors: 8
Server Name: web5
Instance Name: coldfusion

Anyway, I will get some more data the next time we have another hangup on web1.  Its been running very well this morning.  The load on the server has not been fluctuating and spiking the way it has been the last few days.  In teh past, Ive received dozens if not hundreds of "high processor load" alerts on all five web boxes, from our zabbix monitoring system.  Since making the CFC path changes, I have not received a single one.  Im keeping my fingers crossed.

As for the sales pitch,  shoudl it come to teh point where we need a hired gun, I'd much suggest we go with an individual who's active and quite helpful in the community.  If it comes to this, Im sure Ill be in touch.

Thanks again,

-Tony

Well, its been a day and a half since the last lockup.  The load on the machine has never spiked.  Memory usage has never gotten out of control.  I tried to do some file counts on web1 in the directory in question, and a basic 'ls' command hangs up and poops out.  My guess is that there are an obscene amount of files in a single directory, and when CF was trying to scan for CFCs, this is what caused the memory spike and resulting outage.  Im going to keep monitoring this box closely for the next week or so, but I think we might have this thing in the bag.

Thank you both once again for your help.  Ims ure I can put this information to good use fine tuning the JVM settings once the dust settles.

Thanks,

-Tony