We have a system that receieves real time messages via an event gateway. We file all these messages onto disk as small TXT files. I am trying to get our staging environment to load these TXT file overnight to bring staging back up to date with live. We are about 3 weeks behind at this point with about 45k messages a day.
We have a CFM page that gets a list of all the TXT files, reads each one in and then uses a set of CFC objects to create an object structure from the message (the message has various subcomponents/combinations) it then calls a serialise function on the object which serialises the object into a database, it passes the serialise call down to all the subcomponents.
This page seems to run fine if I run it manually. I can run blocks of 10k messages in comfortably as long as I'm willing to sit and watch it. This takes about an hour.
I wanted to schedule this page overnight so it would process the messages at 3am and thus be up to date by the time we came in about 8.
When I run the page as a scheduled task it gets to just over 1000 messages and crashes with a JVM out of memory error
ROOT CAUSE: java.lang.OutOfMemoryError: GC overhead limit exceeded
javax.servlet.ServletException: ROOT CAUSE: java.lang.OutOfMemoryError: GC overhead limit exceeded at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
Now I can't let this run as it also takes down the gateways that are responding to live messaging and while we can catch up its not really very practical.
I am resetting the object at the bottom of each loop using <CFSET msgObj = JavaCast("null","")> in the hope it wouldn't consume tons of memory, but obviously this isn't wokring. I am saving the scheduled task output to a file so I can see what happened but I don't really understand why its behaving so differently when i run it as a scheduled tasks to when i run the page in a browser,
Anyone have any pointers as to what i might try?
You might need to tweak your server settings a bit. Can you post your jvm.config?
Also, this might be a good late night reading material: