1 Reply Latest reply on Oct 21, 2009 10:51 PM by speedpedal

    run large file processing batches in scheduled task is failing

    Steve Powell Level 1

      We have a system that receieves real time messages via an event gateway. We file all these messages onto disk as small TXT files. I am trying to get our staging environment to load these TXT file overnight to bring staging back up to date with live. We are about 3 weeks behind at this point with about 45k messages a day.


      We have a CFM page that gets a list of all the TXT files, reads each one in and then uses a set of CFC objects to create an object structure from the message (the message has various subcomponents/combinations) it then calls a serialise function on the object which serialises the object into a database, it passes the serialise call down to all the subcomponents.


      This page seems to run fine if I run it manually. I can run blocks of 10k messages in comfortably as long as I'm willing to sit and watch it. This takes about an hour.


      I wanted to schedule this page overnight so it would process the messages at 3am and thus be up to date by the time we came in about 8.


      When I run the page as a scheduled task it gets to just over 1000 messages and crashes with a JVM out of memory error


      ROOT CAUSE: 
      java.lang.OutOfMemoryError: GC overhead limit exceeded


      javax.servlet.ServletException: ROOT CAUSE: 
      java.lang.OutOfMemoryError: GC overhead limit exceeded
           at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70)
           at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
           at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
           at jrun.servlet.FilterChain.service(FilterChain.java:101)
           at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
           at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
           at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
           at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
           at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
           at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
           at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
           at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
           at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)


      Now I can't let this run as it also takes down the gateways that are responding to live messaging and while we can catch up its not really very practical.


      I am resetting the object at the bottom of each loop using <CFSET msgObj = JavaCast("null","")> in the hope it wouldn't consume tons of memory, but obviously this isn't wokring. I am saving the scheduled task output to a file so I can see what happened but I don't really understand why its behaving so differently when i run it as a scheduled tasks to when i run the page in a browser,


      Anyone have any pointers as to what i might try?