5 Replies Latest reply on Jul 5, 2017 6:13 AM by brookd

    Virtual File System Errors - java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification

    brookd Level 1

      This is on a CF9 server on Windows 2008

       

      We do a lot of writing to and then including from, the VFS (Virtual File System). Very intermittently, we get these errors when attempting to write to the RAM disk. Retrying the failed WRITE operation usually succeeds.

       

      The filename is always a unique file name created with createUUID() in the filename.

       

      I have read that this stack trace error is related to deleting items from an array while accessing it at the same time. Do I need to lock all access to the VFS? I feel like that would really hurt performance.  Should I clean up the temporary files created and written to the VFS at the end of the request, or maybe I should schedule it to run daily so there is less contention for those resources?

       

      java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) at java.util.AbstractList$Itr.next(AbstractList.java:343) at org.apache.commons.vfs.provider.ram.RamFileSystem.listChildren(RamFileSystem.java:100) at org.apache.commons.vfs.provider.ram.RamFileObject.doListChildren(RamFileObject.java:81) at org.apache.commons.vfs.provider.AbstractFileObject.getChildren(AbstractFileObject.java:55 7) at coldfusion.vfs.VFile.listFiles(VFile.java:480) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:122) at coldfusion.tagext.io.FileUtils.getDirectorySize(FileUtils.java:112) at coldfusion.runtime.CFPage.GetVFSMetaData(CFPage.java:634) at cfmonitor2ecfm1275412139.runPage(C:\inetpub\wwwroot\LF\web\health\monitor.cfm:44) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:360) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:94) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) at coldfusion.CfmServlet.service(CfmServlet.java:200) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doRequestNoFilter(FusionReacto rCoreFilter.java:712) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFusionRequest(FusionReactorC oreFilter.java:341) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilt er.java:246) at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java: 121) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 ) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

       

       

      Any help appreciated,  the VFS is really not working as well as I hoped it would.

       

      Brook

        • 1. Re: Virtual File System Errors - java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification
          BKBK Adobe Community Professional & MVP

          Like you, I suspect this is caused by race conditions. You could solve it with a design like:

           

          <!--- Use a suitable value for the timeout --->

          <cflock name="someUniqueNameDependingOnTheIndividualFileName" type="exclusive" timeout="2">

          <!--- Code that (writes to the file / reads from the file ) --->

          </cflock>

           

          <cflock name="someUniqueNameDependingOnTheIndividualFileName" type="exclusive" timeout="2">

          <!--- Code that (reads from the file / writes to the file ) --->

          </cflock>

           

          If the problem is indeed caused by race conditions, then you will have little choice but to use a lock. I don't expect the lock to hurt performance. If a process waited at the lock, it would only mean that the lock was necessary.

          • 2. Re: Virtual File System Errors - java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification
            brookd Level 1

            Ok thanks again for the ideas! I don't understand why a lock would be needed if the file name is unique, but... I guess I may have to try this method and see if it helps.

            • 3. Re: Virtual File System Errors - java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification
              brookd Level 1

              Whats weird is using the VFS seems problematic. I am wondering if these problems are related to the use of virtual directories within the VFS. I only just started using virtual sub directories to organize the temp files and be able to more easily delete them based on where they were stored (ie. temp + more perm files). But check this out:

               

              I have a remote monitoring service that checks server health and also checks VFS space remaining by running this code (monitor.cfm)

               

              <cfset vfsmetadata =  getVFSMetadata('ram')>

              <cfdump var="#vfsmetadata#" label="Virtual File System">

              <cfif vfsmetadata.free lt 10000000>

              Alert me.....

              </cfif>

               

              The dump is for me if I want to visually look at remaining space

               

              Intermittently throughout the day I get this error (Weird I get this error just from calling getVFSMetadata()).

               

              ---------------------

               

              An error occurred when performing a file operation listFiles on file /widcardparsingtemplates.

              org.apache.commons.vfs.FileSystemException: Could not list the contents of folder "ram:///widcardparsingtemplates". at org.apache.commons.vfs.provider.AbstractFileObject.getChildren(AbstractFileObject.java:56 1) at coldfusion.vfs.VFile.listFiles(VFile.java:480) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:122) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:131) at coldfusion.tagext.io.FileUtils.getDirectorySize(FileUtils.java:112) at coldfusion.runtime.CFPage.GetVFSMetaData(CFPage.java:634) at cfmonitor2ecfm1275412139.runPage(monitor.cfm:44) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:360) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:94) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) at coldfusion.CfmServlet.service(CfmServlet.java:200) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doRequestNoFilter(FusionReacto rCoreFilter.java:712) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFusionRequest(FusionReactorC oreFilter.java:341) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilt er.java:246) at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java: 121) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 ) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) Caused by: java.lang.ArrayIndexOutOfBoundsException: 2114 at

              org.apache.commons.vfs.FileSystemException: Could not list the contents of folder "ram:///widcardparsingtemplates". at org.apache.commons.vfs.provider.AbstractFileObject.getChildren(AbstractFileObject.java:56 1) at coldfusion.vfs.VFile.listFiles(VFile.java:480) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:122) at coldfusion.tagext.io.FileUtils.recursiveDirectorySize(FileUtils.java:131) at coldfusion.tagext.io.FileUtils.getDirectorySize(FileUtils.java:112) at coldfusion.runtime.CFPage.GetVFSMetaData(CFPage.java:634) at cfmonitor2ecfm1275412139.runPage(C:\inetpub\wwwroot\LF\web\health\monitor.cfm:44) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:231) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:416) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:360) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:94) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:2 8) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) at coldfusion.CfmServlet.service(CfmServlet.java:200) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89) at jrun.servlet.FilterChain.doFilter(FilterChain.java:86) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doRequestNoFilter(FusionReacto rCoreFilter.java:712) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFusionRequest(FusionReactorC oreFilter.java:341) at com.intergral.fusionreactor.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilt er.java:246) at com.intergral.fusionreactor.filter.FusionReactorFilter.doFilter(FusionReactorFilter.java: 121) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42 ) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46) at jrun.servlet.FilterChain.doFilter(FilterChain.java:94) at jrun.servlet.FilterChain.service(FilterChain.java:101) at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106) at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42) at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286) at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543) at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203) at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320) at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428) at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266) at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66) Caused by: java.lang.ArrayIndexOutOfBoundsException: 2114 at

              • 4. Re: Virtual File System Errors - java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification
                BKBK Adobe Community Professional & MVP

                Might the misspelling widcardparsingtemplates have something to do with it?

                • 5. Re: Virtual File System Errors - java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification
                  brookd Level 1

                  No,its not related (its misspelled with consistency )

                   

                  It may be to early to say for sure but locking all access to VFS may have resolved this problem... I'll  write back in a few days if so...