6 Replies Latest reply on Nov 26, 2012 1:33 PM by Tim Goodman

    CQ5 fails to start - Bad File Descriptor

    Tim Goodman Level 1

      Hi,

       

      We are having an intermittant problem on servers running CQ5 (SP1 not installed).

       

      Occaisionly the server stops working - with the Service unavailable message appearing in the CRX and CQ Admin pages.  The Felix console is accessible, but the password has been reset to default.

       

      It appears to be related to the Persistance store - but I'm not sure what to check exactly. 

      Is it related to the server rather than the Software?  Any ideas how we can identify the problem?

       

      Sometimes a restart fixes it, other times you need to rebuild the server

       

       

      Service Unavailable

      AuthenticationSupport service missing. Cannot authenticate request.

       

       

       

      23.11.2012 06:33:36.309 *WARN* [FelixStartLevel] com.day.crx.persistence.tar.ClusterTarSet Could not open java.io.IOException: Bad file descriptor

                at java.io.RandomAccessFile.writeBytes(Native Method)

                at java.io.RandomAccessFile.write(RandomAccessFile.java:453)

                at com.day.crx.persistence.tar.file.TarFile.write(TarFile.java:736)

                at com.day.crx.persistence.tar.file.TarFile.writeData(TarFile.java:629)

                at com.day.crx.persistence.tar.file.TarFile.appendMetaData(TarFile.java:703)

                at com.day.crx.persistence.tar.file.TarFile.append(TarFile.java:585)

                at com.day.crx.persistence.tar.TarSet.switchDataFile(TarSet.java:441)

                at com.day.crx.persistence.tar.TarSet.open(TarSet.java:223)

                at com.day.crx.persistence.tar.ClusterTarSet.reopenCopy(ClusterTarSet.java:1354)

                at com.day.crx.persistence.tar.ClusterTarSet.open(ClusterTarSet.java:787)

                at com.day.crx.persistence.tar.TarPersistenceManager.openTarSet(TarPersistenceManager.java:9 63)

                at com.day.crx.persistence.tar.TarPersistenceManager.init(TarPersistenceManager.java:494)

                at com.day.crx.core.CRXRepositoryImpl.createVersionManager(CRXRepositoryImpl.java:845)

                at org.apache.jackrabbit.core.RepositoryImpl.<init>(RepositoryImpl.java:311)

                at com.day.crx.core.CRXRepositoryImpl.<init>(CRXRepositoryImpl.java:283)

                at com.day.crx.core.CRXRepositoryImpl.create(CRXRepositoryImpl.java:258)

                at com.day.crx.core.CRXRepositoryImpl.create(CRXRepositoryImpl.java:243)

                at com.day.crx.sling.server.impl.jmx.ManagedRepository.activate(ManagedRepository.java:164)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:227)

                at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:38)

                at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:591)

                at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:472)

                at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:146)

                at org.apache.felix.scr.impl.manager.ImmediateComponentManager.createImplementationObject(Im mediateComponentManager.java:226)

                at org.apache.felix.scr.impl.manager.ImmediateComponentManager.createComponent(ImmediateComp onentManager.java:118)

                at org.apache.felix.scr.impl.manager.DelayedComponentManager.createRealComponent(DelayedComp onentManager.java:95)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager$Registered.getService(Abstract ComponentManager.java:1296)

                at org.apache.felix.scr.impl.manager.DelayedComponentManager.getService(DelayedComponentMana ger.java:88)

                at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistratio nImpl.java:310)

                at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.jav a:221)

                at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:292)

                at org.apache.felix.framework.Felix.getService(Felix.java:3010)

                at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:329)

                at org.osgi.util.tracker.ServiceTracker.addingService(ServiceTracker.java:442)

                at org.apache.aries.jmx.whiteboard.Activator$MBeanTracker.addingService(Activator.java:101)

                at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:896)

                at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:261)

                at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:233)

                at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:840)

                at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispat cher.java:871)

                at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java :733)

                at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:662 )

                at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:3765)

                at org.apache.felix.framework.Felix.registerService(Felix.java:2877)

                at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:251)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractCompon entManager.java:456)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerComponentService(Abstr actComponentManager.java:508)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractC omponentManager.java:1157)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractCompo nentManager.java:334)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManage r.java:158)

                at org.apache.felix.scr.impl.config.ImmediateComponentHolder.enableComponents(ImmediateCompo nentHolder.java:313)

                at org.apache.felix.scr.impl.BundleComponentActivator.loadDescriptor(BundleComponentActivato r.java:241)

                at org.apache.felix.scr.impl.BundleComponentActivator.initialize(BundleComponentActivator.ja va:147)

                at org.apache.felix.scr.impl.BundleComponentActivator.<init>(BundleComponentActivator.java:1 11)

                at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:274)

                at org.apache.felix.scr.impl.Activator.bundleChanged(Activator.java:192)

                at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatc her.java:807)

                at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java :729)

                at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:610)

                at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:3754)

                at org.apache.felix.framework.Felix.startBundle(Felix.java:1780)

                at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1156)

                at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:266)

                at java.lang.Thread.run(Thread.java:662)

      23.11.2012 06:33:36.313 *WARN* [FelixStartLevel] com.day.crx.persistence.tar.ClusterTarSet Could not open java.io.IOException: Bad file descriptor

                at java.io.RandomAccessFile.writeBytes(Native Method)

                at java.io.RandomAccessFile.write(RandomAccessFile.java:453)

                at com.day.crx.persistence.tar.file.TarFile.write(TarFile.java:736)

                at com.day.crx.persistence.tar.file.TarFile.writeData(TarFile.java:629)

                at com.day.crx.persistence.tar.file.TarFile.appendMetaData(TarFile.java:703)

                at com.day.crx.persistence.tar.file.TarFile.append(TarFile.java:585)

                at com.day.crx.persistence.tar.TarSet.switchDataFile(TarSet.java:441)

                at com.day.crx.persistence.tar.TarSet.open(TarSet.java:223)

                at com.day.crx.persistence.tar.ClusterTarSet.reopenCopy(ClusterTarSet.java:1354)

                at com.day.crx.persistence.tar.ClusterTarSet.open(ClusterTarSet.java:787)

                at com.day.crx.persistence.tar.TarPersistenceManager.openTarSet(TarPersistenceManager.java:9 63)

                at com.day.crx.persistence.tar.TarPersistenceManager.init(TarPersistenceManager.java:494)

                at com.day.crx.core.CRXRepositoryImpl.createVersionManager(CRXRepositoryImpl.java:845)

                at org.apache.jackrabbit.core.RepositoryImpl.<init>(RepositoryImpl.java:311)

                at com.day.crx.core.CRXRepositoryImpl.<init>(CRXRepositoryImpl.java:283)

                at com.day.crx.core.CRXRepositoryImpl.create(CRXRepositoryImpl.java:258)

                at com.day.crx.core.CRXRepositoryImpl.create(CRXRepositoryImpl.java:243)

                at com.day.crx.sling.server.impl.jmx.ManagedRepository.activate(ManagedRepository.java:164)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:227)

                at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:38)

                at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:591)

                at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:472)

                at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:146)

                at org.apache.felix.scr.impl.manager.ImmediateComponentManager.createImplementationObject(Im mediateComponentManager.java:226)

                at org.apache.felix.scr.impl.manager.ImmediateComponentManager.createComponent(ImmediateComp onentManager.java:118)

                at org.apache.felix.scr.impl.manager.DelayedComponentManager.createRealComponent(DelayedComp onentManager.java:95)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager$Registered.getService(Abstract ComponentManager.java:1296)

                at org.apache.felix.scr.impl.manager.DelayedComponentManager.getService(DelayedComponentMana ger.java:88)

                at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistratio nImpl.java:310)

                at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.jav a:221)

                at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:292)

                at org.apache.felix.framework.Felix.getService(Felix.java:3010)

                at org.apache.felix.framework.BundleContextImpl.getService(BundleContextImpl.java:329)

                at org.osgi.util.tracker.ServiceTracker.addingService(ServiceTracker.java:442)

                at org.apache.aries.jmx.whiteboard.Activator$MBeanTracker.addingService(Activator.java:101)

                at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:896)

                at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:261)

                at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:233)

                at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:840)

                at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispat cher.java:871)

                at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java :733)

                at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:662 )

                at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:3765)

                at org.apache.felix.framework.Felix.registerService(Felix.java:2877)

                at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:251)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractCompon entManager.java:456)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerComponentService(Abstr actComponentManager.java:508)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager$Unsatisfied.activate(AbstractC omponentManager.java:1157)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractCompo nentManager.java:334)

                at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManage r.java:158)

                at org.apache.felix.scr.impl.config.ImmediateComponentHolder.enableComponents(ImmediateCompo nentHolder.java:313)

                at org.apache.felix.scr.impl.BundleComponentActivator.loadDescriptor(BundleComponentActivato r.java:241)

                at org.apache.felix.scr.impl.BundleComponentActivator.initialize(BundleComponentActivator.ja va:147)

                at org.apache.felix.scr.impl.BundleComponentActivator.<init>(BundleComponentActivator.java:1 11)

                at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:274)

                at org.apache.felix.scr.impl.Activator.bundleChanged(Activator.java:192)

                at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatc her.java:807)

                at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java :729)

                at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:610)

                at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:3754)

                at org.apache.felix.framework.Felix.startBundle(Felix.java:1780)

                at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1156)

                at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:266)

                at java.lang.Thread.run(Thread.java:662)

        • 1. Re: CQ5 fails to start - Bad File Descriptor
          Jörg Hoh Adobe Employee

          If that situation happes, check how many files the java process has open. This message might occur, when the predefined limit is hit. In the default start script you can change it by redefining the variable "CQ_FILE_SIZE_LIMIT" and a restart.

           

          Jörg

          • 2. Re: CQ5 fails to start - Bad File Descriptor
            Sham HC Level 7

            Check the data tar files permissions at [1]. If it is 444 correct the permission.

            Please note that a bad file descriptor error is usually due to a low level file system corruption. Sometimes I have see customer

            getting this exception when index file got corrupted. The reason could be "Power failure", "Force shutdown", "Starting CQ with different users account" .... If the bad file might just be a tar index file. Try doing the following:

              -  stop cq

              -  delete all crx-quickstart/repository/workspaces/crx.default/index*.tar files

              -  start cq (the index tar files will be rebuilt)

             

            [1]

            ${CQ_HOME}/crx-quickstart/repository/workspaces/crx.default/data_*.tar

            ${CQ_HOME}/crx-quickstart/repository/tarJournal/data_*.tar

            ${CQ_HOME}/crx-quickstart/repository/version/data_*.tar

            1 person found this helpful
            • 3. Re: CQ5 fails to start - Bad File Descriptor
              Tim Goodman Level 1

              Thanks very much for the suggestions:

               

              @Sham HC

               

              - The data files have permissions of 644.  We assume this is correct, but let us know

              - We have tried deleting the index file, but it didn't get recreated.  I assume this means there is another error with the data file which occurs before the index can be created.

               

              @Jorg

               

              - The default is set at 8142.  We haven't tried extending this, and I'm not sure how we can check how many files are open.  We rebooted the server which we assume would reset any file connections.

               

              Is there any way we can check the validity of the data files?

               

              Thanks

              • 4. Re: CQ5 fails to start - Bad File Descriptor
                Tim Goodman Level 1

                @Sham HC

                 

                we just tried deleting all index files (i.e. not just the workspace, but also in tarjournal and version directories).

                 

                This time CQ5 did recreate them, and restart correctly.

                 

                Is there anyway to diagnose what might be causing this issue?

                 

                Thanks

                • 5. Re: CQ5 fails to start - Bad File Descriptor
                  Sham HC Level 7

                  Good to hear system recovered. Not sure what caused it, analysing all the logs before & after the issue might help.  If you have not encoured

                  unusally activity like "Power failure", "Force shutdown", "Starting CQ with different users account" I would recommend to file daycare with all the logs before & after issue.

                  • 6. Re: CQ5 fails to start - Bad File Descriptor
                    Tim Goodman Level 1

                    Thanks - will do.