I modified a process workflow in livecycle workbench. It was a simple modification that uses a resource. On save, I restarted JBoss instance that holds livecycle and I started have the error:
com.adobe.workflow.template.document.TemplateNodeNotFoundException: Template object: A1271100735162 not found.
I'll add the complete stack trace at the bottom.
The process instance also started stalling when it reached the two workflow actions that were modified in workbench. By checking directly into mysql in the table tb_action_instance, I could see the stalled actions and they were using action templates:
A1271100735161 A1271100735163
so clearly the template A1271100735162 was replaced by A1271100735163. The action instance would stall, they would never display errors or get blocked, they would just not terminate and the whole process is stalled.
I identified all the old action that used the old template: SELECT process_instance_id, status, start_time, complete_time FROM tb_action_instance WHERE action_template_id = 'A1271100735162';
I delete all of them, including their respective processes in the tables: tb_task tb_process_instance tb_job_instance tb_action_instance
I kept getting the error. I then removed all terminated and non terminated processes from the above tables and I kept having the same errors.
By going to workbench and resaving, redrawing parts of the process, I was able to get the stalled processes to run, but I keep having the template A1271100735162 error and it's flooding my logs.
Where does livecycle store the templates and workflow, I'm not sure if its on disk, but I would like to remove everything related to the bad template.
Here's the complete stack trace:
Caused by: com.adobe.workflow.template.document.TemplateNodeNotFoundException: Template
object: A1271100735162 not found.
at
com.adobe.workflow.template.document.DefaultAbstractTemplateDocument.getTemplateNodeById(DefaultAbstractTemplateDocument.java:84)
at com.adobe.workflow.engine.ProcessEngineBMTBean.asyncTerminateActionCommand(ProcessEngineBMTBean.java:998)
... 44 more
2014-03-21 ERROR [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Exception in JMSCI message listener
javax.ejb.TransactionRolledbackLocalException: RuntimeException
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:262)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:399)
at org.jboss.ejb.Container.invoke(Container.java:960)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1139)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:262)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:399)
at org.jboss.ejb.Container.invoke(Container.java:960)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1139)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1452)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266)
at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:891)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
at org.jboss.mq.SpySession.run(SpySession.java:323)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
at java.lang.Thread.run(Thread.java:619)
Caused by: javax.ejb.EJBException: RuntimeException
at com.adobe.workflow.engine.ProcessCommandControllerBean.doOnMessage(ProcessCommandControllerBean.java:206)
at com.adobe.workflow.engine.ProcessCommandControllerBean.onMessage(ProcessCommandControllerBean.java:99)
at sun.reflect.GeneratedMethodAccessor353.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:492)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
... 15 more
Caused by: javax.ejb.EJBException: RuntimeException
at org.jboss.ejb.plugins.LogInterceptor.handleException(LogInterceptor.java:417)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:209)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
at org.jboss.ejb.Container.invoke(Container.java:960)
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
at $Proxy215.asyncTerminateActionCommand(Unknown Source)
at com.adobe.workflow.engine.ProcessCommandControllerBean.doOnMessage(ProcessCommandControllerBean.java:177)
... 24 more
Caused by: com.adobe.workflow.template.document.TemplateNodeNotFoundException: Template object: A1271100735162 not found.
at com.adobe.workflow.engine.ProcessEngineBMTBean.asyncTerminateActionCommand(ProcessEngineBMTBean.java:1028)
at sun.reflect.GeneratedMethodAccessor355.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:173)
at org.jboss.ejb.plugins.TxInterceptorBMT.invoke(TxInterceptorBMT.java:77)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
... 31 more
Caused by: com.adobe.workflow.template.document.TemplateNodeNotFoundException: Template object: A1271100735162 not found.
at com.adobe.workflow.template.document.DefaultAbstractTemplateDocument.getTemplateNodeById(DefaultAbstractTemplateDocument.java:84)
at com.adobe.workflow.engine.ProcessEngineBMTBean.asyncTerminateActionCommand(ProcessEngineBMTBean.java:998)
... 44 more
2014-03-21 12:01:32,905 WARN [org.jboss.ejb.plugins.jms.DLQHandler] Message resent too many times; sending it to DLQ; message id=ID:9-1395417692668461
2014-03-21 12:01:32,912 ERROR [STDERR] Mar 21, 2014 12:01:32 PM com.adobe.workflow.recovery.JMSRecoveryMDB onMessage
WARNING: Could not re-process JMS message Illegal property name: JMSXDeliveryCount
