OutOfMemoryError while Packaging big e-books

[GianpieroPP]

Hi to all,

I'm having a problem while packaging ebook onto ACS4 platform when the pdf file size is up to 90MB. While uploading a pdf e-book (size 90 MB, approx.), I get a console error: "OutOfMemoryError" to indicate that the JVM has no more free heap space available.
The problem is due the implementation of the client (UploadTest). So I've modified it (the client) and I've fixed the problem, on the client side... but, unfortunatley, the same problem is on the server (Package web application). This is the stack trace:

Nov 1, 2009 10:20:10 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet Package threw exception
java.lang.OutOfMemoryError: Java heap space
at java.lang.String.<init>(String.java:208)
at java.lang.StringBuffer.toString(StringBuffer.java:586)
at com.adobe.adept.xml.XMLFieldReader.endElement(XMLFieldReader.java:460)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:633)
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanEndElement(XMLNSDocumentScannerImpl.java:719)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(XMLDocumentFragmentScannerImpl.java:1685)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:368)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:834)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:148)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1242)
at com.adobe.adept.servlet.AdeptServlet.doPost(AdeptServlet.java:180)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:568)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)

Now the problem is "how to fix the problem if the source code is not available?". Maybe I can post my solution for the client app so the ACS4 team can apply the same solution on the Package web application?

Thank you in advance.
Gianpiero Piccolo.

[Guest]

Use the -xml option of UploadTest so that you can specify the path of the using the dataPath element. This greatly reduces the memory usage.

[GianpieroPP]

I'm just using the -xml option. But I don't know what is the "datapath" element.

This is my command line:

java -Xmx1024M -classpath ".\UploadTest\classes;E:\software\apache\java\xalan-j_2_7_1-bin\xalan.jar;E:\software\apache\java\xalan-j_2_7_1-bin\serializer.jar" -Djavax.xml.transform.TransformerFactory=org.apache.xalan.xsltc.trax.TransformerFactoryImpl com.adobe.adept.upload.UploadTest http://************:8080/packaging/Package "E:\Documenti\eBook personali\****\*****.pdf" -pass ******** -xml -jpg

The xalan implementation for xml serialization is more efficient. The -xml that you suggested is already on the command line.

The xml content is:

<package xmlns="http://ns.adobe.com/adept">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
  <dc:title>*******</dc:title>
  <dc:creator>**************</dc:creator>
  <dc:publisher>**********</dc:publisher>
  <dc:format>application/pdf</dc:format>
</metadata>
<permissions>
  <display>
   <device/>
   <until>2013-01-13T13:10:49-07:00</until>
  </display>
  <excerpt>
   <until>2013-01-13T13:10:49-07:00</until>  
  </excerpt>
  <print>
   <count initial="10" max="20" incrementInterval="3600"/>
   <maxResolution>300</maxResolution>
  </print>
</permissions>
</package>

What's "the path of the using the dataPath element"?

Thank you.

Gianpiero

[Guest]

the dataPath element is covered in the Tech Ref, but it is the path (relative to the packaging server) of the content.  Also you will need to add the -dataPath option into the UploadTest command line to tell it not to emit a data element

[GianpieroPP]

Great, dear Jim!

So you're telling me:

1. upload in a different way the unencrypted ebook file on the media server (into a location unreacheable via any URL from users).
2. use the UploadTest client with -dataPath command line option to inform the packaging server that the unencrpyted file in already on the server (see prevoius step)
3. and then, delete the unencrypted ebook file from the media server to free unutilized space.

If above three steps are ok, this night I'm testing this procedure. I'll inform you about the (eventually) successfull.

Thank you.

If this procedure is ok, then I'll write some line of code into the UploadTest client to automatically upload via ftp the ebook file before starting the packaging process (and finally delete it).

Good bye.

Gianpiero

[Guest · Accepted solution]

1 and 3 are up to you.  While the file needs to be a local file path for the packaging service, the file does not need to be accessible via a URL form any body else.  Also the packaging service doesn't really need to be directly accessible outside of your firewall.  For instance what some ACS4 operators have done is to create a service (in their preferred language), that displays a form which allows the user to upload the file, specify any preferred metadata, etc... Such a service will typically:

• authenticate the user from their own internal (non ACS4) account information
• copy the uploaded file to a file share (accessible to the ACS4 packaging service but not HTTP accessible outside of the firewall)
• send a request to ACS4  admin service to create distribution rights
• update the internal billing system with information about customer and content

[GianpieroPP]

The procedure is ok. I can package my ebook (120MB size!!). Thank you very mutch, Jim.

Another help: before I'll implement the web form by myself from scratch, there is anybody that can kindly distribuite to me his j2ee implementation of it?

Thanks in advance.

GianpieroPP

gianpiero.sicurezza.java@fastwebnet.it

[benbiddington]

Decompile it and have a look, doPost is an awesome couple of hundred lines. Pretty depressing reading.