Reader Extensions can apply rights to any conforming PDF document, not just ones created by Adobe. That being said, the bulk of the testing is done with Adobe-generated files, so there can be problems at times if the PDF has errors that Reader overlooks (I'd call that a bug). So you can diagnose this a bit by opening in Acrobat and saving the file and then reader extending. [Note that I'm suggesting diagnostic technique rather than part of a process.]
To run this, you'd create a web-service endpoint for LiveCycle and have your Java code call the server via the web service call.
Keep in mind that you do need to have proper RE licensing in place so that reader-extending is done on a per-document or per-recipient model or other licensing worked out with Adobe. I assume that you have this in place.