Do you use a signing certificate obtained from a Certificate Authority (CA), not a self-signed certificate . The rest of my reply is for the CA case, not for the self-signed case.
When you digitally sign a PDF Acrobat automatically by default embeds revocation information in the signature block in the PDF file. It does that so that a signature can be validated at the signing time long after the signing certificate expires. In some cases the size of revocation information (CA creates it) can be big and that;s the most plausible explanation.
You can check that by right-clicking on the signature, selecting 'Show Signature Properties' and then 'Show Certificate' (do not remember exact wording. When Certificate Viewer comes up, select the Revocation tab on the top an it will most likely have a text, something like this: 'The certificate is valid because it does not appear in a CRL embedded in the signature' (I do not have an Acrobat at hand now and cannot verify the exact wording). If this is what you see, then my explanation is on the mark. If it something else, we need more info.