This is going to get a little esoteric. The signature appearance, in other words what you see on the page, is not the signature, but rather a graphical representation of the actual signature. The actual signature is a blob of hex encoded, ASN.1 formatted data that conforms to the CMS/PKCS 7 standards written into the PDF file and is not something you see unless you open the file in an text editor. If you move the computer clock back and sign a file, and the signature does not contain a time stamp no one would be the wiser. However, if you move the clock too far back the signature creation time would fall out of revocation information's usable time frame, you wouldn't have any valid revocation collateral and the signature would be in an unknown state.
However, you were asking about the delta between the trusted time stamp server and the computer time. If it really is a reputable time stamp service and it's trusted by the document recipient, no one would pay any attention to the computer time displayed in the signature appearance.
That begs the question why can't the timestamp time be displayed in the signature appearance, That's because the signature appearance is created prior to the file being signed so that the appearance is covered by the digital signature. That way were someone to try and modify the signature appearance it would break the signature. So, the appearance is created before the actual digital signature is created, the other part of the equation is the timestamp is procured after the signature is created. In reality, the timestamp is a digital signature over the document signature. That is, the document signature is created and once it exists it is digested (hashed) and the digest is sent to the timestamp server to be signed.
What you end up doing is trusting the time that came from the timestamp server because you know they have no reason to fuss with the computer time that runs the server. Their business model is dependent on supplying accurate times, and as a disinterested third party they have no reason to lie about the time. However, by the time Acrobat gets a hold of the time from the timestamp token (the response to the timestamp request) all of the document signature has been committed to disk (that's another security issue) so the timestamp response has to be written into an unsigned portion of the signature object since it's really not part of the signature proper.
And that's probably way more than you wanted to know.
Nobody looking at the validity of a digital signature should use ANY information on the page. I know they might, but it is quite wrong to do so. Accepting the date is only one step away from accepting the entire appearance at face value, and it would take only minutes to fake that.