July Reader DC Update Introduces Frequent Crashing

[JonathanP13791379]

We are seeing a large number (100+) of crashes in a customer environment that took the July 10th Acrobat Reader DC update (today). The crash is always an Access Violation thrown from your WM_GETMINMAX message handler on one of the background threads and the main UI thread is idle. We monitor the crashes in these environments very closely and did not see a single instance with this pattern until today's update.

  Image path: C:\Program Files (x86)\Adobe\Acrobat Reader DC\Reader\AcroRd32.dll

  Timestamp: Fri Jun 29 09:05:21 2018 (5B363CA1)

  File version: 18.11.20055.27899

Stack:

06 2b63dbe4 77092df3 2b63dc08 00000000 00000000 KERNELBASE!UnhandledExceptionFilter+0x165

07 2b63dc88 770205d6 2b63dd98 2b63dde8 7701ff41 ntdll!LdrpLogFatalUserCallbackException+0x4d

08 2b63dc94 7701ff41 2b63dd98 2b63e51c 2b63dde8 ntdll!KiUserCallbackExceptionHandler+0x26

09 2b63dcb8 7701ff13 2b63dd98 2b63e51c 2b63dde8 ntdll!ExecuteHandler2+0x26

0a 2b63dd80 7702068f 2b63dd98 2b63dde8 2b63dd98 ntdll!ExecuteHandler+0x24

0b 2b63dd80 2e180ecf 2b63dd98 2b63dde8 2b63dd98 ntdll!KiUserExceptionDispatcher+0xf

0c 2b63e274 74b690d1 2e180ecf 000709f2 00000024 AcroRd32+0x30ecf

0d 2b63e308 74b74d95 2e180ecf 00000000 00000024 user32!UserCallWinProcCheckWow+0x18e

0e 2b63e344 2e1817a8 2e180ecf 000709f2 00000024 user32!CallWindowProcW+0x8d

0f 2b63e360 2e181740 000709f2 00000024 00000000 AcroRd32+0x317a8

10 2b63e37c 2e181707 000709f2 00000024 00000000 AcroRd32+0x31740

11 2b63e394 2e1811e2 2b63e534 2e180f3f 00000024 AcroRd32+0x31707

12 2b63e3ac 74b68e71 000709f2 00000024 00000000 AcroRd32+0x311e2

13 2b63e3d8 74b690d1 2e180f3f 000709f2 00000024 user32!_InternalCallWinProc+0x2b                      0x24 = WM_GETMINMAX

14 2b63e46c 74b6932c 2e180f3f 00000000 00000024 user32!UserCallWinProcCheckWow+0x18e

15 2b63e4cc 74b71723 00b4d260 00000000 00000024 user32!DispatchClientMessage+0xdc

16 2b63e50c 77020666 2b63e524 00000000 2b63e9b4 user32!__fnINOUTLPPOINT5+0x43

17 2b63e560 74b77da5 00000000 2b63e8e0 2b63e5b8 ntdll!KiUserCallbackDispatcher+0x36

18 2b63e840 74b77f61 2b63e890 00000000 00000000 user32!VerNtUserCreateWindowEx+0x203

19 2b63e920 74b77fe7 2efde728 00000000 00000000 user32!CreateWindowInternal+0x17c

1a 2b63e95c 2e17d22d 00000000 2efe0f14 2efde728 user32!CreateWindowExW+0x37

1b 2b63e9c4 2e17fa34 00000000 2efe0f14 2efde728 AcroRd32+0x2d22d

1c 2b63eab0 2e17f77c 00000000 2efe0f14 2efde728 AcroRd32+0x2fa34

1d 2b63eb08 2e17f722 aa14302e 00000001 1cb016d8 AcroRd32+0x2f77c

1e 2b63eb30 2e17df4b aa142c7e 00000000 1cb016d8 AcroRd32+0x2f722

1f 2b63f760 2e17a826 aa142cd2 70716ede 749b69e0 AcroRd32+0x2df4b

20 2b63f7cc 2e179fed 2e150000 00000000 1c78e350 AcroRd32+0x2a826

21 2b63f7ec 61e67c7c 2e150000 00000000 1c78e350 AcroRd32+0x29fed

22 2b63ff80 749a7c04 328beb62 749a7be0 0829812c AcroPDFImpl+0x1c6d9

23 2b63ff94 7703ad8f 328beb62 01442815 00000000 kernel32!BaseThreadInitThunk+0x24

24 2b63ffdc 7703ad5a ffffffff 770200a3 00000000 ntdll!__RtlUserThreadStart+0x2f

25 2b63ffec 00000000 61e67961 328beb62 00000000 ntdll!_RtlUserThreadStart+0x1b

[Adorobat]

Hi Jonathan,

As per the issue description mentioned above, Acrobat Reader is crashing after installing the latest update, is that correct?

Could you try rebooting the machine once if you haven't already and check if that resolves the issue?

Does Reader crash when you launch the application or when opening any pdf file in the application?

Also, let us know the operating system installed on the machine?

Thanks,

Shivam

[JonathanP13791379]

It is a provisioned terminal services deployment (Citrix XenApp 7.15CU2) on Windows Server 2012 R2.

Yes, this started with yesterdays update (18.011.20055) and was not happening with the May update (18.011.20040). Nothing else in the provisioned image changed between Monday and yesterday.

From the numerous examples, it always appears to occur when loading an embedded PDF control into an application.

[shaktikeshri]

Hi Jonathan,

In addition to the questions asked by Shivam above, can you please let us know some details of the customer environment and workflows for crashes

Also, if possible can you please share crash dump using window debugger or

logs and dump obtained using Download Acrobat and Reader Process Monitor Tool - Adobe Labs

Thanks,

Shakti

[JonathanP13791379]

This is a health care environment and the memory dumps often contain protected PHI and PII. However, I'm a professional debugger and can execute any analysis you'd like and provide the output (with protected information removed).

The only consistent part of the various workflows involves a user performing an action that results in the creation of a an embedded Reader control. Loading report PDFs, opening billing summary PDFs, etc.

[soumik011]

Hi Jonathan,

In addition to the above queries please let us know what was the version before the July 10th update? Also does Reader protected mode is disabled in all the machines?

Regards

Soumik

[JonathanP13791379]

Here is an Adobe build extract that we collected on 7/2 and would be indicative of the state prior to yesterdays update. Protected mode was disabled and the previous version was your May release.

AdobeVersion : 18.011.20040

bAllowUserToChangeMsgPrefs : 0

bDisableSharePointFeatures : 1

bDisableWebmail : 1

bDisallowDLLUnload : Not found

bDontShowMsgWhenViewingDoc : 1

bProtectedMode : 0

bShowMsgAtLaunch : 0

bShowWelcomeScreen : 0

bToggleAdobeDocumentServices : 1

bToggleSendAndTrack : 1

bToggleWebConnectors : 1

cServices/bUpdater : 0

FeatureLockDown/bUpdater : Not found

[edwinbuza]

Hi Jonathan,

Am I correct thinking this crash ONLY happens if starting an embedded session inside that application? So any other usage of Acrobat is not causing any trouble?

[JonathanP13791379]

Yes, that seems to be true.

[edwinbuza]

Looks like it is trying to resize one or more acrord32 windows without getting the proper size/position first. No I don't know if acrord32 is reading in the wrong place or the method changed.

Another thing to try is updating the graphics drivers as those sometimes can be the cause of weird issues like this.

[JonathanP13791379]

We saw this in two entirely separate environments each with different server hardware and graphics drivers. The drivers and hardware also didn't change when we took the July update that introduced the crash.

[edwinbuza]

I now have frequent (as in daily) BSOD crashes. The only thing I can extract from the minidump is, it appears to be a driver issue. By coincidence I noticed my Acrobat DC was updated to 18.011.20063 _exactly_ when the crashes began even though it has been set _explicitly_ in the registry _not_ to do so with all the bUpdater/Mode/Check keys. In the logfiles however it seems to think it is set for AUTO_ALL. Hey, Adobe, care to explain to me how that is even possible????

Now then, I go to downgrade it back to 20035 and see if that will stabilize things.

I also will keep a _very_ close eye on the Acrobat DC version and will set logging to verbose to find out what is going on here.

[rotemm64520439]

Hi,

We seem to face with the same issue in our system. The client of our system crashes due to an error in the Adobe thread.

We also use the PDF control to show PDF files embedded int he system. Opening PDF files outside the system is functioning properly.

Is there a fix for this issue?

It has been a few months since the original message, and several patches were released.

Anyway,

This is the summary of the analysis of a dump we took (can't post the entire dump for privacy reasons):

and this is the call stack of the problematic thread:

Thread 39 - System ID 5532

Entry point	AcroPDFImpl!DllUnregisterServer+1c3be
Create time	08/10/2018 07:24:14
Time spent in user mode	0 Days 00:00:00.577
Time spent in kernel mode	0 Days 00:00:00.390

This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required.

Call Stack

verifier!VerifierStopMessage+1f8
verifier!AVrfpDphReportCorruptedBlock+1c2
verifier!AVrfpDphCheckPageHeapBlock+161
verifier!AVrfpDphFindBusyMemory+da
verifier!AVrfpDphFindBusyMemoryAndRemoveFromBusyList+20
verifier!AVrfDebugPageHeapFree+90
ntdll!RtlDebugFreeHeap+2f
ntdll!RtlpFreeHeap+5d
ntdll!RtlFreeHeap+142
kernel32!HeapFree+14
msvcr120!free+1a
CoolType!CTInit+5a5f3
CoolType!CTInit+5fb01
CoolType!CTInit+5a2ed
CoolType!CTInit+5a29b
CoolType!CTInit+572cc
CoolType!CTInit+57079
AcroRd32!CTJPEGDecoderReadNextTile+84965
AcroRd32!CTJPEGDecoderReadNextTile+84934
AcroRd32!CTJPEGDecoderReadNextTile+84728
AcroRd32!DllCanUnloadNow+1d65b
AcroRd32!DllCanUnloadNow+1d43f
AcroRd32!AcroWinMainSandbox+7423
user32!InternalCallWinProc+23
user32!UserCallWinProcCheckWow+109
user32!DispatchMessageWorker+3b5
user32!DispatchMessageW+f
AcroRd32!DllCanUnloadNow+1c8a1
AcroRd32!DllCanUnloadNow+1c6ad
AcroRd32!AcroWinMainSandbox+8ed
AcroRd32!AcroWinBrowserMain+18
AcroPDFImpl!DllUnregisterServer+1c6d9
kernel32!BaseThreadInitThunk+e
ntdll!__RtlUserThreadStart+70
ntdll!_RtlUserThreadStart+1b

Any help will be very much appreciated!

Thanks!

Rotem,