AIR Crash on iPad

[William Spence]

So, my first AIR app is complete and I am in the process of profiling it

before I submit it to the various App Stores. On Android, I ran it for over

an hour with no crashes or hangups, although looking at Scout, its memory

seems to run awfully high IMO.  When it reaches around 550MB, AIR kicks it

back down to 200MB with garbage collection. It usually hangs out around

200-350MB on Android.

Apple is a different story.  It just crashes randomly, after 10 minutes,

after 20 minutes.  The crashes happen around 370MB, but I am not convinced

that it is a memory issue.  Looking at the various crash reports generated

by the iPad, I have attached five different crashes that I logged.  In my

estimation, Apple is kicking the app out because it is not responding to

something in the appropriate amount of time, but I have not a clue what it

is, how to debug it, or where to look to address it.  Please let me know if

you have any experience with this at all as I really feel lost at this

level.

Thanks!

Log 1:

Incident Identifier: 473BFE42-EE9B-446E-88C2-FF07573FD341

CrashReporter Key:   4b6e6a5c7a0dd42aa5f46933ea8e9e76f089bbac

Hardware Model:      iPad3,3

Process:             RealDeals [5042]

Path:              

/var/containers/Bundle/Application/CCF0C452-56B6-4562-8B87-559184AC4B56/RealDeals.app/RealDeals

Identifier:          com.RealDeals.tablet

Version:             1.0.0 (1.0.0)

Code Type:           ARM (Native)

Parent Process:      launchd [1]

Date/Time:           2017-10-10 00:11:14.14 -0400

Launch Time:         2017-10-10 00:10:31.31 -0400

OS Version:          iOS 9.3.5 (13G36)

Report Version:      105

Exception Type:  00000020

Exception Codes: 0x000000008badf00d

Exception Note:  SIMULATED (this is NOT a crash)

Highlighted by Thread:  0

Application Specific Information:

com.RealDeals.tablet failed to exit after 5.00s

Elapsed total CPU time (seconds): 2.250 (user 2.250, system 0.000), 23% CPU

Elapsed application CPU time (seconds): 0.339, 3% CPU

Filtered syslog:

None found

Log 2:

Incident Identifier: 2C20580A-15A3-4B1B-B84C-8F3E9152A5FA

CrashReporter Key:   4b6e6a5c7a0dd42aa5f46933ea8e9e76f089bbac

Hardware Model:      iPad3,3

Process:             RealDeals [4028]

Path:              

/var/containers/Bundle/Application/572A85B9-4C40-42B7-8E72-44FA0A4955DB/RealDeals.app/RealDeals

Identifier:          com.RealDeals.tablet

Version:             1.0.0 (1.0.0)

Code Type:           ARM (Native)

Parent Process:      launchd [1]

Date/Time:           2017-10-05 19:40:50.50 -0400

Launch Time:         2017-10-05 19:35:13.13 -0400

OS Version:          iOS 9.3.5 (13G36)

Report Version:      105

Exception Type:  00000020

Exception Codes: 0x000000008badf00d

Exception Note:  SIMULATED (this is NOT a crash)

Highlighted by Thread:  0

Application Specific Information:

<BKNewProcess: 0x15531de0; com.RealDeals.tablet; pid: 4028; hostpid: -1> has

active assertions beyond permitted time:

{(

    <BKProcessAssertion: 0x15634080> id:

47-7C43F87B-C8D2-4041-AE31-1191FF9F89E3 name: Deliver Message process:

<BKNewProcess: 0x15531de0; com.RealDeals.tablet; pid: 4028; hostpid: -1>

permittedBackgroundDuration: 10.000000 reason: suspend owner pid:47

preventSuspend  preventThrottleDownCPU  preventThrottleDownUI

preventSuspendOnSleep ,

    <BKProcessAssertion: 0x15523d10> id:

47-3565E3B3-74C1-48F9-80B0-C942970BFBEB name: Suspending process:

<BKNewProcess: 0x15531de0; com.RealDeals.tablet; pid: 4028; hostpid: -1>

permittedBackgroundDuration: 10.000000 reason: suspend owner pid:47

preventSuspend  preventThrottleDownCPU  preventThrottleDownUI

preventSuspendOnSleep

)}

Elapsed total CPU time (seconds): 20.110 (user 20.110, system 0.000), 6% CPU

Elapsed application CPU time (seconds): 11.832, 3% CPU

Filtered syslog:

None found

Log 3:

Incident Identifier: A88DE2B8-9241-4C28-AD52-AB02905EB348

CrashReporter Key:   4b6e6a5c7a0dd42aa5f46933ea8e9e76f089bbac

Hardware Model:      iPad3,3

Process:             RealDeals [648]

Path:              

/var/containers/Bundle/Application/29E5AFCC-E342-4FDC-9C66-AD2A64E31BFB/RealDeals.app/RealDeals

Identifier:          com.RealDeals.tablet

Version:             1.0.0 (1.0.0)

Code Type:           ARM (Native)

Parent Process:      launchd [1]

Date/Time:           2017-09-05 21:31:00.00 -0400

Launch Time:         2017-09-05 21:26:43.43 -0400

OS Version:          iOS 9.3.5 (13G36)

Report Version:      105

Exception Type:  00000020

Exception Codes: 0x000000008badf00d

Exception Note:  SIMULATED (this is NOT a crash)

Highlighted by Thread:  0

Application Specific Information:

com.RealDeals.tablet failed to scene-update after 10.00s

Elapsed total CPU time (seconds): 2.900 (user 2.900, system 0.000), 14% CPU

Elapsed application CPU time (seconds): 0.664, 3% CPU

Filtered syslog:

None found

Log 4:

Incident Identifier: 05320FA3-4935-4524-9848-957123C5AB11

CrashReporter Key:   4b6e6a5c7a0dd42aa5f46933ea8e9e76f089bbac

Hardware Model:      iPad3,3

Process:             RealDeals [550]

Path:              

/var/containers/Bundle/Application/BC8FACEE-BC28-4680-A614-781C66BCD98C/RealDeals.app/RealDeals

Identifier:          com.RealDeals.tablet

Version:             1.0.0 (1.0.0)

Code Type:           ARM (Native)

Parent Process:      launchd [1]

Date/Time:           2017-09-05 14:42:56.56 -0400

Launch Time:         2017-09-05 14:41:41.41 -0400

OS Version:          iOS 9.3.5 (13G36)

Report Version:      105

Exception Type:  00000020

Exception Codes: 0x000000008badf00d

Exception Note:  SIMULATED (this is NOT a crash)

Highlighted by Thread:  0

Application Specific Information:

com.RealDeals.tablet failed to exit after 5.00s

Elapsed total CPU time (seconds): 2.000 (user 2.000, system 0.000), 20% CPU

Elapsed application CPU time (seconds): 0.359, 4% CPU

Filtered syslog:

None found

Log 5:

Incident Identifier: 99F1A75A-C0F5-4B8C-ABF2-0FB5A82DCFCB

CrashReporter Key:   4b6e6a5c7a0dd42aa5f46933ea8e9e76f089bbac

Hardware Model:      iPad3,3

Process:             RealDeals [511]

Path:              

/var/containers/Bundle/Application/57E26CED-D184-4F41-B056-7B7E39E24011/RealDeals.app/RealDeals

Identifier:          com.RealDeals.tablet

Version:             1.0.0 (1.0.0)

Code Type:           ARM (Native)

Parent Process:      launchd [1]

Date/Time:           2017-09-05 13:29:56.56 -0400

Launch Time:         2017-09-05 13:28:25.25 -0400

OS Version:          iOS 9.3.5 (13G36)

Report Version:      105

Exception Type:  00000020

Exception Codes: 0x000000008badf00d

Exception Note:  SIMULATED (this is NOT a crash)

Highlighted by Thread:  0

Application Specific Information:

com.RealDeals.tablet failed to exit after 5.00s

Elapsed total CPU time (seconds): 2.160 (user 2.160, system 0.000), 22% CPU

Elapsed application CPU time (seconds): 0.358, 4% CPU

Filtered syslog:

None found

[Colin Holgate]

There are various ways to make an app crash. Here are two examples:

1. Having too many unique graphics and setting render mode to GPU. GPU does give amazing performance, but if you're using many different images, like you might with an animation, the GPU can fill up quickly. Using Direct for render mode solves that, but might lower the performance in some cases.

2. Not forgetting things. If you did this:

var mc:MovieClip = new MassiveLibraryMC() as MovieClip;

addChild(mc);

mc.addEventListener(Event.ENTER_FRAME,updatemc);

//then later:

removeChild(mc);

var mc2:MovieClip = new AnotherMassiveLibraryMC() as MovieClip;

addChild(mc2);

mc2.addEventListener(Event.ENTER_FRAME,updatemc2);

and so on, the first MovieClip would still be in memory. Doing that a lot of times could lead to a crash. Something like this would solve that:

mc.removeEventListener(Event.ENTER_FRAME,updatemc);

removeChild(mc);

mc = null;

[William Spence]

Thanks for the reply Colin.  I don't think that the problem is calls to my server, or any overloading of my server, first because I am the only one using it, two because I purposely limit the amount of data that I request, and three because I am using AMF so it is fast.  Garbage Collection is keeping the memory in check, and it isn't crashing on Android, so I don't think it is a memory leak. From the crash logs, it appears that something is taking too long for whatever reason. There are only two things that I feel it could possibly be if it is timing out or something.

1) This is a real estate app, and it uses a webview to display a google map in it.  I use markers in the map.  I wouldn't have thought that it was the map because it is not locked up or unresponsive when the app crashes, and I thought that all the calls were asynchronous.

2) When a user clicks on a property, up to 30 images are downloaded and cached for display.  I do not supply the images, they come from the local MLS. Perhaps they are not downloading in a timely fashion or something. I am not sure what my options are here.  I HAVE to have the images.  Perhaps i could fashion some sort of timeout on image loading, and if it fails, recall the load until the image actually finishes???

So the map and the image loader is where I am going to start, because I really doubt that my AMF backend service calls are the bottleneck.  But I am very intrigued by your idea on RenderMode being set to GPU, because I do have it set to GPU because of performance like you said.  I wish there was another way to improve performance and clear images faster so that GPU was not an issue, because it REALLY helps my app.

Let me know if you have any other ideas!

Thanks

[Colin Holgate]

As most things you're doing are in web view, I would make sure you're up to date with AIR (the later versions have a much better webview), and the render mode setting shouldn't matter too much. Try Direct.

[William Spence]

Actually, the only things that are webView-based are the Google Map, and account authentication to log into a users account.  All of the other real estate activity is done through RemoteObject AMF calls to obtain listing information, and image downloads.  So for me, I feel that 25% of the app is webview based.  I am also using MyFlashLab RichWebView ANE for the webview. I am currently running AIR 27.

Thanks for your advice, I will try the direct RenderMode (although I know I am not going to like the performance hit )

[William Spence]

I am running my app in Direct renderMode and so far, have not experienced any crashes on iOS.  But there is a noticeable performance hit. I am going to run it on a better network tonight and while profiling in Scout.

I would absolutely love to keep my app running in GPU renderMode. I don't think my app is failing due to a memory leak or faulty coding yet.  I think that the GPU is running out of processing memory due to all of the images my app processes.  So here is a follow-up question.  How effective is forcing garbage collection, and does it effect the memory in the GPU as well???

[Colin Holgate]

I remembered something that can help enormously. If everything people are seeing is bitmaps, and not vectors, you could set the stage quality to low, and the app would look identical. If you are using some vectors that would look poor with the stage set to low, make those be bitmaps instead.

Literally this single line can fix GPU rendermode crashing:

stage.quality = "low";

I don't think system.gc(); will help GPU, but it could help normal memory issues.

[William Spence]

Thanks for all your help Colin, very insightful.  Unfortunately, after a lot of experimentation, even with the stage quality set to low and the rendermode set to GPU, eventually, the program would lock up on an older iPad 3.  The only thing I have done that has "guaranteed" no crashes or lockups has been to set the rendermode to "direct".  Bums me out, because I really can notice a performance hit when scrolling my lists that contain images and a few other things, but I think it is much more important that an app works and doesn't break than to look super duper slick.  Wish I could have both HaHa!! It runs pretty fast on a new Android, and would probably look great on a newer iPad Air, but clients with older tablets are going to notice it I think.

[Lars Laborious]

If your are tweening text blocks (you mentioned scrolling lists), you should look into Greensock's BlitMask, if you aren't already using it.

[William Spence]

Lars, thank you for the suggestion.  I have never used BlitMask before.  It sounds very interesting, but I have a feeling that it won't help in my situation.  My ItemRenderers on my lists have to dynamically download Real Estate images, and create buttons to click on as well as to display text, so I am not entirely confident that blitting would help in this instance.  Do you know if it would work under this scenario??

[Colin Holgate]

One of the nice things about Direct mode, other than if you're doing Stage3D or Starling, is that it does do fast blitting. If GPU is not an option for memory reasons, Direct is better than CPU. I ended up just settling on the fact that older devices would scroll more slowly, but at least my app would work.

Here is an example Direct rendermode app of mine, that has a lot of unique animation (therefore would crash GPU rendermode). If you find your way to the credits screen you will see the not ideal scrolling:

Meet Heckerty on the App Store

BTW, I do use GPU rendermode whenever I can. Here's video of another app, which was GPU:

Amazon.com: Fireman Sam Hero Adventures: Appstore for Android

Click the watch video link. The video is a screen recording, a real device was smoother.

[Lars Laborious]

William, I'm not entirely sure, but as long as content are downloaded before you do the scrolling, BlitMask should work. It works well on dynamic text since it is vector-based.

[ASWC]

Blitting does NOT work on mobile device with classic display list because all bitmapdata (drawing) operation are way too slow and blitting is all about that. On the other hand in direct mode using stage3D then blitting (which is really not blitting) would work fine but will be quite irrelevant in that case.

For the user: If you think your app is running out of processing memory then it should be easy to find out and direct mode would not help that much with this. I doubt very much that is the problem and setting your app to DIRECT might seem to work for some reason but will later turn out to be no more than a bandage on a wooden leg. I run on low end device android/ios apps in GPU mode that have nearly 500M of graphics, how much more can you really have that would be the sole reason for a crash? Again I advice skipping the "easy fixes" and other "miracle solutions" and track down manually the real cause of those crashes. Follow the plan I outlined in prior posts, fix the real problem and publish your non crashing app in GPU mode as intended.

[Colin Holgate]

Some of what I said above wasn't quite right, or I hadn't remembered what I knew long ago.

I wrote about this nearly six years ago. If what I said then is roughly right, Direct rendermode works well if not too much of the screen is changing. A full screen scrolling list wouldn't be ideal, but a narrower one should be better than if you used CPU:

RenderMode (gpu) or (direct) - Can someone explain ?

[ASWC]

CPU is out of the question on mobile of course but the question is still about a crash and personally (especially since I do use GPU mode in a few massive mobile apps) I think fixing a crash by changing the display mode is highly suspicious, surely not the kind of fix I want to explain at my weekly app meeting. I really don't see the memory being the source of crash especially since there are so many other ways to crash an IOS app with AIR. Crashes must be investigated and resolved, anything else is finger crossing and usually doesn't provide reliable results. It seems all those known potential crash problems have been discarded and dismissed for convenience.

[William Spence]

ASWC, thank you for your input and ideas on my problem.  In the end, you may be 100%f correct.  But I do have to say that I am a tad bit confused by all of this.  For mobile, Adobe does strongly recommend using Direct mode.  On my newer Android device, I have never had a crash in either Direct or GPU mode.  On iOS, I have an older iPad, and it has ONLY crashed in GPU mode, never in Direct mode.  Since Adobe recommends to avoid GPU on mobile devices, I would hate to go through an enormous debugging process when it could possibly turn out to be the case that my app has nothing wrong with it and that an older iPad just can't handle it.

At this point, my app is enormous.  It is a real estate app that loads and displays a ton of images, a lot of scrolling lists, a lot of AMF RemoteObject database calls.  But I am not a super advanced coder, I do not override a lot of functions or create my own classes, so nothing fancy, I just stick to the basic rules.  But the biggest problem is, it absolutely throws no errors when it crashes, and it takes around 20 minutes to make the app crash, and it never crashes in the same place in the application twice.  So trying to figure out what the problem is, IF THERE IS A PROBLEM, could be a huge time-consuming undertaking.  I wouldn't even know where to start.

You did lay out a nice plan in an earlier post, so I may possibly take that route because you appear to be a much more experienced programmer than I, and GPU would be a huge improvement.  But stability for my clients has to be my priority.

Thanks again!!!!

[ASWC]

Is this is the result of your investigation then so be it, if I'm saying anything is that you need to investigate deeply and be sure that whatever solution you come up with is a definitive fix. If setting your app to DIRECT mode is a definitive and satisfactory fix then that's what it is. My personal tendency would be to run some investigation in my code (put some traces, load/unload views, ect), try to find some patterns for the crashes, ect cos if I like the way my app runs and works in GPU mode then I'll probably want to stick with it as much as possible. It's no question it takes a lot more time and effort but sometimes you narrow down the problem, find it and destroy it and it makes you very happy!

[William Spence]

I would LOVE for that to turn out to be the case.  I would love to find a bug and smash it and continue to use GPU mode.  I will look into this further when time permits.

But I do have to ask, was GPU intended to be used on mobile devices? Is it possible that older mobile devices just have GPU's that were not powerful enough, or did not have proper support by AIR developers to run properly?  Just a thought.  I have never seen an Adobe AIR team member chime in on this GPU/Direct issue on any of the forums.

[ASWC]

It was/is the case, the only problem I never faced so far (which doesn't mean is not a real problem) is crashes due to GPU mode. I couldn't run in GPU mode in Ipad1 and only started doing it with Ipad2, Ipad1 one was way too slow in this mode while a little bit faster in DIRECT mode (using Stage3D), Ipad 2 was better (for the type of application I'm doing) with GPU mode and since then all Ipad/phones have been very good.

Android was better in GPU mode since I started mobile apps so not really anything  to say there, DIRECT mode was sucking CPU in no time with those android devices and you couldn't run anything of value. Android device have better CPU now so DIRECT mode gives a better result/perf depending on the app type.

So yes it's possible you found a typical crash due to GPU and nothing else, all I can say is that I never run into that problem.

[Colin Holgate]

Yes, GPU is for mobile, you can't even select it on desktop. At one point Adobe were trying to push Stage3D a lot, even before Starling was far along, and there were rumors of them dropping GPU. Fortunately that didn't happen.

I think things are as simple as this:

Starling and GPU rendermode are doing the same thing. With Starling, you have to do all the hard work, with GPU Adobe did the hard work.

In both cases you can crash the device if you use too many unique textures. In the case of Starling, where you have to do all the hard work, you're more aware of how much textures you're creating, and so will blame yourself when there are crashes. With GPU, it's all so easy, and you can forget that all that text you're showing, and all those unique vector animations, add up to a ton of GPU memory.

If you thought about it enough, and optimized all of your graphics, you could make GPU mode not crash so soon. But your animation might not be as interesting.

One advantage Starling has over GPU is draw calls. If AS3 could use sprite sheets and atlases like Starling and HTML5 Canvas can, that would help GPU performance a little bit.

[William Spence]

Very interesting.  You have told me a lot I did not know about.  I wish I could try my app in Starling, unfortunately, I use a whole lot of MXML and not just actionscript.  I don't know anything about Starling, but it is my understanding that I would not be able to use it for this reason alone.

Even in GPU mode, I have never had my app crash under standard use. I really have to stress it.  It is a real estate app, so if I just constantly do searches, pull up properties, and scroll through the images as fast as I can, and don't pauses between all of these actions, eventually the app will lock up or crash.  So I have a feeling that it has something to do with image processing memory being stressed beyond an ability to recover that causes the issue. Why Direct mode is able to handle this better, I have no idea, because I have such a poor understanding of how these processes and hardware work. I thought that forcing garbage collection between the states where you search for properties, and ones where you actually view the images, would be helpful.  But I have found no definite ways to reliably force garbage collection, and I have no idea if garbage collection actually affects GPU memory usage as well.

My image loaders always cache the images that I want to display with a maximum limit to these caches, so it should never cache more than 100 property images in the scrollable property list, and never more than 50 in the actual image browser PopUp I created.  In using Scout, it looks like my app stays within acceptable memory limits, but scrolling through all of the photos really fast is what seems to usually cause the problem in GPU mode, and my hunch is that it is the GPU that eventually cries foul. I just have no idea what I could do to try and ease the stress I may be putting on it.

[Colin Holgate]

Just on the question of MXML and Starling, I don't know much about it, but there is Feathers. It appears to be a go between that lets you use MXML and Starling:

https://feathersui.com/help/sdk/getting-started-mxml.html

[ASWC]

As far as garbage collection goes you have extra control with bitmaps and bitmapdata since you can directly use dispose() on a bitmapdata instance. I of course do it constantly in my app to free that memory asap, I also have very few to no vector graphics at all in my apps. If you have vector animation in GPU mode then AIR must create cached bitmap version of all frames and that's super expensive. Now I can see why on Ios if AIR is trying to create a tons of cached bitmaps in a very short period of time and not release any memory that could get a bad memory access eventually and crash. Ios is just not gonna keep giving memory without complaining, when it complains your app gets no memory anymore and might crash. The vector animation example would not create those cached bitmaps for all frames under DIRECT mode and I can see why in that case the app would not crash as easily.

[ASWC]

I do use GPU mode a lot and never had any specific problem with it let alone crashes.

[ASWC]

The list of what can cause crashes on Ios is long, no it's not always memory or CPU related but those need to be looked at first.

If developing from a windows computer then file names is another source of crashes.

Incorrect key in Object or Dictionary.

Removing events that doesn't exist.

And more. And yes Android will likely be fine with all this, moreover an ad-hoc Ios might even be fine with it.

How to track it down?

Remove, assess, add back. If you remove a view and get no crash anymore, the problem is related to that view, publish with that view only > still crash investigate deeper, remove components, events, ect ... track down the problem from views to components, methods, ect ... From top to bottom.

Memory or CPU related? make your app use more memory and more CPU, create twice as many views > crash sooner > you just got a clue. doesn't crash sooner > memory and CPU might be fine.

It's an investigation, it takes time, be organized, logical and you will find the source(s) of the crash.