Why are beta releases older than production versions?

[Mr_J_Rey]

The version currently available through both the beta release direct download & auto-updates is 20.0.0.255 for NPAPI but the production current release version is 20.0.0.267 so why exactly is the comparable beta version number less than the same production release or is everything just out of sync?

Comparing the beta release notes‌ currently available to the production release notes shows that the beta fixes up to Nov 18, 2015 went to the production release for Dec 8, 2015 and the beta fixes up to Dec 22, 2015 (same date as latest beta) went to production on Dec 28, 2015 which the version for that date was 20.0.0.267. But there is no history of the previous beta release notes & what's currently available in the beta release notes doesn't even show the current beta version number since all it has is for 20.0.0.230 (which doesn't match the latest date) so its pretty sketchy on being able to determine why the beta version number scheme has dropped noticeably below the comparable production versions.

Another issue with them being further out of sync is that the production ActiveX version 20.0.0.270 was released on January 1st with no updates to the beta download pages, release notes, etc. I'm using too new of a version of Windows to verify what version for ActiveX is actually available for download although it appears to be as old as the other releases. Oh and this brings up another issue. The main beta page has the date of Dec 22, 2015 but the beta download page has the date of Dec 16, 2015 so they are not reliable either. So it appears that even if you ignore the number mismatches that the latest beta file isn't as new as the latest production version which is even more absurd.

All of these inconsistencies are seriously making me want to ditch the beta programs even though I've found & reported a couple crashes in the last year if they can't even keep track of which is what. Quite unprofessional for very popular software & a large company. Hopefully this will trigger some kind of audit to triple-check that all files, documentation, points of distribution & everything for both release channels are completely in sync. Please either explain or if not then at least just fix it all so things aren't such a mess.

[Colin Holgate]

This slightly confusing situation does happen with both Flash Player and AIR. When the build changes from beta to release, only the release site is updated, the beta site keeps the latest beta. There was a release version a few days ago, and so until there is a new beta you get the situation where the release version is later than the most recent beta.

[jeromiec83223024 · Accepted solution]

Our priority, especially with out of cycle releases (emergency patch releases for security and functional escalations), is to get the ~2.5Bn people using the release channel into a good state.  Once that's taken care of, we typically try to follow up with a beta release that exceeds the release version within a few days.

December/January has been punctuated with an emergency security hotfix (while the entire company was on vacation) and a couple subsequent patches for functional fallout.  The beta channel has not been a priority target during this time.  The goal in general is to move the beta channel to a higher version than the release version within a couple days, but that's not always possible, particularly when we're operating with a skeleton crew.

We *will* be moving the beta channel forward to the next major version of Flash Player shortly. 

Your point is well-taken on the inconsistencies with the documentation.  I'd love for all of this stuff to be an artifact of the build system, but it's a manual, error-prone process at the moment.  It wasn't that long ago that Flash was on an 18-month release cycle, and beta releases came at 30-90 day intervals.  The team was roughly 3x larger as well. 

We've moved to a modern release cadence and have made huge security investments along the way, but the manual documentation processes persists and are fairly time-consuming.  There's clearly room for improvement, but the reality is that we have to make hard prioritization decisions, and things like beta documentation are secondary to keeping end-users safe and making sure that when security patches in the bowels of Flash have unfortunate, unforseen side effects, that those get resolved as quickly as possible.

We really do appreciate the testing effort that the beta community contributes, along with the huge amount of aggregate stability data that it generates.  The documentation and logistics for all releases (beta and otherwise) fall on the shoulders of a small handful of dedicated, hard-working people.  We're keen to help them out through better efficiency and automation, and we'll try to keep a closer eye on it in the interim.

Thanks!

[Mr_J_Rey]

Okay. I figured it was something along that line going on & glad to hear there's a reasonable plan of action. Thanks for the explanation!

My main issue was that the beta version kept crashing when going to full screen & I couldn't figure out heads or tails on if it was a known issue or fixed or what since reasonable research would be assumed before filing a bug report but I totally lost confidence on being able to do that. I've switched back to production version due to the delayed response here but will reconsider switching back to beta now that there is a response here & updates elsewhere. I may just only use the built-in Firefox crash reporting though at this point but we'll see.... Thanks again.

[jeromiec83223024]

Honestly, it's the aggregate crash data that's the big force-multiplier.  There are several million people running the beta, and we (and Mozilla) notice when a new crasher shows up that hasn't been seen before.  The vast majority of those people never file bugs or post on the forums, but just that act of submitting crash reports is really useful in preventing crashes that are bad, but too infrequent to see in lab conditions.  As strategic as we are with our testing approach, there's just no way that we can generate the level of activity that our beta audience generates, much less on the same variety of hardware and software configurations.

In this instance, the beta channel totally did it's job.  Mozilla saw the new signature spike immediate, notified us, and we have someone chasing down the fix.  I can't actually reproduce it on my machine, but I was able to pass along the crash reports, and another poster in the forums provided great data.  That said, the crash dumps don't generally tell us *how* to reproduce the problem, so we're working blind.  If we get a bunch of reports of full screen crashing and can correlate the timing to some new crash signature, that really helps, even if the reports aren't super complete.  If we see a few people report a new symptom out of the blue, we take it pretty seriously.  I troll the boards when I have time too (I'm usually watching pretty closely the week after a release), looking for patterns and clusters of issues, as do others.

For bug reports, we're going to triage them further anyway.  We need to figure out what the actual change was that generated the problem, and there are several hundred changes between most beta builds.  It's totally valid to file a bug and just be like "hey, your data sucks, but here's my best guess.".  That's still super useful, and if you ever feel like you're getting the runaround, or you just want to give us a head's up that you're seeing something weird, but don't have the time to comprehensively track it down, you're more than welcome to send me a private message (just click my name).  I'm always happy to have someone take a look and try to reproduce the problem.  It's way easier to fix problems with repro steps than it is working blind from crash dumps, which aren't always useful and require us to wait until more aggregate data comes back to confirm the absence of the crash in later builds..

Thanks!