Is there a formula to take a final flattened rgb color and if you know the matte color, to extract the original topLayer color and its alpha?
Suppose i have ffff0000 and the matte is ffffffff and i apply 50% blend, the flattened layer would be a rose.
can i take that rose hex if i knew there was 50 alpha to it and bring it back to ffff0000?
result = (alpha * top) + ((1-alpha) * bottom)
That would, of course, be normalized and rounded for whatever bit precision you are using.
There are 4 values involved. If you know 3 of them, you can recover the fourth.
In some cases (composite against black or white), you can estimate the alpha value and thus get a decent result with only 2 known values (result and matte color).