When I free transform and warp a layer to lie flat, sometimes I'll get this harsh pixellation at one end of the layer. It doesn't appear while I'm doing the warp, just suddenly coming up when I hit enter to apply the transformation.
So for example, Ill be transforming this grid, which is originally about 9x this big.
So when I''m running the transformation/warp, it looks like this
Not great, but at least most of the lines are still intact on the left and back (the right side is basically fine looking)
But once I hit enter, I get this
And now like, half the lines I did have are gone, and what's left is super pixelated.
Is there an interpolation setting I could be using? Is it something unavoidable from how distorted I make the layer?
Alright, so I messed with the settings, and with one exception (that being nearest neighboor), they all come out identical
For reference, This is the transformation/warp being done.
This is what nearest neightboor does.
This is what Bilinear, Bicubic, Bicubic Smoother, and Bicubic Sharper all do
Is there anything I can do to preserve the details and quality of the back and left of this?
No, I'm using a grid because it's the most straightforward example of the problem, but I'm doing this with other images. I need to be able to get around thsi problem with bitmap layers.
Or I'm completely misunderstanding what you mean, and if so, I'm sorry.
It is a general AA problem with mapping texture on surfaces in acute angles. In 3D programs this is solved by a technique called mipmaping, The 3D renderer calculates the degree of distortions and applies blurring filter where it is needed based on the depth of the 3D space to solve this problem. Unfortunately Photoshop is not a 3D renderer and can't eliminate this. You can try manually to reduce it yourself by applying the same transformation on several differently blurred copies of the layer and then using masks made with manual brushing over the problem areas to reduce the artifact.
Here's an example image of a 3D program solving the problem.
and a quick attempt I made to illustrate the problem using your grid after making it more dense with several multiplied layers. the bottom part is the grid masked with different degree of blurring at the problem areas. As you can see this is not an easy task and the problem may be considered as impossible to be fixed nicely. On top of that as you can see this approach may require constructing grid patterns on layers so that you can blur the eventual horizontal or vertical directions separately.
Click the image to see it larger without the additional distortion of forum scaling