The interpolation will be the method set near top of Preferences > General. (CS6 transforms of pixel layers have a separate interpolation control).
Alright, so I messed with the settings, and with one exception (that being nearest neighboor), they all come out identical
For reference, This is the transformation/warp being done.
This is what nearest neightboor does.
This is what Bilinear, Bicubic, Bicubic Smoother, and Bicubic Sharper all do
Is there anything I can do to preserve the details and quality of the back and left of this?
Please, show us a screen shot at 100% zoom.
can you make a vector grid?
No, I'm using a grid because it's the most straightforward example of the problem, but I'm doing this with other images. I need to be able to get around thsi problem with bitmap layers.
Or I'm completely misunderstanding what you mean, and if so, I'm sorry.
No I misunderstood, thought you were working with simple line drawings better suited to vector art. Sorry for the diversion.
It is a general AA problem with mapping texture on surfaces in acute angles. In 3D programs this is solved by a technique called mipmaping, The 3D renderer calculates the degree of distortions and applies blurring filter where it is needed based on the depth of the 3D space to solve this problem. Unfortunately Photoshop is not a 3D renderer and can't eliminate this. You can try manually to reduce it yourself by applying the same transformation on several differently blurred copies of the layer and then using masks made with manual brushing over the problem areas to reduce the artifact.
Here's an example image of a 3D program solving the problem.
and a quick attempt I made to illustrate the problem using your grid after making it more dense with several multiplied layers. the bottom part is the grid masked with different degree of blurring at the problem areas. As you can see this is not an easy task and the problem may be considered as impossible to be fixed nicely. On top of that as you can see this approach may require constructing grid patterns on layers so that you can blur the eventual horizontal or vertical directions separately.
Click the image to see it larger without the additional distortion of forum scaling