When I export my PNG image from Photoshop, even at gargantuan proportions, then import it into Dimension, it always looks like a 400 x 300 pixel image:
Ideally I would prefer to not use a rasterized image, but that's the only one that will even work. When I try to import the PSD I just get a broken image icon:
Your online documentation states that SVG images are an option, but not on my install (Win10/64). I only get one choice of "jpg, png, psd" or another of "exr, hdr".
Apologies for the issue in the documentation. .SVG support did not make it in just yet.
In regards to the resolution, there is some behind the scenes work that is happening. Essentially the materials that are on the base model control the size that the decal is rasterized at. So even if you load a 6,000 x 6,000 pixel decal, if your material on the object is 1,000 x 1,000, it uses the base material's size. There is also a maximum of 2,000 x 2,000 pixels for decals.
We are working on removing these limitations. We will support .svg import, and we will render the decal separately from the base material so it can use it's own pixels.
However, as another note, the canvas doesn't have the best results. There's a lot of jagginess with aliasing that happens. Be sure to look at the render preview which will have a much smoother result. It might not be as bad as it looks there in the canvas.