This is very hard to diagnose without details. Basically if you are working with video you should have no lines that are thinner than 2 pixels, 3 is better, the lines should be lined up as precisely as you can to the pixel grid, you should avoid very thin highly saturated lines especially against solid colors because they don't compress well. A PSD should be created at a frame size that is sufficient so that the images never have to be above 100% scale or closer to a camera than the zoom value of the camera. Size is important so lets talk about that a little more. If you have a PSD file and you want to move in on the image to focus on the center 1/4 of the image and your comp is standard HD 1920 X 1080 you would have to have the center 1/4 of the image 1920 X 1080 pixels. That means the entire image would have to be twice that big. Divide a square into quarters and each quarter is half as high and half as wide as the original. If you wanted to move into the center 1/8 screen then the PSD would have to be 4 times as wide as an HD comp... That's why I do most of my illustrations in Illustrator. You can scale vectors to any size needed and Photoshop lines and shapes fall apart very quickly. A photo of a face or a field of wheat can usually be scaled up to about 150% without too much trouble. A thin line that is a bitmap (PSD layer) cannot be scaled much more than 110% without starting to completely fall apart.
Now a little more on Photoshop files for video. PSD and AI files for video should always be RGB so you can predict the colors more accurately. The latest versions of AE will handle CMYK but the files can be buggy. 16 or 32 bit RGB files are even better.
Layer groups will show up as nested comps in the main Photoshop comp created when you import as comps. Not all layer effects applied in Photoshop will be successfully imported into AE.
A screenshot of your comp with all modified properties of the layers giving you problems would help. A screenshot of at least your layers panel in Photoshop would also help. Even better, a dropbox link to the PSD would help us figure out what's going on.
I took a look. You are using shape layers for the corners. They come in correctly as masks. You should import the PSD as a composition retaining layer sizes then open the composition. This is what you will get:
I would have done the whole thing in Illustrator but in this case Photoshop will work just fine.