We really can't know. Your info is at best vague, but actually mostly useless. Just assuming there would be an equalyl "common solution" is pretty pointless therefopre. You have to provide exact render settings, screenshots and so on. that said, certain compression methods simply may expose artifacts one way or the other due to how they work, but again, we know nothing here.
Make your comp 1089 X 1920, nest that comp in a 1920 X 1080 comp and rotate it 90º, render that. Now you can burn a DVD or BluRay.
If you are not playing back through a player that will only take horizontal format movies then consider this. The human eye sees detaill much better horizontally than vertically. That is just the way that it's built. It is why interlacing for TV works. Maybe the problem you are seeing is just perception.
The only way to tell for sure is to render one horizontal and one vertical, cut out a little square detail section from the middle and compare the renders.
Other things that could be going on relate to the thickness of the lines, the movement of the elements, the frame rate, or your display's physical layout. Without info and some screenshots we are in the dark.
Thanks for your responses. I realise I didn't give a great deal of detail was hoping there was some common knowledge dealing with portrait compostion exports.
After some trial and error I found that MPEG4 codec doesn't export well in portrait. The best result is using H.264, artifacts disappeared and motion was smooth.