It's a matter of preference but I like Illustrator because it lets you easily create designs in vector format. Vector format allows you to stretch the design to whatever size you need so you are not limited by the resolution of say a photograph. Again it's a matter of preference but that's where I go. Best of luck!
Start with the printing service you plan on using or the printer you plan on purchasing, find out what formats are supported and go from there.
Both apps are capable of creating vector graphics, however illustrator has a more streamlined approched and better import/export handling of vector files.
whether you go with vector or raster will also depend on what style of graphics you plan on printing. Gradients, drop shadows, etc. are more raster based, line art is more vector based. But they can be blended to a certain degree.
I think most users would say the app to use is the one you are most comfortable using. Because there is some overlap in features between the two apps.
for designing garment & basic patterns it is much simpler in Illustrator.
if you would like to make complex patterns it is recommended to do it in photoshop