That's how threaded text frames work in InDesign
Auto-flowing of text between columns is not available in Photoshop. You'd have to set it up manually.
Do you have access to InDesign for laying pages out?
Yes. As said above, Photoshop is not best suited to that sort of thing. Microsoft Publisher is easy to use, and something that is installed on a lot of computers. If you only have Photoshop, have a look at GuideGuide which is a free Extension that helps you set up columns, margins and gutters.
I don't know what your final output is going to be, but whatever it is, don't flatten the type layers. That would reduce the type to bit map (pictures) which would be blurred and difficult to read. Best option is to Export to PDF using a high quality print preset.
Well, best option is really use InDesign or Publisher, but if you don't have those installed...