I have a video clip that I have masked into a small triangle, then I am adding a kaliadascope tile effect but it is effecting the mask also, so the video is falling outside the mask..
Is there a way to put a mask 'after' the tile effect or will I have to render a new video?
Stock answer no. 1: Pre-compose. Apply the effect in a pre-comp, then use that pre-comp as a layer and mask it out.
This video from the After Effects CS5: Learn By Video series provides a detailed visual demonstration of the render order and how to work with (and around) it.
When you understand the render order, you'll be able to come up with answers like the one that Mylenium gave you on your own.