When you link images in your InDesign file, the size of the document will be kept small. InDesign creates a low-res preview of a linked image so that performance is not heavily impacted. Now if you have numerous large embedded files, those do impact the document size and potentially, performance.
Here are some articles that should help you out:
Please let me know if you need any more assistance.