Welcome to the world of After Effects! As you've noticed, it can be pretty complicated. I highly recommend that you start here. That is a set of resources specifically curated to help people new to AE get started off on the right foot.
To your question, this page in the AE help has what you need. It says to select the layer you want to replace in the Timeline panel then Alt-drag (Windows) or Option-drag (Mac OS) your new footage item from the Project panel onto the selected layer in the Timeline panel.