Many ways to do what you want...
easiest for you maybe...
Bring the clip in via the source monitor and drag video only to the timeline using Video icon only.
Apart from that...you can un target the audio timeline and drag in the entire source frame.
or...you could map the audio of a clip(s) to no audio.
Then you could ALT-Click the audio part of a clip in the timeline and hit DEL
Craig's suggestions are probably all the best for accomplishing what you want, but I'll offer one more suggestion for the fun of it...
Personally, I like to add both the video and audio portions of a clip into a sequence, because even though I may be pretty sure that I won't use the audio, I like having the option. Instead of fussing with turning off tracks and so on, I simply hold down the Alt key and select just the audio portion of a sequence clip (or just video, if you want the audio) and right-click and toggle the Enable option. This leaves that portion of the clip in the timeline, linked to the other portion, but effectively "mutes" it. This way, if I decide I want the audio (or video) later, I just have to reverse the process, and I don't have to turn off everything in a track by muting it, so I can be selective of the clips that pass through to output.
Again, this is a more specialized function, but might be helpful at times.
I most often use Colin's method, with a tiny twist. As I use Audio Mixer so much, I just hit Mute for that Track, 'cause I'm already there, and am lazy.
So many ways, so many ways.