Yes, for multiple reasons, not least of which is that the 32 bit Photoshop can only access up to 2 GB of RAM (31 bits). Another reason is that the 64 bit instruction set proves to be about 15% more efficient for graphics operations.
So would it make sense to just install 64-bit versions and not bother with 32-bit since I don't use too many plugins?
That's entirely up to you - if you're SURE you'll not need any 32 bit plug-ins then you're probably going to be okay. The 64 bit version does everything.
I personally DO have some 32 bit only plug-ins and I occasionally do use the 32 bit Photoshop for that reason, so I have them both installed.
The only other consideration I can think of is that occsasionally a system might have a problem (with drivers or something) to where the 32 or 64 bit version will work better than the other. Probably your best bet, assuming you're trying to minimize disk usage, will be to install just the 64 bit version and see how well it meets your needs. You can always go back, uninstall, and install both (or the other).
A "thinking out of the box" alternative... Maybe you could consider getting a big new SSD drive to replace the drive you have, which will both kick performance up and give you a lot of extra space. Of course, this costs $$$.