It might be that you've added some more stuff to the scene that makes it take up more RAM or you've bumped the resolution up. A preview will cache as much as it can, but the amount it "can" is limited by the amount of RAM in your machine. You can choose to preview every other frame or you can preview at a lower resolution.
After Effects is not like a video editing app where you watch a whole, long thing at once. It works very differently under the hood (I have a long and complicated explanation for this, but suffice it to say there is good reason for it). It's best for working on short shots that will be edited into a larger project in an NLE.
This is true, but this is still a really short time to be rendered I feel and lately I've been getting the same issue that was mentioned by the other poster about needing to allocate more RAM, implying a memory leak.
I have some other project files that have tons of 3rd party plug ins on them, and they are in shorter scene by scene comps so I guess it makes sense that I wouldn't be running into this problem with them at the time.
Is it possible that this is contributing to my low available RAM, despite them being on other projects? Should I de-install/re-install these plug ins in order to fix this if that is the case?
Not familiar with the protocol/possibility here.
Anytime a new version of AE comes out (that is able to be installed alongside other versions of AE), like CC 2015.3, it's a good idea to get new, updated installers for third-party plugins. I'd certainly give that a go.