I don't think so having an older version of the application will cause Lightroom to take time to load.
Also, I would suggest you to update Lightroom to the most recent version: Keeping Lightroom Up-to-Date
To be specific does the application takes time to load or does the computer takes time to boot?
>Could this be because Lightroom 4 is still on my computer.
No. no part of Lightroom 4 is loaded at boot up. If you experience long boot times it is some other thing that is installed. El Capitan should generally boot really quickly.
>If so, how do I remove it?
Go to your applications folder. Find Lightroom 4. Drag it to the trash.
The application is very slow to load.
Oh you mean application startup! not boot up. No that is not impacted by Lightroom 4 being present. Lightroom 6 is a lot slower to load than Lightroom 4 in general.