The initial loading of previews does take some time however, once the previews are generated then the scrolling would show all the previews.
Also, the speed of generation of previews depends on the type of previews that you chose while importing
Refer this article : https://helpx.adobe.com/lightroom/kb/optimize-performance-lightroom.html#topic-2
Also, if you have Lightroom 6, you might want to turn off graphics processor under Lightroom Preferences > Performance and then, monitor the process of previews generation.
Try putting the catalog file and previews on your internal disk, see if that helps.
Try rendering 1:1 previews and keep them indefinitely.
Forget about smart previews, which are irrelevant to this problem.
Thanks - I'll move the catalog to the internal disk, which is SSD. That should help quite a bit. Will also try 1:1 previews.