There are multiple issues at play here that impact the performance: the amount of topics, merged help and server performance.
In the WebHelp output, you can select speed optimization options. Try setting that option to 'Local network', even though you are publishing on the web. This option controls whether you have more and smaller wh* files or larger but fewer wh* files. Network speeds have increased in such a way that you can use the Local Network option. And there is an additional benefit: every file that has to be loaded requires a download from the server. And having a lot of downloads for many smaller files is much slower than fewer calls for larger files. That is because for every file on the server, the server must read the file and send it back to the client. If the server can do this with larger files, you can benefit from the improved internet connections of the last 10 years. Having a fast internet connection doesn't fix the issue where you have to retrieve lots of small files.
Secondly, merged help adds a huge overhead on download times. Basically, RoboHelp loads the master project. And then it will do all the load steps for every merged project as well. This slows down the process immensely. This is due to the scripts in merged help, but also because of the issue I outlined before. Cutting down on the number of merged projects by moving it into fewer, larger projects, will also help.
Thirdly, you can cut down on content to speed up searches. For search, the number of topics is irrelevant. What is relevant is the amount of content. When you have fewer topics with more content, the search database will have the same size as more topics with less content. Of course, having fewer topics also reduces TOC entries and that will speed TOC loading. Searching is further impacted by the number of merged projects. Just remember: merged projects means lots of overhead.
For the client facing site, your computer needs to download the files from the remote server. For local help, the browser can access all content immediately from disk. If your server isn't caching, you get a lot of overhead on the server side, slowing things down. And if the server has a slow upload connection (server upload max is your download max), that will further slow the process down. Especially if you have multiple people trying to access the content at the same time, the server has to process a lot of requests. Though for any moderately modern server, the requests shouldn't be an issue.
Thank you so much, Willam! You even answered some question I hadn't thought of! I will try the LAN option next week, and we will start prioritizing the TOC reorganization (we do have a ton of small topics that could be merged into larger ones).
Some results, in case anyone else is struggling with the same thing...
After regenerating the help with the LAN option, I did get about 6 times fewer package*.js files, but unfortunately it hasn't made a big difference in loading and search times. I think our main problem is the large number of merged projects - I used the Developer Tools in Chrome to see what's happening when the TOC is loading or when you do a search, and it looks like there are some "default" files that are loaded every time, regardless of the number of packages (when searching, 4-5 JS files are always called per merged project; I haven't done the math for loading the TOC).
In the end, we have decided to go with an external search (Zoom), while also trying to cut down the number of merged projects to speed up the TOC loading.