There are a couple of ways to accomplish this goal.
The long way around is to select all the topics that are not in the TOC and look at their properties. On the General tab is an option to exclude the topic from any content being found in the search.
Note that with this method, the topic won't be found even if another topic links to it.
The other way is to look at the Single Source Layout options. For WebHelp, it will be found by first clicking to expand the Content Categories area, then clicking to select the category. Once you do this, there should be an option to "Exclude Unreferenced Topics from Output".
Note that using this method, if another topic links to the unwanted topic, it has then been referenced and will still be found via search.
Hope this helps... Rick
Depending on the version of Rh you are using, you may be able to use Dynamic User Centric Help (Content Categories). The user would choose a TOC and then the search is limited to what is in that build, not quite the same as what is in the TOC but close.
See the RoboHelp Tour on my site and look in Version 9 New Features.
See www.grainge.org for RoboHelp and Authoring tips