Double click the book in the TOC in RH and link a topic to the book. On the Advanced tab, select New Window.
Personally I find it a frustrating way of using help as you have to keep closing the windows. However, its your decision.
See www.grainge.org for RoboHelp and Authoring tips