Are you by chance using Robohelp for Word, rather than Robohelp HTML?
No we are using Robohelp 11, with our project's being generated as Microsoft HTML help (.chm) files.
Sorry, I assumed you were evaluating RH11 and hadn't yet moved to it. Were you using Robohelp for Word previously? I don't ever remember seeing the setting before and I've used RH HTML on and off since X3. However I've never used RH Word. I searched, and the only reference I could find was in a Robohelp for Word (v7) topic. It's about reports rather than the setting itself, but says that the setting is only for WinHelp 4.
Maybe what you need is in the table of contents properties. Open the TOC, right-click on a book or page and select Properties. Then on the Advanced tab there are Window and Frame options.