When you go to record using Captivate you have several options that govern what Captivate does. One of the options will scale whatever window you are recording to fit a specific dimension. But this depends on a few different factors. For example, the application window has to allow scaling.
Definitely Captivate is capable of adjusting the browser window to fit within the recording area, but when it does this it's the same as sizing the browser yourself to that size. If the web pages that are presented inside the browser are wider or longer than the area you will see scroll bars.
Another possibility is to record at full screen, then use Project > Rescale to reduce the recorded size to fit your dimensions.
Helpful and Handy Links
Thanks so much for your quick reply! It was very helpful.
Another option is that after you've selected record in Captivate and you are now looking at the web app you will be recording. Click the page, then press F11. This will remove your Explorer icons/toolbars completely (hides them), along with your system tray at the bottom of the screen (as opposed to the side of the screen). After you are done recording, you will automatically be returned to Captivate. If you end back up in the application window and no system tray appears, just press F11 again to unhide them. This allows for a lot more screen space to be recorded and you don't record which applications are open or have to fuss with resizing your Captivate capture size to the oddball numbers. You can record in straight 1024 x768 or other size.