Use the Optimize PDF command in Acrobat, it gives you more control. I think it's under the Advanced menu in Acrobat 9 Pro and File > Save As > Optimized PDF in Acrobat X Pro.
You might try playing around with different settings. You can also click the Audit Space button at the upper right to see if its from images or other things causing the file size problem.
Unfortunately, InDesign doesn't write interactivity in a very optimized way. You get better results with buttons, for example, creating them in Acrobat Pro and using the Duplicate command to duplicate them across multiple pages.
Thanks. But it burns me that InDesign can't do the job right.