Try making a Copy to test the Try Flattening the PDF (that puts everything one layer.)
Also check to see if you have multiple Fonts embedded, of the same type. Say for example Helvetica Bold Italic In some cases depending upon the document you might have 4, 5 or more copies if the same font. Remove all but one copy of each font type.
If the PDF is designed for the web make sure jpeg images are set at 150 dpi or lower. For book or Magazine publications you have to leave as is. for reports that will looked at two seconds then chucked in file 13 after looked at 150 DPI is adequate.
Also while in Optimizer, go to User data check Discard user information and metadata.
Make all these settings, then optimize. Then open the optimized file and see if looks like the orignal PDF. If so your all set.
For a break down of bytes and percentages of elements such as Images, fonts, color spaces that make up your PDF you can choose, Audit space usage...' in the top right of the PDF Optimizer dialog box. This will help you identify the bulkier elements that you can work on downsizing.
I often noticed a significant drop of size producing a PS with Print using Acrobat Printer and then Distiller with the settings used in Optimizer.