Try making a Copy to test the Try Flattening the PDF (that puts everything one layer.)
Also check to see if you have multiple Fonts embedded, of the same type. Say for example Helvetica Bold Italic In some cases depending upon the document you might have 4, 5 or more copies if the same font. Remove all but one copy of each font type.
If the PDF is designed for the web make sure jpeg images are set at 150 dpi or lower. For book or Magazine publications you have to leave as is. for reports that will looked at two seconds then chucked in file 13 after looked at 150 DPI is adequate.
Also while in Optimizer, go to User data check Discard user information and metadata.
Make all these settings, then optimize. Then open the optimized file and see if looks like the orignal PDF. If so your all set.