The problem is that the outline of each single character is output to the PDF as a *vector object*. So you now have a document with lots of vector data; and vector data cannot be compressed as easily as a bitmap (JPEG, for example, only works so well because it *throws away* large parts of the data).
In addition, you can see the difference between outlined and regular text *on screen* because these two types of object are drawn in a different way. Outlined strokes are drawn as normal vector art, one straight line at a time; "actual" text, on the other hand, is drawn with super-optimized routines, both for efficiency and speed, and to render as legible as possible. (But -- usually -- the difference disappears when you are outputting on a device with a sufficiently high resolution.)
Removing the outline strokes from the text will help immensely. Of course the text will look slightly different, but only when compared directly to the printed matter. On-screen display will be much better and your PDF size will be a lot smaller.