Instead of Time Stretch, you can use Time Remapping. A normal linear keyframe will take the speed down to 0, and you can ramp it down with Graph Editor.
But I have to warn you: AE can't work miracles changing the speed of a given frame rate. I hope it works out okay for you.
The pre-comp is 100% graphics with movement.
So I would reset the time stretch back to normal and do everything with the Time Remapping, correct?
That's right. Get rid of time stretch. Time Remapping will set a keyframe at the beginning and end of the pre-comp. Now set another keyframe where you want to end the speed up. Select the second and the last keyframe and drag them both to the left until you've stretched the length of the first segment to where it needs to be. Now just move the last keyframe closer to the second keyframe to adjust the timing of that section. Verify your speeds with the Graph editor and fine tune the keyframes and add any easing you need to.