35 second sequence??? A sequence is multiple shots, i'm assuming you mean a 35 second shot.
First question, are you going to use the entire 35 seconds in your final comp? If not, why waste time tracking frames that you are not going to use. Trim the shot. This is generally the best workflow for any VFX project. Most of my comps are seven seconds or less and that's pretty standard for a compositing shot in AE.
If you are going to use the entire shot then first make sure AE is up to date and give us OS and AE version details down to the last decimal point. The latest upgrades have fixed a lot of bugs. If you are all current then I would first suggest that you put your shot in a separate comp with nothing else in the timeline and try the tracking. Make sure you don't have too much of the surrounding areas selected. If that still causes problems and all you are trying to do is to track the perspective changes in the face I would probably use Mocha. It's faster for things like adding scars or other things to faces. There are several tutorials on the Mocha website and on Video Co-pilot that show how to successfully track faces using Mocha.
Here's one other thing to think about. Compositing on a face always works better if you plan the shot before you shoot it. Just taking some footage that was shot with no plan and then trying to do some face tracking is asking for trouble. That's why they hire someone like me, a Visual Effects Supervisor, and put me on the set. It's called spending money to save money.
Did you ever solve your problem? If so, how did you end up doing so?