The speed depends on the shot and the video format. It is very memory and CPU intensive. I've had good results with 4K footage when I correct for rolling shutter errors and render a DI as an image sequence or as a production format movie. MP4 video is always harder to process for this kind of thing.
Sorry about the laggy behavior. Are you still experiencing this problem? Please let us know if you are and need additional assistance.
I have this, and I still have this. In After Effects CC 2014 (before the new preview-code, let's call it 'old skool AE CC') it was very workable.
You could select points, delete it, scrub forward half a second (or a second) or so, delete points, and quickly go through a clip like this.
('Auto-delete points in time' only gets you so far, new tracking points appear around the video anyway).
Since the new preview-code (I believe it was AE CC 2015) the process is laggy as hell. Not the selecting of points, but the moment you delete a bunch of tracking points AE seems to freeze for a full two seconds or something, then when you jump to a new position in the timeline it freezes again for a good second or two, and then the interface is fluent again and you can select points and remove.
This makes the 'select the points, delete, jump forward a bit, repeat' way of going through a clip pretty much unworkable, and is actually the reason I don't use the Warp STabilizer that often anymore.
This happens on Prores clips, DNXHD clips, all 1080p25 and even 720p25. It seems the type of clip and the codec have no impact on this.
In the patch notes a few minor versions later (AE CC 2015.1?) something like this was mentioned as being fixed, but it never was fixed for me. This was pretty much perfect responsive and fast in CC 2014 but slow as hell since CC 2015 and after. All systems I experience this with are Core i7 3ghz+ machines, one with an older but heavy AMD card, others with way newer nvidia cards.
It seems I got a bit more responsiveness out of this (a _very_ small bit) by enabling 'show track points' but then disabling the Warp Stabilizer plugin. The track points still show, and you can still manipulate it. But because the plugin is disabled, it won't constantly recalculate it's stabilization after deleting some points. After I'm done, I can enable the plugin again and disable track points to see the effect. But still, even with the plugin disabled (read: the track points are still visible and its possible to edit them this way) deleting some track points causes a freeze for a couple of seconds in the AE interface.
This is still the case with AE CC 2017
I'm running into this laggy problem even with 64gb RAM, i7 quadcore, GEForce graphics card. The video I'm working on was created with a dynamic link to Premiere Pro. I have the effect turned off with "Show Track Points" selected, and it takes over a minute to switch between frames in order to delete the track points. Frustrating for sure.
The footage is .mts files