You could try to track this (probably with difficulty), but it may be worth considering creating multiple masks on a single adjustment layer with a blur effect applied to it, and animating the various maskshapes. One mask per jersey should do the trick. Set a keyframe for all shapes, move forward 10 frames, adjust, etc... then go back and refine/tween as needed. Keep in mind that sometimes it is helpful to work backwards through time.
TimeRemapper's suggestion is very similar to what I was about to say.
You probably have some rotoscoping ahead of you. There are some good tutorials that show how to use rotoscoping together with motion tracking, including some linked to from these pages:
Creating a bunch of animated masks, and using those to assemble a track matte for an adjustment layer, and then putting the blur on the adjustment layer should work well.