You've got Align to Pixel Grid checked on the transform panel. Therefor strokes will snap to the nearest pixel.
You expand, swap fill and stroke, causing the stroke to move slightly in order to align to the pixel grid, then repeat.
All I do is: expand > shift + x (to change the fill to stroke) > repeat process
And it's the Align to Pixel Grid setting which causes strokes to then shift slightly.