Here's one approach:
1. Use your eyedropper to sample color from the shirt, probably from a dark check.
2. Copy that area of the shirt from the red channel and paste it in as a layer.
3. Put an empty lyer above that, set it's blend mode to color, and paint braodly over the yellow area with the samled color from step 1.
4. Use color range to make a selection of the yellow patch, then with the selection active make a curves or levels layer to add contrast to the yellow area, particularly to darken the checks in that area to match the rest of the shirt.
That will take you pretty far, then you'll need to adjust/blend your edges/transitions by tweaking your masks, cloning or dodge and burning.
This is my results using steps 1-4, the transitions would still need some tweaking:
Edit: sorry the image is posting lighter than it appears in PS. Too late for me to figure out why.