Your whole procedure probably is completely inadequate and you don't seem to understand how this works. The tool does not linearly replace colors, it shifts them through the spectrum using whatever color modes you have defined. The colors you pick are reference colors that define the offset and compression of the spectrum. If you want direct matching, use the normal Change Color effect. Otherwise simply learn how to extract the regions you want to recolor using channel operations, keying and even masking. "Dumb" effects rarely ever give the desired results. Trying to play it cheap will not get you what you want...
So which tool can linearly replace colours found in the same frame?
If I have a frame with blue and green, and I want to change the green to the blue, which tool lets me identify the green, then identify the blue, and swap the info for the blue into the info for the green? Change Colour does not, as far as I can tell.