Neither brown or black are secondary colors. Well, brown could be considered a rather desaturated orange...
Anyway, that's neither here nor there.
You can use the Hue/Saturation effect. Help page on it here: After Effects Help | Color Correction effects
You simply pick a channel and adjust the channel range to only select the colors you wish.
Going to (or from) white or black is nearly impossible no matter what effect you're using and is not recommended.