From VFXPedia

Despilling Problem: The Red Fringe

Summary

Sooner or later, you'll have to key something like this. Whether it's a motion-blurred red flag, a pink dress or a defocused fruit cocktail in front of green screen. The problem is the fringe that remains after you have pulled the key which is hard to kill using regular color correction tools.

Download the example comp.

Description

• After pulling the key (which isn't perfect, but it'll do for this example) you'll notice a green fringe that can't be removed even by the "burnt" despill method. Try HueCurves or the Color Corrector, you'll also find that the fringe can't be removed properly.
• If you look at the color channels one by one, red and blue look alright. Just the green channel exhibits the dreaded fringe.

• The solution to this problem is to replace the green channel with another one.
• Copy the red into the green channel using a ChannelBoolean.
• Color correct the new green channel, until the hue matches the flag you've keyed. Other colors will of course be distorted, so the last step is to replace just the fringe.

• This can be done by inverting/blurring the original alpha channel, but this might need further tweaking and a matte that captures the semi-transparent areas exactly.
• A quicker way, especially if the subject is moving quickly, is to simply paint a mask of the fringe area to be replaced. Or plain old roto.

--Tilt