FAQ/Tracker to Masks
From VFXPedia
Basically, you can connect your masks to a Background node in which color is set to solid white.
Then feed it to your Tracker's Foreground input,
Set Tracker.Operation = Match Move
and Tracker.Merge = FG Only
The Tracker's image output can be used as a bitmap mask
But in some cases this method can produce not acceptable results.
Masks are clipped by frame borders and their quality worsens with significant up-scaling
To avoid all that, you can destabilize masks themselves by connecting their Center, Size and Z Rotation to your tracker data like this:
Center >RMB>Connect To> Tracker Unsteady Position
Scale >RMB>Connect To> Tracker Unsteady Scale
Z Rotation >RMB>Connect To> Tracker Unsteady Angle
Thus, tracker animation is applied directly to vector objects, and you get rid of excessive resampling.