[Fuse] SmoothCam

[Tilt]

SmoothCam_v1_0

This Fuse uses Dimension's motion vectors to auto-stabilize an image or smooth the camera motion.

You first need to analyze your footage by simply playing it back inside Fusion while looking at it in one of the viewers (it's recommended to put this node right behind an EXR loader or an optical flow tool for speed reasons). This determines a transformation matrix by least squares fitting to describe each frame's transformation. If you have moving elements that don't follow the camera's motion you can add a garbage matte. The parts where the matte is black will be excluded by the analyzation.

Check out this intro video on Youtube:

(SmoothCam's GUI has changed since but the basic mode of operation is still valid).


Main Inputs

The following inputs appear on the tool's tile in the Flow Editor:

• Input: Source image with optical flow data in its forward vector channels. Required.
• Matte: A black & white mask that defines the area of interest (white) and garbage area (black) when analyzing the vector channel. Use it to exclude objects that move against the camera's motion. Optional.

Analyze Mode

SmoothCam_Interface_Controls1.png

In analyze mode, SmoothCam will examine its input image whenever it is rendered or viewed in one of the viewports. Just play back the whole sequence first. The console will print status messages whenever a frame has been analyzed. When these messages stop, SmoothCam is done analyzing the sequence.

Vector Detail: The image's vector channel contains motion data for every pixel. To determine the overall camera motion, SmoothCam will scale this channel down to just a few pixels, averaging vectors in the process. This slider determines how many points are ultimately used to calculate the dominant image transformation. Higher numbers will make the algorithm more robust against glitches in the motion vectors.

Reset Analyzation Data: The Fuse saves its data in a hidden bezier spline (it is visible in the spline editor but mustn't be modified manually!). Whenever you want to start over with the analysis stage, press this button instead, which will delete the stabilization data.


Stabilize/Smooth Mode

SmoothCam_Interface_Controls2.png

Once you have finished analyzing the clip, switch to "Stabilize / Smooth".

Reference Frame: This slider defines which frame is used as a reference. The chosen frame will not be transformed. Use the "current" button to set this slider to the current frame.

Smoothing (Number of Averaged Frames): The Smoothing slider defines which operation is performed. A value of 0 will freeze the camera as much as possible. A value of one or more, on the other hand, will smooth the camera motion, removing tiny jitter first and creating an increasingly gentle motion if you increase the smoothing value.

Smoothing allows SmoothCam to handle moving shots where the camera isn't always pointing the same way. Complete stabilization is only recommended for shots that should have been stable in the first place (locked of time-lapse shots or hand-held shots where the camera is locked onto an object).

Zoom: Whether you're stabilizing or just smoothing the camera motion - the image transformation that is applied causes black borders to appear in the image. You can choose to fill them (see options below) but you can also choose to get rid of them by scaling the image up by a small amount using the zoom slider.

Transformation Type: Defines what to render. By default, "Basic" mode is enabled which allows you to select the kind of transformation you want to smooth or stabilize: Translation, Rotation, Scaling or any combination of those three. Enable or disable the checkboxes according to your needs.

Full stabilization, however, requires one more operation to smooth camera motion: shear. Since Fusion's SDK doesn't support shearing an image it's recommended to use a PerspectivePositioner tool to do the job. Set the transformation mode to "Full Perspective (External)" and connected the PerspectivePositioner's corners to the SmoothCam's outputs (right-click -> Connect To -> SmoothCam1 -> Unstable Top Left...). This will stabilize/smooth the image better than basic mode.

The coordinates for an external PerspectivePositioner will be available even in "Basic" mode. However, you can save processing power by switching to "Full" (external) mode anyways to prevent the Fuse from rendering an unnecessary output image (performance increase).

Edges: This option is only available in "Basic" transformation mode. It works like Fusion's Transform tool and defines how the image borders should be handled. Canvas fills with the image's canvas color, wrap will copy pixels from the opposite side and duplicate will fill the borders with the nearest edge pixel.


Further info: Development History and Discussion on PigsFly

[Tilt]

Oh damn, just found out it doesn't work anymore in Fusion 7 due to some LUA issue.

It must be a bug in Fusion's LUA bindings for Matrix4.

[xperhz]

hi,

I am here with another noob question
I am looking for a solution to stabilize a bit difficult motion in Fusion. There is a side moving camera with little handshake, and unfortunately the motion was designed to an exact composition, so it cannot be cropped.
I would like to found a way to create a workaround to reach the effect like Warp Stabilizer in AE or Steadiness in Nuke.
I found this script but unfortunately it does not work with Fusion 8.
Is there any way to do this kind of stabilizing effect in Fusion?
In case the cropping would be acceptable, unfortunately the standard match-moving stabilization will also fail because of the camera moving...

[SecondMan]

Depending on how severe the jitter is, no cropping might not be an option unless you're otherwise capable of reconstructing the image edges.

In your case, what I would probably do is track the shot, stabilise it, and then re-apply the camera motion with the jitter removed. You can remove jitter manually, by removing inbetween keyframes and shaking the splines, or by filtering the spline with an algorithm.

[ShadowMaker SdR]

Can you show us the footage? That would help in giving you some better advice.

[xperhz]

SecondMan: I am not sure what do you mean with "jitter" I haven't met this phrase yet.

I uploaded the raw footage to show you what is the problematic shot.

[Chad]

Other than his hair, this is pretty straightforward. His shirt barely moves, even. With a 3D track you could project (with alpha accumulation) onto some meshes. There's enough perspective change that I don't know if a 2D track would suffice. But it's not the focal point of the shot, so might be passable. HVS is your friend sometimes.

For the hair, might need to hand roto that in from whichever frames match his head rotation best. The wall behind is at least solid.

[xperhz]

in the first step, I only want to remove those little handshake effects, to get a smooth motion as possible, without cropping the left/right sides of the image
do I really need roto to get this done?

[Chad]

If you don't want to crop, yes. That's why cropping is a popular option.

[xperhz]

If you don't want to crop, yes. That's why cropping is a popular option.

oh, I see...
then I'll go with crop. I suppose it should work with tiny crop from the sides, however my first test with Fusion was horrible.
I've set two tracker points, but because of the camera motion Fusion cropped approx 20% from the left side of the picture...

a bit off: what do you think about Resolve's stabilizer? Since I will make the final editing and colorgrading there, it could be possible to use it, if it does a good job and a bit more automated.