Third Party Fuses/ROIDS Tutorial

From VFXPedia

Contents 1 Overview 2 Terminology 3 FuRectInt class 3.1 Constructor 3.2 Methods 4 Retrieving the DoD and RoI 5 Code snippets for DoD 5.1 Image creation, no modification of input DoD 5.2 Processing pixels inside the DoD 5.3 Setting a new output DoD 5.4 Transforming the image and its DoD

Overview

This page describes how to leverage the power of DoD and RoI (ROIDS) in a Fuse. Note: It hasn't been written by eyeon themselves, so it might not be completely accurate.

Since Fusion 6.31 it is possible to fully support RoI/DoD in Fuses (previous releases have partial support). The process isn't trivial but there are many Fuses shipping with Fusion that can be used as a template (clBC and clMerge for example). It's not always necessary to implement all aspects of ROIDS. For example, if you don't transform an input image, things remain much more simple. The bare minimum is REGS_SupportsDoD = true in FuRegisterClass(). You should also read up on PreCalc handling.

Here's your path to ROIDS enlightenment:

1. Of the Fuses available on vfxpedia, ELinBC could serve as a simple example (DoD is simply passed through, separate PreCalc handling, all created image objects are based on an existing image's DoD and no per-pixel processing is taking place).
2. RollingShutter also doesn't modify the DoD but this time includes PreCalc handling in its Process() function. It has pixel-by-pixel processing (even using OpenCL) which has to be limited to an image's DataWindow in this case (crash hazard if not done correctly).
3. SparseColor shows how to modify the DoD (clipping to frame borders or setting it to an unlimited size) and restrict processing to the RoI but it doesn't transform its input image. FlareCircle or FlareMulti are also good examples for DoD modification. In these Fuses, the area of a lens flare needs to be calculated beforehand so the DoD can be extended before the actual drawing stage.
4. Overscan and BetterCornerPin transform the image and thus need to request data from beyond the current region of interest. This is only possible since Fusion 6.31. Have a look at their thoroughly commented CheckRequest() functions.

DoD_Demo is an example which draws a gradient inside the DoD and allows you to modify it in certain ways. It spits out some debugging information and is heavily commented to help you understand how Fusion's ROIDS system works. Add it between a Txt+ (which will generates an image with a DoD) and a SetDomain, then play around with those tools.

Terminology

Region of Interest (RoI)

This is the area of an image that the user or another tool is interested in. In a viewer this can be defined using the RoI button and it's displayed by a thin gray line.

Domain of Definition (DoD)

This is the area of an image that contains its pixel data. The area outside the DoD is filled by a solid color called "Canvas". A DoD can be smaller than the image size or reach outside the borders defined by the frame format. In a viewer, this is rendered by a thin dashed line. From a user's point of view, DoD is described on this page. From a developer's point of view, it's the DataWindow (see below) of the image that is returned in a precalc request.

Image.ValidWindow

This property of an image object defines the area that is valid for the current request (specified as a FuRectInt). To set it during image creation use the IMG_ValidWindow attribute. ValidWindow of your output image is usually set to the request's RoI. The actual DoD may be smaller though.

Image.DataWindow

This property of an image object defines the actual dimensions of the image data in memory (specified as a FuRectInt). An image might have a frame format of 1920x1080 pixels, for example, but it might carry less data if a DoD is defined. Pixels outside the area defined by DataWindow must not be accessed (for example by GetPixel) - it might result in a crash. A DataWindow can't be modified once the image has been created which is done by assigning the IMG_DataWindow attribute during image creation. The DoD that is displayed in the viewers is derived from the DataWindow of the output image that has been created during the precalc request. You can return an image with a different DataWindow for the actual render request, but this won't change the DoD of the image. It just creates a mismatch in the viewers.

FuRectInt

This class is used to store the four bounds of a rectangular pixel area. Use the .left, .bottom, .right and .top members to get to the numbers. In proxy mode, these bounds are also reduced accordingly, like the image size. For its methods, see below.

ImageDomain

A helper object that holds not only an image's DoD but all its size, depth and proxy information as well. This is what gets returned by :GetRoI() or :GetInputDoD(). Use the .ValidWindow property to retrieve a FuRectInt of the actual DoD dimensions. There's also an Intersect() method that accepts a FuRectInt or another ImageDomain object plus the :Width() and :Height() methods. Take care, there's also a .Width and .Height member variable, which contains the image width and height!

PreCalc

Before calling the Process() function, Fusion will perform a so-called PreCalc request. This request calls a Fuse's "PreCalcProcess()" function and is usually taken care of internally. You only need to override it for special cases (image size changes, multiple outputs, etc...) and can read more about here: PreCalc and Process.

FuRectInt class

Constructor

f = FuRectInt(left, bottom, right, top)
f = FuRectInt({table_of_four_integers})

If all four parameters are omitted, an empty FuRectInt will be created (internally, all four edges will be set to -1000000). For an infinite DoD (used, for example, when a Transform tool wraps its edges), use the coordinates -1000000,-1000000 and +1000000,+1000000.

Methods

:IsEmpty()    -- returns true if the rectangle is empty
:Intersect(another_FuRectInt_or_int_array) -- returns the intersection with another rectangle
:Union(another_FuRectInt_or_int_array)     -- returns the union with another rectangle
:Inflate(x,y) -- adds x pixels to the left and right edge and y pixels to the top and bottom edge
:Width()      -- returns the rectangle's width
:Height()     -- returns the rectangle's height
:Scale(x,y)   -- scales the rectangle (multiplies left and right edge by x and top and bottom edge by y)
:Offset(x,y)  -- shifts the rectangle x pixels to the right and y pixels up 

You can use print() or dump() to print a FuRectInt's dimensions to the console.

Note: Scale() and Offset() will modify the object in place. Due to the fact that LUA variables are just references, it isn't possible to duplicate a FuRectInt by assigning another variable:

-- both still reference the SAME rectangle!
local datawnd = FuRectInt(10,10,20,20)
local datawnd2 = datawnd
-- to create a copy, use this:
local datawnd2 = datawnd:Intersect(datawnd)
-- or
local datawnd2 = FuRectInt(datawnd.left, datawnd.bottom, datawnd.right, datawnd.top)

Retrieving the DoD and RoI

To make a fuse ready for ROIDS you have to add REG_SupportsDoD = true to the FuRegisterClass call at the top of your fuse. Without it, the requested RoI will always be the full image dimensions.

Note: Adding REG_SupportsDoD makes it necessary to restart Fusion so the Fuse gets reinitialized completely in Fusion's plugin registry. Simply using the Fuse's reload button isn't enough.

If SupportsDoD is enabled, Fusion will provide the desired DoD and RoI values for you via the Request object. Use Request:GetInputDoD() and :GetRoI() to retrieve an ImageDomain object that stores the respective bounding box. You can access the .left, .bottom, .right and .top members of the object which are integer pixel coordinates. If proxy is enabled, these coordinates will also be fractions of the original image dimensions.

The input DoD is taken from the input image: just like the input image brings along with it a frame format and image depth, it also has its domain of definition. If your Fuse combines multiple input images (see the clMerge Fuse), you will have to query and combine multiple input DoDs.

The request's RoI is what the downstream tool (or viewer, if the tool is viewed directly) is interested in.

If you view a tool in one of the viewers without restricting its RoI, you declare interest in the whole image area as defined by the frame format. If you put a SetDomain tool downstream of your Fuse, you're telling Fusion to only look at a specific area from that point on which causes Fusion to tell the upstream tool (your Fuse) that it's really just interested in a subsection of the image. The same thing happens if something is connected to the EffectMask input of your Fuse. The RoI will be no larger than the mask's DoD.

It's up to you, however, to actually limit your pixel processing to the reduced RoI and thus speed up rendering.

Code snippets for DoD

Image creation, no modification of input DoD

The usual procedure to retrieve DoD/RoI and calculate the rectangle that needs to be processed by the current request is as follows (the code assumes that REG_NoPreCalcProcess is true, which makes precalc use the Process() method. The IMG_NoData and "if roi then" statements are necessary to handle the two cases in the same code path). Taken from the clBC and clMerge Fuses that ship with Fusion.

-- get the input image
local img = InImage:GetValue(req)
-- get input's DoD and request's RoI
local dod, roi = req:GetInputDoD(InImage), req:GetRoI()
-- data window, which is the area this Fuse has to process, starts out as input data window restricted to its DoD
local datawnd = dod:Intersect(img.DataWindow)
-- restrict further to region of interest (roi will be nil during precalc and thus datawnd won't be restricted)
if roi then
     datawnd = roi:Intersect(datawnd)
end
 
-- create an output image...
local out = Image({
   IMG_Like = img,                               -- ...that takes most attributes from the input image
   IMG_NoData = req:IsPreCalc(),                 -- ...that doesn't contain data during precalc
   IMG_DataWindow = datawnd,                     -- ...that contains data within the bounds of datawnd
   IMG_ValidWindow = roi and roi.ValidWindow,    -- ...that is valid for a certain RoI
   })

Obviously, this will create different DoDs for precalc than for the regular process request. This is intended. The precalc's data window, which is unrestricted by region of interest, will be the maximum area this tool can produce and will be displayed as the tool's DoD in the viewers. During the process request, the data window may be reduced to only contain the pixels that are actually requested and passed on.

Processing pixels inside the DoD

If your Fuse supports DoD, it must restrict any pixel processing to the dimensions of an image's DataWindow property (that is, its IMG_DataWindow attribute when the image was created). If you write or read Pixels from outside these bounds, Fusion will crash - it won't do any harm, but it's one of the very few occasions where Fuses can cause crashes due to memory access violations.

If you're using the methods of the Image object (like :Gain() or :Fill()) you don't have to care about DoD. These functions will be restricted to an image's DataWindow automatically. Instead, your discretion is required when using :GetPixel(), :SetPixel() or DoMultiProcess(). Here's an example loop, which assumes an image created in a way as described above. A similar example can be found in RollingShutter:

-- calls "mpfunc" for each scanline.
self:DoMultiProcess(nil, { In = img, Out = out }, datawnd.top - datawnd.bottom, mpfunc)

As you can see, the number of chunks is no longer equal to the image height. Instead, the difference between the DataWindow's top and bottom edges are used.

This is dummy code for the multi-threaded processing function. It simply inverts the input image since any further image processing would just complicate this example due to further parameters being involved.

-- multi-threaded pixel function. Parameter 'n' is the current chunk number (zero at the bottom edge of the DoD).
-- receives the following globals in its scope:
-- In: input image
-- Out: output image
function mpfunc(n)
   -- calculate real scanline number
   local y = n + Out.DataWindow.bottom
   -- loop horizontally along scanline
   for x = Out.DataWindow.left, Out.DataWindow.right - 1 do
      local pix = Pixel()
      In:GetPixel(x, y, pix)
      pix.R = 1 - pix.R
      pix.G = 1 - pix.G
      pix.B = 1 - pix.B
      Out:SetPixel(x, y, pix)
   end
end

Setting a new output DoD

To return an image with a different DoD than what the input image has provided, simply assign a new FuRectInt object that contains your desired dimensions to IMG_DataWindow. Many tools in Fusion have a "Clipping Mode" input which can be set to Domain (DoD is unchanged), "Frame" (DoD is clipped/expanded to image dimensions) or "None" (infinite workspace). It makes sense to follow this convention in a custom Fuse. You can look at the code of SparseColor for an implementation of such a button. This is how it's handled in Process():

local img = InImage:GetValue(req)
local clipmode = InClippingMode:GetValue(req).Value     -- as defined in SparseColor example
local dod, roi = req:GetInputDoD(InImage), req:GetRoI()
local datawnd = dod:Intersect(img.DataWindow)
 
-- compared to the previous snippet, this part has been inserted to modify the DoD
if clipmode == "Frame" then
   datawnd = FuRectInt(0, 0, img.Width, img.Height)
elseif clipmode == "None" then
   datawnd = FuRectInt(-1000000, -1000000, 1000000, 1000000)
end
 
-- from here on, it's the same as before:
if roi then
     datawnd = roi:Intersect(datawnd)
end
local out = Image({
   IMG_Like = img,
   IMG_NoData = req:IsPreCalc(),
   IMG_DataWindow = datawnd,
   IMG_ValidWindow = roi and roi.ValidWindow,
   })

Here, the effect of region of interest really kicks in. If no RoI is defined in the viewers, it is set to the image dimensions, which will clip datawnd and thus the created output image to a manageable size even if the user has chosen the infinite workspace option.

Transforming the image and its DoD

Instead of defining a brand new DoD for your output image, you can also modify and transform an existing one. This usually is the case when your Fuse transforms or resizes its input image and creates a problem that the previous examples didn't have to care about: by default, Fusion will use the current request's region of interest to retrieve data from the upstream tool(s). For example, if your Fuse scales the input image down to 50%, the area that needs to be requested from upstream will have to be twice as large. Similarly, if your Fuse shifts the input image 20 pixels to the right, the area that is to be requested from its input has to be shifted 20 pixels to the left. This is possible by overriding the CheckRequest() method which will be called before upstream images are being requested. Have a look at the clMerge Fuse that ships with Fusion for details or check out the Overscan or BetterCornerPin Fuses. Note: This usage of CheckRequest() requires Fusion 6.31 or later.

CheckRequest() will be called based on input priorities, so for this to work you need to assign a low priority to your image input so it gets queried after all the other inputs (e.g. the sliders of your Fuse) have already been processed. The first step is to use the INP_Priority attribute in Create():

InImage = self:AddInput("Input", "Input", {
   LINKID_DataType = "Image",
   LINK_Main       = 1,
   INP_Priority    = -1,	   -- fetch input image after everything else
   })
-- scaling slider, used in the example below
InScale = self:AddInput("Scale", "Scale", {
   LINKID_DataType      = "Number",
   INPID_InputControl   = "SliderControl",
   INP_MinAllowed       = 0.01,
   INP_Default          = 1.0,
   })

The process function remains similar to previous examples. Only this time, the DoD will be scaled by the specified factor:

function Process(req)
local img = InImage:GetValue(req)
local scale =  InScale:GetValue(req).Value
local dod, roi = req:GetInputDoD(InImage), req:GetRoI()
local datawnd = dod:Intersect(img.DataWindow)
 
-- compared to the previous snippet, this part has been added to modify the DoD
if scale ~= 1.0 then
   datawnd:Offset(-img.Width/2, -img.Height/2)
   datawnd:Scale(scale, scale)
   datawnd:Offset(img.Width/2, img.Height/2)
end
 
-- from here on, it's the same as before:
if roi then
     datawnd = roi:Intersect(datawnd)
end
local out = Image({
   IMG_Like = img,
   IMG_NoData = req:IsPreCalc(),
   IMG_DataWindow = datawnd,
   IMG_ValidWindow = roi and roi.ValidWindow,
   })
 
-- this is the part where the image is actually scaled:
if not req:IsPreCalc() then
   img:Transform(out, {
      XF_XSize = scale,
      XF_YSize = scale,
      XF_XOffset = 0.5,
      XF_YOffset = 0.5, })
end
		
OutImage:Set(req, out)
end

To test this Fuse, add it after a Transform tool which scales an image up (so you end up with an overscan DoD). Bring down the scale slider and you'll notice that the image as well as the DoD gets scaled down as expected, but you won't be able to bring back pixels from outside the image borders. This is because Fusion hasn't requested them from upstream since it didn't know about your Fuse's intention to transform the image. To fix this, we'll implement the CheckRequest() function. The following code has a lot of safety-if-statements which make it look more complicated than it actually is. You can find code like this in clMerge or Overscan. The latter is commented extensively.

function CheckRequest(req)
   if (req:GetPri() == -1) and (not req:IsFailed()) then
      local inpdod, reqdod = req:GetInputDoD(InImage), req:GetDoD()
      if (inpdod ~= nil) and (reqdod ~= nil) then
         local reqroi = req:GetRoI()
         if reqroi ~= nil then
            local scale = InScale:GetValue(req).Value
            local datawnd = reqroi.ValidWindow
            -- these are image width and height, not the DoD's width and height
            datawnd:Offset(-inpdod.Width/2, -inpdod.Height/2)
            -- apply inverse scaling factor
            datawnd:Scale(1/scale, 1/scale)
            datawnd:Offset(inpdod.Width/2, inpdod.Height/2)
            -- assign new RoI to input
            req:SetInputRoI(InImage, inpdod:Intersect(datawnd))
         end
      end
   end
end

Using this code, the image that is available to the Process() function will have all the necessary data to transform the image without black borders appearing.