FAQ/Network rendering

From VFXPedia

When rendering with a network of render nodes using Fusion's built-in or standalone render manager, there are a few things to keep in mind.

All of your Loaders and Savers should use network-accessible filenames, so that the paths to all clips are meaningful to each node in the network. This means they should be in UNC format (e.g.: \\fileserver\sharename\footagedir\projectdir\shotdir\element.####.tga), or they should be using drive letters that are mapped identically on all nodes. (e.g. drive E: on your computer is mapped as E: on every node machine.)

First, the network needs to have a Render Master set up. This can be a copy of the standalone RenderMaster app, running locally or on a different machine on the network, or it can be your workstation copy of Fusion, or any of the nodes, with the "Make this machine a Render Master" checkbox enabled, in Preferences/Network. Its job is to co-ordinate the process of assigning frames in the comp to nodes, for rendering. From the Render Manager window, you can use the Scan command to locate all Fusion nodes (and workstations) on your network, or add them manually. These nodes are referred to as Slaves in the Fusion interface.

As far as setting up a node, all that is necessary is to enter the name or IP address of the Master system into the Network Preferences of each node. This is so they can notify the Master when they come online. You should also do this for your local workstation, so that it knows where to submit comps to be rendered. Alternatively, it is possible to select the nodes in the Slave list within Fusion's Render Manager window, then right-click and choose "Set Default Master". This will set the appropriate pref on each node to point to the name/IP of the Master machine you do it from. (Note: Fusion 5.0 and 5.1 have a bug that causes the node's status to always be set to "failed" instead of "Set OK", even when the operation is successful).

You may also need to enter a network-accessible UNC file path (or path with an identically-mapped drive letter) for the Comps:, Previews: and DiskCaches: entries in the Path Maps page of your workstation's preferences.

Checklist

• All Loaders and Savers must use network-accessible paths that nodes will understand.
• The comp itself must be saved to a path that the manager will understand, before submitting. Setting Comps: in Prefs/Path Maps to a network-accessible location makes this easier.
• If the FUSION_NETSENDCOMP environment variable is set to FALSE on the Master machine, the comp must be saved to a path that nodes can also understand.
• For network-rendered previews and disk caches, Previews: should be set to a network-accessible path, as (for previews) that is used as the location for slaves to write frames to. In the case of network-rendered disk caches, the preview subsystem is used to pre-render the cache, and the Previews: path is used as a location to pass the comp on to the manager/slaves.
• DiskCaches: is used to provide the default path for the image files in disk-based caches. This should be a network-accessible location if you want to use the network to pre-render your disk cache, so that the manager (and possibly slaves) can write image files there, in addition to Previews:.
• Make sure all Slaves have all the plugins and fonts installed that the comp requires.
• If you are reading from QuickTime movies or AVI files, make sure that QuickTime and any necessary AVI codecs are installed on that slave.
• Most multi-frame file formats like QuickTime movies and AVI files cannot be written to simultaneously by multiple slaves. However, if you assign a comp to a single slave, that will work fine.
• If you are using a DPS hardware device, use the virtual file system for Savers, rather than attempting to write directly to the .dps file. By using the virtual file system (shared to the network), the host machine will encode all frames as they are requested by the slaves, and allows multiple frames to be written by different slaves, simultaneously and out-of-order. To save a .dps file, create a place-holder .dps file of the correct length first, and then point your saver to the corresponding virtual file system directory. Note that Fusion is able to read .dps files directly and decode them in software, which spreads out the decoding load among the slaves and reduces the bottleneck.

Troubleshooting

Most network rendering problems may be easily diagnosed by looking at the render.log file produced by the Master render manager, found within Fusion's directory. This will in most cases give details as to the cause of the trouble, and the Verbose Logging option in the manager's Misc menu can be used to control the level of detail.

The normal sequence of events to look for is:

• Comp job gets submitted to the master, and added to the queue.
• As slaves become available, they are assigned to the job, and are given the comp to load.
• Once loaded, each slave is assigned a frame to render.
• Slaves report back success or failure, and may be assigned further frames.
• When no unassigned frames remain, slaves may be reassigned to subsequent jobs.
• Once all frames are rendered successfully, the comp is deemed complete.

If a slave is unable to load a comp for any reason (bad path, missing plugin, missing font etc), or if a slave is unable to complete the render of an assigned frame, this is noted in the log and the slave becomes available for other jobs instead. It is possible to instruct a job to retry a failed slave (for remaining frames in the job), by means of the Manager interface or a script command.

In rare cases it may be necessary to consult log files produced by the individual slaves, but most details are communicated back to the Render Manager.