In the olden days, producers knew what visual effects were. Now they’ve gotten into this methodology where they’ll hire a middleman – a visual effects supervisor, and this person works for the producing studio.
They’re middle managers. And when you go into a review with one of them, there’s this weird sort of competition that happens.

It’s a game called ‘Find What’s Wrong With This Shot’. And there’s always going to be something wrong, because everything’s subjective. And you can micromanage it down to a pixel, and that happens all the time. We’re doing it digitally, so there’s no pressure to save on film costs or whatever, so it’s not unusual to go through 500 revisions of the same shot, moving pixels around and scrutinizing this or that.

That’s not how you manage artists. You encourage artists, and then you’ll get – you know – art.

If your idea of managing artists is just pointing out what’s wrong and making them fix it over and over again, you end up with artists who just stand around asking “OK lady, where do you want this sofa? You want it over there? No? Fine. You want it over there? I don’t give a fuck. I’ll put it wherever you want it.” It’s creative mismanagement, it’s part of the whole corporate modality. The fish stinks from the head on down. Back on Star Wars, Robocop, we never thought about what was wrong with a shot.

We just thought about how to make it better.

Phil Tippett

"skip existing" option in Savers:

Moderator: SecondMan

User avatar
avclubvids
Fusioneer
Posts: 95
Joined: Sun Sep 17, 2017 6:51 am
Location: Los Angeles, CA
Been thanked: 1 time
Contact:

"skip existing" option in Savers:

#1

Post by avclubvids » Mon Sep 18, 2017 12:30 pm

We are running a ton of Fusion renders on a commandline queue, and one thing that would make things a lot faster and more fault-tolerant would be a mode / flag / preference where savers can detect if a frame exists before processing and re-rendering it. does something like this exist?

User avatar
SirEdric
Fusionista
Posts: 262
Joined: Tue Aug 05, 2014 9:04 am
Been thanked: 25 times
Contact:

Re: "skip existing" option in Savers:

#2

Post by SirEdric » Mon Sep 18, 2017 7:50 pm

Fusion's RenderManager normally takes care of that by assigning individual frames to only one slave at a time.
If you want to re-render a comp, not overwriting any existing frames, you could use a script that checks for existing frames in the output location of the saver
and sets the comp's RenderRange to only the non-existing frames like (0..5,7,12,20..24) etc.

User avatar
avclubvids
Fusioneer
Posts: 95
Joined: Sun Sep 17, 2017 6:51 am
Location: Los Angeles, CA
Been thanked: 1 time
Contact:

Re: "skip existing" option in Savers:

#3

Post by avclubvids » Mon Sep 18, 2017 11:28 pm

Can Fusion execute scripts once per frame in the commandline exe? We'd need a script that evaluates each frame before rendering or skipping, so it'd have to be dynamic vs. once upon submission.

The issue is that when using a 3rd party render queue (Deadline etc) there is no good way for the existence of frames to be dynamically detected and frame ranges / task assignments adjusted accordingly to fill in gaps while avoiding unnecessary re-rendering. In a larger render farm environment, it is common to have priorities shuffle or machine status change mid-render, leaving some frames rendered and some missing. The only render engine that can smoothly and efficiently recover from this kind of partially complete render is After Effects due to its "skip existing" option- you can let the queue dumbly assign frame ranges to machines, and the AE executable runs a check before rendering each frame to avoid any wasted render time. Another issue this solves is a failed task that gets re-queued: say you have 100 frames in a task that takes an hour to render, but crashes and restarts at frame 99. AE will skip 99 frames automatically and render the 1 missing frame and be done. Fusion (and others) will render 100 frames, even though 99 of them don't need to be. Because most render queues are just dumb commandline submission tools, this kind of thing needs to be in the actual executable.

User avatar
Chad
Fusionator
Posts: 1118
Joined: Fri Aug 08, 2014 12:11 pm
Been thanked: 23 times

Re: "skip existing" option in Savers:

#4

Post by Chad » Tue Sep 19, 2017 4:17 pm

You can't match up comp render ranges to filenames. They don't match up if you have an offset or IFL or whatnot. The savers would have to check their output to determine if the requests need to be made. Not currently possible.

User avatar
avclubvids
Fusioneer
Posts: 95
Joined: Sun Sep 17, 2017 6:51 am
Location: Los Angeles, CA
Been thanked: 1 time
Contact:

Re: "skip existing" option in Savers:

#5

Post by avclubvids » Tue Sep 19, 2017 6:12 pm

let's make the impossible possible! If the savers could have a "check for existing" hook that render decisions can be based on, then we could gain all kinds of efficiencies in render farming and caching situations...

User avatar
French_Fry
Fusionista
Posts: 259
Joined: Tue Aug 05, 2014 7:04 am
Location: Nashville, TN
Been thanked: 3 times
Contact:

Re: "skip existing" option in Savers:

#6

Post by French_Fry » Wed Sep 20, 2017 6:37 am

I'm surprised that Deadline doesn't have that feature. We use Royal Render with AE and Fusion, and Royal Render takes care of seeing if frames exist and deletes them (or overwrites). So in Fusion we don't have to use a feature like that. Renders stop and start back up no problem.

User avatar
Midgardsormr
Fusioneer
Posts: 225
Joined: Wed Nov 26, 2014 8:04 pm
Location: Los Angeles, CA, USA
Been thanked: 44 times
Contact:

Re: "skip existing" option in Savers:

#7

Post by Midgardsormr » Wed Sep 20, 2017 7:25 am

It would probably be possible to customize the Deadline Submission script to do the check then reset the frame range field prior to submission. However, in order for it to work on the render manager's side, you couldn't render in chunks greater than 1. It would be better for Fusion to be able to handle it. There have been times when I would have liked that feature, too, on jobs where some frames inexplicably fail—they could just be deleted and the job resubmitted instead of having to track them all and type them into the Frame Range field.

There's a Wish List subforum here that at least a couple of the devs do read—post the request there, and maybe it will show up in a future version.

In the meantime, it should at least be possible to build a script that enumerates the frames existing in a folder and puts a list of missing ones on the clipboard ready to be pasted into the Frame List field in Deadline.

User avatar
avclubvids
Fusioneer
Posts: 95
Joined: Sun Sep 17, 2017 6:51 am
Location: Los Angeles, CA
Been thanked: 1 time
Contact:

Re: "skip existing" option in Savers:

#8

Post by avclubvids » Wed Sep 20, 2017 9:21 am

on jobs where some frames inexplicably fail—they could just be deleted and the job resubmitted instead of having to track them all and type them into the Frame Range field.
100% this. Super easy to fix render issues when the renderer has a skip existing feature. I'll post a link to this thread in the wishlist forum.

User avatar
French_Fry
Fusionista
Posts: 259
Joined: Tue Aug 05, 2014 7:04 am
Location: Nashville, TN
Been thanked: 3 times
Contact:

Re: "skip existing" option in Savers:

#9

Post by French_Fry » Wed Sep 20, 2017 11:23 am

Midgardsormr wrote:It would probably be possible to customize the Deadline Submission script to do the check then reset the frame range field prior to submission. However, in order for it to work on the render manager's side, you couldn't render in chunks greater than 1.
That's interesting too...Royal Render handles chunk size on the fly for that too. I must be missing something to this problem. I understand that this feature would be great to have in Fusion, especially if rendering from Fusion only or with it's render manager. But all these these requests seem to be handled by a 3rd party manager..in my case Royal Render. But I never figured RR to be an outstanding render manager. I always figured Deadline to be the higher end manager out there.

User avatar
Midgardsormr
Fusioneer
Posts: 225
Joined: Wed Nov 26, 2014 8:04 pm
Location: Los Angeles, CA, USA
Been thanked: 44 times
Contact:

Re: "skip existing" option in Savers:

#10

Post by Midgardsormr » Wed Sep 20, 2017 12:12 pm

Well, I know Deadline has the option to skip existing in some of its other submissions scripts, so I was assuming that it was down to whatever options are in those programs. I don't know a whole lot about the internals of Deadline, so maybe this behavior can be modified somehow, but if you try to do something like render by twos, it makes a task per frame regardless of desired chunk size.

Maybe someone who's actually done some customized submission scripts would have a better idea of what's possible there. It's not something I plan to get into any time soon; I've avoided learning Python specifically so I don't have to support very much pipeline stuff!

User avatar
avclubvids
Fusioneer
Posts: 95
Joined: Sun Sep 17, 2017 6:51 am
Location: Los Angeles, CA
Been thanked: 1 time
Contact:

Re: "skip existing" option in Savers:

#11

Post by avclubvids » Wed Sep 20, 2017 1:49 pm

So the main issue for most commandline execution tools (Deadline etc.) is that they don't know anything about what the programs they are executing are supposed to be doing. They have scripts to gather the information from required to compile all the right arguments to launch a chunk of frames for a slave to process, and they have scripts to help them parse the console/logs back from the executable as it is working to update things like progress etc.

Most of the render managers are pretty dumb. When there are missing frames in a task chunk, they tend to only be able to re-render all the frames in that chunk, or the user has to carefully re-submit the job for just the missing frames (which can be very annoying if the file takes a long time to open, which is common in 3D files with lots of geo etc.)

The specific issue this "skip existing" option would address is scenarios in which, due to an error, some frames are missing. Yes, we could make a pre-flight script that checks for missing frames and formats the commandline flags to render specific frames and ranges, but that does not account for issues that happen during the render. I cannot tell you how many times we have had to re-queue a chunk of frames to fix the one frame that is missing. There's also glorious games you can play on Deadline with renderers that can skip existing, wherein you keep changing the size (and thus number of) tasks to make sure that every machine on the farm is working. Without skip existing, this type of re-tasking leads to unnecessary re-rendering of frames b/c the manager doesn't actually know what frames are done.

BlackRainbow
Posts: 14
Joined: Sat Jul 08, 2017 8:31 am
Been thanked: 1 time

Re: "skip existing" option in Savers:

#12

Post by BlackRainbow » Thu Sep 21, 2017 12:04 am

How exactly do you do your command line renders? That would help a lot with my renders. Any resource or tutorial?

User avatar
avclubvids
Fusioneer
Posts: 95
Joined: Sun Sep 17, 2017 6:51 am
Location: Los Angeles, CA
Been thanked: 1 time
Contact:

Re: "skip existing" option in Savers:

#13

Post by avclubvids » Thu Sep 21, 2017 9:33 am

We are using CGRU (http://cgru.info/) because it is free/open source, but Deadline (http://deadline.thinkboxsoftware.com/)is by far the current most popular render manager. But they are all the same generally - they have plugins for Fusion that help you submit, a slave process that launches Fusion commandline with the right flags on your render slaves, and they have a GUI that lets you see how it's going.

Setting up a farm is a hassle, managing it can be too, so if you're just a small setup with a few people / machines, I would recommend sticking with the built-in Fusion network render solution, it's fantastic!

User avatar
avclubvids
Fusioneer
Posts: 95
Joined: Sun Sep 17, 2017 6:51 am
Location: Los Angeles, CA
Been thanked: 1 time
Contact:

Re: "skip existing" option in Savers:

#14

Post by avclubvids » Mon Sep 25, 2017 9:19 am

More thoughts on this - the BlackMagic render manager does not really have a skip existing mode as best I can tell. It has a record of what it has rendered and can pick up from where it left off and assign unrendered frames to the slaves. This does not help in scenarios where the manager crashes and does not autosave an accurate backup of the queue, nor does it help when the render completes but has errors requiring bad or missing frames to be re-rendered.

The only way to really do this is to add a mode to the Saver wherein it can check for the existence of each frame before processing. For sanity in dealing with render managers, this could be as simple as returning a message that the frame was rendered successfully even though nothing new was processed. It might take a moment to do this, but far less than the time to actually process the frame.

User avatar
Chad
Fusionator
Posts: 1118
Joined: Fri Aug 08, 2014 12:11 pm
Been thanked: 23 times

Re: "skip existing" option in Savers:

#15

Post by Chad » Mon Sep 25, 2017 3:02 pm

How would the Render Manager know which frames to skip without loading the comp and making the requests?