Can someone explain Disk Caching to me?

Moderator: SecondMan

tekgecko
Posts: 11
Joined: Sat May 11, 2019 11:56 am

Can someone explain Disk Caching to me?

#1

Post by tekgecko » Sat May 11, 2019 11:58 am

I have an SSD which I run on my PC to storage is fast, however when I cache a node with many branches to disk I don't see any playback improvement. Does anyone know if I am doing something wrong?

I have just switched from Hitfilm where "pre-rendering" would make the comp run in realtime no matter how complex so I am a little confused!

When I cache a node (see image:

Everything upstream seems to flash green indicating it is re-processing all of the tools before the cached node? I am sure this is wrong isn't it?

Thanks.

User avatar
SecondMan
Site Admin
Posts: 3239
Joined: Thu Jul 31, 2014 5:31 pm
Answers: 1
Location: Vancouver, Canada
Been thanked: 52 times
Contact:

Re: Can someone explain Disk Caching to me?

#2

Post by SecondMan » Sun May 12, 2019 9:43 am

Just so I understand correctly, do you need disk caching explained in general, or help troubleshooting your comp? The manual is a little sparse on it, but it should be fairly straightforward.

However, given the frequency of issues reported with it, it seems it's not. I actually never use it myself, I find simply managing precomps works just fine for my purposes. So far :)

In the meantime there are a couple of things you can try... for starters, here's @Movalex's console tip:
Movalex wrote:Here's a workaround to reliably lock a Disk Cache in Fusion:

1. Add a merge node to a heavy tool that's going to be cached.
2. Do a cache to disk for this Merge node. (You can change DiskCaches Path Map in settings to something appropriate)
3. Disconnect all nodes before cached Merge and add a BG node instead.
4. Profit!
and a while ago, @Kenzor did a great little explainer to the Pirates about the reasons behind failing Disk Caches:
Kenzor wrote:The disk cache fails when a node try's to grab pixels outside of the frame. Certain nodes increase the DOD beyond the DOD defined in a cache. When this happens fusion goes looking for the extra pixels and the whole branch is recalculated.
While in no way intuitive... it kinda makes sense since the cache is based on the DOD at that point in the flow and has no way of knowing if a bigger DOD is needed later
THERE IS A RELIABLE WAY OF USING THE CACHE
Because the recalculation is triggered by the change in DOD you tell fusion that the extra pixels don't exist by using a crop node.
If you want to cache a branch place a crop node where you need the cache and use that node to store the data.
When nodes further down the tree go looking for a bigger DOD the crop node basically says that the the DOD is fixed and protects the branch from recalculation .
I got the DOD info direct from the eyeon guys way back when.
the Crop node is my trick to get the cache working reliably.

So long is the clipping mode is set to 'Frame' (which also crops the DOD) this trick generally works.

Image

There maybe some other triggers that invalidate the cache and cause it to recalculate. But I find that the Crop node works 99% of the time.
I run a dedicated SSD for the fusion cache which really improves speed
Hope any of that helps - let us know how you get on!

tekgecko
Posts: 11
Joined: Sat May 11, 2019 11:56 am

Re: Can someone explain Disk Caching to me?

#3

Post by tekgecko » Sun May 12, 2019 11:15 am

Thanks for the info that explains things a little better. One thing I was doing wrong was pre-rendering when caching which I now know is only used for network rendering.

I basically want to take computationally heavy parts of the node tree and "save them out" so they can be played back in real time and connected up to the rest of the comp until render time. The first suggestion you gave seems to be able to do that for me but I am a little confused as to why you would use a BG node, is this not just a solid colour?

The way I understand it, when caching to disk you just need to click okay and then view that tool in the viewer and let it play through and cache and this will create the saved files?

The only problem is, when I close and reload Resolve it deletes all the files that were cached which seems wrong?

User avatar
SecondMan
Site Admin
Posts: 3239
Joined: Thu Jul 31, 2014 5:31 pm
Answers: 1
Location: Vancouver, Canada
Been thanked: 52 times
Contact:

Re: Can someone explain Disk Caching to me?

#4

Post by SecondMan » Sun May 12, 2019 2:01 pm

tekgecko wrote:
Sun May 12, 2019 11:15 am
caching which I now know is only used for network rendering
No, it's used locally as well. Perhaps even more commonly so.
tekgecko wrote:
Sun May 12, 2019 11:15 am
The first suggestion you gave seems to be able to do that for me but I am a little confused as to why you would use a BG node, is this not just a solid colour?
It's a hack. So in that example, the Merge node is the one being cached, which then becomes your source node for that branch. Nothing before the Merge should be processed anymore. The Background node is simply a stand-in for the part of the comp that came before the Merge. It doesn't do anything, apart from making sure the Merge doesn't fail because it doesn't have an input connected.
tekgecko wrote:
Sun May 12, 2019 11:15 am
The way I understand it, when caching to disk you just need to click okay and then view that tool in the viewer and let it play through and cache and this will create the saved files?
Correct. You can also ask Fusion to prerender the tool in the Cache to Disk dialog:

Image

The difference with using this method is that the Prerender calculates the cache at high quality full scale, whereas the interactive one will cache taking proxy scale settings into account.
tekgecko wrote:
Sun May 12, 2019 11:15 am
The only problem is, when I close and reload Resolve it deletes all the files that were cached which seems wrong?
That seems wrong, yes. Have you confirmed they are deleted from disk, or is it a case of the cache directory being different? Are you seeing the same in Fusion Standalone?

tekgecko
Posts: 11
Joined: Sat May 11, 2019 11:56 am

Re: Can someone explain Disk Caching to me?

#5

Post by tekgecko » Mon May 13, 2019 1:20 am

Ahh okay, I am more likely to skip pre-rendering for the sake of not having to wait the longer cache times.

The cache is completely deleted from the disk when opening, unless I lock the cache which seems to keep the files.

I understand the hack idea but after caching the node and removing everything before it, it just shows up as blank? Doesn't seem to be playing back the cached images.

Thanks so much for your help.

User avatar
SecondMan
Site Admin
Posts: 3239
Joined: Thu Jul 31, 2014 5:31 pm
Answers: 1
Location: Vancouver, Canada
Been thanked: 52 times
Contact:

Re: Can someone explain Disk Caching to me?

#6

Post by SecondMan » Tue May 14, 2019 12:36 am

When I get a moment I'll run some tests in Resolve 16. In the meantime, try your workflow in Fusion Standalone 9? See if that works the way you're expecting (so we can rule out or confirm whether something is broken).

Here's the link to the free Windows version (I'm just going to presumptuously assume you're a Windows user. So am I so don't think anything of it :D).


tekgecko
Posts: 11
Joined: Sat May 11, 2019 11:56 am

Re: Can someone explain Disk Caching to me?

#7

Post by tekgecko » Wed May 15, 2019 4:36 am

SecondMan wrote:
Tue May 14, 2019 12:36 am

When I get a moment I'll run some tests in Resolve 16. In the meantime, try your workflow in Fusion Standalone 9? See if that works the way you're expecting (so we can rule out or confirm whether something is broken).

Here's the link to the free Windows version (I'm just going to presumptuously assume you're a Windows user. So am I so don't think anything of it :D).

Haha no worries at all you are off the hook, Windows user here too ;)


User avatar
Chad
Fusionator
Posts: 1362
Joined: Fri Aug 08, 2014 1:11 pm
Been thanked: 8 times

Re: Can someone explain Disk Caching to me?

#8

Post by Chad » Wed May 15, 2019 2:47 pm

The Cache to Disk dialog predates ROIDS. I wonder if it would be nice to offer an overscan option, like you tell it a certain percentage to render on each side of the image and the requests are "padded" going upstream so the .raw or .exr (only formats I know that support data outside the frame) would have the data requested downstream.

Even better, select a downstream tool and for the render range, have it do all the precalc renders for the integral frames and set the exact requested data window for the disk cache.


User avatar
jedmitchell
Posts: 32
Joined: Mon Feb 27, 2017 5:11 pm

Re: Can someone explain Disk Caching to me?

#9

Post by jedmitchell » Wed May 15, 2019 7:19 pm

tekgecko wrote:
Sun May 12, 2019 11:15 am

The only problem is, when I close and reload Resolve it deletes all the files that were cached which seems wrong?

It does seem wrong, doesn't it? But whether by error or design, that seems to be the way it works in the Fusion Page, as far as I can tell from all the testing I've done.

I remember seeing a post from either Peter or Dwaine over on the official BMD forums to the effect that Fusion Page caching was a big problem they were working on, so maybe it will get better?


tekgecko
Posts: 11
Joined: Sat May 11, 2019 11:56 am

Re: Can someone explain Disk Caching to me?

#10

Post by tekgecko » Wed May 15, 2019 11:28 pm

jedmitchell wrote:
Wed May 15, 2019 7:19 pm
tekgecko wrote:
Sun May 12, 2019 11:15 am

The only problem is, when I close and reload Resolve it deletes all the files that were cached which seems wrong?

It does seem wrong, doesn't it? But whether by error or design, that seems to be the way it works in the Fusion Page, as far as I can tell from all the testing I've done.

I remember seeing a post from either Peter or Dwaine over on the official BMD forums to the effect that Fusion Page caching was a big problem they were working on, so maybe it will get better?

Let's hope so. For now, locking everything seems to stop it from being deleted.