[Fuse] Suck Less Audio File Modifier

areyoudizzzy
Posts: 3
Joined: Mon May 27, 2019 3:34 pm

Re: [Fuse] Suck Less Audio File Modifier

Hi guys, thanks for making these fuses and the whole Reactor setup! I'm only just getting my toes wet but it all seems fantastic!

I'm having some trouble with the Audio modifier in Resolve 16b3. I can get it working on a fusion composition tied to a video clip, so I can assume I'm taking the right steps (audio in correct format, etc.) but I'm trying to work it on the new "Adjustment Clips" and it doesn't seem to work. Using other modifiers, such as the Extended Wave modifier does work on Adjustment Clips but this one doesn't want to play ball. The panel for the modifier input values shows up and the parameter for Amplitude Offset works fine, it just won't react to the audio file.

An example of what I'm trying to do is set up a zooming effect on an adjustment layer over a few clips that reacts to the music. To do this I'm adding a transform node on the Adjustment Layer in the Fusion page and adjusting the "Size" parameter with the Audio modifier.

Any tips for getting this to work?

Many thanks!

EDIT: I'm running Suck Less Audio 2.50 in Resolve 16b3 on macOS 10.14.5

Tags:

SecondMan
Posts: 3383
Joined: Thu Jul 31, 2014 5:31 pm
Been thanked: 70 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

@areyoudizzzy

There's a version 2.6 on Reactor now that may have improved compatibility with Resolve. Try that and see if it works any better?

areyoudizzzy
Posts: 3
Joined: Mon May 27, 2019 3:34 pm

Re: [Fuse] Suck Less Audio File Modifier

Thanks for getting back to me! I realise this is a very new feature of Resolve and that this project operates on donations so thanks for getting to it so quickly!

I've done another test and it seems to be working perfectly in all places apart from adjustment clips I'm affraid! Tested with 16b3 and 16b4.

Any suggestions?

SecondMan
Posts: 3383
Joined: Thu Jul 31, 2014 5:31 pm
Been thanked: 70 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

areyoudizzzy wrote:
Mon Jun 10, 2019 3:07 am
this project operates on donations
Maybe one day...
areyoudizzzy wrote:
Mon Jun 10, 2019 3:07 am
it seems to be working perfectly in all places apart from adjustment clips I'm affraid
So I've looked into Resolve's Adjustment Layers a little bit, and basically what they seem to be - when using Fusion Compositions for them - is a type of comp, where when they are created, they take the render range from the global timeline, and then that range gets "fixed" in the comp. So even when you move the Adjustment Layer around, the render range stays the same.

From a user standpoint, Adjustment Layers are a bit like comp templates for which all the parameters and animations etc are "baked" to the original range of the layer. So if you make an animation inside one, that animation curve moves around with the Adjustment Layer and the global timeline moves "underneath it".

It's a pretty neat feature, it seems very useful. But I can also see how it can get a little messy and confusing when setting them up and moving them around, editing and copying them (I suppose copying them is possible - I haven't tried). From looking at it I wish they would have given this some more thought and come up with a workflow that uncouples Fusion's range from Resolve's edit timeline entirely. So you don't have to deal with potentially really weird keyframe numbers. Like by default I'd like to set at the very least a common start frame for Adjustment Layers, be it frame 0, 1 or 1001 (in VFX everyone is so used to start counting from 1001...).

I'm not sure if I'm explaining this all that clearly, and perhaps some of that is possible already, but never mind all that

For Suck Less Audio this simply means that when setting it up for the first time and loading an audio sample, check the render start number on the left hand side under Fusion's timeline and in Suck Less Audio put that value in the Sample Start Frame (Time Offset) parameter. You might not be able to read the entire parameter name due to Resolve's UI being so modern.

Anyway, that should work - it looks like it did for me

areyoudizzzy
Posts: 3
Joined: Mon May 27, 2019 3:34 pm

Re: [Fuse] Suck Less Audio File Modifier

Yes! It works!! This is going to be awesome! For a bit of detail and maybe some inspiration if you're wondering why I wanted this to work, I'm primarily a musician and want to make some fairly abstract visualizations for my music. I'm planning to break some of my songs down into their main parts (e.g. kick, snare, bass, keys, vocals, etc.) to use with this fuse and apply different effects and adjustments to some footage that all reacts to the musical elements.

So thanks so much for the detailed explanation!

One follow up question, why would the start frame in the fusion composition be 86400.0 when the adjustment clip is all the way to the left. at 01:00:00:00 on the edit page timeline?

SecondMan
Posts: 3383
Joined: Thu Jul 31, 2014 5:31 pm
Been thanked: 70 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

Exciting! That's is exactly the use case I had in mind when I created the Fuse. Unfortunately I have yet to find time to actually do something (properly) with it. I'd be very keen to see your process, if you're happy to share.

Let me know where SLA works for you, and where it doesn't - feedback helps to improve tools a great deal.
areyoudizzzy wrote:
Wed Jun 12, 2019 3:53 am
why would the start frame in the fusion composition be 86400.0 when the adjustment clip is all the way to the left. at 01:00:00:00 on the edit page timeline?
01:00:00:00 is timecode for 1 hour. An hour is 86400 frames at 24fps.

As I said, for adjustment layers I'd much rather see the ability to start from something a little more useful. Need to make a feature request of that...

SirEdric
Fusionator
Posts: 1753
Joined: Tue Aug 05, 2014 10:04 am
Real name: Eric Westphal
Been thanked: 80 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

SecondMan wrote:
Wed Jun 12, 2019 1:42 pm
I'd be very keen to see your process, if you're happy to share.
Plus 10 !!!
Reminds me of the days when I asked for the MIDI modifier, but only used it on a couple projects (where it saved my butt though...)

petersenk
Posts: 3
Joined: Fri Jun 14, 2019 1:29 pm

Re: [Fuse] Suck Less Audio File Modifier

Hi.

Is a hot analysis of a wav file actually the proper approach here? Why not do that offline, and just pass raw modifier data (e.g. a simple csv file) to fuse? Offline analysis (e.g. with some python script, or what not) could do much more, and better. Process more bands/tracks, and end up with more modifier data streams (by frequency analysis/FFT), one per separated band/track.
Then the fuse plugin could focus on what's actually important: take that raw data stream(s) and "modulate" those as needed, by which I mean scaling to have a proper range, or thresholding/limiting to suppress low signals, and make high signals uniform (always reach a certain max. level). Or maybe that's also rather something for offline processing (as in data clean-up).

So at this point we can link parameters to this audio/amplitude modifier, which is already nice to have some continuous automation going on. However, what's IMHO more important would be to have proper triggers, activated by passing an amplitude threshold (and not again until the signal drops below again). But I'm not so sure how this could work in Davinci Resolve/Fuse (I'm super new to this). For example, I'd like to have a subclip (superimposed over the timeline/video) play each time the bass drum hits. Can we do something like this?

EDIT: besides direct/continuous modifiers and triggers (if even possible), another obvious candidate to automate things would be an oscillator (LFO) tuned to a certain beat (that might change at some point in time). Together with control of the phase, this could be pretty useful too. A beat could be programmed manually (some steady bpm would already cover plenty of use cases), or automatically be derived from the data of a given track/wav file.

Still, I'm not quite sure how this would tie into the fuse/node framework, and if we can (re-)start and stop things like subclips in some way.

SirEdric
Fusionator
Posts: 1753
Joined: Tue Aug 05, 2014 10:04 am
Real name: Eric Westphal
Been thanked: 80 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

Interesting.
If you know of a way to create those CSV files offline, there are a couple examples out there how to read them via script and apply the values to FuControls...
In terms of the constant beat, there's the Beat Modifier from the ingenious Krokodove collection.
And don't forget the good old MIDI Extractor...

petersenk
Posts: 3
Joined: Fri Jun 14, 2019 1:29 pm

Re: [Fuse] Suck Less Audio File Modifier

SirEdric wrote:
Sat Jun 15, 2019 12:53 am
there are a couple examples out there how to read them [CSV files] via script and apply the values to FuControls...
Thought so. It would be a more general approach this way too, since you could feed such a fuse plugin any kind of data. Meanwhile that plugin could focus on what's really important: providing easy to use, yet powerful modifiers. For example there might be more useful functions to apply to the raw data stream than simple/linear scaling to get the range under control (a single data stream can be modulated to offer multiple, slightly different modifiers). And then thresholds for trigger control (which I'm still not sure is even currently feasible within the fusion framework).
SirEdric wrote:
Sat Jun 15, 2019 12:53 am
If you know of a way to create those CSV files offline
Shouldn't be too hard to come up with something usable given the sea of libraries available (e.g. for python). I'd imagine something where you can configure the FPS and then your bands to extract the amplitude separately for each band, maybe plot it already too, so you can spot problems early on, and maybe fix them already with different parameters (e.g. bandwidth, different number of bands to isolate certain sounds) and do some clean up even (noise reduction/noise gate, compressing the signal, etc.).
I'm not done skimming through the resolve manual yet (I'm on page 779 of 3000 pages or what not, holy hell), but once I'm more comfortable with the whole application I might have a look at this.

SecondMan
Posts: 3383
Joined: Thu Jul 31, 2014 5:31 pm
Been thanked: 70 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

Very excellent questions!
petersenk wrote:
Fri Jun 14, 2019 1:49 pm
Is a hot analysis of a wav file actually the proper approach here?
It is for my purposes

For Suck Less Audio, I'm not sure I see any benefit from moving the data conversion outside of Fusion to be honest. You would end up with an additional external file, which may not be any faster to read either. I'm also not convinced of the idea that using a .csv file is better suited as the store of data. It's not really any more "human readable" (it's a very large table of numbers after all) and what's great about the .wav files is that you can simply play them back and listen to them (the ability to play the audio right from the modifier is on one of my to-do lists).

SLA is essentially pretty basic, sure. Its task is reading the data contained in a .wav file and make it available for Fusion. And that's exactly what it does (plus shaping it a bit to make it fit your purpose, as you've noticed). In terms of audio, there is no more data embedded in a .wav file than what you get in SLA. You're right in saying that there is ample room for further analysis, like deriving frequency data, but that was not what SLA was designed to do. And because the functionality is "limited" to extracting the data only, it's very fast. I can run multiple ones in real-time on my extremely modest test setup.
petersenk wrote:
Fri Jun 14, 2019 1:49 pm
what's IMHO more important would be to have proper triggers, activated by passing an amplitude threshold (and not again until the signal drops below again)
Agreed! But I don't see this as a task for Suck Less Audio. For two reasons. One is that what you can do with input data like this is only really limited by your imagination! A tool like SLA would become huge and unwieldy after a while if you cram every single idea in there. And that's why at the time, alongside Suck Less Audio, I have also been developing another modifier called Time Machine. Secondly, why limit the processes you describe to audio only? If you have treshold functions, for example, they are just as useful for other types of input data, too.
petersenk wrote:
Fri Jun 14, 2019 1:49 pm
For example, I'd like to have a subclip (superimposed over the timeline/video) play each time the bass drum hits. Can we do something like this?
You certainly can in Fusion using Suck Less Audio and Time Machine. In Time Machine, look for a mode called Trigger. You can find Thresholds in there. Modify Trigger with Audio (WAV) and go from there:

It's this kind of use that Trigger was originally intended for. You can use audio data from SLA to drive it, but also a sine wave for example, or the rotation of a character's hips in a walkcycle.

Another thing that was always on my to-do list is to make more of these atomic Modifiers that can be chained together in Fusion. So you can start with an Audio (WAV) Modifier, then insert - for example - a Band Pass Modifier, then a Trigger Modifier, and then shape and modify the data with all sorts of different operations until it does what you want to do. Plus the process remains visible and accessible that way, too. This is also why many here would like to see Modifiers become nodes in Fusion...

Building this in a more modular fashion also means that the tools may not be limited to what you intend them to do. I find that this is a true general approach to tool building. So rather than building a Plugin or Fuse that extracts audio data and uses that to trigger the playback of clips, break it down into its separate constituents and provide those as separate building blocks. There's no greater joy than seeing something you've built being used in a way that you had never thought of yourself. It's very inspiring. It's also why Fusion doesn't get boring, even after 20 years...

petersenk
Posts: 3
Joined: Fri Jun 14, 2019 1:29 pm

Re: [Fuse] Suck Less Audio File Modifier

SirEdric wrote:
Sat Jun 15, 2019 12:53 am
If you know of a way to create those CSV files offline, ...
Instead of working my way through the Resolve manual, I did a thing over the weekend. Please, take a look:

https://github.com/limstepf/fbase

It's a little python script I've called "Filter bank (automation) signal extractor", or Fb(a)se in short. And it pretty much does what I've been talking about (see README.md on github, there are also some pictures). There's probably still plenty of room for improvements, but it's a start.

If you guys got some time, you'd be welcome to give it a try. And let me now what you think.

SecondMan wrote:
Sun Jun 16, 2019 4:56 pm
For Suck Less Audio, I'm not sure I see any benefit from moving the data conversion outside of Fusion to be honest. You would end up with an additional external file, which may not be any faster to read either. I'm also not convinced of the idea that using a .csv file is better suited as the store of data.
It's not about the file format (which probably doesn't matter too much), but about the automation data itself. All the samples in a WAV file are simply not needed to automate things at 24 FPS, so the signal needs to be downsampled anyways. If you just go ahead and skip every other sample, you're basically implementing a crappy low-pass filter, so a more clever approach would be nice to get clean and consistent automation data. Point being, for this you'll have to examine all the samples, and that's not necessarily something you'd like to do on the fly, but just once.

Please go ahead, and check the output of Fb(a)se. The produced CSV files include a frame number, the corresponding time in seconds, and then columns for the extracted signals which are normalized to [0.0, 1.0], such that further scaling is as simple as it gets. Do you think that would work already? Or would you also need a proper timecode (which can be tricky in some case, from what I'm aware. But there is probably some library that would do that for me...)?

SecondMan wrote:
Sun Jun 16, 2019 4:56 pm
You're right in saying that there is ample room for further analysis, like deriving frequency data, but that was not what SLA was designed to do.
Fair enough. Simple things are nice. I wasn't really suggesting that you change SLA. No, I'd rather see a new, similar, but more generic tool that reads data from CSV files (or other file formats), and offers a nice interface to use the parsed data as modifiers.

SecondMan wrote:
Sun Jun 16, 2019 4:56 pm
I have also been developing another modifier called Time Machine.
Awesome. I'll have a closer look at it. But from what I can tell, that's exactly what I'll need. You think I'll figure out how to use it, or is there some kind of manual, or tutorial around somewhere? Well, guess the thread you linked to would be a good starting point.

SecondMan wrote:
Sun Jun 16, 2019 4:56 pm
break it down into its separate constituents and provide those as separate building blocks.
Agreed. Modular is the way to go.
Carry on.