[Fuse] Suck Less Audio File Modifier

User avatar
areyoudizzzy
Posts: 3
Joined: Mon May 27, 2019 3:34 pm

Re: [Fuse] Suck Less Audio File Modifier

#61

Post by areyoudizzzy » Mon May 27, 2019 4:02 pm

Hi guys, thanks for making these fuses and the whole Reactor setup! I'm only just getting my toes wet but it all seems fantastic!

I'm having some trouble with the Audio modifier in Resolve 16b3. I can get it working on a fusion composition tied to a video clip, so I can assume I'm taking the right steps (audio in correct format, etc.) but I'm trying to work it on the new "Adjustment Clips" and it doesn't seem to work. Using other modifiers, such as the Extended Wave modifier does work on Adjustment Clips but this one doesn't want to play ball. The panel for the modifier input values shows up and the parameter for Amplitude Offset works fine, it just won't react to the audio file.

An example of what I'm trying to do is set up a zooming effect on an adjustment layer over a few clips that reacts to the music. To do this I'm adding a transform node on the Adjustment Layer in the Fusion page and adjusting the "Size" parameter with the Audio modifier.

Any tips for getting this to work?

Many thanks!

EDIT: I'm running Suck Less Audio 2.50 in Resolve 16b3 on macOS 10.14.5


Tags:

User avatar
SecondMan
Site Admin
Posts: 3470
Joined: Thu Jul 31, 2014 5:31 pm
Answers: 5
Location: Vancouver, Canada
Been thanked: 92 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

#62

Post by SecondMan » Sun Jun 09, 2019 6:50 pm

@areyoudizzzy

There's a version 2.6 on Reactor now that may have improved compatibility with Resolve. Try that and see if it works any better?

User avatar
areyoudizzzy
Posts: 3
Joined: Mon May 27, 2019 3:34 pm

Re: [Fuse] Suck Less Audio File Modifier

#63

Post by areyoudizzzy » Mon Jun 10, 2019 3:07 am

Thanks for getting back to me! I realise this is a very new feature of Resolve and that this project operates on donations so thanks for getting to it so quickly!

I've done another test and it seems to be working perfectly in all places apart from adjustment clips I'm affraid! Tested with 16b3 and 16b4.

Any suggestions?

User avatar
SecondMan
Site Admin
Posts: 3470
Joined: Thu Jul 31, 2014 5:31 pm
Answers: 5
Location: Vancouver, Canada
Been thanked: 92 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

#64

Post by SecondMan » Mon Jun 10, 2019 9:22 pm

areyoudizzzy wrote:
Mon Jun 10, 2019 3:07 am
this project operates on donations
Maybe one day... ;)
areyoudizzzy wrote:
Mon Jun 10, 2019 3:07 am
it seems to be working perfectly in all places apart from adjustment clips I'm affraid
So I've looked into Resolve's Adjustment Layers a little bit, and basically what they seem to be - when using Fusion Compositions for them - is a type of comp, where when they are created, they take the render range from the global timeline, and then that range gets "fixed" in the comp. So even when you move the Adjustment Layer around, the render range stays the same.

From a user standpoint, Adjustment Layers are a bit like comp templates for which all the parameters and animations etc are "baked" to the original range of the layer. So if you make an animation inside one, that animation curve moves around with the Adjustment Layer and the global timeline moves "underneath it".

It's a pretty neat feature, it seems very useful. But I can also see how it can get a little messy and confusing when setting them up and moving them around, editing and copying them (I suppose copying them is possible - I haven't tried). From looking at it I wish they would have given this some more thought and come up with a workflow that uncouples Fusion's range from Resolve's edit timeline entirely. So you don't have to deal with potentially really weird keyframe numbers. Like by default I'd like to set at the very least a common start frame for Adjustment Layers, be it frame 0, 1 or 1001 (in VFX everyone is so used to start counting from 1001...).

I'm not sure if I'm explaining this all that clearly, and perhaps some of that is possible already, but never mind all that :)

For Suck Less Audio this simply means that when setting it up for the first time and loading an audio sample, check the render start number on the left hand side under Fusion's timeline and in Suck Less Audio put that value in the Sample Start Frame (Time Offset) parameter. You might not be able to read the entire parameter name due to Resolve's UI being so modern.

Anyway, that should work - it looks like it did for me :)

User avatar
areyoudizzzy
Posts: 3
Joined: Mon May 27, 2019 3:34 pm

Re: [Fuse] Suck Less Audio File Modifier

#65

Post by areyoudizzzy » Wed Jun 12, 2019 3:53 am

Yes! It works!! This is going to be awesome! For a bit of detail and maybe some inspiration if you're wondering why I wanted this to work, I'm primarily a musician and want to make some fairly abstract visualizations for my music. I'm planning to break some of my songs down into their main parts (e.g. kick, snare, bass, keys, vocals, etc.) to use with this fuse and apply different effects and adjustments to some footage that all reacts to the musical elements.

So thanks so much for the detailed explanation!

One follow up question, why would the start frame in the fusion composition be 86400.0 when the adjustment clip is all the way to the left. at 01:00:00:00 on the edit page timeline?

User avatar
SecondMan
Site Admin
Posts: 3470
Joined: Thu Jul 31, 2014 5:31 pm
Answers: 5
Location: Vancouver, Canada
Been thanked: 92 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

#66

Post by SecondMan » Wed Jun 12, 2019 1:42 pm

Exciting! That's is exactly the use case I had in mind when I created the Fuse. Unfortunately I have yet to find time to actually do something (properly) with it. I'd be very keen to see your process, if you're happy to share.

Let me know where SLA works for you, and where it doesn't - feedback helps to improve tools a great deal.
areyoudizzzy wrote:
Wed Jun 12, 2019 3:53 am
why would the start frame in the fusion composition be 86400.0 when the adjustment clip is all the way to the left. at 01:00:00:00 on the edit page timeline?
01:00:00:00 is timecode for 1 hour. An hour is 86400 frames at 24fps.

As I said, for adjustment layers I'd much rather see the ability to start from something a little more useful. Need to make a feature request of that...

User avatar
SirEdric
Fusionator
Posts: 1862
Joined: Tue Aug 05, 2014 10:04 am
Answers: 2
Real name: Eric Westphal
Been thanked: 111 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

#67

Post by SirEdric » Wed Jun 12, 2019 1:49 pm

SecondMan wrote:
Wed Jun 12, 2019 1:42 pm
I'd be very keen to see your process, if you're happy to share.
Plus 10 !!!
Reminds me of the days when I asked for the MIDI modifier, but only used it on a couple projects (where it saved my butt though...:-))

User avatar
petersenk
Posts: 3
Joined: Fri Jun 14, 2019 1:29 pm

Re: [Fuse] Suck Less Audio File Modifier

#68

Post by petersenk » Fri Jun 14, 2019 1:49 pm

Hi.

Is a hot analysis of a wav file actually the proper approach here? Why not do that offline, and just pass raw modifier data (e.g. a simple csv file) to fuse? Offline analysis (e.g. with some python script, or what not) could do much more, and better. Process more bands/tracks, and end up with more modifier data streams (by frequency analysis/FFT), one per separated band/track.
Then the fuse plugin could focus on what's actually important: take that raw data stream(s) and "modulate" those as needed, by which I mean scaling to have a proper range, or thresholding/limiting to suppress low signals, and make high signals uniform (always reach a certain max. level). Or maybe that's also rather something for offline processing (as in data clean-up).

So at this point we can link parameters to this audio/amplitude modifier, which is already nice to have some continuous automation going on. However, what's IMHO more important would be to have proper triggers, activated by passing an amplitude threshold (and not again until the signal drops below again). But I'm not so sure how this could work in Davinci Resolve/Fuse (I'm super new to this). For example, I'd like to have a subclip (superimposed over the timeline/video) play each time the bass drum hits. Can we do something like this?

EDIT: besides direct/continuous modifiers and triggers (if even possible), another obvious candidate to automate things would be an oscillator (LFO) tuned to a certain beat (that might change at some point in time). Together with control of the phase, this could be pretty useful too. A beat could be programmed manually (some steady bpm would already cover plenty of use cases), or automatically be derived from the data of a given track/wav file.

Still, I'm not quite sure how this would tie into the fuse/node framework, and if we can (re-)start and stop things like subclips in some way.

User avatar
SirEdric
Fusionator
Posts: 1862
Joined: Tue Aug 05, 2014 10:04 am
Answers: 2
Real name: Eric Westphal
Been thanked: 111 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

#69

Post by SirEdric » Sat Jun 15, 2019 12:53 am

Interesting.
If you know of a way to create those CSV files offline, there are a couple examples out there how to read them via script and apply the values to FuControls...:-)
In terms of the constant beat, there's the Beat Modifier from the ingenious Krokodove collection.
And don't forget the good old MIDI Extractor...:-)

User avatar
petersenk
Posts: 3
Joined: Fri Jun 14, 2019 1:29 pm

Re: [Fuse] Suck Less Audio File Modifier

#70

Post by petersenk » Sat Jun 15, 2019 4:46 am

SirEdric wrote:
Sat Jun 15, 2019 12:53 am
there are a couple examples out there how to read them [CSV files] via script and apply the values to FuControls...:-)
Thought so. It would be a more general approach this way too, since you could feed such a fuse plugin any kind of data. Meanwhile that plugin could focus on what's really important: providing easy to use, yet powerful modifiers. For example there might be more useful functions to apply to the raw data stream than simple/linear scaling to get the range under control (a single data stream can be modulated to offer multiple, slightly different modifiers). And then thresholds for trigger control (which I'm still not sure is even currently feasible within the fusion framework).
SirEdric wrote:
Sat Jun 15, 2019 12:53 am
If you know of a way to create those CSV files offline
Shouldn't be too hard to come up with something usable given the sea of libraries available (e.g. for python). I'd imagine something where you can configure the FPS and then your bands to extract the amplitude separately for each band, maybe plot it already too, so you can spot problems early on, and maybe fix them already with different parameters (e.g. bandwidth, different number of bands to isolate certain sounds) and do some clean up even (noise reduction/noise gate, compressing the signal, etc.).
I'm not done skimming through the resolve manual yet (I'm on page 779 of 3000 pages or what not, holy hell), but once I'm more comfortable with the whole application I might have a look at this.

User avatar
SecondMan
Site Admin
Posts: 3470
Joined: Thu Jul 31, 2014 5:31 pm
Answers: 5
Location: Vancouver, Canada
Been thanked: 92 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

#71

Post by SecondMan » Sun Jun 16, 2019 4:56 pm

Very excellent questions! :)
petersenk wrote:
Fri Jun 14, 2019 1:49 pm
Is a hot analysis of a wav file actually the proper approach here?
It is for my purposes :)

For Suck Less Audio, I'm not sure I see any benefit from moving the data conversion outside of Fusion to be honest. You would end up with an additional external file, which may not be any faster to read either. I'm also not convinced of the idea that using a .csv file is better suited as the store of data. It's not really any more "human readable" (it's a very large table of numbers after all) and what's great about the .wav files is that you can simply play them back and listen to them (the ability to play the audio right from the modifier is on one of my to-do lists).

SLA is essentially pretty basic, sure. Its task is reading the data contained in a .wav file and make it available for Fusion. And that's exactly what it does (plus shaping it a bit to make it fit your purpose, as you've noticed). In terms of audio, there is no more data embedded in a .wav file than what you get in SLA. You're right in saying that there is ample room for further analysis, like deriving frequency data, but that was not what SLA was designed to do. And because the functionality is "limited" to extracting the data only, it's very fast. I can run multiple ones in real-time on my extremely modest test setup.
petersenk wrote:
Fri Jun 14, 2019 1:49 pm
what's IMHO more important would be to have proper triggers, activated by passing an amplitude threshold (and not again until the signal drops below again)
Agreed! But I don't see this as a task for Suck Less Audio. For two reasons. One is that what you can do with input data like this is only really limited by your imagination! A tool like SLA would become huge and unwieldy after a while if you cram every single idea in there. And that's why at the time, alongside Suck Less Audio, I have also been developing another modifier called Time Machine. Secondly, why limit the processes you describe to audio only? If you have treshold functions, for example, they are just as useful for other types of input data, too.
petersenk wrote:
Fri Jun 14, 2019 1:49 pm
For example, I'd like to have a subclip (superimposed over the timeline/video) play each time the bass drum hits. Can we do something like this?
You certainly can in Fusion using Suck Less Audio and Time Machine. In Time Machine, look for a mode called Trigger. You can find Thresholds in there. Modify Trigger with Audio (WAV) and go from there:

Image

It's this kind of use that Trigger was originally intended for. You can use audio data from SLA to drive it, but also a sine wave for example, or the rotation of a character's hips in a walkcycle.

Another thing that was always on my to-do list is to make more of these atomic Modifiers that can be chained together in Fusion. So you can start with an Audio (WAV) Modifier, then insert - for example - a Band Pass Modifier, then a Trigger Modifier, and then shape and modify the data with all sorts of different operations until it does what you want to do. Plus the process remains visible and accessible that way, too. This is also why many here would like to see Modifiers become nodes in Fusion...

Building this in a more modular fashion also means that the tools may not be limited to what you intend them to do. I find that this is a true general approach to tool building. So rather than building a Plugin or Fuse that extracts audio data and uses that to trigger the playback of clips, break it down into its separate constituents and provide those as separate building blocks. There's no greater joy than seeing something you've built being used in a way that you had never thought of yourself. It's very inspiring. It's also why Fusion doesn't get boring, even after 20 years... :)

User avatar
petersenk
Posts: 3
Joined: Fri Jun 14, 2019 1:29 pm

Re: [Fuse] Suck Less Audio File Modifier

#72

Post by petersenk » Mon Jun 17, 2019 9:02 pm

SirEdric wrote:
Sat Jun 15, 2019 12:53 am
If you know of a way to create those CSV files offline, ...
Instead of working my way through the Resolve manual, I did a thing over the weekend. Please, take a look:

https://github.com/limstepf/fbase

It's a little python script I've called "Filter bank (automation) signal extractor", or Fb(a)se in short. And it pretty much does what I've been talking about (see README.md on github, there are also some pictures). There's probably still plenty of room for improvements, but it's a start.

If you guys got some time, you'd be welcome to give it a try. And let me now what you think.

SecondMan wrote:
Sun Jun 16, 2019 4:56 pm
For Suck Less Audio, I'm not sure I see any benefit from moving the data conversion outside of Fusion to be honest. You would end up with an additional external file, which may not be any faster to read either. I'm also not convinced of the idea that using a .csv file is better suited as the store of data.
It's not about the file format (which probably doesn't matter too much), but about the automation data itself. All the samples in a WAV file are simply not needed to automate things at 24 FPS, so the signal needs to be downsampled anyways. If you just go ahead and skip every other sample, you're basically implementing a crappy low-pass filter, so a more clever approach would be nice to get clean and consistent automation data. Point being, for this you'll have to examine all the samples, and that's not necessarily something you'd like to do on the fly, but just once.

Please go ahead, and check the output of Fb(a)se. The produced CSV files include a frame number, the corresponding time in seconds, and then columns for the extracted signals which are normalized to [0.0, 1.0], such that further scaling is as simple as it gets. Do you think that would work already? Or would you also need a proper timecode (which can be tricky in some case, from what I'm aware. But there is probably some library that would do that for me...)?

SecondMan wrote:
Sun Jun 16, 2019 4:56 pm
You're right in saying that there is ample room for further analysis, like deriving frequency data, but that was not what SLA was designed to do.
Fair enough. Simple things are nice. I wasn't really suggesting that you change SLA. No, I'd rather see a new, similar, but more generic tool that reads data from CSV files (or other file formats), and offers a nice interface to use the parsed data as modifiers.

SecondMan wrote:
Sun Jun 16, 2019 4:56 pm
I have also been developing another modifier called Time Machine.
Awesome. I'll have a closer look at it. But from what I can tell, that's exactly what I'll need. You think I'll figure out how to use it, or is there some kind of manual, or tutorial around somewhere? Well, guess the thread you linked to would be a good starting point.

SecondMan wrote:
Sun Jun 16, 2019 4:56 pm
break it down into its separate constituents and provide those as separate building blocks.
Agreed. Modular is the way to go.
Carry on. :D

User avatar
SecondMan
Site Admin
Posts: 3470
Joined: Thu Jul 31, 2014 5:31 pm
Answers: 5
Location: Vancouver, Canada
Been thanked: 92 times
Contact:

Re: [Fuse] Suck Less Audio File Modifier

#73

Post by SecondMan » Sun Jul 28, 2019 1:41 pm

A quick tutorial video on how to install and use Suck Less Audio in DaVinci Resolve has been spotted in the wild:



Thanks @AndrewHazelden for the tip! 👍

User avatar
Skuggrekyl
Posts: 6
Joined: Wed Sep 04, 2019 12:46 am

Re: Issues with Reactor 3, modify with audio. Resolve 16 on Windows 10.

#74

Post by Skuggrekyl » Wed Sep 04, 2019 6:11 am

Thanks for the reply!

I tried to export the sound file again from Reaper to WAV 16-bit PCM (see settings below) and tried again. I got the same result as before, unfortunately.

I am able to install, open and muck around with the other things. I installed Blurry edges and played around with that for a bit and it worked without issues.


User avatar
SecondMan
Site Admin
Posts: 3470
Joined: Thu Jul 31, 2014 5:31 pm
Answers: 5
Location: Vancouver, Canada
Been thanked: 92 times
Contact:

Re: [Fuse] Time Machine Modifier Fuse

#75

Post by SecondMan » Wed Sep 04, 2019 9:51 am

Skuggrekyl wrote:
Wed Sep 04, 2019 6:11 am

Thanks for the reply!

Continued from this topic: viewtopic.php?p=25206#p25206

Suck Less Audio only accepts a single flavour of WAV files, which is 16 bit PCM as defined by Microsoft. I'm doing a bit by bit read on the samples as I go along, so if anything in the sample structure is different, it will not work.

The console should give you a hint as to why it fails, if the problem lies with Suck Less Audio. Check if you have any messages there?

Audacity saves the PCM format without issue. I've never used Reaper, so I can't guarantee if that will work. From your screenshot, I would recommend deselecting Write BWF ('bext') chunk on export.

Hope that helps a little.

PS. I see you're using Imgur for your image sharing. Imgur is also built into the forum, which makes that process even easier. You can read all about that here :)