Psyop's Cryptomatte In Fusion

User avatar
Kristof
Fusionista
Posts: 431
Joined: Thu Aug 07, 2014 1:30 pm
Been thanked: 5 times

Psyop's Cryptomatte In Fusion

#1

Post by Kristof » Sat Oct 15, 2016 7:38 am

For those of you looking for the released version, jump to this post: https://www.steakunderwater.com/wesuckle ... 8591#p8591
Cryptomatte is a tool created at Psyop by Jonah Friedman and Andy Jones. It creates ID mattes automatically with support for motion blur, transparency, and depth of field, using organizational information already available at render time. This organizational information is usually names, object namespaces, and material names.
As usual, the toolset comes with a version for Nuke and the rest of us are left out in the cold. Anyway, this is a comp along with one demo image you can download from the GitHub page, to show how the matte extraction part works (it is surprisingly simple and also shows you how to use an ID with a coverage pass in general).

So this is not a working Cryptomatte yet, but it is in the works. The hashing part is also covered by a colleague of mine but currently not being demoed in the comp.

http://dump.indeherberge.org/cryptoMatte.zip
Image
Last edited by Kristof on Sun Jan 22, 2017 1:50 pm, edited 3 times in total.

Tags:

User avatar
pingking
Fusionista
Posts: 637
Joined: Thu Aug 14, 2014 9:10 am
Been thanked: 9 times

Re: Psyop's Cryptomatte In Fusion

#2

Post by pingking » Sat Oct 15, 2016 8:38 am

really cool that you work on this

User avatar
SecondMan
Site Admin
Posts: 2406
Joined: Thu Jul 31, 2014 5:31 pm
Location: Vancouver, Canada
Been thanked: 60 times
Contact:

Re: Psyop's Cryptomatte In Fusion

#3

Post by SecondMan » Sat Oct 15, 2016 9:01 am

Oooh pretty neat!

Thanks for sharing!

User avatar
Kristof
Fusionista
Posts: 431
Joined: Thu Aug 07, 2014 1:30 pm
Been thanked: 5 times

Re: Psyop's Cryptomatte In Fusion

#4

Post by Kristof » Sat Oct 15, 2016 10:55 am

It's a team effort. At Grid, Cédric Duriau is coding and a great help in translating what the code actually does. And thanks to Boris Martinez to shed some light on the Nuke ins and outs.

mdharrington
Fusioneer
Posts: 144
Joined: Thu Sep 04, 2014 4:56 pm
Been thanked: 1 time

Re: Psyop's Cryptomatte In Fusion

#5

Post by mdharrington » Sat Oct 15, 2016 2:36 pm

Sorry, I'm interested in this but a little slow on the understanding....

Are these mattes created by a specific renderer, or some kind of plugin that works with a specific render?
And what special is stored in the mattes? Just object ID, or are there layers stacked in depth or something else?

Cause in my mind this is blurring the lines with deep compositing in the way you can cleanly isolate an antialiased object.

User avatar
Kristof
Fusionista
Posts: 431
Joined: Thu Aug 07, 2014 1:30 pm
Been thanked: 5 times

Re: Psyop's Cryptomatte In Fusion

#6

Post by Kristof » Sun Oct 16, 2016 1:14 am

mdharrington wrote:Sorry, I'm interested in this but a little slow on the understanding....
Ask away, I don't mind. On the contrary, that's how we can fully understand what is going on. And I don't want to pretend I understand all of it, but the goal is to fully understand all aspects of it. That's why I feel like sharing even the crudest snippets of workflow as I did in this thread. Community driven...
Are these mattes created by a specific renderer, or some kind of plugin that works with a specific render?
Currently, yes. You need the alshader for Arnold: http://www.anderslanglands.com/alshader ... matte.html
Chaos Group is working on V-Ray’s implementation of the Anders Langlands’ alSurface shader, so maybe this could end up in V-Ray too:
https://labs.chaosgroup.com/index.php/r ... ce-shader/
And what special is stored in the mattes? Just object ID, or are there layers stacked in depth or something else?
Two things, stacks of (or ranked) ID-Coverage pairs so that multiple objects per pixel are covered (not based on depth but on weighted pixel contribution, as I understand it) and a labeling system so you can have human readable labels stored in metadata (hashing).

mdharrington
Fusioneer
Posts: 144
Joined: Thu Sep 04, 2014 4:56 pm
Been thanked: 1 time

Re: Psyop's Cryptomatte In Fusion

#7

Post by mdharrington » Sun Oct 16, 2016 9:13 pm

thanks....i think I get the idea now :)

Hopefully there can be wider render support in the future

User avatar
French_Fry
Fusionista
Posts: 299
Joined: Tue Aug 05, 2014 8:04 am
Location: Nashville, TN
Been thanked: 2 times
Contact:

Re: Psyop's Cryptomatte In Fusion

#8

Post by French_Fry » Mon Oct 17, 2016 7:45 am

I think the Redshift 3D guys are part of the cryptomatte group. Hopefully we'll see some support there also! :)

User avatar
miaz3
Fusioneer
Posts: 173
Joined: Sat Jan 03, 2015 1:43 am
Location: Angoulême / France
Contact:

Re: Psyop's Cryptomatte In Fusion

#9

Post by miaz3 » Tue Oct 18, 2016 11:41 am

fiew projects can't work in Fu, but its good to know !

jonandy
Posts: 4
Joined: Wed Oct 19, 2016 5:37 pm

Re: Psyop's Cryptomatte In Fusion

#10

Post by jonandy » Fri Oct 21, 2016 1:40 pm

Hi,

I'm Andy Jones, one of the creators of Cryptomatte. Jonah actually spotted this thread the other day, and I just wanted to chime in to share a few thoughts, and make myself available if you guys have any questions I can answer about Cryptomatte.

A bit of background -- I'm actually a big fan of Fusion, and was a Fusion user myself for a while. So, it's really exciting to see interest in Cryptomatte on the Fusion forum. I apologize we don't have a Fusion version. Since Cryptomatte is really more like a specification than a code library, our hope with the Nuke release was that it would serve as a template, and would lead to versions in the other packages (which seems to be happening, as far as we can tell). Unfortunately, there are only so many hours in the day, and although we DO have Fusion running at Psyop, making Cryptomatte for Fusion hasn't really been something we could justify spending time on at work yet.

That said, this past spring I did spend a little bit of time looking into how Cryptomatte would fit best into Fusion. It's really great to see a working Cryptomatte node graph getting posted on a user forum, as it helps show how simple Cryptomatte really is -- especially on the comp side. Our Nuke version is basically just a gizmo (the equivalent of a Fusion macro, I think), plus some expressions and a python library. As far as making an full-fledged plugin, I didn't ever get too deep into the API back when I was comping in Fusion, but it seems like writing something with the Fuse API would probably make sense... OFX could work as well, but my overall impression is that OFX plugins never work quite as well as native plugins, unless a package is specifically designed around OFX. And OFX would probably mean distributing binaries (which isn't a bad thing, but can require some overhead to keep up with new releases.)

In any case, I think there are some good options to get something working via the api. However, my overall assessment was that the BEST way to get Cryptomatte working in Fusion would be to extend the existing Object ID and Material ID channels to support Cryptomatte-style channels natively. That seems much more in keeping with how Fusion is designed to work, with each node providing a lot of value-added functionality. Given that Cryptomatte works pretty much exactly like Object IDs (but without edge artifacts and with a table of names for IDs), it would just make a lot of sense to upgrade the existing features, rather than add something functionally superior that's not as well-integrated. IDs also require special handling, because they can't be re-sampled the same way as regular RGB data. That sort of thing would be much easier for users to navigate if the IDs are in a dedicated ID channel. Also, I'm under the impression that one of the big reasons Nuke has "unlimited" channels is to allow mattes to be passed through the comp. With Cryptomatte, you don't really need that, so a limited, but sensible set of RPF channels can actually start making more sense than Nuke's solution.

With that in mind, at SIGGRAPH this past year, Jonah and I visited the Fusion booth and pitched Cryptomatte to Steve Roberts, the Fusion product manager. We had a great chat, and he did seem genuinely interested. But as always, software development at the application level is a complex beast with budgets, release cycles, and marketing departments, so it can be a long journey from 2 people cold-pitching an idea to actually seeing something implemented in the software. So, if Fusion users are interested in seeing Cryptomatte support, I'm sure communicating that to the developers would help make it happen sooner.

In the meantime, I wholeheartedly encourage the use of Cryptomatte AOVs in Fusion using setups like what you posted, and (hopefully) the creation of a plugin using the API. Jonah and I are more than happy to answer any questions. I unfortunately can't really promise contributing dev time to a plugin project at this point, but it's DEFINITELY something we'd take an active interest in. I do suspect it would be a pretty easy project for someone who is already familiar with the Fusion API. For example, a similar discussion on BlenderArtists.org led to a prototype in about a day. I think a lot of people spend more time than that just making old-school mattes on one 3D project :)

https://blenderartists.org/forum/showth ... -released!

Anyway, I'd be very interested to hear people's thoughts, and I really hope this helps Fusion users feel less in the cold!

User avatar
French_Fry
Fusionista
Posts: 299
Joined: Tue Aug 05, 2014 8:04 am
Location: Nashville, TN
Been thanked: 2 times
Contact:

Re: Psyop's Cryptomatte In Fusion

#11

Post by French_Fry » Mon Oct 24, 2016 2:13 pm

Thanks for the extra info Andy! Awesome stuff you got going! :)

User avatar
Kristof
Fusionista
Posts: 431
Joined: Thu Aug 07, 2014 1:30 pm
Been thanked: 5 times

Re: Psyop's Cryptomatte In Fusion

#12

Post by Kristof » Wed Oct 26, 2016 2:32 pm

Hi Andy,

Thanks for reaching out. I wanted to respond sooner but work kept getting in the way. We've been eye-balling Cryptomatte with great interest from the get-go and discussed it here on the forum, along with the alternatives. There was a bit of back and forth between the company and Anders, initially asking him to help us understand what was being showcased in the papers as we were considering deploying something similar. He kinda caught us by surprise when he replied that he'd ask for the source code as he knew you guys, followed by a mail that the al implementation would be put on hold. Fortunately it was green lit after all and many thanks for doing that.

Nice to have a bit of context too.The Fusion devs used to hang around on the forum and Crypto' has been pitched for sure, but it is always hard to tell if ideas are being picked up and the consensus is that it is up to the developers of the tool to come up with a working version for a particular host application. As Nuke is the leader on the market, those users always get the goodies. I can understand that, but at the same time it makes it harder for the alternatives out there. And any market benefits from having healthy and thriving alternatives--just look at how the free version of Fusion put The Foundry under pressure which in turn culminated in a non-commercial Nuke. With that said, I would like to give the Guerilla people some props too. Their mission statement was clear: have openexr/id working in any host. In practice, it turns out that there is still some ground to cover although they'll get there.

The Fusion version of Crypto has been put on hold for now, but development will be picked up again soon-ish (I hope) by Cédric. Keeping in mind that he has yet to convert my little custom tool setup (the equivalent of your expression node in the gizmo) which code wise is not hard to do, I do have my doubts about the speed of such an implementation in a fuse. I was also worried about the rounding errors, but Cédric told me about the single precision function in the original code so maybe that will get us passed that hurdle. If needs be, it will be a plugin that hopefully will be picked up by BMD so that free users can have fun too.

Also thanks for offering help, Andy. And for coming up with a clever way so that we do get to have all those mattes in the end. :)

I'll keep on poking Cédric and make him join the thread sooner or later.

User avatar
Cedric
Posts: 39
Joined: Tue Sep 13, 2016 7:26 am
Location: Ghent
Been thanked: 4 times

Re: Psyop's Cryptomatte In Fusion

#13

Post by Cedric » Wed Oct 26, 2016 3:22 pm

Ok ok I give in Kristof.

First of all, thanks a lot Andy for tuning in and for your contribution to Cryptomatte.

Small introduction, my name is Cédric Duriau and I am a Pipeline TD responsible for database, production and compositing at Grid, and of course the amazing journey of porting Cryptomatte to Fusion.

As Kristof mentioned, so far I have been able to shove all the features in a single fuse. Upcoming inplementations are the conversion of float values too big for Fusion (UI) to display and the expressions to control the "matte only" visual feedback. I'm curious if I will be able to implement Kristof's comp functionalities above controlling the expressions inside the fuse whilst keeping it performant and responding fast. When these features will be looked at, I think we will clearly see if we will have to go the plugin way or keep it the fuse way.

I certainly will keep this thread up to date with my development. Andy, thanks a lot for offering your help, highly appreciated.

Cheers
Cédric

User avatar
ChristopherNodal
Fusioneer
Posts: 164
Joined: Mon Jan 11, 2016 1:42 pm
Been thanked: 12 times

Re: Psyop's Cryptomatte In Fusion

#14

Post by ChristopherNodal » Wed Oct 26, 2016 3:45 pm

This is a great thread already. I've subscribed.

Aside from the free argument, the nice thing about keeping it in a Fuse is code sharing. On this forum there has been at least one example of a Fuse becoming 25 times faster between versions, after one of the Fusion developers had taken a look at it and changed a few lines of code.

Please don't take this as me being presumptuous about your programming abilities. If anything i something like that would be due to lack of proper documentation.

Looking forward to your future updates,

JonahFriedman
Posts: 7
Joined: Tue Jul 12, 2016 5:55 am

Re: Psyop's Cryptomatte In Fusion

#15

Post by JonahFriedman » Thu Oct 27, 2016 11:07 am

Hello, Jonah here (the other creator of Cryptomatte). A Fusion implementation is directly in line with our goals Cryptomatte, which have always been to have an ecosystem of encoders and decoders and not just Nuke and Arnold. Please feel free to contact us directly with any questions, and we'll be on this thread too.

As far as I know nothing was ever waiting for a "green light", and certainly not on our end. Psyop's always been behind this project but things simply take time. After publishing our poster we organized a Cryptomatte Committee, which was made up of developers from various software packages. The goal was to create an open standard out of our in house tool with some peer review. When Anders reached out and told us he wanted to do an implementation in AlShaders, we volunteered our source code to make that easier. Anders, of course has a full time job, and AlShaders is a huge project in itself.

Anyway, for some nitty gritty implementation details, the keyable image we show users in Nuke is two channels of human-readable images (eg: CryptoObject.g, CryptoObject.b) and one channel of pure ID (eg: CryptoObject00.r), resulting in an image that still looks reasonable to the eye (even though the values in one channel are often huge or negative). Then we put the extracted matte into the alpha where it won't interfere with the other channels. When users use the color picker, thankfully Nuke doesn't mess with the floating point values in the ID channel that we care about, so we can use the value from the keyed color directly. If that didn't work, we would have to get the value directly from selected coordinates on the image, which would also work. I'm not sure if that helps you or not Cedric.

Beyond extracting mattes there's also selecting by name, and showing users their matte selections based on names, and so on, but one step at a time.