Small Team folder structures

Moderator: SecondMan

User avatar
theotheo
Fusionista
Posts: 300
Joined: Thu Aug 07, 2014 8:35 am
Been thanked: 10 times

Re: Small Team folder structures

#16

Post by theotheo » Fri Jun 15, 2018 12:57 am

Midgardsormr wrote:
Thu Jun 14, 2018 1:34 pm
We each have a local SSD for caching, and sometimes an artist will pull a heavy sequence to their local storage for performance. Usually that's followed by "Hey, why is my job failing on the farm?" an hour later. :D

Since everyone is demanding UHD or better these days, our network infrastructure is really groaning. And at least one client is in 12-bit, which makes it just that much worse. 37MB per frame for 12-bit UHD DPX!

Nuke does this pretty well with localization. It keeps the localization up to date with sync from the server, but the hashing is locally tied. So if you render the script on the farm it picks up the network references and not the local ones. Saves so much bandwith and speed.

Also, its usually never the network itself that gets sticky, more often than not its the disks not able to keep up with the throughput. I'm surprised at how much can be pushed around with just 1GigE. Let alone 10.

RBemendo wrote:
Thu Jun 14, 2018 8:00 pm
Theotheo, I'm currently on Windows Server 2012. I've tried using windows file shortcuts in the past to try and keep everything in "one" structure. It does work OK, but moving to ZFS file system has been on my mind lately for dependability and less likely data rot.

I run one of our test servers on ZFS, its been super stable and fast, very impressed with how easy it is to set up pools and resources. Well worth checking out. Also, setting up a pipeline with linux is super rewarding and fun.
RBemendo wrote:
Thu Jun 14, 2018 8:00 pm
I think a big shift for me is to have one file structure for editorial, and have a separate structure for VFX/MoGraph within that folder structure(shot based approach). This certainly would require massive changes in the server structure, but I'd been thinking of a shot based approach for some time now. This discussion just helps me re-confirm my ideas about the shot based approach. Now it's time to flesh out the particulars for implementing such a structure.

Shot based workflows are the norm for shot based production, but thats why you also run a /dev/ and /asset/ folder so non sequence specific work can happen outside the context of individual shots, but be pulled from when needed. Assetize, reuse and avoid dupes as much as possible
RBemendo wrote:
Thu Jun 14, 2018 8:00 pm
In larger workflows, is it a VFX editor or assistant editor the one that is doing shot assignments and "starting" the VFX portion of the workflow?

In VFX companies its the prod crew that does shot creation and first assignments with supes. On film production side its often the vfx editor/supe/assist that does the shot setup for VFX.

But for naming convention, you really want to innoculate yourself against outside variables like conforming to different naming conventions, structures, colorspace handling etc. You can have your own setup the way you want for all shows as long as you can reverse back to whatever expected format/name is required at the delivery stage. Do this to avoid having to change your pipe to account for stupid client surpises.

User avatar
miaz3
Fusioneer
Posts: 204
Joined: Sat Jan 03, 2015 1:43 am
Location: Angoulême / France
Been thanked: 1 time
Contact:

Re: Small Team folder structures

#17

Post by miaz3 » Fri Jun 15, 2018 4:32 am

I was talking mostly about an approach like shotgun or ftrack for the folder hierarchy (eg: deep structure).
Conversely, whether it is a small, medium, large company I think it should not reduce to its strict minimum.
Maybe I was "formatted" to put everything in the right folder, but it's so much simpler and scalable.
Even if you do not have much to put in such or such department I think it is important to keep a rigor on this point.

The structure is the skeleton, the keystone, it is better to tidy up than not enough.
it's only my point of view.

Added in 44 seconds:
Nuke does this pretty well with localization. It keeps the localization up to date with sync from the server, but the hashing is locally tied. So if you render the script on the farm it picks up the network references and not the local ones. Saves so much bandwith and speed.
+1

User avatar
Midgardsormr
Fusionator
Posts: 1002
Joined: Wed Nov 26, 2014 8:04 pm
Answers: 1
Location: Los Angeles, CA, USA
Been thanked: 47 times
Contact:

Re: Small Team folder structures

#18

Post by Midgardsormr » Fri Jun 15, 2018 7:01 am

theotheo wrote:
Fri Jun 15, 2018 12:57 am
It keeps the localization up to date with sync from the server, but the hashing is locally tied.

I don't understand this sentence. Should that be caching? Or is there another definition of hashing that I'm not aware of?

User avatar
theotheo
Fusionista
Posts: 300
Joined: Thu Aug 07, 2014 8:35 am
Been thanked: 10 times

Re: Small Team folder structures

#19

Post by theotheo » Mon Jun 18, 2018 2:09 am

Midgardsormr wrote:
Fri Jun 15, 2018 7:01 am
theotheo wrote:
Fri Jun 15, 2018 12:57 am
It keeps the localization up to date with sync from the server, but the hashing is locally tied.

I don't understand this sentence. Should that be caching? Or is there another definition of hashing that I'm not aware of?
Sloppy wording. It does a checksum on the files to see if there's a need to re-sync/update files from the central server/storage.

User avatar
Chad
Fusionator
Posts: 1381
Joined: Fri Aug 08, 2014 1:11 pm
Been thanked: 8 times

Re: Small Team folder structures

#20

Post by Chad » Mon Jun 18, 2018 6:42 am

Each Nuke instance can't be doing the checksum, that would take more time than just copying the files. Would need to be server or filesystem based.

RBemendo
Fusioneer
Posts: 147
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Re: Small Team folder structures

#21

Post by RBemendo » Thu Jun 28, 2018 9:11 am

Finally getting around to approaching this again and re-structuring the servers. I've experimenting with naming conventions within this new folder structure as well. Moving the structure sure as much work as the one I first built if not harder, as I keep finding caveats to structures I try. I'm exploring the collaborative "version control script" as well as the old Eyeon script "change path".

For the most part I've used similar naming conversions to the structure found here:
http://www.designimage.co.uk/vfx-naming-convention/

If anyone has another other tips/resources/scripts etc that would help my transition to changing to a shot based approach it's greatly appreciated. Thanks again for all the help so far.

User avatar
Midgardsormr
Fusionator
Posts: 1002
Joined: Wed Nov 26, 2014 8:04 pm
Answers: 1
Location: Los Angeles, CA, USA
Been thanked: 47 times
Contact:

Re: Small Team folder structures

#22

Post by Midgardsormr » Thu Jun 28, 2018 10:49 am

I've been thinking on this topic quite a lot the past couple of weeks. I've been developing a script to fetch CG renders and pre-merge the additive buffers (Diffuse, Reflect, SSS, etc) in order to make it easier and faster to quickly set up comps with large numbers of layers. I'm not sure if I'll get permission to share it, but if I do, a shot-centric organization principle is built in. As I refactor, I may give some thought to what might need to be changed in order to support pipelines in which scene files are on a different volume than renders.

Something else to consider is the semantic meaning of separator characters. In our pipeline, like the one shown in the example link, we use the underscore character to delimit different functional parts of a file or folder name. Hyphens are indicative of sub-categories within a function. For example, we might have an outputs like these:

proj_010_010_matte-characterA_v001.####.png
proj_010_010_matte-characterB_v001.####.png

From most scripts' perspective, 'matte-characterA' is one piece, but if we needed to distinguish between characterA and characterB outputs, the hyphen serves as a marker. I use the same logic internally.

One thing we do that is differently from the designimage example is we use a two-part shot code: sequence and shot. That makes it easier to filter for all of the related shots in a given sequence. We also frequently use a two-part project code since we work mostly on episodics: TW_605 = Teen Wolf Season 6, episode 05. The optional existence of a season folder in the structure means that we can't count our folders using the project folder as the root—most of the time I count from the comp file down to the shot folder and treat that as the root of my tree. I don't write much that operates at the project level, though. I'm not sure what kind of logic @FredP uses on the project management side.

RBemendo
Fusioneer
Posts: 147
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Re: Small Team folder structures

#23

Post by RBemendo » Thu Jun 28, 2018 12:47 pm

Your thought on fetching CG renders and pre-merging everything is exactly what I'm hoping to accomplish when moving to this shot based approach. Making broad templates for repeatable type shots. This thread discussion first entering my mind after watching Image Engines presentation of Jurassic world. Where there were many similar shots that needed repeatable set ups in the compositing stage. That presentation can be found here:

The hyphen is an interesting approach. And certainly one I had not thought of. This could add another complexity to my logic, but it's great to have this feedback to keep thinking the best way to organize. However in your example, I would have thought the 'character would be a subset of the shot and the matte a subset of the character. So I think I would have structured it as:

proj_010_010_CharacterA-matte_v001
proj_010_010_CharacterB-matte_v001

More to keep exploring ......

User avatar
Midgardsormr
Fusionator
Posts: 1002
Joined: Wed Nov 26, 2014 8:04 pm
Answers: 1
Location: Los Angeles, CA, USA
Been thanked: 47 times
Contact:

Re: Small Team folder structures

#24

Post by Midgardsormr » Thu Jun 28, 2018 2:37 pm

It depends on how you're using it, I suppose. In our work, the client frequently asks for matte outputs as part of the deliverable, so it makes sense to sort by type of output first. So they'd see this:

Code: Select all

Outputs\
    comp\
    element-a\
    element-b\
    matte-a\
    matte-b\

For matte passes rendered out of CG, they'd come with the rest of the AOVs, so the compositor gets this:

Code: Select all

Renders\
    CharacterA\
        Camera1\
            v001\
                Oid010203\proj_010_010_CharacterA-Camera1_Oid010203_v001
    CharacterB\
        Camera1\
            v001\
                Oid010203\proj_010_010_CharacterB-Camera1_Oid010203_v001

Oid010203 means "Object IDs 01, 02 and 03".

RBemendo
Fusioneer
Posts: 147
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Re: Small Team folder structures

#25

Post by RBemendo » Fri Jun 29, 2018 6:05 am

That makes sense. For most of my workflows I'm basically delivering to editorial which can be me many times, so my output folder is usually editorial based rather than composite based and I just work from the render folder when needed in editorial. But I do like differentiating them from within the shot based structure though.

One thing I've found with some of the scripts I'm using for updating loaders, is the path map string ends up being different from shot to shot. So once the shot is set up and multiple versions start being rendered out the scripts work great. But starting new shots, I have to manually set up each one. This certainly gets a bit tedious with 5-8 multipass layers for each element in a comp.

One thing I've found is if I rely on folder names for naming convention and keep all the actual rendered files named the same between shots the scripts work quite well. I'm afraid this could snowball in to disaster if all the files end up basically being named the same between shots and episodes, and the folder name is the only indication of what's inside.