Small Team folder structures

Moderator: SecondMan

RBemendo
Fusioneer
Posts: 93
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Small Team folder structures

#1

Post by RBemendo » Thu Jun 14, 2018 5:40 am

For years I've been organizing projects by department, but I'm finding more often than every before, departments are blurring. 3D modelers are part of animation, animation is part of compositing, composting is part of lighting and editorial, etc.... I'm re-thinking my folder structure organization on the server to account for this change in departments.

My first thought is to change the organization by shot. Everything that would go into a single shot is in one root folder. If the naming structures are the same based on shot rather than by department, I think I would be able to have minimal sub-folders because file types would dictate organization.

Anyone have experience with small teams (4-6) that can share any insight on approaching this?

User avatar
miaz3
Fusioneer
Posts: 173
Joined: Sat Jan 03, 2015 1:43 am
Location: Angoulême / France
Contact:

Re: Small Team folder structures

#2

Post by miaz3 » Thu Jun 14, 2018 7:00 am

Hi,
as ftrack or shotgun approch ?

It can give ideas (PLEX - open source pipeline) :
https://github.com/alexanderrichtertd/plex

User avatar
ISOTRON
Fusioneer
Posts: 149
Joined: Mon Aug 04, 2014 11:54 am
Location: Germany, Munich
Contact:

Re: Small Team folder structures

#3

Post by ISOTRON » Thu Jun 14, 2018 7:15 am

yes, your are right, more and more, departments are blurring..... furthermore clients needs also (they going back to the edit, after seeing the pre-final shots )
BUT u cant organize it only by shots. What u will do, if u have f.E. one 3d Asset for 3 different shots. Duplicate the stuff?
ftrack and shotgun are to big, to much stuff, to much features, for small team... my opinion

but maybe this , like plex: https://prism-pipeline.com/

User avatar
Midgardsormr
Fusionista
Posts: 582
Joined: Wed Nov 26, 2014 8:04 pm
Location: Los Angeles, CA, USA
Been thanked: 45 times
Contact:

Re: Small Team folder structures

#4

Post by Midgardsormr » Thu Jun 14, 2018 7:56 am

I prefer to organize by shot. Global assets go to an Assets folder at the same level as the Shots folder, and are otherwise treated as though they were a shot:
Project/Shots/ShotName
Project/Assets/AssetName

An Asset's internal folder structure is similar to a shot's, so any pipeline scripting can apply equally well to both. Assets can include the obvious 3d models, but also 2d matte paintings, comp templates, show-specific Fusion macros, and so on.

For small teams it doesn't really make sense, in my opinion, to organize by department because you're likely to have generalists who can work both ends. It also makes archiving a bit more work since you'd have to go to two places to get everything you need to pull a shot out of cold storage.

RBemendo
Fusioneer
Posts: 93
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Re: Small Team folder structures

#5

Post by RBemendo » Thu Jun 14, 2018 8:28 am

Thanks Miaz3 and ISOTRON,

I'll take a look at plex. I'd never heard of it before this. Looks promising, but most of my software packages are not supported. Looks like the standalone might be a good starting place to see how he's structuring everything.

I've looked into Shotgun and Ftrack before, but I agree. It's much too big for a small team. The duplication is what I'm concerned about approaching everything as a shot base approach. Reading through the WSL forums, this is a very similar topic to the Generation topic "anyone found a good Generation alternative" viewtopic.php?f=15&t=2169


I guess I'm hopeful that scripts will be developed for Resolve15 that will significantly help the naming conventions and "department" blurring. For now I'll take a look at Plex and more than likely stay with the department approach.

If there's any more tips or suggestions you have, I'm open to new ideas.

User avatar
theotheo
Fusioneer
Posts: 226
Joined: Thu Aug 07, 2014 8:35 am
Been thanked: 46 times

Re: Small Team folder structures

#6

Post by theotheo » Thu Jun 14, 2018 9:07 am

Resolve 15 doesnt give a shit about anything not resolve, don't expect it to work with external naming and folder structure like the ones below. Everything is in one database, while shots and shows tend to modular in their structure.

Bryan is spot on tho. Follow his advice.

Here's a simplified structure that we've been using. Its procedurally built and scales pretty well. tested on shows with 300+ shots

Code: Select all

./reference
./reference/creativeref
./reference/shootref
./review
./film
./film/zz999_9999
./film/zz999_9999/work
./film/zz999_9999/work/nuke
./film/zz999_9999/work/nuke/scripts
./film/zz999_9999/work/nuke/misc
./film/zz999_9999/work/houdini
./film/zz999_9999/work/houdini/hip
./film/zz999_9999/work/houdini/misc
./film/zz999_9999/work/fusion
./film/zz999_9999/work/fusion/scripts
./film/zz999_9999/work/fusion/misc
./film/zz999_9999/work/blender
./film/zz999_9999/work/blender/scripts
./film/zz999_9999/work/blender/misc
./film/zz999_9999/camera
./film/zz999_9999/pipeline
./film/zz999_9999/images
./film/zz999_9999/images/renders
./film/zz999_9999/images/sourceplates
./film/zz999_9999/images/sourceplates/pl01
./film/zz999_9999/images/sourceplates/pl01/v001
./film/zz999_9999/images/mastercomp
./film/zz999_9999/images/precomp
./film/zz999_9999/geometry
./incoming
./pipeline
./pipeline/nuke
./pipeline/houdini
./pipeline/fusion
./pipeline/fusion/Defaults
./pipeline/fusion/Macros
./pipeline/fusion/Macros/stx
./pipeline/fusion/Macros/stx/nuke
./pipeline/fusion/Macros/stx/snippets
./pipeline/fusion/LUTs
./pipeline/fusion/LUTs/OCIO
./pipeline/fusion/Comps
./pipeline/fusion/Docs
./pipeline/fusion/Docs/images
./pipeline/fusion/Filters
./pipeline/fusion/Brushes
./pipeline/fusion/Fuses
./pipeline/fusion/Fuses/Flow
./pipeline/fusion/Fuses/Filter
./pipeline/fusion/Fuses/Creator
./pipeline/fusion/Fuses/external
./pipeline/fusion/Fuses/Matte
./pipeline/fusion/Fuses/Miscellaneous
./pipeline/fusion/Fuses/Modifier
./pipeline/fusion/Stamps
./pipeline/fusion/Modules
./pipeline/fusion/Modules/Lua
./pipeline/fusion/Guides
./pipeline/fusion/Layouts
./pipeline/fusion/Plugins
./pipeline/fusion/Scripts
./pipeline/fusion/Scripts/BinView
./pipeline/fusion/Scripts/Bin
./pipeline/fusion/Scripts/Utility
./pipeline/fusion/Scripts/Utility/EYEON
./pipeline/fusion/Scripts/Comp
./pipeline/fusion/Scripts/Comp/_stx
./pipeline/fusion/Scripts/Comp/_stx/dev
./pipeline/fusion/Scripts/Comp/EYEON
./pipeline/fusion/Scripts/Comp/Code Snippets
./pipeline/fusion/Scripts/ReelView
./pipeline/fusion/Scripts/Views
./pipeline/fusion/Scripts/Job
./pipeline/fusion/Scripts/Slave
./pipeline/fusion/Scripts/Tool
./pipeline/fusion/Scripts/Tool/EYEON
./pipeline/fusion/Scripts/Tool/EYEON/Tool Alignment
./pipeline/fusion/Default
./pipeline/fusion/Bins
./pipeline/fusion/Library
./pipeline/fusion/Tools
./pipeline/fusion/Profiles
./pipeline/fusion/Profiles/Default
./pipeline/fusion/Settings
./pipeline/fusion/Config
./edits
./edits/internal
./edits/client
./production
./production/csv
./assets
./assets/3d_assets
./assets/3d_assets/empty_asset
./assets/3d_assets/empty_asset/cache
./assets/3d_assets/empty_asset/textures
./assets/3d_assets/empty_asset/hip
./assets/3d_assets/empty_asset/comp
./assets/3d_assets/empty_asset/hda
./assets/3d_assets/empty_asset/geometry
./assets/3d_assets/empty_asset/misc
./assets/3d_assets/empty_asset/shaders
./assets/2d_assets
./assets/2d_assets/elements
./assets/2d_assets/hdri
./assets/2d_assets/lens_grids
./assets/2d_assets/nuke_templates
./assets/2d_assets/charts
./assets/2d_assets/set_reference
./outgoing
./dev
./dev/testDev
./dev/testDev/work
./dev/testDev/work/nuke
./dev/testDev/work/nuke/scripts
./dev/testDev/work/nuke/misc
./dev/testDev/work/houdini
./dev/testDev/work/houdini/hip
./dev/testDev/work/houdini/misc
./dev/testDev/work/fusion
./dev/testDev/work/fusion/scripts
./dev/testDev/work/fusion/misc
./dev/testDev/work/blender
./dev/testDev/work/blender/scripts
./dev/testDev/work/blender/misc
./dev/testDev/camera
./dev/testDev/pipeline
./dev/testDev/images
./dev/testDev/images/renders
./dev/testDev/images/sourceplates
./dev/testDev/images/sourceplates/pl01
./dev/testDev/images/sourceplates/pl01/v001
./dev/testDev/images/mastercomp
./dev/testDev/images/precomp
./dev/testDev/geometry

Its a good base to build on

RBemendo
Fusioneer
Posts: 93
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Re: Small Team folder structures

#7

Post by RBemendo » Thu Jun 14, 2018 9:11 am

Midgardsormr, that's a good approach. My server config has always had project files and render files in different locations due to file size limitations on the server. But keeping everything together does seam more logical when it's not a department approach like you're suggesting. I could also script our backups to not include certain file types, like EXR for example. I might give your organization a try on my next project:
Project/Shots/ShotName
Project/Assets/AssetName

I'll have to reconfig our servers to accommodate this, but it does seam like a good approach. Thanks!

User avatar
Midgardsormr
Fusionista
Posts: 582
Joined: Wed Nov 26, 2014 8:04 pm
Location: Los Angeles, CA, USA
Been thanked: 45 times
Contact:

Re: Small Team folder structures

#8

Post by Midgardsormr » Thu Jun 14, 2018 9:26 am

We did the same thing at Muse for the same reason for a long time, but with a third server for deliverables. It's very nice to finally have projects on only one volume. And I don't even deal with backups and archives!

You may not want to exclude by filetype; that could have unforeseen consequences, like excluding HDRi and matte paintings. Rather, rules based on location might be a better idea. For instance, if you don't want to save renders or precomps, and they live at:
project/shots/shotname/elements/renders/
project/shots/shotname/elements/precomps/
You could just exclude those folders to prevent the backup from picking them up. We do something similar at Muse with our simulation caches. Everything goes into a temp folder that is excluded, but if it's a very long sim that we're happy with, the artist can move it to a permanent cache that does get backed up.

RBemendo
Fusioneer
Posts: 93
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Re: Small Team folder structures

#9

Post by RBemendo » Thu Jun 14, 2018 9:47 am

Bryan, that's a good idea to have backups rules based... I hopefully would have come to that conclusion as well, because there are workflows that I use EXRs for more than just renders, so thanks for catching that right away!

Yea, one server does sound nice. I've never gotten to work that way, so I've never tested it out. Glad to hear I wasn't the only one that ran into this.... and after awhile it's really hard to think of different approaches when you're painted into a 2 server set up.

Theotheo, That is one massive folder structure! That might be a bit more advanced than I'm used to but I'll dig through it and see if there is a way I can adapt it to my workflow. Thanks for the resource!

I think I've been along time organizer of "scripts, macros, templates, dev, and pipeline tools"as a global approach rather than a project approach. I can have many projects simultaneously going on and I end up reusing a lot of elements directly from my Studio_stockLibrary. Maybe it's time to rethink that strategy and become more project based rather than globally based.

Many thanks for the insights here, it's got me rethinking my techniques. Which is always good not to stay stagnant.

User avatar
Midgardsormr
Fusionista
Posts: 582
Joined: Wed Nov 26, 2014 8:04 pm
Location: Los Angeles, CA, USA
Been thanked: 45 times
Contact:

Re: Small Team folder structures

#10

Post by Midgardsormr » Thu Jun 14, 2018 11:22 am

RBemendo wrote:
Thu Jun 14, 2018 9:47 am
I think I've been along time organizer of "scripts, macros, templates, dev, and pipeline tools"as a global approach rather than a project approach. I can have many projects simultaneously going on and I end up reusing a lot of elements directly from my Studio_stockLibrary. Maybe it's time to rethink that strategy and become more project based rather than globally based.

For the most part we keep our tool and template development global to the facility, but there are times when it's nice to have a macro built for a specific set of shots or an effect that's a sort of trademark look for a show. For instance, I'm right now working on a greenscreen driving sequence right now where the client provided us with 4-way footage from a camera car. I built a macro that color balances and synchronizes the four plates. It's useless for any other show, so it would make sense to keep it in the project. We're not currently set up that way, but we're building out some pipeline to make it feasible. Mostly it's a matter of getting per-project pathmaps to work the way we want.

RBemendo
Fusioneer
Posts: 93
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Re: Small Team folder structures

#11

Post by RBemendo » Thu Jun 14, 2018 1:21 pm

That makes sense..... yea I'll need to dig more into per-project pathmaps and see how I can use that rather than trying to make most things work globally.

Do any of your artists pull down from the server to local SSDs/RAIDs for performance increases. Or is everything pulling from the server directly? I'm getting close an upgrade cycle and trying to weigh my options for performance. I'm already using a 10gig Ethernet network, although but I've been contemplating having faster drives locally for compositing.

User avatar
theotheo
Fusioneer
Posts: 226
Joined: Thu Aug 07, 2014 8:35 am
Been thanked: 46 times

Re: Small Team folder structures

#12

Post by theotheo » Thu Jun 14, 2018 1:28 pm

Also, you can split your servers quite easily by using symlinks if you're on *NIX, so it all looks like one folder but its actually spread out over many servers., makes it very easy to have lets say renders and sim files on a slower disk, and published renders and comp material / plates on a faster one. You can do this with any type of folder structure.

User avatar
Midgardsormr
Fusionista
Posts: 582
Joined: Wed Nov 26, 2014 8:04 pm
Location: Los Angeles, CA, USA
Been thanked: 45 times
Contact:

Re: Small Team folder structures

#13

Post by Midgardsormr » Thu Jun 14, 2018 1:34 pm

We each have a local SSD for caching, and sometimes an artist will pull a heavy sequence to their local storage for performance. Usually that's followed by "Hey, why is my job failing on the farm?" an hour later. :D

Since everyone is demanding UHD or better these days, our network infrastructure is really groaning. And at least one client is in 12-bit, which makes it just that much worse. 37MB per frame for 12-bit UHD DPX!

Jpeg proxies are less than 1MB, though, even at UHD, so one of my near-term goals is to try to build in better proxy handling.

RBemendo
Fusioneer
Posts: 93
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Re: Small Team folder structures

#14

Post by RBemendo » Thu Jun 14, 2018 1:47 pm

I really appreciate your knowledge and your sharing of this information. I too am in the same thought of better proxy workflows. I've been running proxies for our editorial team, and it has really helped speed up versioning and trying different ideas rather than waiting on realtime playback. I would think proxies might be better handled on the second server so network rendering wouldn't effect the speed of the proxies. Fortunately I already moved our pipeline a way from multi-layered EXRs a while ago so individual 3D render channels wouldn't be too hard to set up with proxies. Just need to automate the color profiles to jpg from linear EXR, but shouldn't be too hard to script.

RBemendo
Fusioneer
Posts: 93
Joined: Fri Dec 12, 2014 11:32 am
Been thanked: 1 time

Re: Small Team folder structures

#15

Post by RBemendo » Thu Jun 14, 2018 8:00 pm

Theotheo, I'm currently on Windows Server 2012. I've tried using windows file shortcuts in the past to try and keep everything in "one" structure. It does work OK, but moving to ZFS file system has been on my mind lately for dependability and less likely data rot.

I think a big shift for me is to have one file structure for editorial, and have a separate structure for VFX/MoGraph within that folder structure(shot based approach). This certainly would require massive changes in the server structure, but I'd been thinking of a shot based approach for some time now. This discussion just helps me re-confirm my ideas about the shot based approach. Now it's time to flesh out the particulars for implementing such a structure.

In larger workflows, is it a VFX editor or assistant editor the one that is doing shot assignments and "starting" the VFX portion of the workflow?