Set up an on-set library

VFX On-set Data Wrangling Workflow

This page describes a folder-driven workflow for VFX on-set data acquisition. The goal is to make every file self-describing at ingest by extracting tags and categories from the source path automatically. End of shooting day you only have to copy the captured data in the folders and ingest the entire shooting day folder into Das Element.

How To

Use a consistent folder structure on set so that the ingest process can automatically tag incoming material. During ingest, the pre_ingest_load hook reads the file path, splits it into folders, applies each folder as a tag and uses the last folder as the category. Each ingested file receives consistent, searchable metadata without manual entry. The folder structure becomes the source of truth for tags, while the final folder defines the category used for browsing and filtering. Once you are back at the studio, move all data to the production network storage and convert the database to a production database for other users or third-party VFX vendors.

Workflow

1. Capture on set: save captured data into the agreed folder structure

2. Preserve folder names: avoid renaming folders during copy. The folder names are used as metadata

3. Start ingest: the ingest system calls pre_ingest_load before each file is loaded

4. Parse the path: the hook splits the file path into folder components

5. Assign tags: every folder component becomes a tag on the file

6. Assign category: the final folder component becomes the file category

7. Continue ingest: the file is loaded with the generated category and tags attached
   

Example folder structures

{YYYYMMDD}_day-{shooting day}}/{type}/{asset name or scene}/{data type}

Example:
20261005_day-001/assets/chrAlienMother/photo/IMG_0123.cr3
20261005_day-001/slates/sc12_5/reference/video.mov

Here’s an best practice folder structure for on-set data acquisition work:

⬇️ vfx-on-set-project-structure.zip

• 20261005_day-001

  • assets

    • chrAlienMother

      • hdri

      • photo

      • photogrammetry

      • reference

      • scan

      • texture

  • slates

    • sc12_5

      • hdri

      • photo

      • photogrammetry

      • reference

      • scan

      • screen_capture

      • set_reference

      • texture

      • video

      • witnesscam
        

Get categories and tags

Naming recommendations

• Use stable shoot-day folders, for example 20261005_day-001.

• Use broad department or data-type folders, for example assets, slates, plates, hdris, or witness_cam.

• Use asset, scene, slate, or setup folders where useful, for example chrAlienMother or sc12_5.

• Use the final folder as the intended category, for example photo, reference, scan, or survey.

• Avoid spaces and ambiguous names. Prefer lowercase, camelCase, snake_case, or production-approved naming conventions.
  

Category hierarchy

The category hierarchy is based directly on the folder hierarchy. Each folder level narrows the context for the material.

pre_ingest_load hook behavior

The hook pre_ingest_load.py runs when files are added to the ingest window, before anything is ingested. It receives data per file with at least path and optionally tags, category, mapping, colorspace, media_type, and metadata, and returns the modified list.

This implementation only looks at folders below the hierarchy root folder (ROOT_FOLDER). Each folder name below it becomes a tag, and the final folder becomes the category. Files outside ROOT_FOLDER are left unchanged.

The ready-to-use hook files are available on GitHub in the resources > on-set examples folder (pre_ingest_load.py and pre_hierarchy_load_folder.py).

def get_category_id(folders: list) -> str:
    # return same the category ID as the categories in the hierarchy file 
    return hashlib.md5("/".join(folders).encode("utf-8")).hexdigest()

Full pre_ingest_load.py example:
https://github.com/das-element/resources/blob/main/scripts/hooks/examples/on-set/pre_ingest_load.py

import hashlib
import re

# name of the hierarchy source folder
ROOT_FOLDER = "/path/to/on-set-folder"

# folder names matching this pattern also add a short scene tag: "sc010 hallway" -> "sc010"
SCENE_PATTERN = r"^sc\d"


def get_category_id(folders: list) -> str:
    """Return the category ID das element generates for a folder source.

    folders: folder names from below ROOT_FOLDER down to the category,
             e.g. ["day-001", "assets", "photo"]
    """
    return hashlib.md5("/".join(folders).encode("utf-8")).hexdigest()


def get_folders(path: str) -> list:
    """Return the folder names between ROOT_FOLDER and the file.

    "/path/to/on-set-folder/day-001/assets/photo/img_0038.png"
    -> ["day-001", "assets", "photo"]
    """
    parts = path.replace("\\", "/").split("/")[:-1]
    if ROOT_FOLDER not in parts:
        return []
    return parts[parts.index(ROOT_FOLDER) + 1 :]


def get_tags(folders: list) -> list:
    """Turn folder names into tags: "_sc010_hallway" -> ["sc010 hallway", "sc010"]"""
    tags = []
    for folder in folders:
        tag = folder.lstrip("_").replace("_", " ")
        tags.append(tag)
        if re.match(SCENE_PATTERN, tag):
            tags.append(tag.split(" ")[0])
    return tags


def main(*args, logger=None) -> list:
    items = args[0]

    for item in items:
        folders = get_folders(item.get("path", ""))
        if not folders:
            continue

        tags = item.get("tags") or []
        for tag in get_tags(folders):
            if tag not in tags:
                tags.append(tag)

        item["tags"] = tags
        item["category"] = get_category_id(folders)

    return items

Related pages

• Core Concept

• Library settings

• Libraries