Environment Configuration

Introduction

This document contains specifications for the various file formats that Sgtk uses for its configuration and settings. Please note that this is a reference document which outlines all the various options and parameters available. For best practices on how to manage your configuration, please see the following document:

LINKBOX_DOC:53:Configuration Management best practices.

Shotgun Pipeline Toolkit Environments

Three major components exists in Toolkit:

• An engine provides a translation layer or an adapter between a host application (such as Maya or Nuke) and Sgtk Apps. Apps typically use python and PySide, and it is the responsibility of the engine to present the host application in a standardized fashion and for example add pyside on top of the host application if this doesn't exist already.

• An app provides a piece of business logic, it is essentially a tool that does something. Apps can be hand crafted to work in a specific host application, or they can be designed to run in more than one host application.

• A framework is a library which can be used by engines, apps or other frameworks. A framework makes it possible to more easily manage code or behaviour which is shared between multiple apps.

An environment file contains the configuration settings for a collection of engines, apps and frameworks. Such a collection is called an Environment. Sgtk launches different environments for different files or different people. You can for example have an environment for Shot production and environment for Rigging. Each environment is a single yaml.

Environment files are located at /<sgtk_root>/software/shotgun/<project_name>/config/env

The yaml file has the following basic format:

engines:
    tk-maya:
        location
        engine settings

        apps:
            tk-maya-publish:
                location
                app settings

            tk-maya-revolver:
                location
                app settings

    tk-nuke:
        location
        engine settings

        apps:
            tk-nuke-setframerange:
                location
                app settings

            tk-nuke-nukepub:
                location
                app settings
                
frameworks:
    tk-framework-tools:
        location
        framework settings

Each app and engine defined a bunch of settings. These settings need to correspond with the list of settings that the app expects. This list is defined in a manifest file called info.yml.

In addition to the various specific settings that needs to be defined for each item, each app, engine and framework also needs to define where its code is located. This is done using a special location parameter.

Code Locations

Sgtk currently supports app installation and management using the following sources:

   

Each of the above has a slightly different Location definition in the environment file.

Apps in the local file system

Pointing Sgtk to an app that resides in the local file system is typically something you do when you do development. Use the following syntax:

location: {"type": "dev", "path": "/path/to/app"}

The above syntax means that Sgtk will look for the app in the location specified by path. This normally works fine, however, if you need to do development on multiple platforms at the same time, you need to specify the path in a platform specific fashion. In that case, use the following syntax:

location: {"type": "dev", 
           "windows_path": "c:\\path\\to\\app", 
           "linux_path": "/path/to/app", 
           "mac_path": "/path/to/app"}

You can also include environment variables in the syntax -- this is handy if you want to set up a generic dev area for a small team. Typically, it would look something like this:

location: {"type": "dev", "path": "${HOME}/dev/sgtk/tk-nuke-myapp"}

Sometimes it can be handy to organize your development sandbox relative to a pipeline configuration. If all developers in the studio share a convention where they for example have a dev folder inside their pipeline configuration dev sandboxes, it becomes easy to exchange environment configs. You can achieve this by using the special token {PIPELINE_CONFIG} which will resolve into the local path to the pipeline configuration, like this:

location: {"type": "dev", "path": "{PIPELINE_CONFIG}/dev/tk-nuke-myapp"}

Since Sgtk does not know what version of the app is being run, it will return Undefined for an app referenced using the dev type. Sometimes, especially when doing framework development, it can be useful to be able to specify a version number. In that case, you can specify a specific version number and Sgtk will associate this version number with the app:

location: {"type": "dev", "path": "/path/to/app", "version": "v0.2.1"}

For production configurations, we recommend one of the other location specifiers.

Apps in git (and github)

If you manage your apps in a local git repository, this location syntax will let you work with multiple versions of an app in Sgtk easily! Use the following syntax in the environment file:

location: {"type": "git", "path": "/path/to/repo.git", "version": "v0.2.1"}
location: {"type": "git", "path": "user@remotehost:/path_to/repo.git", "version": "v0.1.0"}
location: {"type": "git", "path": "git://github.com/manneohrstrom/tk-hiero-publish.git", "version": "v0.1.0"}
location: {"type": "git", "path": "https://github.com/manneohrstrom/tk-hiero-publish.git", "version": "v0.1.0"}

As shown in the examples above, the git descriptor handles both local and remote repositories. It also handles private repositories in github, assuming that you have set up your ssh authentication correctly. On Windows, recommend that you use forward slashes.

Note! When you are using the git location, you need to have the git executable in the PATH in order for Sgtk to be able to do an app update check or app download. (the git exeuctable is, however, not needed during daily app execution, only when you are doing upgrades and installs).

When you install and app with this location, Sgtk will look for a repository in the path specified and download the tag version specified into its local app storage. Note that you only need access to the git repository when you perform app maintenance such as updates and installation. During normal operation, Sgtk will load the app from a local app storage. When looking for updates, Sgtk will retrieve all tags from the repository and run a version comparison, trying to determine the highest number.

Apps in the Sgtk Store

We also provide a special app store for Sgtk Apps and Engines. If you want to use an app from the app store, use the following syntax:

location: {"type": "app_store", "name": "tk-nukepublish", "version": "v0.5.0"}

Similar to the git location, Sgtk will locate the repository, download the specified tag into a local app storage and execute the code from there.

Manual management of Apps and Engines

We also provide a manual mode to make it easy to manage production installations of apps and engines without any automation. This is similar to the Sgtk Store approach described above, except it contains no automation whatsoever; it is up to you to install the code in the right place and no automated update checks can take place. The manual mode uses the following syntax:

location: {"type": "manual", "name": "tk-nuke-publish", "version": "v0.5.0"}

It will look for the code in a manual folder, so with the example above, Sgtk will look for the code in the install/apps/manual/tk-nuke-publish/v0.5.0 folder.

Disabling Apps and Engines

Sometimes it can be useful to temporarily disable an app or an engine. The recommended way of doing this is to to add a disabled: true parameter to the location dictionary that specifies where the app or engine should be loaded from. This syntax is supported by all the different location types. It may look like this for example:

location: {"type": "app_store", "name": "tk-nukepublish", "version": "v0.5.0", "disabled": true}

Alternatively, if you want an app to only run on certain platforms, you can specify this using the special deny_platforms setting:

location: {"type": "app_store", "name": "tk-nukepublish", "version": "v0.5.0", "deny_platforms": [windows, linux]}

Possible values for the deny_platforms parameter are windows, linux, and mac.

For App Developers: Manifest file

Apps, engines and frameworks each have a metadata file. It always resides in the root of the app or engine and is always named info.yml. This metadata file contains important information about the item:

• All the settings that the app or engine expects to have been specified in any environment where the item is loaded.
• All custom Shotgun fields that are required by the app or engine code.
• An optional field defining a display name for the app.
• An optional field to specify which engines this app works with.
• An optional set of frameworks that are required in order for this app to run. You can specify an exact framework version to use (e.g. v1.2.3) or you can track against a subset of versions (e.g. v1.x.x or v1.2.x).

Below is an example of what such a file may look like:

configuration:
    # Configuration settings definitions

requires_shotgun_fields:
    # Shotgun fields that this app expects

# List of engines with which this app works (optional)
supported_engines: ["tk-nuke", "tk-maya"]

# List of frameworks which are needed to run the app (optional)
frameworks:
    - {"name": "tk-framework-widget", "version": "v0.1.2"}
    - {"name": "tk-framework-tools", "version": "v0.x.x"}

# More verbose description of this item 
display_name: "Nicely formatted name"
description: "One line description of what the app does."

# optional url links to documentation and support
documentation_url: "http://intranet/path/to/sgtk_app_docs.html"
support_url: "http://intranet/pipeline_team/support.html"

# Required minimum versions for this item to run
requires_shotgun_version: "v4.2"
requires_engine_version: "v0.2.21"
requires_core_verson: "v0.12.4"
 
# what operating systems are supported?
# leaving this out, or blank means that all operating systems
# are supported. Valid choices are windows, linux and mac
supported_platforms: ["windows", "linux"]

# what parts of the context are required for this app?
# possible choices are project, entity, step, task, user
required_context: ["project", "entity", "user"]

Shotgun fields

If your app requires a particular custom field in Shotgun to operate correctly, you can add this to the info.yml manifest. Whenever the app is installed into a setup, toolkit will ensure that this custom field exists in shotgun.

Just create a requires_shotgun_fields in your manifest and add entries on the following form:

requires_shotgun_fields:
    # Shotgun fields that this app expects
    Version:
        - { "system_name": "sg_movie_type", "type": "text" }

Note how the fields are grouped by Shotgun entity type (in the example above Version). The following field types are supported:

• checkbox
• currency
• date
• date_time
• duration
• float
• list
• number
• percent
• status_list
• text
• timecode
• url

For more complex field types (such as entity and multi entity links), you need to set up creation via the post-install hook instead. (The post install hook is a special hook which runs at installation time and allows you to execute arbitrary code as part of the app installation process.)

Supported Engines

Normally, for apps that are designed to run in a single engine, this parameter can be omitted. However, if your app supports multiple engines, you can specify them as a list and Sgtk will validate that the engine which is loading the app is supported.

Frameworks

If a framework is being used by the app, declare it in the info.yml and Sgtk will keep track and make sure that it is installed and available. Once inside the app, use the call module_name = sgtk.platform.import_framework("tk-framework-name", "module_name") instead of a local import (e.g. from . import module_name).

Display name and Description

The optional display_name field defines the name that the user will see for the app. The optional description field is a brief one line description of what the app does.

Version constraints

If your app or engine requires specific versions of shotgun, the core or other things, you can specify this using the three parameters requires_shotgun_version, requires_core_verson and (for apps only) requires_engine_version.

Documentation

If you are developing an app for the Sgtk app store, no need to fill this in - the Sgtk Store manages documentation separately! But if you are building an in-house app for your studio, it can be useful to link it up to a wiki page or similar, so that it is easy for users to jump straight from the app to the documentation. In this case, just add a documentation_url setting to your app's info.yml manifest file:

documentation_url: "http://intranet/path/to/sgtk_app_docs.html"

This will be picked up and displayed in various locations in the Sgtk UIs.

Support

Similar to the documentation url above, it can be useful to connect an app to a url which lets users know how to get help when they have questions. If you have a separate system in your studio, (like an Request Tracker setup), and you want to make it easy for users to reach out from the app to this system, just define a support_url url:

support_url: "http://intranet/pipeline_team/support.html"

This setting is optional, and if left blank, the support url will automatically be the standard Sgtk support location.

Supported Operating Systems

If your app or engine only supports a particular operating system, you can define this using a supported_platforms parameter. Valid values are linux, windows and mac. This is an optional setting - omitting it or leaving it blank means that the app or engine will work on all platforms.

Required Context Fields

Sgtk has a Context which it uses to determine what the current Shot, Project or Asset is. Apps may require a particular feature to be present in the context - for example, a Loader App may require an entity to be present in order to show a list of items for the current Asset or Shot. The required_context settings help defining what fields are needed. Possible values are project, entity, step, task and user.

The Configuration Section

If an app requires a setting in its code, it should be defined in the info.yml. The Sgtk Core will keep tabs on all requested settings and make sure that they have all been defined and are valid before apps are launched. It will also use this metadata when performing upgrades and installations.

A configuration entry typically looks like this:

setting_name:
    type: some_type
    description: "description of the setting"
    default_value: general_default
    default_value_tk-nuke: nuke_default
    default_value_tk-maya: maya_default
    option: option_value
    option: option_value

Data types typically have specific optional settings. All data types and their options are outlines in the following sections.

A note about default values

In the configuration you can define default value for the various app parameters that you define. Please note that these default values are not being read at run-time, but only used to populate the environment configuration when an app is being installed or upgraded. Sgtk's configuration files are non-sparse, meaning that all parameters need to be populated in the configuration. A default value effectively means a value which will be set at install or upgrade time in case no other value is provided.

Default values per engine

It is possible to define defaults per engine using the following syntax:

setting_name:
    type: some_type
    description: "description of the setting"
    default_value: general_default
    default_value_tk-nuke: nuke_default
    default_value_tk-maya: maya_default
    option: option_value
    option: option_value

When the app is being installed or updated, toolkit will first look for a default value for the engine which the app is being installed into. If that is not found, it will look for a general default_value setting. If that is not found, the user will be prompted to manually supply a default value.

Values that are procedurally populated by hooks

For advanced use cases, it is possible to specify a configuration value as a special hook evaluator rather than as a template setting. This means that when you configure your environment, rahter than specifying an actual value for a setting to use with your app, you can specify a piece of code that returns the value to use. This makes it possible to create very complex settings logic. For more information, see the example_template_hook.py located in the core hooks area.

Simple Data Types

A number of simple data types are supported. These do not have any special options.

• str - a string value
• int - an integer
• float - a floating point value
• bool - a boolean, expecting values true or false

An example declaration for a simple data type may look like this:

debug_logging: 
    type: bool
    default_value: false
    description: Controls whether debug messages should be emitted to the logger 

The config_path data type

Use this when your app requires an external file that is part of the configuration. Typically, this settings type is used when you want to allow for a user to associate files with a configuration. These files can be icons or other resource files that should be part of the configuration. These paths should always be defined without an initial slash and using slashes as its path separator. Sgtk will translate it into a valid windows path.

output_icon:
    type: config_path
    description: A config centric path that points to a square icon png file. 

The tank_type data type

Use this when you want a setting which expects a Tank Type - these are typically used when publishing data to Shotgun. Value is a string matching TankType.code.

published_script_tank_type:
    type: tank_type
    description: The string value of the TankType used for published Nuke scripts.

The shotgun_entity_type data type

Value is a string that represents a Shotgun entity type like Task, Sequence, Shot.

entity_type:
    type: shotgun_entity_type
    default_value: Shot
    description: "The entity type to attach to" 

The shotgun_permission_group data type

Value is a string that represents the display name of a Shotgun permission group like Admin, Artist.

permissions_group:
    type: shotgun_permission_group
    default_value: Artist
    description: "Permissions group to use when performing the operation" 

The shotgun_filter data type

Value is a filter that will be passed to the shotgun api when performing a query (e.g. ["sg_status_list", "is", "cmpt"]). As shotgun filters can take multiple forms, no additional validation will be done on values of this type.

publish_filters:
    type: list
    values:
        type: shotgun_filter

The template data type

Value is a string matching an entry in templates.yml. Using the fields option you can tell Sgtk to check that the template that is being used in a configuration has the fields that the code expects. Using the allows_empty option (False by default) you can allow empty template values.

output_render:
    description: Render output location
    type: template
    fields: "context, name, channel, version, [width], [height], [eye]"
    allows_empty: True

When you specify the fields your app requires, it needs to be strict. Sgtk will make sure that the template that the user has chosen to configure the app with exactly matches the values specified in the fields string. Typically, the app will also pull in fields via the context object - you can specify this by using the special context parameter. Optional parameters are represented with [brackets] and the special token * indicates that this parameter accepts an arbitrary number of fields. For example:

• fields: name - only templates containing a single field name will be valid
• fields: context, name - templates containing exactly the number of fields that the context is able to resolve, and name, will be valid.
• fields: context, name, [width], [height] - same as previous but the two fields width and height can be present in the template at the user's discression.
• fields: name, * - name is required, the rest of the fields can be arbitrary.

A typical example illustrating the use of * is when you need to look for things in the scene which do not belong to the current context - for example if you are working on a shot and wants to scan for assets that have been imported into the scene. In this case, you cannot use the context since this is pointing at the current Shot.

input_templates_to_look_for:
    type: list
    description: List of templates to look for when scanning the scene for inputs. 
                A template listed here only needs to have a field called version. This field
                will be used in the comparison to determine what is out of date.
    allows_empty: True
    values:
        type: template
        fields: "version, *"

The hook data type

Hooks makes it possible to create flexible and powerful configurations for toolkit apps. A hook is a python file which contains a single class (deriving from a Hook base class or from other hook classes) and a number of pre-defined methods.

Hooks takes configuration beyond the retrival of a simple string or int value and allows apps to break out parts of the code that are likely to vary across facilities into code that can be customized, either by completely overriding it or by deriving from another hook class.

In your info.yml configuration manifest, a hook could look something like this:

hook_publish_file:
    type: hook
    default: "{self}/{engine_name}_publish.py"
    description: Called when a file is published, e.g. copied from a work area to a publish area.

The default value contains a special {engine_name} keyword and as the app is installed into an environment, it will resolve this into the current engine name. For example, after you have installed the above app into your Shot environment and nuke engine (by executing tank install_app shot tk-nuke tk-multi-myapp), the configuration would look something like this:

tk-nuke:
   
   tk-multi-myapp:
      setting_a: foo
      setting_b: bar
      hook_publish_file: "{self}/tk-nuke_publish.py"

In this case, the value {self}/tk-nuke_publish.py means that toolkit will go to the hooks folder for the current app and look for a file named tk-nuke_publish.py.

Resolving the path to a hook file

At runtime, the app code can use the command self.execute_hook_method("hook_publish_file", "publish") to execute hook code. This would locate the python file defined above, load it in and execute the publish() method. This makes it possible to create generic app code that let's toolkit handle where to load the actual hook file from - all that the app knows is which setting it is using.

The following different syntaxes can be used (inside an environment confiuration, as shown above), when you configure where toolkit should go look for the hook:

• hook_setting: {self}/path/to/foo.py -- This will look for the hook file in the current app's local hooks folder.
• hook_setting: {config}/path/to/foo.py -- This will look for the hook file in the current pipeline configuration's hooks folder.
• hook_setting: {tk-framework-perforce_v1.x.x}/path/to/foo.py -- This syntax will attempt to locate the framework instance inside the current environment and then look for the hook file inside that framework's hooks folder.
• hook_setting: {$HOOK_PATH}/path/to/foo.py -- This syntax will look for an environment variable named $HOOK_PATH and use this to resolve the path. Note that the {$ENV_VAR} syntax is tied to toolkit and not to the currently running operating system. This syntax will work on both windows, mac and linux.

In addition to the above forms, Toolkit also supports a couple of legacy syntaxes:

• hook_setting: foo -- Looks for the hook inside the current configuration - equivalent to {config}/foo.py.
• hook_setting: default -- Looks at he default_value specified in the info.yml manifest and uses this to resolve the path.

Basic structure of a hook

A default hook shipped with an app may look like this:

# example of default hook - resides in APP_ROOT/hooks/validation.py

import sgtk

HookBaseClass = sgtk.get_hook_baseclass()

class Validator(HookBaseClass):
    
    def validate_string(self, value):
        # default logic here...

    def validate_int(self, value):
        # default logic here...

Note the use of the special sgtk.get_hook_baseclass() call above in order to retrieve the base class. This notation makes it easy to partially override hooks using inheritance. Basically, the default hook that comes bundles with the app will get sgtk.Hook returned by sgtk.get_hook_baseclass(). This is what all hooks derive from and this base class contains some handy methods for doing stuff - most noteworthy is self.parent which will return the app object in the case of an app hook.

Customizing a hook

Now if you want to change the behaviour of how strings are validated in your pipeline you need to do the following:

1. Copy the validation.py from the app location into your configuration folder.
2. Change the app config in the environment from its default value validation_hook: {self}/validation.py to validation_hook: {config}/validation.py

Now we have a bunch of duplicated code in our config folder which isn't great! Because we are inheriting from the default hook, our custom hook only really needs to contain the modifications that we are making and not the entire logic. So go in and strip back our custom hook in the config folder:

# customized validation hook

import sgtk

HookBaseClass = sgtk.get_hook_baseclass()

class Validator(HookBaseClass):
    
    def validate_string(self, value):
        # call base class first
        HookBaseClass.validate_string(self, value)
        # now add our extra bit of validation
        if "LOLCAT" in value:
            raise Exception("No lol-cat stuff allowed!") 

Now, as the app code calls self.execute_hook_method("validation_hook", "validate_str", value=val_to_check), Toolkit will do the following:

1. Load the default hook class defined in the manifest ({self}/validation.py)
2. Load the hook defined in the custom setting {config}/validation.py. When it loads this hook, sgtk.get_hook_baseclass() will return the class object found inside the default hook, meaning that you are subclassing the default hook. This makes it possible to write the code above - which effectively extends the existing hook: Our custom extension does not alter the behaviour of the int validation, so whenever the app code needs to carry out an int validation, the default hook code will be used because of inheritance. Whenever the string validation is happening, our custom implementation calls out to the default hook to do its validation first, before carrying out an extra check we added.

This makes it possible to customize parts of hooks, while leaving a lot of the hook code in tact. This keeps customizations light weight, avoids code duplication and allows an app developer to push out bug fixes much more easily.

Multiple levels of inheritance in hooks

For even more complex scenarios, it is possible to declare an inheritance chain in the envionment config: validation_hook: {$STUDIO_ROOT}/studio_validation.py:{config}/project_validation.py. This example will have three levels of inheritance: first the built in hook, after that a studio level implementation and lastly a project level implementation.

Accessing a framework inside a hook

For even more complex hook setups, you can also access frameworks from inside your hooks.

class SomeHook(HookBaseClass):
    
    def some_method(self):
        
        # first get a framework handle. This object is similar to an app or engine object
        fw = self.load_framework("my-framework-library_v123.x.x")
        
        # now just like with an app or an engine, if you want to access code in the python
        # folder, you can do import_plugin
        module = fw.import_module("some_module")

Note how we are accessing the framework instance my-framework-library_v123.x.x above. This needs to be defined in the currently running environment, as part of the frameworks section:

engines:
  # all engine and app defs here...

frameworks:
 # define the framework that we are using in the hook
 my-framework-library_v123.x.x:
    location: {type: dev, path: /some/path}

The list data type

Value is a list, all values in the list must be of the same data type. You must supply values dict that describes the data type of the list items.

entity_types:
    type: list
    values: {type: shotgun_entity_type}
    default_value: [Sequence,Shot,Asset,Task]
    description: "List of Shotgun entity types where this 
                 Sgtk action should be visible on the Actions menu." 

Optionally you can also specify an allows_empty option if an empty list is a valid value:

tank_types:
    type: list
    allows_empty: True
    values: {type: tank_type}

You would then be able to specify an empty list in your environment configuration:

apps:
    tk-multi-loader:
        tank_types: []

The dict data type

Value is a dictionary, keys are always strings, values can be of mixed types.

In info.yml, optionally provide an items dict. Used to mark what keys must be in the config, default values, and type info for the values which will be used for validation.

write_nodes:
    type: list
    description: "A list of dictionaries in which you define the Sgtk write nodes that 
                 are supported in this configuration. Each dictionary entry needs to have
                 the following keys: name - a descriptive name for this node. file_type -
                 the file type to use for the renders (exr, cin, dpx etc). This will be 
                 passed to the Nuke write node when rendering. settings - configuration
                 settings for the given file type, as a dictionary. This too will be 
                 passed to the write node when rendering. Lastly, you need two entries 
                 named render_template and publish_template 
                 - these control the locations where data is written to at various stages 
                 of the workflow. These templates need to include the fields 
                 name, channel and version, and can optionally include the fields width and
                 height which reflect the image resolution of the render. If you are doing
                 stereo rendering and want to use Nuke's %V flag, include an eye field. 
                 This will be replaced by %V in the paths when the Sgtk write node 
                 computes them."
    allows_empty: True
    values:
        type: dict
        items:
            name: { type: str }
            file_type: { type: str }
            settings: { type: dict }
            tank_type: { type: tank_type }
            render_template:
                type: template
                required_fields: [name,channel,version,SEQ]
                optional_fields: [width,height,eye]
            publish_template:
                type: template
                required_fields: [name,channel,version,SEQ]
                optional_fields: [width,height,eye]

Complex Examples

A list of dictionaries, each with keyes type, tank_type, template

movies:
    type: list
    description: list of dictionaries. This list should contain one entry for every type
                 of movies that needs to be rendered at publish time. Each list item
                 should be a dictionary with the keys type and template, defining the name
                 of the movies to make and the template to base its path on. This template
                 should have a subset of the fields available to the render template.
    allows_empty: True
    values:
        type: dict
        items:
            type: { type: str }
            tank_type: { type: tank_type }
            template:
                type: template
                required_fields: [name,channel,version]
                optional_fields: [width,height,eye]

A valid setting could look like this:

movies:
- {tank_type: Quicktime, template: shot_quicktime_full_res, type: standard}
- {tank_type: Quicktime, template: shot_quicktime_half_res, type: half_res}