Improving Python Development for TouchDesigner Projects
Topics
- Types of Python in TouchDesigner
- IDEs and why they’re great
- Problems with using IDEs for TouchDesigner
- Making a TouchDesigner project work in an IDE
- Where we go from here
As the use of Python in TouchDesigner becomes more widespread and important, it’s worth taking a look at how we write it.
Done well, Python can produce reliable, clearly structured, powerful projects.
Done poorly, Python can produce tangled webs of broken code that are hard to understand and even harder to maintain.
Types of Python usage in TouchDesigner
Use cases for Python in TD fall into two main categories:
- Small snippets of code
- Complex systems and infrastructure
Small snippets of code
There are plenty of places where a little bit of Python is used to calculate a value, or make some changes to a DAT, and so on. For these cases it often makes sense to just write it directly within TouchDesigner, or using the configured external text editor. That’s basically all that’s needed for situations when you’re editing a single piece of code at a time and there isn’t much direct interconnection between your chunk of Python and the rest of your system.
Something like Sublime Text, Atom, or Visual Studio Code works well for this sort of thing.
If set up properly, it’s also possible to use those particular editors as lightweight IDEs.
Examples of this type of code:
- Parameter expressions
- Simple
Script DAT|SOP|CHOPscripts - Simple callback scripts for
Parameter|DAT|CHOP|Panel Execute DAT
Complex systems and infrastructure
When your Python expands beyond the scale of small self-contained scripts, the effectiveness of inline editing and embedding in DATs declines.
Examples of this type of code:
- Scripts that construct or manipulate collections of other OPs
- Complex UI / application logic
- Parsing / writing complex data formats
IDEs and why they’re great
Before getting into the specifics of how to work on this kind of code for TouchDesigner, it’s worth taking a more generic look at IDEs.
IDEs (Integrated Development Environments) are code editors that provide a variety of features supporting development of projects in various programming languages. They often include compilers, debuggers, and better handling of code spread across multiple files.
They’ve been around for a while (the first version of Visual Studio was released in 1997, and its predecessors arguably go back to the 1980s). There’s been a lot of… shall we say, “enthusiastic discussion” throughout the programming world about their value, which continues even to this day. But for purposes of this article I’ll assume that you aren’t looking to boost your nerd credibility by writing everything in emacs just to prove that you can. 😜
Some benefits of IDEs
- Catching syntax errors. An IDE will detect code that is structurally invalid for the language and highlight it, with helpful details about which part is wrong. This is especially important if you’re not adept in the language, but even the most experienced programmers can miss a parenthesis.
- Autocompletion. When you type
thing.it'll show you a list of the potential attributes and methods thatthinghas and automatically fill it in as you continue typing.
- Static analysis. This is sort like super-powered syntax error checking, but instead of finding things that are syntax errors, it finds things that are technically valid Python but are likely to cause problems. For example, referring to variables that don’t exist, or calling a function with the wrong number of parameters. PyCharm (and the whole IntelliJ family of IDEs) are especially good at this.
- Finding references. Often you’ll want to figure out where (and if) a function (or class, variable, etc) is being used. For some cases a simple text search is sufficient (find all occurrences of “doSomeAwesomeStuff” in your project files). But in other cases you need something more intelligent. For example, if you have a method named
format, there might be lots of occurrences of that text in your project which aren't actually uses of that method. Or there could even be multiple classes that have methods with that name and only care about one specific class'sformatmethod. IDEs handle this by understanding what things actually refer to instead of just finding chunks of text. - Refactoring. Along the lines of finding references, you’ll sometimes want to make changes to something that is used in a bunch of places. If you’re just renaming a method, maybe find/replace is good enough. But what if you want to reorder the parameters? IDEs provide tools for automating those tasks, and doing so in a way that is (relatively) safe from errors.
- Version control. You’re already using Git for everything, right? If so, you are probably working with a separate Git client (such as GitHub Desktop, GitKraken, SourceTree, or the command line tools). IDEs generally come with built-in tools for Git (and for other version control systems).
Which IDE to use
Use PyCharm. There are others, but PyCharm is amazing. It’s part of the JetBrains IntelliJ suite of IDEs. There’s a free community edition which has all the features that you’ll need for TouchDesigner projects. It has amazing support for code analysis and automated warnings and suggestions. Plus they have their own super nice font (and an amazingly detailed page about that font)!
VS is one of the longest living IDEs out there and it has really solid support for a ton of languages including Python. It is a commercial product, but there’s a free community edition. It’s sort of Windows-only. There’s Visual Studio for Mac, but it’s sort of a separate thing.
It’s important to note that Visual Studio Code is not the same as Visual Studio. They share the name basically just for marketing purposes, much like how JavaScript has almost literally nothing to do with Java.
Eclipse is another elder of the IDE world. PyDev is set of extensions for it that add Python support. You can easily get into arguments about whether it’s a good IDE, but it’s pretty solid. It’s free and open source.
VSCode, Sublime Text, Atom
It’s possible to set these up in a manner that has some degree of IDE features, but they’re a bit limited.
Problems with using IDEs for TouchDesigner projects
There are a few things that are unique to TouchDesigner projects that interfere with IDE features. There are strategies for dealing with these problems discussed below.
Problem 1: Code hidden in binary files
The main benefits of IDEs are based on the fact that they have an actual understanding of the language and your code as code rather than just text. They can tell when you write thingName.upper() that thingName should be a reference to an object that's in-scope and should have a method named upper() which doesn't require any parameters.
They gain that understanding by parsing through the code files in your project. But they can only do that for files that they can find and read. TouchDesigner .toe and .tox files are binary files that nothing aside from TouchDesigner itself knows how to read. So all that code that you stuck in a DAT somewhere in your project is hidden away inside a binary file that can’t be parsed.
Problem 2: Built-In objects
The TouchDesigner runtime provides many objects that are globally available to all code in a project, some of which even behave differently depending on the context where they’re used. One of the big examples of this is op('foo'), which can be used anywhere, and how it interprets the path 'foo' depends on where that code sits. There are tons of other examples, including ext, parent(), constantCHOP, mod, ipar, and so on.
These things are critical to using Python in TouchDesigner and you will find yourself using them constantly. But an IDE doesn’t know anything about them. They are defined internally by TouchDesigner’s C++ code, so there isn’t a Python file that the IDE can look at.
Making TouchDesigner projects work well in IDEs
There are a number of techniques that can be used to work around the problems that TD poses for IDEs. Many of these are also just good practices to follow for larger projects regardless of whether you’re using an IDE.
Create a project
IDEs operate on a project rather than just a file. A project is generally a directory containing the contents of the project along with some configuration files that the IDE will create and update. In PyCharm, you can set up a project from an existing directory and it’ll generate a special .idea/ subfolder with its configuration files.
The .idea/ folder name is because PyCharm is actually a modified version of IntelliJ IDEA.
Once you’ve created your project, to get the full benefits of the IDE, you need to point it at an installation of Python. The IDE will show some warnings with prompts to do this.
For TouchDesigner on Windows, it’s found in:
C:\Program Files\Derivative\TouchDesigner\bin\python.exe (or ...\TouchDesigner099\... depending on which version you're using).
For TouchDesigner on macOS, it’s found in:
/Applications/TouchDesigner.app/Contents/Frameworks/Python.framework/Versions/Current/bin/python3 (or .../TouchDesigner099.app/... depending on which version you're using).
That will tell the IDE which version of the Python language to use, and will include the specific set of libraries that TouchDesigner includes (such as numpy).
Keep code in external files
Keep the bulk of your code in external .py files. You can make a Text DAT synchronize its contents with an external file using the File and Sync to File parameters. Doing that for your python code will allow you to edit those files in an IDE, and it will allow the IDE to parse and understand the code in those files. Exactly which code should be in external files is discussed below.
Extensions
Extensions are a huge topic, which I’m not going to fully cover here, but there’s a lot of good material available about them. They’re a great way to package up the code relevant to a particular component, and make it accessible from elsewhere in the project.
Shared library modules
TouchDesigner supports using pulling in modules of shared code from several different sources using a few different techniques. There’s good documentation for this on the wiki.
mod (a.k.a. "Module on demand"): One of the most common technique is to use mod.whatever.foo(), which will use the globally available mod object provide by TouchDesigner lookup logic to find a DAT named whatever and call the foo() function that it contains. This is great for use in parameter expressions and small snippets of code. But it's Touch-specific magic that IDEs don't understand.
import: The other approach is to use Python import whatever statements to use the same logic to find and load a module, and then use whatever.foo() to make use of that module. This only works in multi-line scripts, and cannot be used for parameter expressions. But IDEs understand it (as long as they know where to look for modules).
Shared library modules vs component-specific code
This is also a huge topic, but for purposes of this article, there are two principles in software development that in some ways are opposed to each other:
- Reuse code when possible. Instead of having the same chunk of code appear basically the same throughout a project, move it into a shared location and have other things refer to that.
- Keep things limited to the scope where they’re needed. It’s much easier to modify things that are only used within a specific context vs things that are used all over the place.
Striking a balance between these two is a bit of an art, but a good guideline to follow is:
- Start by putting things within the specific component where they’re needed, perhaps as methods of an extension class that’s defined inside a particular component.
- If you have a specific need to use it somewhere else, then pull it up to a level that contains both of those things. That might mean creating a shared module that’s specific to some portion of the project that contains the components that need it, or it might mean a project-wide shared module.
Dealing with small code snippets in large projects
Even if the bulk of your code is in extensions and shared modules, there will still be plenty of situations where you need to have a small chunk of very specific code sitting in a DAT somewhere. The main case for this is callback functions for things like Parameter Execute DAT or Script CHOP.
It may make sense to keep those in their own external files, but sometimes that’s overkill. But regardless, if they are particularly complex, they should just be calling out to some function defined in a shared module or an extension, rather than having all the code inline:
def cook(scriptDat):
ext.ThingManager.BuildThingTable(
scriptDat, someParam=scriptDat.par.Whatever,
anotherThing=op('other_stuff'))Stub files
All that’s great for getting the IDE to understand your code. But what about all the built-ins provided by TouchDesigner? The IDE still has no idea what op() is. This means that it will frequently show lots of errors in your code like Unresolved reference "op". You can either train your mind to magically identify those an differentiate them from actual bugs in your code, or you can find a way to tell the IDE about those objects.
This is where things get a bit strange.
You can create stub files that the IDE will parse and understand, but aren’t actually used at runtime. The IDE will often be smart enough to know that from _stubs import * means that the things defined in your _stubs file might be available in a file that uses that import, but it's often not smart enough to know that this code won't ever actually load that file:
# noinspection PyUnreachableCode
if False:
# noinspection PyUnresolvedReferences
from _stubs import *The comments in there are to tell the IDE that it should ignore the fact that there’s code that will never actually run, and to force it to ignore when your file isn’t actually currently making use those imports.
But where do these stub files come from?!
Typing them out. Manually.
Yes, it is tedious. But fortunately I’ve done much of the work for you already!
https://github.com/optexture/td-components/tree/master/lib/_stubs
To use those, pull down the contents of that folder and store them somewhere like lib/_stubs/, and then in PyCharm, right click lib/ and choose "Mark Directory As > Sources Root". Then stick the code snippet above at the top of each file where you want to use the stubs.
Within those files, I’ve written out the structures of many of the built-in objects that touch provides. Some of it I’ve managed to semi-automate by parsing the wiki, but a lot of it is manual.
I’ve also taken some of the widely used modules that are included in TouchDesigner in DATs and pulled them into their own files, which I update when updating to a new version of TD.
Type hinting
One of the main things you may notice in the _stubs/__init__.py file is the use of Python type hints.
Python is a dynamic language, which means that a given variable could potentially contain any type of object. That’s great for a lot of things, but it’s terrible for detecting mistakes. IDEs have gotten increasingly smarter about handling it, but it’s still limited.
To address that, type hints let you mark what type various things are supposed to be. The IDE can then easily tell when you’re trying to use the wrong type of value. This is another huge topic, but suffice it to say that they’re great for keeping your code error-free and for documenting how things are supposed to work.
Configuring your IDE to recognize the stubs
To use the stub files, you need to tell your IDE / editor how to find them.
Option 1: Copy the _stubs/ folder into your site-packages folder
The site-packages folder in Windows is in C:/Users/YOU/AppData/Roaming/Python/Python33/site-packages (that “33” might be different depending on the version)
The default Python setup in macOS might not have a dedicated site-packages folder outside of the TouchDesigner.app package, which would not be a good place to put custom files since it gets overwritten when you update TD.
Make sure that your editor is pointed at the Python interpreter in the TouchDesigner program folder. That should automatically include the relevant site-packages folder.
This approach will make it visible to all of your projects.
Option 2: Copy the _stubs/ folder into your project
You can either put it in the root of the project’s folder, or in a sub-directory such as lib/.
In your IDE / editor, make sure that the parent of the _stubs/ folder is included in the Python path list for your project. In VSCode, there’s a python.pythonPath workspace setting for this. In PyCharm, there’s a list in the project structure settings.
Importing and using the stub files
Once the IDE / editor knows where to look for a package called _stubs, you can use an import like the one described above:
# noinspection PyUnreachableCode
if False:
# noinspection PyUnresolvedReferences
from _stubs import * Or you could use an import like this:
try:
op
except NameError:
from _stubs import *That will cover the main global things exposed by the TD environment such as op and Par.
For the other non-global modules, you need to use an import like this:
try:
TDF = op.TDModules.mod.TDFunctions
except NameError:
from _stubs import TDFunctions as TDFBecause the symbols aren’t global at runtime, you need to set up an alias like TDF that points to op.TDModules.mod.TDFunctions. But since your editor won’t understand what that is, it will fall back on the second part that uses the stub file for that module.
That approach also works for things like TDJSON.
Where we go from here
There are some things that Derivative is doing / can do to help out with all of this.
There has long been discussion of a text-based project format. That would help for doing things like basic text searches within a project, though it likely wouldn’t help an IDE actually parse that code. It sounds like a huge project though so it may be a while before that’s available.
It may be possible to generate interface stub files based on the native C++ built-in objects provided by TouchDesigner, or maintain stub files that are updated when the native code is changed. If not, the community can maintain them.
