Inter-process Communication from DCC Tools 01

Introducing a workaround for using Python 3 assets from Python 2-based systems like Maya and Houdini

Introduction

In the 2020 CEDEC session "Large-scale Game Development Environment with Python ~ Cyllista Game Engine Development Case Study ~", they introduced examples of integration between Python 3-based engines and modules with DCC tools constrained to Python 2. They mentioned subprocess and multiprocessing, but didn't go into detail about how to actually use them. In this article, I'll show you how to put these into practical use. I should clarify that I am not associated with the session, speakers, or organizations (disclaimer) . Also, since this article is about "utilizing Python 3 assets," it will only be half useful if you don't have such assets.
In this article, I'll explore three approaches based on processing complexity:

Fixed processes with limited flexibility
Receiving data and performing processing
Mutually exchanging data Systems for exchanging data across multiple processes are generally called `inter-process communication` (interprocess communication, abbreviated as IPC) . For information on the various forms of inter-process communication, please refer to other pages such as Wikipedia.

Preparation

To use external Python, set up an environment in advance (miniconda, venv, or whatever works for you, but I don't recommend using a directly installed environment). Also, this article won't cover basic explanations of subprocess, multiprocessing, or pickle, so if you're unfamiliar with them, please check their references first.

Simple Example

Let's start by running the following in the script editor. (This article uses Maya code examples, but the basic format is similar in other environments.) Using subprocess, we launch Python 3 as a separate process from the Python within DCC.

import subprocess
# Adjust paths to match your environment
py_exe = r"C:\bin\python\win\python38.exe"
# The -c option allows you to provide the command to execute as a string
cmd = '''{py_exe} -c "print 'hello python3' "'''.format(**locals())
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in proc.stdout.readlines():
    print(line)

You should get a result like this:

  File "<string>", line 1
    print 'hello python3' 
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print('hello python3')?

This error is intentional. In Python 3, print changed from a statement to a function, so you need to wrap it in parentheses. This error actually confirms that we're running Python 3. So let's try again:

import subprocess
py_exe = r"C:\bin\python\win\python38.exe"
cmd = '''{py_exe} -c "print('hello python3')"'''.format(**locals())
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in proc.stdout.readlines():
    print(line)

Now you should see:

hello python3

Also, you can directly check the version of the execution environment with:

cmd = '''{py_exe} --version"'''.format(**locals())

If you're not familiar with using python.exe from the command line, there are several other startup options available, so try python -h to see them.

A Slightly More Complex Example: Passing Data

Was that too simple? Let's look at a more complex example where we pass data to the script at startup:

script_to_be_executed = "C:/hoge.py"
cmd = '''{py_exe} "{script_to_be_executed}"'''.format(**locals())
...or
module_to_be_executed = "cool_module"
cmd = '''{py_exe} -m "{module_to_be_executed}"'''.format(**locals())

This executes a file or module as a script, so you can do:

script_to_be_executed = "cool_module"
some_arguments = "hoge"
cmd = '''{py_exe} "{script_to_be_executed}" {some_arguments}"'''.format(**locals())

This allows you to pass string data as command line arguments. Since you can pass strings, you can also exchange data via JSON.

But what if you want to pass more complex data? You'll need some form of serialization/deserialization, and in this case, we'll use pickle. (There are some pitfalls, but we'll put those aside for now.) Let's also switch from command line arguments to using standard input.

Here's the code on the DCC side:

# DCC side
import pickle
import subprocess
hoge = 1
fuga = 2
py_exe = r"C:\bin\python\win\python.exe"
scriptpath = r"C:\yamahigashi.dev\content\text\dcc\2020-09-22-dcc-interprocess-communication01\pickle_test01.py"
cmd = '''{py_exe} {scriptpath}'''.format(**locals())
proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out, err = proc.communicate(input=pickle.dumps({"hoge": hoge, "fuga": fuga}))
print(out)
print(err)

And here's the external code being executed:

# ------------------------------------------------------------------------------
# External side
import sys
import pickle
s = sys.stdin.buffer.read()
d = pickle.loads(s)
print(d["hoge"])  # => 1
print(d["fuga"])  # => 2

This way, you can send objects from the DCC side using standard input (stdin). Of course, you can also send back results via standard output.

In this example, we serialized and deserialized a simple built-in type: a dictionary of integers. What about user-defined types? We'll divide this article into two parts and cover that in the next one. Please look forward to the next article.

Notes on Using Japanese or Unicode

As a side note, I want to caution about a common pattern I've seen when researching subprocess that can easily cause errors in Japanese environments:

# Avoid this
exe = "python"
script = "bad_habit.py"
arg = "hogemage"
subprocess.Popen([exe, script, arg])

Instead of passing a list to Popen, do this:

# Do this instead
exe = "python"
script = "better_approach.py"
arg = "hogemage"
cmd = "{exe} {script} {arg}".format(**locals())
subprocess.Popen(cmd)

There doesn't seem to be much difference, and you might think the first approach is more readable and easier to understand. However, if any list element contains Unicode, it can cause errors. You often need to process Japanese files unintentionally, and it's usually in someone else's environment.

This can cause problems when connecting with other systems. While it's not ideal to work with unpredictable types, sometimes you can't avoid it. In such cases, concatenating strings in advance makes errors occur in the application rather than within libraries like subprocess, making them easier to notice. That's why I prefer this approach.

While there's no room for error in the example above, when dealing with paths that might contain Japanese in Python 2, it's better to handle them internally as bytestrings and only make them displayable when showing them. Here's an example:

def _fsencoding():
    # type: () -> Text
    """Get the system's filesystem encoding. On Windows, this is always
    UTF-8 (not MBCS) but cp932 in maya.
    """
    encoding = sys.getfilesystemencoding() or sys.getdefaultencoding()
    if encoding == 'mbcs':
        # On Windows, a broken encoding known to Python as "MBCS" is
        # used for the filesystem. However, we only use the Unicode API
        # for Windows paths, so the encoding is actually immaterial so
        # we can avoid dealing with this nastiness. We arbitrarily
        # choose UTF-8.
        if not cmds.about(q=True, batch=True):
            encoding = "cp932"
        else:
            encoding = "utf8"
    return encoding
WINDOWS_MAGIC_PREFIX = u'\\\\?\\'
def bytestring_path(path):
    # type: (Union[bytes, Text]) -> bytes
    """Given a path, which is either a bytes or a unicode, returns a str
    path (ensuring that we never deal with Unicode pathnames).
    """
    # Pass through bytestrings.
    if isinstance(path, bytes):
        return path
    # On Windows, remove the magic prefix added by `syspath`. This makes
    # ``bytestring_path(syspath(X)) == X``, i.e., we can safely
    # round-trip through `syspath`.
    if os.path.__name__ == 'ntpath' and path.startswith(WINDOWS_MAGIC_PREFIX):
        path = path[len(WINDOWS_MAGIC_PREFIX):]
    # Try to encode with default encodings, but fall back to UTF8.
    try:
        return path.encode(_fsencoding())
    except (UnicodeError, LookupError):
        return path.encode('utf8')
def displayable_path(path, separator=u'; '):
    # type: (Union[bytes, Text], Text) -> Text
    """Attempts to decode a bytestring path to a unicode object for the
    purpose of displaying it to the user. If the `path` argument is a
    list or a tuple, the elements are joined with `separator`.
    """
    if isinstance(path, (list, tuple)):
        return separator.join(displayable_path(p) for p in path)
    elif isinstance(path, unicode):
        return path
    elif not isinstance(path, bytes):
        # A non-string object: just get its unicode representation.
        return unicode(path)
    try:
        return path.decode(_fsencoding(), 'ignore')
    except (UnicodeError, LookupError):
        return path.decode('utf8', 'ignore')
exe = bytestring_path("python")
script = bytestring_path("bad_habit.py")
arg = bytestring_path("hogemage")
cmd = "{exe} {script} {arg}".format(**locals())
subprocess.Popen(cmd)