Magic xpi IronPython Connector

Magic xpi is a functional, extensible, and quick to learn integration platform. (It is also unfair to compare it to BizTalk so I won’t.) I ran into a couple limitations recently, such as no regular expression support and no way to generate unique identifiers. You know, basic functionality that no developer should live without. So I decided to leverage IronPython and give CLR control back to the devs. SOUCE CODE LINK

First, we need to configure the connector using the Connector Builder application. This will create a new folder under Runtime\addon_connectors. Before we leave this screen, make sure to generate the example UI and Runtime projects.

Connector Builder User Interface
Your Connector Config

Open the Runtime and UI solutions in Visual Studio and lift them from .NET 4.0 to .NET 4.5.

Example Projects
Target .NET 4.5

I also like to change my build output path so I don’t have to dig deep for my binaries.

Runtime Debug Output Path
UI Debug Output Path

Next you will want to add IronPython, IronPython.StdLib, and Newtonsoft.Json NuGet packages to the Runtime project.

Runtime Project NuGet Packages

Before going into the source code, let me explain how I understand the Data Mapper connector interface to work. If you were to build these projects as they are right now, you will end up with several prompts and a Data Mapper when you configure the connector in a Magic xpi Studio flow. The Data Mapper component presented to you is actually the Blob that will be sent into the invoke(StepGeneralParams StepParams) method of the Runtime project. You can retrieve this Blob’s bytes from StepParams.PayloadOBject. In Studio, the Data Mapper component overlaid a schema over this Blob for you. So inside the invoke method you will need to deserialize it back into an instance of that schema. Same thing needs to happen if you decide to have a Blob as an output variable from your connector code back to Studio. You will need to serialize the results you want to send back to Studio and provide a schema that can overlay the Blob in Studio in a subsequent Data Mapper.

I thought about how to make this connector flexible so that I could pass in multiple variables, regardless of what Python script I would be running. After some trial and error, I came up with this.

Input Blob
PythonExpression With Newlines
Python Expression Without Newlines
Python Expression With Flow Variable

And the contents of the Python script.

X = A + B
from System import Guid
GUID = Guid.NewGuid().ToString()

What makes this approach flexible is that I can still leverage the Expression Editor to create variables for the Python script to access. In this example, we are setting variable A to 1000 and variable B to 337 prior to script executing. When the Python script executes it will assign the sum of A and B to a new variable X. The script will also create a unique identifier and assign it to a new variable GUID. You could also bypass the actual script file and put the entire script contents into the Python expression field.

No Script File, Just an Expression

The vanilla IronPython implementation to execute the expression and script file is simple.

var pyEngine = Python.CreateEngine();
var pyScope = pyEngine.CreateScope();
pyScript = pyEngine.CreateScriptSourceFromFile(@"C:\temp\");
pyEngine.Execute(@"A=1000\nB=337\n"), pyScope);
var X = pyScope.GetVariable("X");
var GUID = pyScope.GetVariable("GUID");

After the connector code executes, we want access to variables X and GUID back in Studio. We need to define a Blob variable with the OUT direction in our UI project.

PythonOutput Variable

Next, we need to prompt the developer to select a variable in the project to assign the result to.

PythonOutput Variable Assignment

Since we do not want to limit on the number of variables that can be returned, we will construct a JSON schema that will return all variables in IronPython’s current scope, which will end up looking like this. The JSON schema file is Runtime\addon_connectors\IronPython\schema\PyOutput.json.

Output Blob

To make this happen, we simply iterate all variables in the IronPython scope, serialize, and assign to our output Blob. This Blob is now available in the variable we chose previously while configuring the connector in Studio.

Assign all Python variables to output Blob

In Studio the entire flow consists of two nodes. First passes flow variables into the Python script, and executes the Python script. Second maps Python results back into flow variables.

And here is an overview of the flow within Studio.

IronPython Flow

Here is the connector and test project SOURCE CODE.

UPDATE: Explicit Variable and Search Paths

I wanted to retrieve events from the Windows Event Log and return them back to Studio from the Python execution results. Generic K,V pairs would not work in this case. The representation of each event contains its Id, Timestamp, Provider Name, and Event Data.

The simplest way to get the events back to Studio was to serialize the Python dictionary to a JSON string and assign it to a variable. In order to retrieve the specific variable I added the Explicit Python Output expression to the connector’s configuration. If you leave this expression empty then the Python output will fall back to the generic K,V pairs, retrieve all Python variables within the executing scope, and you can overlay PyOutput.json schema over the Blob. If you set it to a specific variable name that exists in the Python execution scope, then only that variable’s (in this case ‘X’) value will be retrieved and assigned to the output Blob. You can then choose to overlay an appropriate schema over the Blob back in Studio.

Explicit Python Output (name the variable you want back)

While testing this, I ran into a “JSON module not found” error. The fix was to install the latest IronPython on the server and set additional search paths for the IronPython Engine. Each line in Runtime\addon_connectors\IronPython\schema\paths.conf will be added to IronPython’s search paths during execution.

Additional Search Paths