Magic xpi IronPython Connector
Magic xpi is a functional, extensible, and quick to learn integration platform. (It is also unfair to compare it to BizTalk so I won’t.) I ran into a couple limitations recently, such as no regular expression support and no way to generate unique identifiers. You know, basic functionality that no developer should live without. So I decided to leverage IronPython and give CLR control back to the devs. SOUCE CODE LINK
First, we need to configure the connector using the Connector Builder application. This will create a new folder under
Runtime\addon_connectors. Before we leave this screen, make sure to generate the example UI and Runtime projects.
Open the Runtime and UI solutions in Visual Studio and lift them from .NET 4.0 to .NET 4.5.
I also like to change my build output path so I don’t have to dig deep for my binaries.
Next you will want to add IronPython, IronPython.StdLib, and Newtonsoft.Json NuGet packages to the Runtime project.
Before going into the source code, let me explain how I understand the Data Mapper connector interface to work. If you were to build these projects as they are right now, you will end up with several prompts and a Data Mapper when you configure the connector in a Magic xpi Studio flow. The Data Mapper component presented to you is actually the Blob that will be sent into the
invoke(StepGeneralParams StepParams) method of the Runtime project. You can retrieve this Blob’s bytes from
StepParams.PayloadOBject. In Studio, the Data Mapper component overlaid a schema over this Blob for you. So inside the
invoke method you will need to deserialize it back into an instance of that schema. Same thing needs to happen if you decide to have a Blob as an output variable from your connector code back to Studio. You will need to serialize the results you want to send back to Studio and provide a schema that can overlay the Blob in Studio in a subsequent Data Mapper.
I thought about how to make this connector flexible so that I could pass in multiple variables, regardless of what Python script I would be running. After some trial and error, I came up with this.
And the contents of the Python script.
X = A + B
from System import Guid
GUID = Guid.NewGuid().ToString()
What makes this approach flexible is that I can still leverage the Expression Editor to create variables for the Python script to access. In this example, we are setting variable
A to 1000 and variable
B to 337 prior to
my.py script executing. When the Python script executes it will assign the sum of
B to a new variable
X. The script will also create a unique identifier and assign it to a new variable
GUID. You could also bypass the actual script file and put the entire script contents into the Python expression field.
The vanilla IronPython implementation to execute the expression and script file is simple.
var pyEngine = Python.CreateEngine();
var pyScope = pyEngine.CreateScope();
pyScript = pyEngine.CreateScriptSourceFromFile(@"C:\temp\my.py");
var X = pyScope.GetVariable("X");
var GUID = pyScope.GetVariable("GUID");
After the connector code executes, we want access to variables
GUID back in Studio. We need to define a Blob variable with the
OUT direction in our UI project.
Next, we need to prompt the developer to select a variable in the project to assign the result to.
Since we do not want to limit on the number of variables that can be returned, we will construct a JSON schema that will return all variables in IronPython’s current scope, which will end up looking like this. The JSON schema file is
To make this happen, we simply iterate all variables in the IronPython scope, serialize, and assign to our output Blob. This Blob is now available in the variable we chose previously while configuring the connector in Studio.
In Studio the entire flow consists of two nodes. First passes flow variables into the Python script, and executes the Python script. Second maps Python results back into flow variables.
And here is an overview of the flow within Studio.
Here is the connector and test project SOURCE CODE.
UPDATE: Explicit Variable and Search Paths
I wanted to retrieve events from the Windows Event Log and return them back to Studio from the Python execution results. Generic K,V pairs would not work in this case. The representation of each event contains its Id, Timestamp, Provider Name, and Event Data.
The simplest way to get the events back to Studio was to serialize the Python dictionary to a JSON string and assign it to a variable. In order to retrieve the specific variable I added the Explicit Python Output expression to the connector’s configuration. If you leave this expression empty then the Python output will fall back to the generic K,V pairs, retrieve all Python variables within the executing scope, and you can overlay PyOutput.json schema over the Blob. If you set it to a specific variable name that exists in the Python execution scope, then only that variable’s (in this case ‘X’) value will be retrieved and assigned to the output Blob. You can then choose to overlay an appropriate schema over the Blob back in Studio.
While testing this, I ran into a “JSON module not found” error. The fix was to install the latest IronPython on the server and set additional search paths for the IronPython Engine. Each line in
Runtime\addon_connectors\IronPython\schema\paths.conf will be added to IronPython’s search paths during execution.