IBM INTEGRATION BUS (IIB) — Developing an API to extract data from SAP
The history of IIB began in 2000 when it was launched as MQSeries Integrator (MQSI). MQSI was an application integration solution that allowed organizations to connect and integrate heterogeneous systems. In 2002, MQSI was added to the WebSphere family and renamed WebSphere MQ Integrator. In 2004, it was renamed again to WebSphere Message Broker.
WebSphere Message Broker was a message-based service integration solution that enabled organizations to securely and efficiently connect and integrate systems, applications, and data. It was a comprehensive solution offering a wide range of features to meet integration needs of any size or complexity. In 2010, WebSphere Message Broker was rebranded as IBM Integration Bus and has since continued to evolve over the years with the release of new versions.
IIB Overview as a Tool
As mentioned, IIB is a message-based service integration platform that enables organizations to securely and efficiently connect and integrate systems, applications, and data. It is a comprehensive solution that offers a wide range of features to meet integration needs of any size or complexity.
IBM Integration Bus is built upon a message bus that provides a common infrastructure for data exchange between systems and applications. The bus consists of a set of components that enable message routing, transformation, mediation, and management.
In practice, it functions as a block-based tool where blocks are dragged and connected. Each block has its properties and configurations, and in some cases, it is possible to include code developments, as we will see throughout this post.
The Case
During the development of an API for extracting data from the SAP system, I noticed that there was not much information available for reference. Additionally, the IIB documentation suffers from a lack of visual aids, a significant oversight considering it is a drag-and-connect type of tool.
Therefore, this post aims to demystify the initial process and, most importantly, how to connect with SAP.
Application Execution
The execution process within IIB involves the orchestration of flows and subflows. These components serve as the building blocks where each step of the application is defined.
To draw a parallel with programming, the flow can be likened to the main code (similar to main.py in a script), while subflows can be seen as functions that are imported and utilized within the flow.
In this context, the SAP connection adapter is incorporated into the subflow. This adapter contains the necessary configuration details to establish a connection to SAP, which can be shared across all flows and subflows. , ensuring a seamless integration with the SAP system during the execution of the application.
API Creation
The initial step in our project is to create the API, from which we will proceed to develop all the necessary modules. With the software open, simply navigate to File > New > REST API. Following this, assign a name to the API, as illustrated in Figure 1.
Creating Endpoints
With the API project created, the next step is to generate the endpoint(s) as needed. In this project, I opted to create one for each SAP table, and I will demonstrate creating one for the “MARA” table, which is part of the standard set of tables.
In the API definition screen, which appears after its creation, click on the area marked in red. A window will appear, where you can provide a name for the endpoint and choose the operation type. In this case, the endpoint is named “MARA,” and the operation type is set to “GET”. Figure 2 illustrates the API definition screen.
Creating API Parameters
After creating the endpoint, we can define parameters for our API. In the extractions we are developing, we want users making API calls to provide information on how many rows they want to extract and how many rows they want to skip. Figure 3 illustrates the API definition screen after creating the endpoint.
By clicking on the area marked in green, we can create a new parameter, which will appear similar to the one marked in purple. Choose the parameter’s name, type, format, and whether it is mandatory. In my case, I created two parameters named “maxrows” and “skip,” both set as the integer type.
Creating the Subflow
After creating the necessary parameters, you can click on the area marked in red in Figure 3. A new screen will open, and from there, you will develop the majority of the logic involved in your extraction.
Subflow Development Area
The development area is where the logic of the flow is created. In the centre-right, there is a palette of functions that can be dragged into the blank area. Functions are represented by blocks that need to be connected from the Input to the Output. There are two possibilities to configure blocks individually:
- Adjust their properties;
- Double-click to open a new block editing tab (this option is not available for all blocks).
Figure 4 illustrates the development area with the function palette highlighted in red. Note that in the development area, the properties of function blocks appear in the area marked in green.
Creating the SAP Connection Adapter
Drag the “SAP Request” block to the development area, and in its properties, locate the one named “primary adapter component.” Click on “Browse,” and a window will open as shown in Figure 5. In this window, click on “Import/Create New.”
On the next screen, click “Next,” as shown in Figure 6.
On the following screen, click “Next” again; there is no need for modifications, as depicted in Figure 7.
Next, as shown in Figure 8, specify the configuration files for SAP access (if you don’t have them or are unsure, request them from the SAP administrator) and click “Next.”
Afterwards, a screen with information to fill in will be presented, as shown in Figure 9 below. Fill in the fields: Host Name (Nome do Host), System Number (Número do sistema), Client (Cliente), User Name (Nome de Usuário), Password (Senha), and select QISS as the SAP Interface Name.
Right after this, the software will connect to SAP, and the existing tables in SAP will be displayed. By clicking on the three arrows in the left corner, indicated in red, it is possible to filter by name, as shown in Figure 10.
When you find a table you wish to extract, select the table and click on the arrow in the center, indicated in green in Figure 10. A window will open with information about the table in question and all its columns, as shown in Figure 11. In this window you should:
- Delete the line “Include a WHERE clause. TABLE <table name>: (Incluir uma cláusula WHERE.)”, marked in red;
- Check the box for “use SAP field names to generate business object attribute names (Usar nomes de campo SAP para gerar nomes de atributos de objetos de negócios)”, in green;
- Choose which columns you want to extract, in purple;
- Press “OK”.
Repeat the filtering and table selection process for all desired tables. Once completed, click on “Next”. In the next object configuration screen, as shown in Figure 12, DO NOT enter anything in the “custom recovery function name (Nome da função de recuperação customizada)” field, simply click “Next.”
In the next screen, you are required to fill in the connection details once again. Following that, on the publishing properties screen, it is necessary to choose a name for the adapter in the “adapter component name,” as shown in Figure 13.
Clicking Next concludes the construction of the adapter. Simply select it and click “OK” and it should then appear in the properties of the SAP Request block.
Extraction
Adding logic and testing the extraction in the subflow
To test if our adapter is functioning, we will perform an extraction using the mapping block. This block is responsible for creating the request message sent to SAP. To edit its settings, follow these steps:
Double-click on the mapping block, and a window will appear as shown in Figure 14. In the appearing window, choose the option “simple message map called by a message flow node.”
In the following screen, navigate to the “select map output” area. Expand the information of your API, and then do the same with the “DFDL and XML schemas” tab. Finally, choose the schema named “SapMara.” Figure 15 displays the screen with the selected schema.
After choosing the schema, click “Next,” and then “Finish.” A new tab will appear. In this tab, expand the area marked in red in Figure 16. Scroll down to the bottom of the page and also expand the area called Sap<table name>Querybo.
Click on the area marked in green in Figure 16 to add environment variables to the mapping. On the left side, expand the variables area, and with a right-click, add the variables named “myMaxRows” and “mySkipRows,” as shown in Figure 17.
Finally, connect the created variables on the left, “myMaxRows” and “mySkipRows,” to the variables “SapMaxRows” and “SapSkipRows”, respectively. Both are located at the bottom of the page on the right side.
To complete the logic for our test extraction, we will include a compute block. It will be responsible for transmitting the parameters from our API to the “myMaxRows” and “mySkipRows” variables. For a more robust solution, I included a condition where, if no argument is passed, it will use a value defined by me. This step is necessary for extraction because, without a definition of the maximum limit of rows, the SAP Request will throw an error.
When you double-click on the compute block, a new window will open with the example code. Modify the code to match what is presented below.
CREATE COMPUTE MODULE getMara_compute1
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
CALL CopyMessageHeaders();
CALL CopyEntireMessage Headers();
RETURN TRUE;
END
CREATE PROCEDURE CopyMessageHeaders() BEGIN
DECLARE I INTEGER 1;
DECLARE J INTEGER;
SET J = CARDINALITY(InputRoot.*[]);
WHILE I < J DO
SET OutputRoot.*[I] = InputRoot.*[I];
SET I = I + 1;
END WHILE
END;
CREATE PROCEDURE CopyEntireMessage() BEGIN
IF InputLocalEnvironment.REST.Input.Parameters.maxrows IS NULL THEN
SET Environment.Variables.myMaxRows = 100;
ELSE
SET Environment.Variables.myMaxRows = InputLocalEnvironment.REST.Input.Parameters.maxrows;
END IF;
IF InputLocalEnvironment.REST.Input.Parameters.skip IS NULL THEN
SET Environment.Variables.mySkipRows = 0;
ELSE
SET Environment.Variables.mySkipRows = InputLocalEnvironment.REST.Input.Parameters.skip;
END IF;
END;
END MODULE;Finally, click on the Sap Request block, and in the “default Method” property, choose “retrieveallSapMara.”
Testing
To perform the test, we must first connect the blocks as shown in Figure 18.
With everything saved, we can deploy our API for testing. Right-click on the API’s name and choose Deploy. Then, select the node where you want to deploy, as shown in Figure 19.
With the deployment completed, we need to find the URL to make the test call. From the home screen, right-click on the node where you have deployed the application for testing, and in the options that appear, click on “Start Web User Interface,” as shown in Figure 20.
By following these steps, a page will open in your browser. Navigate through it as indicated in the left panel in Figure 21, and you will find the base URL for the API.
With this information at hand, you only need to make the request to see if there are results. I chose to use Postman, but there are various other ways to do it. By entering the base URL and the ‘mara’ endpoint and making the request, you should see a result similar to the one presented in Figure 22.
Final Thoughts and What’s Next
Building an application in IBM Integration Bus might feel a bit tricky if you’re used to regular coding, but don’t worry — after some time you will get used to it. As you keep going with your API development journey, here are some practical ideas: add a token system to protect your API, make data pagination better, and throw in filters and clear and customized error messages. These steps not only beef up your API’s security but also make it more user-friendly.
Note of advice
You might notice in the images that some parts are in English but others don’t. This happened because the development had to be done in a Portuguese version of the IBM Integration Bus, but the software wasn’t fully translated. To help you find your way, in many steps, I have included the names in English, followed by their Portuguese version in parentheses.
Troubleshooting
Even when configuring the node through software, it may be necessary to do it through the terminal. To do this, have on hand the location on your computer of the SAP configuration files and the name of the node where you are deploying. Then, type the following commands to follow the steps to be executed:
mqsiprofile.cmdmqsichangeproperties.exe TESTNODE_L459 -c EISProviders -o SAP -n nativeLibs -v C:\IBM\SAPLibsmqsichangeproperties.exe TESTNODE_L459 -c EISProviders -o SAP -n jarsURL -v C:\IBM\SAPLibs