Use Google Cloud Storage as an application file directory for SAP

Although APIs have become the predominant method for system communication, file transfer is still used by many enterprises running SAP. For example, if the amount of data is large or when real-time data exchange is not crucial, file based transfer of data can be the ideal solution. Also, some industries such as banking and insurance still use well established file transfer processes as part of their business operation. In many cases, legacy systems and external partners with limited integration capabilities use file transfer as the primary communication mechanism.

Traditionally, customers have been exchanging files with external systems by setting up:

• A file directory on the application server where files will be stored
• A service that transfers the file between SAP and external system using protocol such as SFTP (SSH File Transfer Protocol)

This approach has some disadvantages.

• A file transfer service has its own added cost of operation and maintenance.
• Regular housekeeping is required for file system in SAP to avoid File system full error.
• Some SFTP implementations in SAP rely on external Unix scripts for functionality, which can add complexity to setup, maintenance, and monitoring.
• SFTP requires opening multiple ports in the firewall, which can increase attack surface in SAP.

In this blog, I will showcase how you can use Google Cloud Storage as an application file directory and securely access files within a Cloud Storage bucket from SAP using ABAP SDK for Google Cloud. The solution also features SFTP connector, which can be used as a task in Application Integration to securely exchange files from Cloud Storage with your external system using SFTP.

The blog covers 2 main use cases:

1. Inbound to SAP, where an external system sends files to a Cloud Storage bucket, which are processed in SAP.
2. Outbound from SAP, where SAP creates files in Cloud Storage bucket, which are then sent to the external system.

Inbound to SAP: Processing files sent from external system

Using the below integration flow, you can transfer file data from an external system into a file object in Cloud Storage bucket and use ABAP SDK for Google Cloud to process the file.

Integration flow for inbound file transfer from external system to SAP

This integration has 3 main components:

1. An external system which produces files, which should be transferred to Cloud Storage.
2. An instance of Application Integration that uses SFTP Connector to read files from the external system and write the file data to Cloud Storage
3. SAP ERP system where an ABAP extension (Batch job, BADI etc) uses ABAP SDK for Google Cloud to process the files stored in Cloud storage.

Let’s look at each component in detail.

External System producing files

Create a credential in your external system for SFTP connection from Google Cloud. SFTP connector supports Username and password or SSH_PUBLIC_KEY.

Google Cloud Integration for file transfer to Cloud Storage

• Create a bucket in Cloud Storage, which will be used to store file data from external system.

Files from the external system will be stored in this bucket

• Set up an instance of SFTP Connector in Integration Connectors by entering the external system details including credential for SFTP connection.
• Create an Application Integration that uses SFTP Connector from the previous step. Refer to community post / Github repo for detailed instructions on how to set up this integration between external system and Cloud Storage

By triggering the Application integration instance, files will copied from external system into the Cloud Storage bucket files_from_ext_system-landing. For testing, below files have been uploaded to the bucket

Files with order data

File data

Processing files in Cloud Storage from SAP by using ABAP SDK

By using ABAP SDK Cloud Storage API class /GOOG/CL_STORAGE_V1, files written to Cloud Storage can be processed in SAP. If not already done, use our public documentation to download, install and configure the SDK in your SAP system.

We will create 2 additional buckets to move files based on processing status:

1. Files successfully processed will be moved to bucket files_from_ext_system-archive
2. Files that had errors will be moved to bucket files_from_ext_system-error

Below code snippets can be used to list, read and move files from Cloud Storage bucket.

First, list files from the bucket files_from_ext_system-landing

DATA(lt_files)  = VALUE /goog/cl_storage_v1=>ty_t_string( ).
TRY.
    DATA(lo_gcs) = NEW /goog/cl_storage_v1( iv_key_name = 'ABAP_SDK_DEV' ).

    lo_gcs->list_objects(
      EXPORTING
        iv_p_bucket = 'files_from_ext_system-landing'
      IMPORTING
        es_output   = DATA(ls_object) ).

    LOOP AT ls_object-items ASSIGNING FIELD-SYMBOL(<ls_item>).
      APPEND <ls_item>-name TO lt_files.
    ENDLOOP.

  CATCH /goog/cx_sdk INTO DATA(lo_cx_sdk).
    "Add error handling logic
ENDTRY.

A quick look at lt_filesin ABAP debugger shows the list of files in bucket files_from_ext_system-landing

Files with order data

Using below logic:

• Read data from the file.
• After processing the file data, copy the file to files_from_ext_system-archive or files_from_ext_system-error bucket based on the processing status of file
• Delete the file from files_from_ext_system-landingbucket.

LOOP AT lt_files ASSIGNING FIELD-SYMBOL(<ls_file>).
  DATA(ls_raw_data) = VALUE xstring( ).
  TRY.

      "Set parameter to read data from file
      lo_gcs->add_common_qparam( iv_name = 'alt'
                                    iv_value = 'media' ).

      lo_gcs->get_objects(
        EXPORTING
          iv_p_bucket = 'files_from_ext_system-landing'
          iv_p_object = <ls_file>
        IMPORTING
          es_output = DATA(ls_output)
          es_raw = ls_raw_data ).

      lo_gcs->clear_common_qparam( 'alt' ).

      DATA(lo_convin) = cl_abap_conv_in_ce=>create( input = ls_raw_data ).

      DATA(lv_file_data) = VALUE string( ).
      lo_convin->read(
       IMPORTING
         data = lv_file_data ).

      DATA(lt_file_data) = VALUE /goog/cl_storage_v1=>ty_t_string( ).

      SPLIT lv_file_data AT cl_abap_char_utilities=>cr_lf 
      INTO TABLE lt_file_data.

      DATA(lv_error) = space.
      "Add logic to process the data in file. 
      "If there are errors, set lv_error = 'X'.

      DATA(lv_destination_bucket) =
        COND string( WHEN lv_error IS INITIAL
                       THEN 'files_from_ext_system-archive'
                     ELSE 'files_from_ext_system-error' ).

      "Copy file to destination bucket
      lo_gcs->rewrite_objects(
        iv_p_destination_bucket = lv_destination_bucket
        iv_p_destination_object = <ls_file>
        iv_p_source_bucket = 'files_from_ext_system-landing'
        iv_p_source_object = <ls_file> ).

      "Delete file from landing
      lo_gcs->delete_objects(
          iv_p_bucket = 'files_from_ext_system-landing'
          iv_p_object = <ls_file> ).

    CATCH /goog/cx_sdk.
      "Add error handling logic
  ENDTRY.
ENDLOOP.

A quick look at lt_file_data while debugging shows the data read from the file.

After processing, the files have been moved to files_from_ext_system-archive

As you can see, you can easily up a cloud-native application directory on Google Cloud storage that can be easily and securely accessed from SAP using ABAP SDK.

While this sample shows a basic flow, you can make this integration more robust and secure by adding:

• Malware scanning for files received from external system.
• ABAP authorizations before Cloud Storage API call to restrict access to limited list of users.
• Email notification to external system by sending a message to a Pub/Sub topic and using a Cloud Function listener.

Outbound from SAP: Sending files to an external system

Using the below integration flow, you can create files in Cloud Storage bucket from SAP and transfer those files to an external system by using Application Integration.

Integration flow for outbound file transfer from SAP to external system

Similar to inbound flow, this integration has 3 main components:

1. SAP ERP system where an ABAP extension (Batch job, BADI etc) uses ABAP SDK for Google Cloud to create files in Cloud Storage.
2. An external system to which files from Cloud Storage are written
3. An instance of Application Integration that reads file data from Cloud Storage and then uses SFTP Connector to write files in the external system

Let’s look at each component in detail.

Creating files in Cloud Storage bucket by using ABAP SDK

First, create a bucket files_from_sap-landing in Cloud Storage.

Below logic creates a CSV file object with order data in the Cloud Storage bucket files_from_sap-landing

TYPES:
  BEGIN OF t_order_data,
    order_number TYPE string,
    material     TYPE string,
    quantity     TYPE string,
    unit         TYPE string,
  END OF t_order_data,

  tt_order_data TYPE STANDARD TABLE OF t_order_data WITH EMPTY KEY.

DATA(lt_order_data) = VALUE tt_order_data( ).

"Write logic to populate lt_order_data

DATA(lv_file_string) = VALUE string( ).

LOOP AT lt_order_data ASSIGNING FIELD-SYMBOL(<ls_order_data>).
  DATA(lv_row_data) = <ls_order_data>-order_number && ',' &&
                      <ls_order_data>-material && ',' &&
                      <ls_order_data>-quantity && ',' &&
                      <ls_order_data>-unit.
  IF lv_file_string IS INITIAL.
    lv_file_string = lv_row_data.
  ELSE.
    lv_file_string = lv_file_string && cl_abap_char_utilities=>cr_lf && lv_row_data.
  ENDIF.
ENDLOOP.

DATA(lv_file_data_xstring) = VALUE xstring( ).
DATA(lo_convout) = cl_abap_conv_out_ce=>create( ).
lo_convout->write( lv_file_string ).
lv_file_data_xstring = lo_convout->get_buffer( ).

TRY.
    DATA(lo_gcs) = NEW /goog/cl_storage_v1( iv_key_name = 'ABAP_SDK_DEV' ).

    lo_gcs->insert_objects(
      EXPORTING
        iv_q_name       = 'SAPOrders' && '_' && sy-datum && sy-uzeit
        iv_p_bucket     = 'files_from_sap-landing'
        is_data         = lv_file_data_xstring
        iv_content_type = 'text/csv' ).
  CATCH /goog/cx_sdk.
    "Add error handling logic
ENDTRY.

For demo purposes, below sample data was inserted into lt_order_datawhile debugging.

File was successfully created in the bucket files_from_sap-landing

File in Cloud Storage bucket

File data

External System receiving files

Similar to inbound scenario, create a credential (Username and password or SSH_PUBLIC_KEY) in your external system for SFTP connection from Google Cloud.

Google Cloud Integration for file transfer from Cloud Storage to external system

This flow can be built similar to inbound scenario by creating SFTP Connector and Application Integration instances.

• Set up an instance of SFTP Connector in Integration Connectors by entering the external system details including credential for SFTP connection.
• Create an Application Integration that uses SFTP Connector from the previous step. Refer to community post / Github repo for detailed instructions on how to set up this integration between external system and Cloud Storage
• Configure a Pub/Sub notification for bucket files_from_sap-landing, such that Application Integration is triggered immediately when a file is created in the bucket.

By running Application Integration, files will be copied from Cloud Storage to external system using SFTP.

Conclusion

As described above, using ABAP SDK for Google Cloud and Cloud Storage, you can build secure and seamless file transfer mechanism between SAP and external systems.

Here are some key resources to learn, collaborate and contribute:

Developer Tools

SAP and Google Cloud: Innovating Together

Happy Learning and Innovating with Google Cloud !