Efficient Financial Pre-Screening Workflows Using AWS Step Functions 2/2

If you haven’t read the first part of this deep dive, you can start with Efficient Financial Pre-Screening Workflows Using AWS Step Functions 1/2.

Image by freepik.com

Architectural Challenges and Implemented Solutions

1. System Configuration and Management

We encountered significant complexity in managing configurations across distributed systems and multiple AWS consoles. Key issues included:

1. Inconsistent configurations leading to environment-specific bugs
2. Difficulty in tracking changes and maintaining version control
3. Increased time spent on troubleshooting configuration-related issues
4. Lack of standardisation in resource provisioning across teams

These challenges resulted in reduced developer productivity and increased operational overhead.

Resolution

We implemented a Infrastructure as Code (IaC) approach using the Serverless Framework, with custom Go plugins for extended functionality.

Implementation Details

Serverless Framework Configuration (serverless.yml):

{
  "service": "my-go-application",
  "provider": {
    "name": "aws",
    "runtime": "go1.x",
    "stage": "${opt:stage, 'dev'}",
    "region": "${opt:region, 'us-west-2'}"
  },
  "package": {
    "patterns": [
      "!./**",
      "./bin/**"
    ]
  },
  "functions": {
    "myFunction": {
      "handler": "bin/myFunction",
      "events": [
        {
          "http": {
            "path": "myEndpoint",
            "method": "get"
          }
        }
      ]
    }
  },
  "resources": {
    "Resources": {
      "MyDynamoDBTable": {
        "Type": "AWS::DynamoDB::Table",
        "Properties": {
          "TableName": "${self:service}-${self:provider.stage}-table",
          "AttributeDefinitions": [
            {
              "AttributeName": "id",
              "AttributeType": "S"
            }
          ],
          "KeySchema": [
            {
              "AttributeName": "id",
              "KeyType": "HASH"
            }
          ],
          "BillingMode": "PAY_PER_REQUEST"
        }
      }
    }
  },
  "custom": {
    "myCustomSetting": "someValue"
  }
}

Main Application Code (main.go):

package main

import (
    "context"
    "encoding/json"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

func handler(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    response := map[string]string{
        "message": "Hello from serverless Go function!",
    }
    body, err := json.Marshal(response)
    if err != nil {
        return events.APIGatewayProxyResponse{StatusCode: 500}, err
    }
    return events.APIGatewayProxyResponse{
        StatusCode: 200,
        Body:       string(body),
        Headers: map[string]string{
            "Content-Type": "application/json",
        },
    }, nil
}

func main() {
    lambda.Start(handler)
}

This solution provides several benefits:

1. Unified Configuration: All AWS resources and function configurations are defined in a single serverless.yml file, improving manageability.
2. Version Control: Infrastructure changes can be tracked alongside code changes in the same repository.
3. Standardisation: Teams can use this template as a baseline, ensuring consistency across projects.
4. Simplified Deployment: The Serverless Framework’s CLI tools streamline the deployment process, reducing manual steps and potential errors.
5. Environment Management: Easy switching between development, staging, and production environments using stage parameters.

By adopting this approach, we significantly reduced configuration-related issues, improved deployment reliability, and enhanced our ability to manage and scale our serverless applications effectively. The Serverless Framework’s built-in plugins and extensive ecosystem provide additional capabilities for monitoring, testing, and optimising our serverless applications without the need for custom Go plugins.

2. Event-Driven Step Function Orchestration

We needed to determine the most effective method for triggering AWS Step Functions for our application pre-screening process. Key considerations included:

1. Ensuring timely initiation of the pre-screening workflow
2. Maintaining loose coupling between services
3. Handling high volumes of applications efficiently
4. Minimising latency between application creation and workflow initiation

Resolution

After evaluating various options, we decided to implemented an event-driven architecture leveraging AWS Kinesis for stream processing and AWS Lambda for event handling and Step Function initiation.

Implementation Details

1. Kinesis Stream Configuration (via AWS CLI or CloudFormation):

{
  "StreamName": "ApplicationCreationStream",
  "ShardCount": 1,
  "RetentionPeriodHours": 24
}

Lambda Function for Triggering Step Functions (main.go):

package main

import (
    "context"
    "encoding/json"
    "log"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/sfn"
)

func handler(ctx context.Context, kinesisEvent events.KinesisEvent) error {
    cfg, err := config.LoadDefaultConfig(ctx)
    if err != nil {
        log.Fatalf("Unable to load SDK config, %v", err)
    }
    sfnClient := sfn.NewFromConfig(cfg)
    for _, record := range kinesisEvent.Records {
        var appEvent struct {
            ApplicationID string `json:"applicationId"`
        }
        err := json.Unmarshal(record.Kinesis.Data, &appEvent)
        if err != nil {
            log.Printf("Error unmarshaling event: %v", err)
            continue
        }
        _, err = sfnClient.StartExecution(ctx, &sfn.StartExecutionInput{
            StateMachineArn: aws.String("arn:aws:states:region:account-id:stateMachine:PreScreeningWorkflow"),
            Name:            aws.String("PreScreening-" + appEvent.ApplicationID),
            Input:           aws.String(record.Kinesis.Data),
        })
        if err != nil {
            log.Printf("Error starting Step Function: %v", err)
        } else {
            log.Printf("Started Step Function for application: %s", appEvent.ApplicationID)
        }
    }
    return nil
}

func main() {
    lambda.Start(handler)
}

Serverless Framework Configuration (serverless.json):

{
  "service": "application-prescreening-trigger",
  "provider": {
    "name": "aws",
    "runtime": "go1.x",
    "stage": "${opt:stage, 'dev'}",
    "region": "${opt:region, 'us-west-2'}"
  },
  "functions": {
    "stepFunctionTrigger": {
      "handler": "bin/stepFunctionTrigger",
      "events": [
        {
          "stream": {
            "type": "kinesis",
            "arn": "arn:aws:kinesis:${self:provider.region}:${aws:accountId}:stream/ApplicationCreationStream",
            "batchSize": 100,
            "startingPosition": "LATEST",
            "enabled": true
          }
        }
      ],
      "iamRoleStatements": [
        {
          "Effect": "Allow",
          "Action": [
            "states:StartExecution"
          ],
          "Resource": "arn:aws:states:${self:provider.region}:${aws:accountId}:stateMachine:PreScreeningWorkflow"
        }
      ]
    }
  }
}

This event-driven approach offers several benefits:

1. Loose Coupling: The application creation service simply publishes events to Kinesis, decoupling it from the Step Functions workflow.
2. Scalability: Kinesis can handle high volumes of events, ensuring efficient processing even during peak periods.
3. Real-time Processing: Events are processed as they arrive, minimising latency between application creation and workflow initiation.
4. Reliability: Kinesis provides at-least-once delivery semantics, ensuring no events are lost.
5. Flexibility: Additional consumers can be added to the Kinesis stream without modifying the application creation service.

By implementing this event-driven architecture, we achieved efficient and reliable triggering of our Step Functions workflows, while maintaining a loosely coupled and scalable system design.

3. Idempotent Execution Control

In our AWS Step Functions-based pre-screening process, we faced the risk of duplicate executions for the same customer application. This could occur due to:

1. Retry mechanisms in distributed systems
2. Parallel triggering of workflows
3. Network issues causing redundant requests

Duplicate executions could lead to:

• Inconsistent data states
• Unnecessary resource consumption
• Potential customer confusion or financial discrepancies

Resolution

We implemented a robust idempotency mechanism leveraging AWS Step Functions’ native features:

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/sfn"
    "github.com/aws/aws-sdk-go-v2/service/sfn/types"
)

const (
    preScreeningStateMachineARN = "arn:aws:states:region:account-id:stateMachine:CustomerPreScreening"
)

type CustomerData struct {
    ID   string `json:"id"`
    Name string `json:"name"`
    // Add other relevant fields
}

func generateStepFunctionName(applicationID string) string {
    return fmt.Sprintf("customer-prescreening-%s", applicationID)
}

func startPreScreening(ctx context.Context, applicationID string, customerData CustomerData) error {
    cfg, err := config.LoadDefaultConfig(ctx)
    if err != nil {
        return fmt.Errorf("unable to load SDK config: %v", err)
    }
    client := sfn.NewFromConfig(cfg)
    input, err := json.Marshal(customerData)
    if err != nil {
        return fmt.Errorf("failed to marshal customer data: %v", err)
    }
    params := &sfn.StartExecutionInput{
        StateMachineArn: aws.String(preScreeningStateMachineARN),
        Name:            aws.String(generateStepFunctionName(applicationID)),
        Input:           aws.String(string(input)),
    }
    _, err = client.StartExecution(ctx, params)
    if err != nil {
        var executionAlreadyExists *types.ExecutionAlreadyExists
        if ok := errors.As(err, &executionAlreadyExists); ok {
            log.Printf("Execution already in progress for application %s", applicationID)
            // Handle accordingly (e.g., return existing execution status)
            return nil
        }
        return fmt.Errorf("failed to start execution: %v", err)
    }
    log.Printf("Started pre-screening for application %s", applicationID)
    return nil
}

func main() {
    ctx := context.Background()
    applicationID := "APP-12345"
    customerData := CustomerData{
        ID:   applicationID,
        Name: "John Doe",
        // Add other relevant data
    }
    if err := startPreScreening(ctx, applicationID, customerData); err != nil {
        log.Fatalf("Error in pre-screening process: %v", err)
    }
}

This Go implementation ensures:

1. Unique Execution Naming:
   We generate deterministic step function execution names using the application’s UUID.
2. Execution Initialisation:
   We use the AWS SDK for Go v2 to start the Step Function execution with the unique name.
3. Duplicate Handling:
   We catch the ExecutionAlreadyExists error and handle it gracefully, preventing duplicate executions.
4. Error Handling and Logging:
   Proper error handling and logging are implemented for better observability.

4. Comprehensive DynamoDB Logging for Step Functions

Insufficient persistence of both execution-level and application-level logs beyond AWS Step Functions’ CloudWatch logs, hindering effective monitoring and debugging.

Resolution

We implemented a comprehensive external logging system using Amazon DynamoDB to store both execution-level and application-level logs. This approach offers:

1. Persistent storage of detailed logs beyond CloudWatch’s retention period
2. Easier querying and analysis of log data
3. Separation of concerns between execution and logging

This enhanced solution provides:

1. Separate logging for execution-level and application-level events
2. Detailed task results stored in application-level logs
3. A flexible schema that accommodates both log types
4. A global secondary index for efficient querying by log type

This comprehensive logging system enables deep insights into both the execution flow and the application-specific data, significantly enhancing our ability to monitor and debug Step Function executions.

Complete Workflow Diagram

Here’s how the entire workflow looks in a flowchart:

Pre-Screening workflow using AWS Step Functions

Conclusion

By using AWS Step Functions and AWS Lambda, we created an efficient and scalable workflow to pre-screen customers for financing. This approach allows us to automate the validation process, ensuring accuracy and reducing manual efforts. AWS Step Functions’ ability to coordinate various services into a single workflow makes it a powerful tool for similar use cases in different industries. Transitioning from BPMN to AWS Step Functions has allowed us to leverage our existing AWS infrastructure more effectively while simplifying our workflow management.