API Gateway, DynamoDB and Discogs.com API

If you have a big collection of records and you enjoy buying, selling and exploring records in Discogs.com you might have invested a lot of your time building your record collection to be available in the Discogs.com site.

Discogs.com has an API , that you can leverage to have some high availability, or to operate it in a more automated way.

What if Discogs.com had some kind of internal issue and your collection were lost or let’s say they just stop providing their fantastic database.

I thought I could operate simultaneously both APIS by using AWS API Gateway along some lambdas and DynamoDB replicating the data on both AWS and Discos.com

From there more features can be added and even a frontend like ReactJS in S3 static hosting.

I’m providing a CloudFormation template that contains the necessary to build an API that allows you to sync your Discogs.com collection to DynamoDB as well as search, add and delete records on both your Discogs.com collection and DynamoDB table.

In order to launch this solution which is defined in the CloudFormation template found in Git and inserted at the bottom of the article you will need to get a Discogs.com Token, and provide it along your Discogs.com user and the code files names uploaded to S3 for each lambda , which will respond performing a function for each API Gateway method.

The CloudFormation template has SAM parts so you can launch it from the command line doing the transformation required by SAM, you would execute the command below (or create and execute a change set using the console):

Below, you have the only CloudFormation template where we define all the API Gateway components needed (resources, methods, models, lambda functions and permissions).

You will enter in the parameters section the name of the S3 bucket where you have uploaded the lambda functions code you can find in my Git repo as well .

Due to the amount of code of some functions is required to store them in S3 instead of defining them directly in the template.

On the other hand you will need to pass in the parameter section your Discogs.com username and Token used by the lambdas to perform interact with Discogs.com API. You can get the Token for your user in the Discogs.com developer section.

First step would be to invoke the sync method to populate your current Discogs.com records into your DDB table. No parameter is needed just go ahead.

Now you can add records passing the release id like passing the Discogs.com release id of the record in the request headers:

In order to search a record we will pass the band_name:Artist

To delete a record will pass the instance id, there’s one generated for each record we add from discogs, in this case 423749971 to the proper method in our API.

Another cool thing is that each time we add or delete a record through API gateway it gets added or deleted on both Dynamodb and our Discogs.com collection

Finally there’s a scan method to retrieve all your records for a later pagination with ReactJS

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: Api that allows to sync you discogs.com collection to DynamoDB, and also allows to search, add, delete a record being this operation performed on both discogs.com and dynamoDB databases in a Serverless way.
Parameters:
  BucketName:
    Type: String
    Default: yours3bucketwhereyouuploadthecodeoflambdas
  CodeKeySync:
    Type: String
    Default: sync.zip
  CodeKeyRecordAdd:
    Type: String
    Default: addrecord.zip
  CodeKeyDelInstance:
    Type: String
    Default: delinstance.zip
  CodeKeyRecordSearch:
    Type: String
    Default: searchrecord.zip
  CodeKeyScanTable:
    Type: String
    Default: scantable.zip
  DDBName:
    Type: String
    Default: Records
  Token:
    Type: String
    Default: yourowntokennnnnnnnnnnnnnncDqypbeNPyHHYJMHQvn
    NoEcho: True
  User:
    Type: String
    Default: yourdiscogssuserrname
Resources:
  DDBTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Ref DDBName  
      AttributeDefinitions:
        -
          AttributeName: "instance_id"
          AttributeType: "N"
        -
          AttributeName: "band"
          AttributeType: "S"
KeySchema:
        -
          AttributeName: "instance_id"
          KeyType: "HASH"
   
      ProvisionedThroughput:
        ReadCapacityUnits: 2
        WriteCapacityUnits: 2
GlobalSecondaryIndexes: 
        - 
          IndexName: "bandGSI"
          KeySchema: 
            - 
              AttributeName: "band"
              KeyType: "HASH"
          
          Projection: 
            ProjectionType: "ALL"
          ProvisionedThroughput: 
            ReadCapacityUnits: "1"
            WriteCapacityUnits: "1"
CloudWatchRole: 
    Type: AWS::IAM::Role
    Properties: 
      AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement: 
          - Effect: Allow
            Principal: 
              Service: 
                - "apigateway.amazonaws.com"
            Action: "sts:AssumeRole"
      Path: "/"
      ManagedPolicyArns: 
        - "arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs"
Account: 
    Type: AWS::ApiGateway::Account
    Properties: 
      CloudWatchRoleArn: 
        "Fn::GetAtt": 
          - CloudWatchRole
          - Arn
MyRestApi:
    DependsOn: 
      - Account
      - DDBTable
    Type: AWS::ApiGateway::RestApi
    Properties:
      EndpointConfiguration:
        Types:
          - REGIONAL
      Name: discsapi
Deployment:
    DependsOn: SyncMethod
    Type: AWS::ApiGateway::Deployment
    Properties: 
      RestApiId: !Ref MyRestApi 
      Description: "My deployment"
      StageName: Prod
SyncResource: 
    Type: AWS::ApiGateway::Resource
    Properties: 
      RestApiId: !Ref MyRestApi
      ParentId: 
        Fn::GetAtt: 
          - MyRestApi
          - RootResourceId
      PathPart: "SyncResource"
RecordAddResource: 
    Type: AWS::ApiGateway::Resource
    Properties: 
      RestApiId: !Ref MyRestApi
      ParentId: 
        Fn::GetAtt: 
          - MyRestApi
          - RootResourceId
      PathPart: "RecordAddResource"
DelInstanceResource: 
    Type: AWS::ApiGateway::Resource
    Properties: 
      RestApiId: !Ref MyRestApi
      ParentId: 
        Fn::GetAtt: 
          - MyRestApi
          - RootResourceId
      PathPart: "DelInstanceResource"
RecordSearchResource: 
    Type: AWS::ApiGateway::Resource
    Properties: 
      RestApiId: !Ref MyRestApi
      ParentId: 
        Fn::GetAtt: 
          - MyRestApi
          - RootResourceId
      PathPart: "RecordSearchResource"
ScanTableResource: 
    Type: AWS::ApiGateway::Resource
    Properties: 
      RestApiId: !Ref MyRestApi
      ParentId: 
        Fn::GetAtt: 
          - MyRestApi
          - RootResourceId
      PathPart: "ScanTableResource"
     
  SyncMethod:
    Type: "AWS::ApiGateway::Method"
    Properties:
      AuthorizationType: NONE
      HttpMethod: POST
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Sub
          - "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}/invocations"
          - lambdaArn: !GetAtt SyncLambda.Arn
      ResourceId: !Ref SyncResource
      RestApiId: !Ref MyRestApi
ScanTableMethod:
    Type: "AWS::ApiGateway::Method"
    Properties:
      AuthorizationType: NONE
      HttpMethod: POST
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Sub
          - "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}/invocations"
          - lambdaArn: !GetAtt ScanTableLambda.Arn
      ResourceId: !Ref ScanTableResource
      RestApiId: !Ref MyRestApi
    
  AddRecordMethod:
    Type: "AWS::ApiGateway::Method"
    Properties:
      AuthorizationType: NONE
      HttpMethod: POST
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Sub
          - "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}/invocations"
          - lambdaArn: !GetAtt AddRecordLambda.Arn
      ResourceId: !Ref RecordAddResource
      RestApiId: !Ref MyRestApi
      RequestModels:
        application/json: !Ref ModelRecordAdd
DelInstanceMethod:
    Type: "AWS::ApiGateway::Method"
    Properties:
      AuthorizationType: NONE
      HttpMethod: POST
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Sub
          - "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}/invocations"
          - lambdaArn: !GetAtt DelInstanceLambda.Arn
      ResourceId: !Ref DelInstanceResource
      RestApiId: !Ref MyRestApi
      RequestModels:
        application/json: !Ref ModelDelInstance
RecordSearchMethod:
    Type: "AWS::ApiGateway::Method"
    Properties:
      AuthorizationType: NONE
      HttpMethod: POST
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Sub
          - "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}/invocations"
          - lambdaArn: !GetAtt SearchRecordLambda.Arn
      ResourceId: !Ref RecordSearchResource
      RestApiId: !Ref MyRestApi
      RequestModels:
        application/json: !Ref ModelRecordSearch
ModelRecordAdd:
    Type: AWS::ApiGateway::Model
    Properties:
      ContentType: 'application/json'
      Description: RecordAdd  
      Name: RecordAdd
      RestApiId: !Ref MyRestApi
      Schema:
        type: object
        properties:
          release_id:
            type: string
        title: RecordAddModel
ModelDelInstance:
    Type: AWS::ApiGateway::Model
    Properties:
      ContentType: 'application/json'
      Description: DelInstance  
      Name: DelInstance
      RestApiId: !Ref MyRestApi
      Schema:
        type: object
        properties:
          instance_id:
            type: string
        title: DelInstanceModel
ModelRecordSearch:
    Type: AWS::ApiGateway::Model
    Properties:
      ContentType: 'application/json'
      Description: RecordSearch
      Name: RecordSearch
      RestApiId: !Ref MyRestApi
      Schema:
        type: object
        properties:
          band_name:
            type: string
        title: RecordSearchModel
SyncLambda:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: sync.lambda_handler
      Runtime: python3.7
      CodeUri: 
        Bucket: !Ref BucketName
        Key: !Ref CodeKeySync
      MemorySize: 128
      Timeout: 100
      Policies:
        - AmazonDynamoDBFullAccess
      Environment:
        Variables:
          table: !Ref DDBTable
          my_token: !Ref Token
          user: !Ref User
          curr_region: !Ref "AWS::Region"
ScanTableLambda:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: scantable.lambda_handler
      Runtime: python3.7
      CodeUri: 
        Bucket: !Ref BucketName
        Key: !Ref CodeKeyScanTable
      MemorySize: 128
      Timeout: 100
      Policies:
        - AmazonDynamoDBFullAccess
      Environment:
        Variables:
          table: !Ref DDBTable
          curr_region: !Ref "AWS::Region"
AddRecordLambda:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: addrecord.lambda_handler
      Runtime: python3.7
      CodeUri: 
        Bucket: !Ref BucketName
        Key: !Ref CodeKeyRecordAdd
      MemorySize: 128
      Timeout: 100
      Policies:
        - AmazonDynamoDBFullAccess
      Environment:
        Variables:
          table: !Ref DDBTable
          token: !Ref Token
          user: !Ref User
          curr_region: !Ref "AWS::Region"
DelInstanceLambda:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: delinstance.lambda_handler
      Runtime: python3.7
      CodeUri: 
        Bucket: !Ref BucketName
        Key: !Ref CodeKeyDelInstance
      MemorySize: 128
      Timeout: 100
      Policies:
        - AmazonDynamoDBFullAccess
      Environment:
        Variables:
          table: !Ref DDBTable
          token: !Ref Token
          user: !Ref User
          curr_region: !Ref "AWS::Region"
SearchRecordLambda:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: searchrecord.lambda_handler
      Runtime: python3.7
      CodeUri: 
        Bucket: !Ref BucketName
        Key: !Ref CodeKeyRecordSearch
      MemorySize: 128
      Timeout: 100
      Policies:
        - AmazonDynamoDBFullAccess
      Environment:
        Variables:
          table: !Ref DDBTable
          curr_region: !Ref "AWS::Region"
lambdaApiGatewayInvokeSync:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: "lambda:InvokeFunction"
      FunctionName: !GetAtt "SyncLambda.Arn"
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${MyRestApi}/*/POST/*"
lambdaApiGatewayScanTable:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: "lambda:InvokeFunction"
      FunctionName: !GetAtt "ScanTableLambda.Arn"
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${MyRestApi}/*/POST/*"
lambdaApiGatewayInvokeRecordAdd:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: "lambda:InvokeFunction"
      FunctionName: !GetAtt "AddRecordLambda.Arn"
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${MyRestApi}/*/POST/*"
lambdaApiGatewayInvokeDelInstance:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: "lambda:InvokeFunction"
      FunctionName: !GetAtt "DelInstanceLambda.Arn"
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${MyRestApi}/*/POST/*"
lambdaApiGatewayRecordSearch:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: "lambda:InvokeFunction"
      FunctionName: !GetAtt "SearchRecordLambda.Arn"
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${MyRestApi}/*/POST/*"

https://github.com/pabloperfer/discogs2dynamodb