Daniel Ko

Extending Hypixel Skyblock API with AWS Lambda and Amazon API Gateway

Exploring APIs by creating an API from game data.

Overview

In a previous article I used the hypixel skyblock API to determine profitable transactions in an auction house. We’ll be using AWS Lambda and Amazon API Gateway to create an API for this information.

Hypixel Skyblock Economics 101

AWS Lambda Function

First we’ll make an AWS Lambda function that returns our data. The function takes in max_price, min_volume, and min_profit and returns items that fit that criteria, along with the profit. The function’s timeout and memory limits should be adjusted in general configurations as necessary.

import json
import requests
import pandas as pd

def auction_flips(max_price, min_volume, min_profit):
    endpoint = 'https://api.hypixel.net/skyblock/auctions'

    params = {
        "page": 0
    }
    all_auctions = []

    while True:
        response = requests.get(endpoint, params=params)

        if response.status_code != 200:
            return {
                'statusCode': response.status_code,
                'body': json.dumps(response.text)
            }

        auctions = response.json()["auctions"]
        all_auctions.extend(auctions)
        if response.json()["totalPages"] <= params["page"] + 1:
            break

        params["page"] += 1

    bin_auctions = [auction for auction in all_auctions if auction['bin'] == True]

    item_to_price = [(auction['item_name'], auction['starting_bid']) for auction in bin_auctions]
    item_to_price = [(item_name, price) for item_name, price in item_to_price if price <= max_price]

    item_to_volume = {}
    for item_name, price in item_to_price:
        if item_name not in item_to_volume:
            item_to_volume[item_name] = 0
        item_to_volume[item_name] += 1

    item_to_price = [(item_name, price) for item_name, price in item_to_price if item_to_volume[item_name] >= min_volume]
    item_to_price = sorted(item_to_price, key=lambda x: x[1])

    df = pd.DataFrame(item_to_price, columns=['item', 'price'])
    profit = df.groupby('item')['price'].unique()
    profit = pd.DataFrame(profit)
    profit = profit[profit['price'].apply(len) >= 2]
    profit['cheapest_price'] = profit['price'].apply(lambda x: x[0])
    profit['second_cheapest_price'] = profit['price'].apply(lambda x: x[1])
    profit['tax'] = profit['second_cheapest_price'] * 0.01 + 350
    profit['profit'] = profit['second_cheapest_price'] - profit['cheapest_price'] - profit['tax']
    profit = profit.drop(columns=['price', 'cheapest_price', 'second_cheapest_price', 'tax'])
    profit = profit.sort_values('profit', ascending=False)
    profit = profit.loc[profit['profit'] >= min_profit]

    return profit.to_json()

def lambda_handler(event, context):
    print("Received event:", json.dumps(event))
    # max_price = event['max_price']
    # min_volume = event['min_volume']
    # min_profit = event['min_profit']
    max_price = event.get('queryStringParameters', {}).get('max_price')
    min_volume = event.get('queryStringParameters', {}).get('min_volume')
    min_profit = event.get('queryStringParameters', {}).get('min_profit')
    print(f'max_price:{max_price}')
    print(f'min_volume:{min_volume}')
    print(f'min_profit:{min_profit}')
    if not max_price or not min_volume or not min_profit:
        return {
            'statusCode': 400,
            'body': json.dumps('Missing required parameters.')
        }
    try:
        max_price = int(max_price)
        min_volume = int(min_volume)
        min_profit = int(min_profit)
    except ValueError:
        return {
            'statusCode': 400,
            'body': json.dumps('Invalid parameter type.')
        }
    profit = auction_flips(max_price, min_volume, min_profit)
    return {
        'statusCode': 200,
        'body': json.dumps(profit)
    }

Using this repo made it easy to add python packages as AWS Lambda layers.

GitHub — keithrozario/Klayers: Python Packages as AWS Lambda Layers

Amazon API Gateway

Now that the AWS Lambda function is complete, we can add API Gateway as a trigger from the function overview page.

Function Overview

All that’s left now is to call the function.

import json
import requests
import pandas as pd

max_price = 5_000_000
min_volume = 30
min_profit = 50_000
url = f'https://kgu794y59g.execute-api.us-east-2.amazonaws.com/default/auctionFlips?max_price={max_price}&min_volume={min_volume}&min_profit={min_profit}'
response = requests.post(url)

df = pd.read_json(response.json())
sorted_df = df.sort_values(by=['profit'], ascending=False)
sorted_df.head()

The request took ~15 seconds and returned the results nicely. Unfortunately json doesn’t hold order so I had to sort it again.

sorted_df.head(0

Conclusion

We made an extension of the hypixel skyblock api by processing the data and creating our own new api. AWS Lambda and Amazon API Gateway were both used to make this process extremely simple.

Challenges

Initially there were internal server errors but looking through the logs on CloudWatch helped me realize there were differences in the event structure between AWS Lambda and the API Gateway. This led to the function working during testing but not with API Gateway.

For the future

In the future, I want to try out API aggregation. Since Threads released recently, maybe an api for posting to both Threads and Twitter. 🤷🏻

To readers

This was an exercise for me to learn and mess around with Amazon web services and APIs. If anyone has any suggestions, please let me know. Anything helps !

Thanks for reading !

personal site