Load Testing API Endpoints: Simulating real-world API traffic with Poisson Distributions
At Sage AI, we serve models as services, enabling Sage products to send data into those services and get ML predictions out. Ensuring that our services can handle the demand from our Sage products before we push changes or new services into production is crucial. The natural way to do this is API load testing.
In the realm of software development and API performance optimization, load testing plays a crucial role in ensuring that systems can handle heavy traffic and maintain optimal response times. While traditional load testing techniques provide valuable insights, incorporating statistical models such as Poisson distributions and exponential decay can offer a more realistic representation of real-world scenarios. In this blog post, we will explore a load testing technique that combines these statistical concepts to simulate realistic traffic patterns and evaluate API endpoint performance.
Why Poisson Distributions?
Poisson distributions are widely used to model random events occurring over a fixed interval of time or space. In the context of load testing, we can leverage Poisson distributions to simulate the arrival rate of requests hitting an API endpoint. The key characteristics of a Poisson distribution are;
- Arrival Rate (λ): Represents the average number of requests arriving per unit of time. It can be calculated by dividing the total number of requests by the duration of the load testing period.
- Independence: Each request arrival is independent of others, meaning the occurrence of one request does not affect the probability of another request arriving.
- Randomness: The timing of each request follows a random pattern, allowing us to replicate real-world traffic scenarios.
Simulating request arrival patterns
To simulate the arrival patterns of requests, we can generate a sequence of inter-arrival times using the Poisson distribution. This sequence of time intervals between consecutive requests provides a realistic depiction of traffic patterns. We can achieve this by following these steps:
1. Determine the average arrival rate (λ) based on the desired load for the API endpoint being tested.
2. Generate a series of random inter-arrival times using the Poisson distribution formula.
Formula for Poisson Distributions
Where:
- P(X = k) is the probability of observing exactly k events.
- e is the base of the natural logarithm, approximately equal to 2.71828. It is sometimes referred to as Euler’s number.
- λ (lambda) is the average rate of events occurring in a fixed interval of time or space.
- k is the number of events observed.
In this context, λ represents the average arrival rate of events (in our case; requests hitting an API endpoint) per unit of time. The formula calculates the probability of observing k events in a given interval.
Incorporating Exponential Decay
Real-world scenarios often involve variable traffic loads, with periods of high and low demand. By incorporating exponential decay into our load testing approach, we can mimic these traffic fluctuations more accurately. Exponential decay introduces a decreasing trend in the arrival rate over time.
1. Define a decay rate (α) that represents the rate at which the arrival rate decreases over time.
2. Combine the Poisson distribution with exponential decay by modifying the arrival rate (λ) over time. As time progresses, the arrival rate decreases exponentially, resulting in a more dynamic and realistic simulation of varying traffic loads.
Formula for Exponential Decay
Where:
- y is the value at time t.
- y0 is the initial value at time t = 0.
- e is the base of the natural logarithm, approximately equal to 2.71828. It is sometimes referred to as Euler’s number.
- k is the decay constant.
- t is the time elapsed.
Tying it all together
Before jumping into our implementation, let’s make the relationship between these concepts more concrete;
- The Poisson distribution is our way of modeling the frequency of events (e.g. arrival rate) in a given period of time.
- Exponential distribution is our way of modeling the time between the events in a Poisson process.
- The average arrival rate (λ) in the Poisson distribution corresponds to the mean for the Exponential distribution (1/λ) which dictates the inter-arrival times when we sample from the distribution.
The code below is an arbitrary example. To make use of this technique in practice, the user should define the desired load profile by specifying the average arrival rate (λ) and decay rate (α).
import math
import random
import time
from typing import List
"""
Function to generate inter-arrival times using Poisson
distribution with exponential decay.
"""
def generate_inter_arrival_times(arrival_rate: int,
decay_rate: float,
duration: int) -> List[float]:
inter_arrival_times = []
time = 0
while time < duration:
# Calculate the arrival rate at the current time using exponential decay
current_rate = arrival_rate * math.exp(-decay_rate * time)
# Generate the inter-arrival time using the Poisson distribution
inter_arrival_time = random.expovariate(current_rate)
# Ensure larger values don't pollute the set.
if inter_arrival_time >= duration:
break
inter_arrival_times.append(inter_arrival_time)
time += inter_arrival_time
return inter_arrival_times
# Set up API calls...
# Configure inter arrival time and arrival rate semantics.
arrival_rate = 10 # Average arrival rate of 10 requests per second
decay_rate = 0.2 # Decay rate of 0.2 requests per second
duration = 60 # Duration of the load test in seconds
inter_arrival_times = generate_inter_arrival_times(arrival_rate,
decay_rate,
duration)
for i, inter_arrival_time in enumerate(inter_arrival_times):
time.sleep(inter_arrival_time)
# Submit call to API endpoint.The code above demonstrates how to generate a sequence of inter-arrival times based on the Poisson distribution with exponential decay and use them to dictate the cadence of calls to the API. You can adjust the arrival_rate, decay_rate, and duration variables according to your specific load testing requirements.
Conclusion
Load testing API endpoints using Poisson distributions and exponential decay offers a more realistic and accurate simulation of real-world traffic patterns. By incorporating these statistical concepts into load testing, developers can gain valuable insights into how their API endpoints perform under varying loads. This technique can aid in identifying and addressing performance bottlenecks, optimizing resource utilization, and ultimately enhancing the overall reliability and responsiveness of API-based systems. Having confidence in the performance of service APIs is pivotal in delivering value to our Sage products, and most importantly; our customers.
