Implementation of the API Gateway Layer for a Machine Learning Platform on AWS
“Quality is decided by the depth at which the work incorporates the alternatives within itself, and so masters them.” Theodor W. Adorno. In other words, to reliably build the API Gateway layer, an in-depth study is required to consider the different available alternatives.
After defining some of the main concepts in the API world in the previous article, I will talk about the different ways of deploying an API Gateway for the Machine Learning platform.
In this article, I will use the infrastructure and software layers designed in one of my previous articles. You may want to go through it to have a clearer view of the platform’s architecture before proceeding.
As a reminder, the scope of this series of articles is the model serving layer of the ML platform’s framework layer. In other words, its “API Gateway”.
Now let’s start designing!
A question may arise: If we are in an AWS environment, why not just use the fully managed and serverless AWS API Gateway?
You never know if you don’t try. So let’s try this!
1 | Just the AWS managed API Gateway
Here’s how AWS API Gateway could be placed in front of an EKS cluster.
First of all, AWS API Gateway is a fully managed service and runs in its own VPC: so we don’t know what’s happening between the scenes or any details about the infrastructure.
Thanks to AWS documentation, we know that we can use API Gateway private integrations¹ to get the traffic from the API Gateway’s VPC to our VPC using an API Gateway resource of VpcLink².
The Private VpcLink is a great way to provide access to HTTP(S) resources within our VPC without exposing them directly to the public internet.
But that’s not all. The VpcLink is here to direct the traffic to a Network Load Balancer (NLB). So the user is responsible for creating an NLB which serves the traffic to the EKS cluster.
With the support of NLB in Kubernetes 1.9+, we can create a Kubernetes service of type LoadBalancer With an annotation indicating that It’s a Network load balancer³.
That would be a correct setup for the AWS API Gateway on EKS. We could as well benefit from WAF support for the managed API Gateway.
The problem with this setup is we have the power of an API Gateway, but it’s far away from our cluster and our services. If we want to use a specific deployment strategy for each service, it would be great if this is done very close to the service (like in the service’s definition itself!).
2 | API Gateway closer to our ML models!
Here’s another way of doing things.
Design components: AWS API Gateway + NLB + Ambassador API Gateway.
In this setup, we put an open-source API Gateway solution closer to our services. I will talk in detail about Ambassador in a future article. For now, let’s just say it’s a powerful open-source API/Ingress Gateway for our ML platform that brings the API Gateway’s features closer to our models.
So do we really need the AWS API Gateway? Not really… One downside though, we will lose the WAF advantages for sure if we don’t use the AWS API Gateway.
But maybe we can optimize it more!
3 | Eliminate the AWS API Gateway!
So let’s eliminate the AWS API Gateway.
Design components: NLB in public subnet + Ambassador API Gateway.
We just need to put the NLB in a public subnet so that we can receive the public traffic.
However, NLB doesn’t understand HTTP/HTTP(s) traffic, it allows only TCP traffic, no HTTPS offloading, and they have none of the nice OSI’s layer 7 features of the Application Load Balancer (ALB).
Plus, with an NLB, we still can’t have the advantages of WAF.
4 | Our final design!
So, here’s the final setup.
Design components: ALB in public subnet + WAF + NLB in private subnet + Ambassador API Gateway.
As WAF integrates well with Application Load Balancer (ALB), why not get an ALB in front of the NLB. We can get that NLB back to its private subnet as well.
One thing to pay attention to though: In this setup, AWS ALB cannot be assigned a static public IP address. So, after some time, the ALB’s IP changes and we lose access to the platform.
Two possible solutions:
1. Summon the almighty Amazon Route53: We need to use the DNS name of the ALB instead of its changing IP addresses. To do this:
a. We have to migrate our nameservers to Route53 if it’s not already the case.
b. Pay attention to mails redirection: Route53 is only a DNS resolver and does not redirect emails. A solution for this could be to use an MX record and a mail server (like Amazon WorkMail).
2. Use AWS Global Accelerator: we never get bored with Amazon. Recently, Amazon launched this new service which could easily solve such a problem. A global accelerator with 2 fixed IPs and a unique DNS name will receive the traffic and direct it to an endpoint group containing our ALB.
Here’s a detailed guide on how to use this new feature.
Conclusion
In this article, I tried to study different deployments of an API Gateway for the Machine Learning platform. Starting from simply using an AWS API Gateway, I tried to find an optimal setup with maximum use of AWS advanced features like WAF.
In the next article, I will discuss in detail Ambassador and various concepts behind its existence.
If you have any questions, please reach out to me on LinkedIn.
[1] https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-private-integration.html
[2] https://docs.aws.amazon.com/apigateway/api-reference/resource/vpc-link/
[3] https://kubernetes.io/docs/concepts/services-networking/service/#aws-nlb-support
