I DDoSed myself using AWS CloudFront and Lambda@Edge and got a $4.5k bill
I am using NextJS and serverless-nextjs, and deploy my app to CloudFront and Lambda@Edge.
I made a mistake and accidentally created a serverless function that called itself. In a recursive loop, with a 30s timeout. I thought I fixed it and deployed the code to the dev environment.
I have had an AWS Billing alert (Budgets) set up to prompt me when my monthly budget goes over $300 (my usual bill is $200/month).
Imagine the terror when I woke up the next day to see the AWS Billing alert email saying I already owed $1,484! I removed a function and deployed it again in 30 minutes, but it was too late. It has already run for 24 hours, using over 70 million Gb-Second!
Only after that I’ve learned that AWS Billing alerts do not work this way for CloudFront. You get delayed information on charges because they collect them from all regions.
On the following day, the bill settled at a shocking $4600. This is more than we have ever spent on AWS all time.
CloudFront includes the AWS Shield Standard feature, but somehow, it was not activated for this case (Lambda@Edge calling itself via CloudFront).
Now, I understand that I should have created CloudWatch alarms, which would alert me when the number of requests exceeds the limit. The problem is, that they need to be set up per region, and I got CloudFront charges from all points of presence.
I am a big proponent of the serverless approach. It makes it easy to scale and develop things (e.g., you get PR review version branches for free, both frontend and backend code like Vercel does). But now, I am unsure because such unexpected charges can ruin a side-project or emerging startup.
Now I am waiting on a response from AWS Support on these charges; maybe they can help me waive part of that.
What is your experience with it? Would you recommend to use to build a new product if you are bootstrapped, 3-person startup?
UPDATE 28.6.2022: Posted on YCombinator News, follow the comments there.
UPDATE 8.7.2022: Here is the follow-up story, there I explain about AWS refund and what that was my next actions.
About author: I am Co-Founder and CTO of Valosan, media relationship app to help PR & Comms professionals get earned media visibility. Reach me out on Twitter or signup for Valosan at valosan.com/signup.
