There are a couple of nuances with aws services I encountered along the way that I wanted to highlight here. AWS Lambda Ephemeral Storage One of the first issues I encountered after getting everything setup was that the Process lambda would only work once. After the first execution, each subsequent…

In my last post I showed how we can create and store a csv file in an S3 bucket. Let us now see how we can get the status of a request & get a downloadable link to the csv for completed requests. The Get Status lambda Under the src…

In the last post I explained how to scrape a url with selenium and extract the number of search results returned by google for a query string. Let us now see how to insert these results into S3 as a csv file. Setting up the S3 bucket Update the template.yaml…

In my last post I created a lambda that accepts a request, stores it in a dynamodb table and sends a message to an SQS queue. Let’s now create another lambda to read from that queue and process the request by scraping the url using selenium. Installing Selenium Create a…

In my last post I covered creating a lambda & API Gateway to accept a POST request. Now lets save that request to a table in DynamoDB and send it to a queue in SQS for processing. Creating the Orders Table We create a new table by updating the template.yml…

In my last post I covered the overall architecture we are building and how to get setup with SAM CLI locally & deploy the app to AWS. Setting up a virtual environment Since we will be using python to build our app, lets create a virtual environment where we can…

I recently found myself trying to build a backend architecture that would allow me to scrape data from a website and not only store its status in a database but also allow a csv dump of the data to be downloaded. I decided to leverage the AWS free tier and…