Handling bursts of one million SNS messages in Lambda

Recently at work we were building a system that needed to handle events coming from an upstream system with the capability for high burst throughput (e.g. going from zero messages to one million within a few seconds). The code itself was a good candidate for a Lambda, and the events we needed to listen to would be coming from an SNS topic. Conveniently, AWS offers a direct integration between SNS and Lambda so that you can process SNS messages in a Lambda without any additional infrastructure whatsoever. Sounds great, right?

Image for post
Image for post

However, every system must have its limits, and the AWS documentation doesn’t always make it perfectly clear how their integrations will react when placed under heavy load. Hence, we really needed an answer to the following…


Image for post
Image for post

This year I participated in my first ever Hacktoberfest. Hacktoberfest is a yearly event run in October by DigitalOcean that presents developers with a simple challenge: create 5 pull requests to open-source projects on GitHub, and get a free t-shirt!

As a developer with a few solid years of experience but barely any open-source contributions under my belt, I decided that this year I would finally have a crack at the challenge and force myself to get deeper into the world of open-source.

Why? In my time as a developer, I’ve already taken advantage of a great many open-source projects to help with my work or professional development. It feels right to give something back to open-source when I’ve gotten so much out of it already. However, making first-time contributions can be quite intimidating, which is why I hadn’t really done any before now — and Hacktoberfest gave me the perfect excuse to finally break that barrier. …


Image for post
Image for post

Earlier this year, the company I work for as a software developer suffered a security incident. What followed was a very unique few months that I’ll remember for the rest of my career.

Prior to the incident, I’d had no real clue what to expect if one were to happen. There seem to be few first-hand accounts of what it’s like to be in that kind of situation published online. …


Let’s talk about happiness.

Happy developers are better developers, right? If you’re feeling stressed or upset, it only makes it more difficult for you to do your best work.

Now, there are a lot of things that can impact your happiness. Happiness for you might be about having a really supportive team at work; it might also be about having realistic deadlines that don’t place you under pressure. Maybe a priority for you is having a workplace that provides you lots of opportunities to learn new things and to grow in your career.

Happiness could be about all of the above, or any number of other countless personal and professional factors. These are all really big and important things to keep in mind if we want to be happy at work! …


My team at work has only recently started to utilise Docker for running our applications in both development and production, and as I’ve become more familiar with it, I’ve been quick to notice the commonly large sizes for even simple application images. With disk space as plentiful as it is these days, this is rarely an issue. However, there are still some scenarios where the size of your Docker images can play a large role:

  • Build/deploy times. If you’re using a continuous integration system (such as Travis CI) with Docker, it will probably be constantly downloading and uploading Docker images to build your application and then upload the resulting image for deployment. …


Image for post
Image for post

Ever built your own keyboard before?

Well, up until a few months ago, I didn’t even know that it was possible. I’ve been using a mechanical keyboard at work for nearly two years now - a Ducky Zero with Cherry MX Blue switches that I’ve quite liked - but it was only recently that I got sucked into the mechanical keyboards subreddit where users show off all kinds of weird and wonderful keyboard designs and layouts. Soon enough, the urge hit me to try and build an awesome-looking keyboard of my own!

The Why

Well, for starters, it’s not to save money. My keyboard has cost me about the same as my Ducky Zero, but depending on what parts you buy, you could easily spend triple what I have. …


Image for post
Image for post
A typical continuous deployment setup

This post is based on my 10-minute lightning talk for Melbourne’s DDD By Night of October 2017, with some more detail and discussion. If you’re looking for the related code samples, they are located here on GitHub.

In this post I’ll be discussing the benefits (and some pitfalls) of integrating Infrastructure-as-Code (IaC) tools with your Continuous Integration/Deployment (CI/CD) tools. I’ll primarily be discussing the tools Terraform and Travis in particular, but I expect the learnings can be applied with any similar toolset.

Motivation

If you’re not familiar with the concepts of Infrastructure-as-Code, let me provide a motivating example.

Consider your typical Continuous Deployment workflow: a developer commits some changes, which are picked up by a build server. The build server picks up these changes, builds them, runs all the tests if applicable, and once complete, deploys the built application to some kind of application server. …


When I start learning a new tool or framework, I like to try and keep it as isolated from other concepts as possible, so I can focus just on that one thing and nothing else. It avoids frustration with auxiliary tooling and helps me make sure I understand exactly what is happening in any given example.

In the case of learning React, I wanted to have a full working example in a plain HTML/Javascript file that I can simply open in any browser- no development webserver or anything involved.

Sounds simple enough, right? Well, it is once you know how, but the React documentation makes it a little difficult to tell how best to do this. …


This is a post I actually wrote up 2 years ago when I was finishing up development on Face Invaders as part of Electric Mammoth. It may not be very relevant by now, but it seemed like a waste to just have it sitting around all that time for no reason- so here it is!

Memory Budgeting

Regardless of whether you’re trying to make a large 3D game or a simple 2D one, if you’re developing on mobile, you should be super-duper conscious of your memory budget. Keep a close eye on your memory usage through Xcode or anything else that will help. …


Image for post
Image for post

In my team at PageUp we’ve recently started delving into serverless architectures (like AWS Lambda). These architectures help solve problems of scale and to introduce more event-driven behaviours into our system.

Fail safely — our focus

Naturally, a large focus of our implementation has been to ensure any errors occurring will cause the system to fail gracefully and pass the bad data away for later examination and diagnosis by a developer. Thankfully, a lot of our needs in this regard can be taken care of with AWS’s other integrated services, for example:

  • Dead Letter Queues to take in all messages that cannot be successfully handled by Lambda (now directly supported with SQS in Lambda…

About

Chris Lewis

Team-oriented software developer. Game dev hobbyist. Always eager to learn and expand my skillset! http://chrislewisdev.com/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store