Photo by Simon Abrams on Unsplash

The technology driving an over-architected job board site

A few months back, my business partner (dillonchanis) and I released our first product under our umbrella company “Everlook”.

He and I both have around 5+ years of professional experience but have never actually released our own product before. I’m sure many of you can relate. We spent a ton of time in the past building half-baked ideas and watching them slowly collect dust in private repos on GitHub.

It makes sense though.. It’s tough coming home from your day job as a developer and finding the time and motivation to grind on your own products when we all have hobbies, families, and other obligations to focus on.

We decided to actually stick to a project for once. We finally released everlookjobs.com a few months back, and here are a few interesting technologies that we used, and some lessons learned that may intrigue you.

When building a product, the first thing you obviously need is an idea. We wanted to build something that could get our feet wet when it came to releasing products, so we decided on an already proven concept. A job board.

The reason we chose a job board was that it’s a relatively easy project to build. Simple CRUD operations, database manipulations, CRON / background tasks, etc.

So after we had our idea of an MVP, we decided we were going to need the following list of major tasks to take into consideration:

• Payment processing
• A dedicated server (preferably something managed so we don’t have to worry about scaling down the road)
• Authentication
• Databases
• File storage
• Hosting
• Cron / background tasks
• Analytics
• A frontend codebase with solid SEO
• A CDN to quickly serve content

After we defined the main concepts of our application, we figured it would be smart to write up some architecture diagrams that outlined the responsibility of each service. They are as follows:

1. Each service was going to be a microservice.
2. Our UI codebase was going to reach out via REST to a facade layer, which would act as an authentication middleware. This way we don’t expose our Microservices behind the hood.
3. A service that strictly deals with payments / customer information.
4. A service that handles file management.
5. A service that handles job posting functionality.
6. A service that handles user /company information.

At that point, we knew how we were going to structure our application but not what technologies and services we would leverage.

Languages of choice:

When thinking about which technologies we would use for our code, we automatically gravitated to using Node / JavaScript due to our familiarity with the language.

Servers:

When debating which technologies we would use, we knew we needed something that could scale, but also didn’t want the hassle of maintaining a long running server. We decided on AWS Lambda due to the pretty cheap pricing model, ease of use, and language support.

Although Lambda seems easy to work with, it can definitely be tricky due to a few nuances. I discuss this more when I speak about our lessons learned.

Looking past the few oddities about Lambda, some great aspects of the service include auto-scaling, the pay-per-request model, the fact that you can trigger lambdas on actions like S3 file drops, DynamoDB integrations, queue executions, to name a few.

Payments:

For payment processing, we decided on utilizing stripe since it’s super easy to use, API docs are easy to follow, and we all know custom PCI compliancy is a pain to handle. All of this payment logic is isolated to our ‘payments’ micro-service.

Authentication:

We opted into using Auth0 for all of our user authentication. Much like payment processing, user management can be a bit of a headache, so we deferred to using a third-party service instead of rolling our own solution. This saved us time and avoided us from having to set up sign up / login, reset password, verification emails, social media connections, etc.

User information is isolated to our User’s Microservice and user info is referenced in other services in a foreign key manner.

Databases:

For our data stores, we decided to use DynamoDB. This is another tricky service but it is fully managed (on-demand scaling is nice). We didn’t have anything insanely relational in our data stores, so this was a perfect fit for us. AWS’s SDK, and the Dynamoose package on NPM make interacting with this service a breeze. You just define your tables in a MongoDB-esque way, and get to reading & writing.

File Storage:

Due to our requirement of managing logos and other files, we decided on AWS S3. This service is super cheap, has solid authentication methods built around it including signed URLs, and is also very easy to interact with via the SDK. We use this for storing deployed Lambda code, cached JavaScript files for our CDN, and a dump for our event payloads as well.

Scheduled Tasks:

A few pieces of our functionality require tasks that execute on an interval (CRON), and AWS’s CloudWatch Rules were the perfect solution. This is due to the fact that you can create a simple schedule expression and have it trigger a lambda task however often you desire. Our use cases for this include deprecating listings after a certain time period, warming our lambdas to reduce API latency, and running other analytics-specific jobs.

Logging & Monitoring:

Obviously logging is needed in order to debug and monitor code in environments. Opting into CloudWatch logs was an obvious choice for us. You simply create a log group which matches your lambda, and all logs are piped to there in AWS. There are also great third-party integrations for monitoring logs (DataDog is one).

CDN:

In order to deliver our content quickly, we knew we needed some type of CDN. Luckily AWS has a great CDN service called CloudFront. This sits in front of our S3 buckets (which store our UI code) and delivers cached versions to our users. When we run deploys from CircleCI, we run CloudFront invalidations to update these caches.

UI Libraries:

Since both Dillon and I are both familiar with React we chose it as our UI library of choice. SPA’s are notoriously bad with SEO performance, though this may improve with Google’s more modern crawlers, so we decided to leverage NextJS to get the best of both worlds. You may be thinking, “Are you running NextJS on a long-running server?” well, the answer is no. We are utilizing AWS Lambda to server-side render our React pages which ended up being more efficient than we initially thought.

Just to give you more clarify, the order of operations for a normal UI interaction goes as follows:

1. A user lands on the desired page.
2. A lambda spins up and SSR’s the requested page (‘/job-detail’, for example).
3. CloudFront intercepts + serves the cached content from S3.
4. The same lambda that served the above content stays warm for subsequent requests and spins down after traffic slows down.

Some other UI packages that we use include:

1. Redux — State management
2. Tailwindcss — CSS utility library
3. Formik — For our forms
4. Lodash — JavaScript utilities

Analytics:

Analytics is something we are diligently working on to deliver to our users like targeted listings and listing statistics, and we knew we needed to feed chronological data into a database to query and run background processes on. AWS’s managed ElasticSearch was our choice due to quick searches, and other helpful analytics plugins.

Hosting:

We have the domain everlookjobs.com for our production URL, (we have lower-level environments as well which exist on other domains). Route53 and AWS cert-manager were great choices for managing certificates and registering domains. All of our URL’s point directly to our CloudFront distributions. We also use S3 to redirect www.everlookjobs.com -> https://everlookjobs.com.

CI/CD:

When deploying code to environments, it’s recommended to use some type of service to automate deployments which we’re using CircleCI. This is mainly due to their cheap pricing model and friendly user experience.

Infrastructure Configuration:

All of our AWS resources (databases, S3 buckets, Lambdas, etc) are all managed using AWS CloudFormation (infrastructure configuration) behind the hood, which is deployed during our CI/CD processes for each micro-service.

When deciding to use “hot” technologies, you will learn a lot of lessons. I’ll outline a few we experienced while building Everlook Jobs.

1. SSR (server-side rendering) with React is weird after writing SPA’s (single page applications) for so long. Things like ‘local storage’ and other Browser APIs don’t exist when serving content from a server, which can definitely bite you if you’re not on the lookout for it. It is easy to forget this when you are conditioned to writing client-side JavaScript code. It’s worth reading through the entire docs of NextJS before diving in. It may help save some frustration.
2. Writing and deploying code to AWS can be a pain but there are a few great open source libraries out there to help with this. Our choice was the Serverless Framework. Check it out!
3. Serverless (^) is your best friend. There are a ton of great plugins and boilerplates out there that can help you get up and running quickly (here is ours).
4. Lambda cold starts are annoying, and can definitely lead to latency issues. This is due to Docker being used behind the hood. After a container has gone a period of time without invocation, the lambda is considered “cold”, which means the underlying Docker container needs some time to spin up when requested. We got around this by “warming” our lambdas, or pinging them every minute or so to avoid them from going cold.
5. Some AWS services cost a lot. This is mainly due to the convenience and power that these services provide out of the box. Make sure to turn off any resources that you aren’t using! We found ourselves with a few chunky bills when forgetting to turn off test databases & EKS clusters (oops).
6. DynamoDB is kind of weird. It’s a distributed database, so managing concepts like pagination, hotkeys, and indexing can be tricky. Make sure to do research before diving into the surface level simplicity of this service. Also, make sure to turn on point-in-time recovery as well. It can be a lifesaver when utilizing DynamoDB.
7. Manage your resources with CloudFormation or some other type of configuration language (Terraform is nice too) to avoid manually creating/updating/destroying resources in your environments.
8. TailwindCSS + component libraries are the huge time savers.

Hopefully some of this was interesting, and maybe it will give you a few ideas when building your next project. We also know that a few of the things mentioned in this article may seem odd, but neither of us are perfect and we know that. We’re simply having fun building things :)

If you think of any improvements or have any questions, please feel free to leave a comment!

Also, if you don’t mind, please check out our application at everlookjobs.com. We’re aiming to deliver great experiences to our fellow job-hunting friends, along with robust toolsets and analytics to our partners looking to hire. We would also like to extend a limited time offer for 50% off job listings (our prices will increase over time). Use the code WELCOME2020 at checkout.

If you notice anything weird, you spot a bug, or would like to request some type of functionality for the future please drop us an email at hello@everlookjobs.com and we’ll check it out.

Thanks,

- Your friends at Everlook