Building Software at Scale

Paylocity is growing at a massive rate, and has been for the almost 7 years that I’ve been with the company. It’s a huge part of what makes Paylocity such an exciting place to work! With this growth comes incredible responsibility, especially considering the mission critical nature for our customers our systems, such as payroll and time and attendance. While we tend to mission-critical applications, we’re spinning off new product modules and services constantly. With each deployment, we need to be confident that we can handle everything our users throw at us, and at the same time, keep innovating with new technologies, patterns, and practices. Below I cover some of the foundational keys to how we build at scale here at Paylocity.

Distributed, asynchronous processing
Every time someone clocks in or out of our system, we have to determine whether that time is payable, and at what rate (e.g. is there overtime or holiday pay involved). To do this as users clock in or out would be extremely time intensive. Additionally, throughout the day, there are certain periods where the system as a whole is receiving large spikes of clock in/out requests (e.g. 9 am, 4 pm, lunch time, etc…). During this time, if we were to process each punch as it comes in, not only would our users see slow responses, but it would cause undo stress to parts of the system. To resolve this, we created a distributed processing system called Work Engine which can quickly distribute load across a number of nodes and communicate asynchronously with users’ browsers. This way when processing has completed, the user can see the change. In 90%+ of our transactions, this occurs seemingly instantaneously to our users, with the remaining amount completing within just a few seconds. Using punch processing as the proving ground, we’ve since been able to leverage the distributed/asynchronous processing power of Work Engine across our entire suite of applications to enable the heavy lifting of large scale processing loads.

Knowing Our Customers
As our user base has grown tremendously, monitoring, alerting, and logging have become paramount in deeply understanding our customers experience from each click and transaction. When issues arise in production, although rare, we need to know before our users do. In order to be able to do this, we leverage logging and monitoring tools, such as App Dynamics and the ELK Stack to give us that real-time insight. This level of insight into how our applications are performing in production allows us to proactively take action so that our customers rarely experience an issue, or if they do, it is easily recoverable and helps us ensure a continuous positive user experience.

Automation
The biggest key to scaling software is automation and there is no bigger boost to productivity and production than from automating key processes. At Paylocity, we work to automate processes as much as possible. When we talk about automation at Paylocity, we’re actually talking about several independent concepts, including: Continuous Integration (CI) builds, consistent deployments to staging and production environments, multiple kinds of testing (e.g. unit tests, functional/integration tests, load tests, etc…), as well as automating the ability for some of our support teams to scale their services — for example, a chat bot that can kick off application pool recycling in environments on which our developers don’t have access. No matter what the process is, automation has shown the ability to improve our execution of it. This gives, most importantly, gives us confidence in delivering only the best user experience to our customers.

Feature Toggles
Agile and Lean software development is often equated to changing the tires on a moving car. It’s clear that being able to deploy more frequent, but smaller change sets is key to ensuring that stability and functional issues are not introduced, but how do you do that when you have large features that may not be ready for user broader consumption? At Paylocity we use feature toggles to help mitigate that challenge. The primary benefit that feature toggles have given us is the ability to separate a deployment from a release. A deployment is the physical movement of files to a production environment — a risky proposition in an of itself. We want to be able to increase the number of deployments so that we can nail down and automate our processes, but also because it allows us to deploy smaller change sets, thereby decreasing the risk of each individual deployment. A release, on the other hand, can now become a planned, scheduled, and measured event in which we carefully roll out new features to early adopters, get feedback, incorporate that feedback, and reapply it to what early adopters are seeing. In this way, we are able to ensure that the new features we roll out are solid, stable, and — most importantly — meet the needs of our customers.

Being open as an organization to these new ideas has allowed us to accept and mount the challenges of extreme growth head on

Perhaps the most fulfilling aspect of working at Paylocity is feeling like your voice is always heard. The way this manifests is in our ability to innovate at the grass roots level. All of the techniques, tools, and processes that help us to build world-class software at scale were once just an idea in someone’s head, or a blog post that someone read, or a conference presentation that someone attended. Being open as an organization to these new ideas has allowed us to accept and mount the challenges of extreme growth head on. We’re still only scratching the surface of what we can do at scale here at Paylocity, but with our unique blend of talent and culture, I’m extremely excited to be a part of that.