Doing it Right: Solid Engineering to Take Down Phish
Paul East |@Area1Security| Area 1 Security
Being a member of the engineering team at Area 1 Security is an adventure in tracking down the bad guys and eliminating the phishing threat for our customers. That mission means understanding attackers and finding the small, telltale patterns in big data and scaling systems. We have established the following principles: Limit the programming languages in use; test the code individually; as a team, employ a standardized deployment pipeline; and implement a standard set of monitoring and alerting systems.
Java is our primary language for backend micro services and systems. This is where our expertise and foundation have been laid. We use a combination of Python and React.js to deliver our web applications, as well as some purpose-built systems in C / C++. The small number of languages translates to a team familiar with a core set of languages, with the ability to provide meaningful code reviews to each other. Having a standard set of languages across the engineering organization reduces bottlenecks, improves feature velocity, and provides opportunities for engineers to work on different parts of the system. This also avoids the situation where systems are orphaned when the original author can no longer support them.
Three layers of testing are in use, in order to deliver high-quality software consistently. The most basic layer is unit testing, a fundamental engineering practice with proven results. The next layer is automated integration testing, supported by a test automation framework that’s purpose-built to achieve our objectives. These two layers are built into our continuous integration (CI) system, so no code change is merged without all tests passing. The final phase is user interface testing of things our automated testing does not cover. This is minimal, and our goal is to eliminate it with automation. Each of these layers is engineer-supported, with an expectation for delivering features and functionality to the systems.
Leveraging Chef (chef.io) for all instance deployments using a common set of recipes across our systems ensures consistent deployment, with all necessary pieces for user management, monitoring, and alerting (more on that later). Engineers can run local instances leveraging the same recipes that are used in production, which allows for like environments at all phases of development. This process enables all engineers to have a common understanding of how the code is deployed, and they can easily learn another system. We employ Docker in many instances and plan to transition to Kubernetes in the coming year.
Alerting and monitoring have to be done well, and we are very passionate in avoiding alert fatigue. If everybody is ignoring the alerts because they always occur, the really important issues will be missed. This means monitoring critical metrics and alerting on the really important ones. As part of the recipes we deploy, standard monitoring and alerting functionality is provided. Engineers can focus on the metrics they collect and the thresholds for alerts, rather than the mechanics of doing so. Because engineers are responsible for the support of the production systems, they have all the incentives to build high-quality software and to alert themselves when critical issues facing customers occur.
If this is the kind of engineering organization you want to join, grow, and learn with, while solving the toughest engineering problems, check out our openings https://jobs.lever.co/area1security or reach me at paul@area1security.com
