Hackathon to production
How we built an enterprise scale integration in 3 days
Beamery are Agile evangelists. We believe in delivering working software in small increments with regular reviews of the requirements and customer needs. Most teams at Beamery implement a Scrum methodology to achieve this, and it works well for our day-to-day work. Sometimes, however, the most effective way of tackling a hard problem is to break out of the sprint cycle and swarm on it together in the form of a hackathon.
As well as providing a refreshing shift from the normal way of working, Hackathons also help improve collaboration between team members (especially important while we’re all working from home), and allows our developers to focus on what they do best: building great software. In this article, we’ll talk about how we used a hackathon to deliver a brand new integration to production in just three days.
At Beamery, integrations are a core part of our product strategy. We run an entire Product vector — consisting of 3 people in Product Management, and two scrum teams — dedicated to evolving our Public API and integrations built using it. As a result of this focus, we have developed a depth of experience and expertise in integration design with the software platforms that form the Beamery Partner Ecosystem. This means we are well placed to quickly and efficiently build new integrations using our integration platform, Horizon (more on this later).
ATS (Applicant Tracking Systems) are one of the key types of systems our customers use in their day-to-day work. Specifically, they use them to manage and track Candidates as they apply to and progress through the job application process. Given this, we have built a number of integrations with the key ATS software platforms in the market, which placed us in good stead for building an integration with SAP’s ATS, SuccessFactors.
SAP SuccessFactors is a key ATS, used by Fortune 500 customers globally. As part of our focus on building a robust ecosystem of technologies that improve recruiter efficiency and candidate experience, building an integration to SAP SuccessFactors was the next logical step for us.
Our pedigree in ATS integrations
Our design approach with ATS integrations relies on two simple principles:
- Beamery is the source-of-truth for the Candidate
- The ATS is the source-of-truth for the application process
To expand on that, Beamery is the place where recruiters source, nurture and engage with potential Candidates. The ATS is where recruiters manage the application process from the time that the Candidate applies to a job, through screening and interview, to the eventual hire. See the below diagram for a visual representation:
Our tech made it easy
Horizon is Beamery’s main integration platform. It exists to provide a fast and flexible platform on which to build data pipelines to facilitate integrations with our partners. At the time of writing, Horizon comprises 66 Google Cloud Functions across 63 repositories, and 10 storage buckets, all replicated over 7 environments. While this might seem like a lot to manage, this complexity is by design: by breaking the platform into very small parts (some functions are only a few dozen lines of code), it’s easily extendable and reusable when we have a new problem to solve. We follow the mantra of ‘complex, but not complicated’.
Over the last two years, we have also worked to transform our deployment process and DevOps culture, adopting a Deploy Quickly Everywhere philosophy. By building agility into the design of our platform, we were well placed to build a new integration from scratch with minimal cognitive overhead.
The Hackathon
Day 0 — Spike and scope
Ahead of the hackathon, we worked on defining the scope of the integration and learning about the SAP Successfactors API. We needed to align our product ambitions with the capabilities of the API ahead of time, to reduce the risk of hitting blockers during the hackathon. Thankfully, SAP provides a powerful and flexible API that would allow us to build and maintain a high quality integration with very little custom logic.
We decided to scope an MVP integration that would provide value for our customers and allow us to easily extend the functionality in the future. Our MVP would be a vacancy integration, pulling vacancies from SAP into Beamery. This integration would allow our customers to focus on creating vacancies in SAP, and have them automatically sync to Beamery in near-real-time. From a recruiter’s point of view, working in Beamery to identify the best potential candidates for a given vacancy is made a lot easier by being able to see which open vacancies they are working to fill, all in the same platform (in this case, Beamery).
Imagine that the recruiter has identified a great pool of candidates for a role. Now, by having the vacancy visible in Beamery, they can simply invite the candidate to apply for the role, saving them time and the need to switch between systems.
Aside from clear API documentation, SAP’s partner team supported us throughout the build process with clarifications and questions on how to solve particular problems.
Day 1 — Start Work
On day 1, we focussed on building a client to pull data from SAP. We wrote the client as a Google Cloud Function in Go, to be invoked by a message published to a corresponding Pub/Sub topic on GCP. This message would contain all the necessary information to call a customer’s SAP API and pull their vacancy data into our Horizon integrations platform. We use Google Cloud KMS encryption to ensure that our customer’s data is protected.
Day 2 — Write the transformers and ingest the data into Beamery
Our new SAP client was working, but at this point it was outputting data in the format provided by SAP. To make it usable by Beamery, we needed to transform it ready for ingestion by Beamery’s APIs. We reused Horizon’s transformation layer for this purpose. Written in Node.js, our unifier function allows for fast and easy writing of new data transformations, and provides a powerful suite of automated tests to ensure their accuracy. Once we’d written the data transformation was written, the vacancy data was ready to be ingested into Beamery.
Day 3 — Plumbing and testing
With the individual components written, all that was left was to connect the different pieces of the integration pipeline together. Horizon contains a coordinator system for connecting functions, which we extended for the new integration.
Once the pipeline was ready, we started QA on the new integration to ensure it was working as intended. This raised a couple of small bugs in our data transformation, which we fixed and retested in a couple of hours. With this done, we were ready to demo the product to our internal stakeholders.
Post hackathon
After the hackathon, we focused on writing technical and customer-facing documentation. With clear documentation for our customers, we help make the onboarding process as smooth and pain-free as possible. We also took the opportunity to plan the roadmap of features we want to add in the near future.
Go to market
After building customer-facing collateral, we kicked off the due diligence process for having our app checked and cleared for listing on the SAP App Store. Take a look here.
Want to work alongside some great humans and inspiring leaders to solve big problems? We’re hiring! Click here to join the #BeamTeam and change the future of work.
