Exercising is good for your (service’s) health
Author: Joe Buck - SDE III, Global Delivery Network
Dr. Joe Buck is one of the senior engineers on the Global Delivery Network (GDN) team within Remitly. The team’s role is to build and maintain our world-class disbursement platform and its connections to our partners that enables Remitly to disburse to more than 170 countries and territories around the world.
Last summer, the GDN team at Remitly hosted Solomon, a Code2040 fellow. During his 12-week internship, we worked together to design and build a service that addressed a significant gap in our engineering toolkit. This service, aptly named “services-exerciser”, enables GDN engineers to quickly define logic for generating a consistent and varied workload on our services in our preproduction and production environments. In the eight months that we’ve been using services-exerciser, we’ve used it to: ensure consistent test coverage of our public services on an on-going basis, quickly validate data integrity during data migrations and API changes, and extract performance data that we would otherwise not have easy access to. Because we now have the functionality that services-exerciser provides, GDN has been able to move faster, and with more confidence, in expanding and improving Remitly’s disbursement network.
Throughout the internship, Solomon was asked to learn a variety of topics that spanned phases of the software development process. He learned to develop software using the Go programming language, learned about GDN’s approach to capturing run-time exceptions (bugsnag), and iterated on the design of the data structures used to configure his service. These iterations produced a simpler and more user-friendly interface. This simplicity was largely achieved by pushing complexity into the internals of the service, a worthy trade-off but one that incurred additional work of greater complexity for Solomon.
The original idea behind Solomon’s project was to create a service where any GDN engineer could easily configure an arbitrary set of calls to endpoints within a service and have those calls executed on a regular basis, thereby generating the load that we desired in our preproduction environment. As we built out the first version of the service, we added (1) support for the GET, PUT, POST, and DELETE REST verbs,(2) support for capturing output from one call and using that data as input for subsequent calls, and (3) the ability to configure both positive (expected to succeed) and negative (expected to fail) tests.
We expanded the original use case to include alerts when there are endpoints for a configured service that are neither under test nor contained within a list of endpoints explicitly excluded from testing. This functionality was achieved by extending our service configuration to include the endpoint providing that service’s API documentation (all GDN services generate an OpenAPI specification for their endpoints). We used regular expressions to match the endpoints from the specification against those in the test, where a failure to find a match triggers an alert for the GDN team to address.
During the development of Solomon’s project, additional use cases arose that we are currently extending our work to address. One such update involves modifying Solomon’s service to be runnable on-demand (as opposed to running on a fixed-schedule). This change allows us to run the tests for a given service as part of that serivce’s deploy pipeline, with test failures signaling the need to rollback the deploy that is currently in-progress.
A second extension is to refactor the test configurations to enable making calls to multiple services in the same unit of work. We will then trigger this batch of calls if any service included in the set of endpoints starts a new deployment, thereby enabling GDN to run complex, multi-stage integration test workloads. This will more quickly surface regressions induced by, but not occurring in, the service being updated.
Solomon’s project adds to GDN’s track record of bringing in high-achieving students, teaming with them to build meaningful projects over their time on the team, and sending them back to their programs with expanded skills and experiences that they can apply to their remaining coursework as well as to their future employment (ideally back at Remitly). In addition to the code that Solomon and our prior interns produced, GDN benefits from Remitly’s internship program by exposing our engineers to newer techniques and alternative approaches that the students bring in with them and we also benefit from the act of passing on our collected knowledge to our interns (it’s hard to teach without learning something in the process).
“My internship at Remitly was my first ever software engineering internship. It was an unforgettable experience, and I learned so much. I grew a lot as a developer and became proficient in new technologies like Golang and Docker.
Interning at Remitly gave me hands-on experience that I couldn’t have had at school. What I’m most proud of was getting to deploy code to the Remitly codebase. It was quite surreal to see that code I wrote was out there doing stuff in the real world. I also enjoyed visiting the office in Seattle and meeting the team. They made me feel like I belonged there, and I felt valued, even as an intern. I had the unique chance to meet the CEO, CTO, and cofounder of Remitly, which isn’t something I thought I’d ever do. I learned a lot in just 12 weeks interacting with other professionals in the field I’m interested in, and it was an experience I’ll cherish for a long time.” — Solomon
We’re looking forward to Solomon rejoining Remitly this summer as a full-time Software Development Engineer.
Remitly’s Intern Program regularly employs students who are looking to gain new skills in software engineering and want to experience a role quite similar to a full-time Software Development Engineer (SDE). Our teams aim to provide projects that will be enjoyable and will solve real problems that will improve the experience for our immigrant customers. We also hire interns for other roles such as marketing, analytics and in other parts of our business. All of our open opportunities can always be found here.
