Automating B2B payouts

OneFootball as a content platform strives to provide value to not only the millions of football fans around the globe by enabling them to access a plethora of exciting content but also to our B2B partners by providing them access to our 100M+ monthly user base via their content and sharing a portion of our revenues with them.

We recently finished an overhaul of how we handle the payouts to our B2B content partners. We automated the entire process both for internal (i.e. our B2B partners) and external (i.e. our Finance department) users of our payouts system.

This was a true cross-team endeavor and it’d be remiss of me to not mention the amazing collaboration and support from our finance department, legal department and the platform infrastructure team.

Old way of working

Old architecture

Before we embarked on this project, the payouts to our partners was a time consuming manual process.

Our B2B partners had to manually fill in an invoice template and upload it to our B2B portal. This meant that our partners had to make sure that they were cross checking all the details such as payout amount and bank account etc. before uploading the invoices, and many times human error cropped up.

And, once the invoices were uploaded, they were transferred to our finance department and then the finance department manually processed them. There was no visibility to our B2B partners over their payouts, as we couldn’t convey back the status of the transactions to them, and this problem was amplified due to the fact that payouts could take up to 30 days to be processed.

Clearly, this wasn’t the most efficient process and there was room for a lot of improvement.

Mission

We started the project with two main objectives in mind: automating the manual process and along the way making it more secure.

Automating the process meant that we had to stop requiring our partners to manually fill and upload an invoice every month, and also save the finance department from having to manually process hundreds of invoices each month.

In terms of the security of the process, we wanted to make sure that access to all the sensitive information (i.e. PII and banking data) conforms to the “Principle of least privilege” (i.e. only those entities have access to the data who absolutely need it), and that all payout transactions were traceable to identifiable entities with no discrepancies.

New way of working

New architecture

Automation

We leveraged PayPal as a payment processing platform to automate the process.

We made use of the following components from PayPal’s platform:

• Drop-in UIs: We use them to enable our B2B partners to convey all the sensitive information (e.g. banking data) directly to PayPal’s platform, thereby, reducing the risk area.
• REST API: We use it to trigger payout transactions and orchestrate other tasks in the process.
• Webhooks: We use them to track the status of payouts and any potential failures, thereby, increasing the visibility over the transactions and consequently enabling us to take proactive measures in case of failures.
• Back office system: We use it to enable our Finance department to have full control over the payout transactions, all the while saving them from having to do manual bookkeeping.

Security

We also paid due diligence to ensure that our systems were secure and couldn’t be exploited as a backdoor to our payment processing platform. As part of this effort (and a parallel engineering wide push for strengthening the security of our platform) we made a number of important changes to our infrastructure:

• AWS Secrets Manager for application secrets: We restricted access to our application secrets by migrating them from our deployment cluster to AWS Secrets Manager. This meant that the secrets were no longer accessible through VCS and CI/CD systems.
• AWS IAM access for DB: We hardened connections to our B2B database by authenticating them through AWS IAM role. This alleviated the need to persist database credentials in our B2B applications.
• k8s RBAC authorization via AWS IAM: We introduced role based access to our k8s cluster (which is where we host our B2B applications). This change meant that only a limited (and necessary) number of entities could gain access to the cluster, and through that to our application secrets and B2B database.

Challenges

The project was anything but smooth sailing. We faced a number of product & engineering challenges, some of which we overcame, others reinforced known best practices, and the rest were valuable learning lessons for the future.

A few tips based on this experience:

• It’s essential to have high-fidelity UX design artefacts before starting the implementation of a user story, even if they’re expected to be minimal.
• All optional features need to be triaged and only prioritized if they block a required user story.
• A high-level architecture of a new project helps with uncovering any technical limitations of the project ahead of time.
• The database driver should refresh the AWS authentication token when opening new database connections via AWS IAM access. This is not a problem when an application is started, but as the token expires after 15 mins, any new connections added afterwards to the connection pool would fail, if AWS authentication token is not refreshed.

Impact

As a result of the changes mentioned above, we not only fully automated the B2B payouts to our partners, but in general made our B2B systems more secure.

The new process is a huge benefit to our Finance department who no longer have to manually process hundreds of invoices from our B2B partners every month.

It’s an amazing value for our B2B partners, who can now receive instant payouts on a single click of a button.

And, it’s a big boost to the observability of our systems, as we’re now in a position to accurately track the lifetime of a payout transaction and keep all relevant parties informed.

Join us!

Does this sound like something that you would like to work on?

Are you excited by creating value for different domains of a business?

Do you like working with cutting edge technology and leveraging it to build seamless solutions for users?

Are you a football fan and believe in the values of sportsmanship?

If the answer to any of these questions is a resounding YES!, then go ahead and apply for our open positions (in our B2B engineering team, and all other amazing teams & departments), we look forward to hearing from you!