How do we track money at Fiverr

On Fiverr, we can now track every cent in our system. We can see where it resides, when it came in, and by whom. We also are able to link this cent to its respective payment transaction. In fact, we can map out the entire journey of this specific cent in Fiverr’s marketplace.

Fiverr is the world’s largest marketplace for digital services. It enables you to browse a large selection of freelancers, who offer their services, and to place orders in just one click.

The finance R&D team at Fiverr is responsible for everything related to moving money on Fiverr’s global marketplace. Our technology powers our massive daily transaction volume to collect payments from buyers and distribute payouts to sellers.

Fiverr’s in-house financial infrastructure allows for international transactions with various payment processors and currencies. Two main components of the financial ecosystem are the payment system and banking system. The first handles end-to-end payment and payout flows in the marketplace. The latter is responsible for managing Fiverr’s internal banking transactions and for delivering accurate, reliable, and comprehensive financial data to our stakeholders.

How do we track the money?

This is all about trees. I’m going to explain the implementation in detail; but first, let me give you a very brief introduction to our banking system architecture.

The Bank of Fiverr

The banking data pipeline consumes product events, such as new payment creation, service rendering, order cancellations and many more event types from the marketplace, moves money across internal accounts, and produces accounting information.

Each customer on Fiverr, both buyer or seller, has multiple virtual bank accounts. Every account plays a different role. For instance, the ‘Revenues’ account holds sellers’ income from delivered orders; the ‘Taxes’ account represents the amount of taxes users paid for Fiverr services; and the ‘Reversals’ account holds money from reversed transactions caused by order cancellation.

The user’s shopping balance is the sum of a subset of his banking account balances.

When a buyer clicks on the pay button, our banking system gets notified that a new payment was created and the incoming money starts flowing and splitting between the accounts of the user. Eventually, the money arrives at the seller’s revenues account, once the order is delivered.

Here’s a simple example of a payment for $10:

• $7 of the payment goes towards the service, which will eventually be delivered to the seller
• $3 goes towards fees and taxes

The transaction from the ‘Cash’ account to the ‘Transient’ account is the first transaction in the payment flow, triggered when a buyer clicks on the pay button.

Next, the incoming money is processed by transactions into ‘Taxes’, ‘Fees’ and ‘Revenues holds’ accounts respectively, splitting the overall $10 into $3 for taxes and fees, and $7 which are intended to be delivered to the seller. Once the order is delivered, the last transaction transfers the money to the seller’s ‘Revenues’ account.

The Need for Money Tracking

Although the banking transactions represent our operational flow, they still don’t show the full picture, in several aspects. We cannot link existing money in the system to the payment it came from; thus, we can’t categorize the funds in the system. Some of the pain points we have encountered are:

Credits Aging: Fiverr is about to launch a new feature that will allow us to grant users credits to spend in a predefined time span. This feature requires one ability that our engineering team needed to develop, which is prioritization of money in the banking flow. In case a user has multiple money types in his balance (cash, credits, etc.), we must make sure that credits with the closest expiration date will be used first while purchasing.

Payouts: We need a better way to link the payouts to the payments in the system. The lack of this information affected the quality of financial reporting for payouts and added extra manual validation work to our accounting department.

To support those features, we added a whole new entity to our architecture: The Money Bag. Incoming money to Fiverr’s system is represented as a Money Bag. Each Bag holds Money Bag Entries that are being created on each internal transaction of the Money Bag in the banking flow. Here’s how it works.

Each incoming payment creates a Money Bag. The Money Bag is linked to the payment and holds a money type (cash, credit, etc.), amount and other money transfer related metadata.

When a transaction moves the Money Bag to another account (for instance, the transaction from the ‘Cash’ account to the ‘Transient’ account in the example above), a Money Bag Entry is created. The entry is related to the parent Money Bag and represents a single step of the bag in the banking flow. The entry holds a reference to the parent Bag, the account it was created in, and its respective transaction.

The Algorithm

We can think about the problem of adding entries to the Money Bag’s course in the banking flow as a problem of constructing a tree, when the transactions are the edges and the Money Bag entries are the nodes. The input of our program is a transaction and the output of it is a tree of Money Bag Entries.

An incoming transaction moves entries from transaction.from_account to transaction.to_account by creating a new entry related to transaction.to_account. In other words, the transaction creates an edge from a node related to transaction.from_account to a newly created node, related to transaction.to_account.

Transactions that insert new money into the system, such as new payments and credit funding, create a new Money Bag with an initial entry that represents the root of the tree. The rest of the transactions move the Money Bag Entries among internal banking accounts, constructing the rest of the tree. Any data inserted into the tree is immutable and any operation applied to the data is idempotent, meaning that the history always stays intact.

In the above diagram, you can see a side by side simulation of the example payment flow from the beginning of the article with it’s Money Bag Entries construction.

Going into the details

If the amount of the transaction is smaller than the amount of the parent node, the algorithm will split the Money Bag by constructing two child nodes for the processed node. The result will be one child node related to transaction.to_account with transaction.amount and a second child node that represents the money remained in transaction.from_account. Otherwise, if the amounts are equal, one child node will be created with the exact amount of the parent. It follows that the constructed tree is a binary tree.

• For simplicity, we will skip scenarios where transactions’ amount is greater than the amount of the Money Bag Entry. In those cases, we process multiple trees in a single flow.

The leaves of the tree represent the active parts of the money bag in the system and the sum of their amounts is always the amount of the parent Money Bag. In our example, the leaves hold the amounts 2, 1 and 7 dollars, indicate that 2 dollars of the payment are in ‘Taxes’ account, one dollar is in the ‘Fees’ account and the rest are in the seller’s ‘Revenues’ account.

Purchases with shopping balance done with already existing money in the system, so there is no need to create new Money Bag at the beginning of the payment flow. In that case, the algorithm will have to determine with which already existing bag in user’s balance to proceed. The decision is made using a set of simple criteria as Money Bag type(credit, cash) and expirations date. In that way, we solve the credit prioritization problem, by prioritizing credit money with the closest expiration date.

The Data

The Money Bags data set provides us diverse insights using some simple queries. Here are some real-life queries we use.

• A query by payment will give us the full money trail of this payment in the system.
• Querying all active entries belong to a certain account will reveal the different origins of the money lays in this account — their age and type.
• To know where in the system resides the money from a certain payment, we search leaf nodes of a Money Bag that was created as a result of the payment.
• To get all expired credit money, we query leaf nodes by expiration date.

By getting the course of payments to a certain account, we have sufficient information to solve the payouts problem. A simple DB query but an important time saver for our accounting team.

Implementation Details

Storage: MySQL database fits well with our Banking System requirements, providing a wide range of essential abilities for banking operations (DB level transactions, versioning, easy joins and more). For the Money Tracking feature, we use MongoDB for it’s built-in sharding solutions and the performance advantages over MySQL.

Decoupling: Adding Money Bag creation to the existing Banking flow has one main advantage — being able to update the entries in real time and to be as up to date as possible. On the other hand, we would need to add extra workload and complexity to a very sensitive system. That’s why we decided to decouple the Money Tracking flow from the existing Banking system and process the construction of the trees on a dedicated process. In fact, the Banking system is unaware that the Money Bags exist at all.

Idempotence: Each operation made on the data set is idempotent, means that rerunning it over and over again will produce the same result. In that way, we avoid the risk of processing duplicate messages, simplify failure recoveries with the ability to easily replay previously processed messages, and in general, make our API more fault-tolerant.

Consuming Events: Decoupling means that we need to find a way to be synced with the Banking System transactions. Consuming banking events with shared resources (accounts) from a large scale, distributed system requires an extra consideration for concurrency.

Race conditions can occur when multiple transactions triggered by one or more customers are trying to move nodes related to the same account. The order matters and different trees will be constructed depending on which transaction acquired the lock first.

We have to make sure we calculate transactions by the order they entered the database, and cannot take the risk of having race conditions. For that purpose, we use a single writer architecture that leverages the global sequence offered by MySQL.

The banking transactions MySQL database maintains an auto increment id which allows us to easily iterate in sequential order over the recently created transactions. The iteration triggered every couple of minutes, processing the recently created transactions.

For now, we don’t have the need to serve real-time data to the consumers of Money Bags, so the periodic iterations work well for us. In case we need real-time, there are various solutions we can choose without changing our core architecture (MySQL triggers, for example).

Conclusions

Using a set of simple engineering tools, we solved a complicated problem in our product. With Money Bags, we now have greater financial integrity and full control over the money in our system. This ownership enables us to add the trust and the validity for Fiverr’s marketplace to serve millions of freelancers and customers.