Portfolio Tracking at AngelOne
Introduction
Portfolio is a feature provided by AngelOne, to help customers track the performance of their equity holdings.
Below is a sample portfolio screen from the AngelOne app. It shows the total amount invested, its current value and the unrealised profit / loss made. Similar values are also provided at the level of individual holdings along with additional information like the average purchase price and the number of units held.
This is one of the most important screens of the app, and receives tens of millions of views every day.
This feature has been in our offering for more than a decade now. Over this time, the application collected a lot of technical debt. Numerous attempts to improve it, only extended the system with more workflows and more layers of caches, adding to the debt instead of fixing it. Clearing the eventual debt became tantamount to a rewrite. So, we decided to rewrite it. We followed the strategy of Asset Capture, taking over bits of holdings, live feeds, average price, API etc. from the main application till eventually the whole service was migrated to the newer setup.
Rendering the portfolio at a high level, requires us to process realtime position change feed, trade files from exchanges and holdings from the depository as shown in the diagram below. We tried to keep our design simple and hence -
- Kept the pipelines for the 3 sources separate till the final sink, and did not mix them in any way.
- Kept the overall touch points from source to destination minimal, so that failure points would be less.
- Implemented a REST API layer, which would merge the sources at the time of query.
The rest of the article will talk about the average price pipeline, the most complex of the 3 pipelines, followed by some interesting findings we had while implementing the API layer.
Calculating average price of holdings
FIFO
Indian tax laws require us to use FIFO methodology to calculate the holding value of equities. Running FIFO for average price requires us to match the oldest purchase made by a customer with the oldest sale made, and then calculate a weighted average on the quantity remaining.
E.g. Let us say you purchased 10 shares of RELIANCE for 1600 on 16–06–2020 and another 10 shares for 2000 on 23–07–2020. You then sold 5 shares for 2500 on 20–06–2022. This sell will be knocked off against the earlier buy of 16–06–2020. Then the average price of the holdings would be — (5*1600+10*2000)/(5+10) = 1866.66
We need to run this at scale across all the transactions, to calculate the new average prices on a daily basis. The earlier setup had implemented the algorithm through stored procedures, but would take close to an hour to finish execution. We soon realised a need to run this calculation on a distributed data setup, and since AngelOne has a lot of pipelines running on Apache Spark, we decided to run the FIFO algorithm on the same.
The calculation of FIFO is complex enough that we could not run it fully using the native SQL like APIs provided by Spark. For the complex bits, we used Spark API’s applyInPandas, which allowed us to run the calculation on native Python, without loosing on the distributed nature of the overall execution. However applyInPandas is an IPC via Arrow. Hence its performance cost will be high, and it is best to use the native SQL APIs to filter and transform the data to the maximum extent possible, before handing it off to the pandas. The FIFO calculations are now down to ~6 mins.
Corporate Actions
The shares held by the customer can change due to numerous events collectively called corporate actions. These are usually done in the form of split, bonus, merger and demerger. When these events happen, the price of the stock changes to reflect the new value of each share. The customer does not loose any of his invested amount, but the purchase price of the stock needs to be appropriately changed to reflect the revised price of the share.
E.g. Let us say you bought 50 shares of TATASTEEL some time before 28–07–2022, for 600 each. On 28–07–2022, when the stock underwent a split of 1:10 you would own 500 shares, and the average price of the stock needs to be changed from 600 to 60, to reflect the new attribute (Face Value) of the share.
Our earlier mentioned Spark setup, is reused here, to recalculate the newer purchase prices for all the shares that go through these corporate actions.
Off market Transactions
Sometimes the transaction happens completely outside the exchanges. E.g. IPO, FPO, Exercising rights, Expiry of bonds etc. These transactions are very essential to keep the overall portfolio in balance. With insights from our senior team members, we managed to find the sources for these transactions, and keep the portfolio value accurate.
We will now discuss how we present this average price to our customers.
The Portfolio API
Hosting and implementation
The earlier API was hosted in a private cloud, and had scaling limitations commonly associated with the private cloud ( self managed infrastructure, acquiring hardware takes days etc.) Also the earlier API had implemented materialised views joining all the core entities, with workflows to keep the data fresh. Further caches were implemented on top of the materialised view. However on running performance tests, we realised that this was early optimisation, and the application could scale very well, without the cache and the materialised view. So we implemented the newer API through a Go server, which on each request, queries all the entities through parallel Goroutines, and combines them into the response object.
The application is now hosted completely on AWS ECS. We did face some performance issues during our testing, but found it to be a known issue with the native cloud watch based logging. Once we switched the logging to AWS Firelens, the performance returned to acceptable numbers.
A lot of our customers have large portfolios, so we also enabled HTTP compression on our responses reducing payload sizes by close to 90%. While this added an additional latency of 2–3 ms at the backend to compress the payload, the latencies on the customer side reduced by ~100ms, at P90.
Choice of storage
Given the high throughput expectation from the service, we did consider NoSQL stores like DynamoDB and Elasticache. But their support for our use cases like secondary indexes, and ease of use with Apache Spark was less than ideal. Hence we chose to go with MySQL Aurora. Aurora provides ability to add slaves dynamically, and once we clustered our data by customer, the latencies and scale were comparable with other NoSQL stores.
Conclusion
The newer Portfolio service is available on the AngelOne iOS app, for some time now, and will soon be available on Android also. This project could not have been completed successfully, without the help from our senior staff engineer Rakesh K K, business insights from Rahul Shah and the overall delivery from the Portfolio team.
