Integrating Third Party Technology to DANA Microservices: A Backend Engineer Perspective
In DANA microservices system sometimes certain functionalities are delivered by integrating with third party technology. There are some challenges and risks that need to be mitigated properly. Some of those are inherent in third party system integration. The others are specific to microservices.
For conciseness, in this article the first party DANA microservices system is referred as to internal system and third party technology is referred as to third party.
[Technical] Dependency to a single vendor
When a certain functionality is solely dependent to a certain third party, a failure to this third party system can easily jeopardise the functionality. In ideal system, nearly all third party dependent functionalities have third party dependency alternatives. For example, internal system need to have multiple SMS Gateway providers. When a certain gateway is failed, the system can just use the other gateways. Having multiple third party providers is also enable us to choose the best solution by analysing their actual performance and pricing.
[Technical] User tracking without data sharing
Generally, user data is not to be shared with third party. This user data includes the seemingly uninteresting userId. In fintech, most of the uses cases involves user. Most of the time, it’s about someone doing something.
When it is needed to do some kind tracking or maintain a stateful process that is tied to individual user, a shared reference between internal system and third party system is to be used. In actual implementation, this shared reference can be named: sharedId, externalId, entityId, etc. This shared reference can be generated by either internal system or third party. This shared reference is mapped by internal system to the actual user data. This mapping should not be disclosed third party or public.
If the shared reference generated by third party, the mapping to user data is calculated and stored after receiving shared reference from third party. This shared reference shall be used to interact with third party. Internal user data is still undisclosed and shall be kept undisclosed.
If shared reference is generated by internal system, the basic idea is still the same as third party generated shared reference. The differentiating part is that the shared reference is generated before submitting it to third party.
[Technical] Microservices complexity
Microservices environment is where monolith never appears 😁.
Using microservices is inevitable in financial technology. Microservices separate concerns and break down complexity which good for a complex system like fintech. However, they also introduce a new problem: intercommunication complexity. New development in microservices can easily introduce multiple communication paths.
Third party services usually treat internal system as single entity. There is just simply no way to tailor made a third party service based on internal system’s microservices configuration. Third party is not aware and should not be aware of the microservices configuration.
To handle this complexity the followings need to be considered:
- Each microservice need to have clear responsibility and not overlaps with one another.
- Each microservice has clear data input source and data source destination.
- Each microservice ideally need to be testable on its own (with mocked data path).
- Communication that doesn’t require immediate response can be implemented as asynchronous.
- Persistent logging mechanism need to be implemented for analysis and troubleshooting.
[People] Team size and selection
During development, team will be having intensive communication both internally and with third party staff. Team size should not be too big because it will introduce some communication issues. Development-wise, there should be around 4 developers working together in one sprint. If the development is too big, adding more sprints are preferred compared to adding more team member.
Team members that have been working together and have knowledge regarding internal system are preferred. Such team members only need to learn the new third party system. However, new team members need to learn internal system, new third party system, and team work with new team. Sometimes, we don’t have the luxury to choose preferred team members and the sprint scope and timeline need to be adjusted accordingly.
[People] Trust in third party staff transparency
Humans are emotional creatures. A total honesty is simply not possible. Often, staffs of third party system are incentified to make their service got used. Also, It’s nearly impossible to implement a bug-free service. Overpromissed and underdelivered service is to be expected. The development plan should put this into consideration.
On technical side, the development should include case where there is condition outside the service specification. Let say the specification says the message SUCCESS or FAILED shall be delivered under 5 seconds. The developement should also handle the message other than SUCCESS and FAILED, and implement timeout mechanism. A robust logging mechanism also helps mitagating this issue.
On non-technical side, a communication channel between internal staff and third party staff need to be established to get a proper support. This is usually a part of agreement.
