HLD caching framework in Mobile Application At Simplilearn
Simplilearn is a provider of online training and professional-certification courses intended to help working professionals achieve their career goals.
The Simplilearn Android app consists of a vast catalog of courses, their details, and a seamless learning environment where our users can learn on the go.
Fundamentals:
The typical app architecture contains three layers: UI Layer, Domain Layer & data layer. Let's just brief about three layers first and deep dive into the Data layer.
UI Layer: Also called a presentation layer is used to show the data on the screen in an organized way, with which the user can interact. UI will reflect accordingly when there is a change in data, either for preloaded local cache/network response or with user interaction such as scrolling/clicking.
Domain Layer: In Simple words, this layer encloses the business logic of the module.
Data Layer: Data layer holds the logic to transform data from network model to room model. This transformation is what gives value to your app — it’s made of rules that determine how your app creates, stores, and changes data. The data layer at Simplilearn includes 3 main things.
This blog is deep diving into the data layer to explain the framework setup that is created to perform multiple operations on fetching, storing, caching and emitting the data to the next layer.
Requirement:
The requirement was to cache the network response on all the screens of the app, not just to show cached data when the user is offline more but to have a better user experience for the user by preloading the data on the screen while the api makes a fresh call. However not all the screens will update the UI after api call depending on the module’s configuration data manager will behave differently. For instance, a course list whose data will not change regularly and is not critical can have a silent api call(Makes api call, updates the DB but doesn’t update UI), this data reflects on the next launch. Whereas the home screen where content is personalized requires updates on every launch.
We maintain a cache period of 7 days and critical updates will override the cache data at any given time.
One important thing to note is that we always fetch data from DB and never consume api data directly. Also, we maintain 2 models viz, Room & Retro model to differentiate network and local data. We do this because multiple modules demand data transformation & data creation to operate on the business logic, this also creates a clear separation of the network model and database columns. A follow-up question would be, do we save the network response directly as JSON, No. We create tables with entity models as we have to query the tables from multiple locations.
Repository fetches data from Remote Data Source[Retrofit] layer and inserts data into Model[Room DB] layer then Repository passes data from Model layer to UI Model in this case Model layer is our Single Source of Truth(SSOT).
Android documentation describes an algorithm for providing data to an app by either retrieving recent data from a local cache or loading the latest data from the network. The article also provides code snippets showing how such an implementation might look. The NetworkBoundResource class is at the core of the implementation, and using the algorithm only requires one to subclass NetworkBoundResource and override a few methods, usually requiring only a few lines of code.
While working with large-scale applications it’s obvious that Application has to call multiple Api’s from a single screen and large applications have multiple screens, but calling multiple Api’s every time while navigating is not a good practice which can create bottlenecks and unnecessary load to the server.
To solve this problem. If the cached data is no longer fresh then it creates a network call to get fresh data from the server. Due to the Single Source of Truth, we can handle no internet or low internet connection problems because data is always available from local storage.
Cache Framework:
Let’s take an example of a repository, which handles personalized dynamic home screen data, HomeScreenRepository, could depend on other repositories such as Persistence and Service to fulfill its requirements. Below is a simple architectural diagram explaining the components, data flow, and methodology.
Let’s discuss some of the abstract methods defined in NetworkBoundResource,
loadFromDb(): this method returns data from a local database.
createCall(): this method is responsible for creating a new request call to retrieve fresh data from the server.
shouldFetchFromServer(): this method checks if the data in the local database is stale or not, if yes it provides instructions to create a new network call which is also responsible to avoid multiple network calls.
saveAPIResult(): once the request call is made then the caching logic is implemented in this method to insert fresh data into the cache storage.
To expose our network state we can use the Resource class which is a generic class responsible for emitting, loading, success, and error state.
1. Initially our NetWorkBoundResource emits a loading state till we get the data from the room database, data might be null if it’s a fresh install in that case network call is made, once the data is fetched then it emits a Success state to the UI, and displays the fetched data from the Room Database.
2. The shouldFetchFromServer() method checks the condition if it needs to fetch the data from the server or not or if it returns true the network call gets initiated once the network call gets completed and Response from the server is not empty it will emit success State and the fetched data gets cached in room database.
3. Now cache data is updated and NetWorkBoundResource is responsible for sending this data to the UI model to display the fresh data.
4. If the response is empty or anything that goes wrong with the API call the NetWorkBoundResource emits an Error State and it passes previously cache data to the UI Model, if data doesn't exist empty screen is shown.
5. If shouldFetchFromServer() returns false it means no need to call API again and just emit the previously cached data to the UI layer.
6. If an API call is made the first time, shouldFetchFromServer() should return true, and subsequent calls should handle the logic of caching and expiry. If data doesn’t exist and no internet connection it’s our responsibility to show an Error message on the screen.
7. If cache exists and it’s not expired then shouldFetchFromServer() will return false which means no need to create an API call, and if cache exists and it expired then shouldFetchFromServer() returns true which will create an API call and update the cache data.
Let’s look at the code snippet below,
Previous Implementation:
Reworked Implementation:
Rework code snippet explanation :
With reworked implementation, you still see the data flow and operations remain the same as before. It adheres to the architecture that we have defined above. However, with latest releases from Kotlin which is coroutines and flow has made it much more simpler to handle background tasks and great support for observer patterns. With flow, we were able to get rid of manual thread handling(We used Thread pool executors to manage API/DB IO calls) as it packs coroutines under the hood, reduced boilerplate code at anonymous class usage place, Unlike enum where we can’t define different functions in each enum constant we used sealed class to overcome that.
Conclusion:
Once you define the above function, developers can purely concentrate on the business logic rather than thinking about how to save and retrieve data. However, developers have to write additional code, to handle cache time period, api calls, db calls by simply defining functions accordingly and passing higher-order functions to networkBoundResource’s loadFromDb, getDataFetchDate, shouldFetchFromServer, createCall, and saveCallResult.
I hope this article was helpful and gave some insights into how we cache things in our Simplilearn Application.
At Simplilearn, It’s quite intrinsic to adopt and implement new tech stacks and bring up the technology to the next level. This approach helps the project to be more modern and tech upgraded and reduces the complexity of managing multiple lines of code to a few.
