Treating an API like a real world store.
I’m working on a platform which involves an API with huge amounts of data, like a movie API, or an API which provides names of cities around the world. I wondered how do you work with all this data?
Do you always contact the API when a user needs a piece of data? Do you store as much information as possible in our own database? Do you store some info and query the API when you want more info then what is stored in your database?
I’m starting to think of this as a real world example. For example, what if I open a store that sells beer . All my beers are in the basement downstairs
( the API ) and I sell the beers upstairs in the store ( our database )
When the first customer arrives I will have to run downstairs to pick up a beer, lets say that five more customers arrive to buy the same beer. I might get the idea of putting a crate of that famous beer upstairs so I can give it to the customers fast. For our application that would mean that if an item is asked a lot, we save it in our database.
We can start thinking further, what if the customer wants more from that beer. What if they want a brewing system made for that beer? The brewing system is large and takes up a lot of space, so it’s not convenient to put it upstairs. In this case we might say that we always keep the brewing system downstairs. Translated to our application that would mean we save some of the data upstairs, but the request for all the data will always go through the API.
So how much types of beer will we store upstairs, knowing that our storage space is limited. For now I have the following in mind: Let’s say we have a cleaning crew that comes by every night and put’s everything left over upstairs, back in the basement. In that way, every time a customer comes by and asks for a specific kind of beer, we can bring a whole crate up and leave it upstairs.. It doesn’t matter if only one client wants this kind of beer, when we come back in the morning this process can start all over.
This means that every time the user queries the API, we will save the basic information in our database ( and cache, but that’s another story ). And we erase the database every week. If the user wants all information about a subject we query the API for the remaining information that not all users need.
(to be continued)