Looking for a data fetching survival kit for all the REST single page apps of your life ?
Some of our technical choices probably deserve to be shared for the coming experiences of state startups ; the source code of the pass is available on the github of betagouv and we have developed freely accessible npm modules centralising features used on our both web apps. Two of them are about normalising and synchronising the frontend data from our fetch api: fetch-normalize-data and its little brother redux-saga-data. All of it is licensed under the Mozilla license.
N.B. : Had you not had a global presentation of the project, we suggest starting off by reading our first post.
What on earth is the pass Culture ?
In January 2018, the first lines of code were laid down. After six months, we had a first draft, enabling us to initiate the experimentation phase. The architecture is mainly made up of 3 services:
- An API server with a relationship database, that handles recording and redistributing all necessary information to both web apps when required.
- A professional back office website for professionals, that is at the origin of manual data creation within the pass Culture. At that point any actor can declare a cultural offer (offer, in the database) with necessary descriptions. One can enter associated stocks, availabilities, price and dates of the occurrence when dealing with an event. It is also strongly recommended to add an image, acting as the first depiction of the offer.
- A progressive webapp aimed at young beneficiaries of the pass. Where the app puts forward offer recommendations (recommendations) through two interfaces: a carousel of the most relevant recommendations according to the context, as well as a browsing page, allowing to find offers dedicated to one’s request.
The web applications of the pass Culture
In order to enable universal access for beneficiaries, the app should foremost be a web app.
Various arguments have lead us to develop the front in React. It is vital to deliver an attractive user’s experience right from the start for young testers who are accustomed to social media (snapchat, instagram, facebook…). Just as in those apps, our goal is to implement soft transitions in browsing and make interactions with the backend look invisible. Despite web constraints, it is possible to recreate such an experience by coding the website like a progressive application with single page application format.
React is obviously not the only tool allowing this, but its current popularity alongside satellite libraries (react-router, react-redux, …) has worked in its favour. We also liked the thought of being able to easily switch to react-native if ever coding even more resource intensive animations came around. We needed to code the back office website for professionals simultaneously, gathering a great number of form pages with complex fields.
The React Components are very useful in this context to write reusable bits of code in between pages (note for the remaining of the article: the word “view” shall be used to refer to any function or React Component allowing to render html in the app).
react-redux
Within a small team, it is best to reduce API surfaces to keep and to spread to newcomers . The back office and the beneficiary app are two distinct web services but it has been important their external used libraries, structures and syntax rules do not diverge too much.
Following this logic, organising flow management the same way has been useful thanks to a react-redux and redux-saga management. Each application holds a data cache (state) created dynamically when setting up the service, evolving according to interactions on the website. The magic of react-redux spares us from coding the warning of different views when the cache changes throughout time: we write how the react components select the specific data in the state to fulfill their display functions, and this suffices.
In most practices, each app codes its reducer functions. These very functions will decide on how to make state evolve based on their observation on actions emitted by views. Let’s take a back office example:
The “Manage dates and stocks” button emits a SHOW_MODAL action by clicking on it, informing a modal reducer to shift the state.modal.isActive value from false to true. The change of this variable automatically tells the Modal Component to appear on screen, as well as the elements behind the window not to be active for the users as long as the window is visible.
Your mission, should you accept it…
The whole idea starts to get tricky when dealing with interactions in charge of modifying data in the backend database in a persistent way. Let’s take the example where a cultural actor adds extra stock in the back office for one of his offers:
Here is the issue at hand: clicking on the “OK” button triggers a request for the api to create a new date. If nothing else is coded on the frontend, we fall into a situation where the screen shows two dates, whereas three now exist in the database. As long as the user doesn’t refresh the page himself, the back office will display a desynchronised state of the data.
To avoid it, we have written a data reducer in charge of aggregating all information related to the database entities. It is in charge of keeping an up to date record of offers, stocks and recommendations manipulated during a session. That last point was ground to find interesting management rules to share. Let’s break down the key steps of the code allowing adding this date:
- On the url /offres/F5UQ, F5UQ being the id of a particular offer. The main view sends the REQUEST_DATA_GET_/OFFERS/F5UQ redux action asking specific information related to this offer to the backend. Reminder; the code of the pass Culture is open: if you look closely, you will notice that an action creator is used, requestData, through which necessary setups are patched for the request: apiPath (path of api) and the method of protocol (DELETE, GET, PATCH, POST).
- redux-saga, receptive to this action, executes in a synchronous way a fetch GET /offers/F5UQ.
This request returns a data package which shows the first level details of the offer (name, duration of the event, venue…) and in encapsulated levels, precisions on entities related to the offer like stock.
- redux-saga finished up its work by asking for a new action sealing the request’s success: SUCCESS_DATA_GET_/OFFERS/F5UQ, in charge of enriching the state.data with new data.
- Recorded entities in the state data are normalised (the stocks encapsulated in the payload are laid flat in the cache), this point is explained further below.
- The StocksManager Component is now able to fetch data to display two stocks in the window via selectors.
Next come the steps that define adding a new stock:
- At the time someone is done filling in the adding form and clicks on OK, the code triggers an action which is similar to the first sent REQUEST_DATA_GET_/OFFERS/F5UQ. The action is known as REQUEST_DATA_POST_/STOCKS as in this case we define apiPath /stocks with a POST method. Furthermore , the body of the request holds the information of the forms.
The data flow is undertaken the same way: the return of the request POST/stocks triggers an action SUCCESS_DATA_POST_/STOCKS that refreshes the global state of the app by adding a stock object in its state.data.stocks table.
- The StocksManager and the stocks counter are receptive to state.data.stocks get updated thanks to the react-redux stream.
One solution: generalising
To sum up, we have realised that we were confronted with the same needs when dealing with synchronizing the state of each webapp with the api’s. We have come up with triggered standard actions for the beginning as well as the start of each transaction with the backend. A redux-saga logic ensures an asynchronous hook in between actions REQUEST_DATA_(GET|PATCH|POST|DELETE)_<COLLECTION NAME> and SUCCESS_DATA_(GET|PATCH|POST|DELETE)_<COLLECTION NAME > (we also have FAIL actions of the same type to deal with requests managing statuses 400 or 500).
That way the state.data becomes a continualy synchronised state of the item of the backend data. All that is left for the developers to code is calling requestData in the right places (particularly by writing this logic in the mapDispachToProps), as well as the specific access to this data via the writing of selectors in mapStateToProps.
redux-saga-data contains the three mandatory items to make any project work in this framework: createDataReducer and watchDataActions to respectively use at the time of the setup of redux and saga in the app, along with previously mentioned requestData. A codesandbox is accessible here to show its use. This proof of concept reenacts in a simpler way the example of adding stock as shown above.
One can understand the point of having normalised the stock data in this case.
Without it the offer’s stock would still be held in the object state.data.offers equal to [{ id: “F5UQ”, stocks: […] }] and we would have had to write in the file mapDispatchToProp.js a resolution method allowing to update this sub list by adding a new stock. One way to proceed is shown down below l.43–45:
Instead of writing a resolver, specifying the normalizer option is preferred in this particular case, allowing to lay flat stock entities in state.data. See above in the commented code, the “stocks” key in the normalizer which means that the table of objects from the payload collected at that place will be laid in state.data at the key specified by stateKey (equals to “stocks”). This particular case enables action.payload.datum.stocks to merge into state.data.stocks.
How about even more generalisation ?
Updating a normalised state of the backend data is a recurrent need in the world of single page apps. We would like to tell ourselves that whatever the action management library that we choose, it would ideal be to own the code of the concept in a heart-library. This is what we have done with fetch-normalize-data. redux-saga-data is in fact just a wrapper of this to make it work with redux-saga.
Furthermore we have chosen to use redux-saga to deal with asynchronous actions. It is to be honest a debate-able option, regarding other libraries that would be lighter to perform this, as redux-thunk. So for those who prefer, a redux-thunk-data also exists (see the codesandbox). Lastly for all the “reacto centered” people ( who want to lighten their bundle) we have developed a react-hook-data mimicking and using the new api from React as useEffect and useState along with Context (see its codesandbox).
Conclusion
Pass Culture is a technical project playing with the latest “trending” tools (webpack, React, redux, sass…) for frontend development. Some working days developing resemble a research lab for Javascript, which partakes in the approach of the betagouv community and on a broader scale that of open source development. We would like to see our work contribute to the idea that building shared objects can facilitate the development of common software.
We hope that this article will draw that attention to those curious of the technical stakes encountered in this project. There are plenty more npm libraries with open source libraries on betagouv that we have developed, but this shall be the topic of other posts.
“How about simply using graphql or react-appollo? Wouldn’t it work with realm, to do a bit more privacy by design? Vue.js is great as well!” They’re all options. The pass Culture is still a minimal state product, and we have shown here just a few of the tools that have enabled us to reach the first step of technical consolidation.
But it is a highly ambitious project — There are also python stakes with the recommendation algorithm under construction — one thing is for certain: we’re hiring! If you wish to contribute to the discussion, maybe even partake in the technical evolution of the pass, do send us an email (support [at] passculture.app)
Thanks to Gregoire Berthon who did the translation of this article from this post.
