How we created a custom Android view which draws a glowing progress ring with rounded ends.

With Glose you can set a goal of reading a given number of pages each day. You can then follow your daily progress with the daily goal widget on the home screen:

The first implementation of the widget used a circular ProgressBar to represent the progress ring. This did the job but missed a few features we wanted:

1. The ends of the progress ring should be rounded. With the circular ProgressBar the ends are flat and do not match the overall style of the app.
2. We should be able to use any drawable for the progress ring, like a gradient for instance, and not be limited to a solid color.
3. The progress ring should project a glow when you’ve exceeded your goal. The more the goal has been exceeded, the bigger the glow effect.

With these features in mind, we set out to update our implementation of the daily goal widget.

Rounding the ends

We couldn’t find a way to create a drawable for the ProgressBar that would draw rounded ends on the progress ring. So we replaced the ProgressBar with a custom view inspired by this StackOverflow answer.

This custom view sets up the two Paint objects that will be used to draw the ring:

ringPaint is configured with the ROUND stroke cap, which is how the progress ring will get its ends rounded.

The drawing of the ring is then performed by the onDraw() override using simple drawArc() primitives:

arcRect is a RectF that caches the bounds of the arc used to draw the ring. It is updated each time onSizeChanged() is called:

And that’s all it takes to get nice rounded ends:

Using a drawable for the progress ring

The progress ring is also used in a popup panel showing historical data. In this popup the ring must be drawn using a gradient that is lighter at the top and darker at the bottom.

To achieve this, we’ve updated the onDraw() method so that the arc it draws acts as a mask on top of the ring drawable:

The key here is ringMaskPaint, which is used as the paint for the second Canvas.saveLayer() call. It is configured with the PorterDuff.Mode.MULTIPLY transfer mode, and as a result anything that is drawn in this layer (the progress ring) will act as a mask over what is drawn in the layer below (the ring drawable). See Porter/Duff Compositing and Blend Modes by Søren Sandmann Pedersen for an illustrated explanation of these concepts.

The onSizeChanged() method is also modified to adjust the bounds of ringDrawable appropriately:

And here is the result:

Adding the glow effect

The progress ring is expected to glow when you’ve exceeded your reading goal, and the more you’ve exceeded your goal, the more the ring glows. We achieve this in the onDraw() override by drawing an arc with a Paint object configured to draw a shadow layer below the arc. We’ve also added the following attributes on the custom view:

• inset defines the dimension by which the progress ring is inset inside the view bounds to leave space for the glow effect.
• glowColor is the color used to draw the glow effect.
• glowPeak is the ratio beyond maxProgress where the glow effect reaches its peak.

And that is what it looks like in the daily goal widget:

Animating the ring and putting it all together

Lastly, we added a setProgressAnimated() method on the custom view that we use to animate the progress ring from zero to its current value. This uses a ValueAnimator that updates the progress value from zero to the desired value. The duration of the animation is adjusted so that the progress ring grows approximately at the same speed across the progress value spectrum.

This is how the progress ring is animated in the popup panel:

And that is how it is animated in the daily goal widget:

Here’s the same animation in slow-motion:

Curious about what it looks like in real life? Go ahead and install Glose for Android, and let us know what you think!

A glowing progress ring with rounded ends for Android was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

We (the Glose’s Android team) wanted to share with you some of the latest fun we had developing. As an e-book reader getting offline is an important part of our app, we were already able to bring our books off the grid, but we wanted to improve how our users can add annotations, reactions and highlights to their favorite books even without internet connection. After some thinking we decided to go with the new WorkManager library and to see where it leads us. We ended up with something we found interesting and wanted to share it with you and have your thoughts on it.

Intro to Redux

First of all a quick recap for those who aren’t familiar with the Redux Architecture which is the architecture we are using for our Android app and it will have its importance. It’s an architecture coming from JavaScript and React world.

It is mainly based on 6 kinds of components:

1. Store: is holding an immutable instance of the app state
2. State: an immutable instance describing the state of the application
3. Actions: are representing how we want to change State
4. Reducer: a pure function that creates a new state given a previous state and a dispatched action.
5. Middleware: are components that also intercept actions like Reducers but their goal is to perform other actions than changing state like deleting files, dispatching others actions, logging etc. For example most of our job on WorkManager will happen in a middleware
6. UI: is all your views, components that are listening for state modification and rendering accordingly to those changes. They also trigger actions on user interactions.

Main benefits of this are the single source of truth coming from Store and States, we have a simple unidirectional data flow, and all our components are easy to test.

I won’t be covering any details on how to implement Redux on your Android app here but if you want to learn more about it I recommend some nicely written articles about it at the end of this article.
However here it’s important because the abstraction given by Reducers will make the offline switch way easier.

Intro to WorkManager

WorkManager is a library part of Android Jetpack responsible for covering background’s work. There are many reasons that make WorkManager a strong tool, I will try to give you a short list of them:

• It offers a unified interface and is backwards compatible up to API 14, that’s been said you won’t have to choose between JobScheduler and AlarmManager.
• It’s highly configurable and really easy to use: you can both use one-time tasks or periodic tasks, you can constrain your task to only be executed on some conditions depending on battery status and in our case network status. You can specify backoff criteria, set delay and more.
• It deals with power-saving features so we won’t have to think about it.
• As the documentation states: “WorkManager ensures task execution” you are sure your job will be done one day, that’s the complicated part you can’t really know when it happens.
• It also supports task chaining (we won’t be using here but can be very convenient)

WorkManager is a very powerful toolbox for background processing but considering the fact you can’t know when it will run it might not be the best choice to run network requests initiated by UI actions. That’s where the Redux magic does the trick, and some thinking, the reason we could choose WorkManager for our problem was the fact we won’t apply it to all our requests but only for some user’s inputs like posting quotes, highlights and such.

Let’s jump into some code now that basic logic has been set.

What we had before

To understand how we achieve our actions independently of the current network connectivity let’s see what we had before using the Redux pattern. To keep this short I will only use the annotation’s case because it’s pretty much the similar process for other parts.

What we call an Annotation in Glose is adding a comment on part of a quote from a book.

So somewhere in our UI in a click listener we had this line:

This a typical Redux call where we dispatch an Action (here the Post action), this will produce two things:

1. Middlewares that are listening for this kind of action will do their job. Here as Post is a particular kind of action of our own (a RequestAction) it will perform the corresponding HTTP call (I’m simplifying a little bit but those are the large lines).
2. The action will be reduced, that means if one or more reducers listen for this kind of action the associated states will change accordingly to the reducer effect. At this time there was no reducer for this action.

Let’s have a look at Post implementation:

Basically Post described the Retrofit call to perform and what to do on success and on error. Note here that we also dispatch results via a Redux action so there will have a reducer changing the app state according to the newly posted annotation like this:

This what a reducer looks like, it just creates a new state depending on the action being dispatched. And then the UI listening to those changes could render them (not really interesting here).

Here is what happened:

What we have now

So the goal is to add some magic somewhere with the WorkManager to free ourselves from network connectivity status. To do so we are going to wrap the HTTP call inside a job for the WorkManager and have an optimistic approach and proceed immediately as if it was a success.

Let’s see what changed from the last chapter.

First of all we now have:

So here instead of directly dispatching the Post action we are dispatching an Enqueue action that will enqueue an offline action with some dedicated “ActionParams” as described below:

The local id is used to identify offline content (not synchronized yet), it’s also used if we need to identify a job to cancel (for opposite operations like post and then delete post etc. which will be defined in each implementation). And the buildAsyncAction() function will be used to create the actual request action to perform using the params in each implementation. So let’s add those action params to our previously described Post:

So we moved all parameters from the request action class to the Params and this last one is able to build the source action when buildAsyncAction() will be called (in the worker, it’s coming soon I swear).

But what really happens when we dispatch the Enqueue action with those params? This is where the Redux magic happens :) This is operated by two kind of components:

Reducer

A reducer watching for the Enqueue action in the optimistic way to simulate this Post. Previously no reducer were watching for the post action, but now we have one watching for the Enqueue with some Post parameter:

You guess the logic is pretty much the same and mostly shared between “boring internal logic” I will skip the detail but we had to add a cleaning offline shreds in the post annotation to keep our states clean and the fully new one was really the same than what was done before except we faked items coming from the api with some local data (mostly local id we were talking about before).

If we only kept to this part everything would have happened locally and so far our users wouldn’t be able to tell the difference between before and now on their device locally.

Middleware

As described before a middleware is in charge of different kinds of behaviors that are not changing states (that’s the reducer’s responsibility) so here it will be in charge of managing our WorkManager.

So when it catches an offline action we create a one-time work request (no need for a periodic one here :D ) send the parameters data through the input data logic so that the worker can have access to them and use them (in next code sample), we specify a backoff criteria to retry if something goes wrong, and finally we add the network constraint. We can enqueue this work request as a unique work with the local id generated before. Last lines are here if we want to cancel a job in some cases (changing highlight colors etc.). I won’t talk too much about this but you can cancel work this way :)

This will enqueue a work to be done somehow when constraint will be satisfied and here is the worker:

So our worker first grabs the previously serialized params, those two lines aren’t perfect but they seriously do the job. As we are supposed to have network connectivity now we can dispatch our request through a request action just like before so here in our sample:

Is the equivalent of:

And we wait for the the result (asyncAction.awaitCompletion() is a wrapper for a deferred.await()) thanks to deferred job and coroutines (might develop this part in another article if you are interested in), but notice that your work can have three kind of results:

• Success obviously means that job has been completed successfully, in case of chained work, those whose depend on this can be executed.
• Retry mean this one failed but should be retried when possible (because of a transient network failure or an 5xx HTTP error from the backend, for instance).
• Failure means this is a permanent failure, so every work depending on this one will be a failure also.

As we dispatched the same action than before the migration to WorkManager we didn’t have to change what happened next on success though Redux logic ;)

To give you an overview of what happened here is a schema, a little bit more complex than before maybe:

Conclusion

Combining brand new WorkManager and Redux we ended up in this which gives us an improvement on how we are behaving without network connectivity. From what we have seen when you enqueue your work request, most of the time, it’s done immediately but when you get in a state where you match your constraints it’s more random this is the bad part of this approach on the problem. Knowing this point it matches our needs so far and works pretty well on that kind of feature. In this article we only take the annotation process as an example but it worked the very same way for Highlights (quoting text with highlight color) and Reactions.

If you have any questions, or feedback on this let us know in the comments. Thanks for reading have a nice day.

Articles about Redux:

https://hackernoon.com/lessons-learned-implementing-redux-on-android-cba1bed40c41

https://medium.com/@trikita/writing-a-todo-app-with-redux-on-android-5de31cfbdb4f

https://jayrambhia.com/blog/android-redux-intro

https://netflixtechblog.com/making-our-android-studio-apps-reactive-with-ui-components-redux-5e37aac3b244

Articles about WorkManager:

https://androidwave.com/android-workmanager-tutorial/

https://medium.com/androiddevelopers/introducing-workmanager-2083bcfc4712

https://developer.android.com/topic/libraries/architecture/workmanager

Getting offline with WorkManager in a Redux world was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Text comprehension tests

A text comprehension test is a set of questions regarding a text, that allow to evaluate reader’s text comprehension. For example, for this paragraph of The Little Prince, which does not lack irony:

a text comprehension test could be:

1. Which job did the author originally choose?
2. What did the grown-ups advise him to do?
3. Which job did he finally choose?
4. Was what he had studied useful for this job?

However, generating such questions and correcting them is very challenging and still an open research problem ([1], [2], [3], [4] or [5]). For a start, we decided to only generate a specific kind of tests, called cloze tests.

Cloze tests. A cloze test is a text where some words are hidden. Usually, 4 answer propositions (1 true, 3 wrong) are given for every hidden word. Here is an example:

Generating and correcting cloze tests is easier than generating and correcting other types of tests where questions may be open, while still giving a good evaluation of text comprehension. However, there still are some difficulties:

1. Which words to hide? Ideally, we would like to hide a set of words that is small and contains the most important words for text comprehension.
2. Which distractors (i.e. relevant incorrect words) to propose? We can neither propose words that are obviously wrong, nor words that could be correct in the context (e.g. synonyms).

Now that we have explained the objective and the main difficulties, let us dive into the technical solution that we have developed.

Step 1: Which words to hide?

The first step in making a cloze test consists in selecting which words to hide. Here is the process we follow:

Step 1.a. We have decided to only hide common nouns because, along with verbs, they contain most of the meaning of a text.

Step 1.b. We give an importance score i to every common noun, based on the assumption that:

The more difficult it is to guess a hidden word, the more important it is.

or to put it differently:

The easier it is to guess a word after hiding it, the less important it is.

In practice, we use BERT ([6]), a deep neural network developed by Google, to predict the nouns after hiding them (more precisely, we use this Pytorch implementation). BERT was in part trained to infill cloze tests. This makes it an algorithm of choice for the task we are describing here. Let us see how we can use it to give an importance score to every common noun in this sentence:

s = “I love playing tennis with my cat.”

The first common noun is tennis. If we hide it (or mask it), the sentence becomes:

s_tennis = “I love playing [MASK] with my cat.”

Then, when we feed s_tennis to BERT which tries to predict the masked word. For this, it outputs a prediction score p_w for every word w in its vocabulary:

The higher the score of a word, the more BERT thinks it is the correct one. From these prediction scores, BERT misprediction of tennis can be evaluated with the following formula: (max p_w) — p_tennis. Hence, following our previous assumption, it gives the definition of the importance of tennis in the sentence:

i_tennis = (max p_w) - p_tennis

Numerically, i_tennis = 3.205.

The same process can be done for cat, the second common noun of the sentence, i.e. :

1. Hiding (or masking) cat in s giving s_cat.
2. Feeding s_cat to BERT and retrieving its prediction scores for every word in the vocabulary.
3. Computing i_cat. Numerically, i_cat = 8.327.

In this example, a higher importance score had been given to cat than to tennis because it is a much more unexpected word.

Step 1.c. We remove common nouns that are not important enough, i.e. common nouns with an importance score lower than a fixed threshold i_min. Indeed being able to guess them is not a good indicator of text understanding. We experimented with different values of i_min and ended up taking i_min = 2.5 because it gave us the best filtration.

Step 1.d. We only keep the X best common nouns, and finally hide them. Because we do not want too many words of the text to be hidden, we limit the ratio by taking X = number of words x r_max. Again, we experimented with different values of r_max = 0.05 because it gave us the best results.

Step 2: Which distractors to propose?

The second step consists in proposing distractors for every hidden word. We want them to be neither trivially wrong, nor correct in the context (e.g. synonyms). Let us use again the sentence s_tennis = "I love playing [MASK] with my cat." to present how we propose distractors for the hidden word tennis:

Step 2.a. We only keep predictions that are not the correct word or its singular / plural. If the correct word is among the distractors, then twice the same word will appear in the propositions, making it obvious it is the answer. If the singular or plural of the correct word is among the distractors, the answer also becomes obvious.

Step 2.b. We only keep predictions with the same casing as the correct word. The casing of a word is either lowercase when all the letters of the word are lowercased, or title when the first letter is uppercased and the others lowercased, or other. We do this not to get distractors that are trivially wrong (e.g. if the hidden word is the first one of a sentence and the distractors are not in the title casing).

In our example, tennis is lowercased, so we will only keep distractors in this casing.

Step 2.c. We order the remaining predictions by their prediction scores in decreasing order.

In our example, the best remaining predictions when hiding tennis are:

Step 2.d. We keep the 3 predictions with the best scores that are constantly spaced, i.e. we take the prediction with the best score, p_1, then the prediction with the best prediction score, p_2, such that p_2 < p_1 — p_gap, then the prediction with the best score, p_3, such that p_3 < p_2 — p_gap. Spacing predictions prevents having distractors that are synonyms because synonyms usually get the same prediction score. It also helps to have distractors with a higher diversity of meaning. We experimented with different values of p_gap and ended up taking p_gap = 2 because it gave the best results.

In our example, the 3 predictions with the best scores spaced by 2 are:

Conclusion

To sum up, the process we follow to generate cloze tests is two-fold:

1. Selecting which words to hide. The main idea is to assume that the more difficult it is to guess a word after hiding it, the more important it is.
2. Proposing distractors for every word to hide. The main idea is to take the predictions with the best scores that are constantly spaced.

This process leads to cloze tests that are satisfactory, although not optimal. Here are possible improvements:

1. A new deep neural network, XLNet ([7]), developed by researchers from CMU and Google Brain, has been released one month ago. It outperforms BERT on numerous tasks including cloze tests infilling. Replacing BERT by XLNet could lead to more relevant predictions. However, XLNet is only available for english language in contrary to BERT that has a multilingual version.
2. A new paper ([8]), from Facebook researchers, presents a way to turn a cloze test into a series of questions. Adding it to our process could lead to an improved version of cloze tests where hidden words are replaced by questions.

References

[1] Smart Question Generation

[2] Automatic Gap-fill Question Generation from Text Books

[3] Improving Neural Question Generation using Answer Separation

[4] A Framework for Automatic Question Generation from Text using Deep Reinforcement Learning

[5] Recent Advances in Neural Question Generation

[6] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

[7] XLNet: Generalized Autoregressive Pretraining for Language Understanding

[8] Unsupervised Question Answering by Cloze Translation

How to evaluate readers text comprehension? was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

One of the most common questions we get asked is “what is your stack?” We are of course always glad to discuss our technology choices in person, but we thought an article was the occasion to have a more comprehensive description of what we use. Each section here will also be a way for us to give an abstract of the more in-depth descriptions that we will continue to publish.

To give credit where it’s due: the format of this article is freely inspired by the great article from the Instagram team named “What powers Instagram.”

OS / Hosting

We are hosting almost everything on Google Cloud Platform, and we orchestrate our services with Kubernetes. We migrated to this solution from dedicated servers on OVH as soon as Kubernetes became a viable option, and the k8s “magic” has allowed us to grow beyond anything we could have managed on raw servers without having to dedicate too many resources to sys-ops roles.

The cost of hosting everything on GCP is of course way higher than the equivalent fire-power on a more classic hosting provider such as OVH, but the time saved building and maintaining servers has been well worth it until now. We also had the chance to be part of the “Google for startups” program, with the Surge package, which helped a lot with the bill while growing.

We might revisit this decision in the future, especially now that OVH is starting to launch k8s packages with very aggressive price points.

Backend

Architecture

Our backend is split into dozens of microservices (around 57 at the time of writing). Without going into too many details here on the pros and cons of a “microservices” architecture, this allows us to be very agile when deploying updates, and it lets us distribute responsibilities and ownership of part of the stack to each member of our engineering team.

Most of those services communicate with very simple HTTP calls, authenticated when it makes sense. Some “many to many” messages are sent using RabbitMQ exchanges, to avoid creating dependency loops between services; and some services publish jobs to RPC queues in RabbitMQ for workflows where HTTP calls would not be appropriate.

And to have some kind of consistency between all these services, they are all named after A Song of Ice and Fire castles.

Frameworks

Most of our services are run by Python 3.7. Some are still using 3.6, and some are taking advantage of the more advanced garbage collection done by Pypy (Python does not release memory, even once the objects are garbage collected. This isn’t an issue for most use cases, and even allows some performance gains, but for services that might need huge amounts of RAM for short periods of time, not being able to release it once the job is done is a serious issue).

We also decided early on to use asynchronous programming, with asyncio in Python. This choice was first motivated by our ebooks ingestion pipeline that received files on an FTP server, and we would then need to monitor those files on the disk to trigger updates whenever there was a change. Doing this with inotify in Python can be easily achieved with threads, but that solution does not scale to the millions of files we had to watch (millions of threads is a no-go…), so we instead switched our HTTP server to Tornado (by Facebook), and started using coroutines.

Since then, our ingestion pipeline was re-written entirely to expose a custom FTP server that simply directly sends the files for processing instead of writing them on a disk and hoping another process picks up the change. The Tornado framework was also dropped for the more mainline aiohttp library and the asyncio implementation in the Python stdlib.

We found that this asynchronous approach is very compatible with both our database (Mongo is made for tons of small concurrent requests) and our microservices architecture (which relies on aggregating data from many different small HTTP requests) and we applied this basic principle and to (almost) all our backend services.

Data storage

Most of our data lives in MongoDB; lots of pros and cons here too, but all-in-all this has worked well for us until now, mainly by allowing us to iterate very quickly and very easily without having to write too many migrations.

We are hosting those mongo servers inside our k8s cluster, backing them with gcloud SSD drives which give us sufficient performance for now.

Some of our data points don’t fit well in the Mongo space, or the performance was too poor for our use case: for instance, the detailed reading progress metrics we are aggregating to provide our user with insightful data dashboards are more adapted to a time-based database (we use InfluxDB for this). Some of our data also live in PostgreSQL, and we use Elasticsearch to power our search engine and store the related data.

We are also using two distinct Redis clusters: one is used to cache some data in an LRU cache (“least recently used”, a cache that only keeps the last items it can store, and drops old items when it reaches its limit) ; and we use the other system as a “cluster-wide” locking mechanism, to share locks between processes that might run on different servers.

And of course, since we are in the GCP ecosystem, our files are hosted on Google Cloud Storage.

Task Queue

Some of our backend tasks are CPU-bound (ie their execution time is constrained by how much CPU we can allocate to the task). This includes most of the NLU tasks (Natural Language Understanding) done by our Research team, and some expensive statistic computations for the data reports we expose to our users.

Locking a whole CPU does not fit well in our asynchronous approach to the backend, and they need to be dispatched to special workers configured for this. We use RabbitMQ queues to connect the dots here.

We also use RabbitMQ exchanges when a service needs to broadcast updates that might concern other services, such as metadata changes for a book. This lets us publish from service A, and subscribe from service B without directly having a dependency loop between A and B which would make updating both tricky.

And finally, our logs ingestion stack also uses RabbitMQ to connect the log producers to our ingestion project. More on that in the logging section.

Logging

All those running parts produce around 500 lines of logs per second, that we need to store somewhere accessible and usable to build our internal reporting dashboards and to investigate issues.

While benchmarking different solutions, we found that building our ELK (Elasticsearch Logstash Kibana) pipeline ourselves was both manageable and cheaper than relying on an external dedicated solution. So we built an in-house Elasticsearch cluster dedicated to our logs with six instances of Elasticsearch 7 — this gives us around 1800 logs/second when ingesting full speed — and we collect the logs with fluentd. The collectors push the logs to a RabbitMQ queue, where they are picked up by some dedicated workers, which will push them to Elasticsearch after some transformations (geolocalisation for the IP addresses of our access logs for instance).

We are not using Logstash for this: we found that to process our 500 logs per second, the cluster of Logstash services used around 20 cores of our CPUs; and the in-house service doing the same “pick up logs from RabbitMQ, process, push to Elasticsearch” job uses between 0.5 and 1 cores using Pypy. We initially used Go for this service, to use as little CPU as possible, but we found that the performance gain (around a factor 2) was not worth maintaining a project in another language (and nobody in the team had any experience maintaining a Go project beyond a PoC).

Mobile

Redux

Mobile apps for iOS and Android are structured around a Redux architecture and its three fundamental principles:

1. The state of the whole application is described by an object tree within a store providing a single source of truth.
2. The application state is read-only and the only way to change it is to emit an action, which will result in a different state being produced, thus ensuring unidirectional data flow.
3. New states are produced by pure functions (a function that does not have any side effect), called reducers, which take the previous state and an action as its input and return the next state as its output.

Components in the apps that depend on the state connect to the store via an in-house implementation of the connect() functionality from React Redux. A connected component provides two lambdas to the connector: a pure function that derives the component properties from a state object tree and another function which renders the components from a given properties object. The connector then takes care of listening to store state changes, deriving the component’s properties, and requesting the component to render when its properties have changed. This provides a simple and efficient way to propagate app state changes to the UI layer.

Asynchronous operations such as HTTP requests are implemented with Redux: an action is dispatched to initiate the operation (more details here), and another action is later dispatched to handle its result. The initial action is handled by a middleware, a function that is called before reducers and which is allowed to have side effects (but not to modify the state).

iOS

Glose for iOS is in Swift. The Redux part is implemented using ReSwift. Every object stored in the state is a struct conforming to Equatable. As Swift does not provide automatic copy function (like in Kotlin), reducers actually return a mutated copy of the state and connect functions use our Equatable implementation to compare and dispatch changes.

The whole flux parts (action initialization, middlewares, and reducers) are executed in a serial OperationQueue.

This was a necessary implementation because it can be quite costly to compare a complex object graph, even if our connectors are small and very targeted. So the networking, JSON encode/decode, mutation and state changes all happen asynchronously and are then dispatched to the connected component on the UI thread. One of the drawbacks is that your UI has to handle asynchronous data almost everywhere. While this is easy in a “static” screen, it’s more complex for layout pass in UITableView and UICollectionView. As cells layout (height) is not computed dynamically, you have to notify your UITableView that you need to update the height of specific cells once your dynamic content is loaded.

With Redux we can make highly re-usable and self-contained components. Glose for iOS use extensively UICollectionView & UITableView and provide various “Provider” so you can easily connect whole data source logic and delegate actions in any screen of the app.

Our reader uses a custom WKWebView, into which we inject a custom content controller, so we can scope and intercept javascript messages. We use this bridge system to communicate between Swift and Javascript code. The custom WKWebView render the book as a pure HTML content, while we inject annotations, reaction, etc. as native UIKit views.

We also make extensive use of TextKit to display custom emojis and now even inject custom views in order for our upcoming animated reactions. It’s a powerful tool, and you can quite easily display anything in your UITextView with custom NSTextAttachment and LayoutManager.

Android

Glose for Android is written 100% in Kotlin. The Redux architecture is built upon ReKotlin on top of which we’ve built an in-house connect() functionality (see above). We rely on the ease and power of Kotlin coroutines for asynchronous operations where appropriate. We use various Android Jetpack components that make our job easier, including Android KTX for idiomatic Kotlin code, lifecycle-aware components that reduce coupling, and paged lists for on-demand loading of data sets.

We also rely heavily on the awesome Epoxy library by Airbnb, which makes it easy and painless to build complex screens declaratively on top of recycler views.

Also involved: Chromium for the rendering of book contents; Retrofit and OkHttp for HTTP requests; Koin for lightweight dependency injection; Picasso for image loading; Firebase for push notifications; Stripe for payment processing; Mockito and Espresso for unit tests and UI tests; Gradle Play Publisher for automated releases from our CI.

Web

Coming soon

Monitoring

Our monitoring stack is pretty standard:

We monitor exception in production with Sentry for all our Python and Javascript code, and through Crashlytics (formerly fabric.io) for our mobile applications.

Server status metrics and basic application metrics are collected using Prometheus.

And we use Grafana dashboards that aggregate both the Prometheus data and data extracted from our logs Elasticsearch cluster to display relevant graphs and trigger alerts when necessary.

You?

We are a (still) small but growing fast team and we are always looking for new talents.

If this description of our platform interests you, or if you are dying to tell us all that we did wrong here, we would love to hear from you!

What powers Glose: Hundreds of pods, dozens of technologies was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Reader engagement is a recurrent problem among all types of readers: adults/children and teachers/students. They all face the same issue: finding books close to their current readability ability, either for casual reading (easy level) or to improve and learn (hard level) without being flooded by too much difficulty which usually results in a harsh experience for most of us.

At Glose, our goal is to enable readers to access books that are gradually more difficult and recommend books that fit their current reading ability.

In this article, we will show how we developed a machine learning system that objectively evaluates text readability.

Text Complexity: Facets and Usage

Text complexity measurement estimates how difficult it is to understand a document. It can be defined by two concepts: legibility and readability.

Legibility ranges from character perception to all the nuances of formatting such as bold, font style, font size, italic, word spacing …

Readability, on the other hand, focuses on textual content such as lexical, semantical, syntactical and discourse cohesion analysis. It is usually computed in a very approximate manner, using average sentence length (in characters or words) and average word length (in characters or syllables) in sentences.

A few other text complexity features do not depend on the text itself, but rather on the reader’s intent (homework, learning, leisure, …) and cognitive focus (which can be influenced by ambient noise, stress level, or any other type of distraction).

Why is it crucial to be able to measure text readability ?

In the context of conveying important information to most readers (drugs leaflets, news, administrative and legal documents), an evaluation of readability helps text writers to adjust their content to their target audience’s level.

Another use case is the field of automatic text simplification, where a robust readability metric can replace standard objective functions (such as a mixture of BLEU and Flesch-Kincaid) used to train text simplification systems.

In this article, we will focus solely on estimating text readability using annotated datasets and machine learning algorithms. We implemented them using the scikit-learn framework.

Data

The starting point of any machine learning task is to collect data. In our case, we extract it from two sources:

1. Our database at Glose contains more than 1 million books which include around 800,000 english books.
2. A dataset of 330,000 book identifiers, graded on the Lexile text complexity scale ∈ [-200, 2200].

This dataset is biased in two ways:

1. The distribution of book genres in our merged dataset is unbalanced (figure above).
2. It assumes that the Lexile score is close to the true readability perception of the average human, which might not be, due to their usage of mainly two features: sentence length and words frequency.

Book identifiers (namely ISBN), are unique to a book’s edition. Each book can have multiple ISBNs due to the large number of editors distributing the same content. In short, each identifier in our dataset maps to multiple identifiers of similar content.

In order to have a unique mapping between ISBN, book content, and Lexile score, we select an intersection subset (where we have both a book’s content in our database and a Lexile annotation) of 17,000 english books.

Book representation

In the first step of our natural language processing pipeline, we clean and tokenize the text into sentences and words. Then we have to represent text as an array of numbers (a.k.a. feature vector): here we choose to represent text by hand-crafted variables in order to embed higher level meaning than a sequence of raw characters.

Each book is represented by a vector of 50 float numbers, each of them being a text feature such as:

• Mean number of syllables per word,
• Mean number of words per sentence,
• Mean number of words considered “difficult” in a sentence (a word is “difficult” if it is not part of an “easy” words reference list),
• Part-of-Speech (POS) tags count per book,
• Readability formulas such as Flesch-Kincaid and
• Number of polysyllables (more than 3 syllables).

These features are all on different scales (c.f. figure above), however we would like to have a similar scale from -1 to 1 because some of the algorithms we use during modelling (Support Vector Regression with a Linear kernel and Linear regression) assume that the data given as input follows a Gaussian distribution. This process, namely standardisation, is about removing the mean and dividing by the standard deviation of a dataset.

Feature selection

Now that we built a set of features representing a text, we would like to truncate that vector to the most salient features ; the ones that discriminate the most our annotations. Using features that do not carry information related to the target variable (the readability score) is a computation time burden to the model, because the inference is done with more features than necessary.

To perform this feature selection step, we use the LASSO method (scikit-learn implementation) with cross-validation (CV is the process of training and testing models with different data splits to avoid a bias from a specific dataset order) because the difference between execution time with and without 10-fold CV is negligible. Moreover, it guarantees to have a model that is less subject to variance when confronted to real data.

The LASSO method is performed by creating multiple subsets of our feature set. For each feature set a regression function is fitted using our training data. Then a correlation is computed (using a metric such as Person, Kendall-Tau or Chi-Square) between each set’s regression function and the readability score. Feature sets are ranked by correlation performance and the best one is selected.

Choosing the right model

Our output variable is numerical and continuous which narrows the spectrum of machine learning models applicable to our dataset (regression task). To select an appropriate model, there is several indicators that may guide one’s choice, such as the number of features or the number of samples available.

In the case of constrained bayesian algorithms such as Naive Bayes variants (simple or tree augmented), performances are likely to decrease with large number of features. This is due to their inability to build large variable dependencies between an output variable and an explanatory variable. Naive Bayes is built under the assumption that variables are independent, which is less likely the case with longer feature vectors. Tree Augmented Naive Bayes (TAN) allows only one explanatory variable as a dependency of another to predict an output variable. This lack of feature intrication makes these algorithms bad candidates for our feature vector length (50), we will not use them in this article.

However, Decision Tree (DT) based algorithms cope very well with high dimensional data (more features) but need lots of data samples (varies as a function of algorithm hyperparameters). DTs build rules (for example: average number of words per sentence > 5) and these rules are split when a given amount of data samples fit them. For example: 10 samples fit the previous rule, we consider that there is too much samples in this rule, so we build two other rules > 5 AND < 10 and > 10 where we fit respectively 4 and 6 samples instead of 10 in one rule. In decision tree algorithms, the number of data samples is a function of model granularity, by handling overfitting correctly, the more data and features there is, the better a DT based model is.

Another approach to model selection that we choose to use is Grid Search, this technique is a training and testing brute force over a set of models and a set of hyper parameters for each model.

Pros: Easy to setup, less preliminary analysis of dataset, specific model knowledge isn’t much needed, empirical evidence (you won’t know unless you try).

Cons: Hyper parameter sets definition needs specific model knowledge and literature review to reduce computation time, time-consuming search (e.g. next figure), no global optimum guarantee.

In our Grid Search, three algorithms compete: a Random Forest Regressor (4 hyper parameters), a Linear Regression and a Support Vector Regressor (2 hyper parameters), the best model is generated through Random Forest regression.

Interpreting readability scores

We now have a production grade model that takes a book’s feature vector (obtained through pre-processing) as input and gives a readability score as output. In order to display a comprehensible metric to users (especially pre-college students), we would like to have a more meaningful representation of this score by converting it to grade level bins, we use the following formula to define those bins.

On the following figure we can see the most interesting sections of the readability scale for the students that will read their books on Glose. A teacher can follow a student’s progression on this scale by monitoring the mean grade level of the books he reads.

Performance

Overall our best model achieves around 0.88 for the metric R² which explains 88% of our test set variance. R², also known as coefficient of determination, is the metric we use to test our regression algorithm. The resulting value we get from it ranges from 0 to 1 and Random Forest is optimised to converge to 1. This value is the explained variance accounted by our model: the higher it is, the less test data samples we find outside of our model’s prediction error range.

On the figure above we see that most of our predictions (60%) fall in the right grade level, whereas nearly 35% in only one grade level above or below ground truth. Adjacent precision is equal to 95%, this metric is more relaxed than precision as it allows up to one grade level error.

However, when we inspect the residuals per grade level and the distribution of grade levels over our test set, we realise that most of our errors (yellow, orange and red bars) happen on grade levels with fewer samples (levels 7 to 12 included).

Statistically, our results seem satisfactory. However we have room for improvement with this approach and we are going to evaluate the robustness of our model with human experts giving their feedback in the loop.

Conclusion and outlook

As a TL;DR and a takeaway of this post, you should have learned:

• What is text complexity, and why is it meaningful.
• How a machine learning pipeline is designed to create a production model.
• A few specifics about parts of this pipeline such as features and models selection.
• That this post’s readability score is 878 which is lower compared to TIGS: An Inference Algorithm for Text Infilling with Gradient Search that reaches the score 992 on our scale, whereas In Search of Lost Time by Marcel Proust stands at 1441.

As a premise of our next article, we are currently working on another approach to evaluate text readability using neural language models as comprehension systems to infill Cloze tests (text chunks with blank words). The training phase of this other approach is unsupervised and has the advantage of being language agnostic.

How to Evaluate Text Readability with NLP was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

When processing large amounts of text (more than 1 million books) like we do at Glose, it is crucial to optimize every step of the pipeline.

In this post, we will be focusing on a very common Natural Language Processing (NLP) task that involves counting the number of occurrences of every word in a text: building a bag-of-words (BoW).

🎒 BoW applications and a simple example

NLP pipelines usually start by converting a text to an array (or several arrays) of numbers (vectors). This vectorial representation is crucial because it is much easier to manipulate vectors than raw strings in machine learning algorithms (such as document classification, sentiment analysis, Part-of-Speech tagging, Named Entities Recognition…), or in recommendation systems that compute similarities between items based on their vectors.

Building the BoW representation is often the first step in obtaining the vectorial representation. For instance, in topic modeling, where every number in the vector corresponds to the contribution of a topic to the text, we start by the BoW, then compute the words’ frequencies (Tf-Idf), and finally compute the topics (using methods like LSA or LDA).

Let us start with a simple example, using this post’s introduction:

When processing large amounts of text, we optimize every step. In this post, we will be focusing on counting the number of occurrences of every word in a text.

Before building the BoW representation, we convert the text to lowercase, remove all the stopwords (meaningless words) and punctuation, and replace all words by their lemmas, which gives:

process large amount text optimize step post focus count number occurrence word text

Then we count the occurrences of each word, which gives:

{
    'process': 1,
    'large': 1,
    'amount': 1,
    'text': 2,
    'optimize': 1,
    'step': 1,
    'post': 1,
    'focus': 1,
    'count': 1,
    'number': 1,
    'occurrence': 1,
    'word': 1,
}

If we associate an integer identifier to every word (‘process’ ↔ 0, ‘large’ ↔ 1 …) — a.k.a build a dictionary — we can write the BoW representation in a more compact manner [(id, count), …]:

[(0,1), (1,1), (2,1), (3,2), … , (11, 1)]

We can then use this BoW representation in further processing steps, as mentioned above.

🐍 A naive implementation

Let us first write a simple python program that transforms a preprocessed text into a compact BoW representation:

We use python’s built-in collections.defaultdict to count the number of occurrences of words, and build the dictionary by iterating on all the words, and adding the missing ones with their integer identifier.

We can now try our BoW implementation on the previous preprocessed example:

sample_text = 'process large amount text optimize step post focus count number occurrence word text'

dictionary = {}

print('BOW representation:', text2bow(sample_text.split(), dictionary))

print('Dictionary:', dictionary)

We get:

BOW representation: [(0, 1), (1, 1), (2, 1), (3, 2), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1), (10, 1), (11, 1)]

Dictionary: {'process': 0, 'large': 1, 'amount': 2, 'text': 3, 'optimize': 4, 'step': 5, 'post': 6, 'focus': 7, 'count': 8, 'number': 9, 'occurrence': 10, 'word': 11}

which is exactly what we got in the previous section, by manually counting the occurrences.

Finally let us time the text2bow function on a preprocessed text of 26893 words, corresponding to a single book:

4.36 ms ± 29.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

While this seems like a reasonable processing time for a single book, it is important to remember that it is only the first step of our NLP pipeline, and that it will be applied on ~1M books, which takes us to ~ 1 hour 12 minutes of processing time just to create BoW representations of all our books.

We will see in the following sections how to use Cython and spaCy’s Cython API to speed up this code.

⚙️ About cython and spaCy

Working with pure Python code is great for fast iteration and experimentation. However there are some use cases where one can benefit from a statically typed and compiled language. Mainly:

1. When developing a production module, that needs to work at full speed.
2. When a bottleneck is identified in Python code (profiling is key!). This is often related to portions of Python code with plain or nested for-loops.
3. When native parallelism is needed (that means releasing Python’s Global Interpreter Lock).

This brings us to Cython, which is defined as follows in its homepage:

Cython is an optimising static compiler for both the Python programming language and the extended Cython programming language (based on Pyrex). It makes writing C extensions for Python as easy as Python itself. […]

The Cython language is a superset of the Python language that additionally supports calling C functions and declaring C types on variables and class attributes. This allows the compiler to generate very efficient C code from Cython code. The C code is generated once and then compiles with all major C/C++ compilers […].

Cython thus solves (1) and (3) by generating pure C/C++ code, and (2) by being a superset of Python, which means that Python code is valid Cython, and allows to only optimize the bottlenecks.

Leveraging all of Cython’s benefits for NLP can however be tricky. It is mentioned in the documentation that one should not use C strings because they require manual memory management and are more cumbersome than Python strings.

This is where spaCy comes of help. It is a fast NLP library written in Python/Cython, which uses a clever method to manage strings, that manipulates 64-bit hashes internally instead of Python strings, making the code much faster while keeping the flexibility of Python strings from the user’s perspective.

spaCy’s StringStore

Most of spaCy’s strings management is taken care of in the file strings.pyx, and its StringStore. As mentioned in the documentation’s header, the StringStore’s purpose is to:

Look up strings by 64-bit hashes. As of v2.0, spaCy uses hash values instead of integer IDs. This ensures that strings always map to the same ID, even from different StringStores.

This means that instead of manipulating strings internally, spaCy computes and manipulates their 64-bit hashes only. They are converted back to their Python string equivalent only when necessary (for instance when a user wants to print the output of the spaCy pipeline).

Hence, the StringStore is the data structure where the mapping between the 64-bit hashes and their Python unicode string equivalent is stored. It is in fact a Cython extension type, which means that from a Python module’s perspective it behaves like a Python class, but internally it can have cdef statements (either static attributes, or C methods), and is built as a C struct.

In order to leverage the speed of spaCy’s low level C structures for our fast BoW program, we need to understand what happens when a Python unicode string is added to the StringStore. Most of this behavior is located in the StringStore’s methods add, intern_unicode, and _intern_utf8.

Here are the main steps of this process:

1. The unicode string is encoded in utf-8 (code)
2. The encoded string is hashed to a 64-bit integer by using MurmurHash2 (code)
3. The encoded string is transformed into a C char array (or pointer). This is done in a way that optimizes memory usage (by either using a char array of 8 elements if the string is small enough, or using a pointer array otherwise) (code for the _allocate function, that converts the encoded string to a C array/pointer, line where the _allocate function is called)
4. The (64-bit hash, C char array/pointer) couple is stored as key-value in a Cython Hash Table for Pre-Hashed Keys, which is an attribute of the StringStore (code)
5. The 64-bit hash is stored in a list attribute of the StringStore (code)

💡 The main lessons from this analysis are that in order to be able to use a fast C level hash table the unicode strings need to be converted to a C char array (or pointer)¹, and that all fast computations must be performed on the 64-bit hashes, not on the strings.

🏎️ Cythonized version of BoW

Now that we know how spaCy manages strings internally, we can start implementing our own Cython version of the BoW.

First, we will import the necessary C types and classes (or extension types):

Among all these imports let us comment on a few:

• The cymem package is used to tie memory to a Python object, so that the memory is freed when the object is garbage collected.
• The preshed package contains both the Hash table where we store the (64-bit hash, C char array/pointer) couples, and a fast counter extension type (PreshCounter) that we will use to perform the BoW counting.
• We use cimport instead of import to access the extensions types’ C methods and attributes.

Let us now write the counting function:

The _insert_in_hashmap function (used in fast_count) is defined below, as well as other utility functions, directly inspired from spaCy’s strings.pyx:

Finally, we compile the Cython file with the following simple setup.py, by executing python setup.py — build-ext -if (more details on Cython compilation here):

This time, by executing text2bow on the example text we get:

BOW representation: [(2751841902330220293, 1), (6601561424492272668, 1), (11916616154811659322, 1), (4645701992108298564, 1), (15099781594404091470, 2), (9470301821735104089, 1), (9556349622722280057, 1), (6437996555066804658, 1), (1020421249059553464, 1), (12460925685579008443, 1), (18223104521466393082, 1), (8530854408006191868, 1)]

Representation of the hash table using unicode strings: {
    6601561424492272668 :  post,
    11916616154811659322 :  word,
    1020421249059553464 :  process,
    4645701992108298564 :  step,
    6437996555066804658 :  amount,
    12460925685579008443 :  focus,
    18223104521466393082 :  number,
    15099781594404091470 :  text,
    9470301821735104089 :  optimize,
    9556349622722280057 :  occurrence,
    8530854408006191868 :  count,
    2751841902330220293 :  large,
}

We can see the 64-bit hashes in the BoW, and a representation of the hash table using unicode strings.

Processing the same 26893-words book is now around 4 times faster:

1.1 ms ± 6.68 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Computing the BoW of 1M books now only takes ~18 minutes 🚀 compared to the previous 1 hour 12 minutes!

Conclusion

In this post we have presented both a naive implementation of Bag-of-Words, as well as an optimized Cython one, directly inspired by spaCy’s internal way of managing strings.

Exploring Cython and spaCy is a good way to write more optimized NLP pipelines, and we definitely advise you to read more about these two. In particular some great introductory Cython material can be found in Kurt Smith’s video, and his book, while more advanced topics can be studied in Cython’s documentation. For spaCy, same thing goes with the great documentation, and don’t hesitate to delve into the code!

¹ Which is exactly what Cython’s documentation was warning us about! The tricky manual memory management of the C strings is in fact taken care of in the _allocate function.

🏎️ Fast bag-of-words using spaCy and cython was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Note: This post is a continuation, but also a better approach of my previous post about reusing UITableView

I had some fun writing some of our user interface code for the next version of our iOS application at Glose books. Writing clean UI code is essential, writing independent, clearly defined and confined UIViewController and interface flow IS fundamental, if you want to pack your applications with a lot of features, while maintaining a clean, understandable, and reusable codebase.

By the past, I’ve made the mistake to not refactor my code early on, always postponing some stuff to “When I’ll need to use it a third time”, and in some old legacy codebase, it was clearly a mistake. I never want to write spaghetti code ever again. You know, that little specificity you want to reuse and re-wire without rewriting the whole interface logic behind… anyway those sorts of things are old stories now.

In the following example, I’ll try to teach you how to have a clean, less than 100 lines UIViewController, which display a UICollectionView with a complicated user interface and business logic. A UICollectionView configuration that you can literally connect in any UIViewController in a few lines of code.

Let’s say you want to to display some books in a UICollectionView, so you want to display them as a grid, but also as list, so you’ll need a least two different UICollectionViewCell . But you also want to provide a custom title for your UIViewController navigation. Your cells have some delegate for forwarding action to your business logic, and you also want to provide a few customisation options. On top of that you also want to support 3D touch previewing. Oh and I forgot, it’ll also support pagination. As you can understand, you’ll not want to write the code to handle all that more than once.

And I’ll need to use this UICollectionView in a view controller which contain only that, a simple grid and list of books, but also in a more complexe one, where I’ll have a more complicated user interface as the header of the said collection view.

And now the code

The Inteface Provider

Let’s write the code and wire all that together. First, here is the class which will register as the delegate and datasource of your UICollectionView, it’ll provide its own delegate, so you can customize it wherever you’ll need it. It’ll be much shorter to implement in your final UIViewController than the full UICollectionView delegate and datasource. I call them the InterfaceProvider:

As you can see in the code above, the delegate provide method where you can return a customized header, you can also listen to the UIScrollView did scroll events. Useful in some case.

This class will manage the integrity of your UICollectionView from the moment you instanciate it from your controller. You pass your view controller and collection view as weak, because you view controller is responsible to keep, well… itself and his UICollectionView alive.

In my case the datasource is just some id of objects I load from a shared store. Very easy and lightweight to pass around.

The Data Provider

Now, let’s see how you actually configure and build your datasource, alongisde some other data for your UIViewController, it’ll be the provider for the business logic, I call them DataProvider:

It’s defined by a very simple protocol, that you later use in concrete classes:

It also provide a base implementation, because in my case, it need a user from where to feed its datasource, it’s very handy, you can provide your own initialisers, properties, and default implementation if needed.

In your implementation, you’ll have to return the title, which is used as the navigation bar title, your datasource, and also implement the method to actually load your data and follow page navigation.

Now, here is one of the concrete implementation of this protocol, for a collection view which will basically show all the books of the user:

The View Controller

And now, the part you were all waiting for, the UIViewController wiring all that together, which is only 75 lines long:

Note: I’m using ReSwift, a Swift implementation of Redux, it fits very well with this interface pattern because you don’t have to wire any of your data update and changes. Everything is live.

Using it

And to instantiate this UIViewController, you simply inject your DataProvider as a dependency like this, if you don’t, your app will crash, because the data provider is declared as a var! in your view controller. And believe it or not, that is totally fine! You won’t forget a dependency, would you?

Here you go, now you can create various concrete implementation of your DataProvider, and inject it in your BooksCollectionViewController without writing all new controllers.

It’s also very decoupled, for example I’m using the InterfaceProvider but not the DataProvider in another controller, because I needed the interface, but the business logic was too complicated and is handled inside the controller itself.

You can also push it one step further and use a system of contained and container view controllers, that you can assemble like legos. More on that later!

Feel free to comment or ping me at @Dimillian if you have any question!

Happy coding!

Building reusable user interfaces in Swift was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Glose, we are building the reading platform of the 21st century, serving millions of eBooks to millions of readers across the world, giving them access to an incredible amount of knowledge right under their finger.

Our interfaces can be complex as we offer a lot of interactions for the readers, and that imply a certain amount of HTTP requests between client and server. That’s why we need to have an efficient flow to handle it.

React and Redux allow us to have a great and efficient control over our state and user interface. When it comes to asynchronous actions that interact with an API (or whatever, API is just the most common example), things can became redundant and laborious. We will see a way to have a great async flow without writing tons of identical code.

This article will only deal with thunks, which are actions that return functions. If you are not familiar with this concept, I invite you to have a look at redux-thunk middleware.

The naive way

The most common way to deal with asynchronous actions is with thunks. Thunks are actions that are not objects but functions that can dispatch other actions — and eventually do other things, but let’s keep it simple for now.

Taking a simple action we want to perform, getting some info about an organization on GitHub. We want to know two things:

• when the request starts
• when the request ends (and if it succeeded or failed)

With a thunk, it will look like the following:

First we are dispatching aFETCH_ORG_REQUEST action, so that the user interface can handle the loading state.

Then, the fetch action is performed. It’s the actual HTTP request. If you don’t know about fetch, check it out, it’s now native on modern browsers and there is a good polyfill for older ones.

When the fetch is over, we have two possibilities:

• either it has failed, then an error is raised an the FETCH_ORG_FAILED action is dispatched
• or everything went well (yay!) and the FETCH_ORG_SUCCESS action is dispatched with the result of the request

Pretty simple isn’t it? Only 26 lines of code (yes it could be less, but I like to be able to read my code later without having a headache). Now imagine a second action to fetch the repositories of an organization. Yup, same stuff, probably around 25–30 lines of code too.

Now think about a large scale project with dozens or hundreds of this kind of HTTP requests. Three actions: ‘request’, ‘success, ‘failed’. Same code, hundreds of times. Pretty boring, and massive. It will represent a lots of code, for absolutely no interest, it’s only mechanic.

We have to factorize that.

Factorize all the things!

Okay, what can be factorized here? The three kinds of actions for a single one. Let’s start with that.

We should make a function that returns our action creator (that itself return a function — a thunk). Our function, let’s say asyncAction will take the name of actions to dispatch and the URL, the suffix will be added automatically.

That’s not so bad. But, there are a lot of issues with that:

• We don’t handle arguments. This action would have been a lot better if we could give facebook as an argument to it, and simply call fetchOrganization('facebook').
• Event if we took all arguments with an ...args at the function returned on line 2, we would lose the information of the argument name, and this is a really important information. We don’t want to refer to our arguments with their position on args array.

Function is not so bad, but it didn’t work. Do you know what has been a cool addition to ES6? Classes.

Do it classy

Even if React and Redux world put a lot of accent on functional programming, I find OOP very elegant in this kind of situation.

What do we want here? A way to express easily all of our asynchronous actions that talks to our API without repeating ourselves.

We can make a class per action, its constructor handling the arguments (if any), a method to perform the call. And that’s it. The parent class takes care of all the stuff discussed before.

This kind of action will look like this:

Pretty neat isn’t it?

Now, here is the magic behind AsyncAction parent class.

The parent class only defines a toThunk method, that we will use in just a minute. The logic is the same as before, dispatch an action, make the request, dispatch according to the result.

Before we can make this work, we have to make a small transformation, because Redux doesn’t know what to do with a class. That’s why we defined a toThunk method, because Redux understands thunks with redux-thunk middleware.

We just have to intercept all actions that are AsyncAction instances, and continue working with the thunk of it. To do that, let’s create a small middleware:

Don’t forget to add this middleware before redux-thunk middleware when creating your redux store (or else transformation to thunk will occur after the action is treaded by redux-thunk middleware), and we’re done.

If you are not familiar with Redux middlewares, I invite you to read the documentation about middlewares.

Now, you can simply dispatch your action like dispatch(new GetOrganization('facebook')). It seems unusual to have a class instanciation with a dispatch, but it’s nothing and you know what is working under, so it’s not a big deal.

Going further

This way to handle asynchronous actions is one among a lot of another. There are great libraries with different approaches like Redux-Saga for example. It’s up to you to learn different ways to deal with your asynchronous flow and to choose the one that better fits your needs.

What I like about the approach I am presenting here is that it’s lightweight and very extensible. At Glose, we are using it, with some additions for error handling, before request hooks, etc.

The code is simple and you can easily add some logic to answer different issues about your asynchronous flow.

Efficient asynchronous flow with React and Redux was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Depuis mon plus jeune âge, j’aime les livres.

J’aime d’abord ce qu’il y a dedans. Je ne compte pas le nombre de journées passées à dévorer roman après roman, mes parents me fustigeant d’aller jouer dehors durant les vacances d’été, alors même que suivre les péripéties de tous ces personnages constituait pour moi la plus belle des aventures estivales. Je me revois aussi dans mon lit, me cachant sous la couverture avec une lampe de poche pour pouvoir lire en cachette jusqu’à une heure avancée de la nuit –voire du matin, tout en sachant pertinemment que le réveil serait difficile. Peu importe, c’était le prix de l’évasion, et ça en valait largement la peine. C’est toujours le cas d’ailleurs.

J’aime aussi l’objet. Le livre que l’on aime regarder, explorer, toucher, exposer, conserver dans sa bibliothèque. Le livre que l’on est heureux d’avoir, que l’on a envie d’ouvrir, dont on veut parcourir toutes les pages, les unes après les autres, ou que l’on souhaite simplement posséder. Le livre que l’on va ajouter à sa collection et exposer comme un trophée. Le livre que l’on dévore. Le livre qui nous dévore. Ce livre qui finalement, devient une part de nous-même.

Le livre d’hier à aujourd’hui

Aujourd’hui, c’est un objet des plus communs, mais il n’en a pas toujours été ainsi. Jusqu’au XIXe siècle, le livre était un objet incroyablement précieux. Ouvragé à l’extrême, avec des matériaux nobles, parfois serti d’or et/ou de pierres précieuses, il contenait et protégeait des textes sacrés, réservés à une population restreinte d’érudits qui savaient lire. Entre le XVIe et le XIXe siècle, c’était l’acheteur qui faisait façonner la couverture par un relieur, le livre étant vendu nu, sans rien pour le protéger. Dès 1820, la production de livres s’industrialise, les nouvelles presses permettent de produire plus à moindre coût. On imite le style du cuir et de l’or et par le même temps, la littérature devient profane, les auteurs s’accordent plus de liberté de création, les illustrateurs de couvertures font de même. Le livre devient un objet tendance et la couverture sert à décrire et mettre en valeur son contenu. Fin XIXe-début XXe, les penny dreadfuls (des histoires effrayantes qui ont donné leur nom à la série TV du même nom) et yellowbacks côtoient les beaux-livres pour lesquels les artistes-illustrateurs s’en donnent à cœur joie.

Certains, comme Alfons Mucha, ou George Wharton Edwards, deviennent de véritables stars. La couverture prend une signification de plus en plus culturelle et à visée communicative. Plusieurs courants comme l’Art Nouveau et le Dadaïsme se succèdent. Après-guerre, la concurrence entre éditeurs fait rage, et la couverture devient un outil marketing, une sorte de teaser du livre, qui tend à attirer l’attention du lecteur.

Pour une histoire du livre plus détaillée, je ne peux que conseiller l’excellent article de Graphéine. D’autres sources sont disponibles en bas de page.

Don’t judge a book by its cover. Bien que l’expression se révèle d’une grande sagesse, je ne peux pas m’empêcher de ne pas être d’accord si je la prends au sens littéral. Imaginons des clients qui déambulent aux hasard parmi les rayons d’une librairie, cherchant sans l’aide de leur libraire ni idée préconçue leur prochaine lecture. C’est bien la couverture qui la première, attirera leur attention vers tel ou tel ouvrage, qui saura donner une idée du style du livre, de son sujet, du ton employé. C’est elle qui attirera leur regard par des couleurs vives et chatoyantes ou leur indiquera un certain sérieux par sa sobriété. Un essai politique ne ressemblera pas à un polar scandinave, une romance Harlequin non plus.

Beau livre et livre numérique

Quand je suis arrivée chez Glose, qui est une librairie numérique, j’étais –et je suis toujours– animée par une passion des beaux livres et des belles couvertures. Alors quand nous avons décidé de mettre à disposition gratuitement plusieurs dizaines d’ouvrages libres de droits, j’y ai vu l’occasion rêvée de partager ma passion pour la littérature en sublimant tous ces livres par leur couverture.

Pourtant, au vu de ce qui se faisait chez la concurrence, la création de couvertures pour les livres numériques libres de droits semblait reléguée au second plan. Dans le meilleur des cas, l’éditeur ou le distributeur reprenait la version imprimée de la couverture, la réadaptait parfois. Mais si l’édition avait le malheur de n’être que numérique, l’effort était moindre, la couverture, pauvre. Lorsque l’on produit plusieurs centaines de livres numériques dans le but de les distribuer gratuitement, il est compréhensible que le budget alloué soit moins conséquent que pour des productions payantes (et oui, le design graphique a un prix !). Pour autant, l’automatisation et la standardisation ne sont pas forcément synonymes de médiocrité pour peu que l’on y mette un peu du nôtre. En l’occurence, le fait que ces livres ne soient pas imprimés, qu’ils ne se transforment pas en objets physiques, semblait être un prétexte pour les reléguer au statut de sous-produit, alors même que leur contenu était identique.

Mon (notre) envie est de rendre la lecture plus facile, plus agréable, plus fun. La première étape n’est-elle pas de la rendre plus désirable ? À ce titre, quoi de mieux qu’une belle couverture, pour donner envie de découvrir ce qui se cache derrière ?

Rendre la lecture désirable

J’ai donc commencé à créer une poignée de couvertures pour la collection de livres libres de droits de Glose. Ça a été l’occasion pour moi de découvrir (ou re-découvrir) certains grands classiques de la littérature que je ne connaissais parfois que de nom. En effet, pour concevoir la couverture d’un livre, il est nécessaire de s’imprégner pleinement de son contenu, afin d’en saisir les subtilités, le ton et l’ambiance générale. Pour Le Cid de Corneille par exemple, outre les rideaux qui évoquent le fait qu’il s’agit d’une pièce de théâtre, j’ai mis en corrélation une épée et une rose, symboles du dilemme de Rodrigue lorsqu’il doit choisir entre son amour pour Chimène et son devoir en battant en duel le père de cette dernière. L’ombre portée sous le titre souligne la dimension dramatique de l’œuvre.

Pour Le Dernier Jour d’un condamné de Victor Hugo, plaidoyer contre la peine de mort, j’ai assimilé le titre et le nom de l’auteur à des barreaux devant la silhouette déshumanisée et anonyme du prisonnier. J’ai repris les mêmes codes graphiques pour Quatrevingt-treize et L’Homme qui rit, deux autres romans à forte portée politique. Dans le premier, qui a pour trame de fond la Révolution Française et la Terreur, j’ai situé le contexte en montrant le chateau des Lantenac (d’après un dessin de Victor Hugo), famille au sein de laquelle vont se confronter deux visions politiques opposées : celle de la tradition monarchique et celle de la révolution républicaine. Dans le second, j’ai représenté les deux éléments qui mettent l’intrigue en place : le naufrage du bateau d’où le protagoniste réchappe et le rictus qui le défigure. Les couleurs sont volontairement ternes et tristes, en accord avec la dimension dramatique et politique de ces trois œuvres.

La couverture des Malheurs de Sophie de la Comtesse de Ségur a raisonné en moi comme un souvenir d’enfance. Outre le livre que j’ai lu lorsque j’étais plus jeune, j’ai également été bercée par la série d’animation que je regardais chaque soir après l’école (et non, je ne faisais pas que lire 😄). L’image qui m’est donc immédiatement venue en tête en pensant à Sophie a été le tissu rose et la dentelle de sa robe, son ruban vert, ainsi que l’épisode des sourcils coupés, qui est de loin celui qui m’a le plus marquée.

D’autres couvertures ont suivi, chacune tendant à retranscrire l’univers de chaque livre : la vie malheureuse d’épouse et de mère de Jeanne dans Une Vie de Guy de Maupassant, les batailles Napoléoniennes de La Chartreuse de Parme de Stendhal, Le Rouge et le Noir du même auteur, etc.

La création d’une collection

Glose étant régulièrement utilisée en milieu scolaire, nous avons décidé de mettre en plus à disposition un grand panel de livres libres de droits à étudier en classe. S’ensuivait donc un grand nombre de couvertures à réaliser, mais nous ne voulions pas que cela se fasse au détriment de la qualité.

Contrairement aux livres précédents, j’ai donc choisi de rester dans la tradition éditoriale française, réputée pour la sobriété de ses couvertures. Afin que chacune d’elle soit unique, j’ai défini un ensemble de constantes et de variables, dans l’optique de créer la collection des Classiques Glose.

La structure

Elle est identique pour tous les livres, avec la présence des informations principales sur le tiers haut de la couverture, et l’image évocatrice de l’œuvre occupant les deux tiers restants.

L’image et la couleur

Le choix des images s’est révélé d’une importance capitale, le but étant que ces visuels transmettent l’univers et l’ambiance inhérents à chaque œuvre : la bourgeoisie parisienne de Pot-Bouille, l’ambiance exotique du Supplément au Voyage de Bouguainville, la passion adultérine de Thérèse et Laurent dans Thérèse Raquin, etc. De là est ressortie une ambiance colorée que j’ai appliquée à la marge et au nom de l’auteur, le but étant de donner de la singularité à chacune des couvertures.

La typographie

J’ai choisi de tirer parti de la grande variété de longueurs de titres en m’octroyant une certaine liberté dans leur composition, ce qui a permis de donner des rythmes et impacts différents aux ouvrages.

Certains de ces livres sont d’ores et déjà disponibles sur Glose.

Qu’elle soit un exercice entièrement créatif ou plus contraint, la création de toutes ces couvertures de livres consiste vraiment à dévoiler une part de moi-même : j’y retranscris ce qui m’a marqué dans le livre, ce qu’il m’évoque, ce qu’il me fait ressentir, dans le but de transmettre tout cela au futur lecteur et de le guider, lui donner envie de découvrir l’œuvre. C’est aussi une l’occasion d’appréhender l’œuvre d’une autre manière, de la comprendre et de s’en imprégner pour mieux la représenter visuellement, sans la dénaturer pour autant. L’expérience se révèle donc extrêmement enrichissante à titre personnel.

Et vous, qu’est-ce que vous attendez d’une bonne couverture de livre ? Quelles sont celles qui vous ont le plus marqué ?

Sources : https://www.grapheine.com/histoire-du-graphisme/histoire-graphisme-couvertures-livres-1
http://histoirevisuelle.fr/cv/icones/1818
https://www.lamaisondubourg.net/single-post/2016/03/04/Explorons-en-profondeur-Les-couvertures-de-livre
http://www.lib.msu.edu/exhibits/historyofbinding/20thcentury/

Créer de belles couvertures de livres… numériques was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

There is even a “beautiful” warning message, and it’s been there for 4 years now… Well, you know what? Most apps still use UIWebView, because it’s simple to use and do the job. But, you should really migrate to the WKWebView, because it’s backed directly by the WebKit framework, it have more features, and it use a faster javascript engine.

At Glose books, as part of our reader, we extensively use UIWebView, even if it’s extremely custom and fuse native and web interactions and UI. Yes, the new Javascript bridge (called WKScriptMessageHandler) is one of the most wonderful thing of the WKWebView. The core content of a book is still in HTML format. So it makes sense to use a browser view(be it a UIWebView or anything else) to display it. In the current version of our application, our reader still run on UIWebview (shame, shame, shame), because at the time, it was the only solution available.

In our in development version, we’ve made a totally new mobile reader, and this time it’s based on WKWebView. Why did it took us so long? Because until the iOS 11 SDK there was no way to handle custom url scheme loading with a WKWebView. There were still so many advantages of using WKWebView, but the custom scheme loading was a total blocker.

UIWebView support custom NSURLProtocol, it means that in order to provide a custom way to load a custom url scheme, you simply had to create a subclass of NSURLProtocol and then register your class with NSURLProtocol. Then anything calling your custom scheme (like helloworld://) would invoke your custom NSURLProtocol class. Then you’re responsible for loading and forwarding your content. This is very powerful, and in our case we use it to load various assets within a book, such as images, videos etc… So a total no go if we can’t do that. There is a lot of workaround we could have used, but they were mostly hacks, and not really ok for a production application.

WKWebview doesn’t support custom NSURLProtocol, so you can register any classes you want, it’ll never be called because of a WKWebView requests. Well REJOICE ! In iOS 11 Apple added WKURLSchemeHandler. It works almost the same as NSURLProtocol!

Here is some code example for you to understand, because let’s be honest, you don’t really care about all the bullshit above :)

So first you create your WKURLSchemeHandler subclass, you have to implement the two methods, start and stop. Only the start method is relevant, because this is where you are going to do the work.

The stop method can be left empty, or used for you to do any necessary cleanup.

You have access to the full request, so you can check, extract anything from the url, load your necessary data and forward them to the WKWebView. Your content should then load.

And after your class is created, you have to register it with the WKWebView, and you have to do that when you create its configuration:

You’re all set, now you’re ready to load any custom content in your WKWebView!

Custom scheme handling and WKWebview in iOS 11 was originally published in Glose Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.