Making Wikipedia Available by Text Message — Part 3
This project was on hiatus for a while, but I was recently inspired to finish it and get it live. A few things have changed in my approach, but most prominently I am now running the entire application locally on an android phone. Thus, to provide similar facilities all one really needs is an android phone, a sim card and WiFi. The most exciting aspect of this new setup is the incredibly low running cost, and the potential reach of the system.
The Reach:
The population of Pakistan is ~197M, the number of cell phone users is ~152M, however only about ~55M of these cell phone users have access to 4G/3G/GPRS facilities. Given mobile phones are the primary means of internet access in Pakistan, there are ~100M+ people who do not have access to the internet. Accounting for the number of people who cannot afford to, or are not literate enough to use internet services this number is even larger. Moreover, many service providers naturally choose to put 4G facilities where they expect the greatest usage, i.e. greatest return for the installation of the service. This means that many small villages and impoverished areas are left out. Areas which would perhaps benefit the most socioeconomically.
As I mentioned in Part 1 of this series, not having access to the internet means not having access to the largest resource of information in human history, a debilitating handicap. We cannot wait for the economics to be right for the telecom companies, nor can we wait for the economic standing of the individuals to rise. And so, I address this problem by solving one small part: Access to Wikipedia.
The Economics:
Now that all the loops (Twilio, ngrok, etc.) are closed, the platform is unbelievably cheap to run and use. This platform runs entirely locally on a smart phone for about <1USD per month, and the number of people it can serve is limited only by the volume of messages the platform/network can handle. For 70rs (0.50USD) I can get a monthly package of 5GB data and 12000 outgoing sms texts. This roughly translates to serving 400 people per day, at ~0.13c per individual per month. For the current application that is more than enough, and with a more robust approach, this method could perhaps be scaled to accommodate ~100M people. (for 130kUSD, with no economies of scale accounted for).
On the user end, only standard sms charges apply. For most providers this is less than 0.01rs/sms (equivalent to 15k messages for a 1USD) —practically free. (With a short code this could be absolutely free.) So for an average user making 200 queries/mo., this is is less than 2rs/month. And so, the entire library of knowledge on Wikipedia, in English and in Urdu, can be made available at the cost of a few sms.
Technical Details:
As mentioned in part 2, the game changer for me was the discovery of EnvayaSMS. This is a mobile app that helps automate sms messaging services on android phones. The Envaya sms app takes all incoming messages and forwards them to a server, and can reply to the sms according the to the servers response. On the server sits the piece of code which reads the sms, formulates the appropriate queries to the Wikipedia, and returns the results in a structured manner.
Another game changer was learning that I could run this server locally on the phone itself, eliminating the need for ngrok(etc.) or a persistent external server. To accomplish this I used Termux, an app which can simulate a Linux environment on android phones. Termux runs a python script which sets up a local Flask server, and serves the queries provided by the Envaya app. The Flask server does not currently scale well (particularly it cannot handle concurrent requests), but is easily adaptable to a more scalable implementation. Moreover, the Envaya app has a queuing routine so the server is not currently overwhelmed (I have yet to explore the limitations of this)
Future Implementations:
There are several things which I would like to change in future implementations. Firstly, I want to avail more services than Wikipedia. Some of these include: an English to Urdu dictionary, geo-location facilites, government services, weather forecasts, news (and many more services which the current USSD facilities cannot provide). Secondly, I would like to obtain an sms shortcode to make the platform completely free for the end user. Lastly, the technical implementation needs to improved to handle a large number of queries, this is not a technically challenging task but is currently beyond my capabilities. Someone with even limited experience in large scale deployment of applications would be able to adapt the framework in a matter of days. [More scalable server implementation, AMQP server implementation etc.] (If this is you, please contribute, links to repository below)
Where I am right now:
The project is in alpha testing, and live at +923164948334. I am currently testing it with students at NJV High School and some friends. (See the Usage references below if you would like to try it out!). I think we should be ready to beta test in a month or so.
Usage of Platform:
The platform is targeted towards individuals who don’t have English as their first language, but are proficient enough to use sms. Thus the interface is kept very simple, and the instructions as explicit as possible. For the most part the interface is the same as before, the user simply has to message the number above with the following the body:
- search <search term>: Searches Wikipedia for the search term, and replies with the top 5 corresponding page names. e.g:
search Albert Einstein
search Manto
- wiki <page name>: Opens the Wikipedia page corresponding to the page name (or best match), and sends the article piecewise over a few messages. e.g:
wiki Albert Einstein
wiki Saadat Hasan Manto
- more: Request the next piece of the article.
- urdu: The user would text this to change the language of the articles to Urdu (There are vast number of Wikipedia articles in Urdu).
- english: As above, it would change the language of articles to English
- how: This yields a simplified usage guide
- define <word>: This would return the dictionary definition of the word. e.g:
define pendulum
define complexity
define intricate
How to implement this yourself:
The implementation is fairly straightforward, and should be clear from the code provided below. However I will supplement this with some technical details and resources.
Flask: https://flask.palletsprojects.com/en/1.0.x/quickstart/
Termux: https://wiki.termux.com/wiki/Main_Page
EnvayaSMS: https://sms.envaya.org/
Wikipedia API: https://wikipedia.readthedocs.io/en/latest/code.html#api
Virtualenv: https://virtualenv.pypa.io/en/latest/
- Write a script to run the Flask app and handle the incoming requests from the EnvayaSMS app and send them to your ‘sms-app’. In my implementation this is called “envaya_test_4.py”.
- Write an ‘sms-app’ script which takes the message body as an input and performs the function you wish to implement. (Wikipedia access, chatbot, etc.) In my implementation this is the “Wikihelper2.py” script.
- Point the EnvayaSMS app to your local server through the app settings. Using the settings provided this would be “http://localhost:5000/sms”
- Set up the local server using Termux by first installing python: Type ‘pkg install python’ in the Termux console. And then installing relevant dependencies with (pip install virutalenv, pip install flask, pip install wikipedia etc.)
- Create a virtual environment on the phone, and run the local flask server script ( “envaya_test_4.py”) from there.
If you need any help setting this up, please drop me line at tuniomurtaza (AT) gmail. If you want to help contribute, see the GitHub repository.
Repo: https://github.com/MirTunio/BurningDaylight/tree/master/SMSWIKI
Conclusion:
I look forward to testing this platform further and making it accessible to as many people as possible. It would address a crucial gap in the availability of information to large number of people in Pakistan and around the world.
