Open Source technologies running U-Report

My first post on Medium.

I have been working on U-Report in Burundi (Africa ): A system which collect data by mean of SMS and help “your voice be heard”.

What made me want to talk about it ( and write my first Medium post); is the similarity between the stack of open source technologies used to run U-Report and a post I read about the technologies behind the Instagram platform.

I don't know if it still the case, … maybe they have changed those open source components they used at the time of the writing of that blog, but what has always amazed me is … Well ! You needn't so much expensive tech products to run a successful piece of software!

Let’s review that.

OS

U-Report run on Linux : Ubuntu as well as Instagram. We, developers, we program, test and deploy on linux machines. By the way, “ubuntu” in Kirundi ( our language) means either “humanity and mankind” or “free” ( as in free beer ! )

Servers

As Instagram, we use nginx . Most peoples in Burundi love Apache … That’s because they don’t know all the goodies of nginx : lower memory usage, serve more concurrent clients … We also just use the uWSGI server which comes with Django.

And for deployment, we rely on Fabric . I was really delighted to see that Instagram uses it.

All the instance are supervised by …Supervisor ! Trop fort!

Backend Applications

We also run Django ! Not really Django, but RapidSMS !

At the beginning there was Python : a very “nice looking” programming language which makes reading code source an easy “pain”. Then came Django : a high-level Python Web framework that encourages rapid development and clean, pragmatic design . A framework is like … Well! A collection of useful functions, methods, objects … and other stuffs provided to you by other developers so you won't reinvent the wheel …

In fact we use RapidSMS : a free and open-source framework for rapidly building mobile services for scale based on Django and Python.

So, at last we have : U-report > RapidSMS > Django > Python.

Data storage

Well. Postgresql is the king : most advanced, SQL-compliant and open-source objective-RDBMS.

I may not like it’s mentality of sticking to the rules ( especially in migrations ) but sometimes it has shown me that I need to make sure the integrity of my database is preserved. No permissive code is allowed!

We may some days use Pgbouncer but till now, we haven’t.

We uses Redis : we need it for Entity-Attribute-Value storage model for django apps.

Task Queue

The big difference you learn between your “bedroom” developed app and a “realworld” : problems of memory and processor usage.

The more you define some action to be taken on each incoming message, the more you’ll have memory “hunger”. You need to perform not only some actions on SMS, but there is also some tasks that need to be performed each 1 second, 2 second, ... , 2 minutes, etc… Even each 24 hours! How to handle that ?

So we use Celery : a task queue with focus on real-time processing, while also supporting task scheduling. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).

Kannel

kannel in my humble opinion, is the greatest open source software to ever been invented to talk to an SMS.

Kannel is a compact and very powerful open source WAP and SMS gateway, used widely across the globe both for serving trillions of short messages (SMS), WAP Push service indications and mobile internet connectivity.

Well. All those technologies are there over the internet open sourced. You're free to contribute to their success ! All the code source are on GitHub !

P.S : U-Report has been upgraded to a new crazy-insane-awesome tech replacing RapidSMS : RapidPro . Kudo!