OpenWISP 2 Network Topology, GSoC ’17
For the last three months, I’ve been working with OpenWISP, an open source organisation selected for the Google Summer of Code program. It has been an amazing experience and I’ve learnt a lot over the summer under the mentorship of Federico Capoano.
This post links to all the work I’ve done during the program. There have been a total of 42 PRs, around 90 commits, 8000+ additions and 8 version releases. This summer has definitely boosted my self confidence and also my Github activity.
About
“OpenWISP is a software platform that can be used to implement a complete Wi-Fi service. The OpenWISP software suite includes five main applications, derived from tools used to offer public Wi-Fi service in Italy and other European countries. OpenWISP 2 is also used as a Wi-Fi controller.”
- openwisp.org
One of the use case scenarios of OpenWISP 2 is mesh networking. In these scenarios, being able to collect and visualise the topology information is crucial.
My work was to create a new module for OpenWISP 2 named openwisp-network-topology, a reusable Django application. It consisted of pulling django-netjsongraph and adding features on top of it. As most of the logic has already been implemented in django-netjsongraph, it was decided that all the new features would be first added to it and then be integrated into openwisp-network-topology. Learn about django-netjsongraph here.
This application, in substance is a web interface to setup the collection of network topology data from mesh networks as well as a mesh network visualiser and a very basic monitoring system.
Final Product
The final product out of my GSoC is a whole new module of OpenWISP 2 called openwisp-network-topology and many improvements in the existing dependencies. The work done can be summarised as follows,
- Refactored the code of django-netjsongraph to make it reusable. The PRs referencing this issue are #26, #35, #42, #44, #45, #48 and #51.
- Integrated django-netjsongraph into openwisp-network-topology with multitenancy features. PRs #4 and #5 add these features.
- Created a new signal in django-netjsongraph that is sent when the status of a link changes. PR #28 implements it.
- Added a way to customize the theme of the visualizer. PR #30 adds it.
- Added netjsongraph.js visualization to admin site. PR #36 adds it.
- Moved all the logic shared between openwisp modules to a new package named openwisp-utils and integrated it into openwisp-network-topology, django-netjsongraph, openwisp-controller and django-netjsonconfig, hence removing all the duplication.
- Implemented a way to save daily snapshots of the network topology so that users can go back in time and visualize the past network topology.
- Added a switcher in the visualizer that allows to go back in time and visualize the past snapshots. PR #49 implements the above two features.
- Added a netdiff parser for parsing OpenVPN status logs. PR #44 references it.
- Integrated openwisp-network-topology into openwisp2. PRs #19, #21 and #23 implement it.
- Documented all the features and achieved a test coverage of 100% in all the packages.
Check out the weekly reports posted by me in OpenWISP mailing list here.
Notable Features
All the OpenWISP2 modules use customised Django admin interface as the UI. Therefore, most of the work involved developing features in the admin interface. This was challenging because the admin interface has a different structure and many predefined methods.
- Multitenancy in admin interface
“The term “software multitenancy” refers to a software architecture in which a single instance of software runs on a server and serves multiple tenants. A tenant is a group of users who share a common access with specific privileges to the software instance.”
- Wikipedia
This module can be hosted on a server with multiple organizations being able to use it simultaneously and grant access to their users at different levels. The packages openwisp-users and openwisp-utils have been used for implementing this feature.
- netjsongraph.js visualization in the admin interface
Network topology can now be visualized from the admin interface too. A lot of admin site customization has been done to implement this which involved, adding a button to open the overlay, defining new URLs and writing corresponding views.
- Topology Snapshots (Way back machine) in admin interface
This feature allows the user to visualize the network topology of the past. A switcher has been added to visualizer which helps in navigating back to a previous date. It makes it easier to monitor and know the status of the network. This has been integrated into OpenWISP 2. A cron job has also been set up to take snapshots at regular intervals.
Building it
All the OpenWISP 2 modules are built in the Python ecosystem. The major technologies used in this project are Python, Django, Django REST Framework, JavaScript and NetJSON.
I spent a good amount of time in understanding the code bases of django-netjsongraph and other OpenWISP modules. The code was well written and documented which made my task easier and made me realise the importance of writing good code.
- Refactoring
“DRY ‐ Don’t Repeat Yourself ‐ Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.”
- Andy Hunt & Dave Thomas from “The Pragmatic Programmer”
A lot of logic has already been implemented in django-netjsongraph and we strictly decided not to duplicate the code but reuse. This was when my first task had begun. I had to provide base classes to improve re-usability by third party apps. This task helped me learn a lot about refactoring, writing neat code, maintaining orthogonality and re-usability.
All the code of django-netjsongraph is now reusable and can be used by third party apps by following the instructions given in the documentation.
- Automated and Reusable Test Suite
“If you’re actually doing TDD, you’re throwing away tests all the time, as your understanding of what the code is changes.”
- Kerri Miller on “Test Driven Development”
My mentor never pushed a feature upstream until the corresponding tests were written and all the test cases passed. I realised the importance and utility of automating the test suite when one addition broke so many features of the project. We integrated Github with TravisCI, coveralls.io and requires.io for better results. I learnt a lot about TDD and developing an extensive test suite.
The code base has a test coverage of 100% with all the tests reusable.
- Documentation and styling
One really good habit I’ve developed is documenting the feature immediately after it has been developed. This helped me keep the documentation up to date and served as a great reference later.
We followed all the official style guidelines including PEP8 for python and Google style guide for JavaScript. We use tools like flake8 and isort integrated with TravisCI for online style checks.
- Tech
I’ve been able to learn a bunch of new technologies and a lot of advanced software engineering concepts. I’ve boosted my development and programming skills to a different level. Ansible and Django REST Framework are some really cool technologies I’ve worked with.
Ansible is an open-source automation engine that automates software provisioning, configuration management, and application deployment.
The best way to install openwisp-network-topology on any server is by using ansible-openwisp2, Ansible role that installs and upgrades the modules of OpenWISP 2.
Contributions
I’ve contributed to many repositories over my GSoC period. I authored openwisp-network-topology and openwisp-utils and am one of the major contributors of django-netjsongraph. I’m currently the python package index maintainer of all the three repositories.
I’ve also contributed to ansible-openwisp2, openwisp-users, django-netjsonconfig and django-sortedm2m.
Status
Almost all the goals of the projects have been met. I’ve worked extra in some areas and lesser in the others which overall makes it a good GSoC contribution. There is one feature in it’s development stage which shall be integrated soon after proper testing. It can be viewed at #44.
openwisp-network-topology is now integrated into openwisp2 and is being tested extensively in production. More people are using django-netjsongraph now and have submitted feature requests.
Important Links
Not ‘the end’
I’d like to thank all my mentors and community members for the constant support and motivation. Special thanks to Federico Capoano for all the guidance. Had he not been, the project wouldn’t have been a successful one or I hadn’t done GSoC in the first place.
It has been a thrilling summer over all. I’m now familiar with a large part of code base and would continue to contribute to the project and improve the performance. GSoC is not the end.
Not to forget, Thanks to Google for organising such an awesome program.
