The traveling patroller

At Airtel Digital our mandate is simple. We need to optimize efficiencies by going digital.

As part of the Product Engineering, one of the main responsibilities that came to me and my team was to digitize the Network Inventory Capture and the “fault fixing” process in our physical fiber lines. In simple words reducing user frustration by minimizing the time it took to fix damaged broadband cables. This would also solve the problem of capturing/updating the network inventory digitally and help us plan our roll-outs better by creating a national database for network inventory. To do that we had to take a look at how the existing process currently worked.

Network maintenance and fault fixing is taken care of by a large network of Patrollers, NMT (Network Maintenance & Transmission) Engineers and Area Managers. Patrollers are outsourced partners whose job it is to travel along the route of the network pipeline everyday, to check on the condition of the cables. One patroller usually took take care of 100 kms of fiber length. This entire system included a team of over 5000 patrollers, 1000 NMT Engineers, all rolling up to Area Managers.

When the patroller finds a cut in the cable or any other disruption, he would inform the nearest NMT Engineer and wait until the Engineer showed up and fixed it.

To truly understand how the process worked we had to take a field trip. We jumped into a jeep and followed a Patroller making his normal rounds. We also spoke to the engineers and area managers to understand what their pain points were.

The main things that came out were.

• Patrollers wouldn’t always strictly follow the specified route but still claim full expenses and all reimbursements.
• There was no single/ common knowledge system for the fixing of faults. It was all done on various devices, sometimes on paper and often knowledge was lost leading to longer fix times.
• There was no digitally captured route plan. It was all paper based.
• The network inventory capture was not accurate, paper based, manually updated by NMT engineers after job was completed.
• Network planning did not have accurate data for future expansion

Back in the office my team started strategizing a solution for all of the above issues. It was obvious to us that this had to be a mobile driven solution as all of our stakeholders were always on the move. We started with a simple mobile app which can be used by patrollers, NMT engineers as well as area managers.

Solving the patroller problem.

For the patrollers the app was a simple way to create transparency in their daily routes. The app pinged the location of the patroller back to the server every 5 seconds. This meant we had to open nearly 10000 parallel connections to all our patrollers and NMT engineers. We initially went with a simple web server with Tomcat & weblogic but soon realized that the overhead on every request made it inefficient and expensive to run. We then moved to MQTT and websockets, and finally settled on using the Netty framework. The system was then expanded to be able to support 50,000 parallel connections to allow for growth.

The second issue we ran into was to actually figure out what a route deviation was. If a patroller double backed to a previous location to fix another fault, it was confusing our system which read it as 0 distance travelled. We then mapped all the data into aerospike to be able to calculate the actual distance travelled by the patrollers.

The last issue we ran into was the management of all this in real time data to make sense of it and let the area managers know in case of any deviation. We finally chose Storm,Kafka and Aerospike for real time processing and notifications.

Solving the NMT problem.

We enhanced the mobile app to now allow both patrollers and engineers to photograph and document all faults and corrections made.

This allowed flexibility in which NMT could respond to which faults, because all the history was now available. Previously NMTs used to only work on areas they had worked before because they had “contexts” of the faults. Now the nearest NMT could be routed and had all the details needed to fix the issue.

So effectively NMT engineers job went from

• Getting on a call to find out the location of the cut from the Patroller
• Then get on another call with the Area Manager to find out how to fix the problem
• Scribble it down on a piece of paper or a notebook
• When he’s in the manhole fixing the issue, keep referring to this notepad for instructions!
• Updating the back-end database on inventory details by calling someone

To

• Just opening the app and fixing the problem and moving on to the next one

The projected impact of this system is close to 2000 crores in savings over the next 2 years and more importantly happier customers.