CSR Tale #6: A masterclass in collaboration!
This CSRTale comes from Nipun Batra (post-doc at the University of Virginia), Jack Kelly (researcher at Google DeepMind), and Oliver Parson (Data Scientist at Centrica). The tale starts when all of them were each a PhD student at a different university. It shows how you can often have excellent remote collaborations and how such collaborations result in impactful work! The tale is told in the first person by Nipun, with inputs from Jack and Oliver.
This is the story behind NILMTK (Github, Paper). I was a second-year PhD student working towards my long-term dream of accessible electricity for everyone. Having grown up without access to 24x7 electricity, this topic was close to my heart.
There are various ways to ensure accessible electricity. One such way is to target and reduce the energy consumption of buildings, which contribute roughly one-third of the global energy consumption. One way of reducing the consumption is to inform occupants about their detailed appliance usage. Studies indicate that such fine-grained feedback can help them take measures to reduce their energy usage. The most scalable way and least intrusive way of providing such a feedback is by using what is called non-intrusive load monitoring (NILM) to break down the total building energy into constituent components using machine learning.
NILM is an old field (established in the early 1980s!). However, I had not heard about it until my second year of PhD. While looking at various building energy-related problems, I found a blog by a British researcher, Oliver Parson (whom I refer to as Oli throughout the blog). His blog (called Disaggregated Homes) seemed like exactly what I wanted to read about and work upon, and provided an excellent overview of the field and important resources and problems. Upon reading more, I found another British researcher, Jack Kelly, who was a regular commenter on Oli’s blog talking about his preliminary work in NILM, his passion for carbon reduction, and his DIY energy monitoring deployment. I was busy doing a smart meter and multiple energy-related sensing deployments of my own. So, I regularly started visiting Jack’s blog too. Sitting almost halfway across the world, I felt that Jack was probably the best person to understand what I was doing. I started discussing my work with him by commenting on his blog. I felt that we both had a similar style of work — open source mindset, caring about carbon emissions, liked to do a bit of hardware, and blogged regularly.
A few months had passed and I really committed to working on NILM. I had been warned by some people that since NILM is not exactly a new field, I better be really sure before diving into it, since there is limited scope for novelty. Somehow, I felt that I should take the risk!
In one of my yearly evaluations, someone on my thesis committee commented, “Oh! Why don’t you use the best NILM algorithm and build some cool applications?”. I didn’t have a really good answer for this. So, next, I spent about 3–4 months implementing two basic NILM algorithms, rather than starting any new research projects. Surely, there was a way I could prevent others from wasting 3–4 months on similar problems. I could also see dozens of new NILM papers coming up on Google Scholar feed every week. But, I felt I was no closer to my goal of finding the best NILM algorithm.
One fine day in November, on an email thread where I was discussing with Jack about potential NILM venues, I mentioned to him — “..let me tell you about a paper that I was planning for E-Energy 2014. The theme I had thought was to compare different data sets and bring out the differences if any. I also envision developing a NILM framework using some basic algorithms and compare it over all the datasets”. Jack really liked the idea and thought that it would be a great chance to collaborate. He also mentioned that Oli would be great to have on the team as Oli had more experience in NILM. Jack made a formal introduction to Oli and presented our idea. Oli loved the idea of a toolkit. The three of us discussed the idea with our advisors and collaborators. We all received a resounding go-ahead, which was very encouraging.
Just to put the timeline into context, the deadline for E-Energy was early January. We had little time to focus our efforts and produce a high-quality submission. Oli had attended a few NILM conferences and he agreed that a toolkit would be highly valuable to the community. We thus decided to build the NILM Toolkit (or NILMTK) — an open-source toolkit to enable easy comparative analysis of NILM algorithms across multiple data sets. We focused on addressing three core problems that the NILM community faced: i) hard to assess generality given the differences in datasets used, and non-trivial to set up similar experimental conditions ii) lack of benchmark algorithms iii) inconsistent usage of metrics. In NILMTK, we i) encoded multiple NILM data sets in a convenient to use common format ii) provided a suite of standardised accuracy metrics iii) provided two benchmark NILM algorithms and iv) standardized appliance nomenclature (among other contributions).
Oli took over the metrics work, and also led the writing efforts, while both Jack and I assisted him. Jack worked on the appliance standardisation and converting the existing datasets to a common format. I assisted Jack in dataset conversation and worked on the NILM benchmark algorithms. From day 1, we decided to have the project up on Github in public. We wanted it to be a community-driven project.
These were some of the most productive days of my research life. If I remember correctly, we finished our set tasks for the E-Energy deadline a few days before the deadline, which was quite a rarity too! Oh, and our first version was 16 pages when the allowed limit was 12 pages. Producing 16 pages of manuscript in such a short time was amazing!
We circulated our paper over to colleagues and got good feedback despite the unusual contribution. I also learnt the differences between American and British English. The paper looked in great shape by the time we submitted. We also put the paper up on arXiv as we wanted the tool to be useful for the community from the first day. We circulated the paper to a few NILM colleagues who all seemed to like the paper. Our spirits were really high!
Interestingly, by the time the reviews came out, our paper had already been cited a few times, which was a real first for me. Cheers to arXiv and Github and the community! The reviewer comments we received were the best we had ever received, and all the reviewers felt that our work was an important contribution to the field. We also submitted a short version of the paper to the newly formed NILM workshop, where its positive reviews encouraged the organisers to create a session dedicated solely to the toolkit. By the time the E-Energy conference had rolled around, the toolkit had already undergone a serious rewrite and we’d begun writing a second paper to send to ACM BuildSys. Jack took the lead with the rewrite, with the aim of simplifying the toolkit and making it more extensible while also allowing it to handle arbitrarily large data sets. The toolkit received a lot of attention at the demo session, and ultimately won a prize for the best demonstration!
Fast forward four years, NILMTK has been cited 157 times and continues to get a lot of love on Github (250+ stars) despite the niche research field. While Oli, Jack, and I have moved on to other things, it gives us great pleasure to see other researchers extend the toolkit with new data sets and algorithms. It was hard to hide our happiness when a newcomer to the NILM field expressed how easily they could get up and running with a publicly available data set and NILMTK.
I think NILMTK stands as a good project to showcase the possibilities of open collaboration, for tools in research, for scientific reproducibility!
CSRTales: Note the ease with which three students from three universities collaborated together for the project. This kind of collaboration is possible only in academia, and is a great example of the research freedom valued so highly by academics.
