Real-World AI: How Collaboration And Iteration Practices Help to Define Project Objectives
Explained through a project on crop yield prediction in Senegal where 40 collaborators scoped down a broad problem statement.
From seven broad project goals, to two clear deliverables.
Authors: Ernitia Paramasari, Precioso Gabrillo
In real-world projects, the final deliverable is often not clear at the project start. This is due to the need to first understand the data availability, data quality, as well as additional insights through exploratory data analysis. All of which, will help to test and overthrow hypotheses and create new ones.
Such an iterative process is described in this article about the Omdena project with The Global Partnership for Sustainable Development Data where the collaborative team narrowed down the project's scope from seven objectives to two deliverables.
The overall goal of the project was to to leverage machine learning to help farmers cope with increasingly erratic weather, model the fastest route to markets and mobilities across livelihood zones, and detecting problems in fields with drones and others tools.
The project kick-off
An Omdena project is always unique and comprises several activities. Chief among them is the first meeting, called kickoff meeting, where collaborators, as well as the partners, gather to greet each other for the first time and discuss the challenge. The challenge problem statement comprises an overview, objectives, expected results, and result’s implementation. It is at this juncture that collaborators realize what the challenge becomes.
The problem and objectives
This project is was quite different from other ones, since it had more objectives than average, which were quite abstract (at least for us, in the beginning). Therefore, several weeks of the project was spent figuring out how to best answer the objectives with different tasks. We ended up with nine tasks to answer seven objectives with four guidelines.
The solutions
One task was to predict the weather from climate data, which would further be used by another task to predict the crop yield of a specific area in Senegal. Another one created an app that captures an image of a crop and identifies the type of disease of the crop, while another uses satellite images and specific data about an area to identify the types of crops in that area, and another one predicts crop yield, especially groundnut and rice, in an area. The next one also used satellite images and data to check forest cover reduction over time, for forest management. Additionally, there was also a problem of the language barrier, since most of the necessary research papers and data found on Senegal were in French. So, another task was created to overcome this hindrance with a translation notebook.
The power of collaboration to help each other
There were so many amazing data scientists and machine learning engineers I had the chance to work with, who came up with all the awesome solutions to the problems. As a junior machine learning engineer, I felt like I was not contributing enough. But it turned out that it was alright. I did not have to be as advanced as them, I just had to do what I could that would be useful for the project. Then, after recommending a type of solution for the project, I was suggested to manage a task.
My task: Digital Climate Advisory Services
The task I managed was aimed to provide a prototype of Digital Climate Advisory Services (DCAS), a way for farmers, government officials, and practically anyone interested to find data-driven information about the weather and how it might affect the way they do their farming practices. DCAS can be very helpful in sharing information, and in this case leveraging AI and ML, to increase farmers’ resilience from changing climate. It would also serve as an effective portal to analyze and consolidate data from various sources and provide advise adapted to the needs of farmers in specific areas. For example, how the predicted weather in the next few days would suggest whether they should water more or less, or whether they should add fertilizer and what type, or how the different types of crops, location, and weather can determine how much they will yield, and which would be the shortest route to deliver their crop yield to a client in a certain area. A diagram of all the solutions and how they are connected is shown below.
For this particular project, creating the DCAS prototype meant combining all the different data, images, maps, algorithms, models, and apps created for the different objectives into one nice user interface to be easily accessed by interested parties. We created a simple website for that purpose.
Deploying Machine Learning apps
After doing some research, we were surprised to find that there was not an open-sourced, free, and easy way to deploy a machine learning model as a web app to a publicly accessible website. However, we did find out that we could use Voila! and Heroku to do so. And so, we did. The different tasks resulted in notebooks to process data and show results. We used Voila to turn those notebooks into standalone apps. In this case, Voila made use of Jupyter widgets so the end-user can make changes to their inquiries without being able to run arbitrary code as in a notebook. Then, we deployed the apps with Heroku. Therefore, the end-user will be able to access the apps online.
Putting it all together
We created a website with HTML, CSS, JavaScript, and Bootstrap. HTML was used to create the web pages on different objectives of the project, while CSS was used to stylize those pages. Bootstrap, an open-source front-end toolkit, was then used to make them responsive, utilizing different libraries from jquery, d3.js, to summernote. I did a Nanodegree on programming where I learned different languages including Python, HTML, CSS, and JavaScript, and created some website design projects before this. Another collaborator is a more experienced computer scientist with more advanced front-end web developing skills, so we worked together on this task.
The website showcases results from different tasks and the main site has links to various machine learning apps. The website was also what we used for the final presentation of the project to show the partner how everything could be put together nicely into real DCAS in the future, after further work. And it worked! Everyone seemed happy with the results, and so were the collaborators.
Diversity in Omdena’s Community
So, again, each collaborator is different, with their own skills, knowledge, and experience. And it is their unique skill set that is valuable to the team. One is probably a beginner in machine learning but knows a lot about research and/or data wrangling. One can know very little about Python or R programming but is very knowledgeable in the field the project is on. Together, that kind of team, with all their differences, is a solid one. Brought together to solve a real-world problem, it is a true interpretation of AI for good.
A side note
After the presentation of the final results, we had an after-party. An experienced collaborator who had been in three Omdena projects suggested it. At the after-party, we had a chance to get to know more about other collaborators and talk about things other than work, including the lessons we got from the challenge. It went well, and we even had e-coffee 🙂. However, I also found out that it was not just me who experienced impostor syndrome. I could not believe it. Those very cool data scientists and machine learning engineers with all their swift answers and smart solutions, had the same feeling? This goes to say that no matter how insignificant you think you (or your contributions) are; the most important thing is that you contribute to the best of your capabilities. In the end, we all contribute something. And together, we can say this, “Collaboration is power”.
Iterations, iterations, and more iterations
Our most common approach became an iteration of solving the problem, finding the data, discovering little to no data, redefining the problem, and repeating.
The absence of specified requirements led to unclear data objectives. This created two issues, data availability, and area of substitution. The subject matter, Senegal, lacked data resources available online. Specifically, data aimed at solving the main effort of climate, weather, crop yield, soil, plant types, crop types, & food accessibility. Searches online did not yield any substantial amount of data. Of the data found, it was not enough for a sample count to allow the machine learning algorithms to produce meaningful results, i.e. accuracy ranging from negative values to under 50%.
Another approach was to use different locations that have available data. However, since the problem involves climate, this meant not just any location but a location that is close to the climate and geographical conditions of Senegal. To increase the complexity, Senegal has two distinct climate zones: from The Sahel (desert) region to the north and the tropical zone to the south. Therefore, finding an equivalent geographic location necessitates the application to a portion of the country.
Final thoughts
The all-encompassing objective to the food security issue for this challenge was a challenge in itself. The multitude of aspects of the problem statement created multiple dilemmas dividing the tasks into multiple pieces stretching collaborator resources. In addition, data availability increased the division of work among collaborators, exacerbating the decreasing number of collaborators available for each task.
Therefore, knowing and defining a clear and concise requirement from the beginning is crucial. Especially when multiple varying objectives exist, it becomes important to identify specific goals and outcomes early on.
This project pushed each collaborators’ determination due to its multiple complexities. Not every collaborator made it to the end. To the few, many contributed more than once, more than one task, to ensure that this project would meet its stated goals and outcomes. The two perspectives in this article provide a historical experience that many might encounter and may give insights into the type of complexity within an Omdena project.
