Unlocking Generative AI for Agriculture: Farmer.chat
Rajsekar Waseempasha Ramaskanda Sanjeev Mishra Jayasankar G K
At Digital Green, we are passionate about empowering smallholder farmers, who play a critical role in global food security. Timely access to accurate information is essential for these farmers to make informed decisions and improve their yields. However, farmers and agriculture extension workers often face challenges in accessing the right information which is timely and contextual due to the scattered agricultural data. Farmer.chat, our AI-powered chatbot, aims to bridge this gap by offering extension workers and farmers a smarter tool to access and share crucial agricultural knowledge which is trustworthy, contextual and in the local language in multiple channels.
While farmer.chat is relatively recent innovation, Digital Green has been at the forefront of innovations in agriculture extension for more than a decade and a half. Our journey started with farmer first approach: capturing and curating locally made videos, facilitating screenings in farmer groups and enabling them with software tools for feedback and data collection.
Through our interventions, we were able reduce the cost of adoption of agriculture practice from $35 to $3.5.
Over the last fifteen years, we have been able to reach 6.3 Mn farmers, through a network of > 100K agriculture extension agents which has resulted in estimated 24% increase in income.
While these numbers are impressive, we still have to push the envelop. There are estimated 500 Mn smallholder farmers globally and agriculture extension agents who interact with farmers at typically in the ratio of 1:1000. Reaching the agriculture extension workers and farmers through partnership and technology remains the key.
Challenges of agricultural extension
Imagine an extension worker in a rural village, helping a farmer struggling with a crop disease. To provide the best advice, they need to juggle a vast amount of information — from local soil conditions to government regulations on pesticides. But sifting through research papers, policy documents and websites can be time-consuming, especially when a farmer needs a quick answer.
Here’s the problem in a nutshell:
- Unorganised information: There’s a lot of agricultural data out there, but it is generic and staggered and farmer being a practitioner, finding the right answer can be like searching for a needle in a haystack.
- Context matters: Farming success depends on local factors like soil type and weather. Conversational interface helps narrow down the question to the specific context.
- Accuracy is key: Bad advice can hurt a farmer’s livelihood. Extension workers need trustworthy information sources not only for the farmers but for themselves to gain confidence and reaffirm their knowledge.
Unlocking Answers in the Field: Smarter Tools
We have experimented with chatbots to augment our work for long but with the advances of generative AI, these AI tools can be game-changers. We immediately saw the potential of developing intelligent chatbots powered by Retrieval-Augmented Generation (RAG) on approved, curated information. These intelligent chatbots can utilise advancement in generative AI to enable:
- Fast answers: GenAI outperforms conventional ways of ingesting unstructured content formats in docs, pdfs, crop tables, images, videos etc. Chatbots can quickly find the most relevant information based on a farmer’s question.
- Conversational flow: GenAI can be trained to drive a conversational flow to narrow down the problem. This enables gathering necessary contextual information on one hand and generating byte sized relevant responses on the other.
- Reliable sources: RAG enabled chatbots can be linked and trained to generate answers grounded in databases of officially approved agricultural content, guaranteeing accurate and trustworthy advice for farmers.
By empowering extension workers with smarter tools, the agentic RAG-enabled chatbots can help bridge the knowledge gap between farmers and the information they need to succeed.
Our findings point to potential of decreasing cost of adoption further from $3.5 to $0.35
In this article, we present a detailed description of information flow in farmer.chat.
Farmer.chat overview
Farmer.chat is an end to end solution that can be deployed by any organisation for enhancing outreach to the farmers . It is highly customisable and can be plugged in IVR or messenger or an app and can ingest any kind of structured or unstructured knowledge source.
In the span of over a year, we have continuously evolved our product. In this small timeframe we have been able to:
Reached more than 16k extension workers
Answering > 200k questions
> 6 languages , 40 value chain crops
~ Zero biases or toxic responses
Here is an example of advanced agent driven chatbot:
This is what it happens here if you pay closer look. The user starts with a query seeking a resource: video on wheat rust. The bot understands and presents the relevant video link. The user further asks about the specific chemical for seed treatment . The bot gets the chemical from what is being spoken in the video. The user further asks about the weather without specifying the necessary information like the location. The bot understands that the information is incomplete and prompts back to the user to share location. The user enters the location in free form (if you notice, makes a grammatical error). The bot is able to extract the district name and responds in a structured and readable weather summary. The user further asks about the possibility of pest and disease attack and it looks in the PDF documents and respond with the previously extracted weather information.
This is how a GenAI powered intelligent bot seamlessly uses different tools: video search, video generated answers, weather API and answer generation from PDF.
Next, let’s look how farmer.chat works under the hood, step by step.
Step 1: Knowledge base builder
This is the heart of the system. It is what powers the entire bot by ingesting content and datasets from different sources.
Here are the functional blocks:
This is a web interface powered by FarmStack, which is a digital public good developed by Digital Green and currently enabling DPI like kadp.kalro.org.
The key components of knowledeg base builder are:
- Connectors: act as pathways that allow the knowledge base builder to ingest various types of data from outside (eg, Googel drive, youtube)
- Content processing engine: the ingested data is processed and transformed into a format suitable for the knowledge base and involves auto summarisation, content categorisation, auto tagging
- Semantic chunking: the organised files with metadata are processed with sentences similar in meaning grouped together in text chunks
- Vector db creation: each text chunk is converted into vector representation using an embedding model
- Video processing: auto transcription and translation to create description of the videos
- Video indexing: LLM to generate description of videos and automated sections
Using the above components, the knowledge base builder serves to ingest, organise and convert the data into vectordb format which can be used by AI application.
Step 2: AI modules
This is the module which is the brain of the system. It is where the query of the user is understood, processed, the necessary information to respond to the user query is obtained and the apt response is generated that is grounded in the knowledge base.
2.1 Response Generator (RAG):
This sub-module utilises a Retrieval-Augmented Generation (RAG) approach to create a response for the user. It is what ensures that the information delivered is grounded in the content and not outside.
RAG as many are aware of combines two broad streams:
- Information retrieval: Searching the knowledge base for relevant information that matches the user’s query. This involves calling the vector database API created in knowledge base builder to get necessary text chunks.
- Generation: Using the retrieved information in the text chunks and the user query, the generator calls LLM and generates a human-like response that addresses the user’s needs.
2.2 Query Orchestration (Generative AI agent):
This sub-module leverages large language model (LLM) to process the user query and plan what steps need to be taken. There are three main components:
- Planning and reasoning agent: understand what user is trying to ask and which tools can be used to answer the user question.
- Execution agent: call the specific tools with necessary parameters so as to synthesise the responses.
- Tooling service: Use the API end points to create tools and their description that the LLM can use.
This sub module is the core to create conversational flow and helps narrow down the user input.
2.3 Local Language & Voice ( Traditional AI):
This sub-module employs language and ASR modules to:
- Detect language : Identify the language used in the user query.
- Translation: If necessary, translate the query into a language understood by the system.
- Speech recognition (for voice queries): Convert spoken user queries into text.
- Text to speech: to convert the final response to an audio note.
Step 3: Integrating with frontend system
Finally, we have the frontend API handlers consisting of two main modules:
3.1 User Interaction
The process starts with a user interacting with the chatbot through various channels like a messenger app, IVR (Interactive Voice Response) system, or even a mobile application.
3.2 Frontend API handlers
This module handles the user interface (UI) aspects of the chatbot. It ensures a smooth user experience by tailoring the interaction based on the chosen platform (messenger, IVR, etc.)
Step 4: Plugging it all back in knowledge base
Although the bot can function with the flow in step 3, the major addition here is feedback of all the conversations back to knowledge base builder.
It is important that the conversations with associated logs are fedback in the knowledge base that can be reviewed manually to curate golden question — answers.
4.1 Conversation Logs:
The system keeps track of user interactions and chatbot responses. These conversation logs are valuable for:
- Reviewing and Refining Responses: Human evaluators can analyze logs to identify areas where chatbot responses can be improved.
- Generating Golden Q/A Pairs: Subject matter experts in the field of language, climate, gender, agronomy, extension can co-create high-quality interactions from the logs as question-and-answer pairs. These “golden” Q/A pairs are then used to further train and improve the LLMs used for query orchestration and response generation.
4.2 Analytics:
This module provides real-time insights into the performance of the chatbot system. These insights can be used to:
- Identify areas for improvement in content.
- Monitor user engagement.
- Track the effectiveness of the chatbot in answering user queries.
Conclusion
This article we presented the flow of information to make farmer.chat come to life. The generative AI space is evolving fast and so are we. This is just the overview and in the next articles we will take you to deeper dive into each of the modules. Stay tuned!!!
Big Thanks
To our gracious donors and government partners for making this possible.
To the readers!!!
