Designing Farm-driven APIs for API-driven farms
How do you build big, complex, domain-driven APIs? Why does the API schema matter? How to manage API-as-a-product?
When our Syngenta Machine Learning platform dedicated to Computational Agronomy reached a reliable infrastructure state, and several advanced predictive models were ready to be onboarded there to be used in Digital Ag applications, we faced one crucial question: how do we get the APIs right?
At Syngenta Digital, we want our Ag AI (Agriculture Artificial Intelligence) solutions to be usable, useful, and re-usable so that they can scale over time and reach many growers, no matter which application they use. And considering that this success lies in our clients’ experience when interacting with our APIs, it is evident that we need to take special care of our API design.
Dealing with the specific and unique domain of Agronomy presents several challenges for which we struggled to find precedent learnings and experience. But if we got this wrong, it could lead to growers making mistaken decisions, jeopardizing crop yield, their annual income, and, ultimately, our food supply. No pressure!
Should you read this article?
If you are a tech enthusiast, continue reading if you are curious about an edge-case for API-as-a-product innovation and management. Regardless of whether you apply it to farming, our story can undoubtedly inspire other less-traditional API product managers.
If you are into the agricultural world, well, you are likely to experience and maybe even contribute to the digital agriculture revolution. If you want to understand how computing technology is becoming mindful of the complexities of farming, keep reading.
A unique context
Computational agronomy is a relatively new science that aims at transforming agriculture by helping farmers to make better farming decisions with the support of AI-driven insights. It is an interdisciplinary field that applies computational and data-driven approaches to codify and enhance agronomy knowledge. It involves developing models, algorithms, and software tools to analyze large-scale agricultural data and using this information to return advanced farm insights or recommendations.
So, what is and what does an API product mean in our case?
Essentially, it conducts a data transaction from the field back to the field (so far, we only deal with GET method). This means that it is a system that collects the necessary field and crop data required to run ML (Machine Learning) models and returns a potentially massive amount of data that supports the delivery of information or recommendations about farming strategy.
The challenges in front of us.
Firstly, our direct clients are Digital Agriculture applications in which the APIs will be integrated. While we acknowledge only a few will have the agronomic skills to verify domain coherence, we want to empower any software engineer to set up this data transaction right and with maximum autonomy.
Secondly, our API products are about handling highly specialized data in both API requests and responses. Misunderstanding and misusing this data can irrevocably damage the industry’s trust in our innovative solutions.
Thirdly, most of our API products require managing a large volume of data and a wide variety of data formats within these requests and responses. And we don’t want our clients to suffer from the burden of data load and complexity.
So, what do we need to get right?
- Ensure API clients send the right field data
- Ensure API clients use the API data right
- Not making it a nightmare for our API clients to figure out which and where to send and find the data in our API contracts
After many experiments and failures, here is what we now feel confident to advise.
Design early — Make it useful
Assuming the market fit and value proposition are set up, it is crucial to carry out API design before starting model development. The API success also depends on the involvement and collaboration between the business stakeholders and all functions involved in delivering the computational agronomy model. Although the benefits and necessity of API first approach are widely discussed online, I will share below the specificities met in Computational Agronomy.
Step 1 — Ensure you have a crystal-clear understanding of the insights needed
“If I had an hour to solve a problem and my life depended on it, I would use the first 55 minutes determining the proper question to ask.”
Albert Einstein
How is the data generated going to be translated into actionable grower insights? Whether it’s by supplying prompt pest occurrence alerts or a season diagnosis based on yield forecast time-series, it is a different story and it will lead to different data crafting. In the first case, qualitative outputs may be handy, while in the second case, normalized numerical values will be correct.
To build a strong understanding of those nuances, you must empathize with end-users (the growers). This level of understanding requires investing a fair amount of time with all business stakeholders and supercharging your curiosity by asking many questions. You should also consider the user interface that the API client will use to deliver the insights.
Additionally, think about re-usability. Additional API clients may be supplying the solutions in different contexts. For example, will a small grower use a pest alert system in the same way that an agribusiness agronomist will? Any slight adaptation now may ease future upgrades and the acquisition of new clients.
Nevertheless, maintain a well-balanced intuition between addressing core market-driven benefits and anticipating game-changing opportunities. While building an innovative product vision, you need to consistently deliver solutions in the present.
Step 2 — Get the end-user inputs right
“At the heart of every product person, there’s a desire to make someone’s life easier or simpler. If we listen to the customer and give them what they need, they’ll reciprocate with love and loyalty to your brand.”
Francis Brown, Product Development Manager at Alaska Airlines
While the reliability of computational agronomy insights depends on the richness and accuracy of field information inputs, those are challenging to obtain. As highlighted earlier, the quality of those inputs determines the reliability of the outputs. Costly inputs can arise from reluctance or impracticability for farmers to share detailed field information.
By engaging with the scientific model owners and the business stakeholders, you need to identify which inputs are mandatory for prediction compliance and which are great-to-have to increase model accuracy. For instance, a particular model might not be able to run without knowing the planting date but will still be of value if planting depth is still optional. Then, the management of default values must be thought ahead for optimal model behavior.
You may also need to transform raw input parameters into user-friendly ones. For example, many crop models require a set of crop variety traits such as maturity groups or pest sensitivity traits. However, it is often more convenient for the users to share a crop variety name. In such a case, you might consider building a system to retrieve the necessary traits from an industry-sourced crop variety catalog.
Step 3 — Understand the usage dynamics
“Design is not just what it looks like and feels like. Design is how it works.”
Steve Jobs
In agriculture, the consumption of crop insights is seasonal and timely. This implies the ability to customize prediction supply patterns, such as the prediction window or frequency of predictions. A second category of inputs results from this: the configuration inputs. By designing them smartly, your API becomes adaptable to variable use cases and provides flexibility for API clients to optimize grower insights. Configuration inputs can either trigger the model system (like customizing the lookback period for the inference dataset) or the response features (like customizing the forecast window required or the prediction frequency). Lining these requirements before development fosters effective collaboration between data scientists and ML engineers to build a smarter and highly performant system.
Parse data in your schema in a standardized and agronomy logic way — Make it usable and re-usable
If you went through the above steps, you gained the necessary knowledge to craft a practical and transparent API schema. Treat the schema as an interface: it must be user-friendly for integrating developers and easily understood by agronomy stakeholders. Despite dealing with a large amount of complex data, it is possible to establish an organized and navigable schema taxonomy.
What did that mean in our farming world?
First, we decided to develop a standard API schema that can apply to any new computational agronomy model and support a consistent user experience across all our platform APIs.
Regarding the API requests, most of our predictive models require the same field data categories. And within any field data category, the requested data granularity will depend on the models. Take, for example, water supply: basic models may only need to know whether the crop is irrigated. In contrast, advanced ones could require a detailed log of irrigation dates, volumes and irrigation methods.
Ensuring the taxonomy of the request body aligns with the mind of an agronomist and sorts out data in a farm-centric way will ensure readability of the contract.
"fields": [
{
"location": {},
"crop": "MAIZE",
"crop_variety": {
"relative_maturity": "Early"
},
"planting": {
"date": "2022-09-10T19:00:28Z",
"density": {
"value": 4,
"unit": "plants/m2"
},
"depth": {},
"row_spacing": {},
"field_water_capacity": {}
},
"water_supply": {
"is_irrigated": true
},
"time_period": {
"forecast": {
"start_date": "2022-09-10T19:00:28Z",
"end_date": "2022-12-19T19:00:28Z"
},
"historical": {
"start_date": "2013-05-21T19:00:28Z",
"end_date": "2023-07-05T19:00:28Z"
}
}
}
]Regarding response data, our goal is to prevent data misuse. In such a domain-driven context, we should never assume that explicit labels will make the data understanding intuitive enough. To be safely understood and exposed, prediction data will often require additional metadata such as statistical values, unit descriptions, or crop-related details. Since the data content in the prediction response is very variable and model-specific, we designed high-level taxonomy rules that do not prevent broad flexibility for each API.
- We parse different types of insights into different ‘feature categories’
- We advocate for the ‘type’ / ‘value’ / ‘attributes’ taxonomy for each prediction value
While this may increase response size, the benefits of standardization, transparency, and manageable size through configuration inputs outweigh the drawbacks. Nevertheless, we are also exploring approaches to handle large prediction outputs, such as heavy time-series.
"predictions": [
{
"feature_category": "potential_yield_loss",
"features": []
},
{
"feature_category": "disease_level_current_season",
"features": []
},
{
"feature_category": "disease_level_previous_season",
"features": []
}
]"predictions": [
{
"feature_category": "growth_stage_prediction",
"features": [
{
"type": "growth_stage:ritchie_scale",
"value": "VT"
},
{
"type": "growth_stage:BBCH_scale",
"value": "59"
},
{
"type": "start_days_since_planting:mean",
"value": 69
},
{
"type": "start_days_since_planting:median",
"value": 69
},
{
"type": "start_days_since_planting:standard_deviation",
"value": 0.866
},
{
"type": "growth_stage:start_date",
"value": "2022-11-18T00:00:00Z"
}
],Document richly — Make it self-serving and trusted
For any API that deals with complex domains and aims at being commercialized, documentation is the foundation of trust and compliance. Regarding computational agronomy, no Digital Agriculture product manager would display data to growers based on superficial assumptions alone. Your API adoption is at stake, so create high-quality API documentation where the appropriate depth of description is supplied for each parameter.
Neglecting documentation quality will also create a burden on support, diverting teams from delivery and innovation. If your clients seek support to clarify contract parameters, it indicates poor documentation and calls for an upgrade. Emphasize strict adherence to this principle and actively seek early feedback directly from clients.
On top of implementing usage routines that will enhance your clients’ user experience, standardizing your API also contributes to creating consistency over the multiple API your platform is about to serve. Standardized APIs also open the door for building interoperability with other platforms or ecosystems.
Conclusion
Think of an API as an item packaging safeguarding your core product value. It serves as a tool for product identification, preservation, and convenience. On one hand, considering the ever-expanding diversity of content and tasks API should handle, a one-size-fits-all design won’t cut it. On the other hand, when used across well-defined and formalized contexts, it makes sense to establish consistency in order to optimize product usage.
First, invest time in deeply understanding the purpose of your APIs and what makes them unique. With this knowledge, you can create a tailor-made design that will elevate the value of the service provided.
Secondly, consider templatizing your industry-specific API patterns.
By treating your API design as such, your API template or standard becomes a product on its own. It will require documentation, adaptation, and ongoing evolution as it exposes and learns from more use cases. To successfully navigate these processes, your guiding principles should revolve around prioritizing client usability and effectively managing the risks associated with your product.
What are the competitive advantages you will gain for applying this mindset?
- Your clients will gain efficiency from re-usable integration routines across all your solutions.
- Smartly designed API standards will contribute to the brand and user experience differentiation of your platform.
- Standardized APIs also open the door for building interoperability with other platforms or ecosystems, fostering strong relationships with your clients.
You might also be interested in reading the following articles:
- To get the basics about APIs: https://medium.com/design-bootcamp/apis-a-breakdown-for-non-technical-product-managers-6c3d7051107d
- Another nice POV about domain-driven APIs: https://nordicapis.com/how-domain-driven-design-benefits-apis/
