Building a Knowledge Base on Climate Impacts and Green Entrepreneurship in the Artisan Sector
In partnership with PJMF, Nest is leading the first comprehensive exploration of the impacts of climate change on artisan enterprises, their workers, and their communities. Through investments in the organization’s data infrastructure, PJMF’s Data and Society Accelerator Program is supporting an effort to collect and analyze relevant data from handcraft businesses throughout the Global South. These findings will become the foundation of an argument for greater public and private investments in these vulnerable communities to strengthen their disaster preparedness and resiliency.
By Shalini Suresh, Ashley Post, and Urvi Awasthi
For more than 16 years, Nest, a nonprofit organization, has supported the responsible growth and creative engagement of the global artisan and maker economy to advance gender equity and economic inclusion. Through programs that support the well-being of artisans in the United States and worldwide, Nest is bringing radical transparency and opportunity to the informal handworker economy. Beyond this, we are a leading advocate for this significant but under-recognized sector shining light on timely challenges and opportunities facing artisan communities around the world.
One of these challenges is the impact of climate change on craft communities, particularly those in the Global South. Artisan and handworker businesses represent a major employment sector in developing economies, the second-largest employer after agriculture, but are often not well researched or included in sustainable economic development strategies. This sector, which is conservatively estimated to employ 300 million individuals, the majority of whom are women, is uniquely positioned to inform climate solutions as many artisans live within vulnerable communities and are dependent on the land and local resources to produce their goods. Additionally, it is widely known that climate change and climate disasters disproportionately impact low-income households and communities. The majority of artisan enterprises and individuals in these supply chains live and work in low-income communities and are exposed to increased climate risks and adverse effects.
While there are countless anecdotes about how artisan businesses have experienced negative impacts on production and worker well-being from recent natural disasters across the globe, there is no comprehensive dataset highlighting these impacts on an aggregate level. Given our broad international network of artisan partners, Nest is well-positioned to lead in collecting data and information related to the impact of climate-related disasters on home-based artisans and handcraft producers. Earlier this year, PJMF awarded the organization a grant of funding and technical support to support the development of a data infrastructure that advances the systems we need to collect, retain, and analyze climate-related data appropriately.
This partnership aims to derive critical insights from Nest’s climate datasets by analyzing trends around risks to artisan enterprises and their workers, and identifying the key variables that may put a community at increased risk of adverse climate impacts to mitigate these changes. These discovery and analysis processes will also reveal innovative solutions currently being implemented by artisan enterprises to minimize their own environmental footprints and address real-time climate threats.
By performing a comparative analysis of the existing climate risks and solutions identified by Nest data, with publicly available climate-related data from the same regions, we will create an increasingly comprehensive knowledge base on the impacts of climate change in artisan communities and developing economics throughout the Global South. These findings will be publicly available and shared with key governmental, private, and public stakeholders to inform greater investment in scaling or designing new climate solutions.
Founded in 2006, Nest’s organizational growth and prominence within the artisan sector have accelerated quickly over the last 16 years. In 2015, the organization founded the Artisan & Maker Guild, a global network of handcraft enterprises. Through pro-bono membership in the Guild, artisan business leaders gain access to a library of free, expert-led resources on topics like business planning, accounting, fulfillment and logistics, marketing, and e-commerce. They may also be considered for Nest’s more intensive business development initiatives like the Artisan Accelerator or Ethical Handcraft program. Over the last seven years, the Guild has grown exponentially and now boasts more than 1,900 craft businesses across 120 countries, employing almost 300,000 individual artisans.
Nest collects a wide range of data from this diverse network of artisan businesses and individual artisans at various time points. In parallel to the growth of the network in the last few years, our process for data collection has evolved and our data sources have grown from a few sparse spreadsheets to large, dynamic databases with thousands of variables. Until our partnership with PJMF, we have relied heavily on basic tools such as Google sheets for data management and our scope of analysis has been limited due to our capacities and access to technology. As a growing non-profit, our pressing challenge is with data management, utilization, and accessibility. We collect hundreds of data points on a weekly basis however we face challenges in synthesizing the data effectively, leveraging all the data points, and deriving key highlights from exploratory analyses rather than targeted analyses. Due to the sheer size of the datasets we have and the type of methodology we have used in the past, we tend to focus on key indicators and reactive data analysis. We are eager to set up a more proactive approach to analyzing all the data we collect and we believe there is a lot more we can do with the right tools.
With this grant and technical support from PJMF, Nest is exploring the migration of our existing data architecture to a cloud-based data management solution, which will greatly expand the capabilities of our Data & Research team. As this is a large undertaking, we are starting this process with a pilot test of our climate datasets which are a smaller subset of our overall databases. Earlier this year, Nest collected the first-ever climate risk, impact, and solutions data from our beneficiaries. We are working with this dataset to derive key insights that can form a foundational evidence base on climate impacts in the handworker sector and inform future research and programming initiatives.
Our process started with the development of our tactical roadmap, an exercise that helped us map out our project goals and identify the optimal methodology and approach for the project. We contracted a data scientist to support with the technical elements of the project, such as coding for data cleaning and setting up the AWS tools, and she was able to propose a data pipeline for our ideal data architecture state, shown in the image below:
The data sources for the pilot test include the internal datasets from Nest detailing climate impact on artisans, as well as external climate data that will help map the situations of artisans to the bigger picture of climate change and gain insight into the future of the effects of climate on handworkers’ livelihood. To analyze the datasets selected and extract actionable insights, the first step we took was to consolidate the data into a data lake that stores the raw ingested data, as well as the processed data after preliminary processing steps such as binning or creating additional and inferred variables from the raw dataset. This processing was accomplished by Python scripts that have been configured to run using AWS Lambda, AWS Glue, or EC2 instances before the data is written to Amazon S3.
With the guidance of the expert data engineers at PJMF, we were able to operationalize this plan and are currently midstream in the pilot testing phase with our climate datasets. To date, we have identified several key lessons and best practices while building out this pipeline, including:
Creating effective data systems
Our storage solution of choice was AWS S3 and as we started working with S3, we realized that it was important to organize it in such a way that the climate datasets could meaningfully interact with Nest’s other extensive datasets. This is essential to draw correlational findings, for example, to understand variations in demographics among businesses that have faced different climate impacts. Simultaneously, we wanted to ensure we are leaving room for future initiatives that could give us new climate data, as well as align with public climate datasets. Hence, we wanted to make working with Lambda functions across buckets intuitive. A big takeaway was the importance of considering the future state of datasets and interactions between varied internal and external databases.
Utilizing Layers
We are using Lambda functions to process raw survey data from our beneficiaries, and as such, we had to select between A) creating a Docker Image for the data or B) using a layer with AWS Data Wrangler. We chose to go with option B) — using AWS Data Wrangler, because it already contains most of the functionalities that would be needed for any data processing. While Docker allows for 10GB of space, we did not feel this was necessary for any of the data processing environments and there were no use-cases that would require that much memory. This also allows us to get around implementing the Lambda Runtime API. Layers will be useful for sharing code between Lambda functions because they allow for packing utilities into a layer that can be used all over Lambda functions. This will be very useful for the way that the Lambda functions will be organized.
Organizing Content of Lambda Functions
Some of Nest’s primary internal datasets have been created following a specific format that allows ease of use with the tools we have been using to date, such as Google Sheets and Stata. The majority of our data cleaning and processing has been done manually, so we have had a system to download raw data, manually clean it (for example: checking data consistencies and formatting), and merge new data into the existing dataset. These processing and transformation functions will need to be replicated on our AWS platform to mirror the same formatting needs. As such, we are setting up Lambda in such a way that each major dataset will have its own dedicated Lambda function to save computation and memory, and when each dataset is updated, there will be an update to the processed data that it maps to.
As of mid-July 2022, we have set up processing tasks on AWS and are finishing up our rounds of exploratory data analysis. Our next steps are to complete this analysis and load the data into a visualization tool such as Tableau to create dashboards where key insights can be presented optimally with visuals and this platform would be used to monitor key metrics on a regular basis. In parallel, we are adapting the lessons learned from this pilot with the climate datasets to our other internal datasets, preparing for an eventual migration of our larger architecture to AWS over the next few months.
Identifying these solutions has been innovative and groundbreaking for our data systems — both with scaling our internal capacity and the potential it unlocks for our organization to share unprecedented data on the handworker sector to external stakeholders. The resources and guidance provided by the PJMF team have been instrumental in allowing us access to this architecture upgrade and we look forward to continuing to share our process and the outcomes of this project.
