Web Mining And Data Analysis on Nonprofits Across India — Part 2

Published in

Sensewithai

7 min readJan 28, 2023

Studying the growth of nonprofits in particular sectors across handpicked states and across India

Brief

This is part 2 of the series where in part 1 we presented our findings on nonprofits data in state of Karnataka, did analysis, shared challenges and the path forward.

We are now ready with more data, all in all around 1.5 Lakh NGOs which have been registered with Ngodarpan pan India till date.

We split them sector wise and state wise to understand the growth of nonprofits through the years in more detail.

What we did ?

We collected publicly available data on all registered nonprofits which is available on the web. This came to around 1.5 lakh of them, each with details like Registration date, State, Key Issues, etc.

We chose to use the 3 main features for further analysis, namely registration date, the state of registration and key issues/ sectors in which the NGO operates.

Since most non-profits have more than one ‘key issues or sector’ mentioned in the ‘sector’ category , which were comma separated values, we first flattened the entire data to have a distribution in a way that represents nonprofits against sectors .

Next, we wanted to observe the growth of nonprofits year by year in all sectors hence data processing and grouping was done on the date of registration for all nonprofits.

We have 45 sectors all in all, which made it little difficult with analysis and interpretation hence for ease of analysis and better visualization , we chose 11 sectors out of them and moved ahead in our analysis .

We also chose few states namely — Delhi, Bihar, Uttar Pradesh, Madhya Pradesh, Rajasthan, Maharashtra, Karnataka, Tamil Nadu and Kerala.

What we found ?

Pan India Analysis [along with newly registered NGOs through periodic webcrawling and data collection and analysis]
State wise analysis

1. Pan India NGOs registered with the Indian Govt.

We mapped the yearly number of new NGO registrations throughout the country against time and found the following trend :

The cumulative growth mapped against time

Cumulative growth of nonprofts across India

We then plotted the growth to observe other factors such as speed of growth in various sectors of NGOs, co-occurrence of growth amongst sector and saw the following coming up :

Few sectors were then filtered out to do better analysis on growth of sectors through the years :

One peculiar finding was that even after efforts by NITI Aayog and govt bodies to spread awareness amongst NGOs to fill up registration details with complete data, the key issues/sector field is left blank by NGOs and this number does not seem to decrease with every passing year, the curve shows that with increasing number of NGOs registered, the ratio of NGOs filling incomplete data roughly remains constant. Are the efforts not being effective ? Or do we need further analysis to understand whats the reason for the blank field ?

Another finding as a result of our periodic intelligent crawl was that we could quickly see the newly registered NGOs on the platform. Filtering out the entire data based on year of registration gave us the exact number of NGOs registered just in January 2023 , along with details of those NGOs like name, state, key issues ,contact details etc.

Here are the details : [ some details like contact etc of NGOs have been intentionally hidden]

16 new NGO registrations in the year 2023 as on Jan 22nd 2023

Newly registered NGOs of 2023 statewise distribution

2. State wise analysis

NGO registration data was collected for following states and analysed separately to understand state specific trends and behaviour of the growth of nonprofits.

a. Delhi

i. Growth trends of NGOs — absolute numbers per year:

last 10 year numbers :
2012 : 379
2013 : 460
2014 : 544
2015 : 611
2016 : 680
2017 : 738
2018 : 832
2019 : 806
2020 : 648
2021 : 811
2022 : 541

ii. Cumulative growth of NGOs

Cumulative growth of registered nonprofits in Delhi

iii. Sector wise cumulative growth of NGOs

Sector wise growth of NGO registrations across Delhi

We observed that the lines of 2 sectors seems to coincide namely ‘Women Development’ and ‘Children’

More analysis on this such as co-occurrence and correlations would lead to greater insights into this observation. Something we plan to take up in future.

iv. Growth across selective sectors

b. Bihar

i. Growth trends of NGOs — absolute numbers per year

2005 was a peculiar year , with a sudden drop in new registrations from 188 to just 20.

2005 was a very significant year for Bihar, where a 15 year govt changed. In 2005 Bihar Assembly election, Janata Dal(United) or JD(U) and BJP won the majority of seats and JD(U) leader Nitish Kumar took over as the CM of Bihar.

But can this be attributed to the sudden drop in NGO registrations ? One of those examples when data analysis can give you the right questions and tease you to go after finding answers and dig further :)

The slowdown seems to have affected all sectors of NGOs .

Another observation which was made was the coinciding lines of 2 sectors namely ‘Women’s Development’ and ‘Vocational Training’ , are these 2 related in a way that NGOs would put both together in their “key issues” ?

pic credit : https://www.patnaonline.in/city-guide/women-welfare-organisations-in-patna

While Patna has come up with a lot of NGOs catering to training women and making them independent, it needs further data and domain analysis to conclude that the 2 coinciding lines speak of the same story !

Again, co-occurrence analysis would lead to clearer answers.

ii. Cumulative growth of NGOs