Why we need Azure Data Lake Store

In this article, I’ll explain about why we need Azure Data Lake Store. And what is it’s significance when we’re analyzing data and getting insights from it. I’ve also created a pictorial representation of Data Lake’s role in the whole process below.

Big Data Analytics

In today’s world we all have so many smart devices like smart phones and smart watches. Almost everyone today have them and some people may have more than one, so there are many devices and each of them generates data all the time. So you can imagine that there’s a lot of data. And we can surely get valuable insight by processing it using services like Spark or Hadoop. And that’s what Big Data Analytics is.

Now you can see in above image at first we have lot’s of devices that are generating data but we can’t directly perform processing over this data. First we need to store it at some place. Once all this generated data is at some centralized place we can then perform analysis over it.

So How exactly Azure Data Lake Store is Related with this?

For this centralized store we have Azure Data Lake Store. It’s a cloud based file system that can store any data of any format of any size. This the reason why we have Data Lake Store. It is integrated with Azure Active Directory so there is security and it can easily be integrated with other services like Hadoop, Spark, HDInsight cluster or it’s complementary Azure Data Lake Analytic that can process all the data and we can get valuable insight.

As I’ve shown in diagram, we can then connect services to Data Lake and it will generate some output.

Later we can also integrate service like PowerBI that will catch our generated output data and give visual insight in it.