Big Data is also about breaking coworking silos not just data !

Big Data is it always just a buzzword ?

As the number of enterprise Big Data projects still growing year by year, I would say “No”. But someone could argue me that the market growth doesn’t necessarily mean that it’s a reality (and I agree) because sometimes business is really different from reality ( dot-com bubble crisis). Despite this fact, I think that Big Data could represent a huge opportunity for companies but not really as the same degree than enterprises could think. I will explain below .

So what is it ?

May be it’s a set of tools: Apache Hadoop ? Or as it’s name suggests a data explosion ? But it could also be a way to achieve best KPI(Key Perfomance Indicator) , access to new markets, enrich enterprise data with external data sources( social networks,blogs) , accelerate scientific research

That sounds really weird :Big Data means everything . And as we know from life experience : ”if something is everything, it also means that it’s nothing”. It’s really a big issue .

However is it really a problem?

From my opinion, it’s also a strenght . Being able to have different meaning per environment for an object, it’s not the first time that it happens in the mankind story. We have a word to qualify this kind of objects in our common vocabulary, we called it a “high level concept” aka an “abstraction”… Their power lies in the fact that everyone could build his own representation but everyone one of them are complementary .

Why ?Because all of these representations are true at the same time(each one is the reality of the owner) and that could not be ignored. The consequence is that they can exchange information more easily as they share the same keys concepts.

Think about abstract class in java , a vehicle could be a boat , a car or a plane… And each of this class could some abstract method & parameters (keys concepts).

What does it mean for Big Data ?

It means that all definitions given above are all true at the same time. The key concepts behind Big Data are Volume,Variety, Velocity also knows as the 3V and each of these has different representation depending of the actor reality:

  • Infrastructure team : 3V means a set of tool to maintain/install

But even if all these representations are different, they share key concepts that would facililate communication between every of them. No actor could improve his understand of the overall concept just by his own, it should discuss with other actors to complete his representation. That’s what I called the “convergence”.

A fisrt example

As an Computer science engineer, let’s sat i would try to explain to a marketing team: what is a RDBMS ? I should firstly explain him the keys concept behind but I would with my engineer experience. So, it would bec just a simple information transfer from me to him and not really an interesting discussion.

But if it would not be the same for an exchange about Big Data as they already have each other own definition. A real discussion could be establish and it would be possible for the two to share their respective context . It’s a simple thing but that’s awesome regarding coworking silos in entreprise.

A second example

From my personnal experience as dev engineer, working with Big Data forces me to explore other aspects of IT work like Operationnal , Analytics, Infrastructure .

Why ? Because from my point of view, you can’t really write an efficient Spark application ( I’m Big fan of Spark :) ) if you don’t care about network issues( latency/bandwith) ; storage( HDD/SSD) and so on…

What’s the impact ?

Big Data could be the meeting point between different enterprise silos as a facilitator to exchange context between co-workers.

So, a big data project is also an opportunity to experiment this convergence.Thus even if the projectfails ,it would have one success: “make co-workers share each others” and that’s great.

Thanks for the reading :).

DevSecOps engineer . Passionate about designing complex distributed systems :) .