Mapreduce

As a Data Engineer, I have developed a lot of complex Hive queries for huge datasets. Since Hive is built on an Apache Hadoop framework, it comes with its own set of analytical functions and optimization techniques such as partitioning, bucketing, compression, denormalization, vectorization, etc. …

Consider you own and work in a carpenter workshop. You design and sell varieties of chairs and tables for offices across the states. Now you have received a big order from a XYZ firm for 1000 chairs. You start to work towards preparing the order. You first procure the raw…

Imagine you work in stock exchanges suppose NSE. Your boss comes to you and gives you 1 terabyte of data which has the information of each stock traded in the exchange. He asks you to filter out the data of each stock with the maximum value traded over the period…

MapReduce is a component of the Apache Hadoop ecosystem, a framework that enhances massive data processing. Other components of Apache Hadoop include Hadoop Distributed File System (HDFS), Yarn, and Apache Pig. The MapReduce component enhances the processing of massive data using dispersed and parallel algorithms in the Hadoop ecosystem. …

A complete overview on Apache Hadoop — This article is born from my desire to share some notes taken during the Big Data Specialization course offered by University of California San Diego on Coursera (here the link). In my opinion the first course, called Introduction to Big Data, might be a bit boring for those already in…

I’m a Data Science grad student at UW La Crosse and my final project in my big data class was to find a large dataset, pick it apart, and return interesting results. Easy right? Finding a dataset that was both interesting and large was a bit tricky for me. I…

Ölçeklenebilirlik, Artan iş yükünüz ile nasıl başa çıkacaksınız sorusunun cevabıdır. İş yükümüz en çok hangi katmanda birikiyor, yada bu iş yükünü eritmek için çalışırken nelerden ödün verebiliriz gibi sorularla sistemimize en uygun ölçeklenebilir sistem tasarım paternini bulabiliriz. Sık kullanılan ölçeklenebilir sistem tasarım paternlerine birlikte göz atalım. Load Balancer En sık…

In this Article , We are going to see basic introduction about Aggregation Framework of MongoDB and Map- Reducer Function. What is MongoDB? MongoDB is an open source NoSQL database management program. NoSQL is used as an alternative to traditional relational databases. NoSQL databases are quite useful for working with large sets of…

Reading, writing and partitioning in Spark — Let’s say you need to filter sentences of a file that contain a particular word. File needs to fit in the disk to process it. Processing power of the machine is limited to the number of cores the machine has. What if the file is bigger than your disk space…