BIG DATA — ANALYTICS (abstract / study)
Translation (portuguese to english) of https://medium.com/@anagauna/big-data-analitycs-resumo-estudo-81504c6ff6b9
Above, a sample design layout drawing of a BIG DATA type computer network. This design was done using Trace-Route software used by the Cisco Networking Academy of Cisco Systems. This layout design was done by Bunyamin Önel shared this Cisco Packet Tracer scenario… (If you have #PacketTracer scenarios to share, send them over to firstname.lastname@example.org with some info and we will share them on this page) — Anastasia at #Cisco https://lnkd.in/dMJu7Yd
Below is my summary of the BIG DATA — ANALYTICS study, removing a degree of technical complication, to simplify my understanding of this subject.
DATA ANALYSIS (STATISTICS)
Statistical studies are related to situations involving planning, data collection, organization of information, analysis of information collected, interpretation and disclosure in a clear and objective manner. Research methods can be classified in two ways: opinion surveys or market research. In opinion polls, the main goal is to gather information about determining subject based on personal interviews. Market research is conducted through the market analysis of a particular product.
Collection, organization, description of data, calculation and interpretation of coefficients belong to Descriptive Statistics, while analysis and interpretation of data, associated with a margin of uncertainty, is the responsibility of the Inductive or Inferential Statistics, also called As the measure of uncertainty or methods that are based on probability theory.
The use of tables and graphs are frequent in Statistics. The tables serve to organize and tabulate the data, and the graphics convey the information with clarity and transparency, contributing to an objective reading.
DATA ANALYSIS (INFORMATION SYSTEM)
DATA ANALYSIS — What is it?
It is the process by which order, structure and meaning are given to the data (information).
It consists in transforming the collected data into useful and true conclusions and / or lessons.
From the pre-established topics, the data are processed, looking for trends, differences and variations in the information obtained.
The processes, techniques and tools used are based on certain assumptions and as such have limitations.
The process is used to describe and summarize the data, identify the relationships and differences between variables, compare variables and make predictions.
QUALITATIVE DATA ANALYSIS
Inquires why certain fact or problem is occurring;
Study the motivations;
It is inductive;
Helps to define hypotheses;
It is exploratory;
It allows to know trends, behaviors, attitudes, etc .;
Provides detailed information on trends, activities, etc;
Provides detailed information to questions or problems about a project or activity thereof;
It does not allow to infer the results to a whole population.
Describes a population sample using frequency tables with sex, age group and occupation.
Organize comments and responses in similar categories (example: concerns, suggestions, strengths, weaknesses, etc.)
Identify patterns, trends, relationships, as well as cause-effect associations.
FORMS OF DATA PRESENTATION
Narratives of participants’ responses
Diagram of relations of the various categories and their meaning given by participants
TOPICS OF THE SCRIPT AND ITS MAIN QUALITATIVE OBJECTIVES:
Identify the characteristics common to the work group, and the differences in relation to the other groups.
Inferring on the processes of socialization for the work of the co-workers and whether such processes would be associated in some way with the ways they are perceived at work today.
Identify common experiences, based on their cooperative insertion, and their impact on personal, family and social life
QUANTITATIVE DATA ANALYSIS
It studies the actions or interventions;
It is deductive;
Provides data to prove hypotheses;
It is conclusive;
Measures the level of interventions, trends, activities, etc .;
It produces quantifiable information about the magnitude of a problem but does not provide information on why the problem is occurring;
It is possible to infer the results to a whole population.
Statistical methods are used to represent the data (information)
Descriptive statistics involves: collecting data, presenting data and characterizing data, for the purpose of describing the data.
Inferential statistics involves: estimation and hypothesis testing, with the purpose of making decisions about the characteristics of a population from the sample.
STAGES OF ANALYSIS OF QUANTITATIVE DATA
Organize the data, taking care to give a logical order to the data, putting all the elements of the sample and the variables under study in a table.
Grouping and summary of data using frequency tables, eg age = (fi = ni / n)
Summary of the main statistics (variable, mean, median, fashion, standard deviation and variance)
Measures of central tendency
Measures of dispersion
Analyze and interpret data
Do a correlation analysis
ANALYSIS AND INTERPRETATION OF QUANTITATIVE DATA
Generally, data analysis and interpretation involves making comparisons of statistical quantities of the variables of interest.
The conclusions of these comparisons are based on the rejection or acceptance of hypotheses formulated during the evaluation questions.
The acceptance or rejection of hypotheses is based on the results obtained in the so-called statistical tests.
The most used tests are: T-student, Chi-Square and Anova
Baseline vs. Results Achieved: compares the situation before and after the implementation of the program.
Target Group versus Control Group: compares attitudes or practices between participants and non-participants in a program.
The treatment group — is a group of intervention participants and whose outcome measures are compared with those of a control group.
Control group — is a group of “untreated” targets that are compared with experimental groups in outcomes.
TOOLS FOR DATA ANALYSIS (information system)
Currently there are several technology companies that provide software for data analysis and treatment.
Big Data Frameworks and Platforms
The big data ecosystem includes many libraries and frameworks that interoperate with each other. Libraries usually provide solutions to specific problems; for instance, applying neural-network methods on your data. Frameworks integrate various libraries to provide even more functionality. Here are a few examples:
- Frameworks: Hadoop Ecosystem, Apache Spark, Apache Storm, Apache Pig, Facebook Presto
- Patterns: MapReduce, Actor Model, Data Pipeline
- Platforms: Cloudera, Pivotal, Amazon Redshift, Hortonworks, IBM, Google Compute Engine
The Hadoop ecosystem is complemented and surrounded by many different tools. Some of them are covered in our series of courses, such as:
- Apache Mahout: a scalable machine learning and data mining library
- Apache Pig: a high-level data-flow language and execution framework for parallel computation
- Apache Spark: a fast and general compute engine for Hadoop data. Spark provides a simple and expressive programming model that supports a wide range of applications, including Extract, Transform and Load (ETL), machine learning, stream processing, and graph computation.
Where to go for information
The company website is a good place to start when you need detailed information about a particular tool. The websites we’ve listed below provide user documentation and other sources of support. We’ve also compiled a list of recommended books for you. These are books you’re likely to find on a big data ninja’s bookshelf that you might like to borrow from your library.
DATA ANALYSIS = DATA SCIENCE
- Gregory Piatetsky-Shapiro (Analytics, Data Mining, Data Science Expert, KDnuggets President) com exemplos de Fórmula de Cálculo em Which Big Data, Data Mining, and Data Science Tools go together?
- Kirk Borne (Principal Data Scientist at Booz Allen Hamilton) em With Prescriptive Analytics, the future ain’t what it used to be
- Kirk Borne @KirkDBorne (Principal Data Scientist at Booz Allen Hamilton) em 50+ Free #DataScience Books: http://bit.ly/1rFM68K #abdsc #BigData#MachineLearning #DataMining
- Bernard Marr @BernardMarr (Advanced Performance Institute) em Big Data Possibilities — What is Big Data?
- Bernard Marr @BernardMarr (Advanced Performance Institute) em Supervised V Unsupervised Machine Learning — What’s The Difference? via @forbes
- Ronald van Loon @Ronald_vanLoon (Top10 Influencer #BigData #DataScience #IoT #Analytics) em My Brief Guide to Big Data and Predictive Analytics for non-experts | #BigData#PredictiveA… http://bit.ly/1N01U9l
- Ronald van Loon @Ronald_vanLoon (Top10 Influencer #BigData #DataScience #IoT #Analytics) em 12 Statistical and Machine Learning Methods that Every Data Scientist Should Know | #MachineLearning #DataScientis … http://bit.ly/29zeP8m
- Evan Sinar, PhD @EvanSinar (Chief Scientist & VP at Development Dimensions International (@DDIworld); Author & Top Influencer on #leadership | #dataviz | HR #analytics | #bigdata | #iot) em Awesome tutorial from @rnlanders on Natural Language Processing — check it out! http://ow.ly/iomt30at0FU #nlp #analytics #datascience
- Evan Sinar, PhD @EvanSinar (Chief Scientist & VP at Development Dimensions International (@DDIworld); Author & Top Influencer on #leadership | #dataviz | HR #analytics | #bigdata | #iot) em The seven deadly sins of statistical misinterpretation, and how to avoid them @DataScienceCtrl http://ow.ly/BvUW30ao0XR #datascience
Bibliography Recommendation Data Science = Gregory Piatetsky-Shapiro (Analytics, Data Mining, Data Science Expert, KDnuggets President) em More Free Data Mining, Data Science Books and Resources
The list below based on the list compiled by Pedro Martins, but we added the book authors and year, sorted alphabetically by title, fixed spelling, and removed the links that did not work.
- An Introduction to Data Science by Jeffrey Stanton, Robert De Graaf, 2013.
An introductory level resource developed by Syracuse University
- An Introduction to Statistical Learning: with Applications in R by G. Casella, S, Fienberg, I Olkin, 2013.
Overview of statistical learning based on large datasets of information. The exploratory techniques of the data are discussed using the R programming language.
- A Programmer’s Guide to Data Mining by Ron Zacharski, 2012.
A guide through data mining concepts in a programming point of view. It provides several hands-on problems to practice and test the subjects taught on this online book.
- Bayesian Reasoning and Machine Learning by David Barber, 2012.
focusing on applying it to machine learning algorithms and processes. It is a hands-on resource, great to absorb all the knowledge in the book.
- Big Data, Data Mining, and Machine Learning: Value Creation for Business Leaders and Practitioners by Jared Dean, 2014.
On this resource the reality of big data is explored, and its benefits, from the marketing point of view. It also explains how to storage these kind of data and algorithms to process it, based on data mining and machine learning.
- Data Mining and Analysis: Fundamental Concepts and Algorithms by Mohammed J. Zaki, Wagner Meira, Jr., Data Mining and Analysis: Fundamental Concepts and Algorithms, Cambridge University Press, May 2014.
A great cover of the data mining exploratory algorithms and machine learning processes. These explanations are complemented by some statistical analysis.
- Data Mining and Business Analytics with R by Johannes Ledolter, 2013.
Another R based book describing all processes and implementations to explore, transform and store information. It also focus on the concept of Business Analytics.
- Data Mining Techniques: For Marketing, Sales, and Customer Relationship Management by Michael J.A. Berry, Gordon S. Linoff, 2004.
A data mining book oriented specifically to marketing and business management. With great case studies in order to understand how to apply these techniques on the real world.
- Data Mining with Rattle and R: The Art of Excavating Data for Knowledge Discovery by Graham Williams, 2011.
The objective of this book is to provide you lots of information on data manipulation. It focus on the Rattle toolkit and the R language to demonstrate the implementation of these techniques.
- Gaussian Processes for Machine Learning by Carl Edward Rasmussen and Christopher K. I. Williams, 2006.
This is a theoretical book approaching learning algorithms based on probabilistic Gaussian processes. It’s about supervised learning problems, describing models and solutions related to machine learning.
Read the full post on KDnuggets: http://www.kdnuggets.com/2015/03/free-data-mining-data-science-books-resources.html
Gregory Piatetsky-Shapiro (Analytics, Data Mining, Data Science Expert, KDnuggets President)
Very interesting compilation published here, with a strong machine learning flavor (maybe machine learning book authors — usually academics — are more prone to making their books available for free). Many are O’Reilly books freely available. Here we display those most relevant to data science. I haven’t checked all the sources, but they seem legit. If you find some issue, let us know in the comment section below. Note that at DSC, we also have our free books:
- Data Science by Analyticbridge (internal to DSC, one of the first books about data science)
- Data Science 2.0 (internal to DSC, check the red-starred articles)
- 27 free data mining books
There are several sections in the listing in question:
- Data Science Overviews (4 books)
- Data Scientists Interviews (2 books)
- How To Build Data Science Teams (3 books)
- Data Analysis (1 book)
- Distributed Computing Tools (2 books)
- Data Mining and Machine Learning (29 books)
- Statistics and Statistical Learning (5 books)
- Data Visualization (2 books)
- Big Data (3 books)
Here we mention #1, #5 and #6:
Data Science Overviews
- An Introduction to Data Science (Jeffrey Stanton, 2013)
- School of Data Handbook (2015)
- Data Jujitsu: The Art of Turning Data into Product (DJ Patil, 2012)
- Art of Data Science (Roger D. Peng & Elizabeth Matsui, 2015)
Distributed Computing Tools
- Hadoop: The Definitive Guide (Tom White, 2011)
- Data-Intensive Text Processing with MapReduce (Jimmy Lin & Chris Dyer, 2010)
Data Mining and Machine Learning
- Introduction to Machine Learning (Amnon Shashua, 2008)
- Machine Learning (Abdelhamid Mellouk & Abdennacer Chebira)
- Machine Learning — The Complete Guide (Wikipedia)
- Social Media Mining An Introduction (Reza Zafarani, Mohammad Ali Abbasi, & Huan Liu, 2014)
- Data Mining: Practical Machine Learning Tools and Techniques (Ian H. Witten & Eibe Frank, 2005)
- Mining of Massive Datasets (Jure Leskovec, Anand Rajaraman, & Jeff Ullman, 2014)
- A Programmer’s Guide to Data Mining (Ron Zacharski, 2015)
- Data Mining with Rattle and R (Graham Williams, 2011)
- Data Mining and Analysis: Fundamental Concepts and Algorithms(Mohammed J. Zaki & Wagner Meria Jr., 2014)
- Mining the Social Web: Data Mining Facebook, Twitter, LinkedIn, Goo…(Matthew A. Russell, 2014)
- Probabilistic Programming & Bayesian Methods for Hackers (Cam Davidson-Pilon, 2015)
- Data Mining Techniques For Marketing, Sales, and Customer Relations…(Michael J.A. Berry & Gordon S. Linoff, 2004)
- Inductive Logic Programming: Techniques and Applications (Nada Lavrac & Saso Dzeroski, 1994)
- Pattern Recognition and Machine Learning (Christopher M. Bishop, 2006)
- Machine Learning, Neural and Statistical Classification (D. Michie, D.J. Spiegelhalter, & C.C. Taylor, 1999)
- Information Theory, Inference, and Learning Algorithms (David J.C. MacKay, 2005)
- Data Mining and Business Analytics with R (Johannes Ledolter, 2013)
- Bayesian Reasoning and Machine Learning (David Barber, 2014)
- Gaussian Processes for Machine Learning (C. E. Rasmussen & C. K. I. Williams, 2006)
- Reinforcement Learning: An Introduction (Richard S. Sutton & Andrew G. Barto, 2012)
- Algorithms for Reinforcement Learning (Csaba Szepesvari, 2009)
- Big Data, Data Mining, and Machine Learning (Jared Dean, 2014)
- Modeling With Data (Ben Klemens, 2008)
- KB — Neural Data Mining with Python Sources (Roberto Bello, 2013)
- Deep Learning (Yoshua Bengio, Ian J. Goodfellow, & Aaron Courville, 2015)
- Neural Networks and Deep Learning (Michael Nielsen, 2015)
- Data Mining Algorithms In R (Wikibooks, 2014)
- Data Mining and Analysis: Fundamental Concepts and Algorithms(Mohammed J. Zaki & Wagner Meira Jr., 2014)
- Theory and Applications for Advanced Text Mining (Shigeaki Sakurai, 2012)
- DSC Resources
- Career: Training | Books | Cheat Sheet | Apprenticeship | Certification | Salary Surveys | Jobs
- Knowledge: Research | Competitions | Webinars | Our Book | Members Only | Search DSC
- Buzz: Business News | Announcements | Events | RSS Feeds
- Misc: Top Links | Code Snippets | External Resources | Best Blogs | Subscribe | For Bloggers
- Additional Reading
- 50 Articles about Hadoop and Related Topics
- 10 Modern Statistical Concepts Discovered by Data Scientists
- Top data science keywords on DSC
- 4 easy steps to becoming a data scientist
- 13 New Trends in Big Data and Data Science
- 22 tips for better data science
- Data Science Compared to 16 Analytic Disciplines
- How to detect spurious correlations, and how to find the real ones
- 17 short tutorials all data scientists should read (and practice)
- 10 types of data scientists
- 66 job interview questions for data scientists
- High versus low-level data science
- Follow us on Twitter: @DataScienceCtrl | @AnalyticBridge
The information management big data and analytics capabilities include :
- Data Management & Warehouse: Gain industry-leading database performance across multiple workloads while lowering administration, storage, development and server costs; Realize extreme speed with capabilities optimized for analytics workloads such as deep analytics, and benefit from workload-optimized systems that can be up and running in hours.
- Hadoop System: Bring the power of Apache Hadoop to the enterprise with application accelerators, analytics, visualization, development tools, performance and security features.
- Stream Computing: Efficiently deliver real-time analytic processing on constantly changing data in motion and enable descriptive and predictive analytics to support real-time decisions. Capture and analyze all data, all the time, just in time. With stream computing, store less, analyze more and make better decisions faster.
- Content Management: Enable comprehensive content lifecycle and document management with cost-effective control of existing and new types of content with scale, security and stability.
- Information Integration & Governance: Build confidence in big data with the ability to integrate, understand, manage and govern data appropriately across its lifecycle.
Fonte: IBM.COM = http://www-01.ibm.com/software/data/bigdata/
Por: ANA MERCEDES GAUNA (13/10/2015)
Senior System Analyst | Webmaster | MCSE | MCDBA | CCNA2
Rio de Janeiro/RJ — Brazil — 27 anos de experiência profissional (CLT)