Missing data, missing mechanisms, and missing data profiling — Earlier this year, I started a piece on several data quality issues (or characteristics) that heavily compromise our machine learning models. One of them was, unsurprisingly, Missing Data. I’ve been studying this topic for many years now (I know, right?!) but along some projects I contribute to in the Data-Centric…

with bytewax and ydata-profiling — In this blog post, we will be covering how you can combine and leverage the open-source streaming solution, bytewax, with ydata-profiling, to improve the quality of your streaming flows. Buckle up! Stream processing enables real-time analysis of data in-flight and before storage, and can be stateful or stateless. Stateful stream…

The Top 5 Features for Efficient Data Manipulation — Due to its extensive functionality and versatility, pandas has secured a place in every data scientist’s heart. From data input/output to data cleaning and transformation, it’s nearly impossible to think about data manipulation without import pandas as pd, right? Now, bear with me: with such a buzz around LLMs over…

Best Practices, Techniques, and Tools to Fully Understand Your Data — Introduction Exploratory Data Analysis (EDA) is the single most important task to conduct at the beginning of every data science project. In essence, it involves thoroughly examining and characterizing your data in order to find its underlying characteristics, possible anomalies, and hidden patterns and relationships. This understanding of your data is…

Exploring the Streamlit App introduced in ydata-synthetic — Generating synthetic data is increasingly becoming a fundamental task to master as we move towards a Data-Centric paradigm of AI development. Synthetic Data truly has a tremendous potential, but it also comes with its own challenges, especially in what concerns fully capturing the complexity and intricacies of real-world domains, namely…

5 Open Source Python Packages for EDA and Visualization — This is a column series focusing on open-source tools for data science: each article focuses on a specific topic and introduces the reader to a set of different tools, showcasing their features with a real-world dataset. This piece focuses on data profiling and reviews ydata-profiling, dataprep, sweetviz, autoviz, and lux…

Get or create that dream job with Adam Ross Nelson — Data Science Consultant and Career Coach — The Data Crusaders is an interview series featuring domain experts and thought leaders in the field of Data Science, Machine Learning, and Artificial Intelligence conducted by me as part of the monthly activities of the Data-Centric AI Community. After my “debut crusade” with Ben Epstein, I had the pleasure to…

Navigating the complexity of imperfect data — This is a column series that focuses on data quality for data science. This constitutes the first piece and focuses on Imbalanced Data, Underrepresented Data, and Overlapped Data. Garbage in, garbage out. That is the curse of learning from data. In this piece, I’ll go over the importance of feeding…

A breakpoint with Ben Epstein — Founding Engineer at Galileo — The Data Crusaders is an interview series featuring domain experts and thought leaders in the field of Data Science, Machine Learning, and Artificial Intelligence conducted by me as part of the monthly activities of the Data-Centric AI Community. With 2023 just around the corner, and following up with a fantastic…