Great Expectations

Charles Dickens’s novel ‘’Great Expectations’’ tells the story of Philip Pirrip Jr, also known as Pip, a young orphan living with his sister Joe and her husband Joe Gargery in the southwest of England. At the beginning of the story, Pip spends his childhood in his hometown, unaware of the…

Introduction: Data validation is the process of ensuring data has undergone data cleansing to make sure it has data quality, which means that it’s both correct and useful. Data validation uses routines, often called “validation rules,” “validation constraints,” or “check routines.” …

Great Expectations is a Python package that helps data engineers set up reliable data pipelines with built-in validation at each step. By defining clear expectations for your data, it ensures that your data meets those expectations, making it more reliable and easier to work with. In today’s data-driven world, ensuring…

Ensuring Accurate and Reliable Data Pipelines with dbt-expectations Common Data Quality issues Data quality issues can lead to errors in analysis and reporting, which can affect decision-making. Common data quality issues in source data include inconsistent data formats, invalid data values, missing or incomplete data, and incorrect data types. Transformations can also introduce issues…

In today's world, data has become a crucial element for organizations and individuals in making informed business decisions. Data retrieval and generation have become common practices. The most crucial part is ensuring that our data is correct and precise, as other business decisions are entirely dependent on the same. …

The short answer is yes. The long answer is a bit more complicated. For data related processes in the current market (2023), the go-to programming language is Python. The main reason for this is the high number of available packages for data manipulation, like pandas, numpy, and more. These packages…

Introduction Building reliable and reproducible Machine Learning systems in production is a tough challenge that involves many facets beyond just the development of the ML model. Among these, data quality controls are an essential component for achieving high-quality models and predictions. Data quality or data validation analysis includes identifying data inconsistencies…

Who are we/what do we do? We’re Benji Lampel from Astronomer and Tino Pietrassyk from FactoryPal, here to tell you how FactoryPal has used Great Expectations and Airflow to ensure their factory optimization recommendations are the right ones. What is the problem we are trying to solve? FactoryPal goes into factories and optimizes their processes. The company recommends optimal values for settings on machines on…

Poor data quality can be a major issue for businesses, resulting in costly errors and incorrect decisions being made. To avoid these problems, it is important to take steps to ensure the quality of data being used. …