A Comprehensive Approach to Efficient Data Engineering

Wix Engineering
Wix Engineering
Published in
2 min readApr 19, 2023

Wix Engineering’s meetups focus on deep technical insights, whether it’s front-end, back-end, mobile, testing, or anything in between.

This time we focused on optimizing Spark with Iceberg, ensuring data quality with great expectations, and elevating code review practices using game theory with Almog Gelber, Itai Sevitt and Ziv Levy:

Using advanced Iceberg abilities with Spark pipelines — Almog Gelber

In this session, we will discuss how to use Iceberg capabilities to optimize spark pipelines and queries, improve performance with use of less resources, reduce cost and eventually run much faster.

Data Quality at Scale: Challenges and Best Practices at Wix — Itai Sevitt

In this talk, we will discuss how ‘Great Expectations’ (an Open Source data quality framework) is integrated into Wix’s data pipelines. We will focus on the use of ‘Great Expectations’ together with common frameworks like Trino, Airflow and Open Metadata to continuously test, document and ensure the quality of our data assets.

The Game Theory of Software Decision Making — Ziv Levy

In the process of building software engineering solutions, we face numerous decisions on a daily basis. These decisions can often lead to heated discussions and create a negative team atmosphere, wasting time and causing unnecessary stress. However, what if these daily decisions could be simplified?

In this talk Ziv will try to attack and eliminate the pain points of decision making in software engineering and share his personal experience and techniques of how he helped his team benefit from a lighter decision making process.

— —
For more engineering updates and insights:

--

--

Wix Engineering
Wix Engineering

Published in Wix Engineering

Architecture, scaling, mobile and web development, management and more, this publication aggregates blog posts written by our very own Wix engineers. Visit our official blog here: https://www.wix.engineering/

Wix Engineering
Wix Engineering

Written by Wix Engineering

Wix Engineering. We develop innovative cloud-based web applications that influence our Wix.com 150M+ users worldwide