A Comprehensive Approach to Efficient Data Engineering
Wix Engineering’s meetups focus on deep technical insights, whether it’s front-end, back-end, mobile, testing, or anything in between.
This time we focused on optimizing Spark with Iceberg, ensuring data quality with great expectations, and elevating code review practices using game theory with Almog Gelber, Itai Sevitt and Ziv Levy:
Using advanced Iceberg abilities with Spark pipelines — Almog Gelber
In this session, we will discuss how to use Iceberg capabilities to optimize spark pipelines and queries, improve performance with use of less resources, reduce cost and eventually run much faster.
Data Quality at Scale: Challenges and Best Practices at Wix — Itai Sevitt
In this talk, we will discuss how ‘Great Expectations’ (an Open Source data quality framework) is integrated into Wix’s data pipelines. We will focus on the use of ‘Great Expectations’ together with common frameworks like Trino, Airflow and Open Metadata to continuously test, document and ensure the quality of our data assets.
The Game Theory of Software Decision Making — Ziv Levy
In the process of building software engineering solutions, we face numerous decisions on a daily basis. These decisions can often lead to heated discussions and create a negative team atmosphere, wasting time and causing unnecessary stress. However, what if these daily decisions could be simplified?
In this talk Ziv will try to attack and eliminate the pain points of decision making in software engineering and share his personal experience and techniques of how he helped his team benefit from a lighter decision making process.
— —
For more engineering updates and insights:
- Follow us on: Twitter & LinkedIn
- Join our Telegram channel
- Subscribe to our monthly newsletter
- Subscribe to our YouTube channel
- Follow our Medium publication
- Listen to our podcast on Apple, Spotify or Google
