Originally published on lakeFS blog on January 4, 2023. 2022 has been an incredible year, in the same way, that roller coasters are thrilling. Our industry has seen many shifts and rapid changes — which we experienced together. In 2022, we witnessed how data engineering teams became increasingly central to…

A typical routine of a data engineer includes writing code, choosing and upgrading the compute infrastructure, and testing both the new and changed data pipelines. Most of the time, engineers looking to test the changes they wish to apply need to run their tested pipelines in parallel to production. Every…

Data Exploration tools for Data Scientists — So, you are a data scientist, you work with data and need to explore it and run some analytics on the data before jumping into running extensive machine learning algorithms. Let’s start by examining what Serverless is. Serverless: According to Wikipedia, serverless computing is a cloud computing execution model where the…

Today, I’m excited to share a project I’ve worked on for the last couple of months. An interview-based video series: “Tech Exceptions”, highlighting Startups technical stories. In the first episode, we interview Alex Kendall, CEO of Wayve, a startup that develops artificial intelligence for autonomous vehicles capable of driving in…

To become a data engineer, there are various skills and software to learn, but the basics are the same! Today, there are multiple data engineers’ certificates and courses that you can take. …

If there is a framework that super excites me, it’s Apache Spark. If there is a conference that excites me, it’s the Spark & AI summit. This year, with the current COVID-19 pandemic, the North America version of the Spark & AI summit is online and free. No need to…

The 2019 edition of ITNEXT Summit took place on Oct 30, 2019, at Amsterdam, NL. The event had around 450+ passionate attendees that came to learn from great experts and get inspired. There were 3 tracks: JavaScript DevOps Data Engineering Each track was led and curated by an MC:

First thing first, what is TensorFrames? TensorFrames is an open source created by Apache Spark contributors. Its functions and parameters are named the same as in the TensorFlow framework. Under the hood, it is an Apache Spark DSL (domain-specific language) wrapper for Apache Spark DataFrames. It allows us to manipulate…

TensorFlow is now available on Apache Spark framework, but how do you get started? It called TensorFrame — TL;DR This is a step by step tutorial on how to get new Spark TensorFrame library running on Azure Databricks. Big Data is a huge topic that consists of many domains and expertise. All the way from DevOps, Data Engineers to Data Scientist, AI, Machine Learning, algorithm developers and many more…