Geek Culture
Published in

Geek Culture

Datasets in Spark

Spark has become the ubiquitous platform for data processing and has taken over the traditional MapReduce framework. In fact, some technologists would go so far as to declare MapReduce dead. Spark has been proven to outperform MapReduce by several orders of magnitude in numerous benchmarks and performance studies. Below, we briefly recount the history behind Spark’s dominance in the big data space.

In this article, we will cover the Spark Dataset in Brief

Photo by Lukas Blazek on Unsplash

Below is the definition of a Dataset from the official Databricks documentation:

“A Dataset is a strongly-typed, immutable collection of objects that are mapped to a relational schema. Datasets are a type-safe structured API available in statically typed, Spark supported languages Java and Scala. Datasets are strictly a JVM language feature. Datasets aren’t supported in R and Python since these languages are dynamically typed languages”.


Dataset is a data structure in SparkSQL which is strongly typed and is a map to a relational schema. It represents structured queries with encoders. It is an extension to data frame API. Spark Dataset provides both type safety and object-oriented programming interface.
Dataset clubs the features of RDD and DataFrame. It provides:

  • The convenience of RDD.
  • Performance optimization of DataFrame.
  • Static type-safety of Scala.

After Spark 2.0, RDD was replaced by Dataset, which is strongly-typed like an RDD, but with richer optimizations under the hood.

DataFrame and Dataset in spark

In the context of Scala we can think of a DataFrame as an alias for a collection of generic objects represented as Dataset[Row]. The Row object is untyped and is a generic JVM object that can hold different types of fields. In contrast, a Dataset is a collection of strongly typed JVM objects in Scala or a class in Java. It is fair to say that each Dataset in Scala has an untyped view called DataFrame, which is a Dataset of Row. The following table captures the notion of Datasets and DataFrames in various Spark supported languages.

Datasets are possible because of a feature called the encoder, which converts from JVM types to Spark SQL’s specialized internal (tabular) representation. Encoders are highly specialized and optimized code generators that generate custom bytecode for the serialization and deserialization of your data. Encoders are required by all Datasets. The encoder maps domain-specific types to Spark’s internal representation for that type. For instance, an Int that is one of the fields in a Row will be mapped or converted to IntegerType or IntegerType() respectively for Scala or Java and Python. Domain-specific types can be expressed as beans for Java and as case classes for Scala. The tabular representation is stored using Spark’s internal Tungsten binary format, allowing for operations on serialized data and improved memory utilization. Whilst using the Dataset API, Spark generates code at runtime to serialize a Java object into an internal binary structure and vice versa. This conversion can have a slight impact on performance, but there are several benefits to be had. For example, Spark understands the structure of data in Datasets, and it can create a more optimal layout in memory when caching Datasets.

Photo by nting xie on Unsplash

Features of Dataset in Spark

After having the introduction to dataSet, let’s now discuss various features of Spark Dataset-

Optimised Query

Dataset in Spark provides Optimized query using Catalyst Query Optimizer and Tungsten. Catalyst Query Optimizer is an execution-agnostic framework. It represents and manipulates a data-flow graph. Data flow graph is a tree of expressions and relational operators. By optimizing the Spark job Tungsten improves the execution. Tungsten emphasizes the hardware architecture of the platform on which Apache Spark runs.

Analysis at compile time

Using Dataset we can check syntax and analysis at compile time. It is not possible using Dataframe, RDDs or regular SQL queries.

Persistent Storage

Spark Datasets are both serializable and Queryable. Thus, we can save it to persistent storage.

Faster Computation

The implementation of the Dataset is much faster than the RDD implementation. Thus increases the performance of the system. For the same performance using the RDD, the user manually considers how to express computation that parallelizes optimally.

Differences with DataFrames

We’ll need to contrast DataFrames and Datasets to gain a better understanding of the two. Datasets check if types conform to the specification at compile time. DataFrames aren’t truly untyped, as their types are maintained by Spark, but the verifying whether the types conform to the specification in the schema is done at runtime. Said another way, DataFrames can be thought of as Datasets of type `Row`, which is Spark’s internal optimized in-memory representation for computation. Having its own internal representation of types allows Spark to skip JVM types that can be slow to instantiate and have garbage collection costs.

Use-cases for Datasets

It may seem redundant to have Datasets when we already have DataFrames, but there are certain scenarios in which Datasets are a better fit than DataFrames. That is, some operations can’t be expressed with DataFrames but only with Datasets. Consider also the desire for type-safety. For example, attempting to multiply two string variables in code will fail at compile-time instead of at run-time. Additionally, development may be helpful as IDEs and other tools can provide auto-complete and other hints when objects are strongly typed. Another reason to use Datasets is if all of one’s data and transformations accept case classes (Scala); it is trivial to reuse them for both distributed and local workloads.

In this article, we have scratched the surface on what Spark Datasets to offer. Hope, this is an introductory article helpful, Until next time cheers!




A new tech publication by Start it up (

Recommended from Medium

One Feature to Supercharge Your Excel in 2020

How I started my career in Data Analytics — Shantanu Kumar

Weight Initialization in ANN

COVID-19 Pharmacies Availability Alert In Tirupati

Work Without Fear Or Heroism



Setting Up Kaggle in Google Colab

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
DhineshSunder Ganapathi

DhineshSunder Ganapathi

Data Engineer, Published Author, Book Worm, Tech Blogger, Intrigued to learn new things

More from Medium

Spark Datasets take head and limit


Filling NULL values with next available data in Spark SQL — Data Engineering Series

Spark: Repartition vs Coalesce, and when to use which