Big Data File Formats Explained — Data Engineer Beginners

8 min readDec 29, 2023

JSON (JavaScript Object Notation)

JSON (JavaScript Object Notation) is a lightweight data-interchange format that is easy for humans to read and write, and easy for machines to parse and generate. JSON is often used to transmit data between a server and a web application, as well as to store configuration settings and exchange data between different programming languages.

JSON represents data as key-value pairs, similar to a dictionary or an associative array in other programming languages.
Data is organized in a hierarchical and nested structure using objects and arrays.
JSON uses a simple and readable syntax. Data is enclosed in curly braces {} for objects and square brackets [] for arrays.
Key-value pairs are separated by colons (:), and elements in an array are separated by commas.
JSON supports several data types, including strings, numbers, objects, arrays, booleans, and null.
JSON is widely used for configuration files, APIs, and as a data storage format for various applications.

Example of a simple JSON object representing information about a person:

{
  "name": "John Doe",
  "age": 30,
  "city": "New York",
  "isStudent": false,
  "hobbies": ["reading", "traveling"]
}

In the below example JSON file contains an object with a key “employees,” which maps to an array of two objects representing employee information. Each employee object has keys like “firstName,” “lastName,” “age,” and “department.” The structure of JSON allows for flexibility and ease of representation for various types of data.

{
  "employees": [
    {
      "firstName": "John",
      "lastName": "Doe",
      "age": 30,
      "department": "Engineering"
    },
    {
      "firstName": "Jane",
      "lastName": "Smith",
      "age": 28,
      "department": "Marketing"
    }
  ]
}

CSV (Comma Separated Values)

CSV is a simple and widely used file format for storing tabular data (numbers and text) in plain text form. In a CSV file, each line of the file represents a row of data, and the values within each row are separated by commas (or another delimiter)

Structure — Data is organized in rows, where each row corresponds to a record or entry.
Within each row, values are separated by commas or other delimiters (such as semicolons or tabs).
Delimiter — The comma is the most common delimiter used in CSV files, but other delimiters like semicolons or tabs may be used depending on regional conventions or specific requirements.
The choice of delimiter is important to avoid conflicts with the data itself. For example, if the data contains commas, using a comma as a delimiter may cause parsing issues.
Text Qualification — If a field value contains the delimiter or special characters, the value is often enclosed in double quotes to distinguish it from the delimiter used to separate fields. For example: "John Doe",25,"New York, NY", "Male"
Header Row — CSV files often include a header row at the beginning that contains the names of the columns. This row helps to identify the meaning of each column.

Name,Age,City,Gender

CSV files typically have a “.csv” file extension.
CSV is a platform-independent format and can be easily created and read by a variety of software applications, including spreadsheet programs like Microsoft Excel and database systems.
CSV files store data as plain text, so all values are treated as strings. It’s up to the interpreting software to recognize and handle data types appropriately.
CSV is commonly used for data interchange between different systems and applications.

Name,Age,City,Gender
John Doe,25,New York, Male
Jane Smith,30,San Francisco, Female
Bob Johnson,22,Chicago, Male

Parquet

Parquet is a columnar storage file format optimized for use with big data processing frameworks. It is designed to be highly efficient for both storage and processing of large datasets. Parquet is widely used in the Apache Hadoop ecosystem, particularly with tools like Apache Spark, Apache Hive

Columnar Storage — Unlike row-oriented storage formats, such as CSV or JSON, Parquet stores data in a columnar format. This means that values from the same column are stored together, allowing for better compression and improved query performance for analytical workloads.
Compression — Parquet uses compression techniques to reduce storage space requirements. The columnar storage format allows for effective compression because similar data types and values are grouped together.
Common compression algorithms used with Parquet include Snappy, Gzip, and LZO.
Schema Evolution — Parquet supports schema evolution, allowing changes to the data schema over time without requiring the entire dataset to be rewritten. This is beneficial for evolving data structures without significant disruptions to data processing workflows.
Predicate PushDown — Parquet enables predicate pushdown, a feature that allows the filtering of data at the storage level before it is read into memory. This minimizes the amount of data that needs to be processed, leading to improved query performance.
Metadata — Parquet files contain metadata, including schema information and statistics about the data. This metadata is used by processing engines to optimize queries and filter data efficiently.
Data Types — Parquet supports a wide range of data types, including primitive types (integers, floating-point numbers, strings, etc.) and complex types (arrays, maps, structs). This flexibility makes it suitable for diverse data processing needs.
Performance and Scalability — Due to its columnar storage and compression, Parquet is well-suited for analytical processing on large datasets. It allows for efficient scanning of specific columns and parallel processing in distributed environments.
File Extension — Parquet files typically have a “.parquet” file extension.

Example Parquet File Structure Illustration

<Column 1>
  <Value 1>
  <Value 2>
  ...
<Column 2>
  <Value 1>
  <Value 2>
  ...
...

While I can’t provide an actual binary representation of a Parquet file, I can provide a simplified example of the logical structure of a Parquet file along with some sample data. Remember that the actual binary format is more complex due to the use of advanced compression and encoding techniques.
Let’s consider a scenario where we have a dataset containing information about users, and we’ll represent this dataset using a few columns: user_id, name, age, and city.
Here’s a simplified representation of a Parquet file structure with sample data:

+-----------------------------------------------------------+
| Parquet File Header                                        |
+-----------------------------------------------------------+
| Metadata (Schema, Compression, etc.)                        |
+-----------------------------------------------------------+
| Row Group 1                                                |
|   +-----------------------------+-------------------------+
|   | user_id | name    | age | city                       |
|   +-----------------------------+-------------------------+
|   | 1       | Alice   | 25  | New York                   |
|   | 2       | Bob     | 30  | San Francisco              |
|   | 3       | Charlie | 28  | Chicago                    |
|   +-----------------------------+-------------------------+
| Row Group 2                                                |
|   +-----------------------------+-------------------------+
|   | user_id | name    | age | city                       |
|   +-----------------------------+-------------------------+
|   | 4       | Dave    | 35  | Los Angeles                |
|   | 5       | Eve     | 22  | Seattle                    |
|   +-----------------------------+-------------------------+
+-----------------------------------------------------------+

Avro

Avro is a binary serialization format developed within the Apache Hadoop project. It is designed to provide a compact and fast serialization mechanism for data exchange between systems, especially in big data processing environments

Schema-Based Serialization — Avro uses a schema to define the structure of the data being serialized. The schema is often defined in JSON format and is used to encode and decode the data.
Data Types — Avro supports a rich set of data types, including primitive types (int, long, float, double, boolean, string, bytes) and complex types (record, enum, array, map, union, fixed).
Binary Format — Avro serializes data in a compact binary format, resulting in smaller file sizes compared to some text-based formats like JSON or XML. The binary format also contributes to faster data serialization and deserialization.
Compression — Avro files can be compressed to further reduce storage requirements. Common compression algorithms, such as Snappy or deflate, can be applied to Avro data. Compression helps minimize storage costs and improve data transfer efficiency.
Self-Describing Data — Avro data files are self-describing, meaning they include the schema information along with the serialized data. This makes it easy to interpret the data without needing the schema in advance. The schema is stored at the beginning of the Avro file, allowing readers to understand the structure of the data without external schema files.
Forward and Backward Compatibility- Avro supports schema evolution, allowing for changes to the schema over time without breaking compatibility. Both forward and backward compatibility are maintained, meaning new data can be read by old readers, and old data can be read by new readers.
Language-Independent — Avro is designed to be language-independent, meaning it can be used across various programming languages. Avro schemas can be defined in JSON, and libraries for reading and writing Avro data are available in multiple programming languages, including Java, Python, C++, and more.
File Extension — Avro files typically have a “.avro” file extension

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"},
    {"name": "age", "type": "int"},
    {"name": "city", "type": "string"}
  ]
}

Avro Data

{"id": 1, "name": "Alice", "age": 25, "city": "New York"}
{"id": 2, "name": "Bob", "age": 30, "city": "San Francisco"}
{"id": 3, "name": "Charlie", "age": 28, "city": "Chicago"}

In this example, the Avro schema defines a record type named “User” with four fields. The Avro data represents instances of this record with specific values for each field.

ORC

ORC (Optimized Row Columnar) is a columnar storage file format designed for use with the Apache Hive data warehouse system. It is highly optimized for performance, especially for complex query processing in big data analytics. ORC files are often used in conjunction with Apache Hive, Apache Spark, and other big data processing frameworks.

Columnar Storage — Data is stored in a columnar format, which allows for better compression and improved query performance. This is particularly advantageous for analytical workloads where only a subset of columns is often queried.
Compression — ORC supports various compression algorithms, including Zlib, Snappy, and LZO. Compression is applied at the column level, providing efficient storage and reduced I/O.
Predicate Pushdown — ORC files support predicate pushdown, a feature that allows the filtering of data at the storage level before it is read into memory. This reduces the amount of data that needs to be processed during query execution.
Lightweight Indexing — ORC files include lightweight indexes, known as bloom filters, that help skip irrelevant data blocks during query execution. This further improves query performance.
Statistics and Metadata — ORC files store statistics and metadata about the data, including column statistics like minimum and maximum values. This information is used by query engines to optimize query execution plans.
Data Types — ORC supports a wide range of data types, including primitive types (integers, floating-point numbers, strings, etc.) and complex types (arrays, maps, structs). This flexibility makes it suitable for diverse data processing needs.
Hive Integration — ORC is closely integrated with Apache Hive, making it a popular choice for storing and processing Hive tables.

Sample dataset with information about users

| user_id | name   | age | city       |
|---------|--------|-----|------------|
| 1       | Alice  | 25  | New York   |
| 2       | Bob    | 30  | San Fran   |
| 3       | Charlie| 28  | Chicago    |

Example ORC File Structure

<Column 1: user_id>
  <Value 1>
  <Value 2>
  <Value 3>

<Column 2: name>
  <Value Alice>
  <Value Bob>
  <Value Charlie>

<Column 3: age>
  <Value 25>
  <Value 30>
  <Value 28>

<Column 4: city>
  <Value New York>
  <Value San Fran>
  <Value Chicago>

Each column is stored separately, and the values within each column are stored in a compressed, columnar format. This structure allows for efficient compression and retrieval of specific columns during query processing.

Big Data File Formats Explained — Data Engineer Beginners

** JSON (JavaScript Object Notation) **

** CSV (Comma Separated Values) **

** Parquet **

** Avro **

** ORC **