Building the Ultimate Trading Data Pipeline — Part 3: Data pipeline for populating daily historical data

In this article we’re going to create a pipeline to extract historial daily data from EOD HD API and store it using Timescale.

TimescaleDB is an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL and packaged as a PostgreSQL extension, providing automatic partitioning across time and space (partitioning key), as well as full SQL support.

TimescaleDB is an extension to PostgreSQL and one of the biggest benefits is that we can use regular PostgreSQL tables (we can create all tables we did in the previous articles) alongside the specialist time-series tables.

Install TimeScaleDB

Prerequisites: Docker installed.

Use this docker-compose.yaml:

version: '3.1'
services:  
  timescaledb:  
    image: timescale/timescaledb-ha:pg14-latest  
    environment:  
      - POSTGRES_DB=trading-data-db  
      - POSTGRES_USER=<user>  
      - POSTGRES_PASSWORD=<password>
    ports:  
      - "5432:5432"  
    volumes:  
      - timescaledb-ha:/home/postgres/pgdata/data  
volumes:  
    timescaledb-ha:  
      driver: local # Define the driver and options under the volume name…