Today, we will be migrating our oracle database to redshift using Database Migration Service.

Photo by Barth Bailey on Unsplash

Create your schema in the target database

Step 1: Start a new project and connect the source (oracle) and target (Redshift)

Step 2: Right click on…

Trends are moving towards self-service big data processing and analytics

Photo by Joshua Sortino on Unsplash

Big data is transforming the way we make business decisions. It allows far greater insight into our future potential. It illuminates the customer journey in ways that self-reporting doesn’t reveal. It cuts through all the BS to show us where our efforts are paying off or falling flat.

It’s becoming impossible to ignore big data and its business impacts. Big data is proving to be quite profitable. Millions of users are generating new data points each day, and plenty of data analysis tools have been developed to make sense of all the information. Data analytics is invaluable when analyzing market…

Serverless Data Lake

Step-By-Step Walkthrough on ETL Data Processing, Querying and Visualization in a Serverless Data Lake

Photo by Giorgi Shakarashvili on Unsplash

In a previous article, we created a serverless data lake for streaming data. We worked on streaming data, executed windowed functions using Kinesis Data Analytics, stored it on S3, created catalog using AWS Glue, executed queries using AWS Athena, and finally visualized it on QuickSight.

Now let’s complete our lambda architecture by introducing Batch Ingestion

A little background on Lambda Architecture

Load data from Amazon S3 to Amazon Redshift using Copy Commands

Photo by fabio on Unsplash

In this tutorial, you walk through the process of loading data into your Amazon Redshift database tables from data files in an Amazon S3 bucket from beginning to end.

In this tutorial, you’ll do the following:

  • Connect to Amazon Redshift using SQL Workbench/J.
  • Launch an Amazon Redshift cluster and create a database table and schema.
  • Use COPY commands to load the table from the data files on Amazon S3.

Create a cluster

Step 2: On the navigation menu, choose CLUSTERS, then choose Create cluster. The Create cluster page appears.

Using this walkthrough, you can learn how to migrate Oracle database to RDS PostgreSQL using AWS Data Migration Service and AWS Schema Conversion Tool (AWS SCT). In this post we will cover the following:

  • Migrate Schema using AWS SCT
  • Drop foreign keys and secondary indexes on the target database, and disable triggers.
  • Perform full load and change data capture (CDC) using AWS DMS

Create your schema in target database

Oracle to Redshift

In this tutorial, I will be going through oracle schema objects and their alternatives in Amazon Redshift that can help you during database migration.

Migrating your Oracle data warehouse to Amazon Redshift can substantially improve query and data load performance, increase scalability, and save costs. Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using your existing business intelligence tools. But not all oracle database features are supported by Redshift, so you need to look at the alternatives for them. …


Making an Amazon S3 Data Lake on Streaming Data using Kinesis, S3, Lambda, Glue, Athena and Quicksight

This article will cover the following:

  • Write a python producer which will send records to Kinesis Data Stream using KPL aggregator
  • Preprocess records using Kinesis Data Analytics Preprocessor Lambda
  • Run-time aggregation on streaming data using Kinesis Data Analytics
  • Store data in S3 and create catalog in Glue
  • Run queries in Athena and create Views
  • Import datasets in Quicksight and build charts

Note: In this article, I will go through step by step on how I built this pipeline so that anyone interested in…

Shafiqa Iqbal

Big Data Engineer. WomenTech Ambassador. Documenting the journey and sharing knowledge.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store