On-Premise to GCP Data Ingestion Pattern and Data Analytic Pipeline
Discover an Invaluable Practice Course to Elevate Your Success in Any Certification!
Pattern-1
Ingest data to GCP when On-Prem system provides REST end point for your data.
Your OpenShift and/or GKE services can hit that REST end point to get the data. This data can land in BigQuery directly or via PubSub depending upon the business case and architectural requirements.
Pattern-2
Ingest data to GCP when On-Prem system data in Databases like SQL Server, MySQL, Oracle etc…
Your OpenShift and/or GKE services can hit that REST end point to get the data. This data can land in BigQuery directly or via PubSub depending upon the business case and architectural requirements.
Pattern-3
Ingest data to GCP when On-Prem data is in files.
One way of getting these data in files is using Storage Transfer Service. This data is can land in GCS and post ETL processing using Fusion or Dataprep, it can be stored in data ware house — BigQuery.
Pattern-4
Ingestion of on-premise data when data in SFTP server.
Data Analytics Pipeline to ingest structured and unstructured data from anywhere to GCP.
