Elixir alias ErLang is in the programming language category since 2012. It's a dynamic, functional language designed for building scalable and maintainable applications. We recently migrated an Elixr heavy environment from a well known public cloud to Google Cloud Platform. As part of the App and Infra modernization, we have moved all of the application servers from VMs to Kubernetes(GKE clusters). In this blog, we are going to see how the elixir app is maintaining the backend PostgreSQL connections and the main bottleneck with the application-level connection pooling. Then finally how PgBouncer solved our scaling issues.
Moving to the GKE clusters:
Elixir apps are compatible…
Pgbouncer a connection pooler application that is widely used in any PostgreSQL environment. It's very lightweight but handles tons of connections. In the SRE world, now observability is playing an important role. This cool pgbouncer has some inbuilt tables(or views) to view the current status and historical stats. But it’s very difficult to run the commands via psql prompt And if we are thinking of setup monitoring then we need to put more effort to build such monitoring tools specifically for pgbouncer. Most of you heard of the Percona PMM monitoring tool. An opensource monitoring tool for MySQL, PostgreSQL, and…
GCP CloudSQL has 3 different database software which includes one of the best opensource databases PostgreSQL. The common DBA tasks like Backup, Patching, Auto Failover, and a few other things are managed by GCP. It also provides the inhouse stackdriver monitoring for PostgreSQL. But for a mission-critical databases needs a more database level metrics to fine-tune the PostgreSQL server and troubleshoot the performance issues. …
MySQL replication is one of the simplest replication methods that we ever experienced. We have done so many MySQL migrations across multiple Cloud platforms like AWS, GCP, and Azure. In AWS the MySQL migration to RDS is straight forward. We can use our favorite backup tools such as mysqldump, mydumper, Percona Xtrabackup(only for Aurora), and mysqlpump. Or we can use AWS DMS service to migrate the data. In GCP, we have very limited options in terms of using backup tools. We have to use native mysqldump which is always criticized as a Single Thread process and then all the databases…
BigQuery is the serverless data warehouse service in GCP. At the time of writing this blog, GCP introduced BigQuery Omni that will bring BigQuey to a multi-cloud platform. At Google Cloud Next’20 many features got introduced. This will definitely attract the customers to play around with the BigQuery, and migrate their data warehouse to BigQurery. Tons of migrations are already happening. But if you have tens of thousands of tables in your data warehouse, it's very difficult to create all of them manually on BigQuery. In order to make these migrations easier for ourselves and also for our customers, we…
One of our customers, India’s largest broadcast satellite service provider decided to migrate their giant IBM Netezza data warehouse with a huge volume of data(30TB uncompressed) to AWS RedShift. Along with Netezza, it was also decided that all the downstream applications also will be migrated to AWS.
Considering how critical and busy this data warehouse is, there was no room for any downtime and the whole migration had to happen seamlessly. Thanks to our experience with migrations like these, we took this challenge and recently concluded a successful migration. …
AWS RedShift is a managed Data warehouse solution that handles petabyte scale data. This blog post helps you to efficiently manage and administrate your AWS RedShift cluster. It is a managed service, but still, we have to take care of a few things to make it better. Even with Zero administration, RedShift will take care of almost everything. But you can’t get the best performance/Auditing/troubleshooting options.
What we are not covering:
- Table design
- WLM
- Best Practices
- Cost Optimization
Table of Contents:
Whether you are managing a single node Redshift cluster or a huge multi-node cluster, for the business users performance is the important thing. In RedShift, performance optimization starts from the basic table design itself. When you run into performance issues on your production cluster, you may reactively search for diagnostic queries, read through documentation to fix issues. Drawing from our experience of migrating some large enterprise data warehouses to RedShift and managing them on production, we decided to build a utility to proactively run health checks at regular intervals and generate a report with High / Medium / Low priority…
AWS RDS PostgreSQL has logical replication enabled from 9.4+ onwards. Generally in a logical replication once the subscriber is enabled, CDC will be captured in the logical replication slots and also it’ll start sync the historical data(initial sync) from the Publisher itself with the COPY process. If you have a mission-critical database on AWS RDS PostgreSQL, then the logical replication will start COPY from the Master node. This may reduce the performance of the master node. And it's not good for your business. We can’t do read from Read Replica, logical replication only works on the master node. …
Oracle databases are great but for analytics, people need some massively scalable warehouse like BigQuery. In this blog, we are going to see how to sync oracle data to BigQuery with the golden gate bigquery adapter with the initial load. Oracle golden gate has a large variety of big data adapters. You can sync from Oracle to Kafka, ElasticSearch, Hive, or Hadoop. Also, it supports GCP BigQuery as well. BigQuery is a serverless data warehouse that scales for petabytes of data. Let's see how to configure the BigQuery golden gate adapter.
Infra setup:
- Oracle 12c Enterprise
- Goldengate 12c for Oracle will be…
