Non Disruptive DR Drills for Oracle Databases using Pure Storage ActiveDR — Part 3 of 4

Remove the pain, time, complexity and cost from your DR drills, learn how ActiveDR can simplify DR tests without the need to take an outage on your production systems.

In Part 3 of this 4 part series I’ll show some simple scripting to perform non disruptive DR tests for Oracle databases using ActiveDR.

Part 1 — Configure ActiveDR and Protect your DB Volumes (Why & How)
Part 2 — Accessing the DB volumes at the DR site and opening the database
Part 3 — Non Disruptive DR Drills with some simple scripting
Part 4 — Controlled and Emergency failovers/failbacks

In Part 2 we completed the DR database configuration using the replicated volumes that were configure in Part 1.

Part 3 shows how we can use some simple scripting to perform on-demand DR drills without disrupting our production site.

In my opinion this is a game changer, think about it! Zero risk DR testing that can be done anytime (goodbye Saturdays/Sundays stuck in the office) and as frequently as required. How many times have you released a new application into production without checking it’s DR process works? Full scale legacy DR failovers are just too disruptive to run each and every time we release a new application, their DR processes often go untested until the next scheduled DR drill which could be 6 months or more away. The ability to do DR drills anytime without disturbing production operations removes any uncertainty and prevents unwanted surprises should we have a real disaster situation.

Photo by Max Bender on Unsplash

Now that we have all of our ActiveDR and DB configuration in place from Parts 1 and 2 we are ready for on-demand non disruptive DR drills. The process is extremely simple and I can summarise it in 5 steps.

1. Promote Oracle-DR pod
2. Start DR database services and applications
3. Complete any DR verification tests
4. Stop DR applications and database services
5. Demote Oracle-DR pod

This can easily be done with a few clicks on the DR FlashArray GUI and by manually starting the DR database at the command line but in a real world scenario I’m going to want some automation to handle this for me so I’ve got some very simple shell scripts you can take a look at here

Disclaimer: These scripts are for example purposes only, if you wish to use them be aware that they are not supported by Pure Storage and you should add your own error checking and hardening.

The scripts and their function are described below:

dr_inserts.bash     - Simple script to repeatedly insert and view latest rows on thre DR database 
                        - This will report errors while the Oracle-DR Pod is demoted as the database is unavailable
dr_drill_start.bash - Simple script to start a DR drill by promoting the Oracle-DR Pod, Mounting ASM disk groups and Starting the DR database
dr_drill_end.bash   - Simple script to end the DR drill by Stopping the DR database, dismounting the ASM disk groups and demoting the Oracle-DR Pod

Instead of using multiple screenshots to show this DR Drill process in action it can more easily be understood with a short video.

The basic concept to understand is that we have 3 states for the Oracle-DR Pod which are all modified with a single click or CLI command.

1. Demoted — The DR Pod is receiving updates from the source pod via the replica link and the volumes in the pod are read-only.
2. Promoted — The replication is paused/queued and the volumes in the Pod are read/write allowing the database to be opened.
3. Demoted — The volumes in the pod return to read-only, they are restored to their original state at the time of being promoted (by means of hidden snapshots) and the queued replication writes from the source are applied to resynchronise the DR volumes with the latest PROD data.

The FlashArray can have multiple pods that can be used for different purposes so this is not an all or nothing approach. You could have an independent pod for each database or have multiple databases in the same pod. You might have all you SQL databases in one pod and all your Oracle databases in another pod or you have some databases that do not run in a pod at all, instead choosing to replicate them with snapshots for things like downstream reporting systems that need to be up all the time, configuring this depending on how you want to operate is flexible. Oh and did I mention ActiveDR is included in the price of every FlashArray, we don’t nickel and dime with additional charges for software features here at Pure.

So now that we’ve seen how to remove the pain and complexity of our DR drills, we’ve got our weekends back and have reached a general state of bliss, what about if we were to have a real disaster (not just a test) and we need a full failover? This we shall exlore in the final blog post of this series — Part 4.