Non Disruptive DR Drills for Oracle Databases using Pure Storage ActiveDR — Part 2 of 4
Remove the pain, time, complexity and cost from your DR drills, how ActiveDR can simplify DR tests without the need to take an outage on your production systems.
This Part 2 of a 4 part series in which I’ll explore ActiveDR for managing Oracle Disaster Recovery.
Part 1 — Configure ActiveDR and Protect your DB Volumes (Why & How)
Part 2 — Accessing the DB volumes at the DR site and opening the database
Part 3 — Non Disruptive DR Drills with some simple scripting
Part 4 — Controlled and Emergency failovers/failbacks
In Part 1 we configured ActiveDR and protected the database volumes with a Pod that was replicated to the DR Site.
The raw database volumes are now protected so in the event of a disaster we have the peace of mind knowing that our data is safe at another site. But let’s not pat ourselves on the back just yet because we can’t access this data without configuring an Oracle database that can read it. So here in Part 2 we’ll do just that and please note this is a one time process for the intial DR database config.
The DR hosts (fbperf1/2) are already connected to the DR Array and host group with both of the aformentioned hosts has been created so our first step to connected the new replicated volumes to this host grooup on the DR array.
purehgroup connect --vol Oracle-DR::oracle-acrac-actdrdbs-CONTROL_REDO fbperf
purehgroup connect --vol Oracle-DR::oracle-acrac-actdrdbs-DATA fbperf
purehgroup connect --vol Oracle-DR::oracle-acrac-actdrdbs-FRA fbperfThe newly attached volumes showed up instantly on the hosts in /dev/mapper for me although on some environments it may be neccessary to re-scan the host bus adaptors
for f in `ls /sys/class/scsi_host/host?/scan`
do
echo "scanning $f" ; echo "- - -" > $f
doneFor each volume find the serial number with ls -l /dev/mapperand match this with volume serial number shown by the array to add meaningful device names into /etc/multipath.conf on all rac nodes.
multipath {
wwid 3624a937021f018170267441100011491
alias dg_acrac_actdrdbs_control_redo
}
multipath {
wwid 3624a937021f01817026744110001148f
alias dg_acrac_actdrdbs_data
}
multipath {
wwid 3624a937021f018170267441100011490
alias dg_acrac_actdrdbs_fra
}Reload the mulipath devices as root using multipath -r and the friendly names that we will use for the ASM configuration are now visible in /dev/mapper
ls -l /dev/mapper | grep actdrdbs
lrwxrwxrwx 1 root root 8 May 5 17:30 dg_acrac_actdrdbs_control_redo -> ../dm-45
lrwxrwxrwx 1 root root 8 May 5 17:30 dg_acrac_actdrdbs_data -> ../dm-46
lrwxrwxrwx 1 root root 8 May 5 17:29 dg_acrac_actdrdbs_fra -> ../dm-44I’m using the ASM Fiter Driver (As opposed to ASMLib) which is the recommended way of preparing devices for ASM these days so I’ll add this path string into the ASM Filter Driver config file so the devices can discovered by ASMFD on all nodes.
$ORACLE_HOME/bin/asmcmd dsset 'AFD:*','/dev/mapper/dg*'
$ORACLE_HOME/bin/asmcmd afd_scan
$ORACLE_HOME/bin/asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
ADR_CONTROL_REDO ENABLED /dev/mapper/dg_acrac_actdrdbs_control_redo
ADR_DATA ENABLED /dev/mapper/dg_acrac_actdrdbs_data
ADR_FRA ENABLED /dev/mapper/dg_acrac_actdrdbs_fraNote that once we have scanned for new disks asmcmd afd_scan when we list them the disk labels are already present as these were written into the header of the device on the PROD site and have bene replicated across to DR.
To mount the diskgroups so that we can register the new database instance in Oracle Clusterware we’re going to temporarily need the the Oracle-DR Pod in read/write mode which means promoting the pod… any writes hitting the PROD database will be queued up and not applied to the DR database until the POD is demoted again after we have finished configuring the database at the DR site.
As the grid user mount the diskgroups on all rac nodes.
echo "alter diskgroup ADR_CONTROL_REDO mount force;" | sqlplus -s / as sysasm
echo "alter diskgroup ADR_DATA mount force;" | sqlplus -s / as sysasm
echo "alter diskgroup ADR_FRA mount force;" | sqlplus -s / as sysasm
We will need to recreate the directory structure from the production hosts at the DR site before we can add it to the clusterware configuration. As the oracle user create the relevant directories on all rac nodes at the DR site.
mkdir -p /u01/app/oracle/local/admin/ACTDRDBS/adump
mkdir -p /u01/app/oracle/local/diag/rdbms/actdrdbs/ACTDRDBS1/cdumpNow find some of the important clusterware configuration details on the production site database using srvctl config database -d <DBNAME>
srvctl config database -d ACTDRDBS
Database unique name: ACTDRDBS
Database name: ACTDRDBS
Oracle home: /u01/app/oracle/product/db/12cR2
Oracle user: oracle
Spfile: +ADR_CONTROL_REDO/ACTDRDBS/PARAMETERFILE/spfile.270.1136027663
Password file: +ADR_CONTROL_REDO/ACTDRDBS/PASSWORD/pwdactdrdbs.256.1136027291
Domain: puresg.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: ADR_FRA,ADR_CONTROL_REDO,ADR_DATA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: oper
Database instances: ACTDRDBS1,ACTDRDBS2
Configured nodes: acrac1,acrac2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services:
Database is administrator managedWe’ll use some of these details to configure the RAC settings on the DR Site.
The DR site has a slightly different path for the ORACLE_HOME which is reflected below, also note we are using role PRIMARY as this database is not being managed by DataGuard and we are using the policy MANUAL as we want to control when the database is started rather than having RAC attempt to start it automatically because we can only start it when the volumes are read/write by way of the Oracle-DR pod being promoted
srvctl add database \
-db ACTDRDBS \
-dbname ACTDRDBS \
-oraclehome /u01/app/oracle/product/db/12.2.0 \
-spfile +ADR_CONTROL_REDO/ACTDRDBS/PARAMETERFILE/spfile.270.1136027663 \
-pwfile +ADR_CONTROL_REDO/ACTDRDBS/PASSWORD/pwdactdrdbs.256.1136027291 \
-startoption OPEN \
-stopoption IMMEDIATE \
-role PRIMARY \
-policy MANUAL \
-diskgroup ADR_FRA,ADR_CONTROL_REDO,ADR_DATA \
-dbtype RACAdd the database to the /etc/oratab file on both nodes e.g.
vi /etc/oratab
ACTDRDBS1:/u01/app/oracle/product/db/12.2.0:N
+ASM1:/u01/app/grid/product/18.0.0:NWe can now attempt to start the Database on the DR Site
srvctl start database -d ACTRDBS
The database is ready on the DR site so the final step is to create a RAC service for client sessions to connect to it
srvctl add service -db ACTDRDBS -service ACTDRDBS_DR -preferred "ACTDRDBS1,ACTDRDBS2"
srvctl start service -db ACTDRDBS -service ACTDRDBS_DRAn example TNS Connect string to connect to the datbase via this service would be
(DESCRIPTION= (TRANSPORT_CONNECT_TIMEOUT=5) (RETRY_COUNT=6) (FAILOVER=ON) (ADDRESS = (PROTOCOL = TCP) (HOST = fbperf-scan) (PORT = 1521) ) (CONNECT_DATA= (SERVER = DEDICATED) (SERVICE_NAME = ACTDRDBS_DR.puresg.com) ) )
We can now shut the database down and unmount the ASM diskgroups before demoting the Oracle-DR pod to allow the replication flow to catch-up and continue streaming writes made on the production database over to our DR Pod.
srvctl stop database -d ACTDRDBS -o immediateas the grid user
echo "alter diskgroup ADR_CONTROL_REDO dismount force;" | sqlplus -s / as sysasm
echo "alter diskgroup ADR_DATA dismount force;" | sqlplus -s / as sysasm
echo "alter diskgroup ADR_FRA dismount force;" | sqlplus -s / as sysasm
So there we have it, Our production database is being replicated and we’ve completed all the necessary configuration at the DR site to open this database on demand (without interfering with production) in a few simple steps. In part 3 we’ll put together some very simple scripts to allow for on-demand non disruptive DR-drills.
