OCR , OLR

 ORACLE LOCAL REGISTRY(OLR) contains node-specific information required by OHASD . Every node has its own dedicated OLR file.(not shared between the nodes).

As OCRs files are present in ASM diskgroup, While starting the CRS, it wont be able to Access OCR file to find the cluster resource information.

Because at this point ASM instance would also be down. So at this point OLR file ( which is present at normal file system) is accessed to start the required resources to start CRS

1. The default location of OLR:                  $GRID_HOME/cdata/*olr

2.View the available backups of OLR:     $GRID_HOME/bin/ocrconfig -local -showbackup

3. Take the backup of OLR manually: ( Need to done from ROOT user)

root# /crsapp/app/oracle/product/grid12c/bin/ocrconfig -local -manualbackup

 4. Verify integrity of the OLR

cluvfy comp olr

 5. Check the current OLR location:

$GRID_HOME/bin/ocrcheck -local

6. View OLR in text format:

$GRID_HOME/bin/ocrdump -local -stdout

 

OCR (Oracle Cluster Registry)

OCR will have all the cluster information. If the OCR is loss, the entire cluster will be affected. OCR is the central repository which stores the metadata, configuration, status of the clusterware resources.OCR is used to maintain application.resources and availability.It also stores configuration information for CRS daemons and clusterware managed applications.

Below are the information stored in the OCR.

·        Node membership information ex: which nodes are the part of the cluster

·        Software active version

·        Server pools

·        Status/Configuration for the cluster resources such as RAC databases, listeners, instances and services

·        ASM instance and disk groups

·        CRS application resource profiles such as VIP addresses etc..

·        Database services characteristics e.g preferred/available nodes, TAF, Load balancing, goal etc..

·        Information about clusterware processes

·        Information about interaction and management about third party applications controlled by clusterware

·        Details about network interfaces

·        Information about OCR backups

In Oracle Clusterware 11g Release 2 an additional component related to the OCR called the Oracle Local Registry (OLR)

You can view the status of the OLR file on each node by using the ocrcheck command with the –local parameter as seen here:

#ocrcheck -local

ocrdump can be used to dump the contents of the OLR to tthe text terminal:

#ocrdump -local -stdout

You can use the ocrconfig command to export and import the OLR as seen in these examples:

#ocrconfig -local -export <export file name >

#ocrconfig -local -import <file name>

And you can repair the OLR file should it become corrupted with the ocrconfig command as seen in this example:

#ocrconfig -local -repair olr <file name>

The OLR is backed up at the end of an installation or an upgrade. After that time, you can only manually back up the OLR. Automatic backups are not supported for the OLR.

To manually back up OLR:

# ocrconfig –local –manualbackup

To view the contents of the OLR backup file:

#ocrdump -local -backupfile olr_backup_file_name

To change the OLR backup location

#ocrconfig -local -backuploc new_olr_backup_path

To restore OLR:

 # crsctl stop crs

# ocrconfig -local -restore file_name

# ocrcheck -local

# crsctl start crs

$ cluvfy comp olr

Purpose of OLR

————–

It is the very first file that is accessed to startup clusterware when OCR is stored on ASM. OCR should be accessible to find out the resources

which need to be started on a node. If OCR is on ASM, it can’t be read until ASM (which itself is a resource for the node and this information is

stored in OCR) is up. To resolve this problem, information about the resources which need to be started on a node is stored in an operating system 

file which is called Oracle Local Registry or OLR. Since OLR is a file an operating system file, it can be accessed by various processes on the node

for read/write irrespective of the status of the clusterware (up/down). Hence, when a node joins the cluster, OLR on that node is read, various

resources ,including ASM are started on the node . Once ASM is up , OCR is accessible and is used henceforth to manage all the clusterware

resources. If OLR is missing or corrupted, clusterware can’t be started on that node!

Where is OLR located?

———————

The OLR file is located in the grid_home/cdata/<hostname>.olr . The location of OLR is stored in /etc/oracle/olr.loc.and used by OHASD .

What does OLR contain?

———————-

The OLR stores data about

ORA_CRS_HOME

localhost version

active version

GPnP details

OCR latest backup time and location

information about OCR daily, weekly backup location

 node name etc.

This information stored in the OLR is needed by OHASD to start or join a cluster.

A quick peek at the backup of the olr shows the resources that are being maintained.

[root@host01 ~]# ocrconfig -local -manualbackup

host01    2013/01/18 01:20:27    /u01/app/11.2.0/grid/cdata/host01/backup_20130118_012027.olr

[root@host01 ~]# strings /u01/app/11.2.0/grid/cdata/host01/backup_20130118_012027.olr |grep -v type |grep ora!

ora!drivers!acfs

ora!crsd

ora!asm

ora!evmd

ora!ctssd

ora!cssd

ora!cssdmonitor

ora!diskmon

ora!gpnpd

ora!gipcd

ora!mdnsd

 It is the very first file that is accessed to startup clusterware as/when OCR is stored on ASM.

Source : Linkedin Article

Is it possible to use OLR without OCR

 Answer is No, interesting and thought provoking 

This is introduced in Oracle 11gR2 which will have all the resource information about the node and it can't shared across nodes.

Why Oracle introduced OLR in the sense of, while starting the clusterware some of the resources needs to be started first. Before 11gR2, everything stored in OCR which is residing in OS files. But from 11gR2, OCR and voting disk Saved in ASM. So, to start the clusterware, need to start the ASM first. Without knowing resource information Oracle can't start ASM. So for that reason, Oracle is maintaining Local registry to start the resources and start the clusterware.

If OLR is missing/corrupted, the clusterware won't get started.

To answer this, Oracle introduced a component called OLR.

Ø It is the first file used to startup the clusterware when OCR is stored on ASM.

Ø Information about the resources that needs to be started on a node is stored in an OS file called ORACLE LOCAL REGISTRY (OLR).

Ø Since OLR is an OS file, it can be accessed by various processes on the node for read/write irrespective of the status of cluster (up/down).

Ø When a node joins the cluster, OLR on that node is read, various resources, including ASM are started on the node.

Ø Once ASM is up, OCR is accessible and is used henceforth to manage all the cluster nodes. If OLR is missing or corrupted, clusterware can’t be started on that node.

Oracle Local Registry is OCRs local counterpart. It stores information about the local node only, mainly related to the OHASd

Source : Linkedin Article

RAC sequence

 Sequence >>>>>>>>>

 

STOP

1. emctl stop dbconsole

2. srvctl stop listener -n racnode1

3. srvctl stop database -d RACDB

4. srvctl stop asm -n racnode1 -f

5. srvctl stop asm -n racnode2 -f

6. srvctl stop nodeapps -n racnode1 -f

7. crsctl stop crs

 

START

1. crsctl start crs

2. crsctl start res ora.crsd -init

3. srvctl start nodeapps -n racnode1

4. srvctl start nodeapps -n racnode2

5. srvctl start asm -n racnode1

6. srvctl start asm -n racnode2

7. srvctl start database -d RACDB

8. srvctl start listener -n racnode1

9. emctl start dbconsole

OCR OLR VOTING DISK ASMDISK

 We DBAs would have think of it at-least once whenever we come across RAC 11gr2.

Here are my consolidations in this regards.

GPNP failed - 
Whenever GPnP profile is not present, it is automatically copied from existing nodes by GPnPd as pending.xml . 
Whenever, profile is updated, it is automatically renamed to profile.xml.
profile.xml file is created automatically even if other nodes are down.

OLR failed -
running instance would not be impact but it would unable to start after reboot.
OLR can be import from backup (backup created at installation/upgrade times /or ocrconfig –local –manualbackup taken)

Voting Disk failed -
Node evictions. Can be restore (from automatic backed up OCR or replace with new one as it only has network/disk ping data within).

OCR failed -
cluster resources will not run. can be import from backup.
if accidentally deleted then run deleteroot.sh (All nodes),rootdeinstall.sh (master node), root.sh (master node), add other resources manually.

ASM Disk group/ASM Disk - failed
ASM disk having OCR/Voting disk/ datafiles , database will crash.
in all other cases, database would keep running with non-availability of data for affected data files.
ASM disk group (as its logical entity), Meta data can be restore using md_backup/md_restore. 
Whereas ASM disk could be create manually and its data can be restore using RMAN database backup.

Please add your inputs so that it can assist others.

if any queries, kindly feel free to buzz me here.


(Source - internet)

(Note - Purpose is to assemble widely distributed information over internet , at one place in simple language for learning purpose)

Source : Linkedin Article

Oracle RAC START/STOP sequence

 

STOP ORACLE RAC

1. emctl stop dbconsole

2. srvctl stop listener -n rac1

3. srvctl stop database -d RAC

4. srvctl stop asm -n racnode1 -f

5. srvctl stop asm -n racnode2 -f

6. srvctl stop nodeapps -n racnode1 -f

7. crsctl stop crs

check :

ps -ef | grep d.bin

 

START ORACLE RAC

1. crsctl start crs

2. crsctl start res ora.crsd -init

3. srvctl start nodeapps -n racnode1

4. srvctl start nodeapps -n racnode2

5. srvctl start asm -n racnode1

6. srvctl start asm -n racnode2

7. srvctl start database -d RAC

8. srvctl start listener -n rac1

9. emctl start dbconsole

ps -ef | grep d.bin

srvctl status database -d rac

crsctl check cluster -all

crsctl stat res -t

RAC

The Cluster Ready Services (CRS) process, which is the main clusterware process that manages all other clusterware processes and resources, and provides the communication infrastructure for the cluster.

The Cluster Synchronization Services (CSS) process, which manages the distributed lock manager and ensures that data is synchronized and consistent across all nodes in the cluster.

The Global Enqueue Service (GES) process, which manages the allocation and management of enqueues, which are used to coordinate access to shared resources in the cluster.

The Oracle Cluster Registry (OCR) process, which maintains the cluster configuration and membership information, and stores it in a central repository that is accessed by all nodes in the cluster.

The Voting Disk process, which is used to maintain quorum and ensure that the cluster can continue to operate even if some nodes become unavailable.

RAC patching is a process of updating the software that runs an Oracle Real Application Clusters (RAC) database. The high-level steps for patching a RAC database are as follows:

Download and unzip the patch files from the Oracle website.

Stop the RAC database and all related services.

Apply the patch to each node in the RAC cluster.

Start the RAC database and verify that it is functioning properly.

Run any post-patch scripts or steps that are provided with the patch.

These steps may vary slightly depending on the specific patch being applied and the version of the RAC database. It is important to carefully follow the instructions provided with the patch to ensure that the patch is applied correctly and the RAC database continues to function properly.

What is Voting disk

 

In an Oracle RAC (Real Application Clusters) system, a voting disk is a small, shared disk that is used by the cluster to store information about the current state of the cluster. This information includes the IDs of the nodes in the cluster and their current status (up or down). The voting disk is used by the cluster to determine which nodes are currently active and can participate in cluster operations.

The voting disk is an important component of an Oracle RAC system because it ensures that the cluster can continue to operate even if one or more nodes fail. This is because the remaining nodes can use the information on the voting disk to determine the current state of the cluster and take appropriate action.

In the event of a failure or corruption of the voting disk, it is important to replace the disk as soon as possible to ensure the continued operation of the cluster.

If the voting disk becomes corrupted in an Oracle RAC (Real Application Clusters) system, you should immediately shut down the database and replace the corrupted voting disk. Here are the steps to do this:

Shut down the database using the following command:

srvctl stop database -db <database_name>

Replace the corrupted voting disk with a new, uncorrupted disk. This will require access to the server and the storage system where the voting disk is located.

Once the new voting disk is in place, start the database using the following command:

srvctl start database -db <database_name>

Check the status of the database to ensure that it has started successfully and that the new voting disk is being used. You can use the following command to do this:

srvctl status database -db <database_name>

If the database has started successfully and the new voting disk is being used, you can proceed with normal operation. 

Oracle Cluster Registry (OCR) becomes corrupted

 If the Oracle Cluster Registry (OCR) becomes corrupted, it can cause serious problems for an Oracle Real Application Clusters (RAC) system. The OCR is a critical component of the clusterware stack, and it contains important configuration information and other data that is essential for the proper functioning of the cluster.

If the OCR is corrupted, it may be impossible to start the cluster or access the database, and the cluster may become unstable or unavailable. In some cases, the corruption of the OCR may even result in data loss or other serious issues.

To resolve a corrupted OCR, you will need to restore the OCR from a recent backup. This can typically be done using the ocrconfig utility, which allows you to restore the OCR from a backup file. It's important to note that restoring the OCR from a backup will overwrite any changes that have been made to the OCR since the backup was taken, so you should only use this approach if you are sure that the backup is up-to-date and contains the correct information.

In some cases, it may not be possible to restore the OCR from a backup. If this is the case, you may need to rebuild the OCR from scratch. This can be a complex and time-consuming process, and it's generally recommended to seek the guidance of a qualified Oracle DBA or other experienced professional when rebuilding the OCR.

To recover a corrupted Oracle Cluster Registry (OCR), you will need to restore the OCR from a recent backup. This can typically be done using the ocrconfig utility, which allows you to restore the OCR from a backup file. It's important to note that restoring the OCR from a backup will overwrite any changes that have been made to the OCR since the backup was taken, so you should only use this approach if you are sure that the backup is up-to-date and contains the correct information.

Here are the general steps for restoring a corrupted OCR from a backup:

1. Make sure that the clusterware stack (including the voting disk, OCR, and other components) is running on all nodes in the cluster.

2. Locate the most recent backup of the OCR. This should be a file with a .bak extension, typically located in the $ORACLE_HOME/cdata directory.

3. On one of the nodes in the cluster, use the ocrconfig utility to restore the OCR from the backup file. This can typically be done with the following command:

OCR, or Oracle Cluster Registry, is a file that stores information about the configuration of an Oracle Real Application Clusters (RAC) database. If the OCR becomes corrupted, there are a few steps you can try to recover it:

First, try using the ocrconfig utility to repair any errors in the OCR. This utility is included with the Oracle software and can be run using the following command:

$ ocrconfig -repair

If the ocrconfig utility is unable to repair the OCR, you can try restoring it from a backup. If you have a recent backup of the OCR, you can restore it using the ocrconfig utility with the -restore option, followed by the path to the backup file:

$ ocrconfig -restore /path/to/ocr_backup.bak

If you do not have a recent backup of the OCR, you can try recreating the OCR from scratch. To do this, you will need to shut down all of the Oracle RAC instances, then use the ocrconfig utility with the -delete option to delete the existing OCR:

$ ocrconfig -delete

After deleting the existing OCR, you can recreate it using the ocrconfig utility with the -config option:

$ ocrconfig -config

Once the new OCR has been created, you can restart the Oracle RAC instances and check to see if the corruption has been resolved.

It is important to note that these steps are only applicable to Oracle RAC databases. If you are using a single-instance Oracle database, you will need to use a different method to recover a corrupted OCR. Additionally, these steps are intended as a general guideline and may not work in all cases. If you are unable to recover the OCR using these steps.

What is cache Fushion in RAC ?

 Cache fusion is a mechanism used in Oracle Real Application Clusters (RAC) to enable multiple instances to access the same data in a shared cache. It is a key part of the RAC architecture and allows nodes in a cluster to access and modify data in a coordinated and consistent manner.

In a RAC system, each node has its own private cache in which it stores data that it has recently accessed from the database. When a node needs to access data that is not in its private cache, it can request the data from another node that has the data in its cache. This is known as a cache fusion operation.

Cache fusion is a complex process that involves coordinating the transfer of data blocks between the different nodes in the cluster. It uses a variety of mechanisms, such as locks and versioning, to ensure that the data being accessed is consistent and up-to-date. It also ensures that multiple nodes do not attempt to modify the same data block at the same time, which could cause data corruption.

Overall, cache fusion is an essential part of the RAC architecture and is key to the ability of RAC systems to provide high levels of performance, scalability, and availability.

What is split Brain syndrome ?

 Split brain syndrome is a term used to describe a situation that can occur in Oracle Real Application Clusters (RAC) when there is a communication failure between nodes. In a split brain scenario, the nodes in the cluster become isolated from each other and are unable to communicate or coordinate their actions.

In a RAC environment, split brain syndrome can cause serious problems if it is not addressed quickly. Because the nodes are unable to communicate with each other, they may start to act independently, which can lead to data inconsistencies and even data corruption. For example, two nodes may write to the same data block at the same time, resulting in conflicting data being written to the shared storage.

To prevent split brain syndrome, RAC uses a mechanism called I/O fencing, also known as split-brain protection. I/O fencing blocks access to shared storage by a node that is suspected to be out of sync with the rest of the cluster, effectively "fencing" it off from the rest of the cluster. This prevents the node from accessing shared data and helps to prevent data corruption and other issues that can arise from a split brain scenario.

How to resolve Split brain syndrome ?

To resolve split brain syndrome in an Oracle Real Application Clusters (RAC) system, the first step is to restore communication between the nodes. This may involve troubleshooting the network connectivity between the nodes, replacing faulty network hardware, or taking other steps to restore the communication between the nodes.

Once the communication between the nodes has been restored, the cluster manager (CRS) will automatically perform a reconciliation process to ensure that the nodes are in sync with each other and that any data inconsistencies or conflicts have been resolved. This may involve resolving any conflicts between data blocks that were modified independently by different nodes during the period of split brain.

If the split brain syndrome was caused by a failure of the cluster manager (CRS) itself, you may need to restart the CRS on all nodes in order to restore its functionality. This can be done using the crsctl start crs command.

Overall, the steps for resolving split brain syndrome in RAC will depend on the specific circumstances of the situation and the underlying cause of the problem. It's generally recommended to seek the guidance of a qualified Oracle DBA or other experienced professional when dealing with split brain syndrome.