Oracle RAC vs NATIVE Compilation

2 years ago we performed an upgrade of our database from an Oracle 9i single instance database towards Oracle RAC 11gR2 with Oracle Unbreakable Linux as OS. 

From the moment we started running on this new platform we noticed that the filesystem /dev/shm started growing and growing until it hit 100% and the database crashed.

When looking into this directory we noticed tons of PESLD* files. These files were generated by the NATIVE compilation mode.

So for some months we used the work-around which was provided to us :

-          shutdown the database
-          remove the PESLD* files from /dev/shm directory
-          startup the database

Our customer was not to happy with this workaround since  we are hosting a 24/7 application, so we needed to find a definitive solution for this.

At first we created a script to remove all PESLD-files which were not in use by the database :

cd /dev/shm
ls -latr PESLD_proddb*>/tmp/PESLD_Remove.tmp
cat /tmp/PESLD_Remove.tmp |cut -c51-89>/tmp/PESLD_Remove.txt

###
### Start removing files
###

while read line
do
     Found="`lsof| grep $line`"
     if [[ ${Found}X == X ]]
     then
       if [[ -f /dev/shm/$line ]]
       then
         echo mv /dev/shm/$line /tmp
         mv /dev/shm/$line /tmp
       fi
     else
       echo $line found. No action will be done.
       echo $Found
     fi
done < /tmp/PESLD_Remove.txt

But this was very dangerous and it was also no final solution.

Luckily for us, there is a way to go back to INTERPRETED mode :

  

-         Take a full backup of the database or create a restore point

-         Stop the database 

[oracle]$ srvctl stop database –d PRODDB 

-         Start-up an instance on one of the nodes in upgrade mode

[oracle]$ sqlplus / as sysdba

SQL> startup nomount

SQL> alter system set cluster_database=false scope=spfilesid='PRODDB1'; 

SQL> shutdown immediate

SQL> startup upgrade

-         Make sure that the parameter plsql_code_type has the value INTERPRETED

-         Execute the script dbmsupgin.sql. This script will switch all packages/functions,… back to INTERPRETED. When this script is finished startup the database in normal mode and run the script utlrp

[oracle]$ sqlplus / as sysdba

SQL> @?/rdbms/admin/dbmsupgin.sql

SQL> shutdown immediate

SQL> startup

SQL> @?/rdbms/admin/utlrp

-        Check if all components in DBA_REGISTRY are valid.

-        Startup the RAC-database

[oracle]$ sqlplus / as sysdba

SQL> alter system set cluster_database=true scope=spfile sid='*';

SQL> shutdown immediate

SQL> startup

[oracle]$ srvctl start database –d PRODDB