11R2 grid infrastructure gotcha

Version: Oracle Clusterware active version on the cluster is [11.2.0.1.0]

Issues with 11R2 itself

1. ASM diskgroup shows offline after reboot.

SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */

SUCCESS: ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */

WARNING: failed to online diskgroup resource ora.DATADG.dg (unable to communicate with CRSD/OHASD)

WARNING: failed to online diskgroup resource ora.RECOVERYDG.dg (unable to communicate with CRSD/OHASD)

#check status

$GRID_HOME/bin/srvctl status diskgroup -g diskgroupname -a

#To start or sync diskgroup resource:

$GRID_HOME/bin/srvctl start diskgroup -g diskgroupname

upgrade gotchas

1. try to use same user as old version RAC, do NOT try use different user. The upgrade won't success, and have to do a lot troubleshooting to bring RAC to 11R2.
2. If you did use different users, the following are things you need look into and change.
   a. rootUpgrade.sh will fail, then you have to deconfigure it ($GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force)
   b. restore all files belong to old RAC (Restore original /etc/init.d/initcss initcrs

   Restore /var/opt/oracle/

   Restore old version crshome

   c. run rootUpgrade.sh again
   d. this will upgrade your RAC to 11R2, but there are more things need be fixed.
   e. fix ASM by remove asm (srvctl remove asm) you need remove all database and related service first. add asm back(srvctl add asm), then add database and related service back. REF:

   ASMCA Fails When Upgrading to 11.2 due to Different ASM/CRS User or Non-default ASM Name [ID 1113073.1]

   
   f.fix nodeapps (mainly because the ownership not right) srvctl stat res -p > res.log
   ## workaround for nodeapps

cd $grid_home

find . -name log | xargs chmod 775

chmod g+w /u01/app/11.2.0/grid/opmn/logs/

chmod g+w /u01/app/11.2.0/grid/eons/init

chmod -R g+w /u01/app/11.2.0/grid/opmn/conf/

chmod g+w /u01/app/11.2.0/grid/network/admin

chmod g+w /u01/app/11.2.0/grid/network/admin/*

chmod -R g+w /u01/app/11.2.0/grid/ons/

h. fix TNS_ADMIN create a softlink from $GRID_HOME/network/admin to /var/opt/oracle