ExaCS: 19c Upgrade from 12c Database

In this blog, we will discuss step by steps process to upgrade the 12c database to the 19c database on Oracle Exadata Cloud Service. Before the actual upgrade makes sure you take full RMAN backup of the database and backup of init parameter, TNS_ADMIN files. Here are the main steps we need to follow to upgrade to the Oracle 19c database.

1. Validate Upgrade Steps

- Perform the upgrade in the test environment.

- Create a guaranteed restore point.

- Send downtime notifications to business users

- Check the certification of database version and boundary applications.

- Golden Gate 19.1.0.0.4 and higher are certified with Oracle Database 19c.

- APEX will not be upgraded as part of this upgrade

2. Minimum Software Requirements

For smooth upgrade keep the Exadata software tools updated.

2.1 Required Exadata Database Server Software version

Exadata Database Server (DomU) software version needs to be Exadata 19.2 or higher

If the version is less than 19.2, update the stack using the command:

dbaascli admin updateStack --version LATEST

[root@exacs-node1 tmp]# imageinfo -ver

20.1.2.0.0.201125.2

[root@exacs-node1 tmp]# /usr/local/bin/imageinfo -ver

20.1.2.0.0.201125.2

[root@exacs-node1 tmp]#

Required Grid Infrastructure version

[root@exacs-node1]# /var/opt/oracle/exapatch/exadbcpatchmulti -list_patches -oh=$(hostname -s):$(grep ^crs_home /etc/oracle/olr.loc | cut -d= -f2) |grep "gridversion detected"

2.2 Latest Cloud Tooling

Update the cloud tooling if an update is available.

[root@exacs-node1 ~]# dbaascli patch tools apply --patchid LATEST

3. Database Prerequisites

Confirm the database version and running status

[root@exacs-node1 tmp]# dbaascli database status --dbname DEVTESTC

DBAAS CLI version 22.1.1.1.0

Database Status:

Instance DEVTESTC1 is running on node exacs-node1 with online services DEVTEST_srv.testb.testt.oraclevcn.com. Instance status: Open.

Instance DEVTESTC2 is running on node exacs-node2 with online services DEVTEST_srv.testb.testt.oraclevcn.com. Instance status: Open.

Database name: DEVTESTC

Oracle Database 12c EE Extreme Perf Release 12.1.0.2.0 - 64bit Production

4. Install 19c Oracle HOME

Two ways to install the Oracle Home Software

a. 4.1 Using OCI Console

OCI Console > Click on Action Menu> Oracle Database > Bare Metal, VM and Exadata > Exadata VM Cluster > Click on Cluster Name

Click on Database Home in Resources Section

Click on Create Database Home

Select the Database Image as 19c

Click on create.

Once the software installation completes, you can check from the console about the status. Using dbaascli command you can also verify.

Command: dbaascli dbhome info

HOME_NAME=OraHome2

HOME_LOC=/u02/app/oracle/product/19.0.0.0/dbhome_2

VERSION=19.14.0.0

PATCH_LEVEL=19.14.0.0

DBs installed=

OH Backup=NOT Configured

b. 4.2 Dbaascli command to install ORACLE_HOME

1- Login as root and update the stacks

dbaascli admin updateStack --version LATEST

2- List all images

dbaascli cswlib showImages

3- [root@exacs-node1 ~]# dbaascli dbhome create --version 19.14.0.0.0

DBAAS CLI version 22.1.1.1.0

Executing command dbhome create --version 19.14.0.0.0

INFO: OSS details are not provided, Restoring from local acfs image

Please confirm yes/no:

yes

INFO: exabm=1 and oss_uri from return_exabm_url() :https://swiftobjectstorage.ca-montreal-1.oraclecloud.com

INFO : dbimage fixup executed.

Successfully downloaded DB gold image.

INFO: Logfile for this activity => /var/opt/oracle/log/ohome_createlocal

INFO : Creating Oracle Home from Local Image

INFO: Restoring home from image /var/opt/oracle/dbaas_acfs/dbnid/19.14.0.0.0/db19000_bits_EXA.zip for version 19.14.0.0.0

Oracle home installed successfully.

5. Run Pre checks on 12c Home

Prior to performing an update operation, use the --executePrereqs option to run prerequisite checks to ensure that the update can proceed. No changes are made when using the --executePrereqs option.

[root@exacs-node1 tmp]# dbaascli database upgrade --dbname DEVTEST --targetHome /u02/app/oracle/product/19.0.0.0/dbhome_8 –executePrereqs

It will generate the following pre-upgrade and post-upgrade fix scripts.

==================

PRE UPGRADE SUMMARY

==================

/u02/app/oracle/cfgtoollogs/DEVTEST_abcd/preupgrade/preupgrade.log

/u02/app/oracle/cfgtoollogs/DEVTEST_abcd/preupgrade/preupgrade_fixups.sql

/u02/app/oracle/cfgtoollogs/DEVTEST_abcd/preupgrade/postupgrade_fixups.sql

Open the pre-upgrade log file and check the failed and error actions items. Resolve the recommended actions.

Run preupgrade_fixups.sql for the auto fix.

Manual fixes need to be implemented before the upgrade.

The time zone file version error can be ignored before the upgrade. My personal experience on time zone DST errors is to ignore if your 12c DB version and 19c DB version are more than 32.

SQL> SELECT * FROM v$timezone_file;

FILENAME VERSION CON_ID

-------------------- ---------- ----------

timezlrg_34.dat 34 0

6. Perform Upgrade

You can perform the actual upgrade in two ways.

1- 6.1 Console 6.1

OCI Console > Click on Action Menu > Oracle Database > Bare Metal, VM and Exadata > Exadata VM Cluster > Click on Cluster Name > Database

From the database, list clicks on the actual DEVTEST database which needs to be upgraded.

Click on More Actions > Upgrade > Provide the database name

From here, monitor the logs or wait for ~3 hours.

2- 6.2 Using dbaascli tool

sudo su – root

nohup dbaascli database upgrade --dbname DEVTEST --targetHome /u02/app/oracle/product/19.0.0.0/dbhome_8 &

6.3 Monitor the logs

/var/opt/oracle/log/DEVTEST/dbupgrade/….

7. Post Upgrade Steps

Exadata machine is so smart that it runs the post-upgrade fixes along with the upgrade steps. So Exadata machine either using dbaascli or console method of upgrade it runs the post-upgrade script on its own.

Still, we can check the invalids, confirm the versions, and sanity check of application.

Conclusion:

Oracle made the upgrade process seamless and easy with dbaascli / console process. Even though the entire process is automated, we still get many issues as a part of an upgrade. My experience with this upgrade was initially very confusing because of the pre-upgrade recommendation. DST patch recommendation was a false alarm, as both my source and target DB had the latest DST version which is 34. And I have invested an entire day to do research about this issue and eventually, I skipped this recommendation and went ahead with the upgrade. Oracle Support like always was of no use and they take their own time to understand the issue and respond on oracle SR. So for me raising SR was waste of time. In case you struck anywhere in upgrade I would suggest starting the dbaascli way to upgrade and troubleshoot the error.

At last, don't forget to DROP the restore point and check the compatible parameter.