Patching an On-Premise Exadata System to 19.2
So there are definitely Issues when trying to patch older Exadata systems to the Latest version 19.2 and above. One major issue we have encountered is that /boot was originally setup as 100mb for the really old machines like the V2 , X2-2 or X3-2 so even though they are technically capable of upgrading there is an issue that the /boot partition cannot hold two kernel images and two initramfs at the same time...
The latest systems use dracut to create their initramfs . Starting with identical systems we have seen wildly varying initramfs sizes. It may be something to do with compression but this was an issue for us.
I find its simpler to get patchmgr to tell me the correct version but running it once with a dummy version :).Lazy much. Naturally this is a sandbox machine.
[root@entsol23 dbserver_patch_19.190306]# ./patchmgr -dbnodes dbs_group -precheck -iso_repo /backup/exadata/19.2/p28802055_192000_Linux-x86-64.zip -target_version 19.2
************************************************************************************************************
NOTE patchmgr release: 19.190306 (always check MOS 1553103.1 for the latest release of dbserver.patch.zip)
NOTE
WARNING Do not interrupt the patchmgr session.
WARNING Do not resize the screen. It may disturb the screen layout.
WARNING Do not reboot database nodes during update or rollback.
WARNING Do not open logfiles in write mode and do not try to alter them.
************************************************************************************************************
2019-04-10 15:43:20 +0530 :Working: DO: Initiate precheck on 2 node(s)
2019-04-10 15:46:34 +0530 :Working: DO: Check free space and verify SSH equivalence for the root user to node(s)
2019-04-10 15:46:42 +0530 :SUCCESS: DONE: Check free space and verify SSH equivalence for the root user to node(s)
2019-04-10 15:52:09 +0530 :Working: DO: dbnodeupdate.sh running a precheck on node(s).
2019-04-10 15:53:51 +0530 :ERROR : dbnodeupdate.sh precheck failed on one or more nodes
SUMMARY OF WARNINGS AND ERRORS FOR exadb01:
exadb01: # The following file lists the commands that would have been executed for removing rpms when specifying -M flag. #
exadb01: # File: /var/log/cellos/nomodify_results.100419154316.sh. #
exadb01: ERROR: Given 'to release' (19.2) does not match found release (19.2.0.0.0.190225), exiting
2019-04-10 15:53:52 +0530 :ERROR : DONE: dbnodeupdate.sh precheck on exadb01
SUMMARY OF WARNINGS AND ERRORS FOR exadb02:
exadb02: # The following file lists the commands that would have been executed for removing rpms when specifying -M flag. #
exadb02: # File: /var/log/cellos/nomodify_results.100419154316.sh. #
exadb02: ERROR: Given 'to release' (19.2) does not match found release (19.2.0.0.0.190225), exiting
2019-04-10 15:53:52 +0530 :ERROR : DONE: dbnodeupdate.sh precheck on exadb02
2019-04-10 15:54:05 +0530 :INFO : SUMMARY FOR ALL NODES:
2019-04-10 15:53:52 +0530 :ERROR : exadb01 has state: FAILED
2019-04-10 15:53:52 +0530 :ERROR : exadb02 has state: FAILED
2019-04-10 15:54:05 +0530 :FAILED : DONE: Initiate precheck on node(s).
[INFO ] Collected dbnodeupdate diag in file: Diag_patchmgr_dbnode_precheck_100419154316.tbz
-rw-r--r-- 1 root root 1591723 Apr 10 15:54 Diag_patchmgr_dbnode_precheck_100419154316.tbz
2019-04-10 15:54:06 +0530 :ERROR : FAILED run of command:./patchmgr -dbnodes dbs_group -precheck -iso_repo /backup/exadata/19.2/p28802055_192000_Linux-x86-64.zip -target_version 19.2
2019-04-10 15:54:06 +0530 :INFO : Precheck attempted on nodes in file dbs_group: [exadb01 exadb02]
2019-04-10 15:54:06 +0530 :INFO : Current image version on dbnode(s) is:
2019-04-10 15:54:06 +0530 :INFO : exadb01: 19.1.0.0.0.181016.2
2019-04-10 15:54:06 +0530 :INFO : exadb02: 19.1.0.0.0.181016.2
2019-04-10 15:54:06 +0530 :INFO : For details, check the following files in /backup/exadata/19.2/dbserver_patch_19.190306:
2019-04-10 15:54:06 +0530 :INFO : - <dbnode_name>_dbnodeupdate.log
2019-04-10 15:54:06 +0530 :INFO : - patchmgr.log
2019-04-10 15:54:07 +0530 :INFO : - patchmgr.trc
2019-04-10 15:54:07 +0530 :INFO : Exit status:1
2019-04-10 15:54:07 +0530 :INFO : Exiting.
[root@entsol23 dbserver_patch_19.190306]#
Patchmgr syntax
/backup/Exadata/19c/dbserver_patch_19.181025/patchmgr -dbnodes /backup/Exadata/19c/dbs_group -precheck -yum_repo http://entsol23/repo -target_version 19.1.0.0.0 -ignore_alerts
Usage:
./patchmgr -dbnodes <database_node_list_file>
-backup [-rolling] [-unkey] |
-precheck (yum_repo <baseurl> | -iso_repo <zipped_iso_file>) [-target_version version] [-unkey] |
-upgrade (yum_repo <baseurl> | -iso_repo <zipped_iso_file>) [-target_version version] [-rolling] [-unkey] |
-rollback [-rolling] [-unkey]
-cleanup [-unkey]
MANDATORY PARAMETER
-dbnodes db_nodes_list_file
Specifies the name of the database node(s) list file. The file has one
hostname or ip per line.
ACTIONS
-backup
Perform backup only the database node(s) specified in the host list.
Runs backup in non-rolling fashion unless -rolling option is specified.
-precheck
Runs the pre-upgrade validation checks in the database node(s)
specified in the host list in non-rolling fashion.
-upgrade
Upgrades the database node(s) specified in the host list.
Runs upgrade in non-rolling fashion unless -rolling option is specified .
-rollback
Rollback the database node(s) specified in the host list.
Runs rollback in non-rolling fashion unless -rolling option is specifie d.
-cleanup
Cleans up all temporary content on the database node(s)
specified in the host list in non-rolling fashion.
-get "property"
Get information about specific property. Property can be:
- log_dir - assigned directory for all log files.
MAIN OPTIONS
-allow_active_network_mounts
Allow to run db node update with active NFS or/and SMB mounts.
This option is supported starting release 12.1.2.1.1
-no_connection_draining
Disables RHP database connection draining.
Connection draining availibilty depends on Grid Infrastructure home ver sion.
Option only applicable for rolling updates.
-force_remove_custom_rpms
Remove any custom rpms when db node update goes from OL5 to OL6.
-ignore_alerts
Ignore any active hardware alerts on the Exadata cell and proceed with
the patching.
-iso_repo
Zipped ISO image file for the Exadata update repository.
This option must be specified for -backup, -precheck and -upgrade actio ns.
This options cannot be combined with -yum_repo option.
-log_dir absolute_path_to_log_directory or keyword auto
- Absolute path to the log directory or
- auto - generate and set path to the log directory that is based on
launch directory and content of nodes list file.
Note. Specifying of -log_dir enables multiple patch manager invocations
and enables to run patch manager as non-root.
-nobackup
Do not perform backup on the database node(s) for an upgrade action.
-modify_at_prereq
Rpm changes at prerequisite check.
-nomodify_at_prereq
No rpm changes at prerequisite check (default behavior).
-rolling
Applies the backup, upgrade or rollback in rolling fashion, one
database node at a time.
-rolling_backups
Directs patchmgr to backup each node prior to updating that node in a
rolling manner. If this option is not specified (default), then
patchmgr completes the backups of all nodes in parallel before
updating the first node.
This option can only be used in conjunction with the -rolling option.
Otherwise, this option is ignored and the default backup method is used .
This option will be ignored if -nobackup is selected.
-skip_gi_db_validation
Skip certification of Grid Infrastructure/Database home compatibility with Oracle Linux 7.
-target_version
The patch version like 12.1.2.3.0.160207.3.
-unkey
Removes passwordless SSH access to the database node(s) before exit.
-yum_repo
Base URL for the Exadata update repository.
This option must be specified for -backup, -precheck and -upgrade actions.
This options cannot be combined with -iso_repo option.
MAIL NOTIFICATION OPTIONS
-smtp_from "addr"
The address patchmgr notification sent from.
sendmail daemon must be started to send email notification.
-smtp_to "addr1 addr2 addr3 ..."
The address(es) patchmgr notification sent to.
sendmail daemon must be started to send email notification.
-smtp_set_envelope_sender
Use the same from address in Return-Path: mail header.
HELP OPTION
-help
Displays this screen
SUMMARY OF WARNINGS AND ERRORS FOR exadb01:
exadb01: # The following file lists the commands that would have been executed for removing rpms when specifying -M flag. #
exadb01: # File: /var/log/cellos/nomodify_results.121118193316.sh. #
exadb01: ERROR: Detected open hardware alert(s) on the Exadata Database Server. Alerts should be resolved before proceeding the update. Optionally use -ignore_alerts if running from patchmgr.
exadb01: ERROR: Database home /u01/app/oracle/product/11.2.0/dbhome_1 is at 11.2.0.3.0 but needs to be at 11.2.0.3.28 with Grid Infrastructure 12.1.0.2.0 or later to be certified for Oracle Linux 7.
exadb01: ERROR: Database home /u01/app/oracle/product/12.1.0.2/dbhome is at 12.1.0.2.0 but needs to be at 12.1.0.2.180831 to be certified for Oracle Linux 7.
exadb01: ERROR: Custom packages found that require removal before updating to another major Oracle Linux release, see exadb01:/var/log/cellos/unkown_packages-rpt.121118193316.txt for more details.
exadb01: The following known issues will be checked for but require manual follow-up:
exadb01: (*) - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
exadb01: (*) - DBFS mount scripts may fail after a major os upgrade
2018-11-12 19:53:40 +0530 :ERROR : DONE: dbnodeupdate.sh precheck on exadb01
SUMMARY OF WARNINGS AND ERRORS FOR exadb02:
exadb02: # The following file lists the commands that would have been executed for removing rpms when specifying -M flag. #
exadb02: # File: /var/log/cellos/nomodify_results.121118193316.sh. #
exadb02: ERROR: Detected open hardware alert(s) on the Exadata Database Server. Alerts should be resolved before proceeding the update. Optionally use -ignore_alerts if running from patchmgr.
exadb02: ERROR: Database home /u01/app/oracle/product/11.2.0/dbhome_1 is at 11.2.0.3.0 but needs to be at 11.2.0.3.28 with Grid Infrastructure 12.1.0.2.0 or later to be certified for Oracle Linux 7.
exadb02: ERROR: Database home /u01/app/oracle/product/12.1.0.2/dbhome is at 12.1.0.2.0 but needs to be at 12.1.0.2.180831 to be certified for Oracle Linux 7.
exadb02: ERROR: Custom packages found that require removal before updating to another major Oracle Linux release, see exadb02:/var/log/cellos/unkown_packages-rpt.121118193316.txt for more details.
exadb02: The following known issues will be checked for but require manual follow-up:
exadb02: (*) - Yum rolling update requires fix for 11768055 when Grid Infrastructure is below 11.2.0.2 BP12
exadb02: (*) - DBFS mount scripts may fail after a major os upgrade