In-Service Software Upgrade

Information About In-Service Software Upgrade

In-Service Software Upgrade (ISSU) is a procedure to upgrade a wireless controller image to a later release while the network continues to forward packets. ISSU helps network administrators avoid a network outage when performing a software upgrade.

ISSU can also be used to apply cold patches without impacting the active network.

ISSU is supported only on the following Cisco Catalyst 9800 Series Wireless Controllers, and supports only upgrade.

  • Cisco Catalyst 9800-80 Wireless Controller

  • Cisco Catalyst 9800-40 Wireless Controller

  • Cisco Catalyst 9800-L Wireless Controller

  • Cisco Catalyst 9800-CL Wireless Controller (Private Cloud)

High-Level Workflow of ISSU

  1. Onboard the controller software image to the flash memory.

  2. Download the AP image to the AP.

  3. Install the controller software image.

  4. Commit the changes.

Prerequisites for Performing In-Service Software Upgrade

  • Ensure that both Active and Standby controllers are in install mode and are booted from bootflash:/packages.conf.

  • Ensure that the network or device is not being configured during the upgrade.

  • Schedule the upgrade when your network is stable and steady.

  • Ensure uninterrupted power supply. A power interruption during upgrade procedure might corrupt the software image.

Guidelines and Restrictions for In-Service Software Upgrade

  • If you do not run the install commit command within 6 hours of the install activate issu command, the system will revert to the original commit position. You can choose to delay the commit using the Delayed Commit procedure.

  • During ISSU upgrade, while AP rolling upgrade is in progress, the install abort command won’t work. You should use the install abort issu command, instead to cancel the upgrade.

  • During ISSU upgrade, the system displays a warning message similar to: 
    found 46 disjoint TDL objects
    . You can ignore the warning message because it doesn’t have any functional impact.

  • During ISSU upgrade, if both the controllers (active and standby) have different images after the power cycle, an auto cancel of ISSU is triggered to bring both the controllers to the same version. The following is a sample scenario: Install Version1 (V1) software on the active controller and then apply a SMU hot patch and perform a commit. Now, upgrade the software to Version2 using ISSU, and then power cycle the active controller. At this point, the system has a version mismatch (V1 and V2). The active controller reloads at this stage, after the completion of bulk synchronization. Now, both the controllers come up with the same version (V1 and V1).

  • An ISSU upgrade that is canceled because of configuration synchronization failure on the standby controller rolls back to V1 of the software image. However, this information isn’t available in the show install command log. Run the show issu state detail command to see the current ISSU state.

  • To enable the clear install command, you should first run the service internal command in global configuration mode, and then run the clear install command in privileged EXEC mode.

  • Image rollback could be affected if the controller has a stale rollback history and the stack gets formed afterwards. We recommend that you run the clear install state command to clear stale information and boot the controller in bundle mode.

  • The clear install state command doesn’t delete the SMU file from flash or storage. To remove a SMU, use either the install remove file command or the install remove inactive command.

  • When the new active controller comes up, after the image upgrade, it doesn’t retain the old logs on web GUI window as part of show logs.

  • If a stateful switchover (SSO) or a high-availability (HA) event occurs during the rolling AP upgrade procedure of the ISSU feature, the rolling AP upgrade stops. You should then use the ap image upgrade command to restart the upgrade process.

  • If HA fails to form after the ISSU procedure, you should reload any one chassis again to form HA again.

  • Use clear ap predownload statistics command before using the show ap image command. This ensures that you get the right data after every pre-download.

  • Manually cancel the ISSU process using the install issu abort command in the scenarios given below, to avoid a software version mismatch between the active controller and the standby controller.

    • An RP link is brought down after standby HOT during an ISSU procedure and the links remains down even after the auto-abort timer expiry.

    • An RP link is brought down before the standby controller reaches standby HOT during an ISSU procedure.

  • Cisco TrustSec (CTS) is not supported on the RMI interfaces.

  • If a switchover occurs while performing an AP upgrade using ISSU, the upgrade process will restart automatically after the switchover.

Upgrading Software Using In-Service Software Upgrade

Use the following procedure to perform a complete image upgrade, that is, from one image to another.


Note

ISSU is supported only within and between major releases, for example, 17.3.x to 17.3.y, 17.6.x to 17.6.y (within a major release) and 17.3.x to 17.6.x, 17.3.x to 17.9.x (among major releases), that is, for two releases after the current supported release. ISSU is NOT supported within and between minor releases or between minor and major releases, for example 17.4.x to 17.4.y or 17.4.x to 17.5.x or 17.3.x to 17.4.x.

ISSU downgrade is not supported for Cisco Catalyst 9800 Series Wireless Controller platforms.

Note

We recommend that you configure the percentage of APs to be upgraded by using the ap upgrade staggered command.

Procedure

  Command or Action Purpose

Step 1

install add file file-name

Example:

Device# install add file <>

The controller software image is added to the flash and expanded.

Note

 

In Cisco Catalyst 9800 Wireless Controller for Switch, run the install add file sub-package-file-name command to expand the wireless subpackage file.

Step 2

ap image predownload

Example:

Device# ap image predownload

Performs predownload of the AP image.

To see the progress of the predownload, use the show ap image command.

Step 3

install activate issu [ auto-abort-timer timer]

Example:

Device# install activate issu

Runs compatibility checks, installs the package, and updates the package status details.

Optionally, you can configure the time limit to cancel the addition of new software without committing the image. Valid values are from 30 to 1200 minutes.

Step 4

Run either of the following commands:

  • install abort issu 
    Device# install abort issu

    Cancels the upgrade process and returns the device to the previous installation state. This is applicable for both controller and the AP.

  • install commit 
    Device# install commit

    Commits the activation changes to be persistent across reloads.

    Note

     

    If you do not run the install commit command within 6 hours of completing the previous step, the system will revert to the original commit position.

Upgrading Software Using ISSU (GUI)

Before you begin

  1. The device should be in Install mode.

  2. The device should have an HA pair. The standby controller should be online and is in SSO mode.

    You can verify the details using show issu state detail command.

Procedure

Step 1

Choose Administration > Software Management.

Step 2

Under the Software Upgrade tab, check the ISSU Upgrade (HA Upgrade) (Beta) check box.

Step 3

In the AP Upgrade Configuration section, from the AP Upgrade per Iteration drop-down list choose the percentage of APs to be upgraded.

Step 4

Click Download & Install.

This initiates the upgrade process and you can view the progress in the Status dialog box.

Click the Show Logs link to view the upgrade process details.

Note

 

An SSO takes place while activating the image on the active controller. After the SSO, you should login again to the controller.

Step 5

The system enables the Commit and ISSU Abort buttons after the upgrade.

Click Commit to commit the activation changes, or ISSU Abort to terminate the upgrade process and return the device to the previous installation state.

Upgrading Software Using In-Service Software Upgrade with Delayed Commit

Use this procedure to upgrade the controller software with delayed commit, which will help you to run and test the new software without committing the image.

Procedure

  Command or Action Purpose

Step 1

install add file file-name

Example:

Device# install add file <file>

Adds and expands the controller software image to the flash.

Note

 

In Cisco Catalyst 9800 Wireless Controller for Switch, run the install add file sub-package-file-name command to expand the wireless subpackage file.

Step 2

ap image predownload

Example:

Device# ap image predownload

Performs predownload of the AP image.

Step 3

install auto-abort-timer stop

Example:

Device# install auto-abort-timer stop

Stops the termination timer so that the upgrade process is not terminated after the default termination time of 6-8 hours.

Step 4

install activate issu

Example:

Device# install activate issu

Runs compatibility checks, installs the package, and updates the package status details.

Step 5

install commit

Example:

Device# install commit

Commits the activation changes to be persistent across reloads.

Monitoring In-Service Software Upgrade

To view the ISSU state after the install add ISSU and before the install activate ISSU, use the following command:

Device# show issu state detail

-- Starting local lock acquisition on chassis 1 ---
Finished local lock acquisition on chassis 1
Current ISSU Status: Enabled 
Previous ISSU Operation: Abort Successful 
=======================================================
System Check Status
-------------------------------------------------------
Platform ISSU Support Yes
Standby Online Yes
Autoboot Enabled Yes
SSO Mode Yes
Install Boot Yes
Valid Boot Media Yes
=======================================================
No ISSU operation is in progress
show install summary
[ Chassis 1 2 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version 
--------------------------------------------------------------------------------
IMG I 17.1.1.0.432 
IMG C 16.12.2.0.2707
--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

To view the ISSU state after activating ISSU, use the following command:

Device# show issu state detail

Current ISSU Status: In Progress 
Previous ISSU Operation: Abort Successful 
=======================================================
System Check Status
-------------------------------------------------------
Platform ISSU Support Yes
Standby Online Yes
Autoboot Enabled Yes
SSO Mode Yes
Install Boot Yes
Valid Boot Media Yes
=======================================================
Operation type: Step-by-step ISSU
Install type : Image installation using ISSU
Current state : Activated state
Last operation: Switchover
Completed operations:
Operation Start time
-------------------------------------------------------
Activate location standby Chassis 2 2019-09-17:23:41:12
Activate location active Chassis 1 2019-09-17:23:50:06
Switchover 2019-09-17:23:52:03
State transition: Added -> Standby activated -> Active switched-over
Auto abort timer: automatic, remaining time before rollback: 05:41:53
Running image: bootflash:packages.conf
Operating mode: sso, terminal state reached
show install summary
[ Chassis 1/R0 2/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version 
--------------------------------------------------------------------------------
IMG U 17.1.1.0.432
--------------------------------------------------------------------------------
Auto abort timer: active on install_activate, time before rollback - 05:41:49
--------------------------------------------------------------------------------

To view the ISSU state after installing the commit, use the following command:

Device# show issu state detail

--- Starting local lock acquisition on chassis 1 ---
Finished local lock acquisition on chassis 1
Current ISSU Status: Enabled 
Previous ISSU Operation: Successful 
=======================================================
System Check Status
-------------------------------------------------------
Platform ISSU Support Yes
Standby Online Yes
Autoboot Enabled Yes
SSO Mode Yes
Install Boot Yes
Valid Boot Media Yes
=======================================================
No ISSU operation is in progress
show install summary
[ Chassis 1/R0 2/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version 
--------------------------------------------------------------------------------
IMG C 17.1.1.0.432
--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------

To view the ISSU state after terminating the ISSU process, use the following command:

Device# show issu state detail
Current ISSU Status: In Progress 
Previous ISSU Operation: Abort Successful 
=======================================================
System Check Status
-------------------------------------------------------
Platform ISSU Support Yes
Standby Online Yes
Autoboot Enabled Yes
SSO Mode Yes
Install Boot Yes
Valid Boot Media Yes
=======================================================
Operation type: Step-by-step ISSU
Install type : Image installation using ISSU
Current state : Timeout-error state
Last operation: Commit Chassis 1
Completed operations:
Operation Start time
-------------------------------------------------------
Activate location standby Chassis 2 2019-09-17:23:41:12
Activate location active Chassis 1 2019-09-17:23:50:06
Switchover 2019-09-17:23:52:03
Abort 2019-09-18:00:14:13
Commit Chassis 1 2019-09-18:00:28:23
State transition: Added -> Standby activated -> Active switched-over -> Activated -> Timeout-error
Auto abort timer: inactive
Running image: bootflash:packages.conf
Operating mode: sso, terminal state reached
To view the summary of the active packages in a system, use the following command:
Device# show install summary

[ Chassis 1 2 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version 
--------------------------------------------------------------------------------
IMG C 16.12.2.0.2707
--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------

Troubleshooting ISSU

Using install activate issu command before completing AP pre-download.

The following scenario is applicable when you run the install activate issu command before completing AP pre-download. In such instances, you should run the ap image predownload command and then proceed with the activation. 

Device# install activate issu

install_activate: START Wed Jan  8 04:48:04 UTC 2020
System configuration has been modified.
Press Yes(y) to save the configuration and proceed.
Press No(n) for proceeding without saving the configuration.
Press Quit(q) to exit, you may save configuration and re-enter the command. [y/n/q]
y
Building configuration...
[OK]Modified configuration has been saved
install_activate: Activating ISSU
NOTE: Going to start Activate ISSU install process
STAGE 0: System Level Sanity Check
===================================================
--- Verifying install_issu supported ---
--- Verifying standby is in Standby Hot state ---
--- Verifying booted from the valid media ---
--- Verifying AutoBoot mode is enabled ---
--- Verifying Platform specific ISSU admission criteria ---
CONSOLE: FAILED: Install operation is not allowed.
 Reason -> AP pre-image download is mandatory f
or hitless software upgrade.
 Action -> Trigger AP pre-image download.
FAILED: Platform specific ISSU admission criteria
ERROR: install_activate  exit(2 ) Wed Jan  8 04:48:37 UTC 2020