Hitless Software Upgrade on Catalyst 9800 Series Wireless LAN Controllers
Available Languages
Contents
Introduction
This document describes how to perform an N+1 hitless software upgrade on Catalyst 9800 series Wireless LAN controllers.
Contributed by Anand Shandilya, Cisco TAC Engineer.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
-
Catalyst 9800 Wireless LAN Controllers and AP (IOS and ClickOS) platforms
- Catalyst 9800 Wireless LAN Controllers software feature sets
Components Used
The information provided in this document is based on the following software and hardware components.
- Catalyst C9800-40 and C9800-L-F-K9 wireless LAN Controllers
- Click OS and IOS APs
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Background Information
The existing CAPWAP implementation requires the WLC and the AP to be on the same software version. Therefore, a WLC upgrade is followed by APs upgrade causing an inevitable network outage. With the existing implementation, it is impossible to upgrade the WLC without scheduling downtime.
Hitless upgrade leverages the concept of N+1 high availability using a spare WLC (already upgraded to the target version) to upgrade the CAPWAP infrastructure. The APs are then upgraded in a staggered fashion, using the Rolling AP upgrade feature, avoiding network disruption by not letting all the APs upgrade at once. This ensures that the clients are serviced by the neighboring APs while one of the APs undergoes the upgrade process.
WorkFlow
- Upgrade the spare WLC to the target version.
- Establish a mobility tunnel between the production WLC and the spare one.
- Initiate the upgrade on the production controller using the install add file command.
- Pre-download the APs.
- Move the APs to the destination controller (spare WLC). APs are upgraded in a staggered fashion using the Rolling AP upgrade algorithm.
- Once all the APs move to the spare WLC in multiple iterations activate the target image on production WLC.
- The production WLC reloads for the new image to take effect.
- Move all the APs back to the production controller.
Rolling AP Upgrade Algorithm
The algorithm works in three stages.
1. Candidate AP Set Selection
First, a set of candidates are selected based on nearby APs information. Rolling AP Upgrade algorithm selects the configured percentage of APs to be upgraded in each iteration while maintaining RF coverage
For serving wireless clients. maintaining coverage is important and hence, it takes precedence over selecting the required number of APs. Therefore,
For P = 25%, expected number of iterations for all APs to upgrade ~ 6
For P = 15%, expected number of iterations for all APs to upgrade ~ 12
For P = 5%, expected number of iterations for all APs to upgrade ~ 22
2. Client Steering
Clients on the candidate APs are steered to APs which are not in the candidate list prior to rebooting the candidate APs. If the clients still persist on the candidate APs, they will just be sent a de-authentication frame and the AP will reload with the new image.
3. AP Re-load and Re-join
Post the client steering stage, the AP is reloaded with the new image.
At this point, a 3-minute timer is started for the APs to join back. When this timer expires, all candidate APs are checked and marked for the WLC they have connected to (self or the peer).
If at least 90% of the candidate APs have joined back, the iteration is concluded. If not, 3 minutes window is extended and the check is repeated for two more times until the count hits at least 90%.
At the end of the 3rd try, the iteration is concluded anyway and the next iteration is initiated. Hence, each iteration may last for at most 10 mins.
Restrictions
- Non-client serving APs like the ones working in the monitor, sniffer mode, etc are upgraded in one go before the rest of the procedure starts.
- Mesh APs are not supported by rolling AP Upgrade. If the deployment has mesh APs, they are upgraded in one shot and at the end of all iterations.
- 16.10 only had a CLI option to configure.
- AP needs to be registered before GUI shows up the option to enable a hitless upgrade.
- The hitless upgrade is not supported by the controller running in BUNDLE mode.
Topology
Configuration
From GUI
1. Establish the mobility tunnel between the controllers.
2. Initiate the upgrade on the controller. Enable the hitless upgrade option as well. Optionally, enable Fallback after upgrade so that the APs move back to the parent controller (without a swap and reset) after activation of the new image and reloading of the parent controller.
3. Once all the stages are done, WLC prompts for a reload.
From CLI
1. Establish the mobility tunnel between the controllers.
9800-40(config)#wireless mobility group member mac-address d478.9b3c.4ecb ip 10.106.36.78 public-ip 10.106.36.78 group default 9800-L(config)#wireless mobility group member mac-address d4e8.80b2.dc8b ip 10.106.36.110 public-ip 10.106.36.110 group default
2. Initiate the upgrade on the controller.
9800-40#install add file flash:C9800-40-universalk9_wlc.17.01.01s.SPA.bin
Once the installation is successful, the new image should be in an inactive state.
9800-40#show install summary [ Chassis 1 ] Installed Package(s) Information: State (St): I - Inactive, U - Activated & Uncommitted, C - Activated & Committed, D - Deactivated & Uncommitted -------------------------------------------------------------------------------- Type St Filename/Version -------------------------------------------------------------------------------- IMG I 17.1.1s.0.351 IMG C 16.12.2s.0.47 -------------------------------------------------------------------------------- Auto abort timer: inactive --------------------------------------------------------------------------------
3. Initiate the pre-download on APs to load the new image as the backup on the APs.
9800-40#ap image predownload
In order to check the status of pre-download, use this command.
9800-40#show ap image
Total number of APs: 5
Number of APs
Initiated : 0
Predownloading : 1
Completed predownloading : 3
Not Supported : 0
Failed to Predownload : 0
Predownload in progress : Yes
AP Name Primary Image Backup Image Predownload Status Predownload Version Next Retry Time Retry Count
-------------------------------------------------------------------------------------------------------------------------------------------------------
AP3800 16.12.2.132 17.1.1.29 Complete 17.1.1.29 0 0
3800-2 16.12.2.132 17.1.1.29 Complete 17.1.1.29 0 0
4800-1 16.12.2.132 17.1.1.29 Complete 17.1.1.29 0 0
3702I-2 16.12.2.132 0.0.0.0 Predownloading 17.1.1.29 0 0
4. Optionally, if one requires to configure the percentage of APs to be upgraded per iteration, this command can be used. The default value is 15.
9800-40(config)#ap upgrade staggered ? 15 15 percent APs per iteration 25 25 percent APs per iteration 5 5 percent APs per iteration one-shot All APs in one shot, no staggering
5. Once the pre-download is complete on all the APs, you have to move the APs to the spare controller that runs on the updated code.
9800-40#ap image upgrade destination 9800-L 10.106.36.78 fallback
This command will move the APs to the specified destination WLC with a swap and reset command. Swap command interchanges the AP image so that the target code is marked primary image for the APs whereas reset command reloads the AP. It is assumed that the destination WLC is on the same version as the APs backup image.
Optionally, one can use the fallback keyword to enable Fallback after Upgrade option so that the APs move back to the parent controller (without a swap and reset) after activation of the new image and reloading of the source controller.
6. Once all the APs have moved to the destination controller, activate the image on the source WLC.
On Destination WLC, verify if all the APs have moved successfully.
9800-L#show ap upgrade AP upgrade is complete, fallback awaited Fallback type: Fallback only From version: 16.12.2.132 To version: 17.1.1.29 Started at: 04/13/2020 02:32:09 UTC Configured percentage: N/A Percentage complete: 100 End time: 04/13/2020 02:56:09 UTC Progress Report --------------- Iterations ---------- Iteration Start time End time AP count ------------------------------------------------------------------------------------------------ 0 04/13/2020 02:32:09 UTC 04/13/2020 02:32:09 UTC 1 1 04/13/2020 02:32:09 UTC 04/13/2020 02:38:09 UTC 1 2 04/13/2020 02:38:09 UTC 04/13/2020 02:44:09 UTC 1 3 04/13/2020 02:44:09 UTC 04/13/2020 02:47:09 UTC 1 4 04/13/2020 02:47:09 UTC 04/13/2020 02:56:09 UTC 1 Upgraded -------- Number of APs: 5 AP Name Radio MAC Iteration Status Site ---------------------------------------------------------------------------------------------------- AP3800 1880.9021.e0e0 0 Joined default-site-tag 3800-2 1880.9021.e280 1 Joined default-site-tag 9130-1 04eb.409f.9760 2 Joined default-site-tag 4800-1 dc8c.3746.b0e0 3 Joined default-site-tag 3702I-2 fc5b.39f1.c7e0 4 Joined Unknown In Progress ----------- Number of APs: 0 AP Name Radio MAC ------------------------------------------------- Remaining --------- Number of APs: 0 AP Name Radio MAC ------------------------------------------------- APs not handled by Rolling AP Upgrade ------------------------------------- AP Name Radio MAC Status Reason for not handling by Rolling AP Upgrade ----------------------------------------------------------------------------------------------------------------------
On Source WLC, activate the image. Type yes to all the prompts. Once the install is complete, the controller proceeds to reload.
9800-40#install activate install_add_activate_commit: Activating PACKAGE Following packages shall be activated: /bootflash/C9800-L-rpboot.17.01.01s.SPA.pkg /bootflash/C9800-L-mono-universalk9_wlc.17.01.01s.SPA.pkg /bootflash/C9800-L-hw-programmables.17.01.01s.SPA.pkg This operation may require a reload of the system. Do you want to proceed? [y/n]y --- Starting Activate --- Performing Activate on all members [1] Activate package(s) on chassis 1 [1] Finished Activate on chassis 1 Checking status of Activate on [1] Activate: Passed on [1] Finished Activate
Post-reload, commit the image with this command.
9800-40#install commit
7. If one hasn't enabled Fallback after Upgrade option (as mentioned in Step5) use this command on destination WLC to move back the APs to the source WLC, once the source WLC is upgraded to the latest code.
On source WLC
9800-40#show version | i Version Cisco IOS XE Software, Version 17.01.01s Cisco IOS Software [Amsterdam], C9800 Software (C9800_IOSXE-K9), Version 17.1.1s, RELEASE SOFTWARE (fc4)
On Destination WLC
9800-L#ap image move destination 9800-40 10.106.36.110
This command moves the APs back to the source WLC without a swap and reset command.
8. All the APs should join back the source WLC and the latest image should be in a committed state.
9800-40#show install summary
[ Chassis 1/R0 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
--------------------------------------------------------------------------------
Type St Filename/Version
--------------------------------------------------------------------------------
IMG C 17.1.1s.0.351
--------------------------------------------------------------------------------
Auto abort timer: inactive
--------------------------------------------------------------------------------
9800-40#show ap summary
Number of APs: 5
AP Name Slots AP Model Ethernet MAC Radio MAC Location Country IP Address State
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
9130-1 2 9130AXI 04eb.409e.2620 04eb.409f.9760 default location IN 10.106.36.145 Registered
AP3800 2 3802I a023.9fae.f48a 1880.9021.e0e0 default location IN 10.106.37.13 Registered
3800-2 2 3802I a023.9fae.f4a4 1880.9021.e280 default location IN 10.106.36.187 Registered
4800-1 3 4800 dc8c.370e.b2da dc8c.3746.b0e0 default location IN 10.106.36.130 Registered
3702I-2 2 3702I fc5b.39d9.f4b4 fc5b.39f1.c7e0 default location IN 10.106.38.219 Registered
Verify
- Ensure that the WLC is running in INSTALL mode. The hitless upgrade is not supported in the BUNDLE mode.
9800-40#show version | i mode Installation mode is INSTALL
- The mobility tunnel between the controllers should be UP.
9800-40#show wireless mobility summary Mobility Summary Wireless Management VLAN: 36 Wireless Management IP Address: 10.106.36.110 Wireless Management IPv6 Address: Mobility Control Message DSCP Value: 48 Mobility Keepalive Interval/Count: 10/3 Mobility Group Name: default Mobility Multicast Ipv4 address: 0.0.0.0 Mobility Multicast Ipv6 address: :: Mobility MAC Address: d4e8.80b2.dc8b Mobility Domain Identifier: 0x34ac Controllers configured in the Mobility Domain: IP Public Ip MAC Address Group Name Multicast IPv4 Multicast IPv6 Status PMTU --------------------------------------------------------------------------------------------------------------------- 10.106.36.110 N/A d4e8.80b2.dc8b default 0.0.0.0 :: N/A N/A 10.106.36.78 10.106.36.78 d478.9b3c.4ecb default 0.0.0.0 :: Up 1385
- In order to monitor the AP upgrade use these commands.
On Source WLC
9800-40#show ap upgrade AP upgrade is in progress Fallback type: Fallback only From version: 16.12.2.132 To version: 17.1.1.29 Started at: 04/12/2020 21:02:09 India Configured percentage: 15 Percentage complete: 80 Expected time of completion: 04/12/2020 22:22:09 India Progress Report --------------- Iterations ---------- Iteration Start time End time AP count ------------------------------------------------------------------------------------------------ 0 04/12/2020 21:02:09 India 04/12/2020 21:02:09 India 1 1 04/12/2020 21:02:09 India 04/12/2020 21:08:09 India 1 2 04/12/2020 21:08:09 India 04/12/2020 21:14:09 India 1 3 04/12/2020 21:14:09 India 04/12/2020 21:17:09 India 1 4 04/12/2020 21:17:09 India ONGOING 1 Upgraded -------- Number of APs: 4 AP Name Radio MAC Iteration Status Site ---------------------------------------------------------------------------------------------------- AP3800 1880.9021.e0e0 0 Joined Member default-site-tag 3800-2 1880.9021.e280 1 Joined Member default-site-tag 9130-1 04eb.409f.9760 2 Joined Member default-site-tag 4800-1 dc8c.3746.b0e0 3 Joined Member default-site-tag In Progress ----------- Number of APs: 1 AP Name Radio MAC ------------------------------------------------- 3702I-2 fc5b.39f1.c7e0 Remaining --------- Number of APs: 0 AP Name Radio MAC ------------------------------------------------- APs not handled by Rolling AP Upgrade ------------------------------------- AP Name Radio MAC Status Reason for not handling by Rolling AP Upgrade ----------------------------------------------------------------------------------------------------------------------
On Destination WLC
9800-L#show ap upgrade AP upgrade is in progress Fallback type: Fallback only From version: 16.12.2.132 To version: 17.1.1.29 Started at: 04/13/2020 02:32:09 UTC Configured percentage: N/A Percentage complete: 80 Expected time of completion: 04/13/2020 03:52:09 UTC Progress Report --------------- Iterations ---------- Iteration Start time End time AP count ------------------------------------------------------------------------------------------------ 0 04/13/2020 02:32:09 UTC 04/13/2020 02:32:09 UTC 1 1 04/13/2020 02:32:09 UTC 04/13/2020 02:38:09 UTC 1 2 04/13/2020 02:38:09 UTC 04/13/2020 02:44:09 UTC 1 3 04/13/2020 02:44:09 UTC 04/13/2020 02:47:09 UTC 1 4 04/13/2020 02:47:09 UTC ONGOING 0 Upgraded -------- Number of APs: 4 AP Name Radio MAC Iteration Status Site ---------------------------------------------------------------------------------------------------- AP3800 1880.9021.e0e0 0 Joined default-site-tag 3800-2 1880.9021.e280 1 Joined default-site-tag 9130-1 04eb.409f.9760 2 Joined default-site-tag 4800-1 dc8c.3746.b0e0 3 Joined default-site-tag In Progress ----------- Number of APs: 1 AP Name Radio MAC ------------------------------------------------- 3702I-2 fc5b.39f1.c7e0 Remaining --------- Number of APs: 0 AP Name Radio MAC ------------------------------------------------- APs not handled by Rolling AP Upgrade ------------------------------------- AP Name Radio MAC Status Reason for not handling by Rolling AP Upgrade ----------------------------------------------------------------------------------------------------------------------
9800-L#show ap upgrade summary Report Name Start time ------------------------------------------------------------------------------------------ AP_upgrade_from_9800-40_13320202329 04/13/2020 02:32:09 UTC 9800-L#show ap upgrade name AP_upgrade_from_9800-40_13320202329 AP upgrade is in progress Fallback type: Fallback only From version: 16.12.2.132 To version: 17.1.1.29 Started at: 04/13/2020 02:32:09 UTC Configured percentage: N/A Percentage complete: 60 Expected time of completion: 04/13/2020 03:52:09 UTC Progress Report --------------- Iterations ---------- Iteration Start time End time AP count ------------------------------------------------------------------------------------------------ 0 04/13/2020 02:32:09 UTC 04/13/2020 02:32:09 UTC 1 1 04/13/2020 02:32:09 UTC 04/13/2020 02:38:09 UTC 1 2 04/13/2020 02:38:09 UTC 04/13/2020 02:44:09 UTC 1 3 04/13/2020 02:44:09 UTC ONGOING 0 Upgraded -------- Number of APs: 3 AP Name Radio MAC Iteration Status Site --------------------------------------------------------------------------------------------------------- AP3800 1880.9021.e0e0 0 Joined default-site-tag 3800-2 1880.9021.e280 1 Joined default-site-tag 9130-1 04eb.409f.9760 2 Joined default-site-tag In Progress ----------- Number of APs: 1 AP Name Radio MAC ------------------------------------------------- 4800-1 dc8c.3746.b0e0 Remaining --------- Number of APs: 0 AP Name Radio MAC ------------------------------------------------- APs not handled by Rolling AP Upgrade ------------------------------------- AP Name Radio MAC Status Reason for not handling by Rolling AP Upgrade ----------------------------------------------------------------------------------------------------------------------
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)