How to avoid boot loop due to corrupted image on Wave 2 and Catalyst 11ax Access Points (CSCvx32806)

Introduction

This document provides details on the boot loop seen on Wave2 11ac and Catalyst 11ax Access Points (APs) during image upgrade due to AP image corruption. This symptom of boot loop is tracked by the Cisco bug CSCvx32806. The deployments that include APs joined over WAN links are most susceptible for AP image corruption during AP Image pre-download or Efficient Image upgrade. 

Affected Products

• Cisco Wave2 11ac Access Points (1800/2800/3800/4800/1560)
• Cisco Catalyst 91xx Series WiFi 6 and WiFI6E Access Points

Affected Software Versions

Cisco IOS-XE versions

• 16.12.x
• 17.3.1, 17.3.2, 17.3.3, 17.3.4c, 17.3.5a, 17.3.6
• 17.4.1, 17.5.1
• 17.6.1, 17.6.2, 17.6.3, 17.6.4
• 17.7.1, 17.8.1
• 17.9.1, 17.9.2

Problem

Customers looking to upgrading their Catalyst 9800 WLCs leverage features like AP image predownload or Efficient Image Upgrade (only in case of FlexConnect) to get the software image pre-downloaded to AP's flash partition, to reduce downtime needed for image upgrade. For the deployments where APs are located across WAN links, both predownload and efficient image upgrade are susceptible to image corruption. When an image is downloaded on the flash of COS AP, AP detects corruption, reports image verification failure but continues to boot the corrupted image and ends up in boot loop.

Root Cause

The root cause of why image gets corrupted is not yet known and is being tracked via CSCwf09053. The corruption has typically manifested when image was transferred over CAPWAP over WAN link. When an image is downloaded on the flash of COS AP, an upgrade script (upgrade.sh) is executed which verifies the image and returns two success or error codes. In case of first error code, the upgrade is halted but in case of second error code, AP ignores the error and continues to install the corrupted image resulting in AP getting stuck in boot loop. This behavior of AP to bypass the second error is fixed via CSCvx32806. 

Symptoms

To verify if you are running into this problem, you will need to review the syslogs generated by the APs. It is recommended to configure a syslog server (as explained in step 1 of the Recommended Upgrade Procedure section) to receive syslogs from AP when an AP Image Pre-download or Efficient Image Upgrade for FlexConnect deployment in executed. In the syslogs, if you see Image signature verification failure: -3 for a given AP, it indicates that the image that was pre-downloaded is corrupted.

Fixed Software

The image corruption issue is resolved in

• 17.3.6 + APSP6 or higher APSPs
• 17.3.7 and higher
• 17.6.5 and higher 17.6 MRs
• 17.9.3 and higher 17.9 MRs
• 17.10.1 and higher releases

Workaround (for APs already in boot loop)

For AP Models 1800, 2800, 3800, 4800, 1560, 9117, 9124, 9130, 9136 

1. Power up the AP and connect to AP via console.
2. Boot the AP, break to U-BOOT by hitting 'ESC'. This should bring you to (u-boot)> or (BTLDR)#prompt. 
3. Run these commands

(u-boot)> OR (BTLDR)# setenv mtdids nand0=nand0 && setenv mtdparts mtdparts=nand0:0x40000000@0x0(fs) && ubi part fs
(u-boot)> OR (BTLDR)# ubi remove part1  (or part2 if corrupted image is in part2) 
(u-boot)> OR (BTLDR)# ubi create part1  (or part2 if corrupted image is in part2)      
(u-boot)> OR (BTLDR)# boot 

For AP Models 9105, 9115, 9120

1. Power up the AP and connect to AP via console.
2. Boot the AP, break to U-BOOT by hitting 'ESC'. This should bring you to (u-boot)> prompt.
3. Run these commands

(u-boot)> ubi part fs 
(u-boot)> ubi remove part1  (or part2 if corrupted image is in part2) 
(u-boot)> ubi create part1  (or part2 if corrupted image is in part2) 
(u-boot)> boot

Recommended Procedure to Upgrade to Fixed Software

For the scenario where upgrade has not been initiated, Cisco recommends these steps for upgrading WLC software while avoiding COS AP image corruption.

Step 1. Verify SSH is enabled under the AP Join Profile(s) on the C9800 WLC. Setup a syslog server in the network. Configure the IP address of the syslog server under AP Join Profile for all the sites and set the log trap value = Debug. Verify that the syslog server is receiving syslogs from AP.

Step 2. Download the software image to the C9800 WLC to prepare for predownload via CLI:

C9800# copy tftp:// bootflash:
C9800# install add file bootflash: C9800-80-universalk9_wlc.17.03.07.SPA.bin

Step 3. Run the AP image pre-download on the Cisco C9800 WLCs:

C9800# ap image predownload

Note: Depending on the scale and type of deployment this can take anywhere from a few minutes to a few hours.

Step 4. Once the pre-download for all the APs has completed, check for either of the below two logs on the syslog server:

• Image signing verify success.

• Image signature verification failure: -3

Caution: For the APs with the failure message, DO NOT PROCEED FURTHER WITH THE UPGRADE PROCESS. For APs which show the “success” message, APs have correctly downloaded the image.

Step 5 (optional).

APs which have the failure message, have a corrupted image in their backup partition and if that image is activated, it will land the AP in a bootloop. 

To avoid the bootloop you would need to overwrite the image in the Backup partition of the AP with an archive download of a separate AP image by using the following process. 

If the number of failed APs is small, then you can simply SSH to each AP and initiate the following steps.

COS_AP#term mon 
COS_AP#show clock 
COS_AP#archive download-sw /no-reload tftp://<ip-address>/%apimage% 
COS_AP#show version

If the number of failed APs are large, you can use an automated process using the WLAN Poller

Step 5a. Install the WLAN Poller on your MAC or Windows Machine.

Step 5b. Populate the aplist csv file with the relevant failed APs.

Step 5c. Populate the cmdlist file with the below commands (You can always add more at your discretion):

COS_AP#term mon 
COS_AP#show clock 
COS_AP#archive download-sw /no-reload tftp://<ip-address>/%apimage% 
COs_APshow version 

Step 5d. Execute the WLAN Poller.

Step 5e. Once its execution has completed, please check every APs log file for the failure and success messages again to be safe (See Step 3)

Step 6.  Once the archive download process has completed, you can proceed with your upgrade.

Step 6a. Instructs AP to swap primary partition to newer downloaded image and reboot the APs

C9800#ap image swap
C9800#ap image reset

Step 7. Immediately activate image on the C9800 WLC and reload.

C9800#install activate file bootflash:C9800-80-universalk9_wlc.17.03.07.SPA.bin
- Confirm reload when prompted

Step 8.  Commit image on the C9800 WLC. Skipping this step will cause WLC to rollback to previous software image

C9800#install commit

Frequently Asked Questions

 Q1) I ran a predownload some days ago but have not rebooted WLC and APs yet. I dont have syslogs to verify if image is corrupted. How do i verify if image is corrupted?

Check “show logging” on the APs using WLAN Poller and follow step #3. If you see no success or failure messages in the show logging, reach out to TAC for the alternate process.

Q2) I have a centralized deployments with APs in Local Mode. Do I still need to execute the above?

This issue has only been reported when upgrading APs over WAN connection. APs in Local mode and over Local network are highly unlikely to run into this issue, so it is not recommended to follow this procedure for upgrades.

Q3) I have new out-of-box APs. How can i deploy them without encountering this issue?

New out of Box APs downloading code over WAN will be also susceptible to the issue. It is recommended to stage these APs first with Local WLC.