THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.
|Affected OS Type
|Affected Software Product
|Affected Release Number
Unified Computing System (UCS) Drivers
OS Based Driver ISO
VMWare ESXi 6.7
VMWare ESXi 7.0
|NFNIC driver returns target aborted IO as completed successfully
In certain conditions, the Native FNIC (NFNIC) driver might incorrectly mark an aborted IO operation as successful. The driver will not attempt to retry the IOs and as a result data will be missing or corrupted on the storage target.
This is applicable to target aborted IOs only. For example, IOs which are aborted by a storage array. If an IO times out or is aborted by the initiator, the NFNIC driver will correctly handle the operation. Any Unified Computing System (UCS) servers which utilize the NFNIC driver (both rack and blade servers) are impacted.
The NFNIC driver is responsible for handling Fibre-Channel (FC) IOs (frames), which includes error handling and reporting. This driver was introduced in the VMware ESXi 6.7 release and is different from the “legacy” FNIC driver. Only the NFNIC driver handles this condition incorrectly.
This issue can result in data integrity issues on storage targets. Data might be missing, incomplete, or corrupted with no warning prior to loss of data integrity.
Target initiated aborts will mostly occur when Fibre-Channel (FC) frames are dropped or received out of order by the target. This might be triggered by the addition or deletion of FC port-channel members, which can initiate a frame drop. You might see the VMKernel log message "hdr status: FCPIO_ABORTED IO failure!".
An example of a VMKernel log message is shown here:
WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1850: sc: 0x459a92c02ac0 tag: 0x5e5 hdr status: FCPIO_ABORTED IO failure!
This issue requires a software update to the NFNIC driver. The best solution is to upgrade the NFNIC driver to the fixed 18.104.22.168 release or later. Review the HCL Tool for the appropriate driver to run for your hardware/software/firmware configuration. The NFNIC 22.214.171.124 driver for ESXi 6.7 and 7.0 can be downloaded from VMware Customer Connect.
For certain storage arrays, there might be hardened storage array code which can avoid this situation. See the defect notes of Cisco bug ID CSCvx64370 for more details. Cisco still recommends to upgrade to a fixed NFNIC release even if you run hardened storage code.
In order to update your Cisco NFNIC driver, enter this command on your ESXi host:
#esxcli software vib install -v /path/to/driver/
Here is an example:
[root@esxhost1:~] esxcli software vib install -v /tmp/nfnic-126.96.36.199-1OEM.6188.8.131.5269922.x86_64.vib
How To Identify Affected Products
This command can be used on the ESXi CLI shell in order to confirm the version of NFNIC driver on your ESXi host. Note that if the NFNIC driver is not listed, then the NFNIC driver is not in use.
#esxcli software vib list | grep nfnic
An example is shown here:
[root@esxhost1:~] esxcli software vib list | grep nfnic
nfnic 184.108.40.206-1OEM.6220.127.116.1169922 Cisco VMwareCertified 2020-04-04
In this example, the NFNIC is Version 18.104.22.168. It will need to be upgraded. The appropriate NFNIC driver that users should run can be determined with the HCL Tool.
This issue might also (rarely) be triggered as a result of a write queue (WQ) error due to bad hardware. The symptoms of this failure will be the same, and an upgrade of the NFNIC driver will prevent any data loss or corruption. If customers continue to see messages such as the “FCPIO_ABORTED” message shown after an NFNIC driver upgrade, contact the Cisco Technical Assistance Center (TAC).
For More Information
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
Receive Email Notification For New Field Notices
My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.