Troubleshooting PowerOn Auto Provisioning

Switch Does Not Come Up in Time for POAP to Complete

If the switch does not come up in a reasonable duration for POAP to complete, connect to the switch through the serial line and check to see if it is stuck at the following prompt: 

Waiting for system online status before starting POAP ...
Waiting for system online status before starting POAP ...
Waiting for system online status before starting POAP ...

System is not fully online. Skip POAP? (yes/no)[n]:

You can continue with POAP by entering no at the prompt. If POAP does not start properly on the second attempt, proceed with the normal setup by entering yes at the prompt when it returns.

POAP Fails

Take these actions if PowerOn Auto Provisioning (POAP) fails for any reason:

  • Stop the POAP process to continue with the normal switch bring-up steps. It might take a few minutes for a full stop of POAP, so be patient. 

    2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Assigned IP address: 172.23.40.221
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Netmask: 255.255.255.0
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: DNS Server: 172.21.157.5
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Default Gateway: 172.23.40.1
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Server: 172.23.40.6
2013 Oct 29 22:24:59 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Name: /pxelinux.0
2013 Oct 29 22:25:09 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: The POAP Script download has started
2013 Oct 29 22:25:09 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: The POAP Script is being downloaded from [copy tftp://172.23.40.6//pxelinux.0 bootflash:scripts/script.sh vrf management ]
2013 Oct 29 22:25:10 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: POAP boot file download failed.
2013 Oct 29 22:25:10 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: POAP DHCP discover phase failed
2013 Oct 29 22:25:12 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Abort Power On Auto Provisioning and continue with normal setup ?(yes/no)[n]:
2013 Oct 29 22:25:46 switch %$ VDC-1 %$ %POAP-2-POAP_DHCP_DISCOVER_START: POAP DHCP Discover phase started
2013 Oct 29 22:25:46 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Abort Power On Auto Provisioning and continue with normal setup ?(yes/no)[n]:

Abort Auto Provisioning and continue with normal setup ?(yes/no)[n]: yes
  • Check the failure reason in the log files. Two POAP log files are saved on the bootflash. Logs from the POAP process are stored in the file that ends with poap_pid _init.log as shown below. The failure reason should appear toward the end of this file. 
    bash-4.2# tail 20131029_222312_poap_5367_init.log -n 3
Tue Oct 29 22:27:41 2013:poap_net_rx_pkt: Droppping the pakcet due to Ethernet hdrparsing error on if_index - 5000000
Tue Oct 29 22:27:41 2013:DEST IP is not Broadcast
Tue Oct 29 22:27:41 2013:poap_net_rx_pkt: Droppping the pakcet due to Ethernet hdrparsing error on if_index - 5000000
  • Check to see if the POAP script file that is downloaded from your DHCP or TFTP server fails in the process of running. Depending on the stage of the failure, the device might proceed with the normal setup or reboot. 
    2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Assigned IP address: 172.23.40.181
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Netmask: 255.255.255.0
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: DNS Server: 172.21.157.5
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Default Gateway: 172.23.40.1
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Server: 172.23.40.6
2013 Oct 29 22:42:34 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Name: poap.py
2013 Oct 29 22:42:45 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: The POAP Script download has started
2013 Oct 29 22:42:45 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: The POAP Script is being downloaded from [copy tftp://172.23.40.6/poap.py bootflash:scripts/script.sh vrf management ]
2013 Oct 29 22:42:46 switch %$ VDC-1 %$ %POAP-2-POAP_SCRIPT_DOWNLOADED: Successfully downloaded POAP script file
2013 Oct 29 22:42:46 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script file size 21965, MD5 checksum 1bd4b86892439c5785a20a3e3ac2b0de
2013 Oct 29 22:42:46 switch %$ VDC-1 %$ %POAP-2-POAP_SCRIPT_STARTED_MD5_NOT_VALIDATED: POAP script execution started(MD5 not validated)
2013 Oct 29 22:47:57 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: POAP script execution aborted
  • The POAP script file logs are written to a file under the bootflash scheme. The filename starts with poap.log. If there are multiple file logs, look at the one with the most recent time stamp for any errors. 
    bash-4.2# tail poap.log.22_42_46
CLI : show file volatile:poap.cfg.md5.poap_md5 | grep -v '^#' | head lines 1 | sed 's/ .*$//'
INFO: md5sum 46684d8f8b7c5ffac3b37ac8560928e5 (.md5 file)
CLI : show file volatile:poap.cfg md5sum
INFO: md5sum 46684d8f8b7c5ffac3b37ac8560928e5 (recalculated)

CLI : config terminal ; boot nxos bootflash:poap/system.img
CLI : copy running-config startup-config
CLI : copy volatile:poap.cfg scheduled-config
INFO: Configuration successful