- Preface
- New and Changed Information
- Overview
- Troubleshooting Matrix
- Troubleshooting an Installation or Update
- Troubleshooting the Configuration
- Troubleshooting Cisco APIC-EM Single and Multi-Host
- Troubleshooting Services Using System Health
- Troubleshooting Services Using the Controller Admin Console
- Troubleshooting Using the Logs
- Troubleshooting Passwords
- Troubleshooting Commands
- Troubleshooting Log Files
- Contacting the Cisco Technical Assistance Center
- Index
Troubleshooting the
Configuration
The following information may be used to troubleshoot issues with the configuration:
- Troubleshooting the Configuration
- Updating the Configuration Using the Wizard
- Resetting the Cisco APIC-EM
- Restoring the Controller to the Factory Default
- Creating a Support File for a Single Host
Troubleshooting the Configuration
The following table describes recommended actions to take to resolve a Cisco APIC-EM configuration issue.
Symptom |
Possible Cause |
Recommended Action |
---|---|---|
Controller in a single host configuration appears to be in an unstable state. For example, applications are not running, or applications are inaccessible, and/or not appearing in the GUI. |
Controller in unstable state, possibly due to error(s) in entering configuration values with the Cisco APIC-EM configuration wizard. |
Log into the host, check the configuration values, and reenter any configuration values that are incorrect. References: |
Controller in a multi-host configuration appears to be in an unstable state. For example, applications are not running, or applications are inaccessible, and/or not appearing in the GUI. |
Controller in unstable state, possibly due to error(s) in entering configuration values with the Cisco APIC-EM configuration wizard. |
Log into the host, check the configuration values, and reenter any configuration values that are incorrect. References: |
Controller was working fine for a period of time after its initial deployment, but then lapses into an unstable state. For example, applications are not running, or applications are inaccessible, and/or not appearing in the GUI. |
Possible user or other error in configuration values that occurs after the initial deployment. |
Revert back to the first configuration or the factory default configuration. References: |
Controller was working fine for a multi-host configuration, but after a period of time one of the hosts becomes erratic and unstable. |
Possible failed service or services in the multi-host cluster. |
Remove and then reattach unstable host from the multi-host cluster. References: |
Controller was working fine for a multi-host configuration, but after a period of time one of the hosts fails. |
Possible failed service or services in the multi-host cluster. |
Remove and then reattach failed and inoperable host from the multi-host cluster. References: |
Updating the Configuration Using the Wizard
You can troubleshoot the Cisco APIC-EM deployment by running the configuration wizard a second time and updating any earlier configuration entries. The configuration wizard saves and displays your previous configuration settings, so you do not have to reenter them.
![]() Note | When performing this procedure, controller downtime occurs. For this reason, we recommend that you perform this procedure during a maintenance time period. For information about changing settings for a multi-host configuration, see Changing the Settings in a Multi-Host Cluster. |
You have installed the Cisco APIC-EM following the procedure described in the Cisco Application Policy Infrastructure Controller Enterprise Module Installation Guide.
Resetting the Cisco APIC-EM
You can troubleshoot a Cisco APIC-EM deployment by resetting the controller back to configuration values that were originally set using the configuration wizard the first time. A reset of the controller is helpful, when the controller has gotten itself into an unstable state and other troubleshooting activities have not resolved the situation.
![]() Note | In a multi-host environment, you need to perform this procedure on only a single host. After performing this procedure on a single host, the other two hosts will be automatically reset. |
You have installed the Cisco APIC-EM following the procedure described in the Cisco Application Policy Infrastructure Controller Enterprise Module Installation Guide.
Step 1 | Using a Secure
Shell (SSH) client, log into the host (physical or virtual) with the IP address
that you specified using the configuration wizard.
| ||||||
Step 2 | When prompted, enter your Linux username ('grapevine') and password for SSH access. | ||||||
Step 3 | Navigate to the bin directory on the Grapevine root. The bin directory contains the grapevine scripts. | ||||||
Step 4 | Enter the
reset_grapevine
command at the prompt to run the reset grapevine script.
$ reset_grapevine The reset_grapevine command returns the configuration settings back to values that you configured when running the configuration wizard for the first time. The configuration settings are saved to a .JSON file. This .JSON file is located at: /etc/grapevine/controller-config.json. The reset_grapevine command uses the data in the controller-config.json file to return to the earlier configuration settings, so do not delete this file. If you delete this file, you must run the configuration wizard again and reenter your configuration data. The reset_grapevine command will terminate if the SSH connection is disconnected for any reason. To avoid this, we recommend that you use tmux (terminal multiplexer) which is already installed on the controller to run the reset_grapevine command in the session. You can use the following commands for tmux:
After entering the reset_grapevine command, you are then prompted to reenter your Grapevine password. | ||||||
Step 5 | Enter your
Grapevine password a second time.
[sudo] password for grapevine:******** You are then prompted to delete all virtual disks The virtual disks are where the Cisco APIC-EM database resides. For example, data about devices that the controller discovered are saved on these virtual disks. If you enter yes (y), all of this data is deleted. If you enter no (n), then the new cluster will come up populated with your existing data once the reset procedure completes. | ||||||
Step 6 | Enter
n to prevent the deletion all of the virtual disks.
THIS IS A DESTRUCTIVE OPERATION Do you want to delete all VIRTUAL DISKS in your APIC-EM cluster? (y/n):n You are then prompted to delete all Cisco APIC-EM authentication timeout policies, user password policies, and user accounts other than the primary administrator account. | ||||||
Step 7 | Enter
n to prevent the deletion of all authentication
timeout policies, user password policies, and user accounts other than the
primary administrator account.
THIS IS A DESTRUCTIVE OPERATION Do you want to delete authentication timeout policies, user password policies, and Cisco APIC-EM user accounts other than the primary administrator account? (y/n): n You are then prompted to delete any imported certificates. | ||||||
Step 8 | Enter
n to prevent the deletion of any imported
certificates.
THIS IS A DESTRUCTIVE OPERATION Do you want to delete the imported certificates? (y/n): n You are then prompted to delete any backups. | ||||||
Step 9 | Enter
n to prevent the deletion of any backups.
THIS IS A DESTRUCTIVE OPERATION Do you want to delete the backups? (y/n): n The controller then resets itself with the configuration values that were originally set using the configuration wizard the first time. When the controller is finished resetting, you are presented with a command prompt from the controller. | ||||||
Step 10 | Using the Secure Shell (SSH) client, log out of the host. |
Restoring the Controller to the Factory Default
In certain situations, you may want to restore the Cisco APIC-EM to its original factory default settings. For example, if your controller appliance is being replaced or simply has an undesirable configuration that needs to be completely removed. Under these circumstances, you can restore the controller to its factory defaults and then proceed to reconfigure it as a new controller.
This procedure describes how to a restore the factory defaults to the controller.
![]() Caution | This procedure shuts down both the Cisco APIC-EM and the host (physical or virtual) on which it resides. At the end of this procedure, you will need to access the host and restart it. |
You have installed the Cisco APIC-EM following the procedure described in the Cisco Application Policy Infrastructure Controller Enterprise Module Installation Guide.
You have access to the Cisco APIC-EM using either a physical console or a Telnet connection.
Step 1 | Using a Secure
Shell (SSH) client, log into the host (physical or virtual) with the IP address
that you specified using the configuration wizard.
| ||||||
Step 2 | When prompted, enter your Linux username ('grapevine') and password for SSH access. | ||||||
Step 3 | Enter the
reset_grapevine factory command at the prompt.
$ reset_grapevine factory The reset_grapevine factory command will terminate if the SSH connection is disconnected for any reason. To avoid this, we recommend that you use tmux (terminal multiplexer) which is already installed on the controller to run the reset_grapevine factory command in the session. You can use the following commands for tmux:
| ||||||
Step 4 | Enter your Linux
grapevine password a second time to start the reset process.
$ sudo password for grapevine ********* After entering this command a warning appears that the reset_grapevine factory command will shut down the controller. You are then prompted to confirm your desire to run the reset_grapevine factory command. | ||||||
Step 5 | Enter
Yes to confirm that you want to run the
reset_grapevine factory command.
The controller then performs the following tasks: |
What to Do Next
Perform the following tasks:
Creating a Support File for a Single Host
You can troubleshoot the Cisco APIC-EM deployment by creating a root cause analysis (rca) support file. This rca file consists of logs, configuration files, and command output. After you create this rca file, you can then email it to Cisco support for assistance.
You have installed the Cisco APIC-EM following the procedure described in the Cisco Application Policy Infrastructure Controller Enterprise Module Installation Guide.
Step 1 | Using a Secure
Shell (SSH) client, log into the host (physical or virtual) with the IP address
that you specified using the configuration wizard.
| ||
Step 2 | When prompted, enter your Linux username ('grapevine') and password for SSH access. | ||
Step 3 | Navigate to the bin directory on the host. The bin directory contains the grapevine scripts. | ||
Step 4 | To create the
support file, enter the
rca command in this directory.
$ rca [sudo] password for grapevine: mkdir: created directory '/tmp/grapevine-root-172.24.100.15-rca-2016-08-05_18-17-04_UTC+0000' =============================================================== RCA package created on Fri Aug 5 18:17:10 UTC 2016 Max Log Size: 50000000 Skipping collection of '' =============================================================== 2016-08-05 18:17:10 | INFO | Generating log for 'date'... 2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat hosttime'... 2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat speed' .. 2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat session id'... 2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat balloon '... 2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat swap'... 2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat memlimit'... 2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat memres' .... The rca command runs a root cause analysis script that creates a tar file that contains log files, configuration files, and the command output.
|
What to Do Next
Send the tar file created by this procedure to Cisco support for assistance in resolving your issue. For information about contacting Cisco support, see Contacting the Cisco Technical Assistance Center.