Cisco Application Policy Infrastructure Controller Enterprise Module Deployment Guide, Release 1.1.x
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
- Updated:
- February 29, 2016
Chapter: Troubleshooting the Cisco APIC-EM
Troubleshooting the Cisco APIC-EM
- Cisco APIC-EM Components and Architecture
- Troubleshooting an Unsuccessful Installation
- Troubleshooting the Configuration
- Troubleshooting Services
- Troubleshooting Passwords
- Troubleshooting Commands
- Troubleshooting Log Files
Cisco APIC-EM Components and Architecture
This section describes the components and basic architecture of the Cisco APIC-EM. You should review this section to better understand how the controller and its components operate before performing any of the troubleshooting procedures described in this chapter.
The Cisco APIC-EM consists of the following components:
Hosts
A host can be either an appliance, server or virtual machine where Cisco APIC-EM has been installed. Within the hosts are the Linux containers running instances of the Grapevine clients. The Grapevine clients run the different services. The Grapevine root runs as an application on the host itself and not in any Linux containers.
You can set up either a single host or multi-host deployment for your network.
Linux Containers
Linux containers is a virtualization technology that provides a software virtualization system for systems running GNU/Linux. These containers enable multiple virtual environments to be set up and run simultaneously.
The Cisco APIC-EM components (Grapevine roots and clients) make use of the Ubuntu operating system environment and Linux containers (LXC). The Grapevine root runs as an application on the host itself. The Grapevine clients run in LXCs within the host.
Grapevine
The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network, using Grapevine as an Elastic Services platform to support the controller's infrastructure and services. The Grapevine root and clients are key components of this infrastructure.
 Note | You can use a tool called the Grapevine developer console to troubleshoot roots and clients running services. The Grapevine developer console was bundled with the deployment files and installed when you first deployed the Cisco APIC-EM. You can access the Grapevine developer console at this controller IP address and port number: https://<controller IP address>:14141 |
Root and Clients
The Grapevine root handles all policy management in regards to service updates, as well as the service lifecycle for both itself and the Grapevine client. The Grapevine client is where the supported services run.
Note | You can remotely log into the root using SSH (Secure Shell) to troubleshoot any issues. A default idle timeout of 1 hour has been set for an SSH console login. You will be automatically logged out after 1 hour of inactivity on the SSH console. |
Services
The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network. A service in this PaaS environment is a horizontally scalable application that adds instances of itself when demand increases, and frees instances of itself when demand decreases.
Databases
The Cisco APIC-EM supports two databases: application and Grapevine. The application database is used for the application and external networking data. The Grapevine database is used for the Grapevine and internal network data. Both databases are replicated in a multi-host environment for scale and high availability.
Networks
The Cisco APIC-EM architecture requires both external and internal networks to operate:
-
The external network(s) consists of the network hosts, devices, and NTP servers, as well as providing access to the northbound REST APIs. The external network(s) also provides access to the controller GUI.
-
The internal network consists of the Grapevine roots and clients that are connected to and communicate with each other (service to service). For forwarding to or receiving traffic from the larger external network (that consists of the connected devices and hosts, as well as NTP servers), all inbound and outbound traffic for this internal network passes through a subset of clients connected to the external network. The internal network is isolated and nonroutable from the external network(s), as well as any other internal network.
Network Connections and NICs
The network adapters (NICs) on the host (physical or virtual) are connected to the following external networks:
-
Internet (network access required for Make A Wish requests, Telemetry, and trustpool updates)
-
Network with NTP server(s)
-
Network with devices that are to be managed by the Cisco APIC-EM
Note | The Cisco APIC-EM should never be directly connected to the Internet. It should not be deployed outside of a NAT configured or protected datacenter environment. |
Troubleshooting an Unsuccessful Installation
After performing an installation using the configuration wizard, you should have received an ‘installation was successful’ message. If you did not receive this message and your installation was unsuccessful, you can perform the following troubleshooting procedures.
- Confirming that Core Services are Running
- Confirming the Multi-Host Cluster Configuration Values
- Resolving Access to the Cisco APIC-EM GUI
Confirming that Core Services are Running
You should have attempted to deploy the Cisco APIC-EM following the procedure described in this guide.
Confirming the Multi-Host Cluster Configuration Values
You should have attempted to deploy the Cisco APIC-EM following the procedure described in this guide.
| Step 1 | Using a Secure
Shell (SSH) client, log into the host (physical or virtual) with the IP address
that you specified using the configuration wizard.
| ||
| Step 2 | When prompted, enter your Linux username ('grapevine') and password for SSH access. | ||
| Step 3 | Enter the
following command to display the multi-host configuration.
$ grape root display Command output similar to the following should appear.
ROOT PROPERTY VALUE
----------------------------------------------------------------------
4cbe3972-9872-4771-800d-08c89463f1eb hostname root-1
4cbe3972-9872-4771-800d-08c89463f1eb interfaces [{'interface': 'eth0', 'ip': '209.165.200.10', 'mac': '00:50:56:100:d2:14', 'netmask': '255.255.255.0'}, {'interface': 'eth1', 'ip': '209.165.200.10', 'mac': '00:50:56:95:5c:18', 'net mask': '255.255.255.0'}, {'interface': 'grape-br0', 'ip': '209.165.200.11', 'mac': 'ba:ed:c4:19:0d:77', 'netmask': '255.255.255.0'}]
4cbe3972-9872-4771-800d-08c89463f1eb is_alive True
4cbe3972-9872-4771-800d-08c89463f1eb last_heartbeat Wed Sep 09, 2015 11:02:52 PM (just now)
4cbe3972-9872-4771-800d-08c89463f1eb public_key ssh-rsa
c2EAAAADAQABAAABAQDYlyCfidke3MTjGkzsTAu73MtG+lynFFvxWZ4xVIkDkhGC7KCs6XMhORMaABb6
bU4EX/6osa4qyta4NYaijxjL6GL6kPkSBZiEKcUekHCmk1+H+Ypp5tc0wyvSpe5HtbLvPicLrXHHI/TS
Fsa+gLPqg55TflX8RH3i8dHf1Zwq6v4nHVryjAzMXeFYnFHST9e0P62QnkAGh29ktxUpS3fKua9iCVIE
V44t+VvtFaLurG9+FW/ngZwGrR/N0ZJZl6/MQTN3 grapevine@grapevine-root
4cbe3972-9872-4771-800d-08c89463f1eb root_id 4cbe3972-9872-4771-800d-08c89463f1eb
4cbe3972-9872-4771-800d-08c89463f1eb root_index 0
4cbe3972-9872-4771-800d-08c89463f1eb root_version 0.3.0.958.dev140-gda6a16
4cbe3972-9872-4771-800d-08c89463f1eb vm_password ******
(grapevine)
#
ROOT PROPERTY VALUE
----------------------------------------------------------------------
4cbe3972-9872-4771-800d-08c89463f1eb hostname root-2
4cbe3972-9872-4771-800d-08c89463f1eb interfaces [{'interface': 'eth0', 'ip': '209.165.200.101, 'mac': '00:50:56:100:d2:14', 'netmask': '255.255.255.0'}, {'interface': 'eth1', 'ip': '209.165.200.11', 'mac': '00:50:56:95:5c:18', 'net mask': '255.255.255.0'}, {'interface': 'grape-br0', 'ip': '209.165.200.11', 'mac': 'ba:ed:c4:19:0d:77', 'netmask': '255.255.255.0'}]
4cbe3972-9872-4771-800d-08c89463f1eb is_alive True
4cbe3972-9872-4771-800d-08c89463f1eb last_heartbeat Wed Sep 09, 2015 11:02:52 PM (just now)
4cbe3972-9872-4771-800d-08c89463f1eb public_key ssh-rsa
c2EAAAADAQABAAABAQDYlyCfidke3MTjGkzsTAu73MtG+lynFFvxWZ4xVIkDkhGC7KCs6XMhORMaABb6
bU4EX/6osa4qyta4NYaijxjL6GL6kPkSBZiEKcUekHCmk1+H+Ypp5tc0wyvSpe5HtbLvPicLrXHHI/TS
Fsa+gLPqg55TflX8RH3i8dHf1Zwq6v4nHVryjAzMXeFYnFHST9e0P62QnkAGh29ktxUpS3fKua9iCVIE
V44t+VvtFaLurG9+FW/ngZwGrR/N0ZJZl6/MQTN3 grapevine@grapevine-root
4cbe3972-9872-4771-800d-08c89463f1eb root_id 4cbe3972-9873-4771-800d-08c89463f1eb
4cbe3972-9872-4771-800d-08c89463f1eb root_index 0
4cbe3972-9872-4771-800d-08c89463f1eb root_version 0.3.0.958.dev140-gda6a16
4cbe3972-9872-4771-800d-08c89463f1eb vm_password ******
(grapevine)
The following data is displayed by this command:
| ||
| Step 4 | If any of the
fields in the command output appear incorrect, enter the root cause analysis
(rca) command.
$ rca | ||
| Step 5 | Send the tar file created by the rca command procedure to Cisco support for assistance in resolving your issue. |
Resolving Access to the Cisco APIC-EM GUI
You access the Cisco APIC-EM GUI by entering the IP address that you configured for the network adapter using the configuration wizard. This IP address connects to the external network. Enter the IP address in your browser in the following format:
https://IP address
If you are unable to access the Cisco APIC-EM GUI, you must access the Grapevine developer console to check for faulty or failed services.
The Grapevine developer console allows you to monitor the health of your Cisco APIC-EM deployment. The Grapevine developer console is part of the Service Elasticity Platform (Grapevine). You access the Grapevine developer console by entering the IP address that you configured for the network adapter using the configuration wizard, but with a specific port number (14141).
For a multi-host cluster, you do not have to log into each host. In a multi-host cluster, you get a single, consolidated view of all of the services running on all three hosts. Multiple instances of services running on different hosts will appear in the Grapevine developer console in a multi-host cluster.
Note | A default idle timeout of 1 hour has been set for the Grapevine developer console. You will be automatically logged out after 1 hour of inactivity on the Grapevine developer console. |
To access the Grapevine developer console to check for faulty or failed services, follow the procedure described below.
You should have attempted to deploy the Cisco APIC-EM following the procedure described in this guide.
| Step 1 | Access the
Grapevine developer console by opening a browser window and entering the IP
address that you configured for the network adapter using the configuration
wizard.
For example, enter the following IP address with required port number: https://external network IP address:14141 | ||
| Step 2 | Enter your
administrative username and password when prompted.
The administrative username and password were configured by you using the configuration wizard. After you enter the username and password, the Grapevine developer console appears. Each installed service with its version number appears in the console in an alphabetical list. Below each service is a square icon that represents the health of the service. Services that have been installed and are operational are green. Faulty or failed services are red. | ||
| Step 3 | Scroll down the
list and confirm that the following services are installed and running for your
deployment:
Note whether the service is operational or faulty. | ||
| Step 4 | Review the console Tasks tab below the list of services for any error messages about any faulty services. Note the reason given for the faulty or failed service. | ||
| Step 5 | Contact Cisco support with the following information: |
Troubleshooting the Configuration
Perform the following procedures to troubleshoot possible configuration issues.
- Updating the Configuration Using the Wizard
- Resetting the Cisco APIC-EM
- Restoring the Controller to the Factory Default
- Creating a Support File for the Cisco APIC-EM
Updating the Configuration Using the Wizard
You can troubleshoot the Cisco APIC-EM deployment by running the configuration wizard a second time and updating any earlier configuration entries. The configuration wizard saves and displays your previous configuration settings, so you do not have to reenter them.
Note | When performing this procedure, controller downtime occurs. For this reason, we recommend that you perform this procedure during a maintenance time period. For information about changing settings for a multi-host configuration, see Changing the Settings in a Multi-Host Cluster Using the Wizard |
You have deployed Cisco APIC-EM using the procedures described in this guide.
| Step 1 | Using a Secure
Shell (SSH) client, log into the host (physical or virtual) with the IP address
that you specified using the configuration wizard.
| ||
| Step 2 | When prompted, enter your Linux username ('grapevine') and password for SSH access. | ||
| Step 3 | Restart the
configuration wizard using the following command.
$ config_wizard | ||
| Step 4 | Review the
current configuration values in the configuration wizard and click
next>>, until you access the specific step
where you wish to update your previous configuration entry.
For example, if you need to enter a new NTP server IP address, click next >> until you get to the NTP SERVER SETTINGS screen. | ||
| Step 5 | Update the
value that was previously entered in the configuration wizard and is currently
displayed.
For example, you can update the NTP server settings by entering a new IP address. | ||
| Step 6 | Click next>> until the last step of the configuration wizard process. | ||
| Step 7 | Click proceed>> to have the configuration wizard save and apply your configuration changes to your Cisco APIC-EM deployment. |
Resetting the Cisco APIC-EM
You can troubleshoot a Cisco APIC-EM deployment by resetting the controller back to configuration values that were originally set using the configuration wizard the first time. A reset of the controller is helpful, when the controller has gotten itself into an unstable state and other troubleshooting activities have not resolved the situation.
In a multi-host environment, you need to perform this procedure on only a single host. After performing this procedure on a single host, the other two hosts will be automatically reset.
You have deployed the Cisco APIC-EM using the procedures described in this guide.
| Step 1 | Using a Secure
Shell (SSH) client, log into the host (physical or virtual) with the IP address
that you specified using the configuration wizard.
| ||
| Step 2 | When prompted, enter your Linux username ('grapevine') and password for SSH access. | ||
| Step 3 | Navigate to the bin directory on the Grapevine root. The bin directory contains the grapevine scripts. | ||
| Step 4 | Enter the
reset_grapevine
command at the prompt to run the reset grapevine script.
$ reset_grapevine The reset_grapevine command returns the configuration settings back to values that you configured when running the configuration wizard for the first time. The configuration settings are saved to a .JSON file. This .JSON file is located at: /etc/grapevine/controller-config.json. The reset_grapevine command uses the data in the controller-config.json file to return to the earlier configuration settings, so do not delete this file. If you delete this file, you must run the configuration wizard again and reenter your configuration data. You are then prompted to reenter your Grapevine password. | ||
| Step 5 | Enter your
Grapevine password a second time.
[sudo] password for grapevine:******** You are then prompted to delete all virtual disks The virtual disks are where the Cisco APIC-EM database resides. For example, data about devices that the controller discovered are saved on these virtual disks. If you enter yes (y), all of this data is deleted. If you enter no (n), then the new cluster will come up populated with your existing data once the reset procedure completes. | ||
| Step 6 | Enter
n to prevent the deletion all of the virtual disks.
THIS IS A DESTRUCTIVE OPERATION Do you want to delete all VIRTUAL DISKS in your APIC-EM cluster? (y/n):n You are then prompted to delete all Cisco APIC-EM authentication timeout policies, user password policies, and user accounts other than the primary administrator account. | ||
| Step 7 | Enter
n to prevent the deletion of all authentication
timeout policies, user password policies, and user accounts other than the
primary administrator account.
THIS IS A DESTRUCTIVE OPERATION Do you want to delete authentication timeout policies, user password policies, and Cisco APIC-EM user accounts other than the primary administrator account? (y/n): n You are then prompted to delete any imported certificates. | ||
| Step 8 | Enter
n to prevent the deletion of any imported
certificates.
THIS IS A DESTRUCTIVE OPERATION Do you want to delete the imported certificates? (y/n): n You are then prompted to delete any backups. | ||
| Step 9 | Enter
n to prevent the deletion of any backups.
THIS IS A DESTRUCTIVE OPERATION Do you want to delete the backups? (y/n): n The controller then resets itself with the configuration values that were originally set using the configuration wizard the first time. When the controller is finished resetting, you are presented with a command prompt from the controller. | ||
| Step 10 | Using the Secure Shell (SSH) client, log out of the host. |
Restoring the Controller to the Factory Default
In certain situations, you may want to restore the Cisco APIC-EM to its original factory default settings. For example, if your controller appliance is being replaced or simply has an undesirable configuration that needs to be completely removed. Under these circumstances, you can restore the controller to its factory defaults and then proceed to reconfigure it as a new controller.
This procedure describes how to a restore the factory defaults to the controller.
 Caution | This procedure shuts down both the Cisco APIC-EM and the host (physical or virtual) on which it resides. At the end of this procedure, you will need to access the host and restart it. |
You have already deployed the Cisco APIC-EM using the procedures described in this guide.
You have access to the Cisco APIC-EM using either a physical console or a Telnet connection.
| Step 1 | Using a Secure
Shell (SSH) client, log into the host (physical or virtual) with the IP address
that you specified using the configuration wizard.
| ||
| Step 2 | When prompted, enter your Linux username ('grapevine') and password for SSH access. | ||
| Step 3 | Enter the
reset_grapevine factory command at the prompt.
$ reset_grapevine factory | ||
| Step 4 | Enter your Linux
grapevine password a second time to start the reset process.
$ sudo password for grapevine ********* After entering this command a warning appears that the reset_grapevine factory command will shut down the controller. You are then prompted to confirm your desire to run the reset_grapevine factory command. | ||
| Step 5 | Enter
Yes to confirm that you want to run the
reset_grapevine factory command.
The controller then performs the following tasks: |
What to Do Next
Perform the following tasks:
Creating a Support File for the Cisco APIC-EM
You can troubleshoot the Cisco APIC-EM deployment by creating a support file. This support file consists of logs, configuration files, and command output. After you create this support file, you can then email it to Cisco support for assistance.
You have deployed the Cisco APIC-EM using the procedures described in this guide.
| Step 1 | Using a Secure
Shell (SSH) client, log into the host (physical or virtual) with the IP address
that you specified using the configuration wizard.
| ||
| Step 2 | When prompted, enter your Linux username ('grapevine') and password for SSH access. | ||
| Step 3 | Navigate to the bin directory on the host. The bin directory contains the grapevine scripts. | ||
| Step 4 | To create the
support file, enter the
rca command in this directory.
$ rca mkdir: created directory '/tmp grapevine-rca-2014-05-27_04-22-20-PM_PDT-0700' ---------------------------------------------------------------------- RCA package created On Tues May 27 16:22:20 PDT 2014 ---------------------------------------------------------------------- [INFO] Generating log for 'vmstat 1 10' [sudo] password for grapevine: The rca command runs a root cause analysis script that creates a tar file that contains log files, configuration files, and the command output. |
What to Do Next
Send the tar file created by this procedure to Cisco support for assistance in resolving your issue.
Troubleshooting Services
You can use the troubleshooting procedures in this section to assist in troubleshooting service issues.
You can use the Grapevine developer console to review the status of a service instance (active, inactive, or failed) to assist in identifying a service problem. You can also use the Grapevine developer console to check the service logs to identify any problems. After identifying a failed or faulty service, you can then manually grow and/or harvest service instances to try to resolve the problem.
- Grapevine Developer Console
- Logging into the Grapevine Developer Console
- Reviewing the Service Version, Status, and Logs
- Monitoring Services and Clients Using the CLI
Grapevine Developer Console
The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network. A service in this PaaS environment is a horizontally scalable application that adds instances of itself when increasing loads occur on a client within the network. You use the Grapevine developer console to troubleshoot these services. The Grapevine developer console tools were bundled with the deployment files and installed when you first deployed the Cisco APIC-EM.
Note | For a multi-host cluster, you do not have to log into each host to view the Grapevine developer console. In a multi-host cluster, you get a single, consolidated view of all of the services running on all three hosts. |
The Grapevine developer console provides the following windows and functionality:
-
Overview—Provides a list of services with information about their version and status. You can add or remove services in this window.
-
Clients—Provides detailed client information in this window.
-
Hosts—Provides detailed host information in this window.
-
Waiting Queue—Provides information about the waiting queue.
-
Services—Provides detailed service information. You can add or remove services in this window.
-
Logs—Provides detailed task, instance, and client logs
Note | You cannot access the Grapevine developer console as a Linux root user. You can only access the Grapevine developer console using the administrator username and password that you configured during the deployment process. |
Logging into the Grapevine Developer Console
You are able to perform the following tasks using the Grapevine developer console:
Note | Only advanced users should access the Grapevine developer console to perform any troubleshooting tasks for the services. |
You must have successfully deployed the Cisco APIC-EM and it must be operational.
| Step 1 | Access the
Grapevine developer console by opening a browser window and entering the IP
address that you configured for the network adapter using the configuration
wizard.
For example, enter the following IP address with required port number: https://external network IP address:14141 | ||
| Step 2 | Enter your
administrative username and password when prompted.
The administrative username and password were configured by you using the configuration wizard. The console for the Elastic Service Platform (Grapevine) appears. |
What to Do Next
Proceed to review any of the service versions, status, and logs for any troubleshooting purpose.
Reviewing the Service Version, Status, and Logs
You are able to perform the following tasks using the Grapevine developer console:
Note | Only advanced users should access the console to perform the tasks described in this procedure or attempt to troubleshoot the services. |
You must have successfully deployed the Cisco APIC-EM and it must be operational.
| Step 1 | Access
the Grapevine developer console by opening a browser window and entering the IP
address that you configured for the network adapter using the configuration
wizard.
For example, enter the following IP address with required port number: https://external network IP address:14141 | ||
| Step 2 | Enter
your administrative username and password when prompted.
The administrative username and password were configured by you using the configuration wizard. The console for the Elastic Service Platform (Grapevine) appears. | ||
| Step 3 | Review the
status of each service listed in the
Overview window in the console.
Each service is represented by a square. A green colored square represents an active instance of the service, and a red colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running). In a multi-host environment, a service may be represented by two green colored squares, indicating that the service is running on two different hosts within your cluster. Place your cursor over each square to view the host that the service is running on. | ||
| Step 4 | Review the
version of each service in the
Overview window in the console.
The version is located in the header of each listed service. | ||
| Step 5 | Review the service logs by clicking a specific active instance of a service (green square icon) and then viewing the Instance or Client logs located at the bottom of the window. |
What to Do Next
When finished with the Grapevine developer console, click the Logout button to log out of the console.
Monitoring Services and Clients Using the CLI
In addition to the developer console, a command-line interface on the host is also provided for troubleshooting purposes. This CLI allows you to monitor the health of the Cisco APIC-EM from the command line.
You have deployed the Cisco APIC-EM using the procedures described in this guide.
| Step 1 | Using a Secure
Shell (SSH) client, log into the host (physical or virtual) with the IP address
that you specified using the configuration wizard.
| ||
| Step 2 | When prompted, enter your Linux username ('grapevine') and password for SSH access. | ||
| Step 3 | Display all of
the
Cisco APIC-EM
services currently installed by entering the
grape
service display command. Key data about each service is then
displayed separated by hash marks.
$ grape service display
...
#
rabbitmq config {}
rabbitmq core_service True
rabbitmq enabled False
rabbitmq endpoint_config {u'default':
{u'backend_protocol': u'amqp', u'backend_path': u'',
u'frontend_protocol': u'', u'frontend_path': u'', u'frontend_port':
0, u'backend_port': 5672}}
rabbitmq kill_as_group True
rabbitmq max_instances 1
rabbitmq min_instances 0
rabbitmq priority 1
rabbitmq queue_config {u'queues': [],
u'bindings': [], u'exchanges': []}
rabbitmq requirements {u'template_id':
u'default', u'persistent_disk': False}
rabbitmq run_as_group grapevine
rabbitmq run_as_user grapevine
rabbitmq service_type rabbitmq
rabbitmq spare_count 0
rabbitmq start_secs 10
rabbitmq static_load 10
rabbitmq status_interval 60
rabbitmq stop_as_group True
rabbitmq stop_signal TERM
rabbitmq version 1.0.0
#
...
| ||
| Step 4 | Display
running instances of the
Cisco APIC-EM
services by entering the
grape
instance display command. Key data about running instances of
service is then displayed separated by hash marks.
$ grape instance display
...
#
4c4c83db-2da6-4a04-9af6-96c8dac692d1 client_id
4c4c83db-2da6-4a04-9af6-96c8dac692d1
4c4c83db-2da6-4a04-9af6-96c8dac692d1 endpoint_config
{u'default': {u'backend_port': 5672, u'backend_protocol': u'amqp'}}
4c4c83db-2da6-4a04-9af6-96c8dac692d1 interfaces
[{'interface': 'eth0', 'ip': '192.168.0.1', 'mac': '00:50:56:9f:6c:c4'},
{'interface': 'eth1', 'ip': '172.16.0.15', 'mac': '00:50:56:9f:71:46'}]
4c4c83db-2da6-4a04-9af6-96c8dac692d1 is_error None
4c4c83db-2da6-4a04-9af6-96c8dac692d1 service_type rabbitmq
4c4c83db-2da6-4a04-9af6-96c8dac692d1 state running
4c4c83db-2da6-4a04-9af6-96c8dac692d1 task_id None
4c4c83db-2da6-4a04-9af6-96c8dac692d1 timestamp
Fri Oct 03, 2014 02:05:43 PM (4 days ago)
4c4c83db-2da6-4a04-9af6-96c8dac692d1 version 1.0.0
#
...
| ||
| Step 5 | Display all
Grapevine clients currently running in
Cisco APIC-EM
by entering the
grape
client display command. Key data about each client is then
displayed separated by hash marks.
$ grape client display
CLIENT PROPERTY VALUE
----------------------------------------------------------------------
ae63a6c1-a946-4df7-a68d-33227eed8134 client_id ae63a6c1-a946-4df7-a68d-33227eed8134
ae63a6c1-a946-4df7-a68d-33227eed8134 client_version 0.1.0.212.dev633-gf7e21de
ae63a6c1-a946-4df7-a68d-33227eed8134 interfaces [{'interface': 'eth0', 'ip': '192.168.0.32',
'mac': '00:50:56:9f:3a:90'}]
ae63a6c1-a946-4df7-a68d-33227eed8134 is_alive True
ae63a6c1-a946-4df7-a68d-33227eed8134 last_heartbeat Wed Oct 08, 2014 10:22:50 AM (15 secs ago)
ae63a6c1-a946-4df7-a68d-33227eed8134 template_id default
ae63a6c1-a946-4df7-a68d-33227eed8134 vm_id ce0a634a-5475-4450-9dce-f3217d855ac4
#
...
All clients should have a is alive property of True. |
Troubleshooting Passwords
Perform the following procedures to troubleshoot password issues.
- Performing Password Recovery with an Existing Administrator
- Performing Password Recovery with No Existing Administrator
- Performing Password Recovery for the Linux Grapevine User Account
Performing Password Recovery with an Existing Administrator
To perform password recovery for a user (administrator, installer or observer) where there exists at least one controller administrator (ROLE_ADMIN) user account, take the following steps:
-
Contact the existing administrator to set up a temporary password for the user that requires password recovery.
Note
The administrator can set up a temporary password by deleting the user's account and then recreating it with the lost password. The user can then log back into the controller to regain access and change the password once again to whatever he or she desires.
-
The user then needs to log into the controller with the temporary password and change the password.
Note
Passwords are changed in the controller GUI using the Change Password window. For information about changing passwords, see Chapter 4, Managing Users and Roles in the Cisco Application Policy Infrastructure Controller Enterprise Module Configuration Guide.
Performing Password Recovery with No Existing Administrator
The following procedure describes how to perform password recovery where there exists only one controller administrator (ROLE_ADMIN) user account and this account cannot be successfully logged into.
Note | We recommend that you create at least two administrator accounts for your deployment. With two administrator accounts, if one account is locked for whatever reason then the other account can be used to unlock that locked account. |
| Step 1 |
If there are no other existing administrator (ROLE_ADMIN) user
accounts, use an SSH client from your terminal to log into the host (physical
or virtual) with the IP address that you specified using the configuration
wizard.
| ||
| Step 2 | Enter the Linux username ('grapevine') and password when prompted. | ||
| Step 3 | On the
console, enter the following command on the Grapevine root.
$ config_wizard This command starts up the Cisco APIC-EM configuration process. | ||
| Step 4 | Choose the <Create a new APIC-EM cluster> option. | ||
| Step 5 | Proceed through the configuration process until reaching the step to configure the APIC-EM ADMIN USER SETTINGS. | ||
| Step 6 | Specify a new administrator user password. | ||
| Step 7 | Reenter the new administrator user password for confirmation. | ||
| Step 8 | Proceed through
the configuration wizard and its process until completion.
This final step will bring down the cluster and then bring it back up again (similar to running the reset_grapvine command). |
Performing Password Recovery for the Linux Grapevine User Account
You can use the following procedure to recover from the loss of the Linux grapevine user password. This procedure reconfigures the Linux grapevine user password that is required for accessing the host's Linux operating system.
You should be logged into the host (physical or virtual) using a Linux console to access the Linux kernel.
| Step 1 | Reboot the host (physical or virtual) while logged into the Linux console. | ||
| Step 2 | Press "e" upon
seeing the GNU GRUB menu to edit the boot commands.
| ||
| Step 3 | Search for the
line in the GNU GRUB menu output that begins with "linux" and change “ro” to
“rw”, and append "init=/bin/bash" to that line.
For example, search for this line: linux /vmlinuz-3.13.0-24-generic root=/dev/mapper/grapevine--vg-root ro cgroup_enable=memory swapaccount=1 quiet sqlash $vt_handoff And change it to this line: linux /vmlinuz-3.13.0-24-generic root=/dev/mapper/grapevine--vg-root rw cgroup_enable=memory swapaccount=1 quiet sqlash $vt_handoff init=/bin/bash | ||
| Step 4 | Press
Ctrl-x or the
F10 key to proceed with the boot process.
At this point, the host will boot up in root mode. You can now enter the Linux passwd command to reset the password for the Linux grapevine user. | ||
| Step 5 | Enter the Linux
passwd
command to reset the password for the Linux grapevine user.
$ passwd grapevine
| ||
| Step 6 | When prompted, enter a new Linux grapevine password. | ||
| Step 7 | When prompted, confirm the new Linux grapevine password by entering it a second time. | ||
| Step 8 | Enter the
following
reboot
command to reboot the system.
$ /sbin/reboot -f The system reboots and will start up with new configuration and password. At the end of the reboot process, you are presented with the GNU GRUB menu. | ||
| Step 9 | Press Enter to boot up in the Ubuntu OS. | ||
| Step 10 | After booting up
in the Ubuntu OS, log back into the host by entering your Linux grapevine
username and password.
| ||
| Step 11 | Restart the
configuration wizard using the following command.
$ config_wizard Proceed through the configuration wizard process by clicking next>> and accepting the pre-configured values until you reach the LINUX USER SETTINGS step.
| ||
| Step 12 | When prompted
to enter values for the
LINUX
USER SETTINGS, enter the new Linux grapevine password that you
created earlier in step 6.
| ||
| Step 13 | Click
next>> and continue through the configuration
wizard process, until the last step of this process.
| ||
| Step 14 | At the end of the configuration wizard process, click proceed>> to have the configuration wizard save and apply your configuration changes to the Cisco APIC-EM. |
Troubleshooting Commands
You can issue commands on both Grapevine root and clients to troubleshoot the Cisco APIC-EM.
Root Commands
The following table describes commands that you can issue on the Grapevine root to troubleshoot the Cisco APIC-EM.
Note | Enter the grape help command on Grapevine root to view the available commands. |
|
Root Command |
Description |
||
|---|---|---|---|
|
grape backup delete |
Deletes the existing controller backup file. If you are unable to back up your current controller database, then you may want to delete it and restore an earlier smaller controller backup file. |
||
|
grape capacity_plugin display |
Displays the current settings used by Grapevine. |
||
|
grape client display |
Displays the number of Grapevine clients that are currently running. This command also displays any clients that have faulted. |
||
|
grape client status |
Displays the status of the Grapevine clients. |
||
|
grape host evacuate host_id |
The grape host evacuate command harvests all of the services on the host where it is issued. If you issue this command on a host in a multi-host cluster, then the services are harvested and transferred to the remaining two hosts in the cluster. If you issue this command on a host in a single host configuration, all the services are harvested. |
||
|
grape host status host_id |
The grape host status command displays all of the services on the host where it is issued. |
||
|
grape instance display |
Displays the Cisco APIC-EM services that are currently running. |
||
|
grape instance status |
Displays the status of the Cisco APIC-EM services. |
||
|
grape network display |
Displays the current external network configuration used by both Grapevine and the Cisco APIC-EM including: |
||
|
grape release display current |
Displays the versions of the Cisco APIC-EM services that are currently running. |
||
|
grape release display latest |
Displays the latest available versions of the Cisco APIC-EM services. |
||
|
grape release update force |
Updates a service for Cisco APIC-EM. Use this command to force a service update. |
||
|
grape root display |
Displays the Grapevine root properties, including: hostname, interfaces, root ID, and version. |
||
|
grape service display |
Displays the Cisco APIC-EM services that are currently installed. |
||
|
grape update_service display |
Displays the current automatic services update configuration. Additional fields in the command output indicate the last time the Cisco cloud was polled for updates (last_connect_time), as well as the Cisco cloud status(last_connect_status). For example, whether the Cisco cloud is reachable, unreachable, etc. |
||
|
grape version |
Displays the version of Grapevine that the Grapevine root is running. |
||
|
rca |
The rca command runs a root cause analysis script that creates a tar file that contains log files, configuration files, and the command output. After running this command and creating the tar file, you can send the file to Cisco support for assistance in resolving an issue. |
||
|
reset_grapevine |
The reset_grapevine command returns the configuration settings back to values that you configured when running the configuration wizard for the first time. The configuration settings are saved to a .JSON file. This .JSON file is located at: $ /etc/grapevine/controller-config.json. |
||
|
reset_grapevine factory |
The reset_grapevine factory command returns the configuration settings back to their factory defaults.
|
||
|
securityutil openport |
Displays the open ports on the Grapevine root and clients. An open port is any TCP or UDP port, for which a listener exists on one or more IP addresses. |
||
|
sudo service grapevine status |
Determines the status of the Grapevine root core services and whether all of the root core services are running. |
Client Commands
The following table describes commands that you can issue on the Grapevine client to troubleshoot the Cisco APIC-EM.
|
Client Command |
Description |
|---|---|
|
grape version |
Displays the version of Grapevine that the Grapevine client is running. |
|
sudo service grapevine status |
Determines the status of the Grapevine client core services and whether all of the client core services are running. |
Troubleshooting Log Files
You can use log files to troubleshoot the Cisco APIC-EM. Log files are stored on both the Grapevine root and clients.
Note | Logging files are persistent across any system failures and software updates for change management. |
Root Log Files
The following table lists log files located on the Grapevine root that can be used to troubleshoot the Cisco APIC-EM.
|
Directory |
Filename |
Description |
|---|---|---|
|
/var/log/ |
boot.log |
This log file provides you with the details of the boot process and any errors that may occur during this process. Use this log file to troubleshoot any of the following problems: |
|
/var/log/ |
config_wizard.log |
Use this log file to determine if there were any errors during the initial Grapevine configuration and deployment. |
|
/var/log/ |
grapevine_manager.log |
Use this log file to determine if there were any errors during a manual file update using the Update Settings fields in the controller's UI. |
|
/var/log/grapevine |
supervisord.log |
Use this log file to determine whether any of the Grapevine root core services unexpectedly expired. |
|
/var/log/grapevine |
cassandra.log |
Use this log file to determine whether the Grapevine database is healthy. |
|
/var/log/grapevine |
grapevine_root.log |
Use this log file to troubleshoot any of the following problems: |
|
/var/log/grapevine |
grapevine_capacity_manager.log |
Use this log file to determine why Grapevine failed to grow or harvest a Grapevine client. |
Client Log Files
The following table lists log files located on the Grapevine client that can be used to troubleshoot the Cisco APIC-EM.
|
Directory |
Filename |
Description |
|---|---|---|
|
/var/log |
boot.log |
Use this log file to determine the following: |
|
/var/log/grapevine |
supervisord.log |
Use this log file to determine if any of the Grapevine client core services unexpectedly died? |
|
/var/log/grapevine |
grapevine_client.log |
This log file provides you with the details of the Grapevine client daemon bootstrap process and any errors that may occur during this process. Use this log file to determine the following: |
|
/var/log/grapevine/services |
service-name/version/service-name.log |
Use this log file to determine why did a service fail to perform some operation? |
Feedback