- Preface
- New and Changed Information
- Overview
- Troubleshooting Matrix
- Troubleshooting an Installation or Update
- Troubleshooting the Configuration
- Troubleshooting Cisco APIC-EM Single and Multi-Host
- Troubleshooting Services Using System Health
- Troubleshooting Services Using the Controller Admin Console
- Troubleshooting Using the Logs
- Troubleshooting Passwords
- Troubleshooting Commands
- Troubleshooting Log Files
- Contacting the Cisco Technical Assistance Center
- Index
Troubleshooting
Services Using the Controller Admin Console
The following procedures may be used to troubleshoot services using the controller admin console:
- About Cisco APIC-EM Services
- Controller Admin Console
- Reviewing the Service's Version, Status, and Logs
- Removing a Service Instance
- Creating a Service Instance
About Cisco APIC-EM Services
The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network, using Grapevine as an Elastic Services platform to support the controller's infrastructure and services. A service in this PaaS environment is a horizontally scalable application that adds instances of itself when demand increases, and frees instances of itself when demand decreases.
The Cisco APIC-EM controls elasticity at the service level, rather than at the Grapevine client level.
Service Managers and Monitors
The Cisco APIC-EM services that run on the Grapevine Elastic Services Platform provide the controller with its functionality. The Grapevine Elastic Services Platform consists the following components:
-
Grapevine root—Handles all policy management in regards to service updates, as well as the service lifecycle for both itself and the Grapevine client.
-
Grapevine client—Location where the supported services run.
After installation, service functionality is enabled using the following managers and monitors:
-
Grapevine Root
-
Service manager—Starts, stops, and monitors service instances across the Grapevine clients.
-
Capacity manager—Provides on-demand capacity to run the services.
-
Load monitor—Monitors the load and health of services across the Grapevine clients.
-
Service catalog—Repository of service bundles that can be deployed on the Grapevine clients.
-
-
Grapevine Client
Service Features
The Cisco APIC-EM provides the following service features:
- Adding capacity on an existing client—When a service load exceeds a specified threshold on a client, the controller can request another service instance to start on a second, preexisting client.
- Adding capacity on a newly instantiated client—When a service load exceeds a specified threshold on a client, the controller can request a new client to be instantiated and then start another service instance on this client.
-
Allows automatic scaling of services—As the service load increases, the controller instantiates additional service instances in response. As the service load decreases, the controller tears down the number of instances in response.
-
Resiliency for services—When a service fails, the controller starts a replacement instance. The controller then ensures that the service’s minimum instance count requirements are maintained.
Services
The following is a list of default Cisco APIC-EM services for the Cisco APIC-EM Release 1.5.x.
-
access-policy-programmer-service
-
apic-em-event-service
-
apic-em-inventory-manager-service
-
apic-em-jboss-ejbca
-
apic-em-network-programmer-service
-
apic-em-pki-broker-service
-
cas-service
-
cassandra
-
election-service
-
file-service
-
grapevine
-
grapevine-coordinator-service
-
grapevine-log-collector
-
grouping-service
-
identity-manager-pxgrid-service
-
nbar-policy-programmer-service
-
network-poller-service
-
node-ui
-
pnp-service
-
policy-analysis-service
-
policy-manager-service
-
postgres
-
qos-lan-policy-programmer-service
-
qos-monitoring-service
-
qos-policy-programmer-service
-
rabbitmq
-
rbac-service
-
reverse-proxy
-
router
-
scheduler-service
-
task-service
-
telemetry-service
-
topology-service
![]() Note | The Cisco APIC-EM services running on your controller is dependent upon the applications installed and enabled on the host. |
Controller Admin Console
The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network. A service in this PaaS environment is a horizontally scalable application that adds instances of itself when increasing loads occur on a client within the network. You use the Controller Admin console to manage and troubleshoot these services. The Controller Admin console and its tools were bundled with the deployment files and installed when you first deployed the Cisco APIC-EM.

![]() Note | For a multi-host cluster, you do not have to log into each host to view the Controller Admin console. In a multi-host cluster, you get a single, consolidated view of all of the services running on all three hosts. |
The Controller Admin console is directly accessible from the controller's GUI. To access this console, click on Settings (gear) icon in the menu bar at the top of the controller's GUI, then click on the System Administration link in the drop down menu.
The Controller Admin console provides the following windows and functionality:
-
Overview—Provides a list of services with information about their version and status. You can add or remove services in this window.
-
Clients—Provides detailed client information in this window.
-
Hosts—Provides detailed host information in this window.
-
Waiting Queue—Provides information about the waiting queue.
-
Services—Provides detailed service information. You can add or remove services in this window.
-
Logs—Provides detailed task, instance, and client logs.
Reviewing the Service's Version, Status, and Logs
You are able to perform the following tasks using the Controller Admin console:
![]() Caution | Only advanced users should access the Controller Admin console to perform the tasks described in this procedure or attempt to troubleshoot the services. |

You must have successfully installed the Cisco APIC-EM and it must be operational.
You must have administrator (ROLE_ADMIN) permissions and either access to all resources (RBAC scope set to ALL) or an RBAC scope that contains all of the resources that you want to group. For example, to create a group containing a specific set of resources, you must have access to those resources (custom RBAC scope set to all of the resources that you want to group).
Removing a Service Instance
You are able to remove or harvest a service instance by using the Controller Admin console. The Controller Admin console tools are bundled within the ISO image and installed when you first deploy the Cisco APIC-EM.
![]() Caution | Only advanced users should access the Controller Admin console to perform the tasks described in this procedure or attempt to troubleshoot the services. |

You must have successfully installed the Cisco APIC-EM and it must be operational.
You must have administrator (ROLE_ADMIN) permissions and either access to all resources (RBAC scope set to ALL) or an RBAC scope that contains all of the resources that you want to group. For example, to create a group containing a specific set of resources, you must have access to those resources (custom RBAC scope set to all of the resources that you want to group).
Step 1 | In the Home window, click either admin or the Settings icon (gear) at the top right corner of the screen. | ||
Step 2 | Click the System Administration link from the drop-down menu. | ||
Step 3 | Review the list
of operational services in the
Overview window in the console.
Each service is represented by a square. A green colored square represents an active instance of the service, and a red colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running). Placing your cursor over a square displays the IP address of the client where the service is running. In a multi-host environment, a service may be represented by two green colored squares, indicating that the service is running on two different hosts within your cluster. Place your cursor over each square to view the host that the service is running on.
| ||
Step 4 | Locate the
service where you want to remove (harvest) an instance of a service and click
the subtraction sign (-) at the lower right.
You are then prompted to confirm your action to harvest an instance. | ||
Step 5 | Choose
Yes in the dialog box to confirm that you want to
harvest an instance of the service.
The controller then proceeds to spin down the instance of the service. When the process is finished, the square representing the service instance is removed. |
What to Do Next
Manage your services by growing additional instances or removing (harvesting) instances from the services. When finished with the Controller Admin console, click another icon on the Navigation pane to exit the console.
Creating a Service Instance
You can create or grow a service instance using the Controller Admin console.
![]() Caution | Only advanced users should access the Controller Admin console to perform the tasks described in this procedure or attempt to troubleshoot the services. |

You must have successfully installed the Cisco APIC-EM and it must be operational.
You must have administrator (ROLE_ADMIN) permissions and either access to all resources (RBAC scope set to ALL) or an RBAC scope that contains all of the resources that you want to group. For example, to create a group containing a specific set of resources, you must have access to those resources (custom RBAC scope set to all of the resources that you want to group).
Step 1 | In the Home window, click either admin or the Settings icon (gear) at the top right corner of the screen. | ||
Step 2 | Click the System Administration link from the drop-down menu. | ||
Step 3 | Review the list
of operational services in the
Overview window in the
Controller Admin console.
Each service is represented by a square. A green colored square represents an active instance of the service, and a red colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running). Placing your cursor over a square displays the IP address of the client where the service is running. In a multi-host environment, a service may be represented by two green colored squares, indicating that the service is running on two different hosts within your cluster. Place your cursor over each square to view the host that the service is running on.
| ||
Step 4 | Locate the
service where you want to manually grow an instance of a service and click the
addition sign (+) at the lower right.
You are then prompted to confirm your action to grow an instance. | ||
Step 5 | Choose
Yes in the dialog box to confirm that you want to
grow an instance of the service.
The controller then takes a client from the Spare Capacity Pool and spins up an instance of the service. When the process is finished, the square that represents the new service instance turns green. |