Troubleshooting Services Using the Controller Admin Console

The following procedures may be used to troubleshoot services using the controller admin console:

About Cisco APIC-EM Services

The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network, using Grapevine as an Elastic Services platform to support the controller's infrastructure and services. A service in this PaaS environment is a horizontally scalable application that adds instances of itself when demand increases, and frees instances of itself when demand decreases.

The Cisco APIC-EM controls elasticity at the service level, rather than at the Grapevine client level.

Service Managers and Monitors

The Cisco APIC-EM services that run on the Grapevine Elastic Services Platform provide the controller with its functionality. The Grapevine Elastic Services Platform consists the following components:

  • Grapevine root—Handles all policy management in regards to service updates, as well as the service lifecycle for both itself and the Grapevine client.

  • Grapevine client—Location where the supported services run.

After installation, service functionality is enabled using the following managers and monitors:

  • Grapevine Root

    • Service manager—Starts, stops, and monitors service instances across the Grapevine clients.

    • Capacity manager—Provides on-demand capacity to run the services.

    • Load monitor—Monitors the load and health of services across the Grapevine clients.

    • Service catalog—Repository of service bundles that can be deployed on the Grapevine clients.

  • Grapevine Client

    • Service manager—Starts, stops, and monitors service instances on the Grapevine client.

    • Service instance manager—Deploys the service.

Service Features

The Cisco APIC-EM provides the following service features:

  • Adding capacity on an existing client—When a service load exceeds a specified threshold on a client, the controller can request another service instance to start on a second, preexisting client.
  • Adding capacity on a newly instantiated client—When a service load exceeds a specified threshold on a client, the controller can request a new client to be instantiated and then start another service instance on this client.
  • Allows automatic scaling of services—As the service load increases, the controller instantiates additional service instances in response. As the service load decreases, the controller tears down the number of instances in response.

  • Resiliency for services—When a service fails, the controller starts a replacement instance. The controller then ensures that the service’s minimum instance count requirements are maintained.

Services

The following is a list of default Cisco APIC-EM services for the Cisco APIC-EM Release 1.5.x.

  • access-policy-programmer-service

  • apic-em-event-service

  • apic-em-inventory-manager-service

  • apic-em-jboss-ejbca

  • apic-em-network-programmer-service

  • apic-em-pki-broker-service

  • cas-service

  • cassandra

  • election-service

  • file-service

  • grapevine

  • grapevine-coordinator-service

  • grapevine-log-collector

  • grouping-service

  • identity-manager-pxgrid-service

  • nbar-policy-programmer-service

  • network-poller-service

  • node-ui

  • pnp-service

  • policy-analysis-service

  • policy-manager-service

  • postgres

  • qos-lan-policy-programmer-service

  • qos-monitoring-service

  • qos-policy-programmer-service

  • rabbitmq

  • rbac-service

  • reverse-proxy

  • router

  • scheduler-service

  • task-service

  • telemetry-service

  • topology-service


Note


The Cisco APIC-EM services running on your controller is dependent upon the applications installed and enabled on the host.


Controller Admin Console

The Cisco APIC-EM creates a Platform as a Service (PaaS) environment for your network. A service in this PaaS environment is a horizontally scalable application that adds instances of itself when increasing loads occur on a client within the network. You use the Controller Admin console to manage and troubleshoot these services. The Controller Admin console and its tools were bundled with the deployment files and installed when you first deployed the Cisco APIC-EM.

Figure 1. Controller Admin Console


Note


For a multi-host cluster, you do not have to log into each host to view the Controller Admin console. In a multi-host cluster, you get a single, consolidated view of all of the services running on all three hosts.


The Controller Admin console is directly accessible from the controller's GUI. To access this console, click on Settings (gear) icon in the menu bar at the top of the controller's GUI, then click on the System Administration link in the drop down menu.

The Controller Admin console provides the following windows and functionality:

  • Overview—Provides a list of services with information about their version and status. You can add or remove services in this window.

  • Clients—Provides detailed client information in this window.

  • Hosts—Provides detailed host information in this window.

  • Waiting Queue—Provides information about the waiting queue.

  • Services—Provides detailed service information. You can add or remove services in this window.

  • Logs—Provides detailed task, instance, and client logs.

Reviewing the Service's Version, Status, and Logs

You are able to perform the following tasks using the Controller Admin console:

  • Review the status of each service

  • Review the version of each service

  • Review the logs of each service


Caution


Only advanced users should access the Controller Admin console to perform the tasks described in this procedure or attempt to troubleshoot the services.


Figure 2. Controller Admin Console

Before You Begin

You must have successfully installed the Cisco APIC-EM and it must be operational.

You must have administrator (ROLE_ADMIN) permissions and either access to all resources (RBAC scope set to ALL) or an RBAC scope that contains all of the resources that you want to group. For example, to create a group containing a specific set of resources, you must have access to those resources (custom RBAC scope set to all of the resources that you want to group).


    Step 1   In the Home window, click either admin or the Settings icon (gear) at the top right corner of the screen.
    Step 2   Click the System Administration link from the drop-down menu.
    Step 3   Review the status of each service listed in the Overview window in the console.

    Each service is represented by a square. A green colored square represents an active instance of the service, and a red colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running).

    In a multi-host environment, a service may be represented by two green colored squares, indicating that the service is running on two different hosts within your cluster. Place your cursor over each square to view the host (IP address) that the service is running on.

    Step 4   Review the version of each service in the Overview window in the console.

    The version is located in the header of each listed service.

    Step 5   Review the service logs by clicking a specific active instance of a service (green square icon) and then viewing the Instance or Client logs located at the bottom of the window.

    The Instance logs detail information about the instance of the service. The Client logs detail information about the client where the service is located.

    Step 6   Proceed to review the logs under the Tasks tab. The following information is available for the service task:

    ID

    Task identification number

    Operation

    Type of task, for example, growing a service

    Client ID

    Client identification number

    Status

    Status of the task, for example Success or Failed.

    Reason

    Reason given for either a successful or failed task.

    Start Time

    Time task stared.

    Last Modified Time

    Time service was last modified.


    Removing a Service Instance

    You are able to remove or harvest a service instance by using the Controller Admin console. The Controller Admin console tools are bundled within the ISO image and installed when you first deploy the Cisco APIC-EM.


    Caution


    Only advanced users should access the Controller Admin console to perform the tasks described in this procedure or attempt to troubleshoot the services.


    Figure 3. Removing (Harvesting) a Service Instance

    Before You Begin

    You must have successfully installed the Cisco APIC-EM and it must be operational.

    You must have administrator (ROLE_ADMIN) permissions and either access to all resources (RBAC scope set to ALL) or an RBAC scope that contains all of the resources that you want to group. For example, to create a group containing a specific set of resources, you must have access to those resources (custom RBAC scope set to all of the resources that you want to group).


      Step 1   In the Home window, click either admin or the Settings icon (gear) at the top right corner of the screen.
      Step 2   Click the System Administration link from the drop-down menu.
      Step 3   Review the list of operational services in the Overview window in the console.

      Each service is represented by a square. A green colored square represents an active instance of the service, and a red colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running).

      Placing your cursor over a square displays the IP address of the client where the service is running.

      In a multi-host environment, a service may be represented by two green colored squares, indicating that the service is running on two different hosts within your cluster. Place your cursor over each square to view the host that the service is running on.

      Note   

      At the right of the console window are spare clients that are not running any service instances.

      Step 4   Locate the service where you want to remove (harvest) an instance of a service and click the subtraction sign (-) at the lower right.

      You are then prompted to confirm your action to harvest an instance.

      Step 5   Choose Yes in the dialog box to confirm that you want to harvest an instance of the service.

      The controller then proceeds to spin down the instance of the service.

      When the process is finished, the square representing the service instance is removed.


      What to Do Next

      Manage your services by growing additional instances or removing (harvesting) instances from the services. When finished with the Controller Admin console, click another icon on the Navigation pane to exit the console.

      Creating a Service Instance

      You can create or grow a service instance using the Controller Admin console.


      Caution


      Only advanced users should access the Controller Admin console to perform the tasks described in this procedure or attempt to troubleshoot the services.


      Figure 4. Creating (Growing) a Service Instance

      Before You Begin

      You must have successfully installed the Cisco APIC-EM and it must be operational.

      You must have administrator (ROLE_ADMIN) permissions and either access to all resources (RBAC scope set to ALL) or an RBAC scope that contains all of the resources that you want to group. For example, to create a group containing a specific set of resources, you must have access to those resources (custom RBAC scope set to all of the resources that you want to group).


        Step 1   In the Home window, click either admin or the Settings icon (gear) at the top right corner of the screen.
        Step 2   Click the System Administration link from the drop-down menu.
        Step 3   Review the list of operational services in the Overview window in the Controller Admin console.

        Each service is represented by a square. A green colored square represents an active instance of the service, and a red colored square represents a service with a faulty or failed instance. Squares without color represents inactive services (no instances initiated and running).

        Placing your cursor over a square displays the IP address of the client where the service is running.

        In a multi-host environment, a service may be represented by two green colored squares, indicating that the service is running on two different hosts within your cluster. Place your cursor over each square to view the host that the service is running on.

        Note   

        At the right of the console window are spare clients that are not running any service instances.

        Step 4   Locate the service where you want to manually grow an instance of a service and click the addition sign (+) at the lower right.

        You are then prompted to confirm your action to grow an instance.

        Step 5   Choose Yes in the dialog box to confirm that you want to grow an instance of the service.

        The controller then takes a client from the Spare Capacity Pool and spins up an instance of the service.

        When the process is finished, the square that represents the new service instance turns green.