Troubleshooting the Configuration

The following information may be used to troubleshoot issues with the configuration:

Troubleshooting the Configuration

The following table describes recommended actions to take to resolve a Cisco APIC-EM configuration issue.

Symptom

Possible Cause

Recommended Action

Controller in a single host configuration appears to be in an unstable state. For example, applications are not running, or applications are inaccessible, and/or not appearing in the GUI.

Controller in unstable state, possibly due to error(s) in entering configuration values with the Cisco APIC-EM configuration wizard.

Log into the host, check the configuration values, and reenter any configuration values that are incorrect.

References:

Controller in a multi-host configuration appears to be in an unstable state. For example, applications are not running, or applications are inaccessible, and/or not appearing in the GUI.

Controller in unstable state, possibly due to error(s) in entering configuration values with the Cisco APIC-EM configuration wizard.

Log into the host, check the configuration values, and reenter any configuration values that are incorrect.

References:

Controller was working fine for a period of time after its initial deployment, but then lapses into an unstable state. For example, applications are not running, or applications are inaccessible, and/or not appearing in the GUI.

Possible user or other error in configuration values that occurs after the initial deployment.

Revert back to the first configuration or the factory default configuration.

References:

Controller was working fine for a multi-host configuration, but after a period of time one of the hosts becomes erratic and unstable.

Possible failed service or services in the multi-host cluster.

Remove and then reattach unstable host from the multi-host cluster.

References:

Controller was working fine for a multi-host configuration, but after a period of time one of the hosts fails.

Possible failed service or services in the multi-host cluster.

Remove and then reattach failed and inoperable host from the multi-host cluster.

References:

Updating the Configuration Using the Wizard

You can troubleshoot the Cisco APIC-EM deployment by running the configuration wizard a second time and updating any earlier configuration entries. The configuration wizard saves and displays your previous configuration settings, so you do not have to reenter them.


Note


When performing this procedure, controller downtime occurs. For this reason, we recommend that you perform this procedure during a maintenance time period. For information about changing settings for a multi-host configuration, see Changing the Settings in a Multi-Host Cluster.


Before You Begin

You have installed the Cisco APIC-EM following the procedure described in the Cisco Application Policy Infrastructure Controller Enterprise Module Installation Guide.


    Step 1   Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
    Note   

    The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

    Step 2   When prompted, enter your Linux username ('grapevine') and password for SSH access.
    Step 3   Restart the configuration wizard using the following command.
    $ config_wizard
    
    
    Note   

    The config_wizard command is in the PATH of the 'grapevine' user, and not the "root" user. Either run the command as the "grapevine" user, or fully qualify the command as the "root" user. For example: /home/grapevine/bin/config_wizard

    Step 4   Review the current configuration values in the configuration wizard and click next>>, until you access the specific step where you wish to update your previous configuration entry.

    For example, if you need to enter a new NTP server IP address, click next >> until you get to the NTP SERVER SETTINGS screen.

    Step 5   Update the value that was previously entered in the configuration wizard and is currently displayed.

    For example, you can update the NTP server settings by entering a new IP address.

    Step 6   Click next>> until the last step of the configuration wizard process.
    Step 7   Click proceed>> to have the configuration wizard save and apply your configuration changes to your Cisco APIC-EM deployment.

    Resetting the Cisco APIC-EM

    You can troubleshoot a Cisco APIC-EM deployment by resetting the controller back to configuration values that were originally set using the configuration wizard the first time. A reset of the controller is helpful, when the controller has gotten itself into an unstable state and other troubleshooting activities have not resolved the situation.


    Note


    In a multi-host environment, you need to perform this procedure on only a single host. After performing this procedure on a single host, the other two hosts will be automatically reset.


    Before You Begin

    You have installed the Cisco APIC-EM following the procedure described in the Cisco Application Policy Infrastructure Controller Enterprise Module Installation Guide.


      Step 1   Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
      Note   

      The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

      Step 2   When prompted, enter your Linux username ('grapevine') and password for SSH access.
      Step 3   Navigate to the bin directory on the Grapevine root. The bin directory contains the grapevine scripts.
      Step 4   Enter the reset_grapevine command at the prompt to run the reset grapevine script.
      $ reset_grapevine
      
      

      The reset_grapevine command returns the configuration settings back to values that you configured when running the configuration wizard for the first time. The configuration settings are saved to a .JSON file. This .JSON file is located at: /etc/grapevine/controller-config.json. The reset_grapevine command uses the data in the controller-config.json file to return to the earlier configuration settings, so do not delete this file. If you delete this file, you must run the configuration wizard again and reenter your configuration data.

      Important:

      The reset_grapevine command will terminate if the SSH connection is disconnected for any reason. To avoid this, we recommend that you use tmux (terminal multiplexer) which is already installed on the controller to run the reset_grapevine command in the session. You can use the following commands for tmux:

      tmux new -s session_name reset_grapevine

      Command to create a new session using tmux for reset-grapevine.

      For example, you can enter the following command:

      tmux new -s session100 reset_grapevine

      tmux ls

      Command to view a the list of tmux sessions.

      tmux attach -t session_name reset_grapevine

      Command to attach to a tmux session.

      For example, you can enter the following command:

      tmux attach -t session200 reset_grapevine

      To get more information about tmux, you can run the man tmux command.

      After entering the reset_grapevine command, you are then prompted to reenter your Grapevine password.

      Step 5   Enter your Grapevine password a second time.
      
      [sudo] password for grapevine:********
      
      

      You are then prompted to delete all virtual disks The virtual disks are where the Cisco APIC-EM database resides. For example, data about devices that the controller discovered are saved on these virtual disks. If you enter yes (y), all of this data is deleted. If you enter no (n), then the new cluster will come up populated with your existing data once the reset procedure completes.

      Step 6   Enter n to prevent the deletion all of the virtual disks.
      
      THIS IS A DESTRUCTIVE OPERATION
      Do you want to delete all VIRTUAL DISKS in your APIC-EM cluster? (y/n):n
      
      

      You are then prompted to delete all Cisco APIC-EM authentication timeout policies, user password policies, and user accounts other than the primary administrator account.

      Step 7   Enter n to prevent the deletion of all authentication timeout policies, user password policies, and user accounts other than the primary administrator account.
      
      THIS IS A DESTRUCTIVE OPERATION
      Do you want to delete authentication timeout policies, user password policies, 
      and Cisco APIC-EM user accounts other than the primary administrator account? (y/n): n
      
      

      You are then prompted to delete any imported certificates.

      Step 8   Enter n to prevent the deletion of any imported certificates.
      
      THIS IS A DESTRUCTIVE OPERATION
      Do you want to delete the imported certificates? (y/n): n
      
      

      You are then prompted to delete any backups.

      Step 9   Enter n to prevent the deletion of any backups.
      
      THIS IS A DESTRUCTIVE OPERATION
      Do you want to delete the backups? (y/n): n
      
      

      The controller then resets itself with the configuration values that were originally set using the configuration wizard the first time. When the controller is finished resetting, you are presented with a command prompt from the controller.

      Step 10   Using the Secure Shell (SSH) client, log out of the host.

      Restoring the Controller to the Factory Default

      In certain situations, you may want to restore the Cisco APIC-EM to its original factory default settings. For example, if your controller appliance is being replaced or simply has an undesirable configuration that needs to be completely removed. Under these circumstances, you can restore the controller to its factory defaults and then proceed to reconfigure it as a new controller.

      This procedure describes how to a restore the factory defaults to the controller.


      Caution


      This procedure shuts down both the Cisco APIC-EM and the host (physical or virtual) on which it resides. At the end of this procedure, you will need to access the host and restart it.


      Before You Begin

      You have installed the Cisco APIC-EM following the procedure described in the Cisco Application Policy Infrastructure Controller Enterprise Module Installation Guide.

      You have access to the Cisco APIC-EM using either a physical console or a Telnet connection.


        Step 1   Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
        Note   

        The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

        Step 2   When prompted, enter your Linux username ('grapevine') and password for SSH access.
        Step 3   Enter the reset_grapevine factory command at the prompt.
        $ reset_grapevine factory
        
        
        Important:

        The reset_grapevine factory command will terminate if the SSH connection is disconnected for any reason. To avoid this, we recommend that you use tmux (terminal multiplexer) which is already installed on the controller to run the reset_grapevine factory command in the session. You can use the following commands for tmux:

        tmux new -s session_name 'reset_grapevine factory'

        Command to create a new session using tmux for reset-grapevine factory. Note that 'reset-grapevine factory' in this command is a string value and therefore must be entered within single quotation marks.

        For example, you can enter the following command:

        tmux new -s session100 'reset_grapevine factory'

        tmux ls

        Command to view a the list of tmux sessions.

        tmux attach -t session_name 'reset_grapevine factory'

        Command to attach to a tmux session. Note that 'reset-grapevine factory' in this command is a string value and therefore must be within single quotation marks.

        For example, you can enter the following command:

        tmux attach -t session200 'reset_grapevine factory'

        To get more information about tmux, you can run the man tmux command.

        Step 4   Enter your Linux grapevine password a second time to start the reset process.
        $ sudo password for grapevine *********
        
        

        After entering this command a warning appears that the reset_grapevine factory command will shut down the controller.

        You are then prompted to confirm your desire to run the reset_grapevine factory command.

        Step 5   Enter Yes to confirm that you want to run the reset_grapevine factory command.

        The controller then performs the following tasks:

        • Stops all running clients and services

        • Deletes all cluster data

        • Deletes all user data

        • Deletes the configuration files including secrets and private keys

        • Shuts down the controller

        • Shuts down the host (physical or virtual)


        What to Do Next

        Perform the following tasks:

        • Start up the host (physical or virtual).

        • After start up, the configuration wizard appears and prompts you to re-deploy the Cisco APIC-EM.

        • Proceed to re-deploy the Cisco APIC-EM using the configuration wizard.

        Creating a Support File for a Single Host

        You can troubleshoot the Cisco APIC-EM deployment by creating a root cause analysis (rca) support file. This rca file consists of logs, configuration files, and command output. After you create this rca file, you can then email it to Cisco support for assistance.

        Before You Begin

        You have installed the Cisco APIC-EM following the procedure described in the Cisco Application Policy Infrastructure Controller Enterprise Module Installation Guide.


          Step 1   Using a Secure Shell (SSH) client, log into the host (physical or virtual) with the IP address that you specified using the configuration wizard.
          Note   

          The IP address to enter for the SSH client is the IP address that you configured for the network adapter. This IP address connects the host to the external network.

          Step 2   When prompted, enter your Linux username ('grapevine') and password for SSH access.
          Step 3   Navigate to the bin directory on the host. The bin directory contains the grapevine scripts.
          Step 4   To create the support file, enter the rca command in this directory.
          $ rca
          [sudo] password for grapevine:
          mkdir: created directory '/tmp/grapevine-root-172.24.100.15-rca-2016-08-05_18-17-04_UTC+0000'
          ===============================================================
          RCA package created on Fri Aug  5 18:17:10 UTC 2016
          Max Log Size: 50000000
          Skipping collection of ''
          ===============================================================
          
          2016-08-05 18:17:10 | INFO | Generating log for 'date'...
          2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat hosttime'...
          2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat speed' ..
          2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat session id'...
          2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat balloon '...
          2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat swap'...
          2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat memlimit'...
          2016-08-05 18:17:10 | INFO | Generating log for 'vmware-toolbox-cmd stat memres'
          ....
          
          

          The rca command runs a root cause analysis script that creates a tar file that contains log files, configuration files, and the command output.

          Note   

          For a multi-host deployment (three hosts), you need to perform this procedure and run the rca command on each of the three hosts. Additionally, if you need to perform a root cause analysis for a specific service, then to identify which host the service is running on, access the controller's GUI, click admin, click System Administration from the drop down menu, and click Hosts at the top of the window to view the individual hosts and their services.


          What to Do Next

          Send the tar file created by this procedure to Cisco support for assistance in resolving your issue. For information about contacting Cisco support, see Contacting the Cisco Technical Assistance Center.