Refer to the Cisco Unified Serviceability Administration Guide for detailed information of the following different types of tools that Cisco Unified Serviceability provides to monitor and analyze the various Cisco Unified Communications Manager systems.

Use the command line interface (CLI) to access the Cisco Unified Communications Manager system for basic maintenance and failure recovery. Obtain access to the system by either a hard-wired terminal (a system monitor and keyboard) or by performing a SSH session.

The account name and password get created at install time. You can change the password after install, but you never can change the account name.

A command represents a text instruction that caused the system to perform some function. Commands may be stand alone, or they can have mandatory or optional arguments or options.

A level comprises a collection of commands; for example, show designates a level, whereas show status specifies a command. Each level and command also includes an associated privilege level. You can execute a command only if you have sufficient privilege level.

For complete information on the Cisco Unified Communications Manager CLI command set, see the Command Line Interface Reference Guide for Cisco Unified Solutions.

The kerneldump utility allows you to collect crash dump logs locally on the affected machine without requiring a secondary server.

In a Cisco Unified Communications Manager cluster, you only need to ensure the kerneldump utility is enabled on the server before you can collect the crash dump information.

Note

Cisco recommends that you verify the kerneldump utility is enabled after you install Cisco Unified Communications Manager to allow for more efficient troubleshooting. If you have not already done so, enable the kerneldump utility before you upgrade Cisco Unified Communications Manager from supported appliance releases.

The command line interface (CLI) for the Cisco Unified Communications Operating System can be used to enable, disable, or check the status of the kerneldump utility.

Use the following procedure to enable the kernel dump utility:

Use the network management tools for Cisco Unified Communications Manager remote serviceability.

• System Log Management

• Cisco Discovery Protocol Support

• Simple Network Management Protocol support

Refer to the documentation at the URLs provided in the sections for these network management tools for more information.

Typically, you collect sniffer traces by connecting a laptop or other sniffer-equipped device on a Catalyst port that is configured to span the VLAN or port(s) (CatOS, Cat6K-IOS, XL-IOS) that contains the trouble information. If no free port is available, connect the sniffer-equipped device on a hub that is inserted between the switch and the device.

Tip	To help facilitate reading and interpreting of the traces by the TAC engineer, Cisco recommends using Sniffer Pro software because it is widely used within the TAC.

Have available the IP/MAC addresses of all equipment that is involved, such as IP phones, gateways, Cisco Unified Communications Managers, and so on.

The output from debug privileged EXEC commands provides diagnostic information about a variety of internetworking event that relate to protocol status and network activity in general.

Set up your terminal emulator software (such as HyperTerminal), so it can capture the debug output to a file. In HyperTerminal, click Transfer; then, click Capture Text and choose the appropriate options.

Before running any IOS voice gateway debugs, make sure that service timestamps debug datetime msec is globally configured on the gateway.

Note	Avoid collecting debugs in a live environment during operation hours.

Preferably, collect debugs during non-working hours. If you must collect debugs in a live environment, configure no logging console and logging buffered. To collect the debugs, use show log.

Because some debugs can be lengthy, collect them directly on the console port (default logging console) or on the buffer (logging buffer). Collecting debugs over a Telnet session may impact the device performance, and the result could be incomplete debugs, which requires that you re-collect them.

To stop a debug, use the no debug all or undebug all commands. Verify that the debugs have been turned off by using the command show debug.

Cisco Secure Telnet allows Cisco Service Engineers (CSE) transparent firewall access to the Cisco Unified Communications Manager node on your site. Using strong encryption, Cisco Secure Telnet enables a special Telnet client from Cisco Systems to connect to a Telnet daemon behind your firewall. This secure connection allows remote monitoring and troubleshooting of your Cisco Unified Communications Manager nodes, without requiring firewall modifications.

Note	Cisco provides this service only with your permission. You must ensure that a network administrator is available at your site to help initiate the process.

This section contains information about packet capture.

This section provides a quick reference for commands and utilities to help you troubleshoot a Cisco Unified Communications Manager server with root access disabled. The following table provides a summary of the CLI commands and GUI selections that you can use to gather information troubleshoot various system problems.

The following table provides a list of common problems and tools to use to troubleshoot them.

The following tips may help you when you are troubleshooting the Cisco Unified Communications Manager.

Tip	Check the release notes for Cisco Unified Communications Manager for known problems. The release notes provide descriptions and workaround solutions for known problems.

Tip	Know where your devices are registered.

Each Cisco Unified Communications Manager log traces files locally. If a phone or gateway is registered to a particular Cisco Unified Communications Manager, the call processing gets done on that Cisco Unified Communications Manager if the call is initiated there. You will need to capture traces on that Cisco Unified Communications Manager to debug a problem.

A common mistake involves having devices that are registered on a subscriber server but are capturing traces on the publisher server. These trace files will be nearly empty (and definitely will not have the call in them).

Another common problem involves having Device 1 registered to CM1 and Device 2 registered to CM2. If Device 1 calls Device 2, the call trace occurs in CM1, and, if Device 2 calls Device 1, the trace occurs in CM2. If you are troubleshooting a two-way calling issue, you need both traces from both Cisco Unified Communications Managers to obtain all the information that is needed to troubleshoot.

Tip	Know the approximate time of the problem.

Multiple calls may have occurred, so knowing the approximate time of the call helps TAC quickly locate the trouble.

You can obtain phone statistics on a Cisco Unified IP Phone 79xx by pressing the i or ? button twice during an active call.

When you are running a test to reproduce the issue and produce information, know the following data that is crucial to understanding the issue:

• Calling number/called number

• Any other number that is involved in the specific scenario

• Time of the call

  Note	Remember that time synchronization of all equipment is important for troubleshooting.

If you are reproducing a problem, make sure to choose the file for the timeframe by looking at the modification date and the time stamps in the file. The best way to collect the right trace means that you reproduce a problem and then quickly locate the most recent file and copy it from the Cisco Unified Communications Manager server.

Tip	Save the log files to prevent them from being overwritten.

Files will get overwritten after some time. The only way to know which file is being logged to is to choose View > Refresh on the menu bar and look at the dates and times on the files.

This system history log provides a central location for getting a quick overview of the initial system install, system upgrades, Cisco option installations, and DRS backups and DRS restores, as well as switch version and reboot history.

Centralized audit logging ensures that configuration changes to the Cisco Unified Communications Manager system gets logged in separate log files for auditing. An audit event represents any event that is required to be logged. The following Cisco Unified Communications Manager components generate audit events:

• Cisco Unified Communications Manager Administration

• Cisco Unified Serviceability

• Cisco Unified Communications Manager CDR Analysis and Reporting

• Cisco Unified Real-Time Monitoring Tool

• Cisco Unified Communications Operating System

• Disaster Recovery System

• Database

• Command Line Interface

• Remote Support Account Enabled (CLI commands issued by technical supports teams)

In Cisco Business Edition 5000, the following Cisco Unity Connection components also generate audit events:

• Cisco Unity Connection Administration

• Cisco Personal Communications Assistant (Cisco PCA)

• Cisco Unity Connection Serviceability

• Cisco Unity Connection clients that use the Representational State Transfer (REST) APIs

The following example displays a sample audit event:

CCM_TOMCAT-GENERIC-3-AuditEventGenerated: Audit Event Generated UserID:CCMAdministrator Client IP Address:172.19.240.207 Severity:3 EventType:ServiceStatusUpdated ResourceAccessed: CCMService EventStatus:Successful Description: Call Manager Service status is stopped App ID:Cisco Tomcat Cluster ID:StandAloneCluster Node ID:sa-cm1-3

Audit logs, which contain information about audit events, get written in the common partition. The Log Partition Monitor (LPM) manages the purging of these audit logs as needed, similar to trace files. By default, the LPM purges the audit logs, but the audit user can change this setting from the Audit User Configuration window in Cisco Unified Serviceability. The LPM sends an alert whenever the common partition disk usage exceeds the threshold; however, the alert does not have the information about whether the disk is full because of audit logs or trace files.

Tip

The Cisco Audit Event Service, which is a network service that supports audit logging, displays in Control Center—Network Services in Cisco Unified Serviceability. If audit logs do not get written, then stop and start this service by choosing Tools > Control Center—Network Services in Cisco Unified Serviceability.

All audit logs get collected, viewed and deleted from Trace and Log Central in the Cisco Unified Real-Time Monitoring Tool. Access the audit logs in RTMT in Trace and Log Central. Go to System > Real-Time Trace > Audit Logs > Nodes. After you select the node, another window displays System > Cisco Audit Logs.

The following types of audit logs display in RTMT:

• Application log

• Database log

• Operating system log

• Remote SupportAccEnabled log

Application Log

The application audit log, which displays in the AuditApp folder in RTMT, provides configuration changes for Cisco Unified Communications Manager Administration, Cisco Unified Serviceability, the CLI, Cisco Unified Real-Time Monitoring Tool (RTMT), Disaster Recovery System, and Cisco Unified CDR Analysis and Reporting (CAR). For Cisco Business Edition 5000, the application audit log also logs changes for Cisco Unity Connection Administration, Cisco Personal Communications Assistant (Cisco PCA), Cisco Unity Connection Serviceability, and clients that use the Representational State Transfer (REST) APIs.

Although the Application Log stays enabled by default, you can configure it in Cisco Unified Serviceability by choosing Tools > Audit Log Configuration. For a description of the settings that you can configure for audit log configuration, refer to the Cisco Unified Serviceability Administration Guide.

If the audit logs get disabled in Cisco Unified Serviceability, no new audit log files get created.

Tip

Only a user with an audit role has permission to change the Audit Log settings. By default, the CCMAdministrator has the audit role after fresh installs and upgrades. The CCMAdministrator can assign the "standard audit users" group to a new user that the CCMAdministrator specifically creates for audit purposes. The CCMAdministrator can then be removed from the audit user group. The "standard audit log configuration" role provides the ability to delete audit logs, read/update access to Cisco Unified Real-Time Monitoring Tool, Trace Collection Tool, RTMT Alert Configuration, the Control Center - Network Services window, RTMT Profile Saving, the Audit Configuration window, and a new resource called Audit Traces. For Cisco Unity Connection in Cisco Business Edition 5000, the application administration account that was created during installation has the Audit Administrator role and can assign other administrative users to the role.

Cisco Unified Communications Manager creates one application audit log file until the configured maximum file size is reached; then, it closes and creates a new application audit log file. If the system specifies rotating the log files, Cisco Unified Communications Manager saves the configured number of files. Some of the logging events can be viewed by using RTMT SyslogViewer.

The following events get logged for Cisco Unified Communications Manager Administration:

• User logging (user logins and user logouts).

• User role membership updates (user added, user deleted, user role updated).

• Role updates (new roles added, deleted, or updated).

• Device updates (phones and gateways).

• Server configuration updates (changes to alarm or trace configurations, service parameters, enterprise parameters, IP addresses, host names, Ethernet settings, and Cisco Unified Communications Manager server additions or deletions).

The following events get logged for Cisco Unified Serviceability:

• Activation, deactivation, start, or stop of a service from any Serviceability window.

• Changes in trace configurations and alarm configurations.

• Changes in SNMP configurations.

• Changes in CDR Management.

• Review of any report in the Serviceability Reports Archive. View this log on the reporter node.

RTMT logs the following events with an audit event alarm:

• Alert configuration.

• Alert suspension.

• E-mail configuration.

• Set node alert status.

• Alert addition.

• Add alert action.

• Clear alert.

• Enable alert.

• Remove alert action.

• Remove alert.

The following events get logged for Cisco Unified Communications Manager CDR Analysis and Reporting:

• Scheduling the CDR Loader.

• Scheduling the daily, weekly, and monthly user reports, system reports, and device reports.

• Mail parameters configurations.

• Dial plan configurations.

• Gateway configurations.

• System preferences configurations.

• Autopurge configurations.

• Rating engine configurations for duration, time of day, and voice quality.

• QoS configurations.

• Automatic generation/alert of pregenerated reports configurations.

• Notification limits configurations.

The following events gets logged for Disaster Recovery System:

• Backup initiated successfully/failed

• Restore initiated successfully/failed

• Backup cancelled successfully

• Backup completed successfully/failed

• Restore completed successfully/failed

• Save/update/delete/enable/disable of backup schedule

• Save/update/delete of destination device for backup

For Cisco Business Edition 5000, Cisco Unity Connection Administration logs the following events:

• User logging (user logins and user logouts).

• All configuration changes, including but not limited to users, contacts, call management objects, networking, system settings, and telephony.

• Task management (enabling or disabling a task).

• Bulk Administration Tool (bulk creates, bulk deletes).

• Custom Keypad Map (map updates)

For Cisco Business Edition 5000, Cisco PCA logs the following events:

• User logging (user logins and user logouts).

• All configuration changes made via the Messaging Assistant.

For Cisco Business Edition 5000, Cisco Unity Connection Serviceability logs the following events:

• User logging (user logins and user logouts).

• All configuration changes.

• Activating, deactivating, starting or stopping services.

For Cisco Business Edition 5000, clients that use the REST APIs log the following events:

• User logging (user API authentication).

• API calls that utilize Cisco Unity Connection Provisioning Interface (CUPI).

Use the following procedure to verify which Cisco CallManager services are active on a server.

Perform the following procedure if the Cisco CallManager service has been in activated and you want to verify if the service is currently running.