Tracing

Information About Tracing

Tracing Overview

The tracing functionality logs internal events. Trace files are automatically created and saved to the tracelogs subdirectory under crashinfo.

The contents of trace files are useful for the following purposes:

  • Troubleshooting—If a switch has an issue, the trace file output may provide information that can be used for locating and solving the issue.

  • Debugging—The trace file outputs helps users get a more detailed view of system actions and operations.

To view the most recent trace information for a specific module, use the show platform software trace message command.

To modify the trace level to increase or decrease the amount of trace message output, you can set a new trace level using the set platform software trace command. Trace levels can be set for each process using the all-modules keyword in the set platform software trace command, or per module within a process.

Location of Tracelogs

Each process uses btrace infrastructure to log its trace messages. When a process is active, the corresponding in-memory tracelog is found in the directory /tmp/<FRU>/trace/, where <FRU> refers to the location where the process is running (rp, fp, or cc).

When a tracelog file has reached the maximum file size limit allowed for the process, or if the process ends, it gets rotated into the following directory:

  • /crashinfo/tracelogs, if the crashinfo: partition is available on the switch

  • /harddisk/tracelogs, if the crashinfo: partition is not available on the switch

The tracelog files are compressed before being stored in the directory.

Tracelog Naming Convention

All the tracelogs that are created using btrace have the following naming convention:

<process_name>_<FRU><SLOT>-<BAY>.<pid>_<counter>.<creation_timestamp>.bin

Here, counter is a free-running 64-bit counter that gets incremented for each new file created for the process. For example, wcm_R0-0.1362_0.20151006171744.bin. When compressed, the files will have the gz extension appended to their names

Tracelog size limits and rotation policy

The maximum size limit for a tracelog file is 1MB for each process, and the maximum number of tracelog files that are maintained for a process is 25.

Rotation and Throttling Policy

Initially, all the tracelog files are moved from the initial /tmp/<FRU>/trace directory to the /tmp/<FRU>/trace/stage staging directory. The btrace_rotate script then moves these tracelogs from the staging directory to the /crashinfo/tracelogs directory. When the number of files stored in the /crashinfo/tracelogs directory per process reaches the maximum limit, the oldest files for the process are deleted, while the newer files are maintained. This is repeated at every 60 minutes under worst-case situations.

There are two other sets of files that are purged from the /crashinfo/tracelogs directory:

  • Files that do not have the standard naming convention (other than a few exceptions such as fed_python.log)

  • Files older than two weeks

The throttling policy has been introduced so that a process with errors does not affect the functioning of the switch. Whenever a process starts logging at a very high rate, for example, if there are more than 16 files in a 4-second interval for the process in the staging directory, the process is throttled. The files do not rotate for the process from /tmp/<FRU>/trace into /tmp/<FRU>/trace/stage, however the files are deleted when they reach the maximum size. Throttling is re-enabled, when the count goes below 8.

Tracing Levels

Tracing levels determine how much information should be stored about a module in the trace buffer or file.

The following table shows all of the tracing levels that are available, and provides descriptions of the message that are displayed with each tracing level.

Table 1. Tracing Levels and Descriptions

Tracing Level

Description

Emergency

The message is regarding an issue that makes the system unusable.

Error

The message is regarding a system error.

Warning

The message is regarding a system warning.

Notice

The message is regarding a significant issue, but the switch is still working normally.

Informational

The message is useful for informational purposes only.

Debug

The message provides debug-level output.

Verbose

All possible trace messages are sent.

Noise

All possible trace messages for the module are logged.

The noise level is always equal to the highest possible tracing level. Even if a future enhancement to tracing introduces a higher tracing level, the noise level will become equal to the level of that new enhancement.

set platform software trace

To set the trace level for a specific module within a process, use the set platform software trace command in privileged EXEC or user EXEC mode.

set platform software trace process slot module trace-level

Syntax Description

process

Process whose tracing level is being set. Options include:

  • chassis-manager —The Chassis Manager process.

  • cli-agent —The CLI Agent process.

  • dbm —The Database Manager process.

  • emd —The Environmental Monitoring process.

  • fed —The Forwarding Engine Driver process.

  • forwarding-manager —The Forwarding Manager process.

  • host-manager —The Host Manager process.

  • iomd —The Input/Output Module daemon (IOMd) process.

  • ios —The IOS process.

  • license-manager —The License Manager process.

  • logger —The Logging Manager process.

  • platform-mgr —The Platform Manager process.

  • pluggable-services —The Pluggable Services process.

  • replication-mgr —The Replication Manager process.

  • shell-manager —The Shell Manager process.

  • smd —The Session Manager process.

  • table-manager —The Table Manager Server.

  • wireshark —The Embedded Packet Capture (EPC) Wireshark process.

slot

Hardware slot where the process for which the trace level is set, is running. Options include:

  • number —Number of the SIP slot of the hardware module where the trace level is set. For instance, if you want to specify the SIP in SIP slot 2 of the switch, enter 2.

  • SIP-slot / SPA-bay—Number of the SIP switch slot and the number of the shared port adapter (SPA) bay of that SIP. For instance, if you want to specify the SPA in bay 2 of the SIP in switch slot 3, enter 3/2.

  • F0 —The Embedded-Service-Processor in slot 0.

  • FP active —The active Embedded-Service-Processor.

  • R0 —The route processor in slot 0.

  • RP active —The active route processor.

  • switch <number> —The switch with its number specified.

  • switch active —The active switch.

  • switch standby —The standby switch.

module

Module within the process for which the tracing level is set.

trace-level

Trace level. Options include:

  • debug —Debug level tracing. A debug-level trace message is a non-urgent message providing a large amount of detail about the module.

  • emergency —Emergency level tracing. An emergency-level trace message is a message indicating that the system is unusable.

  • error —Error level tracing. An error-level tracing message is a message indicating a system error.

  • info —Information level tracing. An information-level tracing message is a non-urgent message providing information about the system.

  • noise —Noise level tracing. The noise level is always equal to the highest tracing level possible and always generates every possible tracing message.

    The noise level is always equal to the highest-level tracing message possible for a module, even if future enhancements to this command introduce options that allow users to set higher tracing levels.

  • notice —The message is regarding a significant issue, but the switch is still working normally.

  • verbose —Verbose level tracing. All possible tracing messages are sent when the trace level is set to verbose.

  • warning —Warning messages.

Command Default

The default tracing level for all modules is notice .

Command Modes

User EXEC (>)

Privileged EXEC (#)

Command History

Release

Modification

This command was introduced.

Usage Guidelines

The module options vary by process and by hardware-module . Use the ? option when entering this command to see which module options are available with each keyword sequence.

Use the show platform software trace message command to view trace messages.

Trace files are stored in the tracelogs directory in the harddisk: file system. These files can be deleted without doing any harm to your switch operation.

Trace file output is used for debugging. The trace level is a setting that determines how much information should be stored in trace files about a module.

Examples

This example shows how to set the trace level for all the modules in dbm process: 


Device# set platform software trace dbm R0 all-modules debug

show platform software trace filter-binary

To display the most recent trace information for a specific module, use the show platform software trace filter-binary command in privileged EXEC or user EXEC mode.

show platform software trace filter-binarymodules [ context mac-address]

Syntax Description

contextmac-address

Represents the context used to filter. Additionally, you can filter based on module names and trace levels. The context keyword accepts either a MAC address or any other argument based on which a trace is tagged.

Command Modes

User EXEC (>)

Privileged EXEC (#)

Command History

Release

Modification

This command was introduced.

Usage Guidelines

This command collates and sorts all the logs present in the /tmp/.../ across all the processes relevant to the module. The trace logs of all the processes relevant to the specified module are printed to the console. This command also generates a file named collated_log_{system time} with the same content, in the /crashinfo/tracelogs directory.

show platform software trace message

To display the trace messages for a process, use the set platform software trace command in privileged EXEC or user EXEC mode.

show platform software trace message process slot

Syntax Description

process

Tracing level that is being set. Options include:

  • chassis-manager —The Chassis Manager process.

  • cli-agent —The CLI Agent process.

  • cmm —The CMM process.

  • dbm —The Database Manager process.

  • emd —The Environmental Monitoring process.

  • fed —The Forwarding Engine Driver process.

  • forwarding-manager —The Forwarding Manager process.

  • geo —The Geo Manager process.

  • host-manager —The Host Manager process.

  • interface-manager —The Interface Manager process.

  • iomd —The Input/Output Module daemon (IOMd) process.

  • ios —The IOS process.

  • license-manager —The License Manager process.

  • logger —The Logging Manager process.

  • platform-mgr —The Platform Manager process.

  • pluggable-services —The Pluggable Services process.

  • replication-mgr —The Replication Manager process.

  • shell-manager —The Shell Manager process.

  • sif —The Stack Interface (SIF) Manager process.

  • smd —The Session Manager process.

  • stack-mgr —The Stack Manager process.

  • table-manager —The Table Manager Server.

  • thread-test —The Multithread Manager process.

  • virt-manager —The Virtualization Manager process.

slot

Hardware slot where the process for which the trace level is set, is running. Options include:

  • number —Number of the SIP slot of the hardware module where the trace level is set. For instance, if you want to specify the SIP in SIP slot 2 of the switch, enter 2.

  • SIP-slot / SPA-bay—Number of the SIP switch slot and the number of the shared port adapter (SPA) bay of that SIP. For instance, if you want to specify the SPA in bay 2 of the SIP in switch slot 3, enter 3/2.

  • F0 —The Embedded Service Processor slot 0.

  • FP active —The active Embedded Service Processor.

  • R0 —The route processor in slot 0.

  • RP active —The active route processor.

  • switch <number> —The switch, with its number specified.

  • switch active —The active switch.

  • switch standby —The standby switch.

    • number —Number of the SIP slot of the hardware module where the trace level is set. For instance, if you want to specify the SIP in SIP slot 2 of the switch, enter 2.

    • SIP-slot / SPA-bay—Number of the SIP switch slot and the number of the shared port adapter (SPA) bay of that SIP. For instance, if you want to specify the SPA in bay 2 of the SIP in switch slot 3, enter 3/2.

    • F0 —The Embedded Service Processor in slot 0.

    • FP active —The active Embedded Service Processor.

    • R0 —The route processor in slot 0.

    • RP active —The active route processor.

Command Modes

User EXEC (>)

Privileged EXEC (#)

Command History

Release

Modification

This command was introduced.

Examples

This example shows how to display the trace messages for the Stack Manager and the Forwarding Engine Driver processes: 


Device# show platform software trace message stack-mgr switch active R0
10/30 09:42:48.767 [btrace] [8974]: (note): Successfully registered module [97] [uiutil]
10/30 09:42:48.762 [btrace] [8974]: (note): Successfully registered module [98] [tdl_cdlcore_message]
10/29 13:28:19.023 [stack_mgr] [8974]: (note): Examining peer state
10/29 13:28:19.023 [stack_mgr] [8974]: (note): no switch eligible for standby election presently
10/29 13:28:19.022 [stack_mgr] [8974]: (note): Posting event stack_fsm_event_wait_standby_elect_timer_expired, curstate stack_fsm_state_active_ready
10/29 13:28:19.022 [stack_mgr] [8974]: (note): Timer HDL - STACK_WAIT_STANDBY_ELECT_TIMER expired
10/29 13:26:46.584 [btrace] [8974]: (note): Successfully registered module [99] [tdl_ui_message]
10/29 13:26:46.582 [bipc] [8974]: (note): Pending connection to server 10.129.1.0
10/29 13:26:36.582 [evutil] [8974]: (ERR): Connection attempt for sman-ui-serv (uipeer uplink to slot 1) failed, invoking disconnect
10/29 13:26:36.582 [evutil] [8974]: (ERR): Asynchronous connect failed for [uipeer uplink to slot 1] (fd == -1)
10/29 13:26:36.581 [bipc] [8974]: (note): Pending connection to server 10.129.1.0
10/29 13:26:26.581 [evutil] [8974]: (ERR): Connection attempt for sman-ui-serv (uipeer uplink to slot 1) failed, invoking disconnect

Device# show platform software trace message fed switch active
11/02 10:55:01.832 [btrace]: [11310]:  UUID: 0, ra: 0 (note): Successfully registered module [86] [uiutil]
11/02 10:55:01.848 [btrace]: [11310]:  UUID: 0, ra: 0 (note): Single message size is greater than 1024
11/02 10:55:01.822 [btrace]: [11310]:  UUID: 0, ra: 0 (note): Successfully registered module [87] [tdl_cdlcore_message]
11/01 09:54:41.474 [btrace]: [12312]:  UUID: 0, ra: 0 (note): Successfully registered module [88] [tdl_ngwc_gold_message]
11/01 09:54:11.228 [btrace]: [12312]:  UUID: 0, ra: 0 (note): Successfully registered module [89] [tdl_doppler_iosd_matm_type]
11/01 09:53:37.454 [btrace]: [11310]:  UUID: 0, ra: 0 (note): Successfully registered module [90] [tdl_ui_message]
11/01 09:53:37.382 [bipc]: [11310]:  UUID: 0, ra: 0 (note): Pending connection to server 10.129.1.0
11/01 09:53:34.227 [xcvr]: [18846]:  UUID: 0, ra: 0 (ERR): FRU hardware authentication Fail, result = 1.
11/01 09:53:33.775 [ng3k_scc]: [18846]:  UUID: 0, ra: 0 (ERR): SMART COOKIE: SCC I2C receive failed: rc=10
11/01 09:53:33.775 [ng3k_scc]: [18846]:  UUID: 0, ra: 0 (ERR):
SMART COOKIE receive failed, try again
11/01 09:53:33.585 [ng3k_scc]: [18846]:  UUID: 0, ra: 0 (ERR):

show platform software trace level

To view the trace levels for all the modules under a specific process, use the show platform software trace level command in privileged EXEC or user EXEC mode.

show platform software trace level process slot

Syntax Description

process

Process whose tracing level is being set. Options include:

  • chassis-manager —The Chassis Manager process.

  • cli-agent —The CLI Agent process.

  • cmm —The CMM process.

  • dbm —The Database Manager process.

  • emd —The Environmental Monitoring process.

  • fed —The Forwarding Engine Driver process.

  • forwarding-manager —The Forwarding Manager process.

  • geo —The Geo Manager process.

  • host-manager —The Host Manager process.

  • interface-manager —The Interface Manager process.

  • iomd —The Input/Output Module daemon (IOMd) process.

  • ios —The IOS process.

  • license-manager —The License Manager process.

  • logger —The Logging Manager process.

  • platform-mgr —The Platform Manager process.

  • pluggable-services —The Pluggable Services process.

  • replication-mgr —The Replication Manager process.

  • shell-manager —The Shell Manager process.

  • sif —The Stack Interface (SIF) Manager process.

  • smd —The Session Manager process.

  • stack-mgr —The Stack Manager process.

  • table-manager —The Table Manager Server.

  • thread-test —The Multithread Manager process.

  • virt-manager —The Virtualization Manager process.

slot

Hardware slot where the process for which the trace level is set, is running. Options include:

  • number —Number of the SIP slot of the hardware module where the trace level is set. For instance, if you want to specify the SIP in SIP slot 2 of the switch, enter 2.

  • SIP-slot / SPA-bay—Number of the SIP switch slot and the number of the shared port adapter (SPA) bay of that SIP. For instance, if you want to specify the SPA in bay 2 of the SIP in switch slot 3, enter 3/2.

  • F0 —The Embedded Service Processor in slot 0.

  • F1 —The Embedded Service Processor in slot 1.

  • FP active —The active Embedded Service Processor.

  • R0 —The route processor in slot 0.

  • RP active —The active route processor.

  • switch <number> —The switch, with its number specified.

  • switch active —The active switch.

  • switch standby —The standby switch.

    • number —Number of the SIP slot of the hardware module where the trace level is set. For instance, if you want to specify the SIP in SIP slot 2 of the switch, enter 2.

    • SIP-slot / SPA-bay—Number of the SIP switch slot and the number of the shared port adapter (SPA) bay of that SIP. For instance, if you want to specify the SPA in bay 2 of the SIP in switch slot 3, enter 3/2.

    • F0 —The Embedded Service Processor in slot 0.

    • FP active —The active Embedded Service Processor.

    • R0 —The route processor in slot 0.

    • RP active —The active route processor.

Command Modes

User EXEC (>)

Privileged EXEC (#)

Command History

Release

Modification

This command was introduced.

Examples

This example shows how to view the trace level: 


Device# show platform software trace level dbm switch active R0
Module Name                     Trace Level      
-------------------------------------------------
binos                           Notice           
binos/brand                     Notice           
bipc                            Notice           
btrace                          Notice           
bump_ptr_alloc                  Notice           
cdllib                          Notice           
chasfs                          Notice           
dbal                            Informational    
dbm                             Debug            
evlib                           Notice           
evutil                          Notice           
file_alloc                      Notice           
green-be                        Notice           
ios-avl                         Notice           
klib                            Debug            
services                        Notice           
sw_wdog                         Notice           
syshw                           Notice           
tdl_cdlcore_message             Notice           
tdl_dbal_root_message           Notice           
tdl_dbal_root_type              Notice

request platform software trace archive

To archive all the trace logs relevant to all the processes running on a system since the last reload on the switch and to save this in the specified location, use the request platform software trace archive command in privileged EXEC or user EXEC mode.

request platform software trace archive [ last number-of-days [ days [ target location]] | target location]

Syntax Description

last number-of-days

Specifies the number of days for which the trace files have to be archived.

target location

Specifies the location and name of the archive file.

Command Modes

User EXEC (>)

Privileged EXEC (#)

Command History

Release

Modification

This command was introduced.

Usage Guidelines

This archive file can be copied from the system, using the tftp or scp commands.

Examples

This example shows how to archive all the trace logs of the processes running on the switch since the last 5 days: 


Device# request platform software trace archive last 5 days target flash:test_archive

request platform software trace rotate all

To rotate all the current in-memory trace logs into the crashinfo partition and start a new in-memory trace log for each process, use the request platform software trace rotate all command in privileged EXEC or user EXEC mode.

request platform software trace rotate all

Command Modes

User EXEC (>)

Privileged EXEC (#)

Command History

Release

Modification

This command was introduced.

Usage Guidelines

The trace log files are for read-only purpose. Do not edit the contents of the file. If there is a requirement to delete the contents of the file to view certain set of logs, use this command to start a new trace log file.

Examples

This example shows how to rotate all the in-memory trace logs of the processes running on the switch since the last one day: 


Device# request platform software trace slot switch active R0 archive last 1 days target flash:test

request platform software trace filter-binary

To collate and sort all the archived logs present in the tracelogs subdirectory, use the request platform software trace filter-binary command in privileged EXEC or user EXEC mode.

request platform software trace filter-binary modules [ context mac-address]

Syntax Description

context mac-address

Represents the context used to filter. Additionally, you can filter based on module names and trace levels. The context keyword accepts either a MAC address or any other argument based on which a trace is tagged.

Command Modes

User EXEC (>)

Privileged EXEC (#)

Command History

Release

Modification

This command was introduced.

Usage Guidelines

This command collates and sorts all the archived logs present in the tracelogs subdirectory, across all the processes relevant to the module. This command also generates a file named collated_log_{system time} with the same content, in the /crashinfo/tracelogs directory.