Configuring Interswitch Link Diagnostics

This chapter describes how to configure the ISL Diagnostics on the Cisco MDS switches.

Information About ISL Diagnostics

ISL Diagnostics feature helps in validating the health of interswitch links between the Cisco MDS switches in a network.

ISL Diagnostic Overview

ISL diagnostics is supported on the following platforms:

  • Cisco MDS 9500 Series Switches

  • Cisco MDS 9700 Series Switches

  • Cisco MDS 9396S Switch

ISL diagnostics is supported on the following FC modules of the Cisco MDS Switches:

  • Advanced 8-Gbps modules on Cisco MDS 9500 Series Switches

    • DS-X9232-256K9

    • DS-X9248-256K9

  • 16-Gbps modules on Cisco MDS 9700 Series Switches

    • DS-X9448-768K9


Note
The ISL diagnostic support on the modules is limited to the generator and reflector ports only.

  • The diagnostic tests can be performed between two supported modules of different switch families on either side.

The following tests can be performed using ISL diagnostics:

  • Latency test

  • Single Hop Traffic Test

  • Multihop End-to-End Traffic Test

ISL diagnostics is not supported on the following FC modules of the Cisco MDS Switches:

  • DS-X9224-96K9

  • DS-X9248-96K9

  • DS-X9248-48K9

  • DS-X9304-18K9


Note

  • ISL diagnostics are not supported on other non-MDS switches such as Nexus 2000 and Nexus 5000.

  • ISL diagnostics are not supported on Cisco MDS 9148S 16G Multilayer Fabric Switch and Cisco MDS 9250i Multiservice Fabric Switch.

  • ISL diagnostics are not supported on any of the FCoE and IPS modules of the Cisco MDS Switches.

  • ISL diagnostics are not supported on FEC-enabled links.

  • The ISL diagnostic feature is interoperable between the Cisco MDS 9700 and 9500 switches. (For example, for a particular ISL diagnostic test, the generator switch can be a Cisco MDS 9700 switch and the reflector switch can be a Cisco MDS 9500 switch or vice versa.)

Latency Test

Latency test measures the latency of an ISL between two Cisco MDS switches.

The frame is looped back by the reflector switch port to the generator switch where the timestamps are captured.

Timestamps allow the latency of the link to be measured in both directions as well as the latency of the reflector port. The cable length is calculated using only the link latencies. The accuracy of the reported cable length is +/- 2 meters. On Cisco MDS switches, the cable length (for latency test) is validated up to 50 meters of the cable length.


Note
When a latency test is executed, there should not be any other traffic running on the same link.

Figure 1 shows the details for the latency test.

Figure 1. Latency Test

Single Hop Traffic Test

The Single Hop Traffic Test validates the health of an ISL by checking the efficiency of the ISL to handle traffic at various frame rates.

Fibre Channel (FC) frames are generated in the generator switch using the internal traffic generator facility available in the MAC hardware. These frames are transmitted from the generator switch port over the ISL under test. The reflector switch receives the frames, switches them via the normal fabric switching path and transmits the frames back through the received port onto the ISL under test.

The efficiency of the ISL traffic is calculated based on the number of packets received back on the generator switch port.


Note
When a single hop test is executed, there should not be any other traffic running on the same link.

The traffic test returns with an error in the following scenarios:

  • If the ISL is not up.

  • If the generator port does not have an internal traffic generator facility.

  • When the reflector is not put in a loopback mode.

Figure 1 shows the details for Single Hop Traffic Test:

Figure 2. Single Hop Traffic Test

Multihop End-to-End Traffic Test

The Multihop test evaluates the health of the ISL between a host switch and a target switch in a fabric.

Before connecting a host to a target in a fabric, test the fabric path between the host port and the target port, using a Multihop test.

Multiple hops can exist between the host switch and the target switch. No specific configuration is required on the intermediate switches.


Note
The intermediate switches in the fabric can have any interface or link, for example, FC, FCoE, IPS, and so on, between them, as long as a route exists between the generator and the reflector port.

Fibre channel (FC) frames are generated at the generator switch port and transmitted to the first hop link. These frames traverse intermediate switches until they reach the reflector switch. The reflector switch then switches the frames and returns them to the generator switch. Based on the number of packets received on the generator switch, the efficiency of the ISL is displayed.

The Multihop traffic test is based on the domain IDs of the generator and reflector switches.


Note
When a Multihop traffic test is executed, there should not be any other traffic running on both the generator and the reflector ports; however, there can be traffic running over the ISLs that is used by the Multihop traffic test.

Figure 1 shows the details for Multihop End-to-End Traffic Test:

Figure 3. Multihop End-to-End Traffic Test

Configuring ISL Diagnostics

Configuring Latency Test on Cisco MDS 9700 Series Switches

To configure the latency test between the generator and reflector switches perform the following tasks:

Procedure

Step 1

Enable the test interface on the reflector switch for testing latency by setting it to loopback mode using the following command:

switch B# diagnostic isl reflector latency_test loop-back interface interface id enable

Step 2

Configure the generator switch to run the test and display the results:

switch A# diagnostic isl latency-test interface interface id

Step 3

To disable the reflector port for latency testing configure the following command on the reflector switch:

switch B# diagnostic isl reflector latency_test loop-back interface interface id disable

Latency Test

This example shows how to enable a port on a reflector switch for latency testing: 


switch B# diagnostic isl reflector latency_test loop-back interface fc1/13 enable
Reflector Configuration Successful.

This example shows how to run the latency test: 


switch A# diagnostic isl latency-test interface fc4/1
Waiting for sync to be achieved on the link ....
Sync is achieved, Link has been initialized.
Starting the test ....
---------------------------------------------------------
Latency test Result for port:                    fc4/1
Latency in the switch (in ns):                   399
Latency in the cable (in ns):                    39
Length of the cable (accuracy +/- 2m):           4 m
---------------------------------------------------------

This example shows how to disable a port on a reflector switch for latency testing: 


switch B# diagnostic isl reflector latency_test loop-back interface fc1/13 disable
Reflector Configuration Successful.

Configuring Latency Test on Other Supported Platforms

To configure the latency test between the generator and reflector switches perform the following tasks:

Procedure

Step 1

Enable the test interface on the reflector switch for testing latency by setting it to loopback mode using the following command:

switch B# system health isl reflector latency_test loop-back interface interface id enable

Step 2

Configure the generator switch to run the test and display the results:

switch A# system health isl latency-test interface interface id

Step 3

To disable the reflector port for latency testing configure the following command on the reflector switch:

switch B# system health isl reflector latency_test loop-back interface interface id disable

Latency Test

This example shows how to enable a port on a reflector switch for latency testing: 


switch B# system health isl reflector latency_test loop-back interface fc4/25 enable
Reflector Configuration Successful.

This example shows how to run the latency test: 


switch A# system health isl latency-test interface fc 1/13
Waiting for sync to be achieved on the link ....
Sync is achieved, Link has been initialized.
Starting the test ....


---------------------------------------------------------
Latency test Result for port: fc1/13
Latency in the switch (in ns): 5504
Latency in the cable (in ns): 664
Length of the cable (accuracy +/- 2m): 4.816514 m
---------------------------------------------------------

This example shows how to disable a port on a reflector switch for latency testing: 


switch B# system health isl reflector latency_test loop-back interface fc4/25 disable
Reflector Configuration Successful.

Configuring a Single Hop Traffic Test on Cisco MDS 9700 Series Switches

To configure a Single Hop Traffic Test between the generator switch and the reflector switch, perform the following tasks:

Procedure

Step 1

Enable the test interface on the reflector switch for Single Hop Traffic Test by setting it to loopback mode using the following command:

switch B# diagnostic isl reflector traffic_test loop-back interface interface id enable

Step 2

Configure the interface using one of the following options:

  • Configure the interface on the generator switch to run the traffic test for a given frame count, frame size, and rate (link speed) parameters:

    switch A# diagnostic isl generator interface interface id start frame-count number rate value frame_size min minimum size max maximum size step num

  • Configure the interface on the generator switch to run the traffic test for a given duration, frame size, and rate (link speed) parameters:

    switch A# diagnostic isl generator interface interface id start duration seconds rate value frame_size min minimum size max maximum size step num

Step 3

To stop the Single Hop Traffic Test or to display the test result on the generator switch use the following command:

switch A# diagnostic isl generator interface interface id stop

Step 4

To disable the reflector port for Single Hop Traffic Test configure the following command on the reflector switch:

switch B# diagnostic isl reflector traffic_test loop-back interface interface id disable

Step 5

View the results of the Single Hop Traffic Test:

switch B# show diagnostic isl result interface interface id

Single Hop Traffic Test

This example shows how to enable the test interface on the reflector switch for Single Hop Traffic Test by setting it to loopback mode: 


switch B# diagnostic isl reflector traffic_test loop-back interface fc9/37 enable
Reflector Configuration Successful.

This example shows how to run a traffic test on the generator switch for a particular duration, speed, and frame size parameters: 


switch A# diagnostic isl generator interface fc4/5 start duration 100 rate 25% frame_size min 16 max 517 step 1

This example shows how to run and stop a traffic test stop and display the results for duration parameter: 


switch A# diagnostic isl generator interface fc4/3 start duration 10
Waiting for sync to be achieved on the link ....
Link initialized successfully. Starting the test.

switch A# diagnostic isl generator interface fc4/3 stop
---------------------------------------------------------
Traffic test Result for port:                    fc4/3
Packets Transmitted:                             6245142
Packets Recieved:                                6245142
ISL traffic Efficiency (percent):                100.0000
---------------------------------------------------------

switch B# diagnostic isl reflector traffic_test loop-back interface fc9/37 disable
Reflector Configuration Successful.

This example shows the results of the Single Hop Traffic Test:

switch B# show  diagnostic isl result interface fc 5/3
---------------------------------------------------------
Single hop  Traffic test Result for port: fc5/3
Packets Transmitted:                             30621868
Packets Recieved:                                30621868
ISL traffic Efficiency (percent):                100.0000
---------------------------------------------------------

Configuring a Single Hop Traffic Test on Other Supported Platforms

To configure a Single-Hop Traffic Test between the generator switch and the reflector switch, perform the following tasks:

Procedure

Step 1

Enable the test interface on the reflector switch for Single Hop Traffic Test by setting it to loopback mode using the following command:

switch B# system health isl reflector traffic_test loop-back interface interface id enable

Step 2

Configure the interface using one of the following options:

  • Configure the interface on the generator switch to run the traffic test for a given frame count, frame size, and rate (link speed) parameters:

    switch A# system health isl generator interface interface id start frame-count number rate value frame_size min minimum size max maximum size step num

  • Configure the interface on the generator switch to run the traffic test for a given duration, frame size, and rate (link speed) parameters:

    switch A# system health isl generator interface interface id start duration seconds rate value frame_size min minimum size max maximum size step num

Step 3

To stop the Single Hop Traffic Test or to display the test results on the generator switch use the following command:

switch A# system health isl generator interface interface id stop

Step 4

To disable the reflector port for Single Hop Traffic Test configure the following command on the reflector switch:

switch B# system health isl reflector traffic_test loop-back interface interface id disable

Step 5

View the results of the Single Hop Traffic Test:

switch B# show system health isl result interface interface id

Single Hop Traffic Test

This example shows how to enable the test interface on the reflector switch for Single Hop Traffic Test by setting it to loopback mode: 


switch B# system health isl reflector traffic_test loop-back interface fc9/37 enable
Reflector Configuration Successful.

This example shows how to run and stop a traffic test and display the result for duration parameter on a generator switch: 


switch A# system health isl generator interface fc12/16 start duration 100
Waiting for sync to be achieved on the link .....
Link initialized successfully. Starting the test.

switch A# system health isl generator interface fc12/16 stop
---------------------------------------------------------
Traffic test Result for port:                    fc12/16
Packets Transmitted:                             5293153
Packets Recieved:                                5293153
ISL traffic Efficiency (percent):                100.0000
---------------------------------------------------------

switch B# system health isl reflector traffic_test loop-back interface fc9/37 disable
Reflector Configuration Successful.

This example shows the results of the Single Hop Traffic Test:

switch B# show system health isl result interface fc 1/18
---------------------------------------------------------
Single hop  Traffic test Result for port:      fc1/18
Packets Transmitted:                           1019885186
Packets Recieved:                              1019885186
ISL traffic Efficiency (percent):              100.0000     
---------------------------------------------------------

Configuring a Multihop Traffic Test on Cisco MDS 9700 Series Switches

To configure a Multihop Traffic Test between the generator switch and the reflector switch, perform the following tasks:

Procedure

Step 1

Enable the test interface on the reflector switch by setting it to loopback mode for a given VSAN and domain ID of the generator switch for Multihop Traffic Test:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch B# diagnostic isl multi_hop reflector loop-back interface interface id enable vsan vsan id source-domain source id

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch B# diagnostic isl multi_hop reflector loop-back interface interface id vsan vsan id source-domain source id enable

To obtain the source domain use the following command on the reflector switch:

switch B# show fcdomain domain-list vsan vsan id

Step 2

Configure the interface on the generator switch to run the Multihop Traffic Test for a given VSAN, destination domain (domain ID of the reflector switch), frame count, link speed, and frame size parameters:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch A# diagnostic isl multi_hop generator interface interface id start vsan vsan id dest-domain dest id frame-count number rate value frame_size min minimum size max maximum size step num

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch A# diagnostic isl multi_hop generator interface interface id vsan vsan id dest-domain dest id startframe-count number rate value frame_size min minimum size max maximum size step num

Configure the interface on the generator switch to run the Multihop Traffic Test for a given VSAN, destination domain (domain ID of the reflector switch), duration, rate (link speed), and frame size parameters:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch A# diagnostic isl multi_hop generator interface interface id start vsan vsan id dest-domain dest id duration seconds rate value frame_size min minimum size max maximum size step num

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch A# diagnostic isl multi_hop generator interface interface id vsan vsan id dest-domain dest idstart duration seconds rate value frame_size min minimum size max maximum size step num

To obtain the destination domain use the following command on the generator switch:

switch A# show fcdomain domain-list vsan vsan id

Step 3

To stop the Multihop Traffic Test or to display the test results on the generator switch use the following command:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch A# diagnostic isl multi_hop generator interface interface id stop

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch A# diagnostic isl multi_hop generator interface interface id vsan vsan id dest-domain dest id stop

Step 4

To disable the reflector port for Multihop Traffic Test configure the following command on the reflector switch:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch B# diagnostic isl multi_hop reflector loop-back interface interface id disable

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch B# diagnostic isl multi_hop reflector loop-back interface interface id vsan vsan idsource-domain source id disable

Step 5

View the results of the Multihop Traffic Test:

switch B# show diagnostic isl result interface interface id

MultiHop Traffic test

This example shows how to display the domain list on both the generator and reflector switches: 


switch# show fcdomain domain-list vsan 1
Number of domains: 3
Domain ID              WWN
---------    -----------------------
0x85(133)    20:01:00:0d:ec:b7:20:01 [Principal]
0xef(239)    20:01:40:55:39:0c:70:81 [Local]
0x02(2)      20:01:00:0d:ec:b7:28:c1

This example shows how to enable the test interface on the reflector switch by setting it to loopback mode for a given VSAN and domain ID of the generator switch for Multihop Traffic Test: 


switch B# diagnostic isl multi_hop reflector loop-back interface fc9/36 enable vsan 1 source_domain 239

This example shows how to run, stop, and display the results of the traffic test on the generator switch for frame count parameter: 


switch A# diagnostic isl multi_hop generator interface fc4/10 start vsan 1 dest_domain 133 duration 100
switch A# diagnostic isl multi_hop generator interface fc4/10 stop
Generator is stopped.  Clean-up in progress.
Please wait....
---------------------------------------------------------
Traffic test Result for port:                    fc4/10
Packets Transmitted:                             6291024
Packets Recieved:                                6291024
ISL traffic Efficiency (percent):                100.0000
---------------------------------------------------------
switch B# diagnostic isl multi_hop reflector loop-back interface fc9/36 disable

This example shows how to run a traffic test on the generator switch for a particular duration, speed, and frame size parameters: 


switch A# diagnostic isl multi_hop generator interface fc4/10 start vsan 1 dest_domain 133 duration 100 rate 16G frame_size min 16 max 517 step 1

switch#diagnostic isl multi_hop generator interface fc 1/7 stop
Generator is stopped. Clean-up in progress.
Please wait....

---------------------------------------------------------
Traffic test Result for port: fc1/7
Packets Transmitted: 52415159
Packets Recieved: 52415159
ISL traffic Efficiency (percent): 100.0000
---------------------------------------------------------

This example shows the results of the Multihop Traffic Test:

switch#show diagnostic isl result interface fc 4/17

---------------------------------------------------------
Multi hop Traffic test Result for port: 	  fc4/17
Packets Transmitted:                             6131424
Packets Recieved:                                6131424
ISL traffic Efficiency (percent):                100.0000
---------------------------------------------------------

Configuring a Multihop Traffic Test on Other Supported Platforms

To configure a Multihop Traffic Test between the generator switch and the reflector switch, perform the following tasks:

Procedure

Step 1

Enable the test interface on the reflector switch by setting it to loopback mode for a given VSAN and domain ID of the generator switch for Multihop Traffic Test:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch B# system health isl multi_hop reflector loop-back interface interface idenable vsan vsan id source-domain source id

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch B# system health isl multi_hop reflector loop-back interface interface id vsan vsan id source-domain source id enable

To obtain the source domain use the following command on the reflector switch:

switch B# show fcdomain domain-list vsan vsan id

Step 2

Configure the interface on the generator switch to run the Multihop Traffic Test for a given VSAN, destination domain (domain ID of the reflector switch), frame count, link speed, and frame size parameters:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch A# system health isl multi_hop generator interface interface id start vsan vsan id dest-domain dest id frame-count number rate value frame_size min minimum size max maximum size step num

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch A# system health isl multi_hop generator interface interface id vsan vsan id dest-domain dest id start frame-count number rate value frame_size min minimum size max maximum size step num

Configure the interface on the generator switch to run the Multihop Traffic Test for a given VSAN, destination domain (domain ID of the reflector switch), duration, rate (link speed), and frame size parameters:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch A# system health isl multi_hop generator interface interface id start vsan vsan id dest-domain dest id duration seconds rate value frame_size min minimum size max maximum size step num

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch A# system health isl multi_hop generator interface interface id vsan vsan id dest-domain dest id start duration seconds rate value frame_size min minimum size max maximum size step num

To obtain the destination domain use the following command on the generator switch:

switch A# show fcdomain domain-list vsan vsan id

Step 3

To stop the Multihop Traffic Test or to display the test result on the generator switch use the following command:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch A# system health isl multi_hop generator interface interface id stop

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch A# system health isl multi_hop generator interface interface id vsan vsan id dest-domain dest id stop

Step 4

To disable the reflector port for Multihop Traffic Test configure the following command on the reflection switch:

From Cisco MDS NX-OS Release 8.4(1) and later, use the following command:

switch B# system health isl multi_hop reflector loop-back interface interface id disable

For Cisco MDS NX-OS Release 8.3(2) and earlier, use the following command:

switch B# system health isl multi_hop reflector loop-back interface interface id vsan vsan id source-domain source id disable

Step 5

View the results of the Multihop Traffic Test:

switch B# show system health isl result interface interface id

Multihop Traffic test

This example shows how to display the domain list on both the generator and reflector switches: 


switch# show fcdomain domain-list vsan 1
Number of domains: 3
Domain ID              WWN
---------    -----------------------
0x85(133)    20:01:00:0d:ec:b7:20:01 [Principal]
0xef(239)    20:01:40:55:39:0c:70:81 [Local]
0x02(2)      20:01:00:0d:ec:b7:28:c1

This example shows how to enable loop back to the generator switch interface present in a VSAN from the reflector switch for a Multihop Traffic Test: 


switch B# system health isl multi_hop reflector loop-back interface fc9/36 enable vsan 1 source_domain 239

This example shows how to start, stop, and display the results of the traffic test on the generator switch for frame count parameter: 


switch A# system health isl multi_hop generator interface fc3/18 start vsan 1 dest_domain 2 frame-count 1000000
switch A# system health isl multi_hop generator interface fc3/18 stop
Generator is stopped.  Clean-up in progress.
Please wait....
---------------------------------------------------------
Traffic test Result for port:                    fc3/18
Packets Transmitted:                             1000000
Packets Recieved:                                1000000
ISL traffic Efficiency (percent):                100.0000
---------------------------------------------------------



switch B# system health isl multi_hop reflector loop-back interface fc9/36 disable

This example shows how to run a traffic test on the generator switch for a particular duration, speed, and frame size parameters: 


switch A# system health isl multi_hop generator interface fc4/5 start vsan 1 dest_domain 133 duration 100 rate 16G frame_size min 16 max 517 step 1

This example shows the results of the Multihop Traffic Test:

switch B# show system health isl result interface fc 1/18
---------------------------------------------------------
Multi hop Traffic test Result for port: 	  fc1/7
Packets Transmitted:                             3065550
Packets Recieved:                                3065550
ISL traffic Efficiency (percent):                100.0000
---------------------------------------------------------

Debugging ISL Diagnostics

The following table lists the debug commands for this feature. To display the ISL diagnostic test status use one of the following commands:

Table 1. Debug commands

Command

Reference

Cisco MDS 9700 Switches [Cisco MDS NX-OS Release 8.2(1) and earlier]

diagnostic isl show status start index num number

Displays the status of configured ISL diagnostic tests per port.

Cisco MDS 9700 Switches [Cisco MDS NX-OS Release 8.3(1) and later]

system health isl show status

Example: 


switch# system health isl show status
show status of isl_daig:
---------------------------------------------------------
Index: 0 if_index:0x110f000 :is_running: 0 is_reflector:1 is_latency:1 is_multihop:0
Index: 1 if_index:0x0 :is_running: 0 is_reflector:0 is_latency:0 is_multihop:0
Index: 2 if_index:0x0 :is_running: 0 is_reflector:0 is_latency:0 is_multihop:0
Index: 3 if_index:0x0 :is_running: 0 is_reflector:0 is_latency:0 is_multihop:0
Index: 4 if_index:0x0 :is_running: 0 is_reflector:0 is_latency:0 is_multihop:0
Index: 5 if_index:0x0 :is_running: 0 is_reflector:0 is_latency:0 is_multihop:0
Index: 6 if_index:0x0 :is_running: 0 is_reflector:0 is_latency:0 is_multihop:0
Index: 7 if_index:0x0 :is_running: 0 is_reflector:0 is_latency:0 is_multihop:0
Index: 8 if_index:0x0 :is_running: 0 is_reflector:0 is_latency:0 is_multihop:0
Index: 9 if_index:0x0 :is_running: 0 is_reflector:0 is_latency:0 is_multihop:0

Displays the status of configured ISL diagnostic tests per port.

Additional References

For additional information related to implementing online diagnostics, see the following sections:

Related Documents

Related Topic

Document Title

InterSwitch Link Diagnostics CLI commands

Cisco MDS 9000 Family Command Reference

Feature History for Online Diagnostics

Table 1 lists the release history for this feature.

Table 2. Feature History for Online Diagnostics

Feature Name

Releases

Feature Information

ISL Diagnostics

7.3(0)D1(1)

This feature was introduced.