System
This chapter describes how to identify and resolve problems related to the Nexus 1000VE system.
Information About the System
Cisco Nexus 1000VE provides Layer 2 switching functions in a virtualized server environment. Nexus 1000VE replaces virtual switches within ESX servers and allows users to configure and monitor the virtual switch using the Cisco NX-OS command line interface. Nexus 1000VE also gives you visibility into the networking components of the ESX servers and access to the virtual switches within the network.
The Nexus 1000VE manages a data center defined by the vCenter Server. Each server in the Datacenter is represented as a linecard in Nexus 1000VE and can be managed as if it were a line card in a physical Cisco switch. The Nexus 1000VE implementation has two components:
- Virtual Supervisor Module (VSM) – This is the control software of the Nexus 1000VE distributed virtual switch. It runs on a virtual machine (VM) and is based on NX-OS.
- Virtual Services Engine (VSE) – This is the part of Cisco Nexus 1000VE that actually switches data traffic. It runs as VM on a VMware ESX 6.0 and above host.Several VSEs are controlled by one VSM. All the VSEs that form a switch domain should be in the same virtual Datacenter as defined by VMware vCenter Server.
General Restrictions for vCenter Server
When you are troubleshooting issues related to vCenter Server, make sure that you observe the following restrictions:
- The name of a distributed virtual switch (DVS) name must be unique across Datacenters
- You create a DVS in a network folder
- A Datacenter cannot be removed unless the DVS folder or the underlying DVS is deleted.
- A DVS can be deleted only with the help of VSM using the no vmware dvs command in config-svs-conn mode.
- The no vmware dvs command can succeed only if there are no VMs using the DVS port-groups.
- A port group on vCenter Server can be deleted only if there are no interfaces associated with it.
- A sync operation performed in conjunction with the connect command helps VSM keep in sync with vCenter Server.
- Each VSM uses a unique extension key to communicate with vCenter Server and perform operations on a DVS.
Extension Key
The VSM uses the extension key when communicating with the vCenter Server. Each VSM has its own unique extension key, such as Cisco_Nexus_1000VE_32943215
Use the show vmware vc extension-key command to find the extension key of the VSM. It is also listed in the.xml file.
The extension key registered on the vCenter Server can be found through the MOB. For more information, see the Finding the Extension Key Tied to a Specific DVS.
The same extension key cannot be used to create more than one DVS on the vCenter Server.
Recovering a DVS
You can use this procedure to recover a DVS if the VSM VM that was used to create it is lost or needs to be replaced. This section includes the following procedures:
Recovering a DVS With a Saved Copy of the VSM
You can use this procedure to recover a DVS when you have previously saved a back up copy of the VSM configuration file.
BEFORE YOU BEGIN
Before starting this procedure, you must know or do the following:
- Use this procedure if you have previously saved a back up copy of the VSM configuration file. If you have not previously saved a back up copy, the see the Recovering a DVS Without a Saved Copy of the VSM.
- Make sure that the VSM VM switchname is the same as the DVS switchname on the vCenter Server. This allows the VSM configuration to synchronize with the correct DVS on the vCenter Server.
To change the VSM switchname use the switchname newname command.
Step 1
From the MOB, find the DVS extension key.
For more information, see the Finding the Extension Key Tied to a Specific DVS.
Step 2 On the VSM, add the DVS extension key found in Step 1.
The extension key allows the VSM to log in to the vCenter server.
Example:
switch# config t
switch(config)# vmware vc extension-key Cisco_Nexus_1000V_32943215
Step 3 From the MOB, unregister the extension key found in Step 1.
For more information, see the Unregistering the Extension Key in the vCenter Server.
Step 4 From the VC client, register the extension (plug-in) for the VSM.
For more information see the following procedure in the Cisco Nexus 1000V Getting Started Guide.
Step 5 On the VSM, restore the configuration using a previously saved copy of the VSM configuration file.
copy path/filename running-config
Example:
switch# copy sftp://user1@172.22.36.10/backup/hamilton_cfg running-config
Step 6 Do one of the following:
- If the vCenter server connection is not part of the previously saved configuration, continue with the next step.
- Otherwise, go to Step 8.
Step 7 On the VSM, restore the configuration for the vCenter server connection.
Example:
Step 8 Connect to vCenter Server.
Example:
You can now use the old DVS or remove it.
Recovering a DVS Without a Saved Copy of the VSM
You can use this procedure to recover a DVS when you have not previously saved a back up copy of the VSM configuration file.
BEFORE YOU BEGIN
Before starting this procedure, you must know or do the following:
– At the root-level of the Data Center in which it resides.
It cannot be embedded in another folder.
– Of the same name as the VSM.
If the folder does not meet the above criteria, the connection to vCenter server fails with the error, the VSM already exists.
- Use this procedure if you have not previously saved a back up copy of the VSM configuration file. If you have previously saved a back up copy, then see the Recovering a DVS With a Saved Copy of the VSM.
- If you have not previously saved a back up copy of the VSM configuration file, then you may try recreating the old port profiles before connecting to the VC. This procedure has a step for recreating port profiles. If you do not recreate these before connecting to VC, then all the port groups present on the VC are removed and all ports in use are moved to the quarantine port groups.
- Make sure that the VSM VM switchname is the same as the DVS switchname on the vCenter Server. This allows the VSM configuration to synchronize with the correct DVS on the vCenter Server.
To change the VSM switchname use the switchname newname command.
Step 1 From the MOB, find the DVS extension key.
For more information, see the Finding the Extension Key Tied to a Specific DVS.
Step 2 On the VSM, add the DVS extension key found in Step 1.
The extension key allows the VSM to log in to the vCenter server.
Example:
switch# config t
switch(config)# vmware vc extension-key Cisco_Nexus_1000Ve_32943215
Step 3 From the MOB, unregister the extension key found in Step 1.
For more information, see the Unregistering the Extension Key in the vCenter Server.
Step 4 From the VC client, register the extension (plug-in) for the VSM.
For more information see the following procedure in the Cisco Nexus 1000V Getting Started Guide.
Step 5 Manually recreate the old port profiles from your previous configuration.
For more information, see the following procedures in the Cisco Nexus 1000V Getting Started Guide.
- Configuring the system port profile for VSM-VSE Communication
- Configuring the uplink port profile for VM Traffic
- Configuring the data port profile for VM Traffic
Note If you do not manually recreate the port profiles, then all port groups on the vCenter Server are removed when the VSM connects.
Step 6 On the VSM, restore the configuration for the vCenter server connection.
Example:
Step 7 Connect to vCenter Server.
Example:
You can now use the old DVS or remove it.
Problems Related to VSM and vCenter Server Connectivity
Example 15-1 shows the show vms internal event-history errors command that is useful for examining VC errors in detail. It shows whether an error is caused by a VSM (client) or the server.
Example 15-1 show vms internal event-history error Command
Setting the System MTU
Use this procedure to set a system MTU in your existing system uplink port profiles.
BEFORE YOU BEGIN
Before beginning this procedure, you must know or do the following:
- You are logged in to the CLI in EXEC mode.
- The system port profiles are already configured and you know the uplink profile names.
For more information, see the Cisco Nexus 1000V Port Profile Configuration Guide.
- The MTU size you set for the system mtu on the port profile must be less than the size of the system jumbomtu configured on the interface.
For more information about configuring MTU on the interface, see the Cisco Nexus 1000V Interface Configuration Guide.
SUMMARY STEPS
4. show port-profil e [ brief | expand-interface | usage ] [ name profilename]
DETAILED STEPS
Recovering Lost Connectivity Due to MTU Mismatch
Use this procedure to recover lost connectivity due to an MTU mismatch between the physical NIC and the VMware kernel NIC after an ESX reboot.
BEFORE YOU BEGIN
Before beginning this procedure, you must know or do the following:
- You are logged in to the CLI in EXEC mode.
- To verify the ESX MTU settings for corresponding PNICs, use the ESXcfg-nics -l command.
Note Use vemcmds only as a recovery measure and then update the MTU value in the port profile configuration for system uplinks or in the interface configuration for non-system uplinks.
SUMMARY STEPS
2. module vse module_number execute vemcmd show port port-LTL-number
3. module vse module_number execute vemcmd set mtu size ltl port-LTL-number
DETAILED STEPS
|
|
|
|
|---|---|---|
module vse module_number execute vemcmd show port port-LTL-number |
Displays the port configuration including the LTL number needed for Step 3. |
|
|
|
||
module vse module_number execute vemcmd set mtu size ltl port-LTL-number switch(config)# module vse 3 execute vemcmd set mtu 9000 ltl 17 |
Designates the MTU size for the port, using the LTL number obtained in Step 2. |
|
VSM Creation
|
|
|
|
|---|---|---|
Make sure the NIC order is correct: control, management, inband/outband. |
||
The VSM VM can ping the gateway, but not the outside subnet. |
Port Profiles
When creating a port profile, use the following commands to create the corresponding port groups on the vCenter Server:
Profiles that have the system VLAN configuration allow the VSE to communicate with the VSM.
Make sure that the system port-profile is defined with the right system VLANS.
Use the show port-profile and show port-profile usage commands to collect basic required information.
Problems with Port Profiles
Problems with Hosts
Problems with VM Traffic
When troubleshooting problems with intra-host VM traffic, follow these guidelines:
- Make sure that at least one of the VMware virtual NICs is on the correct DVS port group and is connected.
- If the VMware virtual NIC is down, determine if there is a conflict between the MAC address configured in the OS and the MAC address assigned by VMware. You can see the assigned MAC addresses in the vmx file.
When troubleshooting problems with inter-host VM traffic, follow these guidelines:
VSE Troubleshooting Commands
Use the following commands to display VSE information:
- vemlog – displays and controls VSE kernel logs
- vemcmd – displays configuration and status information
- vem-support all – collects support information
- systemctl status nexus1000v – collects status information
- vemcmd show version – collects version information
- vemlog show last number-of-entries – displays the circular buffer
Example 15-2 vemog show last Command
Example 15-3 vemcmd show info Command
Example 15-4 vemcmd help Command
VSE Log Commands
Use the following commands to control the VSElog:
- VEMlog stop – stops the log
- VEMlog clear – clear s the log
- VEMog start number-of-entries – starts the log and stops it after the specified number of entries
- VEMlog stop number-of-entries – stops the log after the next specified number of entries
- VEMlog resume – starts the log, but does not clear the stop value
Error Messages
On the vSphere Client, you can see error messages under the recent tasks tab. You can find detailed description of the error under the Tasks and Events tab. The same messages are also propagated to the VSM.
Table 15-1 lists error messages that you might see on the VSM.
Feedback