You can use System Administration > Trace page (the equivalent of the trace command in the CLI) to debug the flow of messages through the system by emulating sending a test message. The Trace page (and trace CLI command) emulates a message as being accepted by a listener and prints a summary of features that would have been “triggered” or affected by the current configuration of the system (including uncommitted changes ). The test message is not actually sent. The Trace page (and trace CLI command) can be a powerful troubleshooting or debugging tool, especially if you have combined many of the advanced features available on the Cisco appliance .

Note	Trace is not effective for testing file reputation scanning.

The Trace page (and trace CLI command) prompts you for the input parameters listed in the following table.

After you have entered the values, click Start Trace. A summary of all features configured on the system affecting the message is printed.

You can upload message bodies from your local file system. (In the CLI, you can test with message bodies you have uploaded to the /configuration directory. See FTP, SSH, and SCP Access for more information on placing files for import onto the Cisco appliance .)

After the summary is printed, you are prompted to view the resulting message and re-run the test message again. If you enter another test message, the Trace page and the trace command uses any previous values from the above table you entered.

Note

The sections of configuration tested by the trace command listed in the following table are performed in order . This can be extremely helpful in understanding how the configuration of one feature affects another. For example, a recipient address transformed by the domain map feature will affect the address as it is evaluated by the RAT. A recipient that is affected by the RAT will affect the address as it is evaluated by alias table, and so on.

“Sinkhole” listeners allow you to test your message generation systems and to also get a rough measure of receiving performance. Two types of sinkhole listeners are queueing and non-queueing .

• The queueing listener saves the message to the queue, but then immediately deletes it. Use a queuing listener when you are interested in measuring the performance of the entire injection portion of your message generation system.
• The non-queueing listener accepts a message, and then immediately deletes it without saving it. Use the non-queueing listener when you want to troubleshoot the connection from your message generation system to the appliance .

For example, in the following figure, you could create a sinkhole listener “C” to mirror the private listener labeled “B.” A non-queueing version tests the performance path of the system from the groupware client to the groupware server to the appliance . A queueing version tests that same path and the appliance's ability to enqueue messages and prepare them for delivery via SMTP.

In the following example, the listenerconfig command is used to create a sinkhole queueing listener named Sinkhole_1 on the Management interface. This Host Access Table (HAT) for the listener is then edited to accept connections from the following hosts:

• yoursystem.example.com
• 10.1.2.29
• badmail.tst
• .tst

Note	The final entry, .tst , configures the listener so that any host in the .tst domain can send email to the listener named Sinkhole_1 .

If you suspect that the appliance has network connectivity issues, first confirm that the appliance is working properly.

If you suspect problems with injecting email, use the following strategies:

• Confirm the IP address that you are injecting from, and then use the listenerconfig command to check for allowed hosts.

  Is the IP address allowed to connect to the listener you have created? Use the listenerconfig command to examine the Host Access Table (HAT) for the listener. Use these commands to print the HAT for a listener:

  listenerconfig -> edit -> listener_number -> hostaccess -> print

  The HAT can be configured to refuse connections by IP address, block of IP addresses, hostname, or domains. For more information, see “Specifying Hosts that are Allowed to Connect”.

  You can also use the limits subcommand to check the maximum number of connections allowed for a listener:

  listenerconfig -> edit -> listener_number -> limits

• On the machine that you are injecting from, use Telnet or FTP to manually connect to the appliance . For example:

  
  injection_machine% telnet appliance_name
  

  You can also use the telnet command within the appliance itself to connect from the listener to the actual appliance :

  mail3.example.com> telnet
  
  Please select which interface you want to telnet from.
  
  1. Auto
  
  2. Management (192.168.42.42/24: mail3.example.com)
  
  3. PrivateNet (192.168.1.1/24: mail3.example.com)
  
  4. PublicNet (192.168.2.1/24: mail3.example.com)
  
  [1]> 3
  
  Enter the remote hostname or IP.
  
  []> 193.168.1.1
  
  Enter the remote port.
  
  [25]> 25
  
  Trying 193.168.1.1...
  
  Connected to 193.168.1.1.
  
  Escape character is '^]'.
  
  

  If you cannot connect from one interface to another, you may have issues with the way in which the appliance's Management and Data1 and Data2 interfaces are connected to your network. See FTP, SSH, and SCP Access for more information. You can telnet to port 25 of the listener and enter SMTP commands manually (if you are familiar with the protocol).

• Examine the IronPort text mail logs and injection debug logs to check for receiving errors.

  Injection debug logs record the SMTP conversation between the appliance and a specified host connecting to the system. Injection debug logs are useful for troubleshooting communication problems between the appliance and a client initiating a connection from the Internet. The log records all bytes transmitted between the two systems and classifies them as “Sent to” the connecting host or “Received from” the connecting host.

  For more information, see Using Text Mail Logs and Using Injection Debug Logs.

If you suspect problems with delivering email from the appliance , try the following strategies:

• Determine if the problem is domain-specific.

  Use the tophosts command to get immediate information about the email queue and determine if a particular recipient domain has delivery problems.

  Are there problem domains returned when you sort by “Active Recipients?”

  When you sort by Connections Out, does any one domain reach the maximum connections specified for a listener? The default maximum number of connections for a listener is 600. The default maximum system-wide number of connections if 10,000 (set by the deliveryconfig command). You can examine the maximum number of connections for a listener using the command:

  listenerconfig -> edit -> listener_number -> limits

  Are the connections for a listener further limited by the destconfig command (either by system maximum or by Virtual Gateway addresses)? Use this command to examine the destconfig connection limits:

  destconfig -> list

• Use the hoststatus command.

  “Drill-down” using the hoststatus command on the top domains listed from the results listed by the tophosts command.

  Is the host available and accepting connections?

  Are there problems with one specific MX record mail server for the given host?

  The hoststatus command reports the last “5XX” status code and description returned by the host if there is a 5XX error (Permanent Negative Completion reply) for the specified host. If the last outgoing TLS connection to the host failed, the hoststatus command displays the reason why it failed.

• Configure and/or examine the domain debug, bounce, and text mail logs to check if the recipient host is available.

  Domain debug logs record the client and server communication during an SMTP conversation between the appliance and a specified recipient host. This log file type can be used to debug issues with specific recipient hosts.

  For more information, see Using Domain Debug Logs.

  Bounce logs record all information pertaining to each bounced recipient.

  For more information, see Using Bounce Logs.

  Text mail logs contain details of email receiving, email delivery and bounces. These logs are a useful source of information to understand delivery of specific messages and to analyze system performance.

  For more information, see Using Text Mail Logs.

• Use the telnet command to connect from the appliance to the problem domain:

mail3.example.com> telnet


Please select which interface you want to telnet from.

1. Auto

2. Management (192.168.42.42/24: mail3.example.com)

3. PrivateNet (192.168.1.1/24: mail3.example.com)

4. PublicNet (192.168.2.1/24: mail3.example.com)

[1]> 1

Enter the remote hostname or IP.

[]> problemdomain.net

Enter the remote port.


[25]> 25

• You can use the tlsverify command to establish an outbound TLS connection on demand and debug any TLS connection issues concerning a destination domain. To create the connection, specify the domain to verify against and the destination host. AsyncOS checks the TLS connection based on the Required (Verify) TLS setting.

mail3.example.com> tlsverify


Enter the TLS domain to verify against:

[]> example.com

Enter the destination host to connect to. Append the port (example.com:26) if you are

not connecting on port 25:

[example.com]> mxe.example.com:25

Connecting to 1.1.1.1 on port 25.

Connected to 1.1.1.1 from interface 10.10.10.10.

Checking TLS connection.

TLS connection established: protocol TLSv1, cipher RC4-SHA.

Verifying peer certificate.

Verifying certificate common name mxe.example.com.

TLS certificate match mxe.example.com

TLS certificate verified.

TLS connection to 1.1.1.1 succeeded.

TLS successfully connected to mxe.example.com.

TLS verification completed.

If you suspect that there are there are performance problems with the appliance , utilize the following strategies:

• Use the rate and hostrate commands to check the current system activity.

  The rate command returns real-time monitoring information about email operations. For more information, see Displaying Real-time Activity.

  The hostrate command returns real-time monitoring information for a specific host.

• Use the status command to cross-check the historical rates to check for degradation.

• Use the status detail command to check the RAM utilization.

  You can use the status detail command to quickly see the system’s RAM, CPU, and Disk I/O utilization.

  Note	RAM utilization should always be less than 45%. If RAM utilization exceeds 45%, then, the appliance will enter “resource conservation mode;” it initiates a “back-off” algorithm to prevent over-subscription of resources and sends out the following email alert:

  This system (hostname: hostname) has entered a 'resource conservation' mode in order to
  prevent the rapid depletion of critical system resources.
  
  
  RAM utilization for this system has exceeded the resource conservation threshold of 45%.
  The allowed injection rate for this system will be gradually decreased as RAM
  utilization approaches 60%.
  
  

  This situation occurs only with an aggressive injection with poor deliverability facilities. If you encounter RAM utilization exceeding 45%, check the number of messages in the queue and see if a particular domain is down or unavailable for delivery (via the hoststatus or hostrate commands). Also check the status of the system and ensure that delivery is not suspended. If after stopping the injection you continue to experience a high RAM utilization, contact Cisco Customer Support.

• Is the problem specific to one domain?

  Use the tophosts command to get immediate information about the email queue and determine if a particular recipient domain has delivery problems.

  Check the size of the queue. You can delete, bounce, suspend, or redirect messages in the email queue to manage its size, or to deal with recipients to a specific, problematic domain. For more information, see Managing the Email Queue. Use these commands:

  • deleterecipients
  • bouncerecipients
  • redirectrecipients
  • suspenddel / resumedel
  • suspendlistener / resumelistener

  Use the tophosts command to check the number of soft and hard bounces. Sort by “Soft Bounced Events” (option 4 ) or “Hard Bounced Recipients” (option 5 ). If the performance for a particular domain is problematic, use the commands above to manage the delivery to that domain.

See Overriding Internet Explorer Compatibility Mode.

• Troubleshooting Alerts That Miscellaneous Disk Usage is Approaching the Quota

Lights on the front and/or rear panels of your hardware appliance indicate health and status of your appliance . For descriptions of these indicators, see the hardware guides , such as the Cisco x90s Series Content Security Appliances Installation and Maintenance Guide, available from http://www.cisco.com/c/en/us/support/security/email-security-appliance/products-installation-guides-list.html.

Specifications for your appliance , such as temperature ranges, are also available in these documents.

• Technical Support for Virtual Appliances
• Opening or Updating a Support Case From the Appliance
• Enabling Remote Access for Cisco Technical Support Personnel
• Running a Packet Capture