Troubleshooting

The following sections describe the major areas in the Prime Provisioning installation in which troubleshooting might be necessary:

Unable to Find the Hostname

Symptom

Cannot find hostname.

Recommended Action


Step 1blank.gif If you cannot find the hostname, check the /etc/nsswitch.conf file to determine how the hostname is resolved.

Step 2blank.gif Check the /etc/resolv.conf file to determine whether you have a DNS Server IP Address.

Step 3blank.gif If you have a DNS Server IP Address, enter ping < IP Address > to check whether it is reachable.

Step 4blank.gif If the DNS Server is reachable, use nslookup < machine name > to check if it is resolving the name properly.

Step 5blank.gif If it is not working properly, you need a system administrator to fix the DNS entry.

Step 6blank.gif If you are not using DNS, be sure there is an entry for your machine in the hosts file in the /etc directory.


 

Moving a Repository or Renaming an Prime Provisioning Server

If you want to move an existing Repository to a new server with a new Prime Provisioning installation or rename an existing Prime Provisioning installation, your existing configuration must be updated. When renaming the Prime Provisioning installation, the local configuration file needs to be modified. When moving an existing Repository to a new server, the server from which you are moving the Repository and the server to which you are moving the Repository must both be at the same version and patch levels. Otherwise, your Repository needs to be upgraded, as explained in “Using the Repository Upgrade Tool” section. Both when moving an existing Repository and renaming an existing Prime Provisioning installation, the changes must be inserted into the Repository.

Use the following steps:


Step 1blank.gif Stop Prime Provisioning, using the following command:

./prime.sh stop

Step 2blank.gif Update the $PRIMEF_HOME/runtime.properties file with the new values.

Step 3blank.gif Execute the command:

$PRIMEF_HOME/thirdparty/ant/bin/ant -f $PRIMEF_HOME/install.xml

Step 4blank.gif Start the database, using the following command:

./prime.sh startdb

Step 5blank.gif Incorporate the changes into the Repository by initializing the database, using the following command:

./prime.sh initdb.sh


 

Memory Shortage on Large Networks

When running Discovery on a large network (250+ devices or 5000+ tunnels, for example) or an OutOfMemoryException is encountered, it is recommended that the memory setting be changed.

To do this, use the following steps:


Step 1blank.gif Choose Administration > Hosts.

Step 2blank.gif Select a host and click the Config button.

Step 3blank.gif Select watchdog > server > worker > java > flags.

Step 4blank.gif Change the first part of the property string, for example to -Xmx1024m instead of the default value -Xmx512m.

This increases the heap size of the Discovery task, which will clear up the OutOfMemoryException problem.

Step 5blank.gif Revert the watchdog.server.worker.java.flags property back to its original value to reduce the resource usage when no longer needed.


 

Cross-launch to Prime Provisioning Fails.

Symptom

Cross-launching from the Prime Central Suite to Prime Provisioning redirects you to the Prime Central login page instead of the Prime Provisioning login page.

An error message is displayed, for example:

httpd.0 log shows,java.security.cert.CertificateException: No name matching monza-cen2 found

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No name matching monza-cen2 found...

Cause

Mismatch between the value you provided as the Prime Central Hostname and the certificate generated on Prime Central.

Recommended Action

If you are using Mozilla Firefox:


Step 1blank.gif Click the icon that looks like a lock and is displayed on the bottom right corner of your browser.

A dialog box displays the certificate details.

Step 2blank.gif Click View Certificate.
A dialog box displays further details about the certificate.

Step 3blank.gif Note down the value specified in the Common Name (CN) filed.

Step 4blank.gif Using this value as the Prime Central hostname, follow the steps outlined in the procedure in “Integrating Prime Provisioning (Standalone) with Prime Carrier Management Suite” section to re-register Prime Provisioning with the Prime Central server.


 

If you are using IE:


Step 1blank.gif Go to View > Security Report.

Step 2blank.gif Click View Certificates.

Step 3blank.gif Note down the value specified in the Issued to field.

Step 4blank.gif Using this value as the Prime Central hostname, follow the steps outlined in the procedure in “Integrating Prime Provisioning (Standalone) with Prime Carrier Management Suite” section to re-register Prime Provisioning with the Prime Central server


 

Known Installation Issues

Known issues and solutions are as follows:

Symptom 1

Out of disk space.

Recommended Action

The error looks something like the following:

Prime Provisioning 6.1 will be installed in /var/PrimeProvisioning
>Copying files...
>Copying sybase...
>tar:./shared/jre_1.3.1_solaris_sun_sparc/lib/rt.jar: HELP - extract
>write error
>Error copying Sybase

If you see an error like this, it is likely due to the server running out of disk space.

To verify what space is available, run the command df –k < install directory >.

See “Installation Requirements,”for the disk space recommendations.

Symptom 2

The Installation utility GUI never displays.

Recommended Action

This problem should be accompanied with a Java stack dump.


Step 1blank.gif Run the following command to check for the $DISPLAY environment variable being set:

echo $DISPLAY.

If you use the secure shell (ssh), then this will be set up and managed for you.

If you manually change the $DISPLAY environment variable in an SSH environment, the easiest recovery method is to log off and reestablish the SSH connection.

Step 2blank.gif To set the DISPLAY environment variable, do the following:

a.blank.gif For the K or Bourne shell:

export DISPLAY=<machine name>:0.0
 

b.blank.gif For the C shell:

setenv DISPLAY= < machine name > :0.0


 

Symptom 3

Could not find temporary files.

Recommended Actions

If you receive an error that says the temporary file could not be created or found, it usually means the location used to write the temporary file is write-protected or out of disk space.

The two places that Prime Provisioning uses for temporary files are /tmp and /var/tmp.

  • Make sure both locations have write permission by doing a long list on the directories (ls -la). The directory should have wide open permissions: drwxrwxrwx.
  • There is another temporary file problem that can arise, especially in cases where there have been previous aborted installation attempts—existing temp files might be left by previous installations. If this is the case, it is best to clean out all the files in the temp directories after aborted installation attempts.

Symptom 4

Running./install.sh fails.

Recommended Action

Running./install.sh can fail due to the following reasons:

1.blank.gif You are not root.

Although it is possible to install as non-root if you have appropriate permissions in the target directory, this will still have problems since only root can write to /etc/init.d where the startup scripts reside. Therefore, it is easier to install as root.

2.blank.gif You do not have enough disk space in the target directory. To find out the available disk space, issue the following command:

df –k <target directory>

3.blank.gif You do not have enough disk space in the /tmp directory. Issue the command df –k /tmp to determine the available disk space for /tmp.

4.blank.gif You do not have enough disk space in the /var/tmp directory. Issue the command df –k /var/tmp to determine the available disk space for /var/tmp.

5.blank.gif The PATH and LD_LIBRARY_PATH environment variables are incorrect.

Make sure your PATH and LD_LIBRARY_PATH environment variables are correct.

Example:

PATH=/usr/bin:/usr/local/bin
LD_LIBRARY_PATH=/usr/lib:/usr/local/lib
export PATH LD_LIBRARY_PATH
 

a.blank.gif Alternatively, start a clean root shell with this command:

env – ksh

b.blank.gif Then issue a command like the following:

./install.sh /opt/PrimeProvisioning iscadm

Symptom 5

Prime Provisioning does not start on reboot.

Recommended Action

Perform the following steps:


Step 1blank.gif Install Prime Provisioning as the root user.

If you install as root, init.d has a script to start the Watchdog.

If you do not install as root, you do not get the startup on reboot feature.

Step 2blank.gif To become root, enter the following command:

su root

Step 3blank.gif Get the prime.tmpl file from the installation media.

Step 4blank.gif Edit the following fields in prime.tmpl :

OWNER=_owner - replace _owner with the username whom owns prime

PRIMEF_HOME=_vpnsc_home - replace _vpnsc_home with the prime directory

Step 5blank.gif Rename prime.tmpl as prime and then enter the following commands:

mv prime /etc/init.d

chmod 744 /etc/init.d/prime

 

Step 6blank.gif Create the following symbolic links to prime :

a.blank.gif cd /etc/rc1.d

ln -s /etc/init.d/prime K98ISC

b.blank.gif cd to /etc/rc2.d

ln -s /etc/init.d/prime K98ISC

c.blank.gif cd to /etc/rc3.d

ln -s /etc/init.d/prime S99ISC


 

Symptom 6

note.gif

Noteblank.gif Cisco Networking Services (CNS) has been removed as of Prime Provisioning 6.7.


Unable to create or delete IOS devices in the Cisco CNS IE2100 appliance repository when using Cisco CNS Configuration Engine 1.4 software with Prime Provisioning.

Recommended Action

Log into the Cisco CNS IE2100 appliance as root and modify the web.xml file located at /opt/CSCOcnsie/WEB-INF as follows.


Step 1blank.gif Locate the following entry:

<servlet>

<servlet-name> ServletLoadComplete </servlet-name>

<servlet-class> com.cisco.cns.cfgsrv.ServletLoadComplete </servlet-class>

<load-on-startup> 105 </load-on-startup>

</servlet>

Step 2blank.gif Immediately after the entry found in Step 1, insert the following lines:

<servlet>

<servlet-name> ImportDevice </servlet-name>

<servlet-class> com.cisco.cns.cfgsrv.ImportDevice </servlet-class>

<load-on-startup> 100 </load-on-startup>

</servlet>

<servlet>

<servlet-name> ImportTemplate </servlet-name>

<servlet-class> com.cisco.cns.cfgsrv.ImportTemplate </servlet-class>

<load-on-startup> 100 </load-on-startup>

</servlet>

<servlet>
<servlet-name> RemoveDevice </servlet-name>
<servlet-class> com.cisco.cns.cfgsrv.RemoveDevice </servlet-class>
<load-on-startup> 100 </load-on-startup>
</servlet>

<servlet>
<servlet-name> RemoveTemplate </servlet-name>
<servlet-class> com.cisco.cns.cfgsrv.RemoveTemplate </servlet-class>
<load-on-startup> 100 </load-on-startup>
</servlet>

Step 3blank.gif Locate the following entry:

<servlet-mapping>

<servlet-name> ServletLoadComplete </servlet-name>

<url-pattern> /ServletLoadComplete </url-pattern>

</servlet-mapping>

Step 4blank.gif Immediately after the entry found in Step 3, insert the following lines:

<servlet-mapping>

<servlet-name> ImportDevice </servlet-name>

<url-pattern> /ImportDevice </url-pattern>

</servlet-mapping>

<servlet-mapping>

<servlet-name> ImportTemplate </servlet-name>

<url-pattern> /ImportTemplate </url-pattern>

</servlet-mapping>

<servlet-mapping>
<servlet-name> RemoveDevice </servlet-name>
<url-pattern> /RemoveDevice </url-pattern>
</servlet-mapping>

<servlet-mapping>
<servlet-name> RemoveTemplate </servlet-name>
<url-pattern> /RemoveTemplate </url-pattern>
</servlet-mapping>

Step 5blank.gif Reboot the Cisco CNS IE2100 appliance.


 

Symptom 7

Not able to connect to the database.

Recommended Action

Perform the following steps:


Step 1blank.gif Check that the following values are substituted correctly in the installation window:

    • Oracle database server name
  • Oracle port number
  • SID

Step 2blank.gif If everything is correct, check that the server is reachable by entering:

ping < Oracle database server name >

Step 3blank.gif Issue the following to determine whether the database is running:

netstat -an | grep < oracle port number >

If no responses are found, your database is not running and you must restart, as explained in detail in the section, “Launching Oracle and Opening Your Database”.


 

Symptom 8

Unable to access Prime Provisioning with your web browser.

Recommended Action

Check the server status with the command ./prime.sh status.

If any server state is other than started, attempt to restart by entering the command,
./prime.sh wdclient restart < server name >. If this command does not succeed, enter the commands
./prime.sh stop and then ./prime.sh startwd.

note.gif

Noteblank.gif The most common server not to start is the httpd server.


Symptom 9

The web browser does not display certain GUI elements such as the main bar and charts in Prime Provisioning GUI.

Recommended Action

Install Adobe Flash player (version 10.3.183.7) and its plug-in to support the web browser and to enable the main bar and charts in the Prime Provisioning GUI.

Symptom 10

Installation fails due to ANT related errors, such as:

Exception - BUILD FAILED

/opt/Prime Provisioning -6.8/install.xml:57: The signjar type doesn't support the "destdir" attribute

followed by

Error running: /opt/Prime Provisioning -6.8/thirdparty/ant/bin/ant -f /opt/Prime Provisioning -6.8/install.xml jarsigner

Exiting installation.

Cause

Prime Provisioning is unable to use the right version of ANT.

Recommended Action

Update the silent_install.sh file available in the untarred installation folder as follows:

  • Locate the command
    ANT_HOME=${vpnsc_home}/thirdparty/ant
  • Include the following text in the next line
    export ANT_HOME
  • Re-intall Prime Provisioning.

Symptom 11

Prime Provisioning cannot unzip the upgrade tool during installation.

Error: Cannot create upgradeTool

Recommended Action

  • For a new installation ( for new installations, the symptoms of this issue are minor and can be ignored):

blank.gif a) Copy the installation files from cd /cdrom/cdrom0 to a location to which you have Write permissions.

blank.gif b) Initiate the installation procedure:
/install.sh

  • New installation using an existing DB, and the opting to upgrade the DB manually:

blank.gif a) Copy the installation files from the cd /cdrom/cdrom0 to a location to which you have Write permissions.

blank.gif b) Go to the location where you have copied the installation files:
cd <Hand-off location>

blank.gif c) Unzip the isc-upgrade.zip file:
unzip isc-upgrade.zip

blank.gif d) Initiate the installation procedure:
/install.sh

  • Upgrade installation and choosing to perform an ‘upgrade in place’:

blank.gif a) Copy the installation files from the cd /cdrom/cdrom0 to a location to which you have Write permissions.

blank.gif b) Go to the location where you have copied the installation files:
cd <Hand-off location>

blank.gif c) Unzip the isc-upgrade.zip file:
unzip isc-upgrade.zip

blank.gif d) Initiate the installation procedure:
/install.sh

Warning: Unresponsive Script

Warning message: “Warning: Unresponsive script. A script on this page may be busy, or may have stopped....”

Cause: Some operations run longer than the amount of time predefined by the browser. Examples of tasks during which this error message occurs are:

  • editing a customer device with many interfaces,
  • editing user details when there are many users.

Recommended Action

Increase the browser timeout value.

Daylight Saving Time

If Daylight Saving Time (DST) is not working correctly, perform the following steps:


Step 1blank.gif Go to the following URL to determine which patch is needed for your time zone:

http://www.oracle.com/technetwork/java/javase/tzdata-versions-138805.html

Step 2blank.gif To download the Java Runtime Environment (JRE) patch, go to:

http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html#timezone

Step 3blank.gif Go to Prime Provisioning home directory:
cd $PRIMEF_HOME

Step 4blank.gif Enter: ./prime.sh stop

Step 5blank.gif Follow this link to install the missing DST patch that you downloaded from Step 2:

http://www.oracle.com/technetwork/java/javase/tzupdater-readme-136440.html


 

Error - DBSPAWN ERROR: -84

The error: DBSPAWN ERROR: -84 is normally seen when the existing log files are not removed before loading a new repository.db file. The repository.log and sla.log files in the Repository/ directory must be deleted before initiating the ./prime.sh startdb command.

Error - No VPNSC Host Entry in the Database, When Starting Prime Provisioning

To correct the error: No VPNSC Host Entry in the Database, run./prime.sh initdb.sh in the following order:


Step 1blank.gif ./prime.sh stop
Ensure that no other Prime Provisioning processes are running. To do this, you can enter: ps -ef | grep prime

Step 2blank.gif ./prime.sh startdb

Step 3blank.gif ./prime.sh initdb.sh
This step adds the host entry into the repository.

Step 4blank.gif ./prime.sh startwd


 

Error - Could Not Connect to the Name Server, When Starting Prime Provisioning

The error: com.cisco.vpnsc.watchdog.WDRuntimeException: WD_108 :: Could not connect to the name server is normally seen when the domain name cannot be extracted from resolv.conf. The result is that the nameserver does not start, because it fools the system into thinking it is not a Master server.

To correct this error, you must have root privileges. As root, add the correct domain statement to the /etc/resolv.conf file for your server (not $PRIMEF_HOME/etc); for example, domain cisco.com.

Error - This Is Not a Database Server

The following error could occur after you install Prime Provisioning:
<server name> ./prime.sh startdb Master database server is: This is not a database server. There is no need to start the database
Adding this host to the database... com.cisco.cns.security.common.CannotConnectException: Cannot connect to the data store:Cannot connect to the data store. No valid connection to server type: com.cisco.cns.security.dataaccess
at com.cisco.cns.security.dataaccess.ConnectionPool.acquire (ConnectionPool.java:240)

Specifically, this could occur after issuing the command: ./install.bin <directory_where_PrimeProvisioning_is_to_be_installed> iscadm.

The error could be Domain Naming System (DNS) related. In the install.cfg file, <server name> .cisco.com needs to be changed to <server name> only. Then run applycfg.sh followed by ./prime.sh initdb.sh and ./prime.sh startwd.

Error - Cannot Connect to the Data Store

The primary reason for the error: Cannot Connect to the Data Store is DNS related. As root, make sure /etc/resolv.conf (not the $PRIMEF_HOME/etc directory) is correct for your server.

If you need more information, set the Security Policy Engine (SPE) logging to DEBUG and attempt to execute ./prime.sh initdb.sh. This provides more details. If an unknown host exception is created, double check the /etc/hosts file and the /etc/nsswitch.conf file. This controls the flow and sequence of the hostname lookup.

If DNS is not enabled or working, add the IP address to the following files: cns, vpnsc, and HA properties files, to use IP addresses instead of hostnames.

The cns properties files is located at $PRIMEF_HOME/etc/spe/cns.properties.

The vpnsc properties file is located at $PRIMEF_HOME/etc/vpnsc.properties.

The HA properties file is located at $PRIMEF_HOME/etc/HA.properties.

Graphical Installer Mode Not Supported

The following error occurs when you try to install Prime Provisioning in RHEL 6.4 through VNC or Putty:

Graphical installers are not supported by the VM.