Regenerate CUCM Certificate

Introduction

This document describes how to regenerate certificates used in Cisco Unified Communications Manager (CUCM) Release 8.x and later.

Background Information

The Identity Trust List (ITL) enabled per the Security by Default (SBD) feature and the Certificate Trust List (CTL) for Mixed-mode environments are also be covered in this document in order to avoid any undesired outages. For example, how to avoid phone registration issues or phones that do not accept configuration changes or firmware.

Caution: It is always recommended to complete certificate regeneration in a maintenance window.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• CallManager
• CAPF (Certificate Authority Proxy Function)
• IPsec
• Tomcat
• TVS (Trust Verification Service)
• ITLRecovery (only for CUCM 10.X and later)
• phone-vpn-trust
• phone-sast-trust
• phone-trust
• phone-ctl-trust
• LSCs (Locally Significant Certificates)
• MICs (Manufacturer Installed Certificates)

Components Used

The information in this document is based on these software versions:

• CUCM Release 9.1(2)SU2a,
• CUCM Release 8.x and later

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

When to Regenerate Certificates

Most of the certificates used in CUCM after a fresh installation are self-signed certificates issued, by default, for five years. Note that the five-year time range currently cannot be modified to be a shorter range of time on CUCM. However, a Certificate Authority (CA) can issue certificates for nearly any range of time.

The certificates in CUCM are classified in two roles:

• Service certificates: It is possible to regenerate them and are NOT labeled with the word -trust. Each node has its own service certificates, this means that each pub and sub have a CallManager, Tomcat, IPsec, TVS and CAPF certificate.

• Trust certificates: It is NOT possible to regenerate them and are labeled with the word -trust. These certificates can be copies of Service Certificates, certificates installed by default, or certificates from other servers.

There are also some trusted certificates (such as CAPF-trust and CallManager-trust) that are preloaded and have a longer validity period. For example, the Cisco Manufacturing CA certificate is provided on CUCM trust stores to specific features and does not expire until the year 2029.

Certificates must be regenerated before they expire. When the certificates are about to expire you receive warnings in RTMT (Syslog Viewer) and an email with the notification is sent if configured.

An example of a certificate expiration notification that details the CUCM01.der certificate expires on Mon May 19 14:46 on server CUCM02 on the trust store tomcat-trust is shown here:

At Fri Sep 05 02:00:56 CEST 2014 on node 192.168.1.2, the following
SyslogSeverityMatchFound events generated:

SeverityMatch : Critical

MatchedEvent : Sep 5 02:00:06 CUCM02 local7 2 : 864: CUCM02.localdomain:
Sep 05 2014 00:00:06.433 UTC : %UC_CERT-2-CertValidfor7days:
%[Message=Certificate expiration Notification. Certificate name:CUCM01.der
Unit:tomcat-trust Type:own-cert Expiration:Mon May 19 14:46:]
[AppID=Cisco Certificate Monitor][ClusterID=][NodeID=CUCM02]:
Alarm to indicate that Certificate has Expired or Expires in less than seven days

AppID : Cisco Syslog Agent

ClusterID :

NodeID : CUCM02

TimeStamp : Fri Sep 05 02:00:16 CEST 2014

Keep in mind that expired certificates can have an impact on your CUCM functionality, dependent upon the cluster's configuration. Considerations are discussed in the next sections.

Service Impact by the Certificate Store

It is critical for the good functionality of the system to have all certificates updated across the CUCM cluster. If your certificates are expired or invalid they can significantly affect the normal functioning of the system. A list of potential issues you can have when any of the specific certificates are invalid or expired is shown here. The difference in impact can depend upon your system setup.

CallManager.pem

• TFTP not trusted (phones do not accept signed configuration files and/or ITL files).
• Phone services can be affected.
• Secure Session Initiation Protocol (SIP) trunks or media resources (Conference bridges, Media Termination Point (MTP), Xcoders, and so on) does not register or work.
• The AXL request fails.

Tomcat.pem

• Phones are not able to access HTTPs services hosted on the CUCM node, such as Corporate Directory.
• CUCM's web GUI issues, such as unable to access service pages from other nodes in the cluster.
• Extension Mobility or Extension Mobility Cross Cluster issues.
• If UCCX (Unified Contact Center Express) is integrated, due to security change from CCX 12.5 it is required to have upload CUCM Tomcat certificate (self-signed) or the Tomcat root & intermediate certificate (for CA signed) in UCCX tomcat-trust store since it effect Finesse desktop logins

CAPF.pem

• Phones do not authenticate for Phone VPN, 802.1x, or Phone Proxy. 
• Cannot issue LSC certificates for the phones.
• Encrypted configuration files do not work.

IPSec.pem

• Disaster Recovery System (DRS)/Disaster Recovery Framework (DRF) can not function properly. 
• IPsec tunnels to Gateway (GW) to other CUCM clusters do not work.

Trust Verification Service (TVS)

The phone cannot authenticate HTTPS service. The phone cannot authenticate configuration files (this can affect nearly everything on CUCM).

phone-vpn-trust

The phone VPN does not work because the VPN's HTTPS URL cannot be authenticated.

Note: If this does not exist, do not worry. This is only for specific configurations.

phone-sast-trust

Previous CTL/eTokens are unable to update or modify CTL.

Note: If this does not exist, do not worry. This is only for specific configurations.

phone-trust and phone-ctl-trust

Visual Voicemail with Unity or Unity Connection does not work.

Note: If this does not exist do not worry. This is only for specific configurations.

,

LSCs and MICs

Phones do not register. The phone does not authenticate to Phone VPN, Phone Proxy, or 802.1x. 

Note: MICs are on most phone models by default. LSCs are signed by CAPF and last five years by default. Software clients such as CIPC (Cisco IP Communicator) and Jabber do not have a MIC installed.

Create a DRS Backup

It is recommended to create a DRS backup before you perform any major changes like this. The CUCM DRF backup file backs up all the certificates in the cluster. All DRS backup/restore procedures can be found in the Cisco Disaster Recovery System Administration Guide for Cisco Unified Communications Manager.

Caution: Keep in mind Cisco bug ID CSCtn50405, CUCM DRF Backup does not back up certificates.

Determine the Mixed-Mode 

In order to determine if you run a CTL/Secure/Mixed-Mode cluster, choose Cisco Unified CM Administration > System > Enterprise Parameters > Cluster Security Mode (0 == Non-Secure; 1 == Mixed Mode).

If the Cluster is in Mixed-Mode

If you run a CUCM cluster in Mixed-Mode, this means that the CTL file needs to be updated after all certificate changes. The procedure on how to do this is within Cisco's Security Guide Documentation. However, be sure that you have at least one eToken from the original initiation of the Mixed-Mode feature and the eToken password is known.

Note: An update of the CTL does not happen automatically (as it does in the case of the ITL file). It needs to be completed manually by the administrator with either the CTL Client or the CLI command.

In CUCM 10.X and later you can put the cluster into Mixed-Mode in two ways:

• CLI command - if this method is used then your CTL file is signed with the CallManager.pem certificate of the Publisher server.

  admin:show ctl
  The checksum value of the CTL file:
  0c05655de63fe2a042cf252d96c6d609(MD5)
  8c92d1a569f7263cf4485812366e66e3b503a2f5(SHA1)
  
  Length of CTL file: 4947
  The CTL File was last modified on Fri Mar 06 19:45:13 CET 2015
  
  [...]
  
          CTL Record #:1
                    ----
  BYTEPOS TAG             LENGTH  VALUE
  ------- ---             ------  -----
  1       RECORDLENGTH    2       1156
  2       DNSNAME         16      cucm-1051-a-pub
  3       SUBJECTNAME     62      CN=cucm-1051-a-pub;OU=TAC;O=Cisco;L=Krakow;
                                  ST=Malopolska;C=PL
  4       FUNCTION        2       System Administrator Security Token
  5       ISSUERNAME      62      CN=cucm-1051-a-pub;OU=TAC;O=Cisco;L=Krakow;
                                  ST=Malopolska;C=PL
  6       SERIALNUMBER    16      70:CA:F6:4E:09:07:51:B9:DF:22:F4:9F:75:4F:C5:BB
  7       PUBLICKEY       140
  8       SIGNATURE       128
  9       CERTIFICATE     694     E9 D4 33 64 5B C8 8C ED 51 4D 8F E5 EA 5B 6D 21
                                  A5 A3 8C 9C (SHA1 Hash HEX)
  10      IPADDRESS       4
  This etoken was used to sign the CTL file.

• CTL client - if this method is used, then your CTL file is signed with one of the hardware eTokens.

  admin:show ctl
  The checksum value of the CTL file:
  256a661f4630cd86ef460db5aad4e91c(MD5)
  3d56cc01476000686f007aac6c278ed9059fc124(SHA1)
  
  Length of CTL file: 5728
  The CTL File was last modified on Fri Mar 06 21:48:48 CET 2015
  
  [...]
          CTL Record #:5
                    ----
  BYTEPOS TAG             LENGTH  VALUE
  ------- ---             ------  -----
  1       RECORDLENGTH    2       1186
  2       DNSNAME         1
  3       SUBJECTNAME     56      cn="SAST-ADN008580ef ";ou=IPCBU;o="Cisco Systems
  4       FUNCTION        2       System Administrator Security Token
  5       ISSUERNAME      42      cn=Cisco Manufacturing CA;o=Cisco Systems
  6       SERIALNUMBER    10      83:E9:08:00:00:00:55:45:AF:31
  7       PUBLICKEY       140
  9       CERTIFICATE     902     85 CD 5D AD EA FC 34 B8 3E 2F F2 CB 9C 76 B0 93
                                  3E 8B 3A 4F (SHA1 Hash HEX)
  10      IPADDRESS       4
  This etoken was used to sign the CTL file.

Note: You can move between the method used with CUCM Mixed Mode with Tokenless CTL.

Dependent upon the method used to secure your cluster, an appropriate CTL update procedure needs to be used. Either rerun the CTL client or enter the utils ctl update CTLfile command from the CLI.

Verify Security by Default on the Cluster

Avoidance of ITL issues is important because it can cause many features to fail or the phone refuses to abide by any changes to configurations. ITL issues can be avoided in these two ways.

Utilize the Prepare Cluster for Rollback to pre 8.0 Feature

This feature blanks out the ITL entries in the ITL file, so the phones trust any TFTP server. Any HTTPS request from/to phones fails while this parameter is set to True. It is not recommended to have it enabled as it limits phone features like Extension Mobility, Corporate Directory, and so on. However, you are able to make and receive basic phone calls.

Note: This feature does not work for Mixed Mode clusters, as this parameter only clears ITL, not CTL entries.

Note: This feature only prevents, but does not fix ITL issues. If the issue is already in the phone, it does not remove the ITL and the ITL removal needs to be manual.

Note: A change to this parameter causes ALL PHONES TO RESET.

Once this feature is set, all TFTP servers need to be restarted (in order to supply the new ITL) and all phones need to be reset in order to force them to request the new blank ITL. Once the certificate changes are completed and all necessary services have been restarted, this feature can be set back to False, TFTP service restarted, and the phone reset (so the phone can obtain the valid ITL file). Then all the features continue to work as they did previously.

Regenerate Certificates in Specific Order

This procedure provides a TFTP server with a valid/updated ITL file from a trusted TFTP server that is available.

1. Stop TFTP service on the Primary TFTP server.
2. Make changes to the Primary TFTP server's certificates (as needed).
3. Reset the phones (in order to get a new ITL file from the Secondary TFTP server) - dependent upon which certificates are regenerated, this can happen automatically.
4. Once phones have returned, start the Primary TFTP server's TFTP service.
5. Make certificate changes on the Secondary TFTP server.
6. Reset the phones (in order to get a new ITL file from the Primary TFTP server).

Caution: Do NOT edit certificates on both TFTP servers at the same time. This gives the phones no TFTP server to trust and requires the local administrator to manually remove the ITL from all phones.

Regenerate One Type of Certificate at a Time

This is the most used procedure and the recommended one as it prevents phones to lose trust. The process is described in the

Certificate Regeneration Process For Cisco Unified Communications Manager (CUCM) Guide.

Remove and Regenerate Certificates in CUCM

Only service certificates (certificate stores that are not labeled with -trust) can be regenerated. Certificates in the trust stores (certificate stores that are labeled with -trust) need to be deleted, as they cannot be regenerated.

Caution: Be aware of Cisco bug ID CSCut58407 - Devices cannot restart when CAPF / CallManager / TVS-trust is removed.

After all certificate modifications, the respective service needs to be restarted to take on the change. This is covered in the After Regeneration/Removal of Certificates section.

  

Caution:  Be aware of Cisco bug ID CSCto86463 - Deleted certificates reappear, unable to remove certificates from CUCM. This is an issue where deleted certificates continue to reappear after removal. Follow the workaround in the defect.

Regenerate Certificates via the CLI

Caution: Regenerations of certificates triggers an automatic update of the ITL files within the cluster, which triggers a cluster-wide softphone reset to allow phones to trigger an update of their local ITL. This is focused on CAPF and CallManager certificate regenerations but can occur with other certificate stores within CUCM, such as Tomcat.

Regenerate CAPF: Upon regeneration, the CAPF certificate automatically uploads itself to CAPF-trust and CallManager-trust. Also, CAPF always has a unique Subject Name header, thus previously used CAPF certificates are retained and used for authentication.

set cert regen CAPF

Note: If a CAPF certificate expires, phones that use LSC are not able to register to CUCM because CUCM rejects their certificate. However, you can still generate a new LSC for the phone with the new CAPF certificate. When you reboot the phone, it downloads the configuration and then contacts CAPF in order to update LSC. After LSC is updated, the phone registers as it can. This works as long as a new CAPF certificate is in the ITL file and the phone downloaded and trusted the certificate that signed it (callmanager.pem).

Regenerate CallManager: Upon regeneration, the CallManager automatically uploads itself to CallManager-trust.

set cert regen CallManager

Regenerate IPsec: Upon regeneration, the IPsec certificate automatically uploads itself to ipsec-trust.

set cert regen ipsec

Regenerate Tomcat: Upon regeneration, the Tomcat certificate automatically uploads itself to tomcat-trust.

set cert regen tomcat

Regenerate TVS:

set cert regen TVS

What to Expect

When you regenerate certificates via the CLI, you are requested to verify this change. Enter yes and then choose Enter.

admin:set cert regen CAPF

WARNING: This operation will overwrite any CA signed certificate previously imported
 for CAPF

Proceed with regeneration (yes|no)? yes

Successfully Regenerated Certificate for CAPF.

You must restart services related to CAPF for the regenerated certificates to become active.

Remove Certificates via the CLI

Remove CAPF-trust Certificates

set cert delete CAPF <name of certificate>.pem

Remove CallManager-trust Certificates

set cert delete CallManager <name of certificate>.pem

Remove ipsec-trust Certificates

set cert delete ipsec <name of certificate>.pem

Remove Tomcat-trust Certificates

set cert delete tomcat <name of certificate>.pem

Remove TVS-trust Certificates

set cert delete TVS <name of certificate>.pem

Regenerate Certificates via the Web GUI

Regenerate CAPF:

Upon regeneration, the CAPF certificate automatically uploads itself to CAPF-trust and CallManager-trust. Also, the CAPF certificate always has a unique Subject Name header, thus previously used CAPF certificates are retained and used for authentication.

OS Admin > Security > Certificate Management > Find > Click CAPF certificate > Regenerate

Regenerate CallManager:

Upon regeneration, the CallManager certificate automatically uploads itself to CallManager-trust.

OS Admin > Security > Certificate Management > Find > Click CallManager certificate > Regenerate

Regenerate IPsec: 

Upon regeneration, the IPsec certificate automatically uploads itself to ipsec-trust.

OS Admin > Security > Certificate Management > Find > Click ipsec certificate > Regenerate

Regenerate Tomcat:

Upon regeneration, the Tomcat certificate automatically uploads itself to tomcat-trust.

OS Admin > Security > Certificate Management > Find > Click tomcat certificate > Regenerate

Regenerate TVS:

OS Admin > Security > Certificate Management > Find > Click TVS certificate > Regenerate

Remove Certificates via the Web GUI

OS Admin > Security > Certificate Management > Find > Click X certificate within the
 '-trust' store > Remove/Delete

After Regeneration/Removal of Certificates

After you remove or regenerate a certificate from a certificate store, the respective service needs to be restarted in order to take on the change.

Store	Service to Restart	How
Tomcat	Tomcat	CLI: utils service restart Cisco Tomcat Follow steps needed from the CCX environment if applicable https://www.cisco.com/c/en/us/support/docs/customer-collaboration/unified-contact-center-express/118855-configure-uccx-00.html#anc12 https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cust_contact/contact_center/crs/express_12_5/release/guide/uccx_b_uccx-solution-release-notes-125/uccx_b_uccx-solution-release-notes-125_chapter_01.html#reference_2D9122E01C43B6E0AA06AB2A3248B797
CallManager	CallManager; TFTP; CTIManager	Web Gui: Navigate to Cisco Unified Serviceability > Tools > Control Center - Feature Services > (Select Server). Under Cisco CallManager, click Restart. Web Gui: Navigate to Cisco Unified Serviceability > Tools > Control Center - Feature Services > (Select Server). Under Cisco Tftp, click Restart. Web Gui: Navigate to Cisco Unified Serviceability > Tools > Control Center - Feature Services > (Select Server). Under Cisco CTIManager, click Restart.
CAPF	CAPF (On Publisher only)	Web Gui: Navigate to Cisco Unified Serviceability > Tools > Control Center - Feature Services > (Select Server). Under Cisco Certificate Authority Proxy Function, click Restart.
TVS	Trust Verification Service (on the respective server)	Web Gui: Navigate to Cisco Unified Serviceability > Tools > Control Center - Network Services > (Select Server). Under Cisco Trust Verification Service, click Restart.
ipsec	Cisco DRF Local (on all nodes); Cisco DRF Primary (on Publisher)	CLI: utils service restart Cisco DRF Local CLI: utils service restart Cisco DRF Primary

How to Identify no Longer Used -trust Certificates

Before you delete expired certificates in the trust store, it is important to identify the ones that are used and the ones that are not. Keep in mind the next points to select the certificates that must be deleted:

• Most of the -trust certificates are copies of used Service certificates. It is recommended to first regenerate all the expired Service Certificates in all the nodes, and CUCM updates the -trust copy automatically.

• The tomcat-trust VeriSign_Class_3_Secure_Server_CA_-_G3 is no longer used. If the Smart Call Home feature is used, follow the next guide to upload the new certificate: https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/smart-call-home/215210-troubleshooting-certficate-exipry-alert.html

• The Manufacturing -trust certificates are pre-loaded to any CUCM during installation and those are used for CUCM to trust in any Cisco IP phone by default. It is not recommended to remove these certificates:

CAP-RTP-001

CAP-RTP-002

Cisco Root CA 2048

Cisco Root CA M2

ACT2_SUDI_CA

Cisco_Manufacturing_CA

Cisco_Manufacturing_CA_SHA2

• If the domain or hostname was changed, old certificates with an old domain or hostname are listed as "trust". If those hostnames and domains are no longer used, then those certificates are not used and can be deleted.

• If the Common Name of the certificate is from a different server (not CUCM cluster) verify the certificate from the other server is valid. As CUCM cannot regenerate the certificate, that must be done in the other server and then import the certificate as -trust to CUCM.

Install/Update LSC on Phone

If the CAPF certificate has been regenerated, then LSC certificates for all the phones in the cluster need to be updated with LSC signed by the new CAPF certificate.

1. Navigate to CUCM Serviceability > Service Activation. Activate the Cisco CTL Provider and Cisco Certificate Authority Proxy Function on the publisher server.
   
   
2. Under CUCM CCMAdmin, navigate to Device > Phone. Pick the IP Phone you want to provision an LSC on.
   
   
3. In the Device configuration page under Certificate Operation, navigate to Install / Upgrade > By Null String.
   
   
4. Save the phone configuration in CCMAdmin and choose Apply Config.

If the phone has trouble with the installation of the LSC, complete these actions on the phone:

When the phone resets, under the physical phone and navigate to Settings > (6) Security Configuration > (4) LSC > **# (this operation unlocks the GUI and allows us to continue to the next step) > Update (the update is not visible until you perform the previous step). Now, click Submit.

Do not assign any certificates to a phone unless it is a wireless phone (7921/25). Wireless phones use 3rd party Certificate Authorities (CA) in order to authenticate themselves.

Other Certificates' Renewal Process

Certificate Regeneration Process For Cisco Unified Communications Manager (CUCM): the guide describes the process to regenerate the certificates by type, this is the most used and the recommended process.

Certificate Regeneration Process for ITLRecovery on CUCM 12.x and later: the guide describes the process to regenerate the ITLRecovery certificate on a 12.x CUCM cluster. 

Regeneration of CUCM CA-Signed Certificates: the guide describes the process for CA-signed certificates in CUCM and the most common errors displayed when you upload a certificate.

Unified Communication Cluster Setup with CA-Signed Multi-Server Subject Alternate Name Configuration Example: the guide provides an example for Tomcat Multi-san certificate regeneration.

Regenerate Unified Communications Manager IM & Presence Service Self-Signed Certificates: the guide provides the regeneration process and services to restart for IM&P nodes.

UCCX Solution Certificate Management Guide: the guide provides the integration requirements for certificates in UCCX and the process to regenerate them.

Expressway C and E regeneration process is described in these videos:

Installing a Server Certificate to an Expressway

How to Configure Certificate Trust between Expressway-C and Expressway-E

Conclusion

Should you run into an issue or need assistance with this procedure, contact the Cisco Technical Assistance Center (TAC) for assistance. In this case, keep your DRF Backup available as it is used as a last resort in order to restore service if TAC is unable to do so through other methods.