Cisco DCNM OVA Installation Guide, Release 7.x

Automatic Failover

The application-level and virtual machine (VM)-level and switchover process is as follows.

• If any of the applications managed by the load-balancing software (DCNM/AMQP) goes down on OVA-A, the Active node that handles the client requests detects the failure and redirects subsequent requests to the Standby node (OVA-B). This process provides an application-level switchover.
• If the Active node (OVA-A) fails or is powered-off for some reason, the Standby node (OVA-B) detects the failure and enables the VIP address for Cisco DCNM/AMQP on OVA-B. It also sends a gratuitous ARP to the local switch to indicate the new MAC address that is associated with the IP address. For applications not using VIP, the DHCPD runninG on OVA-B detects the failure of DHCPD on OVA-A and activates itself; whereas LDAP running on OVA-B continues running as LDAP is deployed Active-Active. Consequently, a VM-level failover is accomplished for all four applications (DCNM/AMQP/DHCP/LDAP).

Manually Triggered Failovers

An application-level failover can also be triggered manually. For instance, you might want to run AMQP on OVA-B and the rest of the applications on OVA-A. In that case, you can log in to the SSH terminal of OVA-A and stop AMQP by using the appmgr stop amqp command.

This failover triggers the same process that is described in the “Automatic Failover” section; subsequent requests to the AMQP Virtual IP address are redirected to OVA-B

Deploying Cisco DCNM OVAs

You must deploy two standalone OVAs. When you deploy both OVAs, you must meet the following criteria:

• Both OVAs should have their management access (eth0) and enhanced fabric management (eth1) interfaces in the same subnet.
• Both OVAs should be deployed with the same administrative password. This process ensures that both OVAs are duplicates of each other.

After the OVA is powered up, verify that all the applications are up and running by using the appmgr status all command.

After all of the applications are up and running, stop the applications by using the appmgr stop all command.

Note When the OVA is started up for the first time, please wait for all the applications to run before you shut down any of the applications or power off the virtual appliance.

Note For instructions on deploying the Cisco DCNM OVA, see Chapter 2, “Installing Cisco DCNM OVA Management Software”.

Creating an NFS/SCP Repository

The DCNM HA cluster needs a server that has both NFS/SCP capabilities. This server is typically a Linux server.

Note The server has to be in the enhanced fabric management network because the switches will use this server to download images and configurations.

Make sure that the exported directory is writable from both peers. The procedure to export a directory /var/lib/sharedarchive on a CentOS server is listed in the following paragraph. The steps will vary based on your environment.

Note You might need root privileges to execute these commands. If you are a nonroot user, please use them with ‘sudo’.

[root@repository ~]# service nfs restart

The same folder /var/lib/sharedarchive can also be accessed through SCP with SCP credentials.

The /var/lib/sharedarchive * (rw,sync) command provides read-write permissions to all servers on /var/lib/sharedarchive. Refer to CentOS documentation for information on restricting write permissions to specific peers.

Availability of Virtual IP Addresses

Two free IPv4 addresses are needed to set up VIP addresses. The first IP address will be used in the management access network; it should be in the same subnet as the management access (eth0) interface of the OVAs. The second IP address should be in the same subnet as enhanced fabric management (eth1) interfaces (switch/POAP management network).

Installing an NTP Server

For most of the HA functionality to work, you must synchronize the time on both OVAs by using an NTP server. The installation would typically be in the management access network (eth0) interfaces.

Network Management

The data center network management function is provided by the Cisco Prime Data Center Network Manager (DCNM) server. Cisco DCNM provides the setup, visualization, management, and monitoring of the data center infrastructure. Cisco DCNM can be accessed from your browser at http://[host/ip].

Note For more information about Cisco DCNM, see http://cisco.com/go/dcnm

HA Implementation

Cisco DCNMs that run on both OVAs are configured in clustering and federated modes for HA.

• Cisco DCNM clustering is an HA mechanism for LAN devices. Internally it uses JBoss clustering. The first OVA that is HA-enabled becomes the master and takes care of all updates to the database.
• Cisco DCNM federation is the HA mechanism for SAN devices. Groups of SAN devices can be managed by each node in the DCNM federated setup. All the devices can be managed using a single client interface.

You can enable automatic failover in the Cisco DCNM UI by choosing: Admin > Federation . If you enable an automatic failover and the Cisco DCNM that is running on OVA-A fails, the automatic failover moves only the fabrics and shallow-discovered LANs that are managed by OVA-A to OVA-B automatically.

DCNM Virtual IP Usage

An OVA HA setup has two VIP addresses (one for each network) for the Cisco DCNM at the default HTTP port. These VIPs can be used for accessing the DCNM RESTful services on the OVA management network and the enhanced fabric management network. For example, external systems such as Cisco UCS Director can point to the VIP in the OVA management network and the request gets directed to the active Cisco DCNM. Similarly, the switches in an enhanced fabric management network access the VIP address on the enhanced fabric management network during the POAP process.

You can still directly connect to Cisco DCNM real IP addresses and use them as you would in a DCNM in a cluster/federated set up.

Note We recommend that you use a VIP addresses only for accessing DCNM RESTful API. To access the Cisco DCNM Web UI/DCNM SAN/LAN thick client, connect to the server’s real IP address.

Licenses

For Cisco DCNM, we recommend that you have licenses on the first instance and a spare matching license on the second instance.

Application Failovers

Enable an automatic failover option in the Cisco DCNM UI when an OVA HA pair is set up by choosing: Admin > Federation . This process ensures that if the DCNM that is running on OVA-A fails, all the fabrics and shallow-discovered LANs managed by DCNM-A are managed by DCNM-B automatically after a given time interval (usually about 5 minutes after the failure of DCNM on OVA-A).

The Cisco DCNM VIP address still resides on OVA-A. The Representational State Transfer Web Services (REST) calls initially hit the VIP addresses on OVA-A and get redirected to the Cisco DCNM that is running on OVA-B.

Application Failbacks

When the Cisco DCNM on OVA-A comes up, the VIP address automatically redirects the REST requests to DCNM-A.

Virtual-IP Failovers

The VIP address that is configured for Cisco DCNM REST API on OVA-A can fail due to two reasons:

• The load-balancing software running on OVA-A fails.
• OVA-A fails.

In both cases, the VIP address of Cisco DCNM automatically migrates to OVA-B. The only difference is which DCNM will be used after the failover.

– If a load-balancing software failure occurs, the VIP address on OVA-B directs the requests to DCNM-A.

– If an OVA-A failure occurs, the VIP address on OVA-B directs the requests to DCNM-B.

The automatic failover ensures that the ownership of all of the fabrics and shallow-discovered LANs managed by DCNM-A automatically change to DCNM-B.

Virtual-IP Failbacks

When OVA-A is brought up and Cisco DCNM is running, the VIP addresses keep running on the Standby node. The failback of Virtual IP addresses from OVA-B to OVA-A occurs only in the following sequence.

1. OVA-A comes up.

2. Cisco DCNM runs on OVA-A.

3. OVA-B goes down or the load-balancing software fails on OVA-B.

RabbitMQ

RabbitMQ is the message broker that provides the Advanced Messaging Queuing Protocol (AMQP).

Note For more information about RabbitMQ, go to http://www.rabbitmq.com/documentation.html

HA Implementation

Enabling the HA on the OVA creates a VIP address in the OVA management network. Orchestration systems such as vCloud Director, set their AMQP broker to the VIP address.

Enabling the HA on the OVA also configures the RabbitMQ broker that runs on each node to be a duplicate of the broker that is running on the other node. Both OVAs act as “disk nodes” of a RabbitMQ cluster, which means that all the persistent messages stored in durable queues are replicated. The RabbitMQ policy ensures that all the queues are automatically replicated to all the nodes.

Application Failovers

If RabbitMQ-A fails, the VIP address on OVA-A redirects the subsequent AMQP requests to RabbitMQ-B.

Application Failbacks

When RabbitMQ-A comes up, the VIP address automatically starts directing the AMQP requests to RabbitMQ-A.

Virtual-IP Failovers

The VIP address configured for the AMQP broker on OVA-A can fail due to two reasons:

• The load-balancing software running on OVA-A fails.
• OVA-A fails.

In both cases, the VIP address of the AMQP automatically migrates to OVA-B. The only difference is which AMQP broker will be used after the failover.

– In a load-balancing software failure, the VIP address on OVA-B directs the requests to RabbitMQ-A.

– In an OVA-A failure, the VIP address on OVA-B directs the requests to RabbitMQ-B.

“Virtual-IP” Failbacks

When OVA-A is brought up and AMQP-A is running, the VIP addresses keep running on the OVA-B (directing the requests to AMQP-A). The failback of the RabbitMQ VIP from OVA-B to OVA-A occurs only in the following sequence.

1. OVA-A comes up.

2. RabbitMQ runs on OVA-A.

3. OVA-B goes down or the load-balancing software fails on OVA-B.

OpenLightweight Directory Access Protocol

The OVA installs an LDAP server an asset database to the switches.

This section contains the following topics:

• “Using the OVA-Packaged (Local) LDAP Server” section
• “Using the Remote LDAP Server” section

Using the OVA-Packaged (Local) LDAP Server

LDAP HA is achieved through OpenLDAP mirror mode replication. Each LDAP server that is running on one OVA becomes a duplicate of the LDAP server that is running on the other OVA.

DCNM and LDAP Interaction

Both LDAP IP address show up in the Cisco DCNM Web UI ( Admin -> DFA Settings ) in the following order: LDAP-A, LDAP-B.

Cisco DCNM always attempts to write on LDAP-A as follows.

• If the write operation succeeds, the data gets replicated to LDAP-B.
• If the write operation fails, then Cisco DCNM writes to LDAP-B.

The data on LDAP-B eventually gets replicated to LDAP-A when it becomes available.

Switch and LDAP Interaction

When you configure the asset databases, every switch is configured with multiple LDAP servers, as shown in the following example.

The first active LDAP server that is configured in the switch becomes the Active LDAP server. The Active LDAP server is queried first for autoconfigurations.

For every read operation that the switch needs to perform, the Active LDAP server is contacted first, followed by the rest of the LDAP servers.

Use the show fabric database statistics command to find the Active LDAP server, which is marked by an asterisk (*) in the output.

In the previous example, during autoconfiguration, a leaf switch first queries 10.77.247.148, which is the active network database (indicated by “*n”). If that is not available, it automatically contacts the second LDAP server configured as an network database (10.77.247.147 in this example).

Using the Remote LDAP Server

This section describes the behavior when you use a remote LDAP server in an HA environment.

Cisco DCNM and LDAP Interaction

Cisco DCNM allows only two external LDAP servers that are assumed to be synchronized with each other.

Switch and LDAP interaction

The switch and LDAP interaction that use the remote LDAP server is the same interaction as when you are using the OVA-packaged LDAP. The Active LDAP server is contacted first; if it is not reachable, the switch then attempts to read from the next available LDAP server.

DCHP HA

DHCP on both OVAs listen on the interface of the enhanced fabric management network. The native Internet Systems Consortium (ISC) DHCPD failover mechanism is be used for HA. The lease information is automatically synchronized using native code.

DHCP POAP

The switches do a DHCP broadcast and get response from the Active DHCP server.

DHCP Autoconfiguration

When a tenant host or virtual machine (VM) comes up, it sends a broadcast that is relayed by the leaf node. In such a scenario, the VM profiles should be configured with both relay addresses of OVA-A and OVA-B.

Changing DHCP Scope Configurations

Scope changes through the Cisco DCNM UI ensure proper synchronization of scopes among the peers. We do not recommend that you do a manual configuration of the DHCP scope configuration file.

Note You must update the IP range for the default scope before creating the new scope, otherwise DHCP will be unable to star. See the “Starting DHCP in an HA Setup” section for information on updating the IP range for the DHCP scope through the Cisco DCNM UI.

Repositories

All repositories must be remote.

XMPP

Extensible Messaging and Presence Protocol (XMPP) HA is currently not available. The OVA HA configuration does not affect the XMPP servers that are running on either of the nodes in any way.

Configuring the Active Peer

Step 1 Log in to the SSH terminal of the OVA that you want to become the Active peer and enter the appmgr set ha active command.

Step 2 Make sure that each prerequisite is in place and press y ; if not all of the pre-requisites are in place, press n to exit.

A prompt for the root password appears.

Step 3 Enter the administrative password created during OVA installation.

You will now be prompted for the management access interface (eth0 IP address) of the Standby peer.

Step 4 Enter the management IP address of the peer DCNM.

The active OVA generates a pair of authentication keys and transfers it to the peer’s authorized keys.

a. Enter the root password of the Standby peer when prompted.

All of the other network information needed from the Standby peer is automatically picked up by the Active peer and displayed for confirmation.

b. Ensure that it is the correct peer and press y to continue.

Step 5 Enter the VIP addresses for both the management access (eth0) and enhanced fabric management networks (eth1).

Make sure that the VIP addresses are currently not used by any other interfaces in their respective networks.

Step 6 Enter the database URL to set the database. The script uses a JDBC thin driver, so you should enter the URL in the same format.

a. Enter the database password.

b. Enter the database password again for verification.

The script tries to do a sample query from the database to check the details entered. The Cisco DCNM schema and related data are loaded after you confirm that all the data are valid.

Step 7 Enter repository settings:

a. Enter an SCP/NFS repository IP address for the enhanced fabric management network.

b. Enter the IP/exported-directory location.

The script does a test mount and unmounts it shortly after. It is permanently mounted after user confirmation. Similar checks are done for SCP repository users.

c. You will have to enter the SCP password three times (twice for the script and the third time when the script does a test write on the repository).

d. Enter an NTP server IP address. This step is very important for all the applications that run on a cluster.

Step 8 A summary of the details entered will be displayed. If you want to reenter the details, press n.

Once the HA setup is complete, you can check the role of the ha as follows:

Configuring the Standby peer

Step 1 Log in to the SSH terminal of OVA-B and enter the appmgr setup ha standby command.

Step 2 Press y to continue.

The standby OVA generates a pair of authentication keys and transfers it to the peer’s authorized keys.

a. Enter the root password of the Active peer when prompted.

All the other network information entered during active the OVA setup is automatically picked up by the Standby peer and displayed for confirmation.

b. Carefully check if it is the correct peer and press y to continue.

Once confirmed, OVA-B is configured to be a Standby peer, and the following message is displayed.

Note For information about updating default POAP scopes and starting DHCP using HA, please see, Starting DHCP in an HA Setup

Step 3 Check the HA role of the node by entering the appmgr show ha-role command.

Starting Applications in the Active Peer

Step 1 Log in to the SSH terminal of the Active peer (OVA-A) and start all applications by entering the appmgr start all command.

Step 2 Wait for all the applications to start. Once all applications (except dhcpd) are up and running, go to the next procedure.

Note To start DHCP using HA, see the “Starting DHCP in an HA Setup” section.

Starting Applications in the Standby Peer

Step 1 Login to the SSH terminal of the Standby peer and start all applications using the appmgr start all command. Wait for all the applications to start.

Step 2 Once all applications (except dhcpd) are up/running, proceed to the next step.

Note For starting DHCP using HA, please see, Starting DHCP in an HA Setup

Starting DHCP in an HA Setup

In an HA setup, DHCPD will be initially down. In this procedure, you will update the IP range address for the POAP DHCP scope. Use the following procedure to bring up DHCP.

Note You must update the IP range for the default scope before creating the new scope, otherwise DHCP will be unable to start.

Step 1 Log in to Cisco DCNM web UI.

Step 2 On the menu bar, choose Config> POAP > DHCP Scope and enter the free IP range address for the default DHCP scope named enhanced_fabric_mgmt_scope.

Step 3 Click Apply .

DHCP is automatically started on both the OVAs.

Step 4 Verify all applications are running by opening an SSH terminal session and using the appmgr status all command.