The cnSGW-C supports High Availability (HA) of UDP proxy. The HA model of UDP proxy is based on the keepalived virtual IP concepts.

For more information on UDP proxy redundancy, see the High Availability for the UDP Proxy section in the Pods and Services Reference chapter.

This section describes the deployment of cnSGW-C pods and its microservices.

The following figure shows the deployment model of six VMs in cnSGW-C.

In this model, the pods are deployed on VM pairs. Two replicas are available for each protocol pod (for example, rest-ep, protocol-ep, and gtp-ep). One instance is deployed on each protocol VM.

Similarly, service pods and session pods are distributed equally on both the service and session VMs. Such a distribution is controlled by labelling the VMs as well as implementing the K8 affinity and anti-affinity rules during pod scheduling.

This model ensures that, during VM reboot scenarios, at least 50% of the replicas of each pod type are available to handle user signaling.

Graceful pod restart allows pod to complete ongoing processing within 30 seconds. Abrupt pod restart will affect ongoing transactions without impact to PDU sessions.

The following is an example of VM labelling and replica configuration.

k8 label protocol-layer key smi.cisco.com/node-type value smf-proto
exit
k8 label service-layer key vm-type value smf-svc
exit
k8 label cdl-layer key smi.cisco.com/node-type value smf-cdl
exit
k8 label oam-layer key smi.cisco.com/node-type value oam
exit

endpoint pfcp
 replicas 1
 nodes    2
exit
endpoint service
 replicas 1
 nodes    2
exit
endpoint protocol
 replicas 1
 nodes    2
 vip-ip 209.165.201.28
exit
endpoint sbi
 replicas 1
 nodes    2

To verify the configuration, use the following show command:

show running-config instance instance-id instance_id endpoint 

The following is an example output of this show command.

show running-config instance instance-id 1 endpoint 
instance instance-id 1
 endpoint nodemgr
  replicas 1
  nodes    2
 exit
 endpoint gtp
  replicas 1
  vip-ip 209.165.201.29
 exit
 endpoint pfcp
  replicas                2
  enable-cpu-optimization true
  interface n4
   heartbeat
    interval               0
    retransmission-timeout 3
    max-retransmissions    5
   exit
  exit
 exit
 endpoint service
  replicas 2
 exit
 endpoint protocol
  replicas 1
  vip-ip 209.165.201.29
  end 

This command output displays the configurations related to multiple endpoints, such as endpoint names, pod replicas, and nodes.

In active-active mode,

• The inter-rack deployment is transparent to the adjacent NFs.

• The inter-rack deployment contains two instances of the CCG function, each instance manifest itself with a set of interface IPs.

• Each instance support sets of sessions and continue to use the same IP for session consistency.

• At a specific time period, one CCG instance can be primary only on one rack and standby on the other rack.

• The set of interface IPs that are associated with the CCG instance, dynamically route to the primary rack of the instance.

cnSGW-C supports primary/standby redundancy in which data is replicated from the primary to standby instance. The primary instance provides services in normal operation. If the primary instance fails, the standby instance becomes the primary and takes over the operation. To achieve inter-rack redundancy, two primary/standby pairs can be set up where each rack is actively processing traffic and standby is acting as backup for the remote rack.

In an Active-Active inter-rack redundancy deployment, consider there are two racks: Rack-1 and Rack-2 located in the same data center. All the NFs are trying to reach instance-1 and instance-2.

For NFs, both the instances are active. But in real, instance-1 and instance-2 are divided across racks.

Rack-1 has instance-1 and instance-2. In a pre-trigger scenario, instance-1 is local and acts as Primary and instance-2 is in Standby mode.

Rack-2 also has instance-1 and instance-2. In a pre-trigger scenario, instance-2 is local and acts as Primary and instance-1 is in Standby mode.

In case, if Rack-1 goes down, the traffic moves to Rack-2. On Rack-2 both the instances, instance-1 and instance-2 acts as Primary.

This configuration is needed to provide a inter-rack redundancy configuration for multiple rack. With instance ID, endpoint configurations should be configured for each rack.

Sample Configuration 1

The following is a sample configuration for endpoint VIP configuration under one instance:

config 
  instance instance-id gr_instanceId 
    endpoint endpoint_name 
        vip-ip vip_ip_address 
  exit 
exit 

Example:

config
instance instance-id 1
 endpoint sbi
  vip-ip 209.165.201.21
 exit
exit

Sample Configuration 2

The following is a sample configuration to provide information on system-id, cluster-id and slice-name under an instance:

config 
  instances instance instance_id 
   system-id system_id 
   cluster-id cluster_id 
   slice-name cdl_slice_name 
  exit 
exit 

Example:

config
 instances instance 1
  system-id sgw
  cluster-id sgw
  slice-name 1
 exit
exit

Note	It is recommended to have the same values for system-id, cluster-id in the instance, and app-name, cluster-name in deployment.

Only two instances can be configured on each local and remote rack, and corresponding endpoints can be instantiated.

A local instance-id is the identity of the local rack irrespective of if the rack is redundant or not.

Local Instance ID Configuration

The local instance is configured using the local-instance command.

local-instance instance 1

Endpoint configuration must be under instance specified by each unique instance ID.

Endpoint Configuration Example

Following are a few configuration examples.

Note	In the following example, instance-id "1" is a local instance-id, and endpoints configured under it belong to the local rack. Optionally, remote rack instance-id "2" can be configured for endpoints belonging to the inter-rack.

instance instance-id 1
 endpoint li
  replicas 1
  nodes    2
  vip-ip 209.165.201.6
  vip-ip 209.165.201.13
 exit
 endpoint gtp
  replicas 1
  nodes    2
  retransmission timeout 5 max-retry 4
  vip-ip 209.165.201.6
  vip-ip 209.165.201.4
  interface s5
   echo interval          60
   echo retransmission-timeout 5
   echo max-retransmissions 4
  exit
  interface s2b
   echo interval 60
   echo retransmission-timeout 5
   echo max-retransmissions 4
  exit
 exit
exit
instance instance-id 2
 endpoint li
  replicas 1
  nodes    2
  vip-ip 209.165.201.6
  vip-ip 209.165.201.13
 exit
exit
endpoint gtp
  replicas 1
  nodes    2
  retransmission timeout 5 max-retry 4
  vip-ip 209.165.201.6
  vip-ip 209.165.201.5
  interface s5
   echo interval 60
   echo retransmission-timeout 5
   echo max-retransmissions 4
  exit
  interface s2b
   echo interval 60
   echo retransmission-timeout 5
   echo max-retransmissions 4
  exit
 exit
exit

Add instance for PGW FQDN corresponding to local and remote instances.

Example

Following is a configuration example.

Note	In the following example, instance-id "1" is a local instance-id, and the cnSGW-C profile configured under it belongs to the local rack. Optionally, remote rack instance-id "2" can be configured for FQDN belonging to the inter-rack.

profile sgw sgw1
locality LOC1
instances 1 fqdn cisco.com.apn.epc.mnc456.mcc123
instances 2 fqdn cisco.com.apn.epc.mnc567.mcc123

Endpoint configuration is required only for cnSGW-C.

Example

The following is a configuration example.

Note	In the following example, instance-id "1" is a local instance-id, and endpoints configured under it belong to the local site. Optionally, remote site instance-id "2" can be configured for endpoints belonging to the geo-site.

instance instance-id 1
 endpoint nodemgr
  replicas 1
  nodes    1
 exit
 endpoint gtp
  replicas 1
  vip-ip 209.165.201.10
  interface s5e
   vip-ip 209.165.201.29
  exit
  interface s11
   vip-ip 209.165.201.29
  exit
 exit
 endpoint pfcp
  replicas 1
  interface sxa
   heartbeat
    interval               0
    retransmission-timeout 5
    max-retransmissions    3
   exit
  exit
exit
 endpoint service
  replicas 1
 exit
 endpoint protocol
  replicas 1
  vip-ip 209.165.201.29
  interface sxa
   vip-ip 209.165.201.29
  exit
 exit
 endpoint sgw-service
  replicas 1
 exit
exit
instance instance-id 2
 endpoint nodemgr
  replicas 1
  nodes    1
 exit
 endpoint gtp
  replicas 1
  vip-ip 209.165.202.150
  interface s5e
   vip-ip 209.165.201.27
  exit
  interface s11
   vip-ip 209.165.201.27
  exit
 exit
 endpoint pfcp
  replicas 1
  interface sxa
   heartbeat
    interval               0
    retransmission-timeout 5
    max-retransmissions    3
   exit
  exit
exit
 endpoint service
  replicas 1
 exit
 endpoint protocol
  replicas 1
  vip-ip 209.165.201.27
  interface sxa
   vip-ip 209.165.201.27
  exit
 exit
 endpoint sgw-service
  replicas 1
 exit
exit

This section describes how to configure the dynamic routing using BGP.

Configuring AS and BGP Router IP Address

To configure the AS and IP address for the BGP router, use the following commands:

config 
   router bgp local_as_number 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

  In a inter-rack redundancy deployment, you need to configure two Autonomous Systems (AS).

  • One AS for leaf and spine.

  • Second AS for both racks: Rack-1 and Rack-2.

Configuring BGP Service Listening IP Address

To configure the BGP service listening IP address, use the following commands:

config 
   router bgp local_as_number 
      interface interface_name 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• interface interface_name —Specify the name of the interface.

Configuring BGP Neighbors

To configure the BGP neighbors, use the following commands:

config 
   router bgp local_as_number 
      interface interface_name 
      neighbor neighbor_ip_address remote-as as_number 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• interface interface_name —Specify the name of the interface.

• neighbor neighbor_ip_address —Specify the IP address of the neighbor BGP router.

• remote-as as_number —Specify the identification number for the AS.

Configuring Bonding Interface

To configure the bonding interface related to the interfaces, use the following commands:

config 
   router bgp local_as_number 
      interface interface_name 
      bondingInterface interface_name 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• interface interface_name —Specify the name of the interface.

• bondingInterface interface_name —Specify the related bonding interface for an interface. If the bonding interface is active, then the BGP gives a higher preference to the interface-service by providing a lower MED value.

Configuring Learn Default Route

If the user configures specific routes on their system and they need to support all routes, then they must set the learnDefaultRoute as true.

Note	This configuration is optional.

To configure the Learn Default Route, use the following commands:

config 
   router bgp local_as_number 
      learnDefaultRoute true/false 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• learnDefaultRoute true/false —Specify the option to enable or disable the learnDefaultRoute parameter. When set to true, BGP learns default route and adds it in the kernel space. By default, it is false.

Configuring BGP Port

To configure the Port number for a BGP service, use the following commands:

config 
   router bgp local_as_number 
      loopbackPort port_number 
   exit 
exit 

NOTES:

• router bgp local_as_number —Specify the identification number for the AS for the BGP router.

• loopbackPort port_number —Specify the port number for the BGP service. The default value is 179.

Policy Addition

The BGP speaker pods learns many route information from its neighbors. However, only a few of them are used for supporting the outgoing traffic. This is required for egress traffic handling only, when cnSGW-C is sending information outside to AMF/PCF. Routes are filtered by configuring import policies on the BGP speakers and is used to send learned routes to the protocol pods.

A sample CLI code for policy addition and the corresponding descriptions for the parameters are shown below.

$bgp policy <policy_Name> ip-prefix 209.165.200.225 subnet 16 masklength-range 21..24 as-path-set “^65100”

This configuration controls the number of BGP speaker pods in deployment. BGP speaker advertises service IP information for incoming traffic from both the racks.

Note	Use non-bonded interface in BGP speaker pods for BGP peering. BGP peering per Proto node is supported with only two BGP routers/leafs. Considering two Proto nodes, there can be maximum of four BGP neighborships.

instance instance-id  instance_id endpoint bgpspeaker interface { bgp | bfd } internal base-port start base_port_number 

config 
instance instance-id instance_id 
 endpoint bgpspeaker 
   replicas replica_id 
   nodes node_id 
   interface bgp 
     internal base-port start base_port_number  
   exit 
   interface bfd 
     internal base-port start base_port_number 
  exit 
exit 

NOTES:

• instance instance-id instance_id —Specify the GR instance ID.

• base_port_number —Specify the port range only if logical NF is configured. This range depends on your deployment.

Example

The following is a configuration example:

instance instance-id 1
endpoint bgpspeaker
   replicas 1
   nodes    2
   interface bgp
     internal base-port start {24000}
   exit
   interface bfd
     internal base-port start {25000}
 exit

The following section provides IPAM configuraton examples.

cnSGW-C-1 Example

The following is a configuration example for cnSGW-C-1:

ipam
 instance 1
  address-pool pool-1
  vrf-name ISP
   tags
   dnn dnn-1
  exit
  ipv4
    address-range 209.165.201.1 209.165.201.31
 exit
instance 2
 address-pool pool-2
  vrf-name ISP
  tags
   dnn dnn-2
  exit
  ipv4
    address-range 209.165.202.129 209.165.202.159
 exit
exit

cnSGW-C-2 Example

The following is a configuration example for cnSGW-C-2:

ipam
 instance 1
 address-pool pool-1
  vrf-name ISP
  tags
   dnn dnn-1
  exit
  ipv4
    address-range 209.165.201.1 209.165.201.31
 exit
 instance 2
 address-pool pool-2
  vrf-name ISP
   tags
   dnn dnn-2
  exit
  ipv4
    address-range 209.165.202.129 209.165.202.159
 exit
exit

Endpoints must be configured under an instance. Two Geo-Redundancy pods are needed on each rack. You should also configure VIP for internal and external Geo interface for ETCD/CachePod replication.

instance instance-id  instance_id endpoint geo interface { geo-internal | geo-external } vip-ip { vip_ip_address } vip-port { vip_port_number }
 

config 
instance instance-id instance_id 
 endpoint geo 
  replicas replica_id 
  nodes node_id 
  internal base-port start base_port_number 
  interface geo-internal 
   vip-ip vip_ip_address vip-port vip_port_number 
  exit 
  interface geo-external 
   vip-ip vip_ip_address vip-port vip_port_number 
  exit 
exit 
exit 

NOTES:

• instance instance-id instance_id —Specify GR instance ID. One instance ID for local rack and other for another rack.

• vip-ip vip_ip_address —Specify VIP IP address for Internal/External Geo interface.

• vip-port vip_port_number —Specify VIP port number.

• internal base-port start base_port_number —Specify port range only if logical NF is configured.

Example

The following is a configuration example:

instance instance-id 1
endpoint geo
  replicas 1
  nodes 2
  internal base-port start 25000
  interface geo-internal
   vip-ip 209.165.201.8 vip-port 7001
  exit
  interface geo-external
   vip-ip 209.165.201.8 vip-port 7002
  exit
exit

To configure pod monitoring and failover thresholds in the inter-rack setup, use the following sample configuration. The geo pod monitors the configured pod name.

config 
 geomonitor 
  podmonitor pods pod_name 
   retryCount value 
   retryInterval interval_value 
   retryFailOverInterval failover_interval 
   failedReplicaPercent percent_value 
  exit 
 exit 

NOTES:

• pods pod_name —Specify the name of the pod to be monitored. For example, Cache-pod, rest-ep, and so on.

• retryCount value —Specify the retry counter value to retry if pod fails to ping after which pod is marked as down. It should be an integer in the range of 1-10.

• retryInterval interval_value —Specify the retry interval in milliseconds if the pod successfully pings. It should be an integer in the range of 200-10000.

• retryFailOverInterval failover_interval —Specify the retry interval in milliseconds if the pod fails to ping. It should be an integer in the range of 200-10000.

• failedReplicaPercent percent_value —Specify the percent value of failed replica after which the inter-rack redundancy failover is triggered. It should be an integer in the range of 10-100.

Configuration Example

The following is an example configuration.

geomonitor podmonitor pods cache-pod
 retryCount 3
 retryInterval 5
 retryFailOverInterval 1
 failedReplicaPercent 40
exit

Remote cluster monitoring auto corrects roles (it becomes self-primary, when the remote rack is in STANDBY_ERROR state) for uninterrupted traffic flow of traffic. However, this auto role correction gets done only for specific roles.

To configure this feature, use the following sample configuration:

config 
    geomonitor 
        remoteclustermonitor 
            retryCount value 
            retryInterval interval_value 
            end 

NOTES:

• retryCount value —Specify the retry count before making the current rack PRIMARY. It should be an integer in the range of 1-10. The default value is 3.

• retryInterval interval_value —Specify the retry interval in the count of milliseconds, after which the remote rack status gets fetched. It should be an integer in the range of 200-50000. The default value is 3000.

Configuration Example

The following is an example configuration

geomonitor remoteclustermonitor
retryCount 3
retryInterval 3000

The following command is used to monitor the traffic.

config 
 geomonitor 
  trafficMonitor 
   thresholdCount value 
   thresholdInterval interval_value 
  exit 
 exit 

NOTES:

• thresholdCount value —It specifies the number of calls received for standby instance. It should be an integer in the range of 0-10000. Default value is 0. Both UDP-proxy and REST-EP must be considered for the counter value.

• thresholdInterval interval_value —It specifies the maximum duration to hit the threshold count value in ms. It should be an integer in the range of 100-10000. Default value is 3000.

Configuration Example

The following is an example configuration

geomonitor trafficmonitor
thresholdCount 3
thresholdInterval 3000

Before deploying the CDL GR, user must configure the following:

• CDL Session Database and define the base configuration.

• Kafka for CDL.

• Zookeeper for CDL.

In CDL, along with existing GR related parameters, GR instance awareness must be enabled using a feature flag on all the racks. Also, the mapping of system-id to slice names should also be provided for this feature to work on all the racks.

The CDL is also equipped with Geo Replication (GR) failover notifications, which can notify the timer expiry of session data and bulk notifications to the currently active rack. The CDL uses Border Gateway Protocol (BGP) through App-Infra for the GR failover notifications.

The CDL subscribes to the key value on both the GR racks. The App-Infra sends notifications to the CDL when there is any change in these key values. A key value indicates the state of the CDL System ID or the GR instance. The GR instance is mapped to the CDL slices using the CDL system ID or the GR instance ID in the key.

The system ID is mandatory on both the racks. The GR instance ID in the NF configuration must match the CDL system ID.

CDL has instance-specific data slices. It also allows users to configure instance-specific slice information at the time of bringing up.

• CDL notifies the data on expiry or upon bulk notification request from the active slices.

• CDL determines the active instance based on the notification from app-infra memory-cache.

• CDL slice is a partition within a CDL instance to store a different kind of data. In this case, NF stores a different instance of data.

Note	CDL slice name should match with the slice-name configured in GR.

To switch the GR role, initiate the command on the primary rack (for example, role PRIMARY to STANDBY only), and use the following command.

geo switch-role { role primary | standby  instance-id gr_instanceId [ failback-interval  failback_interval ] } 

NOTES:

• role role —Specify the new role for the given rack.

  The roles can be primary or standby . It’s mandatory to trigger manual switchover from primary role for a specific GR instance ID.

• instance-id gr_instanceId —Specify the GR Instance ID

• failback-interval is an optional command to provide backward compatibility of upgrades between releases. The recommended value of failback-interval is 0.

Important

geo switch-role command triggers manual failover from one rack to another rack for specific instance ID. The rack which triggers the failover changes from the PRIMARY role to the STANDBY_ERROR role. In between, the rack which triggers the failover, sends a failover (Trigger GR) message to another rack. The other rack which receives the failover message changes from the STANDBY role to the PRIMARY role.

Note	failback-interval is deprecated from current release and will be discontinued from the subsequent releases.

To reset the GR instance role (for example, roles from STANDBY_ERROR to STANDBY to PRIMARY), use the following sample commands:

geo reset-role role role instance-id gr_instanceId 

NOTES:

• role role —Specify the new role for the given rack.

  The role can be PRIMARY or STANDBY.

• instance-id gr_instanceId —Specify the GR Instance ID.

Important

The command geo reset-role triggers change in the role for the given instance on the local rack. The remote rack does not receive any message for the same command. It is only possible to change the role for the given instance ID from STANDBY_ERROR to STANDBY and STANDBY to PRIMARY. Another role change is not possible.

This section describes show/clear commands that help in debugging issues.

To capture messages for subscriber (gr-instance aware), use the following command:

monitor subscriber [ supi ] [ imsi ] [ imei ] (capture-duration) (internal-messages) (transaction-logs) (nf-service) (gr-instance) 

Note	In 2021.02 and later releases, the namespace keyword is deprecated and replaced with the nf-service keyword.

NOTES:

• supi —Specify the subscriber identifier.

  Example: imsi-123456789, imsi-123*

• imsi —Specify the IMSI value.

  Example: 123456789, *

• imei —Specify the IMEI value.

• Example: 123456789012345, *

• capture-duration —(Optional) Used to specify the duration in seconds during which monitor subscriber is enabled. Default value is 300 secs.

• internal-messages —(Optional) When set to yes, it enables internal messaging. By default, it is disabled.

• transaction-logs —(Optional) When set to yes, it enables transaction logging. By default, it is disabled.

  Note	Messages and transaction logs are mutually exclusive.

• namespace —Deprecated option. Use nf-service instead.

• nf-service —(Optional) Specify the NF service. Possible values are sgw, smf. Default value is none.

• gr-instance —(Optional) Monitor subscriber for a given gr-instance only.

Example

The following is a configuration example.

monitor subscriber imsi 123456789 gr-instance 1
supi: imsi-123456789
captureDuration: 300
enableInternalMsg: false
enableTxnLog: false
namespace(deprecated. Use nf-service instead.): none
nf-service: none
gr-instance: 1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   295  100    98  100   197  10888  21888 --:--:-- --:--:-- --:--:-- 29500
Command: --header Content-type:application/json --request POST --data 
{"commandname":"mon_sub","parameters":{"supi":"imsi-123456789","duration":300,
"enableTxnLog":false,"enableInternalMsg":false,"action":"start","namespace":"none",
"nf-service":"none","grInstance":1}} http://oam-pod:8879/commands
Result start mon_sub, fileName ->logs/monsublogs/none.imsi-123456789_TS_2021-04-09T09:59:59.964148895.txt
Starting to tail the monsub messages from file: logs/monsublogs/none.imsi-123456789_TS_2021-04-09T09:59:59.964148895.txt
Defaulting container name to oam-pod.
Use 'kubectl describe pod/oam-pod-0 -n smf' to see all the containers in this pod.

For more information on Monitor Protocol on cnSGW-C, see the Monitor Subscriber and Monitor Protocol Support chapter in the UCC Serving Gateway Control Plane Function - Configuration and Administration Guide.

To capture packets on different interfaces (gr-instance aware), use the following command:

monitor protocol [ interface ] (capture-duration) (count) (level) (gr-instance)  

NOTES:

• interface —Interface on which PCAP is captured.

  Example: sbi, pfcp, gtpu, gtpc, gtp, radius

• list —Monitor protocol list files.

• capture-duration —(Optional) Used to specify the duration in secs during which PCAP is captured. Default value is 300 secs.

• pcap —(Optional) When set to yes, it enables PCAP file generation. By default, the value is "no" (disabled).

• gr-instance —(Optional) Monitor subscriber for a given gr-instance only.

Example

The following is a configuration example.

monitor protocol interface sbi gr-instance 1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   220  100    95  100   125   8636  11363 --:--:-- --:--:-- --:--:-- 20000
Command: --header Content-type:application/json --request POST --data 
{"commandname":"mon_pro","parameters":{"interface":"sbi","duration":300,"action":
"start","enable_pcap":false,"grInstance":1}} http://oam-pod:8879/commands
Result start mon_pro, fileName ->logs/monprologs/sessintfname_sbi_at_2021-04-30T05:26:22.712229347.txt
Starting to tail the monpro messages from file: logs/monprologs/sessintfname_sbi_at_2021-04-30T05:26:22.712229347.txt
Defaulting container name to oam-pod.
Use 'kubectl describe pod/oam-pod-0 -n cn' to see all of the containers in this pod.

For more information on Monitor Protocol on cnSGW-C, see the Monitor Subscriber and Monitor Protocol Support chapter in the UCC Serving Gateway Control Plane Function - Configuration and Administration Guide.

For GR deployment, the node-monitor pods starts automatically. During RMA procedure, the node-monitor pod automatically shutdown the rack if multi-compute failure is detected when the node is drain and deleted.

For more information on RMA (Return Merchandise Authorization), see SMI Cluster RMA section in the Ultra Cloud Core Subscriber Microservices Infrastructure - Operations Guide.

Before starting RMA process, perform the following:

1. Switch the role for both the instance to other rack using geo switch-role role command and make sure the target rack for RMA is in STANDBY_ERROR role for both the instances.

2. Disable the node-monitor pod.

   1. Take the backup of daemonsets.

      kubectl get daemonsets node-monitor -n cn -o yaml > node-monitor.yaml

   2. Delete node-monitor pods.

      kubectl delete daemonsets node-monitor -n cn

3. Continue with RMA procedure. For more information, see the link.

4. Once RMA procedure is complete, check if the node-monitor pods are already spawned.

   kubectl get pods -n cn -o wide | grep node-monitor

   If the node-monitor pods have not started, restart them.

   kubectl create -f node-monitor.yaml

   Note	node-monitor.yaml file is same as in Step 2.a.

5. Correct the role for the instances accordingly.

Note

For both earlier and current SMI versions: If you are replacing hardware components during an RMA procedure that contain firmware, such as an mLOM card, before adding the repaired or replaced node back to the cluster, you must run the HUU (Host Upgrade Utility) to ensure that the component is compatible with the system before syncing the node back into service. As part of RMA, if you remove a node from the cluster and before you return it to the manufacturer, you must purge all data on the device as per instructions provided by the hardware vendor.

The following section provides information on GR setup health check.

• All critical pods are in good condition to serve user traffic.

  Use the following command to check whether GR and CDL related pods are in Running state.

  kubectl get pods -n cn-cn1 -o wide | grep georeplication-pod
  kubectl get pods -n cn-cn1 -o wide | grep cdl
  kubectl get pods -n cn-cn1 -o wide | grep mirror-maker

• Keepalived pods are in healthy state to monitor all VIPs which are configured for check-interface/check-port.

  Use the following command to check whether keepalived pods in “smi-vips” namespace are in “Running” state.

  kubectl get pods -n smi-vips

• Health-check of pods related to CDL: Check the status of CDL db-endpoint, slot and indexes. All should be in STARTED or ONLINE state for both System IDs 1 and 2.

  cdl show status 
  message params: {cmd:status mode:cli dbName:session sessionIn:{mapId:0 limit:500 key: purgeOnEval:0 filters:[] nextEvalTsStart:0 nextEvalTsEnd:0 allReplicas:false maxDataSize:4096} sliceName:}
  db-endpoint {
      endpoint-site {
          system-id 1
          state STARTED
          total-sessions 4
          site-session-count 2
          total-reconciliation 0
          remote-connection-time 66h37m31.36054781s
          remote-connection-last-failure-time 2021-07-13 11:24:10.233825924 +0000 UTC
          slot-geo-replication-delay 2.025396ms
      }
      endpoint-site {
          system-id 2
          state STARTED
          total-sessions 4
          site-session-count 2
          total-reconciliation 0
          remote-connection-time 66h58m49.83449066s
          remote-connection-last-failure-time 2021-07-13 11:02:51.759971655 +0000 UTC
          slot-geo-replication-delay 1.561816ms
      }
  }
  slot {
      map {
          map-id 1
          instance {
              system-id 1
              instance-id 1
              records 4
              capacity 2500000
              state ONLINE
              avg-record-size-bytes 1
              up-time 89h38m37.335813523s
              sync-duration 9.298061ms
          }
          instance {
              system-id 1
              instance-id 2
              records 4
              capacity 2500000
              state ONLINE
              avg-record-size-bytes 1
              up-time 89h39m11.1268024s
              sync-duration 8.852556ms
          }
          instance {
              system-id 2
              instance-id 1
              records 4
              capacity 2500000
              state ONLINE
              avg-record-size-bytes 1
              up-time 89h28m38.274713022s
              sync-duration 8.37766ms
          }
          instance {
              system-id 2
              instance-id 2
              records 4
              capacity 2500000
              state ONLINE
              avg-record-size-bytes 1
              up-time 89h29m37.934345015s
              sync-duration 8.877442ms
          }
      }
  }
  index {
      map {
          map-id 1
          instance {
              system-id 1
              instance-id 1
              records 4
              capacity 60000000
              state ONLINE
              up-time 89h38m16.119032086s
              sync-duration 2.012281769s
              leader false
              geo-replication-delay 10.529821ms
          }
          instance {
              system-id 1
              instance-id 2
              records 4
              capacity 60000000
              state ONLINE
              up-time 89h39m8.47664588s
              sync-duration 2.011171261s
              leader true
              leader-time 89h38m53.761213379s
              geo-replication-delay 10.252683ms
          }
          instance {
              system-id 2
              instance-id 1
              records 4
              capacity 60000000
              state ONLINE
              up-time 89h28m29.5479133s
              sync-duration 2.012101957s
              leader false
              geo-replication-delay 15.974538ms
          }
          instance {
              system-id 2
              instance-id 2
              records 4
              capacity 60000000
              state ONLINE
              up-time 89h29m11.633496562s
              sync-duration 2.011566639s
              leader true
              leader-time 89h28m51.29928233s
              geo-replication-delay 16.213323ms
          }
      }
  }

• CDL replication status

  Check whether four gRPC connections are established between the CDL EP session pods (of each namespace) across the racks in GRPC_Connections_to_RemoteSite panel of CDL Replication Stats Grafana dashboard. Check Grafana on both racks.

  

• Admin port status between the racks for geo-replication.

  Check heartbeat messages between geo-replication pods across the racks in Periodic_Heartbeat_to_Remote_Site panel of GR Statistics Grafana dashboard.

  

• BGP/BFD link status on rack

  Check whether neighborship with BGP peers is established in BGP Peers panel of BGP, BFD Statistics Grafan dashboard.

  

  Check whether BFD link is in connected state in BFD Link Status panel of BGP, BFD Statistics Grafana dashboard.

  

• Roles of each instances are in healthy state

  Check that in each rack the roles are not in STANDBY_ERROR state at any point of time.

  • Active/Standby Model: Roles should be in the following states on each rack

    Rack-1:

    show role instance-id 1
    result "PRIMARY"
    show role instance-id 2
    result "PRIMARY"

    Rack-2:

    show role instance-id 1
    result "STANDBY"
    show role instance-id 2
    result "STANDBY"

  • Active/Active Model: Roles should be in the following states on each rack.

    Rack-1:

    show role instance-id 1
    result "PRIMARY"
    show role instance-id 2
    result "STANDBY"

    Rack-2:

    show role instance-id 1
    result "STANDBY"
    show role instance-id 2
    result "PRIMARY"

On Rack-1

1. Verify that roles of both instances on Rack-1 are in STANDBY_ERROR.

   show role instance-id 1
   result "STANDBY_ERROR"

   show role instance-id 2
   result "STANDBY_ERROR"

2. Initiate reset role for both instances on Rack-1 to STANDBY. This step transitions the roles from STANDBY_ERROR/STANDBY_ERROR to STANDBY/STANDBY.

   geo reset-role instance-id 1 role standby
   geo reset-role instance-id 2 role standby

3. Verify that roles of both instances have moved to STANDBY on Rack-1.

   show role instance-id 1
   result "STANDBY"

   show role instance-id 2
   result "STANDBY"

4. Initiate switch role for instance-id 1 on Rack-2 to STANDBY with failback-interval of 30 seconds. This step transitions the roles of Rack-2 from PRIMARY/PRIMARY to STANDBY_ERROR/PRIMARY and Rack-1 from STANDBY/STANDBY to PRIMARY/STANDBY.

   geo switch-role instance-id 1 role standby [failback-interval 0]

5. Verify that roles of both instances on Rack-2 are in STANDBY_ERROR/PRIMARY.

   show role instance-id 1
   result "STANDBY_ERROR"

   show role instance-id 2
   result "PRIMARY"

6. Verify that roles of both instances on Rack-1 are in PRIMARY/STANDBY.

   show role instance-id 1
   result "PRIMARY"

   show role instance-id 2
   result "STANDBY"

7. Initiate reset role for instance-id 1 on Rack-2 to STANDBY. This step transitions the roles of Rack-2 from STANDBY_ERROR/PRIMARY to STANDBY/PRIMARY.

   geo reset-role instance-id 1 role standby

8. Verify that the roles of Rack-2 are in STANDBY/PRIMARY.

   show role instance-id 1
   result "STANDBY"

   show role instance-id 2
   result "PRIMARY"

On Rack-2

1. Verify that roles of both the instances on Rack-2 are in STANDBY_ERROR.

   show role instance-id 1
   result "STANDBY_ERROR"

   show role instance-id 2
   result "STANDBY_ERROR"

2. Initiate reset role for both instances on Rack-2 to STANDBY. This step transitions the roles from STANDBY_ERROR/STANDBY_ERROR to STANDBY/STANDBY.

   geo reset-role instance-id 1 role standby
   geo reset-role instance-id 2 role standby

3. Verify that the roles of both the instances move to STANDBY on Rack-2.

   show role instance-id 1
   result "STANDBY"

   show role instance-id 2
   result "STANDBY"

4. Initiate switch role for instance-id 2 on Rack-1 to STANDBY. This step transitions roles of Rack-1 from PRIMARY/PRIMARY to PRIMARY/STANDBY_ERROR and Rack-2 from STANDBY/STANDBY to STANDBY/PRIMARY.

   geo switch-role instance-id 2 role standby [failback-interval 0]

5. Verify that roles of instances on Rack-1 are in PRIMARY/STANDBY_ERROR mode.

   show role instance-id 1
   result "PRIMARY"

   show role instance-id 2
   result "STANDBY_ERROR"

6. Verify that roles of instances on Rack-2 are in STANDBY/PRIMARY mode.

   show role instance-id 1
   result "STANDBY"

   show role instance-id 2
   result "PRIMARY"

7. Initiate reset role for instance-id 2 on Rack-1 to STANDBY. This step transitions the roles of Rack-1 from PRIMARY/STANDBY_ERROR to PRIMARY/STANDBY.

   geo reset-role instance-id 2 role standby

8. Verify that roles of instances on Rack-1 are in PRIMARY/STANDBY.

   show role instance-id 1
   result "PRIMARY"

   show role instance-id 2
   result "STANDBY"

The following section describes KPIs.

ETCD/Cachepod Replication KPIs

The following table lists ETCD/Cachepod Replication KPIs.

Geo Rejected Role Change KPIs

The following table lists Geo Rejected Role Change KPIs.

Monitoring KPIs

The following table lists monitoring KPIs.

BFD KPIs

The following table lists BFD KPIs.

GR Instance Information

Geo Maintenance Mode

The following section provides details on GR-specific bulkstats.

bulk-stats query GR-BGP-Incoming-Failed-Routes
 expression "sum(bgp_incoming_failedrouterequest_total) by (namespace, interface, service_IP, next_hop, instance_id)"
 labels     [ instance_id interface next_hop service_IP ]
 alias      gr-bgp-routes-in
exit
bulk-stats query GR-Geo-Monitoring-Failure
 expression "sum(geo_monitoring_total{ControlActionNameType=~'MonitorPod|RemoteMsgHeartbeat|
RemoteMsgGetSiteStatus|RemoteSiteRoleMonitoring|RemoteClusterPodFailure|RemoteMsgNotifyFailover|
RemoteMsgNotifyPrepareFailover|MonitorVip',status!~'success|monitoring.*'}) by (namespace, 
AdminNode, ControlActionType, ControlActionNameType,  pod, status, status_code)"
 labels     [ pod AdminNode ControlActionNameType status status_code ]
 alias      gr-geo-monitoring-failure
exit
bulk-stats query GR-Geo-Monitoring-Success
 expression "sum(geo_monitoring_total{ControlActionNameType=~'MonitorPod|RemoteMsgHeartbeat|
RemoteMsgGetSiteStatus|RemoteSiteRoleMonitoring|RemoteClusterPodFailure|RemoteMsgNotifyFailover|
RemoteMsgNotifyPrepareFailover',status=~'success|monitoring.*'}) by (namespace, AdminNode, 
ControlActionType, ControlActionNameType,  pod, status)"
 labels     [ pod AdminNode ControlActionNameType status ]
 alias      gr-geo-monitoring
exit
bulk-stats query GR-Geo-Monitoring-Total
 expression "sum(geo_monitoring_total{ControlActionNameType=~'MonitorPod|RemoteMsgHeartbeat|
RemoteMsgGetSiteStatus|RemoteSiteRoleMonitoring|RemoteClusterPodFailure|RemoteMsgNotifyFailover
|RemoteMsgNotifyPrepareFailover|MonitorVip'}) 
by (namespace, AdminNode, ControlActionType, ControlActionNameType,  pod, status)"
 labels     [ pod AdminNode ControlActionNameType status ]
 alias      gr-geo-monitoring
exit
bulk-stats query GR-Geo-Replication-Failure
 expression "sum(geo_replication_total{ReplicationNode=~'CACHE_POD|ETCD|PEER',status!='success',
ReplicationRequestType='Response'}) by (namespace, ReplicationNode, ReplicationSyncType,
ReplicationReceiver,ReplicationRequestType,status,status_code)"
 labels     [ pod ReplicationNode ReplicationReceiver ReplicationRequestType ReplicationSyncType status status_code ]
 alias      gr-geo-replication-failure
exit
bulk-stats query GR-Geo-Replication-Success
 expression "sum(geo_replication_total{ReplicationNode=~'CACHE_POD|ETCD|PEER',
status='success',ReplicationRequestType='Response'}) by (namespace, ReplicationNode, 
ReplicationSyncType,ReplicationReceiver,ReplicationRequestType,status)"
 labels     [ pod ReplicationNode ReplicationReceiver ReplicationRequestType ReplicationSyncType status ]
 alias      gr-geo-replication-success
exit
bulk-stats query GR-Geo-Replication-Total
 expression "sum(geo_replication_total{ReplicationNode=~'CACHE_POD|ETCD|PEER'}) 
by (namespace, ReplicationNode, ReplicationSyncType,ReplicationReceiver,
ReplicationRequestType,pod)"
 labels     [ pod ReplicationNode ReplicationReceiver ReplicationRequestType ReplicationSyncType ]
 alias      gr-geo-replication-total
exit
bulk-stats query GR-Trigger-ResetRole-Api
 expression "sum(geo_monitoring_total{ControlActionNameType=~'TriggerGRApi|ResetRoleApi'}) 
by (namespace, AdminNode, ControlActionType, ControlActionNameType,  pod, status, status_code)"
 labels     [ pod AdminNode ControlActionNameType status status_code ]
 alias      gr-api
exit
bulk-stats query GR-CDL-Index-Replication
 expression "sum(consumer_kafka_records_total) by (pod, origin_instance_id)"
 labels     [ origin_instance_id pod ]
 alias      gr-cdl-index-replication
exit
bulk-stats query GR-CDL-Inter-Rack-Replications-Failures
 expression "sum(datastore_requests_total{local_request='0',errorCode!='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-inter-rack-replications
exit
bulk-stats query GR-CDL-Inter-Rack-Replications-Success
 expression "sum(datastore_requests_total{local_request='0',errorCode='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-inter-rack-replications
exit
bulk-stats query GR-CDL-Inter-Rack-Replications-Total
 expression "sum(datastore_requests_total{local_request='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-inter-rack-replications
exit
bulk-stats query GR-CDL-Intra-Rack-Operations-Failures
 expression "sum(datastore_requests_total{local_request='1',errorCode!='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-intra-rack-operations
exit
bulk-stats query GR-CDL-Intra-Rack-Operations-Success
 expression "sum(datastore_requests_total{local_request='1',errorCode='0'}) by (operation,sliceName,errorCode)"
 labels     [ sliceName operation errorCode ]
 alias      gr-cdl-intra-rack-operations
exit
bulk-stats query GR-CDL-Intra-Rack-Operations-Total
 expression "sum(datastore_requests_total{local_request='1'}) by (operation,sliceName,errorCode)"
 labels     [ errorCode operation sliceName ]
 alias      gr-cdl-intra-rack-operations
exit
bulk-stats query GR-CDL-Session-Count-Per-Slice
 expression sum(avg(db_records_total{namespace=~'$namespace',session_type='total'})by(systemId,sliceName))by(sliceName)
 labels     [ sliceName ]
 alias      gr-cdl-session-count-per-slice
exit
bulk-stats query GR-CDL-Session-Count-Per-System-ID
 expression sum(avg(db_records_total{namespace=~'$namespace',session_type='total'})
by(systemId,sliceName))by(systemId)
 labels     [ systemId ]
 alias      gr-cdl-session-count-per-system-id
exit
bulk-stats query GR-CDL-Slot-Records-Per-Slice
 expression "sum(slot_records_total{pod=~'.*',systemId!=''}) by (pod, sliceName)"
 labels     [ pod sliceName ]
 alias      gr-cdl-slot-records-per-slice
exit
bulk-stats query GR-CDL-Slot-Records-Per-System-ID
 expression "sum(slot_records_total{pod=~'.*',systemId!=''}) by (pod, systemId)"
 labels     [ pod systemId ]
 alias      gr-cdl-slot-records-per-system-id
exit
bulk-stats query GR-CDL-Total-Session-Count
 expression "sum(db_records_total{namespace=~'$namespace',session_type='total'}) by (systemId,sliceName)"
 labels     [ sliceName systemId ]
 alias      gr-cdl-total-session-count
exit

For more information on GR-related statistics, see the following:

• In cnSGW-C statistics, you can filter GR-specific statistics using gr_instance_id label.

  For more information, see the UCC Serving Gateway Control Plane Function - Metrics Reference.

The following section provides details on GR alerts.

BFD Alerts

The following table list alerts for rule group BFD with interval-seconds as 60.

GR Alerts

The following table list alerts for rule group GR with interval-seconds as 60.

CDL Alerts

The following table list alerts for rule group CDL with interval-seconds as 60.

cnSGW-C supports the maintenance mode flag to disable the impact on a cluster if the cluster in GR setup is scheduled for in-service (rolling upgrade). This is useful so that the other mated cluster executes its responsibility and other activities on the targeted cluster without any issue.

If the maintenance mode flag is set to true, cluster role change and GR trigger for the rack is allowed only in case of CLI-based failover.

During the execution, all the monitoring threads check the runtime value for the flag and hold the execution if the maintenance mode flag is set to true. By default, for fresh installation, the flag is set to false. Based on your requirements, to configure the maintenance mode, use the following configuration.

config 
  geo maintenance mode { true | false } 
  end  

NOTES:

• geo maintenance mode { true | false } - Enable/disable the maintenance mode.

  The value for the maintenance mode is stored in.etcd

Both the clusters can be under maintenance at the same time. You can push the system in maintenance mode if the mated cluster is already under maintenance. Before you start the maintenance activity, set the geo maintenance mode flag value to true. When maintenance is complete, reset the flag to false after confirming the health of the system.

When the maintenance flag is set to true:

• All the monitoring activities are paused.

• The mated cluster cannot trigger the failover when it detects the local failure.

• Replication activities continue on the cluster.

• Maintenance mode doesn't change instance roles of the site implicitly. However, role change is possible using geo switch-role role CLI command.

GR trigger is not allowed towards and from the cluster under maintenance. Only CLI-based failover is supported from the cluster under the maintenance. After disabling maintenance mode, start with new data for pod and VIP monitoring. Remote cluster is informed about the maintenance mode value using the NotifyMaintenanceActivity() [Operation 24] message.

Example

The following is a configuration example:

geo maintenance mode true
result "success"

geo maintenance mode false
result "success"

Viewing the Maintenance Mode Status

To check the maintenance mode status, use the following show command.

show geo maintenance mode
result "geo maintenance mode is disabled"