Distributed Automation 2.0 Whitepaper

Cisco Resilient Mesh Solution for Schweitzer Engineering Laboratories (SEL)

Fault Location, Isolation, and Service Restoration (FLISR) application

 

January 28, 2020 9:44 AM

 

 

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company.

© 2020 Cisco Systems, Inc. All rights reserved.

 

Navigator

The document includes the following:

Audience

The audience of this whitepaper is system architects, network/compute/systems engineers, Distribution Engineers, SCADA operators, field consultants, Cisco Customer Experience (CX) specialists, Partners and other customers.

Readers may be familiar with basic knowledge of the DA Fault Location, Isolation and Service Restoration (FLISR) application and radio mesh networks.

Document Objective and Scope

This document explains the functional testing that the Cisco IOT Solution team has recently performed to validate the FLISR application using the Cisco Resilient Mesh solution in an indoor lab environment. It includes information about the testbed equipment and topologies that were used, test case description and test results. 

 

Executive Summary

Cisco Systems is addressing the networking needs of the utility industry. Enhancements to the Cisco Resilient Mesh solution that have increased the available bandwidth on an unlicensed 900-MHz field area network by nearly 10 times over the first generation are discussed in this document. The radio mesh now addresses utility distribution grid use cases such as SCADA transport, Fault Location Isolation and Service Restoration (FLISR), and line voltage monitoring to support applications like Volt/VAR Control and Line Sensor. Cisco has increased the performance of the advanced metering network making it suitable for Distribution Automation and Renewable Energy use cases.

Significant advantages of greater Grid reliability during system faults, reduced outage duration, lower Customer Average Interruption Duration Index (CAIDI), and Momentary Average Interruption Frequency Index (MAIFI)—the industry service reliability Key Performance Indicators (KPIs)—make these new applications attractive for use on Distribution Networks. Customers considering implementing FLISR, Volt/VAR and Line Sensor can eliminate the need for additional infrastructure and legacy grid device replacement, thereby reducing operation and maintenance cost. Additionally remote monitoring and management capability, and support for Distributed Energy Resources (DER) applications make this solution highly attractive.

Cisco Resilient Mesh Solution is based on the unlicensed ISM 900Mhz band and Wi-SUN 1.0 foundations. It supports these Distributed Automation (DA) applications providing a reliable communication network, sufficient bandwidth capacity, and low latency per mesh node to meet system performance requirements.

The Schweitzer Engineering Laboratories (SEL) FLISR products perform optimally with the Cisco Resilient Mesh network. Performance results such as trip time, data alignment, service restoration, and operation consistency are consistently good. From intense testing efforts, a radio mesh based on ISM 902-928MHz and the IEEE802.15.4g/e standard using OFDM modulation with a physical data rate up to 1.2 Mbps was demonstrated to support the performance requirements of the FLISR application.

This white paper describes the Cisco Resilient Mesh network performance supporting the SEL FLISR application successfully over a variety of topologies and locations in the network.

 

Introduction

Early in 2019 Cisco released its next generation Resilient Mesh solution and products enabling utility customers to leverage the same infrastructure for both Advanced Meter Infrastructure (AMI) and Distribution Automation (DA). With a communication infrastructure in place, customers gain visibility into the electrical distribution network for monitoring and control of the grid devices. Utility customers can now deploy new applications to improve the operational, maintenance, and security of their networks.

Cisco Resilient Mesh enhancements have increased the available bandwidth on the 902-928 MHz field area network by nearly tenfold over the first generation. This has reduced the latency between hops to less than 15 milliseconds, enabled peer-to-peer communication, and enhanced edge security features while using IEEE open standards protocols and WiSUN industry-compliant certification.

Cisco is committed to provide industry guidance using this multiservice solution optimized for Distribution Automation applications by testing and validating all major DA use cases in their indoor labs and outdoor pilot locations. This paper describes the validation efforts.

Tests conducted in the indoor labs focused on FLISR using equipment from SEL.

SEL FLISR

SEL developed a comprehensive solution for the DA FLISR application that can be deployed in distributed or centralized architectures. The solution includes a controller device to provide advanced restoration capabilities that can be located in the distribution substation or control center. Combined with Cisco Resilient Mesh communication infrastructure, the FLISR application can operate in fully-automatic mode.

The equipment used in testing is shown in the figures that follow.

 

For additional information about the SEL RTAC product family visit: https://selinc.com/products/3530/

For additional information about the SEL-651R recloser product visit: https://selinc.com/products/651R/

 

The SEL-tested network is shown in Figure 1 that follows.

 

Figure 1 – Cisco Resilient Mesh and SEL FLISR Architecture

 

The SEL reclosers connect to the Cisco Resilient Mesh Industrial Routers (IR510) use either the serial port (RS232 or RS485) or Ethernet port.

The Cisco IR510 DA gateways establish a reliable mesh wireless network based on signal propagation radio design. For instances where the radio signal cannot cover a certain area or signal levels are weak, Cisco Range Extenders (IR530) can be deployed to increase signal coverage for that area.

DNP3 messages between the SEL-651R recloser and the SEL controller are reliably routed through the mesh network using the most optimal path to the field area border router. The Cisco CGR 1000 router, known as the mesh exit point, can then send the signal using either a fiber network or Cellular network to the Control Center location.

The SEL FLISR solution can be deployed in a decentralized architecture in which the RTAC device is installed within the Distribution Substation Local Area Network (LAN) and has a smaller service area and reduced number of feeders leading to faster service remediation and less configuration complexity.

The SEL FLISR solution can also be deployed in a centralized architecture in which one or more RTAC devices are installed in the Control Center and each controller services an area that is not necessarily associated to a substation service area. This approach has a lower deployment cost, but does increase the controller configuration complexity.

 

Cisco Resilient Mesh for DA

Utility customers that select a privately-managed solution can opt to run their DA systems using the Cisco Resilient Mesh solution based on the unlicensed ISM 902-928 MHz band. This solution eliminates any dependencies on third-party services and allows the customer to operate its own network provisioning, configuration management, and troubleshooting. It offers a lower operation cost model because there is no monthly service charge for the transport service.

This solution is depicted in Figure 2 that follows.

Figure 2 – Cisco Resilient Mesh Architecture

 

The Cisco Field Area Network (FAN) DA solution uses the following products to build a radio mesh network based on the IEEE 802.15.4g/e standard, compliant with Wi-SUN 1.0. The radio network requires a radio mesh Personal Area Network (PAN) coordinator. This function is supported by the solution Field Area Routers (FAR). Cisco offers two products that customer can use as FAR devices: the Cisco CGR1240 or CGR1120 shown in the image that follows.

 

 

 

The CGR1240 is an IP67-rated device that can be deployed outdoors, includes 4 module slots, and is AC powered. The slots can be used: 1 for OFDM WPAN and 2 for cellular backhaul.

The CGR 1120 is an IP30-rated device for weatherproof enclosures or indoor substation installations, and is AC or DC powered. The CGR1140 has 2 module slots that can be used: 1 for OFDM WPAN and 1 for cellular interface.

To eliminate single points of failure such as a single CGR chassis or single WPAN module failure, customers may deploy the Field Area Router in High-Availability Active/Standby configuration. By deploying two of these routers within the same location, each with its own WPAN module, a redundancy is created. In this design, only one WPAN module is active at any given time while the second one is in standby mode and ready to become active if the primary WPAN fails. 

The IR510 shown in Figure 3 below, is the next generation DA Gateway that connects utility grid assets to the Cisco Resilient Mesh network. It supports higher data rates based on OFDM modulation and is best suited for DA applications. It has dedicated hardware resources for Edge Compute applications, so that customers can deploy their own applications at the edge of the network.

Figure 3. Cisco IR510 DA Gateway

 

For additional information about the IR510, visit: https://www.cisco.com/c/en/us/products/collateral/routers/500-series-wpan-industrial-routers/datasheet-c78-730550.html

 

When grid assets are not close enough to each other for a clear line of sight or on lines in rural areas with long feeders, customers can deploy the Cisco Range Extender to provide additional signal coverage. The latest range extender product release is the Industrial Router (IR530) which supports the higher OFDM data rates. The IR530 is shown in Figure 4 below.

 

Figure 4.  Cisco IR530 Range Extender

 

 

For additional information about the IR530 visit: https://www.cisco.com/c/en/us/products/collateral/routers/500-series-wpan-industrial-routers/datasheet-c78-740201.html

 

Test Bench and Configuration

The Cisco solution validation team built an indoor conducted-mesh topology using coax cables, signal splitters and attenuators, enabling signal variations to construct a 22-node network. In one section of the topology the team simulated a subset of an urban mesh deployment that has a few aggregation points with a ratio of 1:4, and 4 hops deep. The remaining network was used to simulate a long rural mesh deployment, 10 hops deep.

In mesh networks, nodes that can hear each other, in that the RSSI is within the acceptable range for a specific modulation (OFDM or FSK) and data rate, establish one of the following relationships:

§  Parent

§  Child

§  Neighbor

A parent node offers the best route back to the exit point out of the mesh network, which in the Cisco Resilient Mesh is the CGR 1000 series router. Each route uses a cost metric that reflects the shortest and most reliable path to reach the CGR router indicated by the node rank.

To avoid routing loops, a child node selects the parent node based on the parent rank which must be lower than the child node rank. For resiliency, a child node can select up to 4 parents in the parent table, but only one parent that advertises the best route will be selected as the preferred parent.

A neighbor node is any node within the RF range not selected as a parent or child because of its high rank towards the mesh exit point.

The logical network configuration is shown in Figure 5 that follows.

 

Figure 5.  Logical Cisco Resilient Mesh Topology

 

The RF connectivity between the Cisco DA gateways was designed for IEEE 802.15.4 Option 2, (OFDM Mode 149 on Cisco Resilient Mesh) which corresponds to a physical layer data rate of 800Kbps. The OFDM 800Kbps maximum Received Signal Strength Indicator (RSSI) is -101dB. To avoid node flapping and instability in the network, a new node joining the mesh for the first time must have a minimum link RSSI of -91dB with respect to its neighbors. After a node joins the network, the link RSSI can weaken to the maximum RSSI of -101dB. This RSSI implies a 10% packer error rate before the link is considered unusable and the node is removed from the neighbor node table.

A best practice design rule was that the links between the Cisco DA devices were designed for an average link RSSI range between -80 to -89dB.

The mesh radio parameters were configured for Distribution Automation applications. The size of the mesh was configured as “Small”; this optimizes default parameters using the lower IEEE802.15.4g and Routing Protocol for Low-Power and Lossy Networks (RPL) timers. RPL was also configured to operate in storing mode to support peer-to-peer communication and to help baseline the network performance.

The aggregation section of the network in Figure 5 represents a subset of a single urban distribution substation mesh network where the distances between DA grid devices is shorter and nodes can aggregate traffic from multiple children. The ratio of child to parent is higher and the parent available bandwidth is shared among the children. A benefit of this dense network is an increase in resilience due to the additional backup nodes and paths through which the DA traffic can flow in case of link or node failure. This can also increase the bandwidth for the area because multiple diverse paths will exist through the network.

To simulate this network, the 2^nd, 3^rd and 4^th hop nodes were design to establish multiple layer 1 connections. The nodes above that have a lower rank, enabling multiple parents in the parent table for traffic forwarding.

The nodes in the aggregation section were configured to prefer the first parent starting from the right by using lower link attenuation than the other parent nodes. This simulates the worst case scenario where all children traffic convergences on a single parent node. In an actual deployment, traffic from the children would simply flow to the parent immediately above and multiple routes within the aggregation section of the topology would exist. This situation is depicted in Figure 6.

 

Figure 6.  Aggregation, child path forwarding

 

The linear section of the topology simulates a long rural deployment with 10 hops between nodes. See Figure 7. Each node only has two neighbors, one parent from the higher rank, and one child from the rank below. The links were also designed for the same average RSSI range as in the aggregate topology. The higher the hop count of a path within a mesh network the higher the end-to-end latency, because each additional hop introduces additional processing delay. The end-to-end path throughput decreases as the hop count increases.

Figure 7.  Linear, node path forwarding

The SEL FLISR Testbed

The Cisco team first validated the DA FLISR application using 10 SEL DA devices. For ease of testing the team used the SEL RTAC 3505 controller instead of the SEL-651R recloser, and it was configured with the same number of data points and functions as the actual recloser device to create the same network communication traffic. Using a power simulator to connect all 10 reclosers was determined to be outside the scope of this testing effort.

The SEL FLISR was tested in a centralized configuration in which a SEL RTAC 3530 was installed in the Control Center. The DA FLISR controller was configured to communicate with each SEL RTAC3505 to work as a system to perform service restoration, also known as circuit reconfiguration, during a grid outage event. A second SEL RTAC3530 DA Simulator was used to simulate different grid conditions and to create different failures over a dedicated network referred to as a Simulator Ethernet Network. A laptop running the SEL AcSELerator software was used for SEL device configuration, FLISR topology monitoring, and fault insertion. This simulation network is depicted in Figure 8.

Figure 8.  SEL Simulation Network

 

The team created two FLISR projects in the SEL AcSELerator software to simulate an urban and a rural DA deployment.

The one-line diagram for the urban topology shown in Figure 9.

It included four feeders that were interconnected with reclosers in Normal Open state (green box). Each feeder capacity was designed for 540A and it was sourced from an independent transformer. Substation breakers located at the beginning of each feeder offered protection for the entire distribution line. Different loads were placed on the feeders so that the SEL FLISR controller can select the most optimal feeder as the next power source during an outage and service restoration phase.

 

Figure 9. FLISR Urban One-Line Diagram

 

The SEL reclosers were connected to the IR510s through the Ethernet port based on the diagram in Figure 10 and Table 1 that follow.

 

 

Figure 10.  Urban SEL reclosers to Cisco RM device mapping diagram

 

Table 1.  FLISR Urban Topology Components

 

The Cisco team has also simulated a rural linear distribution line that extended over 10 SEL reclosers between two substations where the recloser Rec5 was in Normal Open state (NO) and all other reclosers were in Normal Close state (NC).

 

Figure 11.  FLISR Rural One-Line Diagram

 

 

The SEL reclosers were connected through Ethernet to each Cisco IR510, but in this scenario the Linear section of the Mesh network was in the configuration shown in Figure 12 and described in Table 2 that follow.

 

Figure 12.  Rural SEL reclosers to Cisco RM device mapping diagram

Table 2.  FLISR Rural Topology Components

 

Test Scenarios and Results

For the SEL FLISR application, the functional validation included these three types of failures:

§  Fault with Lockout

§  Open Phase

§  Loss of Source

which are described in detail in the upcoming section. One-line diagrams are included to show what the FLISR system topology looked like during each Normal, Abnormal, and Restored state.

Fault with Lockout Test Scenarios

The first test case that was executed was for a permanent fault, in which the recloser goes into lockout state. One example of a permanent fault is when a heavy tree branch falls on the distribution line and conductors break. In that case each recloser that senses the fault will trip and send an unsolicited DNP3 message upstream to the DA FLISR Controller. The next step in the FLISR scheme is for the DA Controller to request a Class0123 poll from all the reclosers for an up-to-date view of all the feeder status before proceeding to the FLISR restoration phase. The DA Controller sends DNP3 commands to the respective reclosers that needs to change state in order to restore the services.

The test was executed first on the Urban topology. The topology in Normal State looks like the network depicted in
Figure 13.

 

Figure 13.  Urban Fault with Lockout Normal State Topology

 

A fault was introduced on Feeder number one, between Recloser1 and Recloser2. Once the fault is detected both reclosers change their state from NC to NO as shown in Figure 14.

 

Figure 14.  Urban Fault with Lockout Abnormal State Topology

The faulty segment was isolated and Recloser3 was reconfigured to a Normal Close state and service was restored for the customers downstream the Recloser2.

 

Figure 15.  Urban Fault with Lockout Restored State Topology

 

The test was repeated five times and in each test, the SEL DA FLISR solution was able to successfully use the Cisco Resilient Mesh communication infrastructure to restore services quickly, less than 5 minutes and meet the SEL restoration time requirement.

The SEL and Cisco DA solutions help utilities lower the System Average Interruption Duration Index (SAIDI) and System Average Interruption Frequency Index (SAFI) performance metrics which reflects the reliability of their Distribution network.

 Next, the team also validated the SEL FLISR Fault with Lockout functionality over the Rural topology.

 

Figure 16.  Rural Fault with Lockout Normal State Topology

 

A fault was introduced between Recloser2 and Recloser3 as depicted in Figure 17 which makes the two reclosers trip and change state to Normal Open. All the customers downstream of the Recloser3 up to Recloser5 experience a service outage. Each recloser will send a DNP3 unsolicited message to notify the DA Controller which in term will start the FLISR Service Restoration phase.

 

Figure 17.  Rural Fault with Lockout Abnormal State Topology

 

The DA Controller will poll all the reclosers and determine the best way to restore services. In the linear topology there is only a single restore option therefore the controller will send a DNP3 command to Recloser5 to change state from Normal Open to Normal Close. Service is restored for all the customer segments between Recloser3 and Recloser5 as shown in Figure 18.

 

Figure 18.  Rural Fault with Lockout Restored State Topology

Open Phase Test Scenarios

Open Phase fault applies to three phase circuits, where one of the line voltages is lost. One cause could be due to a bad or loose street pole line jumper that interrupts the line. The loss of voltage will trigger again a DNP3 unsolicited message to the DA Controller that will start the remediation schema.

Because the fault was inserted in the same segment as the previous example, the detailed diagrams were omitted as redundant, and the following diagrams still apply:

            Urban Topology:

§  Normal State: Figure 19 Urban Loss of Source Normal State Topology

§  Abnormal State: Figure 14 Urban Fault with Lockout Abnormal State Topology

§  Restored State: Figure 15 Urban Fault with Lockout Restored State Topology

Rural Topology:

§  Normal State: Figure 16 Rural Fault with Lockout Normal State Topology

§  Abnormal State:  Figure 23 Rural Loss of Power Abnormal State Topology

§  Restored: State:  Figure 24 Rural Loss of Power Restored State Topology

All the five test repetitions resulted in successful operation of the SEL FLISR solution and met the expected recovery time.

Loss of Source Test Scenarios

Loss of source applies to fault that occur within the substation yard, for example a transformer going bad that causes an outage for the entire feeder originating from that substation. In this scenario the relay within the substation will notify the DA controller of the loss of power which will initiate the restoration process.

 

Figure 19.  Urban Loss of Source Normal State Topology

 

The third feeder transformer was taken out of service to simulate a loss of power. The substation feeder breaker (Brk3) tripped and all the downstream customers lost power.

Figure 20.  Urban Loss of Power Abnormal State Topology

The DA FLISR Controller finds the most optimal way to restore the services and, in this case, it chose to change the Recloser8 state from Normal Open to Normal Close. 

Figure 21.  Urban Loss of Power Restored State Topology

The scenario was also repeated for the Rural topology below.

Figure 22.  Rural Loss of Power Normal State Topology

 

The feeder number one transformer was taken out of service to simulate a fault. The substation feeder breaker tripped to Open state and all customers on that feeder loss power as display in Figure 23.

 

Figure 23.  Rural Loss of Power Abnormal State Topology

The Recloser6 status was reconfigured by the DA Controller to a Normal Close (NC) state and service was restored from the second feeder as shown in Figure 24.

 

Figure 24.  Rural Loss of Power Restored State Topology

The Cisco Resilient Mesh support for IEEE 802.15.4g Option 2 profile for OFDM modulation with a link layer bandwidth of 800Kbps proved to enhance the reliability of the SEL FLISR solution. The time it took to locate, isolate, and restore services was drastically improved. The table below summarizes all the FLISR test cases that were conducted and the results.

Table 3.  FLISR Test Results Summary

 

Conclusion

The use of Cisco Resilient Mesh networks for DA FLISR application yields great advantages to utility customers in terms of improved grid reliability, located faults faster, self-healing capabilities during outage events, operational efficiencies of the distribution network by leveraging a reliable, fast, private communication infrastructure with a low OPEX cost.

SEL FLISR equipment and Cisco IR510 devices, together deliver technical excellence and business value by providing a modern communication platform and protection system to meet the immediate needs of today and anticipate evolving requirements far into the future over the lifecycles of protection equipment and communication infrastructure.

 

Contributors

Sean Jian: Utility Director, IoT Solutions Group, Cisco Systems

Caldwel E: Manager, IoT Solutions Group, Cisco Systems

Dave Schmidt: Solution Architect, IoT Solutions Group, Cisco Systems

George Mihalachioaie: Solution Architect, IoT Solutions Group, Cisco Systems

Ashok Thirugnanasambandam: Engineer, IoT Solutions Group, Cisco Systems

Shaikh Jahiruddin: Engineer, IoT Solutions Group, Cisco Systems

Glossary

CAIDI:  Customer Average Interruption Duration Index

FLISR: Fault Location, Isolation and Service Restoration

KPI: Key performance indicator

MAIFI: Momentary Average Interruption Frequency Index

RPL: Routing Protocol for Low-Power and Lossy Networks

SCADA: Supervisory control and data acquisition

 

 

 

 

Additional Resources

Cisco Distribution Automation Solution At-a-Glance

https://www.cisco.com/c/dam/en_us/solutions/industries/resources/energy/utlilities/1584310_AAG_Utilities_DistributionAutomation_corpedit.pdf

 

Cisco Distribution Automation Feeder Automation Design Guide

https://www.cisco.com/c/en/us/td/docs/solutions/Verticals/Distributed-Automation/Feeder-Automation/DG/DA-FA-DG/DA-FA-DG.html