Cisco IoT Connected Grid Network
This section provides an overview of:
The Cisco IoT Field Network Director (IoT FND) is a network management system that manages multi-service network and security infrastructure for IoT applications, such as smart grid applications, including Advanced Metering Infrastructure (AMI), Distribution Automation (DA), distributed intelligence, and substation automation. IoT FND is a scalable, highly-secure, modular, and open platform with an extensible architecture. IoT FND is a multi-vendor, multi-service, communications network management platform that enables network connectivity to an open ecosystem of power grid devices.
IoT FND is built on a layered system architecture to enable clear separation between network management functionality and applications, such as a distribution management system (DMS), outage management system (OMS), and meter data management (MDM). This clear separation between network management and applications helps utilities roll out Smart Grid projects incrementally, for example with AMI, and extend into distribution automation using a shared, multi-service network infrastructure and a common, network management system across various utility operations.
Features
-
Geographic Information System (GIS) map-based, visualization, monitoring, troubleshooting, and alarm notifications
-
Group-based configuration management for routers and smart meter endpoints
-
OS compatible (Cisco IOS, Guest OS, IOx) and provides application management
-
Rule-engine infrastructure for customizable threshold-based alarm processing and event generation
-
North Bound API for transparent integration with utility head-end and operational systems
-
High availability and disaster recovery
Cisco IoT FND provides powerful Geographic Information System (GIS) visualization and monitoring capability. Through the browser-based interface, utility operators manage and monitor devices in a Cisco IoT Connected Grid Field Area Network (FAN) solution, using IPv6 over Low-power Wireless Personal Area Networks (6LoWPANs). The FAN includes the following devices:
-
Cisco 1000 Series Connected Grid Routers (CGRs), also called pole-top or DIN-rail-mount routers. These devices are referred to as routers in this document and identified by model (for example, CGR1000, CGR1120, or CGR1240) on the Field Devices page. Available CGR modules provide 3G, 4G LTE, and Cisco Resilient Mesh connectivity (WPAN). CGR1000s also support the Itron OpenWay RIVA CAM module, which provides connectivity to the Itron OpenWay RIVA electric and gas-water devices.
-
Cisco 800 Series Integrated Services Routers (ISR 800s) are used in most networks as edge routers or gateways to provide WAN connectivity (cellular, satellite over Ethernet, and WiFi) to an end device (energy-distribution automation devices, other verticals such as ATMs, and mobile deployments such as taxis or trucks). These devices are referred to as routers in this document; and identified by product ID on the Field Devices page. You can use IoT FND to manage the following hardened Cisco 819H ISRs:
-
C819HG-4G-V-K9
-
C819HG-4G-A-K9
-
C819HG-U-K9
-
C819HGW-S-A-K9
-
C819H-K9
IoT FND also manages the following non-hardened Cisco 819 ISRs:
-
C819G-B-K9
-
C819G-U-K9
-
C819G-4G-V-K9
-
C819G-7-K9
-
-
Cisco 4000 Series Integrated Services Routers (ISR 4300 and ISR4400) consolidate many must-have IT functions in a single platform, such as network, security, compute, storage, and unified communications to help you build out the digital capabilities in your enterprise branch offices. The platform is modular and upgradable, so you can add new services without changing equipment.
-
Cisco 800 Series Industrial Integrated Services Routers (IR800s) are compact, ruggedized, Cisco IOS Software routers. They offer support for integrated 4G LTE wireless WAN (IR807, IR809 and IR829 models) and wireless LAN capabilities (IR829 only). These devices are referred to as routers in this document; and identified by product ID (for example, IR800) on the Field Devices page. You can use IoT FND to manage the following IR800 models:
-
IR807: Highly compact, low-power industrial router. Well-suited for industrial applications (distribution automation for utilities, transportation, manufacturing) and remote asset management across the extended enterprise.
-
IR809: Very compact, cellular (3G,4G/LTE) industrial routers that enable reliable and secure cellular connectivity for remote asset monitoring and machine-to-machine (M2M) applications such as distribution automation, pipeline monitoring and roadside infrastructure monitoring.
-
IR829: Highly ruggedized compact cellular (3G and 4G LTE with GPS and dual SIM) and WLAN (2.4/5GHz) industrial routers supporting scalable, reliable, and secure management of those IoT applications requiring mobile connectivity such as fleet vehicles and mass transit.
-
-
Cisco 5921 Embedded Services Router (ESR) is designed to operate on small, low-power, Linux-based platforms. It helps integration partners extend the use of Cisco IOS into extremely mobile and portable communications systems. It also provides highly secure data, voice, and video communications to stationary and mobile network nodes across wired and wireless links.
-
The Cisco Wireless Gateway for LoRaWAN (IXM-LPWA-800, IXM-LPWA-900) can be a standalone product that connects to Ethernet switches or routers or connects to LAN ports of the Cisco 800 Series Industrial Integrated Services Routers. This product can be configured as a radio interface of the Cisco Industrial Routers 809 and 829. One or multiple gateways are connected to the LAN port(s) of the IR809 or IR829 via Ethernet or VLANs with encrypted links. Through this configuration, it provides LoRaWAN radio access while the IR809 or IR829 offer backhaul support for Gigabit Ethernet (electrical or fiber), 4G/LTE, or Wi-Fi. You can employ either a default-group tunnel group or a user-defined tunnel group.
-
Cisco Interface Module for Long Range Wide Area Network (LoRAWAN) is an extension module for the industrial routers, Cisco IR809 and IR829, and serves as a carrier-grade gateway for outdoor deployments. The module provides unlicensed low-power wide area (LPWA) wireless connectivity for a range of Internet of Things (IoT) use cases such as asset tracking, water and gas metering, street lighting, smart parking/building/agriculture, and environment monitoring. There are two models that are supported, which are differentiated by their band support (863-870 MHz ISM or 902-928 MHz ISM). The module is identified by product ID (for example, IXM-LORA-800-H-V2).
-
Cisco 500 Series Wireless Personal Area Network (WPAN) Industrial Routers (IR500) supply RF mesh connectivity to IPv4 and serial IoT devices (for example, recloser control, cap bank control, voltage regulator controls, and other remote terminal units).
Note
CGRs, IR800s, IR500s, and other types of Cisco Resilient Mesh endpoints (RMEs) can coexist on a network, but cannot be in the same device group. See Configuring Devices in the Managing Devices chapter.
-
Cisco 800 Series Access Points are integrated with IR800s . These devices are referred to as routers in this document; and identified by product ID (for example, AP800). You can use IoT FND to manage the following AP800 models:
-
AP803 embedded in IR829
-
-
Cisco Aggregation Services Routers (ASR) 1000 series, Cisco Integrated Services Routers (ISR) 3900 series, ISR 4300, ISR 4400, and Cisco 8000 Series Routers are referred to as head-end routers or HERs in this document.
Table 1. PIDs Supported for Cisco 8000 Series Routers Device Type PID Category C8000
C8500L-8S4X
Head-End Routers
C8000
C8000V
Head-End Routers
-
Cisco IPv6 RF (radio frequency) and PLC (power line communications).
-
The IP 67-rated Cisco Catalyst IR8100 Heavy-Duty Series routers is a modular, secure, rugged and outdoor router that is suitable for harsh physical environments. It has multiple WAN (LTE, LTE-Advanced, LTE Advanced Pro, 5G Sub-6GHz1, RJ45/SFP Ethernet) and storage options. The router supports wireless and wired connectivity such as 5G, public, or private LTE, Wi-SUN, LoRaWAN, and has more connectivity options making it more adaptable. It runs on Cisco IOS XE and Cisco IOS XE provides both autonomous and controller (SD-WAN) mode support. In IoT FND, you can find the following IR8100 models:
-
IR8140H-K9
-
IR8140H-P-K9
-
-
Cisco Catalyst IR1800 Rugged Series Routers are secure, 5G routers designed with a high level of modularity that supports private LTE, FirstNet, Wi-Fi6 and Gigabit Ethernet. These routers offer enterprise-grade security from the hardware to the network communications all the way to the industrial assets. The routers are powered by Cisco IOS® XE, Cisco’s fully programmable next-generation operating system. Automotive certifications and features such as Controller Area Network (CAN) bus support, dead reckoning and Global Navigation Satellite System (GNSS), and ignition power management make it ideal for secure, reliable connectivity in transit and public safety applications.
IoT FND supports the following IR1800 models:
-
IR1821-K9
-
IR1831-K9
-
IR1833-K9
-
IR1835-K9
-
IoT FND typically resides in the utility control center with other utility head-end operational systems, such as an AMI head end, distribution management system, or outage management system. IoT FND features enterprise-class fault, configuration, accounting, performance, and security (FCAPS) functionality, as defined in the Open Systems Interconnection (OSI) model.
The Cisco IoT FND North Bound Application Programmable Interface (NB API) allows various utility applications like DMS, OMS, or MDM to pull appropriate, service-specific data for distribution grid information, outage information, and metering data from a shared, multi-server communication network infrastructure. For more information about the Cisco IoT FND North Bound API, see the North Bound API User Guide for Cisco IoT Field Network Director, Release 4.x for your IoT FND installation.
The NB API can send events using HTTPS. NB API clients must subscribe to IoT FND by providing a valid HTTPS URL to send events. IoT FND accepts all SSL and handshake certificates that are published by the NB API client (the event consumer) while making the secure connection.
Cisco IoT FND Features and Capabilities
- Configuration Management — Cisco IoT FND facilitates configuration of a large number of Cisco CGRs, Cisco ISRs, Cisco IRs, Cisco ASRs, C8000, and mesh endpoints. Use Cisco IoT FND to bulk-configure devices by placing them into configuration groups, editing settings in a configuration template, and then pushing the configuration to all devices in the group.
-
Device and Event Monitoring — Cisco IoT FND displays easy-to-read tabular views of extensive information that is generated by devices, allowing you to monitor your network for errors. Cisco IoT FND provides an integrated Geographic Information System (GIS) map-based visualization of FAN devices such as routers and smart meters.
-
Firmware Management — Cisco IoT FND serves as Firmware Management a repository for Cisco CGR, Cisco ISR, Cisco IR, and mesh endpoint firmware images. Use Cisco IoT FND to upgrade the firmware running on groups of devices by loading the firmware image file onto the Cisco IoT FND server, and then uploading the image to the devices in the group. Once uploaded, use IoT FND to install the firmware image directly on the devices. In release 3.0.1-36 and later, a Subnet List view on the Firmware Upgrade page for Mesh Endpoints lets you filter and view subnets by PAN identifier (PAN ID) and Group (details include number of nodes within a group, hops away from the router and operational status). A subnet progress histogram has also been added.
-
OS Migration — The CG-OS to IOS migration is supported until release 4.7.x.
-
Zero Touch Deployment — This ease-of-use feature automatically registers (enrolls) and distributes X.509 certificates and provisioning information over secure connections within a connected grid network.
-
Tunnel Provisioning — Protects data exchanged between Cisco ASRs/C8000 and Cisco CGRs, Cisco ISRs and Cisco IRs, and prevents unauthorized access to Cisco CGRs, to provide secure communication between devices. Cisco IoT FND can execute CLI commands to provision secure tunnels between Cisco CGRs, Cisco ISRs and Cisco IRs and Cisco ASRs/Cisco 8000. Use IoT FND to bulk-configure tunnel provisioning using groups.
-
IPv6 RPL Tree Polling — The IPv6 Routing Protocol for Low-power and Lossy Networks (RPL) finds its neighbors and establishes routes using ICMPv6 message exchanges. RPL manages routes based on the relative position of the mesh endpoints to the CGR that is the root of the routing tree. RPL tree polling is available through the mesh nodes and CGR periodic updates. The RPL tree represents the mesh topology, which is useful for troubleshooting. For example, the hop count information received from the RPL tree can determine the use of unicast or multicast for the firmware download process. IoT FND maintains a periodically updated snapshot of the RPL tree.
-
Dynamic Multipoint VPN and FlexVPN — For Cisco IR800 devices, DMVPN and FlexVPN do not require IoT FND to apply device-specific tunnel configuration to the HER during tunnel provisioning. HER tunnel provisioning is only required for site-to-site VPN tunnels.
-
Embedded Access Point (AP) Management — IoT FND provides management of embedded APs on C819 and IR829 routers.
-
Guest OS (GOS) Support — For Cisco IOS CGR 1000 and IR800 devices that support Guest OS, IoT FND allows approved users to manage applications running on the supported operating systems. IoT FND supports all phases of application deployment, and displays application status and the Hypervisor version running on the device.
-
Device Location Tracking — For CGR 1000, IR1101, IR800, N2450, and IR8100 devices, IoT FND displays real-time location and device location history. Ensure that you enable the router GPS tracking option for this feature.
-
Software Security Module (SSM) — This is a low-cost alternative to the Hardware Security Module (HSM), and is used for signing CSMP messages sent to meters and IR500 devices.
-
Customer Certificates — Cisco IoT FND allows you to use your own CA and ECC-based certificates to sign smart meter messages.
-
Diagnostics and Troubleshooting — The IoT FND rule engine infrastructure provides effective monitoring of triage-based troubleshooting. Device troubleshooting runs on-demand device path trace and ping on any CGR 1000, IR800, Cisco 5921 Embedded Services Router (C5921), range extender, gateway, or meter (mesh endpoints).
-
High Availability — To ensure uninterrupted network management and monitoring, you can deploy the Cisco IoT FND solution in a High Availability (HA) configuration. By using clusters of load-balanced IoT FND servers and primary and standby IoT FND databases, Cisco IoT FND constantly monitors the health of the system, including connectivity within clusters and server resource usage. If a server cluster member or database becomes unavailable or a tunnel fails, another takes its place seamlessly. Additionally, you can add reliability to your IoT FND solution by configuring redundant tunnels between a Cisco CGR and multiple Cisco ASRs/C8000.
-
Power Outage Notifications — Mesh Endpoints (MEs) implement a power outage notification service to support timely and efficient reporting of power outages. In the event of a power outage, MEs perform the necessary functions to conserve energy and notify neighboring nodes of the outage. Routers relay the power outage notification to IoT FND, which then issues push notifications to customers to relate information on the outage.
-
Resilient Mesh Upgrade Support — Over-the-air software and firmware upgrades to field devices such as Cisco CGRs and Resilient Mesh Endpoints (RMEs) (for example, AMI meter endpoints).
-
Audit Logging — Logs access information for user activity for audit, regulatory compliance, and Security Event and Incident Management (SEIM) integration. This simplifies management and enhances compliance by integrated monitoring, reporting, and troubleshooting capabilities.
-
North Bound APIs — Eases integration of existing utility applications such as outage management system (OMS), meter data management (MDM), trouble-ticketing systems, and manager-of-managers.
-
Role-Based Access Controls — Integrates with enterprise security policies and role-based access control for AMI network devices.
-
Event and Issue Management — Fault event collection, filtering, and correlation for communication network monitoring. IoT FND supports a variety of fault-event mechanisms for threshold-based rule processing, custom alarm generation, and alarm event processing. Faults display on a color-coded GIS-map view for various endpoints in the utility network. This allows operator-level custom fault-event generation, processing, and forwarding to various utility applications such as an outage management system. Automatic issue tracking is based on the events collected.
IoT FND Architecture
Zero Touch Deployment Architecture provides a high-level view of the systems and communication paths that exist in a typical utility company operating on a Cisco CGR connected grid network in which Zero Touch Deployment is in use.
For Cisco IOS CGRs, we recommend a tunnel configuration using FlexVPN.
For IR800s, we recommend using Dynamic Multipoint VPN (DMVPN) or FlexVPN.
In this example, the firewall provides separation between those items in the utility company public network (DMZ) and its private network.
The utility company private network shows systems that might reside behind the firewall such as the Cisco IoT FND, the Oracle database server, the Cisco IoT FND North Bound API, the DHCP server, and the Certificate Authority (CA). The Cisco IoT FND Tunnel Provisioning Server proxy (TPS proxy) and Registration Authority (RA) might be located in the DMZ.
After installing and powering on the Cisco CGR, it becomes active in the network and registers its certificate with the RA by employing the Simple Certificate Enrollment Protocol (SCEP).
The Registration Authority (Integrated Service Router (ISR) in Zero Touch Deployment Architecture), functioning as a Certificate Authority (CA) proxy, obtains certificates for the Cisco 1000 Series Connected Grid Router (CGR1240 and CGR1120). The Cisco CGR then sends a tunnel provisioning request over HTTPS to the TPS proxy that forwards it to IoT FND.
Cisco IoT FND manages collection of all information necessary to configure a tunnel between Cisco CGRs and the head-end router ( Cisco 1000 Series Aggregation Services Routers).
Main Components of IoT FND Solution
| Component | Description |
|---|---|
|
IoT FND Application Server |
This is the heart of IoT FND deployments. It runs on an RHEL server and allows administrators to control different aspects of the IoT FND deployment using its browser-based graphical user interface. IoT FND HA deployments include two or more IoT FND servers that are connected to a load balancer. |
|
NMS Database |
This Oracle database stores all information that is managed by your IoT FND solution, including all metrics received from the MEs and all device properties such as firmware images, configuration templates, logs, event information, and so on. |
|
Software Security Module (SSM) |
This is a low-cost alternative to the Hardware Security Module (HSM), and is used for signing CSMP messages sent to meters and IR500 devices. |
|
TPS Proxy |
Allows routers to communicate with IoT FND when they first start up in the field. After IoT FND provisions tunnels between the routers and HER (ASRs/C8000), the routers communicate with IoT FND directly. |
|
Load Balancer |
The load balancer distributes traffic among the IoT FND servers in your network. You can employ a load balancer in your network within a Zero Touch Deployment (ZTD) architecture to provide High Availability (HA). IoT FND uses the BIG-IP load balancer from F5. |
High Availability and Tunnel Redundancy
The example in Zero Touch Deployment Architecture is of a single-server deployment with one database and no tunnel redundancy. However, you could take advantage of Cisco IoT FND HA support to deploy a cluster of Cisco IoT FND servers connected to a load balancer, as shown in IoT FND Server and Database HA. The load balancer sends requests to the servers in a round-robin fashion. If a server fails, the load balancer keeps servicing requests by sending them to the other servers in the cluster.
You could also deploy a standby Cisco IoT FND database to provide another layer of high availability in the system with minimal data loss.
To provide tunnel redundancy, IoT FND allows you to create multiple tunnels to connect a CGR to multiple ASRs/C8000, as shown in IoT FND Tunnel Redundancy.
For more information about HA, see Database High Availability.
List of Standard Ports Used in IoT FND
The table provides the list of standard ports used in IoT FND solution.
| Service | Port |
|---|---|
|
GUI |
443 |
|
FND Demo mode |
80 |
|
Tunnel Provisioning |
9120 |
|
TPS |
9122 |
|
FAR |
9125 |
|
CG-MESH (CSMP) |
61624 |
|
CG-MESH (CSMP CoAP version 18) |
61628 |
|
CG-MESH (Outage) |
61625 |
|
CG-MESH (Restoration) |
61626 |
|
Oracle DB Server |
1522 |
|
PostGreSql DB Server |
5432 |
|
Influx |
8086 |
|
Kapacitor |
9092 |
|
WSMA (for IOS-XE) |
443 |
|
WSMA (for Classic IOS) |
8443 |
|
RADIUS (for authentication) |
1812 |
|
RADIUS (for accounting) |
1813 |
|
FND-RA |
61629 |
|
EST Proxy |
6789 |
|
Registration + Periodic |
9121 |
|
Bandwidth Op Mode |
9124 |
|
PnP — HTTP |
9125 |
|
Web Sockets — Device Communication |
9121 |
|
LwM2M |
5683 |
|
DB Replication for HA |
1622 |
|
DHCP IPv4 |
67 |
|
DHCP IPv6 |
547 |
|
SSH |
22 |
|
NTP Server |
123 |
|
SNMP (for polling) |
161 |
|
SNMP (for notifications) |
162 |
|
Syslog service |
514 |
|
SSM Server |
8445 |
Resilient Mesh Endpoints
The Cisco Field Area Network (FAN) solution brings the first multi-service communications infrastructure to the utility field area network. It delivers applications such as AMI, DA, and Protection and Control over a common network platform.
Advanced meter deployments follow a structured process designed to match the right solution to the needs of the utility company. This process moves in phases that require coordination between metering, IT, operations, and engineering. The first phase for most utilities is identification of goals, followed by analysis of data needs, and business processes. After an evaluation of the business case is complete and a technology chosen, system implementation and validation complete the process.
Once the utility company moves past the business case into system implementation, unforeseen complications can sometimes slow or delay a deployment. The true value of a plug-and-play system is that it saves cost and improves the return on investment by allowing the benefits of advanced metering to be realized sooner.
The features that enable a true plug-and-play RF or PLC mesh network system include:
-
Self-initializing endpoints: CGRs automatically establish the best path for communication through advanced self-discovery – meters and infrastructure deploy without programming.
-
Scalability: This type of network enables pocketed deployments where each Cisco IoT FND installation can accept up to 10 million meters/endpoints. Large capacity enables rapid, multi-team deployments to occur in various parts of the targeted AMI coverage area, while saving infrastructure and communication costs.
In a true mesh network, metering and range extender devices communicate to and through one another and decide their own best links, forming the RF Mesh Local Area Network (RFLAN) or PLC LAN. These ME devices become the network and possess dynamic auto-routing functions that eliminate the need for dedicated repeater infrastructure or intermediate (between endpoint and collector) tiered radio relay networks. The result is a substantial reduction in dedicated network infrastructure as well as powerful and more flexible fixed-network communication capability.
Range extenders are installed by the utility company to strengthen mesh coverage and provide redundancy, supplementing network reliability in difficult environmental settings such as dense urban areas where buildings obstruct the normal mesh signal propagation, or in low-meter-density geographically sparse regions and RF-challenged areas. A range extender automatically detects and connects to the mesh after installation or outage recovery, and then provides an alternate mesh path.
In a normal deployment scenario, these MEs form a stable RFLAN or PLC LAN network the same day they are deployed. Once the collector is installed, placing MEs throughout the deployment area is as simple as changing out a meter. MEs form a network and begin reporting automatically.
Mesh endpoints send and receive information. A two-way mesh system allows remote firmware upgrades, as well as system settings changes and commands for time-of-use periods, demand resets, and outage restoration notifications. Not having to physically “touch the meter” is a major value, especially when entering the advanced demand response metering domain that requires time-of-use (TOU) schedule changes and interval data acquisition changes to meet specific client needs. These commands can be sent to groups or to a specific ME. Meter commands can be scheduled, proactive, on-demand, or broadcast to the entire network.
Communication between the data center/network operations center (NOC) and the collector is accomplished by widely available and cost-efficient mass marketed TCP/IP-based public wide area network (WAN) or with the utility company-owned WAN. The flexibility and open standard public WAN architectures currently available and in the future create an environment that allows continued ongoing cost reduction and future options, without being tied into one type of connectivity over the life of the asset. It is best if the AMI system avoids using highly specialized WAN systems.
After deployment is complete, the system can transmit scheduled hourly (and sub hourly) data to support utility applications such as billing reads, advanced demand response initiatives, load research, power quality, and transformer asset monitoring.
Easy access and reliable on-demand capability allow the utility to perform grid diagnostics and load research system-wide or for selected groups of meters. Other standard features support outage management, tamper detection, and system performance monitoring.
|
Feature Name |
Release Information |
Description |
|
Enhance DB queries to support scaled mesh deployment |
IoT FND 4.8 |
The Oracle DB is scaled up to 8,000/ 8,000,000 routers/ endpoints. Under page, the CSMP optimization settings are introduced to configure the timeout in order to acquire lock when processing CSMP messages. The CSMP optimization setting is available only for Oracle DB set up and not for PostgreSQL DB setup. |
Grid Security
Designed to meet the requirements of next-generation energy networks, Cisco Grid Security solutions take advantage of our extensive portfolio of cybersecurity and physical security products, technologies, services, and partners to help utility companies reduce operating costs while delivering improved cybersecurity and physical security for critical energy infrastructures.
Cisco Grid Security solutions provide:
-
Identity management and access control: Secure utility facilities, assets, and data with user authentication and access control are custom-built for grid operations.
-
Threat defense: Build a layered defense that integrates with firewall, VPN, intrusion prevention, and content security services to detect, prevent, and mitigate threats.
-
Data center security: Turn network, computing, and storage solutions into a secure, shared pool of resources that protects application and data integrity, secures communications between business processes and applications within the utility, and secures connectivity to external resources such as providers of renewable energy.
-
Utility compliance: Improve risk management and satisfy compliance and regulatory requirements such as NERC-CIP with assessment, design, and deployment services.
-
Security monitoring and management: Identify, manage, and counter information security threats and maintain compliance through ongoing monitoring of cyber events.
Feedback