IND.3.2 Remote Maintenance in Industrial Environments

Description

Introduction

The operational technology (OT) of an institution often has a decentralized infrastructure. Various areas of OT can be geographically far apart from one another. In addition, Industrial Control Systems (ICS) typically consist of a large number of products from various manufacturers — i.e., different ICS components and IT systems for OT applications. Therefore, the operational technology of an institution generally requires numerous remote maintenance access points.

These remote maintenance access points are frequently individual solutions in the form of individually assembled hardware and software components. As a result, a large variety of different technologies is used for OT remote maintenance. The life cycles of the remote maintenance solutions generally correspond to those of the products being accessed — meaning OT remote maintenance solutions may be used for considerably longer periods than is common in IT. A wide variety of access points, services, and interfaces exist in parallel. These interfaces communicate using a diverse range of protocols.

Some plant components in OT are also realized as closed units by manufacturers (so-called package units). These plant components frequently contain multiple decentralized access points for remote maintenance, which the manufacturers have integrated from the outset to be operational for their own access.

Remote maintenance access points in industrial environments are generally used by OT Operations and maintenance personnel to configure, monitor, maintain, and repair OT components. Only in exceptional cases — for example, during disruptions — do other employees also use OT remote maintenance access points.

OT components are accessed via remote maintenance from various institutions. Not only the internal personnel of the operator, but also external personnel from manufacturers, integrators, and service providers use remote maintenance access points.

In principle, a remote maintenance access occurs into that security segment of an OT network in which the remote maintenance service is provided. The remote maintenance service then communicates from this segment to the target system being maintained — for example, an ICS component.

Objective

The objective of this building block is to ensure the information security of remote maintenance in industrial environments.

Scope and Modeling

The building block IND.3.2 Remote Maintenance in Industrial Environments MUST be applied once to the entire OT of an institution as soon as this OT contains remote maintenance capabilities. Whether additional requirements for remote maintenance in industrial environments need to be defined that cannot be listed in general terms in this building block depends on the respective areas of application of the ICS. Depending on the area of application, the measures to be implemented to fulfill the requirements may also differ.

To create an IT-Grundschutz model for a specific information domain, the entirety of all building blocks MUST in principle be considered. In general, multiple building blocks are applied to a given topic or target object.

This building block addresses:

• the specific aspects of OT remote maintenance that go beyond the general administration of network components and IT systems via remote maintenance, and
• the specific aspects of OT remote maintenance that deviate from the general administration of network components and IT systems via remote maintenance.

The following content is also relevant and is addressed elsewhere:

• the general administration of network components and IT systems via remote maintenance by IT Operations in office and building IT (see OPS.1.2.5 Remote Maintenance) (The relevant requirements MUST also generally be fulfilled in OT.)
• the general aspects of network architecture and design, as well as segmentation (see NET.1.1 Network Architecture and Design)
• the individual aspects of the hardware and software components that make up the respective remote maintenance solution — e.g., network components, server and client systems, OT applications (The applicable building blocks MUST be modeled for each of these.)
• the general aspects of logging (see OPS.1.1.5 Logging)
• the general aspects of outsourcing (see OPS.2.3 Use of Outsourcing)
• the general aspects of cloud-based remote maintenance solutions (see OPS.2.2 Cloud Usage)
• the general aspects of web-based remote maintenance solutions (see APP.3.1 Web Applications and Web Services)
• the general aspects of securing ICS at the operator level (see IND.1 Process Control and Automation Technology and IND.2.7 Safety Instrumented Systems)

This building block does not address:

• the observation and operation of ICS at the process level, nor
• controlling access via remote maintenance to ICS — e.g., starting or stopping plants. Such access can in principle cause personal injury and property damage on site!

Threat Landscape

Since IT-Grundschutz building blocks cannot address individual information domains, typical scenarios are used to describe the threat landscape. The following specific threats and vulnerabilities are of particular relevance for the building block IND.3.2 Remote Maintenance in Industrial Environments.

Incompletely Documented Remote Maintenance Access Points in OT

The remote maintenance access points in the OT of an institution are generally numerous and diverse. In addition, a large number of internal and external persons access them. The management of remote maintenance access points in industrial environments is therefore fundamentally complex and error-prone. There is a greater risk than in office and building IT that the various access points are inadequately recorded and documented — i.e., are not verifiable.

If unknown, open OT remote maintenance access points are not documented, operators cannot prevent access. As a result, unauthorized users can directly influence the physical processes of an ICS, which can lead — for example — to malfunctions of individual components, to the shutdown of an entire production plant, or even to endangerment of the lives and physical safety of employees on site. If plants including decentralized components for remote maintenance are realized as an integrated system (package unit) by the manufacturer, they are initially not subject to sufficient control by the operators. Operators must record each OT remote maintenance component individually and often actively reconfigure it.

If some OT remote maintenance access points are only recorded and documented locally — for example — there is also the risk that necessary changes to individual access points are not made, or that changes that have been made are not traceable.

Insufficient Availability Due to Dependencies on Office and Building IT

ICS are partly reliant on a fully uninterrupted flow of information. This applies particularly to all real-time data streams. Even very brief interruptions to the availability of data that can be tolerated in office and building IT can be critical in an ICS. Furthermore, dependencies on networks, services, or IT systems can create security vulnerabilities. This is often overlooked during planning when the OT and the office and building IT of an institution communicate with one another.

If dependencies between OT and office and building IT are not taken into account and security vulnerabilities are consequently not closed, security incidents can quickly spread to the entire OT.

If central components and services of the office and building IT are used for remote maintenance of the OT (cross-functional administration), coordination errors can also occur between office and building IT and OT. As a result, the resolution of critical errors in the ICS of the OT can be delayed. One example is a central VPN gateway for remote access that is provided by an organizational unit outside the OT and is not operated in a coordinated manner, so that it is not available quickly enough when needed. The shutdown of a plant can then be considerably prolonged.

Insufficient Rules for the Use of OT Remote Maintenance Access Points

OT remote maintenance access points have a very large potential user base outside the operating institution. At the same time, seamless availability of real-time data in an ICS is essential, whereas the availability of data in office and building IT is generally somewhat less time-critical. General rules — such as those for how remote maintenance access points for office and building IT are used — are often unsuitable for OT or leave too much room for interpretation.

If the use of OT remote maintenance access points is only insufficiently regulated contractually, there are no clear specifications for how the user base of each individual access point is to be restricted. Operators then cannot control who uses their OT remote maintenance access points. For example, access credentials at integrators, manufacturers, and maintenance service providers can be passed on to users who are unknown to the operators. This is not tolerable for an ICS.

If restrictive usage rules specifically for OT are lacking, operators also cannot adequately control how OT remote maintenance access points are used. In the event of a (security) incident, forensic analysis is then made more difficult, for example, because causes cannot be identified quickly enough. This can prolong costly production downtime.

If OT remote maintenance access points are jointly provided and operated with or by the office and building IT, and this shared use of the access point hardware or software is not clearly regulated, inconsistent configurations can result. These then only fulfill the security requirements of the office and building IT, but not the deviating or additional security requirements of the OT.

Insufficient Human Oversight of OT Remote Maintenance Sessions

The data and configuration settings within an ICS, as well as persons, property, and production processes on site, can be endangered by all access from other zones that is not or only insufficiently controlled by internal personnel on site. In particular, the personnel of external maintenance service providers, integrators, and manufacturers possess the necessary specialized knowledge of the plant, machine, or component being maintained, but do not know what harmful effects the specific remote maintenance session can have on the overall system.

If the course and content of OT remote maintenance sessions cannot be adequately controlled by operators, configuration errors with high endangerment potential as well as unintentional or intentional misconduct by maintenance personnel cannot be detected and traced quickly enough. This makes it possible for real-time data streams to fail or be manipulated, or for malware to spread undetected across the entire OT. This can result in injury to or endangerment of the lives of operating personnel on site, significant financial damage from production outages through to the destruction of entire plants, and the disclosure of trade secrets. The integrity of manufactured products can also be impaired — for example, through the manipulation of recipes, resulting in the production of defective goods that may be difficult to detect. If defective products are not identified as such, this can damage the institution’s reputation.

One example is decentralized OT remote maintenance components within plants realized as an integrated system by manufacturers (package units). These remote maintenance access points are often designed so that manufacturers can access the plant at any time. If this occurs without coordination with the operator’s personnel on site, hazardous situations in the current production process can arise, for example.

Direct IP-Based Access Possibilities to ICS from Unsecured Zones

The data within an ICS, as well as persons, property, and production processes on site, are in principle endangered by direct IP-based access possibilities from other zones and networks. This applies, for example, due to the special patch cycles in OT, which can be considerably longer than, for example, in office IT. At the same time, remote maintenance access to an ICS is generally performed from networks that are only partially trustworthy. Such networks include not only private networks of other institutions but also other zones of the institution’s own network. Zones within the institution that are only partially trustworthy include, for example, those of the internal office and building IT or further zones of the OT network (e.g., for operations and production management with MES, ERP).

Direct access from other zones and networks to an ICS — for example, via a local connection or via VPN — can introduce malware and facilitate targeted attacks. As a result, in the OT zones to be protected, there can be dangers to life and physical safety, high financial damage due to downtime, or confidential information such as trade secrets can fall into the wrong hands.

Insecure Alternative OT Remote Maintenance Access Points for Disruptions

Particularly during disruptions, OT remote maintenance access points must be more reliable and high-performing than in office and building IT. In industrial environments, dedicated alternative remote maintenance access points for fast access are therefore set up specifically. These ensure the operational capability of the ICS even when the respective primary access point fails or is insufficiently performant to resolve critical error states at remote plants quickly enough. However, such alternative remote maintenance access points can be particularly vulnerable to attack.

If a fast alternative remote maintenance access point — for example, via mobile network — is created but does not fully meet the institution’s security requirements, attackers can more easily penetrate the OT network.

Insecure Design of OT Remote Maintenance Access Points

ICS often have a decentralized infrastructure with numerous OT remote maintenance access points from different manufacturers. Both the complex, far-reaching distribution of access points and the diverse proprietary remote maintenance solutions make it difficult for operators to adequately centrally secure all OT remote maintenance access points.

If OT remote maintenance access points are located in openly accessible areas, attackers can easily penetrate OT networks via these access points and carry out manipulations.

Even if OT remote maintenance access points are only protected by manufacturer-provided security components, and these do not meet the institution’s security requirements, attackers can easily penetrate OT networks via these access points and carry out manipulations.

Outdated Technical Design of OT Remote Maintenance Access Points

Due to the long life cycles in industrial environments, there are not infrequently OT remote maintenance solutions that have been in use for ten years or more and have security vulnerabilities due to their age. Furthermore, historically developed specific environmental conditions often exist — such as the installation of remote maintenance access points in openly accessible areas.

If OT remote maintenance access points are not adequately secured due to long life cycles, unauthorized access into an ICS and into further networks and systems inside and outside the OT is facilitated. This makes manipulations possible that can lead to endangerment of life and physical safety.

In addition, integrated remote maintenance access points in plants may have been unknown for many years. As a result of increasing dependencies on other networks and systems due to more recent technical changes, unauthorized access with critical consequences can then become increasingly likely.

Requirements

The following are the specific requirements of the building block IND.3.2 Remote Maintenance in Industrial Environments. The Information Security Officer (ISO) is responsible for ensuring that all requirements are fulfilled and reviewed in accordance with the established security concept. The ISO MUST always be involved in strategic decisions.

Additional roles are defined in the IT-Grundschutz Compendium. These SHOULD be filled insofar as this is meaningful and appropriate.

Exactly one role SHOULD be primarily responsible. There may additionally be further responsibilities. If one of these additional roles is primarily responsible for fulfilling a requirement, that role is listed in square brackets after the requirement heading. The use of singular or plural says nothing about how many people SHOULD fill these roles.

Basic Requirements

The following requirements MUST be fulfilled with priority for this building block.

IND.3.2.A1 Planning the Use of Remote Maintenance in OT (B) [Planners]

In industrial environments, a uniform, central remote maintenance concept MUST be created for all remote maintenance facilities for the entire OT of the institution. The following aspects MUST be taken into account in the OT remote maintenance concept:

• specific statutory requirements, e.g., protection of persons,
• specific requirements from plant manufacturers,
• specific requirements due to decentralized infrastructures,
• specific requirements for remote maintenance connections,
• specific requirements for the availability of remote maintenance,
• specific requirements due to environmental conditions, and
• specific requirements due to existing legacy plants.

All of these aspects MUST be agreed upon with all involved internal and external stakeholders.

All remote maintenance access points through which access to an ICS of the institution is possible MUST be recorded in a central documentation.

For newly to be procured remotely maintainable machines, the information security requirements MUST be agreed upon with the suppliers.

The goal SHOULD be standardization of the remote maintenance solutions used. Once standardized remote maintenance solutions are being planned, OT and office and building IT MUST coordinate jointly.

IND.3.2.A2 Consistent Documentation of Remote Maintenance by OT and Office and Building IT (B) [IT Operations, Maintenance Personnel]

In industrial environments, OT and office and building IT MUST jointly record and document all OT remote maintenance access points.

In particular for remote maintenance components integrated in package units, all deactivated access points MUST also be documented.

IND.3.2.A3 Regular Reviews and Exception Approvals for Existing OT Remote Maintenance Access Points (B) [IT Operations]

All plants MUST be regularly checked to verify that all their remote maintenance access points correspond to the target state — i.e., the current remote maintenance concept for the OT.

For necessary deviations from the concept, an approval process MUST be established within the OT.

IND.3.2.A4 Binding Regulations for OT Remote Maintenance by Third Parties (B)

Appropriately restrictive rules for OT remote maintenance MUST be contractually agreed with all external users — i.e., manufacturers, integrators, and maintenance service providers. These contractual rules MUST at all times ensure that external users use all OT remote maintenance access points exclusively in a controlled and coordinated manner.

Internally, it MUST be specified which activities are permissible via which remote maintenance access points by which external users. Furthermore, it MUST be specified which internal employees authorize, observe, and if necessary support remote maintenance access and activities by external parties.

In industrial environments, it MUST be ensured that persons at or in plants and machines cannot be endangered either directly or indirectly by active remote maintenance.

In particular for safety machines, the internal OT employee MUST have both organizational and technical authority over the beginning and end of remote maintenance. It MUST be contractually excluded that remote access can be established and maintained without the explicit consent of the internal OT employees.

IND.3.2.A5 Internal Coordination for OT Remote Maintenance with Office and Building IT (B) [Planners]

OT, office and building IT, and all other involved organizational units MUST establish appropriately restrictive rules for all components and interfaces that directly or indirectly enable OT remote maintenance within the institution. These internal rules MUST at all times ensure controlled and coordinated use of the respective OT remote maintenance access points. The following aspects MUST be regulated:

• Processes
• Responsibilities
• Permissions

IND.3.2.A6 Securing Every Remote Maintenance Access to OT (B) [IT Operations]

In industrial environments, the OT MUST be able to control every access to an IT system that provides a remote maintenance service for the OT. For this purpose, the access MUST be secured by at least one security component within the responsibility of the OT.

Access to remote maintenance SHOULD be standardized for all access points. Each access SHOULD be controlled and explicitly authorized using central authentication components.

If components of OT remote maintenance access points are located decentrally or are integrated into package units, these access points MUST be secured by an additional security component that is itself centrally located and not integrated into a package unit.

Standard Requirements

Together with the basic requirements, the following requirements represent the state of the art for this building block. They SHOULD generally be fulfilled.

IND.3.2.A7 Technical Decoupling of Access (S) [Planners]

Every remote access to any components in an OT zone SHOULD be decoupled. In every remote maintenance access point into the OT, an IT system SHOULD be positioned that terminates the connection before the transition into the OT target zone and establishes new, monitored, and regulated communication to the remote maintenance service.

All tools and programs required for remote maintenance SHOULD be installed and ready to run on the IT system of the remote maintenance access point and SHOULD support multi-user operation. The IT system SHOULD be a jump server or a comparable Application Layer Gateway (ALG) positioned in a dedicated security segment — for example, in a demilitarized zone (DMZ).

OT SHOULD be responsible for the IT system used to decouple access — i.e., it ideally resides in an OT DMZ.

IND.3.2.A8 Explicit Authorization of Each OT Remote Maintenance Session (S) [Employees]

Each remote maintenance session SHOULD be approved in advance by an OT employee of the operating institution who is responsible for the target system of the session. Only then SHOULD the OT employee activate the remote maintenance access point. Explicit authorization SHOULD be maintained both in case of need and during agreed maintenance windows. The authorization SHOULD generally be valid only for a limited period of time — i.e., the responsible OT employees retain authority over the timing of remote maintenance (see requirement IND.3.2.A3 Regular Reviews and Exception Approvals for Existing OT Remote Maintenance Access Points).

Furthermore, external remote maintenance access SHOULD be established exclusively from the inside outward — i.e., from within the OT network.

IND.3.2.A9 Secure File Exchange Accompanying OT Remote Maintenance (S) [Planners, IT Operations]

A secure procedure SHOULD be established for file exchange as part of OT remote maintenance — for example, configuration files, updates, or manuals. This MUST at minimum include a check for malware.

The connection between a file exchange system and the file source SHOULD NOT be automated, but instead initiated and authenticated by the institution’s OT before each file exchange. A file exchange SHOULD generally be logged.

IND.3.2.A10 Observation and Control of OT Remote Maintenance Sessions (S) [Employees]

In industrial environments, it MUST be ensured that neither persons at or in plants and machines, nor the plants or machines themselves, can be endangered either directly or indirectly by active remote maintenance. Furthermore, it MUST be ensured that active remote maintenance does not impair the production process.

If personal injury or property damage is possible, it MUST be ensured that OT employees can monitor the remote maintenance activities on site (four-eyes principle). OT employees SHOULD be able to intervene as needed and SHOULD be able to interrupt a remote maintenance session.

IND.3.2.A11 Central Management of All Accounts for OT Remote Maintenance (S) [IT Operations]

For remote maintenance access in OT, only accounts managed in a central directory service of the OT or the institution SHOULD be used.

Requirements for High Protection Needs

The following are exemplary proposals for requirements that go beyond the level of protection corresponding to the state of the art for this building block. These proposals SHOULD be considered when there are elevated protection needs. The specific determination is made within the framework of an individual risk analysis.

IND.3.2.A12 Dedicated Remote Maintenance Solution in OT (H) [Planners]

For remote maintenance in industrial environments, a dedicated OT remote maintenance solution SHOULD be used that is independent of the office and building IT. All other functions on the IT systems for OT remote maintenance — in particular also functions for administering IT systems and networks outside the OT — SHOULD be deactivated or prevented.

If maximum independence is to be achieved, a dedicated Internet connection for OT remote maintenance SHOULD also be used.

IND.3.2.A13 Logging the Content of Remote Maintenance Access in OT (H) [Planners, Data Protection Officers]

For remote maintenance of OT applications or systems, logging SHOULD be extended such that all activities are traceable without gaps and without delay. To this end, in addition to logging events and session data, the content of remote maintenance access SHOULD also be logged.

IND.3.2.A14 Technical Control of Remote Maintenance Sessions (H) [Planners, Data Protection Officers]

OT remote maintenance sessions SHOULD, in addition to IND.3.2.A10 Observation and Control of OT Remote Maintenance Sessions, be continuously regulated by a technical solution. Activities at the command level — i.e., manual and automated commands — SHOULD be technically monitored and, where necessary, automatically prevented.

In addition, sessions SHOULD be monitored across components. If technical monitoring is in place, an alarm SHOULD be triggered not only for specific rule violations but also for anomalies in usage behavior — for example, as soon as a sudden increase in communication volume is detected.

Additional Information

Good to Know

The Federal Office for Information Security provides an overview in its publication “Remote Maintenance in Industrial Environments” of how remote maintenance access points in industrial environments can be operated securely.

The Federal Office for Information Security describes the securing of industrial systems (ICS, Industrial Control Systems) in its publication “ICS Security Compendium.”