OPS.1.1.6 Software Testing and Approvals

Description

Introduction

The use of IT in institutions requires that automated data processing functions as error-free as possible, since the individual results can in most cases no longer be checked. Therefore, software of any kind must be verified before deployment through software tests. In these tests, it must be demonstrated that the software reliably provides the required functions and also has no undesirable side effects. The subsequent approval of the software by the responsible organizational unit grants the basic permission to use the software productively in the institution. At the same time, this organizational unit assumes responsibility for the IT procedure supported by the software.

Software can be tested at different points in its lifecycle. Thus software tests may become necessary during development, before approval for productive use, or in the course of patch and change management. This applies to both custom software and standardized software. Regression tests play a special role here, because even if only minor aspects of the software are changed, there is a possibility that this will affect entirely different aspects and functions of the software. Regression tests specifically check software for these effects.

This building block describes the testing and approval process for any type of software. The testing and approval process is characterized by the fact that it can be traversed multiple times depending on the result.

Objective

By implementing this building block, the institution ensures that the software used meets the technical and organizational requirements as well as the protection needs of the entire institution or individual organizational units. An essential sub-aspect is that safety-critical software is systematically and methodically checked for existing vulnerabilities.

Scope and Modeling

The building block OPS.1.1.6 Software Testing and Approvals is to be applied once to the information domain.

While building block CON.8 Software Development addresses the software development process and the software tests contained therein that are necessary during the development process, this building block describes the specific requirements placed on a testing and approval management system. This testing and approval management relates not exclusively to software developed in-house or on behalf of the customer, but also to testing and approving any software such as that described in APP.6 General Software, as well as all other software products of the APP Applications layer.

Software tests can also become part of patch or change management or software development. Corresponding requirements are more closely specified in building block OPS.1.1.3 Patch and Change Management and CON.8 Software Development.

Threat Landscape

Since IT-Grundschutz building blocks cannot address individual information domains, typical scenarios are used to illustrate the threat landscape. The following specific threats and vulnerabilities are of particular importance for the building block OPS.1.1.6 Software Testing and Approvals.

Software Tests with Production Data

If software tests are performed with production data, security problems can arise as a result. In particular, confidential production data could be viewed by unauthorized employees or third parties tasked with the respective software test. If testing is performed with the “original” production data rather than with copies of the data, these could be inadvertently changed or deleted.

Software tests conducted in the productive environment could massively disrupt overall operations. Malfunctions of the software to be tested can also affect other applications and IT systems, which are thereby also disrupted. In addition, software testers deliberately test the software at its limits with the intent of exposing possible errors. This in turn increases the risk of overall operations being disrupted.

Missing or Insufficient Testing Procedure

If new software is not tested or only insufficiently tested and approved without installation instructions, errors in the software can go undetected. It is also possible that required and mandatory installation parameters are not identified or not observed as a result.

These software or installation errors resulting from a missing or insufficient software testing procedure can significantly jeopardize the institution’s IT Operations. For example, important data can be lost when a software update is applied.

Missing or Insufficient Approval Procedure

A missing or insufficient approval procedure can lead to software being used that has not been accepted by the specialist side. In this way, the software may contain functions it should not have, functions may not work as desired, or required functions may be missing. The software may also be incompatible with other applications.

Missing or Insufficient Documentation of Tests and Test Results

Software can generally be approved once all tests have been performed and no deviations have been found. However, if the documentation of software tests is incomplete, it cannot be determined afterwards what was tested. If identified software errors or missing functions have been insufficiently documented and therefore not considered during approval, the production data to be processed can be inadvertently deleted or changed due to these deviations. Other IT systems and applications can also be disrupted.

Missing or Insufficient Documentation of Approval Criteria

If approval criteria are not clearly communicated, this can lead to software being approved prematurely or no approval being granted even though it could have been. As a result, on the one hand versions with undetected software errors could be approved that disrupt productive operations. On the other hand, missing or insufficient documentation of approval criteria can lead to project delays and resulting financial damage.

Requirements

The following are the specific requirements of building block OPS.1.1.6 Software Testing and Approvals. The Information Security Officer (ISO) is responsible for ensuring that all requirements are met and verified in accordance with the established security concept. The ISO MUST always be involved in strategic decisions.

The IT-Grundschutz Compendium additionally defines further roles. They should be staffed insofar as this is reasonable and appropriate.

Exactly one role should be Primarily responsible. Beyond that, there may be Additional responsibilities. If one of these additional roles is primarily responsible for fulfilling a requirement, this role is listed in square brackets after the requirement heading. The use of singular or plural says nothing about how many persons should fill these roles.

Basic Requirements

The following requirements MUST be fulfilled as a priority for this building block.

OPS.1.1.6.A1 Planning of Software Tests (B)

The framework conditions for software tests MUST be established within the institution before the tests, in accordance with the protection needs, organizational units, technical possibilities, and test environments. The software MUST be tested on the basis of the requirements of the requirements catalog for the software. If there is also a specification document, this MUST additionally be taken into account.

The test cases MUST be selected such that they verify as representatively as possible all functions of the software. In addition, negative tests SHOULD also be considered that check whether the software does not contain any undesired functions.

The test environment MUST be selected such that it covers as representatively as possible all device models and operating system environments used in the institution. It SHOULD be tested whether the software is compatible and functional with the operating systems used in the existing configurations.

OPS.1.1.6.A2 Performing Functional Software Tests (B) [Testers]

Functional software tests MUST be used to verify the proper and complete functioning of the software. The functional software tests MUST be performed such that they do not affect productive operations.

OPS.1.1.6.A3 Evaluation of Test Results (B) [Testers]

The results of the software tests MUST be evaluated. A target-actual comparison with defined requirements SHOULD be performed. The evaluation MUST be documented.

OPS.1.1.6.A4 Software Approval (B) [Specialist Responsible Persons]

The responsible organizational unit MUST approve the software once the software tests have been successfully performed. The approval MUST be documented in the form of an approval declaration.

The approving organizational unit MUST verify that the software was tested in accordance with the requirements. The results of the software tests MUST match the previously defined expectations. It MUST also be verified that the legal and organizational requirements were met.

OPS.1.1.6.A5 Conducting Software Tests for Non-Functional Requirements (B) [Testers]

Software tests MUST be performed that verify whether all essential non-functional requirements are met. In particular, security-specific software tests MUST be performed when the application has security-critical functions. The test cases performed as well as the test results MUST be documented.

OPS.1.1.6.A11 Use of Anonymized or Pseudonymized Test Data (B) [Data Protection Officers, Testers]

If production data containing protected information is used for software tests, this test data MUST be adequately protected. If this data contains personal information, this data MUST be at least pseudonymized. If possible, the test data with personal reference SHOULD be completely anonymized. If a personal reference could be derived from the test data, the data protection officer and possibly the employee representation MUST be consulted.

Standard Requirements

Together with the basic requirements, the following requirements correspond to the state of the art for this building block. They SHOULD generally be fulfilled.

OPS.1.1.6.A6 Structured Briefing of Software Testers (S) [Specialist Responsible Persons]

Software testers SHOULD be informed by specialist responsible persons about the types of tests to be performed and the areas of software to be tested. Furthermore, software testers SHOULD be informed about the use cases and any further requirements of the software.

OPS.1.1.6.A7 Personnel Selection of Software Testers (S) [HR Department, Specialist Responsible Persons]

Special selection criteria SHOULD be considered when selecting software testers. Software testers SHOULD have the required professional qualifications.

If custom software is reviewed at source code level, the testers SHOULD have sufficient expertise in the programming language to be tested and the development environment. The source code SHOULD NOT be reviewed exclusively by testers who were also involved in creating the source code.

OPS.1.1.6.A8 DISCONTINUED (S)

This requirement has been discontinued.

OPS.1.1.6.A9 DISCONTINUED (S)

This requirement has been discontinued.

OPS.1.1.6.A10 Creation of an Acceptance Plan (S)

An acceptance plan SHOULD document the types of tests to be performed, test cases, and expected results. The acceptance plan SHOULD also include the approval criteria. A procedure SHOULD be defined for the situation where an approval is rejected.

OPS.1.1.6.A12 Conducting Regression Tests (S) [Testers]

If software has been changed, regression tests SHOULD be performed. It SHOULD be checked whether existing security mechanisms and settings have been inadvertently changed by the update. Regression tests SHOULD be performed completely and SHOULD also include extensions and tools. If test cases are omitted, this SHOULD be justified and documented. The test cases performed and the test results SHOULD be documented.

OPS.1.1.6.A13 Separation of Test Environment from Productive Environment (S)

Software SHOULD only be tested in a test environment designated for this purpose. The test environment SHOULD be operated separately from the productive environment. The architectures and mechanisms used in the test environment SHOULD be documented. Procedures SHOULD be documented for how to proceed with the test environment after completion of the software test.

OPS.1.1.6.A15 Review of Installation and Associated Documentation (S) [Testers]

The installation of the software SHOULD be reviewed in accordance with the regulations for software installation and configuration (see building block APP.6 General Software). If available, the installation and configuration documentation SHOULD also be reviewed.

Requirements for High Protection Needs

The following are exemplary proposals for requirements for this building block that go beyond the level of protection corresponding to the state of the art. The proposals SHOULD be considered for high protection needs. The specific determination is made within the framework of an individual risk analysis.

OPS.1.1.6.A14 Conducting Penetration Tests (H) [Testers]

For applications or IT systems with high protection needs, penetration tests SHOULD be performed as a test method. A concept for penetration tests SHOULD be created. The penetration test concept SHOULD document not only the test methods to be used but also the success criteria.

The penetration test SHOULD be carried out in accordance with the framework conditions of the penetration test concept. Security vulnerabilities found by the penetration test SHOULD be classified and documented.

OPS.1.1.6.A16 Security Vetting of Testers (H)

If testers must access particularly protected information, the institution SHOULD obtain evidence of their integrity and reputation. If this involves classified information, software testers SHOULD undergo a security check in accordance with the Security Vetting Act (SÜG). The ISO SHOULD involve the institution’s classified information protection officers or security representatives for this purpose.

Additional Information

Good to Know

The International Organization for Standardization (ISO) provides requirements for secure system development in ISO/IEC 27001:2013, Annex A.14 “System Acquisition, Development and Maintenance,” which also requires a testing and approval process, as well as requirements for test data itself. In addition, ISO has published the standard ISO/IEC 29119-2:2013 “Software and systems engineering - Software testing - Part 2: Test processes, International Organization for Standardization,” which comprehensively addresses requirements for software tests.

The BSI has published the study “Implementation Concept for Penetration Tests,” which can be used as a basis for penetration tests, as well as the BSI guidelines for developing secure web applications, which also include software tests.

The Information Security Forum (ISF) presents in “The Standard of Good Practice for Information Security” aspects of testing and approving for all relevant requirements (areas).

The National Institute of Standards and Technology provides guidelines for software testing in NIST Publication 800-53 in SA 11 Developer Security Testing and Evaluation.

The book “The Art of Software Testing” by Glenford J. Myers, Corey Sandler, Tom Badgett, can be consulted for software tests.

The Common Vulnerability Scoring System can be used as a scoring system for classifying the severity of a security vulnerability and thus presenting the results of software tests with regard to information security.