Skip to content

Latest commit

 

History

History
228 lines (161 loc) · 12.5 KB

06.quality_criteria.md

File metadata and controls

228 lines (161 loc) · 12.5 KB

6. Quality Criteria - Automation {.page_break_before}

The following sections describe the quality conventions and best practices that apply to the development, operation and integration phases of a Service with a production infrastructure for research, such as the EOSC ecosystem. These guidelines rule the Service development and operation process within the framework of the EOSC-Synergy project.

The criteria in this document complements the criteria described in the "Software Quality Assurance baseline" [@https://digital.csic.es/handle/10261/160086], while following the same pragmatic DevOps approach of automation.

6.1. Deployment

6.1.1. Automated Deployment [SvcQC.Dep] {#dep}

The automated deployment of Services implies the use of code to install and configure them in the target infrastructures. Infrastructure as Code (IaC) templates allow operations teams to treat service provisioning and deployment in a similar fashion as developers manage the software code.

Consequently, IaC enables the paradigm of immutable infrastructure deployment and maintenance, where Services are never updated, but deprovisioned and redeployed. An immutable infrastructure simplifies maintenance and enhances repeatability and reliability.

  • [SvcQC.Dep01] A production-ready Service SHOULD be deployed as a workable system with the minimal user or system administrator interaction leveraging IaC templates.

  • [SvcQC.Dep02] Any future change to a deployed Service SHOULD be done in the form of a new deployment, in order to preserve immutable infrastructures.

  • [SvcQC.Dep03] IaC SHOULD be validated by specific (unit) testing frameworks for every change being done.

    • [SvcQC.Dep03.1] IaC (unit) tests MUST be idempotent.

6.2. Dynamic testing - Black box testing

6.2.1. API Testing [SvcQC.Api] {#api01}

Web services commonly use application programming interfaces (APIs) to expose the available features to external consumers, which can be either oriented to the end-user or suitable for machine-to-machine communications.

Accurate implementation of a publicly-accessible API, is driven by a clearly defined specification. The OpenAPI Specification (OAS) [@https://www.openapis.org] provides the most suitable way to describe, compose, consume and validate APIs. The following requirements assume the presence of such an API specification.

  • [SvcQC.Api01] API testing MUST cover the validation of the features outlined in the specification (aka contract testing).

    • [SvcQC.Api01.1] Any change in the API not compliant with the OAS MUST NOT pass contract testing.

    • [SvcQC.Api01.2] The use of OAS SHOULD narrow down the applicable set of test cases to the features described in the specification, avoiding unnecessary assertions.

  • [SvcQC.Api02] API testing MUST include the assessment of the security-related criteria outlined in SvcQC.Sec section.

  • [SvcQC.Api03] API testing SHOULD involve the use of test doubles, such as mock servers or stubs, that act as a validation layer for the incoming requests.

6.2.2. Integration Testing [SvcQC.Int]

Integration testing refers to the evaluation of the interactions among coupled Services or parts of a system that cooperate to achieve a given functionality.

  • [SvcQC.Int01] Whenever a new functionality is involved, integration testing MUST guarantee the operation of any previously-working interaction with external Services.

    • [SvcQC.Int01.1] When using APIs, contract testing MUST detect any disruption in the communication between provider and consumer endpoints, through the validation of the API specification SvcQC.Api01.

  • [SvcQC.Int02] Integration testing MUST NOT rely on non-operational or out-of-the-warranty services.

  • [SvcQC.Int03] On lack of automation, ad-hoc pilot Service infrastructures and/or local testbeds MAY be used to cope with the integration testing requirements.

  • [SvcQC.Int04] In the presence of CI environments, integration tests SHOULD be suitable for the automated testing.

6.2.3. Functional tests [SvcQC.Fun] {#fun}

Functional testing is a type of black-box testing. It involves the verification of the Service identified functionality, based on requested requirements and agreed design specifications. This type of Service testing focuses on the evaluation of the functionality that the Service exposes, leaving apart any internal design analysis or side-effects to external systems.

  • [SvcQC.Fun01] Functional testing SHOULD tend to cover the full scope --e.g. positive, negative, edge cases-- for the set of functionality that the Service claims to provide.

    • [SvcQC.Fun01.1] When using APIs, contract testing MUST detect any disruption in the features exposed by the provider to the consumer, through the validation of the API specification. SvcQC.Api01.

    • [SvcQC.Fun01.2] Functional tests SHOULD include the Web Interface or Graphical User Interface (GUI) of the Service.

  • [SvcQC.Fun02] Functional tests SHOULD be checked automatically.

  • [SvcQC.Fun03] Functional tests SHOULD be provided by the developers of the underlying software.

6.2.4. Performance tests [SvcQC.Per]

Performance testing verifies that the software meets the specified performance requirements and assesses performance characteristics - for instance, capacity and response time [@http://www.swebok.org].

Stress or Load testing, exercises software at the maximum design load, as well as beyond it, with the goal of determining the behavioral limits, and to test defense mechanisms in critical systems [@http://www.swebok.org]. Stress testing is a subset of Performance testing [@https://www.geeksforgeeks.org/difference-between-performance-and-stress-testing].

Scalability testing is a test methodology in which an application’s or Service performance is measured in terms of its ability to scale up and/or scale out the number of user requests or other such performance measure attributes, through an increase in the amount of available resources. The definition is based on [@doi:10.1145/2737182.2737185]. Scalability testing is a subset of Performance testing.

Elasticity is based on how quickly Services in an infrastructure are able to adapt [@doi:10.1145/2737182.2737185], in response to variable demand or workload for those service(s) [@doi:10.1186/s13677-019-0134-y]. Elasticity testing is a subset of Performance testing.

  • [SvcQC.Per01] Performance testing SHOULD be carried out to check the Service performance under varying loads.

  • [SvcQC.Per02] Stress testing SHOULD be carried out to check the Service to determine the behavioral limits under sudden increased load.

  • [SvcQC.Per03] Scalability testing MAY be carried out to check the Service ability to scale up or scale down when its load reaches the limits.

  • [SvcQC.Per04] Elasticity testing MAY be carried out to check the Service ability to scale out or scale in, depending on its demand or workload.

6.2.5. Security [SvcQC.Sec] {#security}

Security assessment is essential for any production Service. While an effective implementation of the security requirements applies to every stage in the software development life cycle (SDLC) --especially effective at the source code level, as discussed in [@https://digital.csic.es/handle/10261/160086], section Security [SQA-QC.Sec]--, the security testing of a Service is also --similarly to the diverse testing strategies previously covered-- a black-box type of testing. Hence, this section focuses on the runtime analysis of security-related requirements, as part of the Dynamic Application Security Testing (DAST) as well as the Interactive Application Security Testing (IAST).

Additionally, the compliance with security policies and regulations complements the analysis, which can be implemented, continuously validated and monitored through the Security as Code (SaC) capabilities. SaC is a particularly suitable tool for endorsing security of Service Composition deployments.

  • [SvcQC.Sec01] The Service public endpoints and APIs MUST be secured with data encryption.

    • [SvcQC.Sec01.1] The Service MUST use strong ciphers for data encryption.

  • [SvcQC.Sec02] The Service SHOULD have an authentication mechanism.

    • [SvcQC.Sec02.1] Whenever dealing with a Service Composition, such as microservice architectures, the Services SHOULD be managed by a centralized authentication mechanism.

    • [SvcQC.Sec02.2] In publicly-accessible APIs, Service authentication SHOULD be handled through an API gateway in order to control the traffic and protect the backend services from overuse.

  • [SvcQC.Sec03] The Service SHOULD implement an authorization mechanism.

    • [SvcQC.Sec03.1] In Service Composition environments, the authorization mechanism SHOULD uniquely grant the essential access permissions for each Service according to the Principle of Least Privilege (PoLP).

  • [SvcQC.Sec04] The Service MUST validate the credentials and signatures.

    • [SvcQC.Sec04.1] Credentials used in the Service MUST be signed by a recognized and trusted certification authority.

  • [SvcQC.Sec05] The Service MUST handle personal data in compliance with the applicable regulations, such as the General Data Protection Regulation (GDPR) within the European boundaries.

  • [SvcQC.Sec06] The Service SHOULD be audited in accordance with the black-box testing criteria identified by de-facto (cyber)security standards and good practices.

    • [SvcQC.Sec06.1] Dynamic application security testing (DAST) checks MUST be executed, through the use of ad-hoc tools, directly to an operational Service in order to uncover runtime security vulnerabilities and any other environment-related issues (e.g. SQL injection, cross-site scripting or DDOS). The latest release of OWASP's Web Security Testing Guide [@https://owasp.org/www-project-web-security-testing-guide/stable] and the NIST's Technical Guide to Information Security Testing and Assessment [@doi:10.6028/NIST.SP.800-115] MUST be considered for carrying out comprehensive Service security testing.

    • [SvcQC.Sec06.2] Interactive Application Security Testing (IAST) [@https://www.veracode.com/security/interactive-application-security-testing-iast], analyzes code for security vulnerabilities while the app is run by an automated test. IAST SHOULD be performed to a service in an operating state.

    • [SvcQC.Sec06.3] Penetration testing (manual or automated) MAY be part of the application security verification effort.

    • [SvcQC.Sec06.4] The security assessment of the target system configuration is particularly important to reduce the risk of security attacks. The benchmarks delivered by the Center for Internet Security (CIS) [@https://www.cisecurity.org/cis-benchmarks] and the NIST's Security Assurance Requirements for Linux Application Container Deployments [@doi:10.6028/NIST.IR.8176] MUST be considered for this task.

  • [SvcQC.Sec07] IaC testing, from SvcQC.Dep02 criterion, MUST cover the security auditing of the IaC templates (SaC) in order to assure the deployment of secured Services. For all the third-party dependencies used in the IaC templates (including all kind of software artifacts, such as Linux packages or container-based images):

    • [SvcQC.Sec07.1] SaC MUST perform vulnerability scanning of the artefact versions in use.

    • [SvcQC.Sec07.2] SaC SHOULD verify that the artifacts are trusted and digitally signed.

    • [SvcQC.Sec07.3] SaC MUST scan IaC templates to uncover misalignments with widely accepted security policies from SvcQC.Sec06 criteria, such as non-encrypted secrets or disabled audit logs.

    • [SvcQC.Sec07.4] SaC MAY be used to seek, in the IaC templates, for violations of security requirements outlined in the applicable regulations from criterion SvcQC.Sec05.

    • [SvcQC.Sec07.5] World-writable files SHOULD NOT be created while the service is in operation. Whenever they are required, the relevant files MUST be documented.

    • [SvcQC.Sec07.6] World-readable files MUST NOT contain passwords.