Skip to content

Latest commit

 

History

History
168 lines (105 loc) · 7.2 KB

07.ops_quality_criteria.md

File metadata and controls

168 lines (105 loc) · 7.2 KB

7. Quality Criteria - Operational {.page_break_before}

This section describes the operational quality criteria that is not fit for automation, but that contribute to the assessment of the quality of the service when it's in an operational production state.

7.1. Files and documents

7.1.1. Documentation [SvcQC.Doc]

Documentation is an integral part of any Software or Service. For example, it describes how and what users can use and interact with it, or how operators can deploy, configure and manage a given Software or Service.

  • [SvcQC.Doc01] Documentation MUST be available online, easily findable and accessible.

  • [SvcQC.Doc02] Documentation SHOULD have a Persistent Identifier (PID).

  • [SvcQC.Doc03] Documentation MUST be version controlled.

  • [SvcQC.Doc04] Documentation MUST be updated on new Service versions involving any change in the installation, configuration or behavior of the Service.

  • [SvcQC.Doc05] Documentation MUST be updated whenever reported as inaccurate or unclear.

  • [SvcQC.Doc06] Documentation SHOULD have a non-software license.

  • [SvcQC.Doc07] Documentation MUST be produced according to the target audience, varying according to the Service specification. The identified types of documentation and their RECOMMENDED content are:

    • [SvcQC.Doc07.1] Deployment and Administration:

      • Installation and configuration guides.

      • Service Reference Card, with the following RECOMMENDED content:

        • Brief functional description.

        • List of processes or daemons.

        • Init scripts and options.

        • List of configuration files, location and example or template.

        • Log files location and other useful audit information.

        • List of ports.

        • Service state information.

        • List of cron jobs.

        • Security information.

        • FAQs and troubleshooting.

    • [SvcQC.Doc07.2] User:

      • Detailed User Guide for the Service.

      • Public API documentation (if applicable).

      • Command-line (CLI) reference (if applicable).

7.1.2. Policies [SvcQC.Pol]

Policy documents describe what are the user's expected behavior when using the Service, how they can access it and what they can expect regarding privacy of their data.

  • [SvcQC.Pol01] The Service MUST include the following policy documents:

    • [SvcQC.Pol01.1] Acceptable Usage Policy (AUP): Is a set of rules applied by the owner, creator or administrator of a network, Service or system, that restrict the ways in which the network, Service or system may be used and sets guidelines as to how it should be used. The AUP can also be referred to as: Acceptable Use Policy or Fair Use Policy.

    • [SvcQC.Pol01.2] Access Policy or Terms of Use: represent a binding legal contract between the users (and/or customers), and the Provider of the Service. The Access Policy mandates the users (and/or customers) access to and the use of the Provider’s Service.

    • [SvcQC.Pol01.3] Privacy Policy: Data privacy statement informing the users (and/or customers), about which personal data is collected and processed when they use and interact with the Service. It states which rights the users (and/or customers) have regarding the processing of their data.

7.2. Support

7.2.1. Support [SvcQC.Sup]

Support is the formal way by which users and operators of the Service communicate with other operators and/or developers of the Service in case of problems, be it operational problems or bugs in the Service or underlying Software. Reporting of enhancements, improvements and documentation issues.

  • [SvcQC.Sup01] The Service MUST have a tracker or helpdesk for operational and users issues.

  • [SvcQC.Sup02] The Service SHOULD have a tracker for the underlying software issues [@https://digital.csic.es/handle/10261/160086], section [SQA-QC.Man01].

  • [SvcQC.Sup03] The Service SHOULD include an Operational Level Agreement (OLA) with the infrastructure where it is integrated.

  • [SvcQC.Sup04] The Service MAY include Service Level Agreement (SLA) with user communities.

7.3. Monitoring and Metrics

7.3.1. Monitoring [SvcQC.Mon]

Monitoring is a periodic testing of the Service. It requires a monitoring service from where tests are executed or sent and results of those tests are shown. The tests can be the same, in part or in total of the Functional, Security and Infrastructure tests. The technology used for the monitoring is left to the developers of the underlying software to decide eventually with input from the infrastructure(s), where the Service is foreseen to be integrated.

  • [SvcQC.Mon01] The Service in an operational production state SHOULD be monitored for functional-related criteria:

    • [SvcQC.Mon01.1] The Service public endpoints MUST be monitored.

    • [SvcQC.Mon01.2] The Service public APIs MUST be monitored. Use functional tests of criteria SvcQC.Fun01.1.

    • [SvcQC.Mon01.3] The Service Web interface MAY be monitored. Use functional tests of criteria SvcQC.Fun01.2.

  • [SvcQC.Mon02] The Service MUST be monitored for security-related criteria:

    • [SvcQC.Mon02.1] The Service MUST be monitored for public endpoints and APIs secured and strong ciphers for encryption. Use Security tests of criteria SvcQC.Sec01.

    • [SvcQC.Mon02.2] The Service SHOULD be monitored with DAST checks. Use Security tests of criteria SvcQC.Sec06.1.

  • [SvcQC.Mon03] The Service MUST be monitored for infrastructure-related criteria:

    • [SvcQC.Mon03.1] IaC (unit) tests SvcQC.Dep02 SHOULD be reused as monitoring tests, thus avoiding duplication.

7.3.2. Metrics [SvcQC.Met]

A metric is a quantifiable measure used to track and assess the status of a specific process.

In the case of Services, some relevant metrics are the number of users registered in the Service, or of active users. Also accounting is important to track resource usage per user or group of users, either or both computing and storage resources.

Although the metrics may be published in external services managed by the infrastructure, this is a common case in federated infrastructures such as EOSC.

  • [SvcQC.Met01] The Service SHOULD implement the collection of metrics.

    • [SvcQC.Met01.1] The collection of metrics SHOULD be cumulative over time and timestamped, so that the values can be queried per time interval.

    • [SvcQC.Met01.2] The metric Number of registered users SHOULD be collected.

    • [SvcQC.Met01.3] The metric Number of active users over a given period of time MAY be collected.

    • [SvcQC.Met01.4] The metric Amount of computing resources per user or per group MAY be collected. The metric unit depends on the type of service and infrastructure. An example is CPU x hours.

    • [SvcQC.Met01.5] The metric Amount of storage resources per user or per group MAY be collected. The metric unit depends on the type of service and infrastructure. An example is TByte x month.