adding documentation

WarnerMedia · Feb 8, 2023 · 50271de · 50271de
1 parent 7ec3936
commit 50271de
Show file tree

Hide file tree

Showing 13 changed files with 509 additions and 4 deletions.
diff --git a/README.md b/README.md
@@ -1,16 +1,33 @@
-# 🛡️ Artemis: Hunt For Security Issues In Source Code
+# Artemis: Hunt For Security Issues In Source Code
+
+![Artemis](docs/images/logo192.png)
 
 Artemis is an extensible source code scanning tool developed by the Warner Bros. Discovery Application Security team that provides a single interface for running multiple security analysis tools against a source code repository, regardless of the contents of the repository. Artemis can scan repositories in different GitHub, GitLab, Bitbucket, or Azure DevOps organizations from a single, unified platform.
 
 ## 📖 Contents
 
 - [Overview](#overview)
 - [License](#license)
+- [Contributors](docs/contributors.md)
 
-## Overview
+## 🔨 Building
 
 Artemis is made up of 3 primary components: the [backend](./backend), the [web UI](./ui), and the [scan orchestrator](./orchestrator). Each of these has its own components, architecture, and development processes, which are detailed in the README files within their subdirectories.
 
-## License
 
-This repository is released under [the MIT license](https://en.wikipedia.org/wiki/MIT_License).  View the [local license file](./LICENSE).
+## 📁 Documents
+
+[Artemis Overview](docs/overview.md)
+
+[Why Shift-Left? Why Artemis?](docs/shiftleft.md)
+
+[API Examples](docs/api-examples.md)
+
+[CI Integration](docs/CI.md)
+
+[GitHub Actions Integrations](docs/actions.md)
+
+
+## ©️ License
+
+This repository is released under [the MIT license](https://en.wikipedia.org/wiki/MIT_License).  View the [local license file](./LICENSE).
diff --git a/docs/CI.md b/docs/CI.md
@@ -0,0 +1,31 @@
+# CI Integration
+
+
+### Considerations
+
+Before integrating Artemis into your CI pipeline you will need to consider a few things:
+
+1. Identify a user with appropriate permissions to make the CI changes.
+1. Identify a user who will "own" the Artemis API key (this may be the same user as #1). The Artemis API key is used by the integration to authorize calls to the Artemis REST API (using the HTTP header x-api-key).
+1. Identify where in the CI pipeline you want to run the scan.
+1. Determine whether you want to gate your build/deployment based on scan results.
+
+### How to setup Artemis in CI
+
+1. Ensure the integration user (see Considerations #2 above) has access to view/scan all necessary repos in the Artemis web UI.
+
+1. Create an API key for the user or service account you want to “own” the integration (see Considerations #2 above).
+
+1. Add the integration to your CI pipeline at the stage determined in Considerations #2 above.  
+
+
+**Note**: The artemis-scan.sh script used in the GitHub Actions CI integration should be generic enough to use in any CI pipeline as long as it supports a Unix shell
+
+Once the CI integration is in-place, you should also be able to view scans initiated via the CI integration in the Artemis UI.
+
+Here is an example of how Artemis can integrate with [GitHub Actions](actions.md)
+
+**[Main](../README.md)**
+
+
+
diff --git a/docs/actions.md b/docs/actions.md
@@ -0,0 +1,74 @@
+# GitHub Actions Integrations
+Create a file like this in your repo’s `.github/workflows` directory and give it a name that is indicative of it’s function, such as `artemis.yml` or `scan.yml`
+
+This scan will occur when a PR is created since it is set to run on the `pull_request` lifecycle event.
+
+
+```
+name: Artemis
+
+on: pull_request
+
+jobs:
+  scan:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Run Artemis scan
+        env:
+          ARTEMIS_API_KEY: ${{ secrets.ARTEMIS_API_KEY }}
+        run: |
+          curl --silent --show-error --fail \
+            https://example.com/ci-tools/shell/artemis-scan.sh \
+            --header "x-api-key: $ARTEMIS_API_KEY" \
+            --output artemis-scan.sh
+          /bin/bash artemis-scan.sh github ${{ github.repository }} ${{ github.head_ref }} critical,high
+```
+
+### Authorizing the scan
+
+In order for the scan to run you will need to plumb the API key from your secrets manager into the GitHub Action (refer to CI Integration step #2 for information on API key creation)
+In GitHub Enterprise, this can be achieved by adding a secret named ARTEMIS_API_KEY in the "Repository Secrets" section of the repository's Settings > Secrets.
+
+**Note**: In the script above, this secret is set in a shell environment variable of the build runner so it is accessible in the initiated artemis-scan.sh script.
+
+### Artemis-scan.sh  
+The artemis-scan.sh script is used in this GitHub Action integration, but it is not limited to use in GitHub integrations, it should be generic enough that any CI system that allows Unix shell scripts should be able to consume this script.
+
+The script takes 4 parameters:  
+1. The `service` (version control system) where the source code lives. This matches the VCS portion of the "Version Control System" field in the Artemis UI. For example, `github` for GitHub Enterprise.   
+1. The repository where the source code lives. This may include an organization name or other pathing to the source code, for example, `testorg/example`.    
+1. The `branch` name.     
+1. A comma-separated list of vulnerability severities that you want to trigger a scan failure.      
+Values include: critical, high, medium, low, negligible
+
+A "scan failure" means that the `artemis-scan.sh` script will return an error message and a non-zero return 
+code if any of the specified conditions are met.
+By default, the script will always return a failure if any secrets are detected in addition to any vulnerability severities the user specifies. 
+
+### Project Status
+
+Whether you decide to gate builds based on scan results, it can be beneficial to add some high-level visibility of scan results to a project so that team members can take appropriate corrective actions.
+
+
+>Important: Result status should just be a high-level pass/fail - it should not expose any of the sensitive findings from the actual scan results.
+
+In GitHub Enterprise, this can be accomplished by:
+
+1. Ensure that the GitHub Action returns a non-zero return code if the scan fails (this is the default behavior of the GitHub Action example above)
+1. Configure GitHub so that the PR may pass even if the Artemis scan action fails:
+Go to the GitHub repo Settings > Branches > Edit Branch Protection Rules
+In the “Protect matching branches” section you’ll notice there’s a checkbox for “Require status checks pass before merging” – if this is checked then all actions specified in the “Status checks that are required” table must pass. Ensure that the artemis/scan action is removed from that list.
+
+    Now the action should run on `pull_request` and may fail, but the failure shouldn’t prevent the PR from being merged.
+
+    You can always enable `Require status checks pass before merging` in the repository's Branch Protection Rules when you decide you want to gate PRs based on scan results.
+
+3. To surface the scan status in the GitHub project page, navigate to the repo's Settings > Actions. Click the artemis/scan action from the “Workflows” column on the left-hand side of the page.
+ If you click the “…” beside the filter box with placeholder text “Filter workflow runs” you can select “Create status badge”
+ This should provide some Markdown code to generate a passing/failing badge for CI status.
+ Copy this Markdown code and add it to top of the `README.md` file for that repo. This should then display a CI scan pass/fail badge at the top of the repository's README. 
+
+![CI Badge](images/ci_badge.png)
+
+**[Main](../README.md)**
+
diff --git a/docs/api-examples.md b/docs/api-examples.md
@@ -0,0 +1,242 @@
+# API Examples
+
+## Authentication
+
+The API accepts a key in the `x-api-key` header.  
+You can create an API key in the User Profile section of the Artemis web UI.
+
+## Initiating a single scan
+
+The primary API interface for creating scans, retrieving reports and scan history, and manipulating the allow list follows the format `/api/v1/SERVICE/ORG/REPO` where `SERVICE` is the name of the VCS the repo resides in, such as github or an hostname, in the case of an internal VCS. `ORG` is the organization within the VCS, such as testorg. Some VCS, such as GitLab allow for arbitrary nesting of subgroups. In that case the set of subgroups is part of the ORG, such as group/group1/group2. Finally, REPO is the repository name.
+
+For example, in the case of a repository named example in the `testorg` GitHub organization the SERVICE is github, the ORG is `testorg`, and the REPO is `example`. Therefore, the API endpoint for scanning the example repository would be `/api/v1/github/testorg/example`
+
+To initiate a scan, POST to the `/api/v1/SERVICE/ORG/REPO` endpoint for the repository you wish to scan. The body of the POST request is a JSON object. If the body is empty all the defaults are used. The object fields are:
+
+* branch: [OPTIONAL, default: HEAD] The branch to scan. If omitted the scan is run against the HEAD ref.
+* depth: [OPTIONAL, default: 500] If a tool allows a GitHub depth limit, the max number of commits to examine. Otherwise this value is ignored.
+* plugins: [OPTIONAL] The specific plugins to run against the repository. If omitted, all plugins are run. If included but empty, no plugins are run.
+* categories: [OPTIONAL] The specific plugin categories to run against the repository. If omitted, all plugins are run. If included alongside plugins (above), all plugins under the specified categories PLUS the plugins within the plugins list will be run.
+* callback: [OPTIONAL] A dictionary containing the URL to POST to when the scan is completed and a client-defined identifier to echo back in the POST body.
+	* url: Callback URL.
+	* **client_id: Client-defined ID string.
+
+## Initiating multiple scans for the same service
+
+You can initiate more than on scan at a time for different repositories, provided all of the repositories are in the same service. They do not need to all be in the same organization. For this, the API endpoint is /api/v1/SERVICE. Instead of the POST body being a single JSON object the body is a JSON array of objects, each being a single repo scan request. The object fields are the same as with the single request above but with two non-optional additions:
+
+* repo: The name of the repository, excluding the owner
+* org: The organization the repository belongs to
+
+### Examples
+
+Scan a repo using default settings:
+
+```
+curl -X POST \
+https://example.com/api/v1/github/testorg/example \
+-H 'x-api-key: APIKEYGOESHERE'
+```
+
+Scan the dev branch of a repo using only the `gitsecrets` plugin, plugins in the inventory category, and a callback:
+
+```
+curl -X POST \
+  https://example.com/api/v1/github/testorg/example \
+  -H 'Content-Type: application/json' \
+  -H 'x-api-key: APIKEYGOESHERE' \
+  -d '{
+    "branch": "dev",
+    "plugins": [
+      "gitsecrets"
+    ],
+    "categories": [
+      "inventory"
+    ],
+    "callback": {
+      "url": "https://example.com/myhandler",
+      "client_id": "12345"
+    }
+  }'
+```
+
+Scan two repos within GitHub, one using only the `gitsecrets` plugin, the inventory category, and a callback, and the other using default settings:
+
+```
+curl -X POST \
+  https://example.com/api/v1/github \
+  -H 'Content-Type: application/json' \
+  -H 'x-api-key: APIKEYGOESHERE' \
+  -d '[
+    {
+      "org": "testorg",
+      "repo": "example",
+      "branch": "dev",
+      "plugins": [
+        "gitsecrets"
+      ],
+      "categories": [
+        "inventory"
+      ],
+      "callback": {
+        "url": "https://example.com/myhandler",
+        "client_id": "12345"
+      }
+    },
+    {
+      "org": "testorg",
+      "repo" "testrepo"
+    }
+  ]'
+```
+
+## Retrieving a report
+
+To retrieve the most recent scan report GET the `/api/v1/SERVICE/ORG/REPO` resource. To retrieve a specific scan report GET the `/api/v1/SERVICE/ORG/REPO/SCAN_ID` resource.
+
+### Examples
+
+Get the latest report for a specific repository:
+
+```
+curl -X GET \
+  https://example.com/v1/repo/testorg/example \
+  -H 'x-api-key: APIKEYGOESHERE'
+```
+
+Get a specific report:
+
+```curl -X GET \
+  https://example.com/v1/repo/testorg/example/abc0123-0abc-0123-0abc-00abc0123abc \
+  -H 'x-api-key: APIKEYGOESHERE'
+```
+
+## Allow-listing Findings
+
+Findings can be allow-listed so that they do not appear in reports or affect the overall success status of the report.
+
+### Operations
+
+* `GET /api/v1/SERVICE/ORG/REPO/whitelist` - Get allowlist for repository. Can be filtered with the type query parameter.
+* `GET /api/v1/SERVICE/ORG/REPO/whitelist/UUID` - Get specific allowlist item.
+* `POST /api/v1/SERVICE/ORG/REPO/whitelist` - Create new allowlist item.
+* `PUT /api/v1/SERVICE/ORG/REPO/whitelist/UUID` - Update existing allowlist item.
+* `DELETE /api/v1/SERVICE/ORG/REPO/whitelist/UUID` - Delete existing allowlist item.
+
+The whitelist payload is a JSON blob:
+
+```
+{
+  "id": "<UUID>",
+  "type": "vulnerability | secret | static_analysis",
+  "value": {}
+}
+```
+The value format is dependent on the type.
+
+### Vulnerability
+
+```
+{
+  "component": "<COMPONENT>",
+  "id": "<ID>",
+  "source": "<DEPENDENCY_SOURCE>"
+}
+```
+### Secret
+```
+{
+  "filename": "<FILENAME>",
+  "line": <LINE_NUM>,
+  "commit": "<COMMIT_HASH>"
+}
+```
+
+**Note:** To allowlist a specific commit, the filename still must be specified, even if the file has already been removed from the repository. An example of this scenario is that if an SSH key is found in a repository and it gets removed, the commit history will still be flagged by Artemis for containing the key that was checked in. You will need to whitelist the commit once the repository owner has removed and changed the key.
+
+### Examples
+**Get** the full allowlist:
+
+```
+curl -X GET \
+  https://example.com/api/v1/github/testorg/example/whitelist \
+  -H 'x-api-key: APIKEYGOESHERE'
+```
+
+**Get** just the allowlisted secrets:
+
+```
+curl -X GET \
+  https://example.com/api/v1/github/testorg/example/whitelist?type=secret \
+  -H 'x-api-key: APIKEYGOESHERE' 
+```
+
+**Create** a new allowlist item:  
+
+```
+curl -X POST \
+  https://example.com/api/v1/github/testorg/example/whitelist \
+    -H 'Content-Type: application/json' \
+    -H 'x-api-key: APIKEYGOESHERE' \
+    -d '{
+      "type": "vulnerability",
+      "value": {
+        "component": "library-1.2.3",
+        "id": "CVE-2020-1234567",
+        "source": "Dockerfile"
+      }
+    }'
+```
+
+**Delete** an existing allowlist item:
+
+```
+curl -X DELETE \
+  https://example.com/api/v1/github/testorg/example/whitelist/abc0123-0abc-0123-0abc-00abc0123abc \
+  -H 'x-api-key: APIKEYGOESHERE'  
+```
+
+## SBOM
+
+### Initiating an SBOM scan
+
+Run a scan and enable the `sbom` plugin category
+
+```
+curl -X POST \
+  https://example.com/api/v1/SERVICE/ORG/REPO \
+  -H 'Content-Type: application/json' \
+  -H 'x-api-key: APIKEYGOESHERE' \
+  -d '{ "categories": ["sbom"] }'
+```  
+
+### List components
+List all components discovered through SBOM scanning
+
+```
+curl -X GET \
+  https://example.com/api/v1/sbom/components \
+  -H 'x-api-key: APIKEYGOESHERE'
+```
+
+List all components with GPL licenses discovered through SBOM scanning
+
+```
+curl -X GET \
+  https://example.com/api/v1/sbom/components?license__contains=GPL \
+  -H 'x-api-key: APIKEYGOESHERE'
+```
+
+### List Repositories
+
+List all repositories using a specific component
+
+```
+curl -X GET \
+  https://example.com/api/v1/sbom/components/COMPONENT/VERSION/repos \
+  -H 'x-api-key: APIKEYGOESHERE'
+```  
+
+
+
+**[Main](../README.md)**