Skip to content

Commit

Permalink
fix: Rework e2e scan summary snapshot (#1539)
Browse files Browse the repository at this point in the history
#### Details

This pull request will clip the e2e scan summary snapshot slightly to
make sure the snapshot is consistent in every build environment.

In #1535 we added an e2e snapshot test to capture the scan summary. When
the pull request merged and triggered the build pipeline, the new test
failed. This is because the build pipeline was able to upload and
capture the artifact URL, which changed the snapshot. The added line to
the snapshot: "To see all failures and scan details, download the
accessibility report (link)".

##### Motivation

Make tests passing.

##### Context

<!-- Are there any parts that you've intentionally left out-of-scope for
a later PR to handle? -->

- I don't think there's a way to mock the artifact URL. While it's
possible to register mocks with the azure-pipelines-task-lib, since the
TaskMockRunner is executing on compiled code, I wasn't able to find a
way to register the mock successfully.

- I considered adding `uploadOutputArtifact: false,` to this e2e test's
inputs object, however, I noticed that the line "To see all failures and
scan details, download the accessibility report (link)" appears in the
console report even if `uploadOutputArtifact` is false. I'm imagining
this is so users who need to upload the artifact separately are able to
access the artifacts.

- I ran the build pipeline to confirm the snapshot tests will pass:
[build
pipeline](https://dev.azure.com/accessibility-insights-private/Accessibility%20Insights%20(private)/_build/results?buildId=42079&view=logs&j=81b7b1a0-9e2e-58a5-1601-69aaad9b82d6&t=5cf72e78-07c0-5125-1262-b1e05b2475fe)

<!-- Were there any alternative approaches you considered? What
tradeoffs did you consider? -->

#### Pull request checklist
<!-- If a checklist item is not applicable to this change, write "n/a"
in the checkbox -->
- [n/a] Addresses an existing issue: Fixes #0000
- [x] Added relevant unit test for your changes. (`yarn test`)
- [x] Verified code coverage for the changes made. Check coverage report
at: `<rootDir>/test-results/unit/coverage`
- [x] Ran precheckin (`yarn precheckin`)
  • Loading branch information
katydecorah authored Feb 13, 2023
1 parent 4dd4950 commit 4ccbcf8
Showing 1 changed file with 3 additions and 13 deletions.
16 changes: 3 additions & 13 deletions packages/ado-extension/e2e/ado-extension.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,23 +10,13 @@ describe('Sample task tests', () => {
inputs = {};
});

it('returns expected markdown report', () => {
it('returns expected scan summary and footer', () => {
inputs = {
url: 'https://www.washington.edu/accesscomputing/AU/before.html',
};
const testSubject = runTestWithInputs(inputs);
expect(filterStdOut(testSubject.stdout)).toMatchInlineSnapshot(`
"Accessibility Insights
20 failure instances
* (10) label: Ensures every form element has a label
* (4) color-contrast: Ensures the contrast between foreground and background colors meets WCAG 2 AA contrast ratio thresholds
* (3) link-in-text-block: Ensure links are distinguished from surrounding text in a way that does not rely on color
* (2) image-alt: Ensures <img> elements have alternate text or a role of none or presentation
* (1) html-has-lang: Ensures every HTML document has a lang attribute
Baseline not configured
A baseline lets you mark known failures so it's easier to identify new failures as they're introduced. See baselining docs (https://aka.ms/ado-extension-usage-baseline) for more.-------------------
"-------------------
Scan summary
URLs: 1 with failures, 0 passed, 0 not scannable
Rules: 5 with failures, 14 passed, 36 not applicable
Expand All @@ -38,7 +28,7 @@ describe('Sample task tests', () => {
`);

function filterStdOut(stdout: string) {
const logs = stdout.match(/Accessibility Insights\n(.|\n)*##\[debug\]\[Telemetry\] tracking a 'ScanCompleted' event/);
const logs = stdout.match(/-------------------(.|\n)*##\[debug\]\[Telemetry\] tracking a 'ScanCompleted' event/);
return logs ? logs[0] : '';
}
});
Expand Down

0 comments on commit 4ccbcf8

Please sign in to comment.