E2E test case framework setup #238

chinmdas · 2025-02-25T13:17:06Z

What It Does
This PR does following:

It sets up framework for E2E Test cases for CICS Zowe client using vscode-extension-tester
Covers some scenarios like:
- Creating New Global Team Configuration file (when NO Config file initially available)
- Editing Global Team Configuration file
- Editing Project Level Team Configuration file

How to Test

Run the below command from parent package.json location
- cd packages/vsce && npm run test:e2e

Review Checklist
I certify that I have:

tested my changes
added/updated automated tests
updated the changelog
followed the contribution guidelines

packages/vsce/src/ui-test/00_CreateNewGlobalTeamConfiguration.test.ts

packages/vsce/src/ui-test/01_EditProjectTeamConfiguration.test.ts

AndrewTwydell

Few things to look at while I actually run the tests locally.

packages/vsce/CHANGELOG.md

packages/vsce/package.json

packages/vsce/settings.json

AndrewTwydell · 2025-02-25T14:27:03Z

I also would like to know if we can put the tests alongside the others and not put stuff in src. I imagine it's part of the extester config - think this would be tidier and more consistent with everything else.

codecov-commenter · 2025-02-26T07:12:54Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 40.78%. Comparing base (2d13589) to head (1ddee3d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #238   +/-   ##
=======================================
  Coverage   40.78%   40.78%           
=======================================
  Files         151      151           
  Lines        5056     5056           
  Branches      817      861   +44     
=======================================
  Hits         2062     2062           
  Misses       2994     2994

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

chinmdas · 2025-02-26T07:16:10Z

I also would like to know if we can put the tests alongside the others and not put stuff in src. I imagine it's part of the extester config - think this would be tidier and more consistent with everything else.

Hi @AndrewTwydell ,
I have now moved the complete e2e test project scaffolding to __test__/__e2e__.

packages/vsce/tsconfig.json

almightyrush

ignore .cics directory for git.

packages/vsce/__tests__/__e2e__/00_CreateNewGlobalTeamConfiguration.test.ts

packages/vsce/__tests__/__e2e__/01_EditGlobalTeamConfiguration.test.ts

packages/vsce/__tests__/__e2e__/01_EditProjectTeamConfiguration.test.ts

chinmdas · 2025-02-27T15:37:54Z

ignore .cics directory for git.

This is handled now..!!

AndrewTwydell

Review part 1/2

AndrewTwydell · 2025-02-28T13:38:30Z

.eslintrc.yaml

@@ -6,7 +6,7 @@ extends:
 env:
  node: true
  es6: true
-ignorePatterns: ["**/scripts", "**/__mocks__", "**/lib"]
+ignorePatterns: ["**/scripts", "**/__mocks__", "**/lib", "**/__tests__/__e2e__"]


Is there a reason we don't want to lint check the e2e tests?? We check the rest of the test files

AndrewTwydell · 2025-02-28T13:41:15Z

packages/vsce/__tests__/__e2e__/00_CreateNewGlobalTeamConfiguration.test.ts

+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License", destination); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */


Where did you get this license header from? The rest of our files have a different license.

You can run npm run license at the top level to ensure they have the correct license header.

AndrewTwydell · 2025-02-28T13:41:22Z

packages/vsce/__tests__/__e2e__/01_EditGlobalTeamConfiguration.test.ts

+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License", destination); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */


AndrewTwydell · 2025-02-28T13:41:29Z

packages/vsce/__tests__/__e2e__/01_EditProjectTeamConfiguration.test.ts

+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License", destination); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */


AndrewTwydell · 2025-02-28T13:44:39Z

packages/vsce/tsconfig.json

This file has now not changed

AndrewTwydell · 2025-02-28T13:48:54Z

packages/vsce/package.json

@@ -866,9 +866,20 @@
    "tsupDep": "node ./scripts/tsup-deps.js",
    "getPrebuilds": "node ./scripts/getSecretsPrebuilds.js",
    "package": "vsce package --dependencies --allow-star-activation && node ../../scripts/mv-pack.js cics-extension-for-zowe vsix",
+    "test:e2e:prepare": "extest get-vscode -s .cics && extest get-chromedriver -s .cics",
+    "test:e2e:install": "extest install-from-marketplace zowe.vscode-extension-for-zowe -s .cics -e .cics",
+    "test:e2e:infra:setup": "mkdir -p ~/e2e-global && export ZOWE_CLI_HOME=~/e2e-global",


I'm still of the opinion we should be creating the test ZOWE_CLI_HOME directory in the same place as our test workspace folder (.cics) - we could have .cics-project and .cics-global folders, both under e2e. I think it would be bad practice to create folders under a user's home directory.

Happy to be disuaded here, what do people think 🤷‍♂️

I like the addition of this though - using a separate ZOWE_CLI_HOME seems a safe option

AndrewTwydell · 2025-02-28T13:50:21Z

packages/vsce/__tests__/__e2e__/settings.json

Purely for my future knowledge, why do we need to provide our test VSCode instance with settings?? Are some of these mandatory for the kind of tests we're doing?

AndrewTwydell · 2025-02-28T13:52:22Z

packages/vsce/__tests__/__e2e__/.mocharc-debug.js

Seems like we could get into a spot where GitHub actions is waiting hours (or months if this is seconds) for a test to timeout... Not good 🤔

Signed-off-by: Chinmay Das <[email protected]>

AndrewTwydell

Generally I like the vision here. Some things to look at though:

The names for the test suites and tests need to be clearer - I can't read them and know what the test is doing (Quick Pick Option test for example)
Repeated tests aren't necessary, and any repeated actions they're taking can be in methods to be called or in the before setup stages
I'd ideally like more assertions before and after to actually prove we're testing things rather than performing actions.

AndrewTwydell · 2025-02-28T15:27:26Z

packages/vsce/__tests__/__e2e__/00_CreateNewGlobalTeamConfiguration.test.ts

+  it("+ (plus) button clickable test", async () => {
+    // selecting cics view and clicking + icon
+    await cicsTree.click();
+    await cicsTree.expand();
+    plusIcon = await cicsTree.getAction(`Create a CICS Profile`);
+    expect(plusIcon).exist;
+    cicsTree.takeScreenshot();
+    await plusIcon.click();
+  }).timeout(5000);


To me, this isn't testing anything - it's performing an action without asserting state before or after.

To make this a test of that plus button, I'd want to know there are no quick picks currently open, then perform the action, then assert that is a quick pick open with the expected content.

AndrewTwydell · 2025-02-28T15:31:51Z

packages/vsce/__tests__/__e2e__/00_CreateNewGlobalTeamConfiguration.test.ts

+  before(async function () {
+    this.timeout(15000);
+    // open the cics extension view
+    const view = await(await new ActivityBar().getViewControl("Zowe Explorer"))?.openView();
+
+    //handle notifications if config file is missing
+    const notifications = await new Workbench().getNotifications();
+    if (notifications.length > 0) {
+      await notifications[0].click();
+      const actions = await notifications[0].getActions();
+      await actions[0].click();
+
+      //select qp option
+      quickPick = await InputBox.create();
+      await quickPick.selectQuickPick(0);
+    }
+
+    cicsTree = await view.getContent().getSection("cics");
+    await cicsTree.click();
+  });


Most of this feels like it should be tests rather than environment setup to me.

Opening the CICS view and asserting it's visible seems like a reasonable environment setup step, but creating the global config file using the notification seems like something we should be testing with assertions before and after we perform the action.

We will want tests that start with no config file so we can test behaviour, as well as situations where the file already exists.

AndrewTwydell · 2025-02-28T15:33:16Z

packages/vsce/__tests__/__e2e__/00_CreateNewGlobalTeamConfiguration.test.ts

+    const label1 = await qpItems[0].getLabel();
+    expect(label1).contains("＋ Create a New Team Configuration File");
+
+    if (qpItems.length > 1) {


This seems too vague for a test - we should know EXACTLY what state our environment has when running commands so we can assert the exact behaviour we expect. Here, we should know if the quick pick has 1 option or more, and probably have a test for each circumstance.

AndrewTwydell · 2025-02-28T15:35:23Z

packages/vsce/__tests__/__e2e__/00_CreateNewGlobalTeamConfiguration.test.ts

+    const titles = await editorView.getOpenEditorTitles();
+
+    // Check zowe.config.json was opened - could check content here
+    expect(titles.some((title) => title.startsWith("zowe.config.json"))).is.true;


This is good. Wonder if we can check more than just the title - perhaps some of the filepath to know we've got the corrct config file (global/project)

AndrewTwydell · 2025-02-28T15:36:52Z

packages/vsce/__tests__/__e2e__/01_EditGlobalTeamConfiguration.test.ts

+  before(async function () {
+    // open the cics extension view
+    this.timeout(15000);
+    const view = await (await new ActivityBar().getViewControl("Zowe Explorer"))?.openView();
+    cicsTree = await view.getContent().getSection("cics");
+  });


So because this test file is focussed on editing an existing global config file, I would expect the before section to be creating and asserting this global config file exists, so we can then test against it.

AndrewTwydell · 2025-02-28T15:39:00Z

packages/vsce/__tests__/__e2e__/00_CreateNewGlobalTeamConfiguration.test.ts

+  }).timeout(5000);
+
+  // now we can check what the dialog says
+  it("Create New Configuration File Test", async () => {


This is more specific than the name suggests right? Doesn't it go through the motions of creating a new global config file, even though one already exists?

AndrewTwydell · 2025-02-28T15:39:55Z

packages/vsce/__tests__/__e2e__/01_EditGlobalTeamConfiguration.test.ts

+  it("CICS Section title Check", async () => {
+    // title check for cics section
+    const title = await cicsTree.getTitle();
+    expect(title).equals("cics");
+  }).timeout(100000);
+
+  it("+ (plus) button clickable test", async () => {
+    // selecting cics view and clicking + icon
+    await cicsTree.expand();
+    const plusIcon = await cicsTree.getAction(`Create a CICS Profile`);
+    await cicsTree.click();
+    expect(plusIcon).exist;
+    await plusIcon.click();
+    cicsTree.takeScreenshot();
+  }).timeout(100000);


If these are idential tests to elsewhere, we shouldn't need them repeated. Any actions they perform that are required for other tests should be done in a before/beforeEach section

Signed-off-by: Chinmay Das <[email protected]>

I, Chinmay Das <[email protected]>, hereby add my Signed-off-by to this commit: 9065c1e Signed-off-by: Chinmay Das <[email protected]>

github-actions bot assigned chinmdas Feb 25, 2025

chinmdas requested review from AndrewTwydell, almightyrush, zFernand0, davenice and enamkhan February 25, 2025 13:17

github-advanced-security bot found potential problems Feb 25, 2025

View reviewed changes

packages/vsce/src/ui-test/00_CreateNewGlobalTeamConfiguration.test.ts Fixed Show fixed Hide fixed

packages/vsce/src/ui-test/01_EditProjectTeamConfiguration.test.ts Fixed Show fixed Hide fixed

AndrewTwydell requested changes Feb 25, 2025

View reviewed changes

packages/vsce/CHANGELOG.md Outdated Show resolved Hide resolved

packages/vsce/package.json Outdated Show resolved Hide resolved

packages/vsce/settings.json Outdated Show resolved Hide resolved

chinmdas requested a review from AndrewTwydell February 26, 2025 07:17

AndrewTwydell reviewed Feb 26, 2025

View reviewed changes

packages/vsce/tsconfig.json Outdated Show resolved Hide resolved

almightyrush reviewed Feb 27, 2025

View reviewed changes

chinmdas requested review from almightyrush and AndrewTwydell February 27, 2025 15:39

chinmdas linked an issue Feb 27, 2025 that may be closed by this pull request

Add end to end testing #225

Open

chinmdas added the no-changelog label Feb 27, 2025

chinmdas force-pushed the e2e-extension-test-branch branch from 28e58ae to 9e5d149 Compare February 27, 2025 16:23

AndrewTwydell requested changes Feb 28, 2025

View reviewed changes

chinmdas force-pushed the e2e-extension-test-branch branch from c3cb212 to 4a25657 Compare February 28, 2025 15:22

chinmdas added 8 commits February 28, 2025 20:57

Rebasing to remove empty commit

9065c1e

Adding package-lock.json

640e510

Signed-off-by: Chinmay Das <[email protected]>

Removing unused imports

a2211b5

Signed-off-by: Chinmay Das <[email protected]>

Fixing ESLint issues

a66ac28

Signed-off-by: Chinmay Das <[email protected]>

ignoring e2e tests from unit test code coverage

434afc6

Signed-off-by: Chinmay Das <[email protected]>

moving the scaffolding to __test__/__e2e__

59538d7

Signed-off-by: Chinmay Das <[email protected]>

moving settings.json to __tests__/__e2e__

5ef2f5e

Signed-off-by: Chinmay Das <[email protected]>

configuring global config location

05bc7a9

Signed-off-by: Chinmay Das <[email protected]>

chinmdas added 2 commits February 28, 2025 20:58

using test folder tsconfig.json to compile

ed3fd64

Signed-off-by: Chinmay Das <[email protected]>

Removed empty commit

84f1697

Signed-off-by: Chinmay Das <[email protected]>

chinmdas force-pushed the e2e-extension-test-branch branch from 4a25657 to 84f1697 Compare February 28, 2025 15:29

chinmdas requested a review from AndrewTwydell February 28, 2025 15:34

AndrewTwydell requested changes Feb 28, 2025

View reviewed changes

chinmdas added 2 commits February 28, 2025 21:13

removing duplicate pretty

a0661cb

Signed-off-by: Chinmay Das <[email protected]>

DCO Remediation Commit for Chinmay Das <[email protected]>

1ddee3d

I, Chinmay Das <[email protected]>, hereby add my Signed-off-by to this commit: 9065c1e Signed-off-by: Chinmay Das <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E test case framework setup #238

E2E test case framework setup #238

chinmdas commented Feb 25, 2025 •

edited

Loading

AndrewTwydell left a comment

AndrewTwydell commented Feb 25, 2025

codecov-commenter commented Feb 26, 2025 •

edited

Loading

chinmdas commented Feb 26, 2025

almightyrush left a comment

chinmdas commented Feb 27, 2025

AndrewTwydell left a comment

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell left a comment

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

AndrewTwydell Feb 28, 2025

E2E test case framework setup #238

Are you sure you want to change the base?

E2E test case framework setup #238

Conversation

chinmdas commented Feb 25, 2025 • edited Loading

AndrewTwydell left a comment

Choose a reason for hiding this comment

AndrewTwydell commented Feb 25, 2025

codecov-commenter commented Feb 26, 2025 • edited Loading

Codecov Report

chinmdas commented Feb 26, 2025

almightyrush left a comment

Choose a reason for hiding this comment

chinmdas commented Feb 27, 2025

AndrewTwydell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndrewTwydell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chinmdas commented Feb 25, 2025 •

edited

Loading

codecov-commenter commented Feb 26, 2025 •

edited

Loading