Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add SetAndCheckTenantSetting functional testing utility function #1448

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

schrolla
Copy link
Collaborator

@schrolla schrolla commented Dec 2, 2024

🗣 Description

Adds a new SetAndCheckTenantSetting to the functional testing utilities available for use in YAML-based testplans. The function supports taking two script blocks: one to change a tenant setting and another to test that the value has changed. If the value has not changed as expected, the set will be retried a user-defined number of times over a specified retry interval.
In addition, both the SharePoint PnP and PowerPlatform test plans have been updated to use the retry functions in tests that have previously been problematic in terms of false positives due to configurations not changing as requested.

💭 Motivation and context

Attempts to resolve issues where a test pre-condition requires changing a tenant setting, but the setting doesn't reliably change or report actual success. The function is meant to ensure that the tenant reflects the requested configuration change prior to running the actual test.

Closes #1364

🧪 Testing

Either run one of the updated test plans or create/modify another testplan to use the function. It does require creating or migrating existing pre-condition commands into the SetBlock and creating a new CheckBlock that retrieves that value and checks its value, returning a boolean result.

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • PR targets the correct parent branch (e.g., main or release-name) for merge.
  • Changes are limited to a single goal - eschew scope creep!
  • Changes are sized such that they do not touch excessive number of files.
  • All future TODOs are captured in issues, which are referenced in code comments.
  • These code changes follow the ScubaGear content style guide.
  • Related issues these changes resolve are linked preferably via closing keywords.
  • All relevant type-of-change labels added.
  • All relevant project fields are set.
  • All relevant repo and/or project documentation updated to reflect these changes.
  • Unit tests added/updated to cover PowerShell and Rego changes.
  • Functional tests added/updated to cover PowerShell and Rego changes.
  • All relevant functional tests passed.
  • All automated checks (e.g., linting, static analysis, unit/smoke tests) passed.

✅ Pre-merge checklist

  • PR passed smoke test check.

  • Feature branch has been rebased against changes from parent branch, as needed

    Use Rebase branch button below or use this reference to rebase from the command line.

  • Resolved all merge conflicts on branch

  • Notified merge coordinator that PR is ready for merge via comment mention

  • Demonstrate changes to the team for questions and comments.
    (Note: Only required for issues of size Medium or larger)

✅ Post-merge checklist

  • Feature branch deleted after merge to clean up repository.
  • Verified that all checks pass on parent branch (e.g., main or release-name) after merge.

@schrolla schrolla added enhancement This issue or pull request will add new or improve existing functionality Testing This issue or task involves testing the automation tool function labels Dec 2, 2024
@schrolla schrolla self-assigned this Dec 2, 2024
@schrolla schrolla force-pushed the 1364-create-test-and-retry-utility-function branch from bb8c279 to b6cf72c Compare December 2, 2024 19:59
@schrolla schrolla force-pushed the 1364-create-test-and-retry-utility-function branch from b6cf72c to bc0359d Compare December 9, 2024 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement This issue or pull request will add new or improve existing functionality Testing This issue or task involves testing the automation tool function
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create functional test utility function to check tenant setting and retry until changed
1 participant