zowe.schema.json is not updated automatically when extenders contribute a new service profile to zowe-explorer-api

[FrankSu1996]

Describe the bug

When extenders use zowe-explorer-api to contribute a new service profile, any currently present zowe.schema.json files (either workspace/global level) do not get updated to contain the new service profile's metadata. Currently, the only way to update the schema is by creating a new team configuration file using the Zowe Explorer views.

To Reproduce

1. Make sure a zowe schema and team configuration files are already present at either workspace or global level
2. Create an extender that registers a new service profile type with zowe-explorer-api using ApiExplorerExtender.initForZowe() and ProfilesCache.registerCustomProfilesType().
3. Call ApiExplorerExtender.reloadProfiles()
4. Without creating a new team configuration file, try and add the new service profile defined in step 2 in the existing team configuration file.

Expected behavior

The newly added service profile should not introduce syntax errors in the team configuration file and should continue to function.

Actual behavior

The zowe team config file will not recognize the profile type since the zowe.schema.json is never updated, introducing syntax errors. ProfilesCache will also fail to load these newly defined profiles.

Screenshots

Desktop (please complete the following information):

• OS:
• Zowe Explorer Version: Zowe-explorer-api v2.14.1
• (Optional) Zowe CLI Version:
• (Optional) Are you using Secure Credential Store?

Additional context

[github-actions]

Thank you for creating a bug report.
We will investigate the bug and evaluate its impact on the product.
If you haven't already, please ensure you have provided steps to reproduce the bug and as much context as possible.

[zFernand0]

Hey @FrankSu1996,
This seems very similar to an issue we encountered in the CICS VSCE.

• Unable to load in Zowe Explorer 2.8 cics-for-zowe-client#34

These issues should be resolved with v2.15.0 of Zowe Explorer 😋

[FrankSu1996]

Thanks, will try with v2.15.0 of Zowe Explorer.

[phaumer]

@zFernand0 where can I find a description of what the fix in 2.15 does? They key use case goes back to the reasons for Zowe switching to team.config files in the first place: an end user does not want to create it themselves, but just receive and use it from an admin. So the use case is

• New developer joins the team and receives team.config file from admin (e.g. via email)
• New developer follows instructions from admin and creates a .zowe folder in home directory and pastes in team.confg.json
• New developer installs Zowe Explorer and other extensions that contribute new profile types. They do not install Zowe CLI or plugins at all.
• The custom profiles just work and the schemas were merged by the extensions without the user doing anything else. (i.e. the user must not be forced to click edit or create team config)

[traeok]

@phaumer This was introduced as a more comprehensive enhancement to fulfill request #2508 - see the 2nd entry in the "New features & enhancements" section of the v2.15 changelog.

There were a few overarching issues that inspired the need for schema management between Zowe client applications:

• Zowe CLI rebuilt the entire schema during zowe plugins install/uninstall, which caused changes by Zowe Explorer extenders to disappear once a CLI plugin was added or removed.
• ProfileInfo needed some additional APIs to allow Zowe Explorer extenders to update the schema on disk.
• Starting with v2.15, we track schema contributions from all Zowe client extenders so that our APIs can check those contributions before adding/removing any types from the schema. As this helps manage profile types across all applications, I've addressed it in the changelogs as "schema management".

As long as extenders are using v2.15 of the Zowe Explorer API, they should be able to successfully contribute to the schema without user intervention.

[FrankSu1996]

Tested with zowe-explorer-api v2.15.0; the schema error sitll persists. My test:

1. Built our vscode extension that contributes a service profile with zowe-explorer-api v2.15.0
2. Opened VsCode with a workspace that already have zowe team configuration files (including schema)
3. Installed our Extension.

The schema's failed to update, and did not include our service profile's metadata. Reloading VsCode did not work either. Creating a new team configuration file still works properly.

[traeok]

Tested with zowe-explorer-api v2.15.0; the schema error sitll persists

@FrankSu1996 To verify, are you also using Zowe Explorer v2.15 for this test? We have added logic to ZE v2.15 during extender registration that will add the new profile types to the schema (if needed). However, without ZE v2.15 the API will not update the schema automatically during registration.

If you want to update the schema manually using the API, we have a function in ProfileInfo named addProfileTypeToSchema that you can use to add to the schema directly.

[FrankSu1996]

Yes, I am using Zowe explorer v2.15.0. I can see the new vscode warning messages about schema versions.

[FrankSu1996]

Is there some sort of new API I need to call for the automated updating of the schema? Or do I just need to register with zowe-explorer-api as always using ApiExplorerExtender.initForZowe() and ProfilesCache.registerCustomProfilesType().

[traeok]

Is there some sort of new API I need to call for the automated updating of the schema? Or do I just need to register with zowe-explorer-api as always using ApiExplorerExtender.initForZowe() and ProfilesCache.registerCustomProfilesType().

You should not have to call a new API as the schema updates are done in ZoweExplorerExtender.initForZowe. Can you attach a screenshot of the warning message you are seeing?

Note that we do not support automatically updating schemas at the project level because they are not easily managed - we only update the global schema. We discussed the potential to implement this as an enhancement, but it would have to be opt-in as we did not want to add files into a project directory to manage schema metadata.

[FrankSu1996]

Using a global level team config also does not work. The warning messages do go away if I specify a version in our profile schema, but seems to have no impact on the schema update functionality.

[traeok]

Using a global level team config also does not work. The warning messages do go away if I specify a version in our profile schema, but seems to have no impact on the schema update functionality.

If possible, can you provide a screenshot of the warning message you've encountered? Based on our discussion so far, there are a couple possibilities in mind:

• The profile type was already present in the Zowe schema before schema management was introduced. The on-disk schema is unversioned, and during the extension registration, the schema is different but also unversioned. In this case, it should still update the global config since we were not previously tracking it using schema management.
• The profile type was recently registered by installing a CLI plug-in using zowe plugins install, so it is already being tracked in the Zowe folder - but, the schema contributed by the VS Code extension is unversioned and has different contents. Having a schema version here would also resolve this scenario, as the schema will only update on-disk if the one from the VS Code extension is newer.

---

As a test for reference, I initialized a team config in sandbox using the Zowe CLI command zowe config init --gc (which generates a global level config & schema).
Then, I opened VS Code and installed Zowe Explorer and the FTP extension. When the FTP extension was registered for the first time, Zowe Explorer added the "zftp" type to my global schema.

[FrankSu1996]

Recording 2024-03-12 120000.zip

I've attached a screen recording of what happens. Basically, we have an extension that contributes a zOpenDebug type of service profile. I create a global team config using the Zowe Explorer views, then install our extension. The schema does not get properly updated with our service profile metadata.

In our extension code, we make the following api calls to zowe-explorer-api:

IApiExplorerExtender.initForZowe()
ProfilesCache.registerCustomProfilesType()
IApiExplorerExtender.reloadProfiles()

and only after those calls successfully complete do we log out "Successfully registered 'zOpenDebug' Zowe service profile type with zowe-explorer-api." as seen in the recording.

[traeok]

Thanks for the recording @FrankSu1996, that helped to spot the problem. This is a bug with ProfileInfo and we will address this ASAP.

Since you have a workspace folder opened containing a project-level config and schema, none of the schemas are updated on disk. This is unintended behavior and describes the bug mentioned above. However, if you close the workspace folder and try to install the extension, it should update the global schema with the profile metadata as expected.

Since this is specific to ProfileInfo, which is a class in Imperative, I'm going to create an issue in the zowe-cli repo as a blocker for this one, and will follow up once the problem is resolved in Imperative. Thanks for your patience and feedback regarding the diagnosis of this issue.

In the meantime, to avoid the "schema warning" messages from occurring in the future, I would recommend deleting the extenders.json file from your Zowe home folder until we release a bug fix.

[FrankSu1996]

No problem, glad I could help Also, I realized that the reason why the schema wasn't getting updated for me (on top of the new issue you just opened) was that extenders.json had already cached zOpenDebug as a profileType, and if that's the case the schema update will not happen. I think this is a seperate issue, since users may install/uninstall an extension multiple times. In that case, if extenders.json is already caching the profileType, then subsequent installs will not update the schema for the user.

When I did delete extenders.json, then installed our extension, the global zowe schema properly updated.

[traeok]

Hi @FrankSu1996 - I just wanted to provide an update.

I've resolved the issue you encountered in Imperative, and it's currently in the review/QA phase. Once the fix is released with the next version of Imperative, I'll prepare a patch version of Zowe Explorer (and its API package) with this fix.

In addition, I've added an optional updateProjectSchema parameter to addProfileTypeToSchema so that extenders can choose to add to project-level schemas by calling the API themselves. However, we do not currently track the latest schema changes to project-level schemas, meaning that extenders will choose to update a project-level schema at their discretion. We are evaluating a project-level opt-in for schema management as a future enhancement.

[traeok]

Closed as completed in #2786

• The global schema will now be updated in cases where a workspace folder is opened and a project-level config is present
• Project-level updates can now be done by extenders calling the ProfileInfo.addProfileTypeToSchema API with the updateProjectSchema optional parameter set to true