HEX / Text Source Code Editing

[davidkcoblentz]

Description of the enhancement requested

Environment

Version: 1.62.0 (user setup)
Commit: b3318bc0524af3d74034b8bb8a64df0ccf35549a
Date: 2021-11-03T15:23:01.379Z
Electron: 13.5.1
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Windows_NT x64 10.0.17763

Text / Hex Editing View

I would like to see the source that I am editing with the hex equivalent below each character like the mainframe has in it's editor. An example is seen below. I understand there are dump style extensions but I would prefer a hex editor like the mainframe has. There are times when coding MFS or assembler or even COBOL programs that may have embedded hex values, that are non-displayable in the source.

Hex Editing MFS Source example that has embedded hex values

MFS Source code that contains Hex values

[FALLAI-Denis]

Hi,

Personally, I think the use of hex characters directly in the sources should be avoided.
This already poses problems on the mainframe if the encoding are incorrectly declared. But things are worse with the EBCIDIC / UTF8 (or ASCII) conversions that occur for work in VS Code.
After conversion, the characters can lose all meaning.

VS Code, and Z Open Editor, is a source code editor (source programs editor). It is not a data file editor. The files handled must only use displayable characters.

You must use the X'nn' syntax in COBOL, or equivalent in other languages, to introduce hex values in the source codes.

However, EBCDIC characters x'4F ', x'6F' should be converted to UTF8 (or ASCII) and displayed correctly in VS Code / Z Open Editor. For me, they are converted to x'7C', x'3F' in UTF8 / ASCII.

There seems to be another problem with your configuration.
Is the encoding conversion correctly set at Zowe CLI?
Is the correct encoding declared in VS Code?
Auto detection problem of the encoding in VS Code?
Try "reopen with encoding" in VS Code.

Otherwise to see the values of hexa or non-displayable characters in VS Code, see the extension Gremlins tracker for Visual Studio Code.

[phaumer]

Yes, on the VS Code side we are working with UTF-8 only. Conversions are handled for us by z/OSMF or RSE API. Microsoft has a Hex extension as well, but this will not help much here I guess.

I would not think that we would try this any time soon for Z Open Editor. As you might know we are also partnering with other companies for data editing for our other editors, but unfortunately VS Code support is behind.

[davidkcoblentz]

Thank you both so much for your quick answers! I see your point that it is a source editor. I am still checking out why the original developers put non-displayable bytes of data in the MFS. I will document it here if I come to understand why. I am suspecting MFS 3270 attribute bytes but am still checking.

I also noticed when I brought my REXX source code into VS Code from the mainframe that the '|' (vertical bar), used as the logical OR operator, is not translating properly so it shows as a non-displayable character. I am also investigating to see if I should be using a different code-page other than English 037 to ASCII 437 when downloading from the mainframe.

[FALLAI-Denis]

Hi,

For EBCDIC / UTF8 character conversions, see
The ebcdic x'CA 'character (IBM-1147) is missing after transfer with ZOWE Explorer (with encoding 1147) / soft hyphen x'AD' not rendered by Chromium

See also if you have the PTFs on z/OSMF which addresses the character conversions problem (by past z/OSMF only worked with IBM-1047, ccsid USS), and if you also have a version of Zowe CLI which supports character conversions (--encoding).

[phaumer]

For RSE API users there is documentation on conversion options here: https://ibm.github.io/zopeneditor-about/Docs/mvs_encoding.html If you find any conversion problems with RSE let us know here as well.

[davidkcoblentz]

Thank you! I just realized you had questions above that I didn't answer.
Questions:
There seems to be another problem with your configuration.
Is the encoding conversion correctly set at Zowe CLI? We are in the process of configuring zOSMF and getting ZOWE connectivity. I am currently downloading each source member with my 3270 Terminal emulator Aviva. I will address this when we get zOSMF / Zowe configured
Is the correct encoding declared in VS Code? Same as above
Auto detection problem of the encoding in VS Code? Same as above
Try "reopen with encoding" in VS Code. # Same as above

[FALLAI-Denis]

Hi @davidkcoblentz

This is probably the origin of the problem.
When you download files, be sure to convert encoding from the ccsid used for your mainframe (the ccsid defined in your 3270 emulator) to UTF8 (without bom).

If you convert from EBCDIC IBM-037 to ASCII IBM-437, it's an error.
If you don't have UTF8, try Windows-1252 in replacement of UTF8.

If you use "ind$file" look at parameters of your emulator, (ftp should be a better solution).

I think you can use Zowe CLI "zftp" without z/osmf. Otherwise use client ftp of your workstation.

[davidkcoblentz]

Great! Thanks, I will try the CLI to see. Thank you, ***@***.*** From: Denis FALLAI ***@***.***> Sent: Wednesday, November 17, 2021 12:19 PM To: IBM/zopeneditor-about ***@***.***> Cc: Coblentz, David ***@***.***>; Mention ***@***.***> Subject: Re: [IBM/zopeneditor-about] HEX / Text Source Code Editing (Issue #183) Hi @davidkcoblentz<https://urldefense.com/v3/__https:/github.com/davidkcoblentz__;!!OgnrLJOUc50AAzM!eO7dW3UjHFRmWdayjcAlwfxP8z5TO3yfRW70m7HWON2-JVsblxlx2qEU3VngQxxUhg$> This is probably the origin of the problem. When you download files, be sure to convert encoding from the ccsid used for your mainframe (the ccsid defined in your 3270 emulator) to UTF8 (without bom). If you use "ind$file" look at parameters of your emulator, (ftp should be a better solution). I think you can use Zowe CLI "zftp" without z/osmf. Otherwise use client ftp of your workstation. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/IBM/zopeneditor-about/issues/183*issuecomment-971896783__;Iw!!OgnrLJOUc50AAzM!eO7dW3UjHFRmWdayjcAlwfxP8z5TO3yfRW70m7HWON2-JVsblxlx2qEU3Vnx4k5NwQ$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ATC2YG7H6EY7WOCIBM4LE6TUMP53LANCNFSM5H3HLS6Q__;!!OgnrLJOUc50AAzM!eO7dW3UjHFRmWdayjcAlwfxP8z5TO3yfRW70m7HWON2-JVsblxlx2qEU3VnmLjODpA$>. Triage notifications on the go with GitHub Mobile for iOS<https://urldefense.com/v3/__https:/apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!OgnrLJOUc50AAzM!eO7dW3UjHFRmWdayjcAlwfxP8z5TO3yfRW70m7HWON2-JVsblxlx2qEU3Vl-7wXefw$> or Android<https://urldefense.com/v3/__https:/play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!OgnrLJOUc50AAzM!eO7dW3UjHFRmWdayjcAlwfxP8z5TO3yfRW70m7HWON2-JVsblxlx2qEU3VkGZfEsLA$>.

[FALLAI-Denis]

PS :

If you want to do an FTP transfer directly with z/OS, you may encounter problems because the z/OS FTP server does not accept to switch from single-byte encoding (IBM-037) to multibyte encoding (UTF-8 / IBM-1208).

One solution would be to transfer via FTP from IBM-037 to IBM-1252, then apply a local transformation to UTF8, for example by opening the transferred file in VS Code in Windows-1252 and then saving it in UTF-8.

[FALLAI-Denis]

Hi @davidkcoblentz, @phaumer

I remain convinced that we cannot transfer a file containing non-displayable characters (binary data or packaged decimal) and apply an encoding translation to it, (go from EBCDIC to UTF8 and vice versa).

Nevertheless, the editing of (small) data files remains a common need, (for large data files it is better to turn to a specific tool which is no longer direct file editing).

For this, I made two proposals for the development of the Zowe Explorer for VS Code extension, which you could consider and support by voting:

• Not all mvs data sets necessarily use the same ccsid #1589
• Editor in hexadecimal dump mode with EBCDIC interpretation #1590

Thanks.

[davidkcoblentz]

Thank you! Great ideas! I up-voted both of these. Thank you, ***@***.*** From: Denis FALLAI ***@***.***> Sent: Tuesday, November 30, 2021 11:59 PM To: IBM/zopeneditor-about ***@***.***> Cc: Coblentz, David ***@***.***>; Mention ***@***.***> Subject: Re: [IBM/zopeneditor-about] HEX / Text Source Code Editing (Issue #183) Hi @davidkcoblentz<https://urldefense.com/v3/__https:/github.com/davidkcoblentz__;!!OgnrLJOUc50AAzM!bMroF6QitA_b3UQx_7fNhEfZHmri84vZweakpk-6_AK9iQLJJH6UzC1kCUVJtllG9w$>, @phaumer<https://urldefense.com/v3/__https:/github.com/phaumer__;!!OgnrLJOUc50AAzM!bMroF6QitA_b3UQx_7fNhEfZHmri84vZweakpk-6_AK9iQLJJH6UzC1kCUXgWSf4XA$> I remain convinced that we cannot transfer a file containing non-displayable characters (binary data or packaged decimal) and apply an encoding translation to it, (go from EBCDIC to UTF8 and vice versa). Nevertheless, the editing of (small) data files remains a common need, (for large data files it is better to turn to a specific tool which is no longer direct file editing). For this, I made two proposals for the development of the Zowe Explorer application, which you could consider and support by voting: * Not all mvs data sets necessarily use the same ccsid #1589<https://urldefense.com/v3/__https:/github.com/zowe/vscode-extension-for-zowe/issues/1589__;!!OgnrLJOUc50AAzM!bMroF6QitA_b3UQx_7fNhEfZHmri84vZweakpk-6_AK9iQLJJH6UzC1kCUX01cOLFQ$> * Editor in hexadecimal dump mode with EBCDIC interpretation #1590<https://urldefense.com/v3/__https:/github.com/zowe/vscode-extension-for-zowe/issues/1590__;!!OgnrLJOUc50AAzM!bMroF6QitA_b3UQx_7fNhEfZHmri84vZweakpk-6_AK9iQLJJH6UzC1kCUVoshIBhw$> Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https:/github.com/IBM/zopeneditor-about/issues/183*issuecomment-983345264__;Iw!!OgnrLJOUc50AAzM!bMroF6QitA_b3UQx_7fNhEfZHmri84vZweakpk-6_AK9iQLJJH6UzC1kCUVPiYbskA$>, or unsubscribe<https://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/ATC2YG3LZBEPNJZPXIMC5BDUOXBT5ANCNFSM5H3HLS6Q__;!!OgnrLJOUc50AAzM!bMroF6QitA_b3UQx_7fNhEfZHmri84vZweakpk-6_AK9iQLJJH6UzC1kCUXR4c7Gmg$>. Triage notifications on the go with GitHub Mobile for iOS<https://urldefense.com/v3/__https:/apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!OgnrLJOUc50AAzM!bMroF6QitA_b3UQx_7fNhEfZHmri84vZweakpk-6_AK9iQLJJH6UzC1kCUVs6fevOA$> or Android<https://urldefense.com/v3/__https:/play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!OgnrLJOUc50AAzM!bMroF6QitA_b3UQx_7fNhEfZHmri84vZweakpk-6_AK9iQLJJH6UzC1kCUUaIZdE4g$>.