[api] Allow extensions to use the syntax highlighter

[sebastian-lenz]

It would be great if the extension API would offer a way to access the syntax highlighter of VS Code.

I am currently building an extension that offers an inline help system, I use a WebviewPanel to display the help in the editor. For consistency it would be great if we could format code samples the same way the editor does but I currently see no way to do that.

[aeschli]

In what format would we give the syntax highlighted code give back?

[sebastian-lenz]

Anything to work with would be great.

My first intention was to search for an API that could resolve a MarkupContent or MarkedString object to html. MarkupContent is the common way extension pass text and code snippets that will be displayed to the user (e.g. from a hover or signature help provider).

This way markdown content could be rendered as well. Currently extension have to ship their own markdown parser (which VS Code already ships with at least two I know of) and if they want to provide highlighted code something like highlight.js and separate definitions for the language (or somehow integrate vscode-textmate). Before posting the issue here I asked it on the Gitter channel and another user indicated that they actually do it like so in their extension.

I also noted that the syntax color scheme of the currently activated theme is not available in the webview and cannot be accessed by extensions either.

[sebastian-lenz]

So, just to add this, all I would need would be a rendered html string.

[aeschli]

Adding @mjbvz as the webview expert.

[mjbvz]

@aeschli To get vscode-like syntax highlighting in a webview we would need an api like @sebastian-lenz proposes. For the markdown preview specifically, since many websites use highlightjs for highlighting code anyways, using highlightjs in the markdown preview isn't to much of a problem imo

[octref]

You might be interested in http://shiki.matsu.io/. It doesn't use oniguruma so no native dependecy, should be quite easy to integrate into your extension.

[usernamehw]

Can shiki be already used for markdown preview code blocks (to solve #59483)?

[octref]

/cc @mjbvz

[sebastian-lenz]

Thanks for the tip, but my core issue is the fact that when including a syntax highlighter in an extension, the results are not visually consistent within VS Code. Although Shiki will use the same specification, we cannot access the theme colors the user has picked.

[DonJayamanne]

We have a similar issues within the python extension
https://github.com/Microsoft/vscode-python/issues/3773
https://github.com/Microsoft/vscode-python/issues/4197
https://github.com/Microsoft/vscode-python/issues/4026
https://github.com/Microsoft/vscode-python/issues/4066

[APerricone]

The token colors aren't even exposed as CSS variables... over 400 variables and the tokens are missing?

[DonJayamanne]

Thought I'd mention this (even though it has been mentioned before #56356 (comment))
I'm currently using http://shiki.matsu.io and this works perfectly well (for generating HTML with styles/themes & custom tokens from VS Code & VS Code extensions).

@octref thanks for the wonderful package

[DarkTrick]

Isn't this also a little related with the question here on StackOverflow? If VSCode would provide the possibility to use the editor object - perhaps including some options like readonly - this ticket would be resolved as well?

[ylg]

@DonJayamanne Would you elaborate on that a little? I don't see how Shiki solves for this (other than by coincidentally supporting themes that may or may not match a user's visual config).

[rangav]

I also require this feature to be implemented, Currently unable to use themes for my extension Thunder Client in the response syntax highlighting

I have hard-coded the default theme colors for now

[DonJayamanne]

I don't see how Shiki solves for this (

Sorry I missed this. I had to write code to load the theme files into Shiki.

[tjpalmer]

The mermaid markdown example by @mjbvz shows where to plug in custom syntax highlighting in markdown preview. And vscode-textmate can be used to adapt tokens to hljs class names. It would be nice if vscode did this automatically, but it's possible to wire up manually at this point. And this works better than shiki for multiple reasons, including because hljs class names pick up matching theme colors automatically.

[Hyperclaw79]

Any progress on this?
While manually wiring in the themes would be doable for a specific language, it will be hard to support a generic case which can accept any language for syntax highlighting.
Moreover, any highlighting changes added by other language extensions will not be reflected this way.

Take a look at this generic Sticky Note which allows codeblocks to be created by copying a selection. Even after copying in the base css files like markdown.css and highlight.css, there's a lot of inconsistency between the note and the actual file.

[dgss]

May I ask how you could make shiki work on an vscode webview extension (that uses webpack)? VSCode webview extensions are not able to make requests which seems necessary for the onig.wasm to be loaded and same for the themes. Themes can be solved somehow but i really wonder to know how you make the shiki work?

[mjbvz]

@aeschli @alexdima For webviews and notebook renderers, we'd like an API where we can give VS Code a string of code and a language identifier, and get back html+css with syntax highlighting for that code. This would work similarly to the existing tokenizeToString function we use in core

Some use cases:

• For webviews, I'd like to use this to highlight code blocks in the markdown preview: Use VS Code's syntax highlighting in Markdown preview #91279

• For notebook renderers, I had to put together a solution for this in core that uses tokenizeToString. It would be nice to move this logic out of core and also let other notebook renderers access this API

[shiqimei]

Hi @mjbvz ! Do you know how GitHub Copilot Chat works? Since its codeblock can be adapted to the theme user selected.

[dgss]

Hi @mjbvz ! Do you know how GitHub Copilot Chat works? Since its codeblock can be adapted to the theme user selected.

They have two different things, in the chat window they are not 100% matching the editor colors. However for the big answer blocks they are using a new editor tab which already has the editor colors as i've seen.
As i've checked last week there is still no css custom properties match with editor colors. But maybe the way they are using is more proper, instead of using a codeblock inside the extension window, using a dedicated editor tab for it. Unfortunately it doesn't fit into our needs.

[Hyperclaw79]

Any updates on this? Will this feature request be considered or has it been abandoned?

[matv-stripe]

I think this would be a great feature to add!

[RedCMD]

tokens may be provided when TreeSitter is fully implemented

• Explore using tree sitter for syntax highlighting #210475
• Support syntax highlighting with tree-sitter #50140

otherwise you'll need to implement your own parser with like vscode-textmate or tree-sitter etc
microsoft/vscode-textmate#244