[Feature] Will multi-modal models support W8A8 quantization in the future?

[MenglingD]

Motivation

Our business model (Internvl 2-26B) outputs very few tokens (1-2 tokens) after prompt optimization, which can be considered as only the prefill stage. Therefore, we hope to use W8A8 quantization to speed up the inference process. However, we found that lmdeploy does not support W8A8 inference for multi-modal models: #2042

Could you please explain why W8A8 quantization is not supported for multi-modal models? Is it due to model accuracy concerns?

Related resources

No response

Additional context

No response

[AllentDan]

There is a pull request #2308 handling this.

[MenglingD]

There is a pull request #2308 handling this.

Thanks, we will try it later and provide timely feedback if any issues arise.

[MenglingD]

There is a pull request #2308 handling this.

Also, I'd like to ask if TurboMind plans to support the w8a8 feature for VLM (Vision-Language Models) models in the future?

[AllentDan]

Turbomind is only responsible for llm. Vision model in lmdeploy used pytorch.

[MenglingD]

Turbomind is only responsible for llm. Vision model in lmdeploy used pytorch.

Excuse me, I made a mistake in my statement. What I actually wanted to ask is whether TurboMind has any plans to support W8A8? Because from the documentation, TurboMind doesn't currently support W8A8：支持的模型

[AllentDan]

Yes, there is a plan that TurboMind will support W8A8.