New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

如何使用下载后本地模型进行Inference #22

Open

ismeyueyue opened this issue Jan 22, 2025 · 0 comments

ismeyueyue commented Jan 22, 2025

你好，非常感谢项目开源。最近发现使用下载后的模型推理失败。

背景：因为服务器无法联网，同时huggingface提供过多的模型，所以我下载个别的模型并上传到模型目录下，在使用FP8量化模型Inference过程中，我将'MODEL_PATH'设置为模型的本地目录，发现推理失败。

请问上面这个是什么原因？具体如何才能正确加载本地模型进行推理呢？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment