Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[shortfin llm] Separate server configs and exported model configs #727

Open
renxida opened this issue Dec 26, 2024 · 0 comments
Open

[shortfin llm] Separate server configs and exported model configs #727

renxida opened this issue Dec 26, 2024 · 0 comments

Comments

@renxida
Copy link
Contributor

renxida commented Dec 26, 2024

Currently, the prefix sharing algorithm is configured from config.json, which makes it necessary to edit it every time we export a model before server would work.

config.json should be only used for exported config keys from sharktank.export_paged_llm_v1

server.py should take additional options either from a separate config file or from the commandline arguments.

Ideally we figure out some project-wide standards for configuration management (e.g. json / yml files in ~/.shortfin with cmdline options to override them).

@renxida renxida changed the title [shortfin llm] Separate config setup for server [shortfin llm] Separate server configs and exported model configs Dec 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant