[WIP] Add GPU Optimizer deployment and update configurations #480
Conversation
config/gpu-optimizer/deployment.yaml
Outdated
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: pod-autoscaler |
There was a problem hiding this comment.
name should change to something related to gpu-optimizer etc
config/gpu-optimizer/deployment.yaml
Outdated
| kind: Role | ||
| metadata: | ||
| namespace: aibrix-system | ||
| name: deployment-reader |
There was a problem hiding this comment.
Similarly, names should be updated
| maxReplicas: 10 | ||
| targetMetric: "avg_prompt_throughput_toks_per_s" # Ignore if metricsSources is configured | ||
| metricsSources: | ||
| - endpoint: gpu-optimizer.aibrix-system.svc.cluster.local:8080 |
There was a problem hiding this comment.
these files are being refactored in #477. we probably can change it later.
config/gpu-optimizer/deployment.yaml
Outdated
| @@ -0,0 +1,75 @@ | |||
| apiVersion: v1 | |||
There was a problem hiding this comment.
can we break these files into more granular files?
- deployment.yaml
- rbac.yaml
- service.yaml
config/gpu-optimizer/deployment.yaml
Outdated
| - protocol: TCP | ||
| port: 8080 | ||
| targetPort: 8080 | ||
| nodePort: 30008 |
| containers: | ||
| - name: gpu-optimizer | ||
| image: aibrix/runtime:nightly | ||
| command: ["python", "-m", "aibrix.gpu_optimizer.app"] |
There was a problem hiding this comment.
if the server is down, what's the autoscaler behavior? Have you tested such behaviors?
nwangfw
force-pushed
the
gpu-optimizer-orchestration
branch
from
c881f59 to
90cd690
Compare
* Move huggingface_token to config.json Add missing zscaler root CA to image for huggingface lib to download tokenizer model successfully. * Remove huggingface token --------- Co-authored-by: Jingyuan Zhang <[email protected]>
* adding timestamp and prompt in/output length to traces * name fix; plotting script fix * update README * addressing comments * addressing comments * add sample workload * add sample workload * update file format * update jsonl option --------- Co-authored-by: Le Xu <[email protected]>
…l namespaces with model label.
|
@nwangfw I fixed k8s access problem on branch issues/484_Controller_failed_to_fetch_metrics_from_MetricSource. I think we should merge changes together. |
…_failed_to_fetch_metrics_from_MetricSource # Conflicts: # development/simulator/deployment-a100.yaml # development/simulator/deployment-a40.yaml
|
Sorry, it looks like this PR includes merged changes from the main. Maybe I should start another PR for a clear view. |
|
As #494 merged, let's close this PR |
Pull Request Description
In https://github.com/aibrix/aibrix/pull/430/files, most of the component are dockerized but have not moved to kubernetes environment, we move them under config/default scope and make sure it can be installed along with other aibrix components.
Related Issues
Resolves: #459
Important: Before submitting, please complete the description above and review the checklist below.
Contribution Guidelines (Expand for Details)
We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:
Pull Request Title Format
Your PR title should start with one of these prefixes to indicate the nature of the change:
[Bug]: Corrections to existing functionality[CI]: Changes to build process or CI pipeline[Docs]: Updates or additions to documentation[API]: Modifications to aibrix's API or interface[CLI]: Changes or additions to the Command Line Interface[Misc]: For changes not covered above (use sparingly)Note: For changes spanning multiple categories, use multiple prefixes in order of importance.
Submission Checklist
By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.