Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Could not start backend on Jetson AGX Orin #466

Closed
2 of 4 tasks
HiddenPeak opened this issue Jan 4, 2025 · 3 comments · May be fixed by #467
Closed
2 of 4 tasks

Could not start backend on Jetson AGX Orin #466

HiddenPeak opened this issue Jan 4, 2025 · 3 comments · May be fixed by #467

Comments

@HiddenPeak
Copy link

System Info

Jetson Orin AGX 64G
SM 8.7
cargo 1.75.0 (1d8b05cdd 2023-11-20)

Information

  • Docker
  • The CLI directly

Tasks

  • An officially supported command
  • My own modifications

Reproduction

  1. curl https://sh.rustup.rs -sSf | sh
  2. sudo apt-get install libssl-dev gcc -y
  3. git clone https://github.com/huggingface/text-embeddings-inference.git
  4. cd text-embeddings-inference
  5. cargo install --path router -F candle-cuda -F http --no-default-features
  6. model=BAAI/bge-reranker-large
  7. text-embeddings-router --model-id $model --port 8080

Expected behavior

Could not start Candle backend: Could not start backend: Runtime compute cap 87 is not compatible with compile time compute cap 87
Error: Could not create backend

Caused by:
    Could not start backend: Could not start a suitable backend
@HiddenPeak
Copy link
Author

in backends/candle/src/compute_cap.rs

I try to add SM87 support, and I need some time to test it.

If anyone want to try it,use [this] (https://github.com/HiddenPeak/text-embeddings-inference ) befor it merged.

@HiddenPeak HiddenPeak mentioned this issue Jan 4, 2025
Open
1 task
@HiddenPeak
Copy link
Author

74f61043ee2b7a9da75e93547f397b7
6519f72d3f88800fd8e89705b2d5fe1
fc2f46532bbe2545f4f7f2c55388190
It works

@HiddenPeak
Copy link
Author

on Jetpack 6.1 with cuda 12.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add Jetson Orin support HiddenPeak/text-embeddings-inference
1 participant
@HiddenPeak