Regarding using Llama 7b and 13b

[TheAthleticCoder]

Hey!

Does anyone have any idea how to use Llama-7b in vllm?
My computing resources do not allow for using the 13b version :(

[amoghmishra-sl]

Tell us the hardware spec?

[TheAthleticCoder]

So I'm using ADA resources provided by my institute, they list the specs as:

"Ada cluster consists of ninety-two Boston SYS-7048GR-TR nodes equipped with dual Intel Xeon E5-2640 v4 processors, providing 40 virtual cores per node, 128 GB of 2400MT/s DDR4 ECC RAM, and four Nvidia GeForce GTX 1080 Ti GPUs, providing 14336 CUDA cores, and 44 GB of GDDR5X VRAM or four Nvidia GeForce RTX 2080 Ti GPUs providing 17408 cores, and 44 GB of GDDR6 VRAM. The nodes are connected to each other via a Gigabit Ethernet network. All compute nodes have a 1.8 TB local scratch and a 960 GB local SSD scratch. The compute nodes are running Ubuntu 18.04 LTS. SLURM software is used as a job scheduler and resource manager. The aggregate theoretical peak performance of Ada is 70.66 TFLOPS (CPU) + 4588 TFLOPS (FP32 GPU)."

[TheAthleticCoder]

@amoghmishra-sl

[guaguace]

Have you encountered a situation where using llama results in the output of /n or other characters being repeated many times?How should I go about resolving this issue?

[phebing]

Maybe a little late, but just stumbled across. Also had a similar problem with codeLlama.
Fixed it by adjusting my ropeconfig:
localAI yaml style (not sure how it is with vllm):
rope_freq_base: 1000000
rope_freq_scale: 1.0

(my original problem / answer: https://huggingface.co/TheBloke/CodeLlama-34B-Instruct-GGUF/discussions/4#6581785198aa9fcdd280dac3 )

[amoghmishra-sl]

Yes, I have. Add EOS term or reduce your max token limit. I think that is what sovled the issue but I am 100% sure about it because it was a long time ago. Lemme know what fixes it though.

…

On Fri, Nov 24, 2023 at 11:14 AM guaguace ***@***.***> wrote: Have you encountered a situation where using llama results in the output of /n or other characters being repeated many times?How should I go about resolving this issue? — Reply to this email directly, view it on GitHub <#637 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZRQUWKGQUP4WSEXVQ45OGTYGBJPVAVCNFSM6AAAAAA3ACPVCWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMNJYGA3TQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>