sample params difference between vLLM and Huggingface transformers

[zhaoyang-star]

We know evaluation is very important for a production env. I tested the code completion ability of starcoder-15b by HumanEval using vLLM and HF as two different backends.

I found it is hard to get the same pass@1 using vLLM and HF transformers. So how to get the same pass@1 as HF when I use vLLM?

Backends	pass@1
HF	0.38
vLLM	0.33

There may be two reasons and the first may be the main part.

• Different definations of sampling params
• Kernel impls diff.

For the first one. I compared the main params as following.

HF	Default value	vLLM	Default value
num_return_sequences	1	n	1
num_beams	1	best_of	1
repetition_penalty	1.0	frequency_penalty	0.0
		presence_penalty	0.0
temperature	1.0	temperature	1.0
top_p	1.0	top_p	1.0
top_k	50	top_k	-1
max_new_tokens	None	max_tokens	16

I use the same sampling params in vLLM and HF.

# vLLM config
sampling_params = SamplingParams(
    best_of=9,
    n=9,
    frequency_penalty=0.0,
    temperature=0.2,
    max_tokens=1024,
)

# HF
        outputs = model.generate(
            input_ids=inputs,
            num_beams=9,
            num_return_sequences=9,
            repetition_penalty=1.0,
            temperature=0.2,
            max_new_tokens=1024,
            remove_invalid_values=True,
        )

[imiraoui]

Did you figure it out?

[chenxu2048]

Here are some merged pull requests focusing on the alignment of sampling parameters and results: #753, #1424, #1577, and #1885.