Bitsandbytes models stopped working in the new 0.7.1 version

[davefojtik]

In 0.7.0 everything works fine. But when I update to 0.7.1 with the same models, code etc. it's broken.

The logs spam a lot of MLA is not supported with bitsandbytes quantization. Disabling MLA., even though the VLLM_MLA_DISABLE env variable is set to true.

Then after the Capturing cudagraphs part, the vllm returns the following error:

"Uncaught exception | <class 'TypeError'>; BatchDecodeWithPagedKVCacheWrapper.plan() got an unexpected keyword argument 'kv_data_type'; <traceback object at 0x6ffe180f4ac0>;

Every update I go through the engine arguments and enviroment variables in the documentation to see if there's something new or changed, but this time I didn't see anything that could cause this. Did I miss some major change?

Here's the full log (we're using custom VLLM on RunPod serverless):

INFO 02-06 04:28:25 __init__.py:183] Automatically detected platform cuda.
Starting Job...
VLLM Engine not initialized, starting...
WARNING 02-06 04:28:31 config.py:605] bitsandbytes quantization is not fully optimized yet. The speed can be slower than non-quantized models.
WARNING 02-06 04:28:31 config.py:975] MLA is not supported with bitsandbytes quantization. Disabling MLA.
INFO 02-06 04:28:31 llm_engine.py:232] Initializing a V0 LLM engine (v0.7.1) with config: model='/workspace/models/Lamarck-14B-v0.7-bnb-4bit/', speculative_config=None, tokenizer='/workspace/models/Lamarck-14B-v0.7-bnb-4bit/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, override_neuron_config=None, tokenizer_revision=main, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=bitsandbytes, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=bitsandbytes, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Lamarck-14B-v0.7-bnb-4bit, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[2,1],"max_capture_size":2}, use_cached_outputs=False,
WARNING 02-06 04:28:31 config.py:975] MLA is not supported with bitsandbytes quantization. Disabling MLA.
INFO 02-06 04:28:31 cuda.py:169] Using FlashInfer backend.
INFO 02-06 04:28:31 model_runner.py:1111] Starting to load model /workspace/models/Lamarck-14B-v0.7-bnb-4bit/...
INFO 02-06 04:28:31 loader.py:1078] Loading weights with BitsAndBytes quantization.  May take a while ...
2025-02-06T04:28:31.736547334Z 
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  1.41it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.40it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.40it/s]
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  1.22it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.23it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.23it/s]
INFO 02-06 04:28:34 model_runner.py:1116] Loading model weights took 9.3543 GB
WARNING 02-06 04:28:35 config.py:975] MLA is not supported with bitsandbytes quantization. Disabling MLA.
WARNING 02-06 04:28:35 config.py:975] MLA is not supported with bitsandbytes quantization. Disabling MLA.
WARNING 02-06 04:28:42 config.py:975] MLA is not supported with bitsandbytes quantization. Disabling MLA.
WARNING 02-06 04:28:42 config.py:975] MLA is not supported with bitsandbytes quantization. Disabling MLA.
INFO 02-06 04:28:42 worker.py:266] Memory profiling takes 7.71 seconds
INFO 02-06 04:28:42 worker.py:266] the current vLLM instance can use total_gpu_memory (23.64GiB) x gpu_memory_utilization (0.98) = 23.17GiB
INFO 02-06 04:28:42 worker.py:266] model weights take 9.35GiB; non_torch_memory takes 0.08GiB; PyTorch activation peak memory takes 4.50GiB; the rest of the memory reserved for KV Cache is 9.24GiB.
INFO 02-06 04:28:43 executor_base.py:108] # CUDA blocks: 1576, # CPU blocks: 170
INFO 02-06 04:28:43 executor_base.py:113] Maximum concurrency for 32768 tokens per request: 1.54x
WARNING 02-06 04:28:43 config.py:975] MLA is not supported with bitsandbytes quantization. Disabling MLA.
WARNING 02-06 04:28:43 config.py:975] MLA is not supported with bitsandbytes quantization. Disabling MLA.
INFO 02-06 04:28:43 model_runner.py:1435] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
WARNING 02-06 04:28:43 config.py:975] MLA is not supported with bitsandbytes quantization. Disabling MLA.
Capturing CUDA graph shapes:   0%|          | 0/2 [00:00<?, ?it/s]
Capturing CUDA graph shapes:   0%|          | 0/2 [00:00<?, ?it/s]
{"requestId": null, "message": "Uncaught exception | <class 'TypeError'>; BatchDecodeWithPagedKVCacheWrapper.plan() got an unexpected keyword argument 'kv_data_type'; <traceback object at 0x6ffe180f4ac0>;", "level": "ERROR"}
[rank0]:[W206 04:28:44.311381868 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())

[davefojtik]

Solved. The problem was installing an outdated Flashinfer in my container image. Version 0.2.0.post2 works fine.