vLLM error when Loading LoRa Adapter on API request #309
Description
I first train meetkai/functionary-small-v3.2 using deepspeed functionary/train/train_lora.py with your provided params. Then I run the following script to serve LoRA adapters at startup
python server_vllm.py --model meetkai/functionary-small-v3.2 --enable-lora --lora-modules {name}={path} --host 0.0.0.0 --port 8000
The issue comes when I try a request to the model vLLM fails with the following message
INFO: 131.226.33.184:53594 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/vllm/lora/worker_manager.py", line 105, in _load_adapter
lora = self._lora_model_cls.from_local_checkpoint(
File "/opt/conda/lib/python3.10/site-packages/vllm/lora/models.py", line 221, in from_local_checkpoint
peft_helper = PEFTHelper.from_dict(config)
File "/opt/conda/lib/python3.10/site-packages/vllm/lora/peft_helper.py", line 80, in from_dict
return cls(**filtered_dict)
File "<string>", line 14, in __init__
File "/opt/conda/lib/python3.10/site-packages/vllm/lora/peft_helper.py", line 45, in __post_init__
self._validate_features()
File "/opt/conda/lib/python3.10/site-packages/vllm/lora/peft_helper.py", line 42, in _validate_features
raise ValueError(f"{', '.join(error_msg)}")
ValueError: vLLM only supports modules_to_save being None.