Reward Not Increasing While trying to use Qwen 2.5 - 0.5B,1.5B Instruct models for training on musique with re-search code

[prasadke20]

Hi I am trying to use smaller LLMs to train re-search with musique data, although i am facing an issue with no reward increase
Here is the config i am using
bash train.sh --train_batch_size 48 --ppo_mini_batch_size 48 --prompt_template_name re_search_template_sys --actor_model_path Qwen/Qwen2.5-0.5B-Instruct --search_url url --nnodes 1 --n_gpus_per_node 2 --save_freq 5 --test_freq 5 --total_epochs 2 --save_path ReSearch --train_files ReSearch/data/musique/train.parquet --test_files ReSearch/data/musique/test.parquet --apply_chat True

Also after a point searches sent to retriever are blank searches

[prasadke20]

Also please suggest any optimizations that can be done so that I can use 7B instruct model with two L40 gpus of 46gb each

[AnselCmy]

Hi @prasadke20. Training reasoning abilities with search is indeed challenging on smaller models such as 0.5B or 1.5B parameters. Training larger models like 7B is not feasible on two GPUs with 46GB each, as you would typically need at least 16×7GB (2× for gradients, 2× for parameters, and 12× for optimizer states) during training. You might consider using LoRA, which is currently supported by the original verl implementation.

[prasadke20]

Hi @AnselCmy , thanks for your reply . Can i use LORA by changing some parameters in your current code of re-search, or does it needs to be built seperately? Can u guide me on the changes that can be done to achieve this?