Questions about Qwen3-4B model settings

[TimeLovercc]

Hi CURE authors. Great job! I want to reproduce the results in Table 1. However, when I use the exact dataset and similar config with Qwen3-4B model. My performce for Qwen3-4B model is consistenly lower than the performance in Table 1. For example, my performance for Qwen3-4B for MBPP for Code column is 77.57%. The number in table is 88.4%. I am curious about the exact setting to reproduce your baseline performance for Qwen3-4B, can you give me some guidance? I want to know

• do you use this version of Qwen3-4B link?
• what is your prompt for code generation?

Thanks again for your great work.

[yinjjiew]

Hi, thank you for your interests in this work! Sorry for the delay, I just found this issue.

Yes, that's the model we used. May I ask what's the response length limit you set? For long CoT model, like Qwen3-4B, it should have 20000 as response length limit (or potentially 30000). The prompt for code generation is just the prompt in our evaluation_config.py.