About the temperature

[jdp22]

Thank you for your great work! May I ask the temperature used during the LLM inference?

[jdp22]

I have tried to test Qwen-2.5B-Instruct on the Math500, 1024 max token, 0.7 temperature. On the TT-MV, I found the the performance is 70.00%, which is ~10% higher than what is reported in the paper. This performance is near the RandOpt and ES+TT-MV.

[sunrainyg]

We used the default setting with temperature = 1. For TT-MV, as you pointed out, performance could likely be improved a lot by trying different temperatures, as we did not tune this parameter. There is also substantial room for improvement in RandOpt, since its hyperparameters were not tuned. I believe that with smarter approaches, methods in prompt space (e.g., GEPA), weight space, and output space (sampling), as well as their combinations, may all converge.