Getting same output text across different UQ methods

[nehiridil]

Hi!
I am experimenting the performance of UQ methods. As the below code shows, I am calling estimate_uncertainty multiple times and get score for a selected UQ method. Will the generated outputs from llm be the same for each call in the loop? As far as I experimented, it is the same but could not figure out if it's coincidence or by implementation side. If generated texts might be different, comparing methods won't make sense in this setting.

from lm_polygraph.utils.manager import estimate_uncertainty

def estimate_uncertainties(model, ue_methods, input_text):
    scores = []
    for method_name in ue_methods.keys():
        ue = estimate_uncertainty(model, ue_methods[method_name], input_text=prompt)
        scores.append( 
            {
            'method': method_name,
            'uncertainty': ue.uncertainty
            }
        )
        torch.cuda.empty_cache()
    return scores, ue.generation_text

Thank you so much for your time!

[IINemo]

Hi, sorry for the long reply, I think it is controlled by the temperature parameter in generation_config. By default we use greedy decoding, which means that it will be always the same.

[nehiridil]

Hi! Thank you so much for reply.

So when I want to get scores for different UQ methods for one prompt, it generates the response multiple times which is unnecessary.
And when I switch the do_sample to True, and try different temperatures, llm will create different outputs. In this setting I cannot compare the UQ methods, since each of them will use different llm output.

Is there a way to generate the llm output once, and use that output across different UQ methods? Since I am using the estimate_uncertainty function, it's not possible to give only the output and evaluate the uncertainty.

Thank you so much for your time,

Warm regards,

[IINemo]

Hi, I see, I think it is possible, but you need to use some low-level code instead of estimate_uncertainty. There are some examples here:
https://github.com/IINemo/lm-polygraph/blob/main/examples/low_level_example.ipynb
and also here:
https://github.com/IINemo/lm-polygraph/blob/main/examples/other/qa_example.ipynb

However, I think I can make a better example later today.

[IINemo]

Hi, please have a look at this example (in Google Colab):
https://drive.google.com/file/d/1_TkY3Ak7gLNmivMuh3QW7BdZgOB-k2HG/view?usp=sharing

It shows how you can launch multiple estimators. Note that plygraph_eval - a benchmarking script, already does the same thing for benchmarking multiple uncertainty scores.

[IINemo]

@nehiridil let us know if we could help somehow further

[nehiridil]

Thanks a lot for taking time and sharing the code! Unfortunately, I cannot look at it yet because of my other project which its deadline is approaching. I'll contact you once I've adapt it to my case in 3 days.

[nehiridil]

Hi!
I thank you once again for sharing the code. As far as I understand, I need to compute some parts before running estimators. For example I also added these lines to get results for MeanPointwiseMutualInformation.

calc_greedylm = GreedyLMProbsCalculator()
deps.update(calc_greedylm(deps, texts=batch, model=model_adapter))

I just could not figure out how to get results for PTrue(). I believe I need to use one of the PromptCalculators but not sure which ones are correct. I tried PromptCalculator() but 'input_texts' field is still missing for me.

Thank you so much for your help once again!

[IINemo]

Hi, sorry, we are at ACL-2025 right now to hold a tutorial and other things. We will be back in a week.