Error with Missing Values in `normed_condition_df`

[odespard]

Describe the bug
Dear Constantin,

I hope you are doing well. I am analysing some proteomics data from a DIA-NN .parquet file using the median condition analysis option in AlphaQuant. I am really enjoying using the software and the code is a pleasure to read. However, it is throwing an Exception on one of the files.

On inspection of the code, it appears the problem is in the calculation of the total_available_comparisons variable in ConditionBackgrounds.select_intensity_ranges(). There are several rows in self.normed_condition_df which have a count of 0. As I understand it, the code assumes that there are completely missing values, and hence subtracts 1 from the total count to get the number of comparisons to the anchor fold change. However, this leads to subtracting 1 from the total_available_comparisons whenever all values are missing. Hence, my final value for total_available_comparisons is negative (around -25000), and this leads to strange incorrect behaviour and eventually the error.

I think something like the following should work to avoid the error:

class ConditionBackgrounds():
...
    def select_intensity_ranges(self, p2z):
    ...
      for idx ,count in enumerate(self.normed_condition_df.count(axis=1)):
          if count <= 1:
              comparisons_to_add = 0
          else:
              comparisons_to_add = count - 1
      
          total_available_comparisons+=comparisons_to_add
          cumulative_counts[idx] = int(total_available_comparisons/2)
    ...

If you think this is sufficient to solve the problem, I am happy to submit a pull request. However, since, as I understand it, the code is written on the assumption that there are no missing values at this step, I am slightly concerned that there might still be unwanted behaviour later on.

It also strikes me as possible that this might be an issue even on data with fewer missing values – it may not cause an error if the total_available_comparisons stays positive, but it will increase the number of ions put into the final bin. I don't think this would be particularly bad for quantification, but it would probably not be ideal. Indeed, I have noticed on other files (which have run without errors) that the final bin tends to include a very large number of ions relative to the previous bin (35000 instead of around 1300); then again, the early bins also contain many more ions, so I don't know if this is intentional (and I would be grateful if you could help me understand!)

Thank you for your attention.

Yours sincerely,
Oscar

To Reproduce
Steps to reproduce the behavior:
Use a file with numerous missing values.

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
The software should be able to cope even if there are lots of missing values.

Logs

The relevant portion of the log is below:

2025-04-19 17:03:07,959 - alphaquant.norm.normalization - INFO - normalized within conditions
2025-04-19 17:03:07,960 - alphaquant.plotting.pairwise - INFO - without missingvals (if applicable)
2025-04-19 17:03:08,020 - alphaquant.plotting.pairwise - INFO - complete dataset
2025-04-19 17:03:08,197 - alphaquant.norm.normalization - INFO - median -0.06401884536364122, mode -0.21795154703963382
2025-04-19 17:03:08,198 - alphaquant.norm.normalization - INFO - using mode for shift
2025-04-19 17:03:08,199 - alphaquant.norm.normalization - INFO - shift comparison by 0.21795154703963382
2025-04-19 17:03:08,456 - alphaquant.norm.normalization - INFO - normalized between conditions
2025-04-19 17:04:46,601 - alphaquant.diffquant.background_distributions - INFO - Created Background Distribution for 37280 ions. SD: 0.962321070842282, fraction of missing values: 0.19
2025-04-19 17:04:49,237 - alphaquant.diffquant.background_distributions - INFO - Created Background Distribution for 34231 ions. SD: 0.8851078871951418, fraction of missing values: 0.15
2025-04-19 17:04:50,815 - alphaquant.diffquant.background_distributions - INFO - Created Background Distribution for 32049 ions. SD: 0.884025335782286, fraction of missing values: 0.11
2025-04-19 17:04:52,358 - alphaquant.diffquant.background_distributions - INFO - Created Background Distribution for 30109 ions. SD: 0.8727434096906822, fraction of missing values: 0.08
2025-04-19 17:04:53,925 - alphaquant.diffquant.background_distributions - INFO - Created Background Distribution for 28465 ions. SD: 0.8648630024679393, fraction of missing values: 0.04

ValueError Traceback (most recent call last)
Cell In[4], line 1
----> 1 run_pipeline(
2 input_file=INPUT_FILE,
3 samplemap_file=SAMPLEMAP_FILE,
4 results_dir=RESULTS_DIR,
5 multicond_median_analysis=True,
6 )

File ~/software/source/alphaquant/alphaquant/run_pipeline.py:215, in run_pipeline(input_file, samplemap_file, samplemap_df, modification_type, input_type_to_use, results_dir, multicond_median_analysis, condpairs_list, file_has_alphaquant_format, min_valid_values, valid_values_filter_mode, min_valid_values_c1, min_valid_values_c2, min_num_ions, minpep, organism, cluster_threshold_pval, cluster_threshold_fcfc, fcdiff_cutoff_clustermerge, use_ml, take_median_ion, perform_ptm_mapping, perform_phospho_inference, enable_experimental_ptm_counting_statistics, outlier_correction, normalize, use_iontree_if_possible, write_out_results_tree, use_multiprocessing, runtime_plots, volcano_fdr, volcano_fcthresh, annotation_columns, protein_subset_for_normalization_file, protnorm_peptides, peptides_to_exclude_file, reset_progress_folder, minrep_both, minrep_either, minrep_c1, minrep_c2)
212 num_cores = get_num_cores_to_use(use_multiprocessing)
214 if num_cores == 1:
--> 215 run_analysis_singleprocess(condpair_combinations=condpairs_list, runconfig=runconfig)
217 else:
218 run_analysis_multiprocess(condpair_combinations=condpairs_list, runconfig=runconfig, num_cores=num_cores)

File ~/software/source/alphaquant/alphaquant/run_pipeline.py:318, in run_analysis_singleprocess(condpair_combinations, runconfig)
315 def run_analysis_singleprocess(condpair_combinations, runconfig):
317 for condpair in condpair_combinations:
--> 318 aqcondpair.analyze_condpair(runconfig=runconfig, condpair=condpair)

File ~/software/source/alphaquant/alphaquant/diffquant/condpair_analysis.py:51, in analyze_condpair(runconfig, condpair)
49 write_out_normed_df(df_c1_normed, df_c2_normed, pep2prot, runconfig.results_dir, condpair)
50 normed_c1 = aqbg.ConditionBackgrounds(df_c1_normed, p2z)
---> 51 normed_c2 = aqbg.ConditionBackgrounds(df_c2_normed, p2z)
53 ions_to_check = normed_c1.ion2nonNanvals.keys() & normed_c2.ion2nonNanvals.keys()
54 ions_to_check = sorted(ions_to_check)

File ~/software/source/alphaquant/alphaquant/diffquant/background_distributions.py:22, in ConditionBackgrounds.init(self, normed_condition_df, p2z)
20 self.init_ion2nonNanvals(normed_condition_df)
21 self.context_ranges = []
---> 22 self.select_intensity_ranges(p2z)
24 self.all_intensities = np.concatenate(list(self.ion2nonNanvals.values()))
25 self.num_replicates = len(next(iter(self.ion2allvals.values())))

File ~/software/source/alphaquant/alphaquant/diffquant/background_distributions.py:67, in ConditionBackgrounds.select_intensity_ranges(self, p2z)
0 <Error retrieving source code with stack_data see ipython/ipython#13598>

File ~/software/source/alphaquant/alphaquant/diffquant/background_distributions.py:124, in init(self, start_idx, end_idx, ion2noNanvals, idx2ion, p2z)
122 self.min_z=0
123 self.max_z=0
--> 124 self.start_idx = int(start_idx)
125 self.end_idx = int(end_idx)
126 self.var = None

File ~/software/source/alphaquant/alphaquant/diffquant/background_distributions.py:147, in calc_missingval_fraction(self, ion2nonNanvals, idx2ion)
141 def calc_missingval_fraction(self, ion2nonNanvals: dict, idx2ion: dict) -> float:
142 """Calculates the fraction of missing values in the background distribution
143
144 Args:
145 ion2nonNanvals (dict): maps the ion to all measured intensities of this ion (no NAs/zero measurements)
146 idx2ion (dict): distinct mapping of the index to the ion name
--> 147
148 Returns:
149 float: fraction of missing values
150 """
151 value_nums = [len(ion2nonNanvals.get(idx2ion.get(idx))) for idx in range(self.start_idx, self.end_idx)]
152 num_replicates = max(value_nums)

ValueError: max() arg is an empty sequence

Version:

• Installation Type: Developer
• If no log is available, provide the following:
  • Platform information
    • system AlmaLinux 9
    • version 9.5
    • machine x86_64
    • processor Intel(R) Xeon(R) Platinum 8380H CPU @ 2.90GHz
    • cpu count 224
  • Python information:
    • alphaquant version 0.1.9.dev0

[mschwoer]

Hey Oscar,
thank you for reporting this. We have finally started looking into it, will come back with a solution next week or so.
best,
Magnus

[mschwoer]

Hey @odespard, would it be possible to provide the data (and parameters) to reproduce?

[odespard]

I can't share the data, unfortunately. Are you having trouble reproducing it?

[mschwoer]

In fact I do :-)
I guess you can't share the data for legal reasons, could you maybe strip it to a mininal example, or rescale it or smth?

[odespard]

Yes, I'll make something synthetic that will reproduce the issue; unfortunately it might take a while as I have some other things to do at the moment, but I'll let you know when I get the chance.