result.pep.xml could not be found

[EmaEtch]

Description of the bug

Dear developers,

I am trying to use mhcquant but I am facing some problems running the initial tests.
I am running the following command in a conda environment where nextflow and singularity have been installed following your guidelines:
nextflow run nf-core/mhcquant -r 3.0.0 -profile test,singularity --outdir /mnt/users/xxx/output_dir -work-dir /mnt/users/xxx/nxf_works/work

But I am getting the following error:

ERROR ~ Error executing process > 'NFCORE_MHCQUANT:MHCQUANT:OPENMSTHIRDPARTY_COMETADAPTER (2)'

Caused by:
  Process `NFCORE_MHCQUANT:MHCQUANT:OPENMSTHIRDPARTY_COMETADAPTER (2)` terminated with an error exit status (1)


Command executed:

  CometAdapter \
      -in HepG2_rep2_small.mzML \
      -database UP000005640_9606_decoy.fasta \
      -out HepG2_rep2_small.idXML \
      -threads 2 \
      -pin_out HepG2_rep2_small_pin.tsv -instrument high_res -spectrum_batch_size 0 -activation_method ALL -precursor_mass_tolerance 5 -precursor_error_units ppm -fragment_mass_tolerance 0.01 -fragment_bin_offset 0.0 -num_hits 1 -digest_mass_range 800:2500 -max_variable_mods_in_peptide 3 -missed_cleavages 0 -precursor_charge 2:3 -fixed_modifications  -variable_modifications 'Oxidation (M)' -enzyme 'unspecific cleavage' -use_X_ions false -use_Z_ions false -use_A_ions false -use_C_ions false -use_NL_ions false
  
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_MHCQUANT:MHCQUANT:OPENMSTHIRDPARTY_COMETADAPTER":
      CometAdapter: $(CometAdapter 2>&1 | grep -E '^Version(.*)' | sed 's/Version: //g' | cut -d ' ' -f 1 | cut -d '-' -f 1)
      Comet: $(comet 2>&1 | grep -E "Comet version.*" | sed 's/Comet version //g' | sed 's/"//g')
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  Could not remove directory 20251211_142039_net69-blade01-slot01_40_1!
  Error: File not found (the file '/tmp/20251211_142039_net69-blade01-slot01_40_1/result.pep.xml' could not be found)
  INFO:    Cleaning up image...

I have tried multiple configurations defining TMPDIR, NXF_TMPDIR, SINGULARITY_TMPDIR, SINGULARITY_HOME, NXF_HOME, but nothing is working correctly.
Do you have some clues to help me,
Thank you in advance,

Command used and terminal output

Relevant files

No response

System information

No response

[jonasscheid]

Could you try running -r dev?

@steffenlem Have you seen this error before?

[EmaEtch]

Hi,
Thanks for your reply.

I have tested both -r 3.0.0 and -r dev.
In both cases, the behavior is inconsistent: sometimes the workflow runs successfully, and sometimes the exact same command fails, even when nothing is changed.

I also observed that if I submit two identical jobs in parallel on different Slurm nodes, one may complete successfully while the other fails with the same result.pep.xml not found error.

This makes me think the issue is related to an unstable or environment-dependent configuration on my side, but I cannot figure out what is causing it.

I have observed a consistent pattern regarding the temporary folders created during the CometAdapter execution.

When the job succeeds, three temporary directories are created in /tmp
(e.g. 20251212_152702_net69-blade01-slot03_37_1, 20251212_152702_net69-blade01-slot03_40_1, 20251212_152703_net69-blade01-slot03_40_1),
and each of them contains a result.pep.xml file.

When the job fails, only two such directories are created
(e.g. 20251212_152704_net69-blade01-slot04_40_1, 20251212_152705_net69-blade01-slot04_40_1),
and each also contains a result.pep.xml.

However, the error then reports that a specific result.pep.xml file cannot be found (for example:
/tmp/20251212_152704_net69-blade01-slot04_40_1/result.pep.xml).
So the directory and the file do exist at some point, but CometAdapter still fails to access it — or perhaps the file is being deleted too early.

If you have any hints about what could trigger such intermittent behavior, I would really appreciate your help.

Thanks again!

[jonasscheid]

This makes me think the issue is related to an unstable or environment-dependent configuration on my side, but I cannot figure out what is causing it.

I would also suggest that

My experience regarding slurm and HPC usage is mostly restricted to this config https://nf-co.re/configs/cfc/
Maybe

scratch = 'true'

helps

[jonasscheid]

I observe this artifact error sometimes with github CI tests, where singularity tests throw this error. I suspect the container download from galaxy had some issues.. Do you still have this issue? @EmaEtch

[EmaEtch]

Hi,

Thank you for your message.

Yes, indeed I still have the issue.

I have tried several approaches to rule out a potential Nextflow or Singularity configuration problem on my side, but I have not been able to resolve the problem.

I also tested another nf-core pipeline (nf-core/rnaseq) using the same Nextflow and Singularity setup, and it runs without any issue. This suggests that my Nextflow–Singularity configuration is generally functional.

I also completely removed and re-downloaded the Singularity cache this morning, but the error persists.

Given this, indeed the issue may be related to the Singularity images used by mhcquant as you mentioned.
Please let me know if you have any recommendation.

[jonasscheid]

Could you try the nf-core download command or pull the singularity images from galaxy manually (at least the openmsthirdparty container)?

[EmaEtch]

I have tried nf-core download command for all the singularity cache or doawnloading openms-thirdparty_3.4.0--h9ee0642_0 from galaxy manually but the error persists

[jonasscheid]

Ok most likely a setup problem. Maybe not enough space for singularity images?

Maybe try

mkdir -p ./singularity_tmp
export SINGULARITY_TMPDIR=$PWD/singularity_tmp

Or add scratch=False via a custom config similar to the cfc config on nf-core/configs repo

[EmaEtch]

Hi,

Thanks for your reply and the suggestions.

I already tested using a dedicated SINGULARITY_TMPDIR on a filesystem with sufficient space, as well as disabling scratch via a custom Nextflow configuration, but unfortunately the issue still occurs.

To rule out a general Nextflow/Singularity setup problem on my side, I also tested another nf-core pipeline (nf-core/rnaseq) using the same Nextflow and Singularity versions and configuration, and it runs without any issues.

I have also cleared and re-downloaded the Singularity cache, but the problem persists.

Given this, it seems unlikely that the issue is related to disk space or a general Singularity setup, and it may be specific to the OpenMS/Comet execution within the mhcquant Singularity images in my environment.

Please let me know if you would recommend any additional tests or if there is a known workaround for this issue.