Pipeline is ignoring some memory constraints in custom config -c

[ryandkuster]

Description of the bug

Thank you so much for developing pairgenomealign, I'm excited to use this pipeline!

During a test run, I noticed that some defined memory constraints from a custom configuration file (-c) are not being utilized. For example, a custom configuration (below) is respecting the queueSize of 2 but the maximum memory limit of 50GB is being far surpassed with >300GB in practice. The input fasta files are rather modest in number and size, so the memory use is a great concern.

executor {
    queueSize = 2
}
process {
  resourceLimits = [
    cpus: 12,
    memory: 50.GB,
  ]
} 

I've had similar results when supplying a more explicit attempt at process management for process_high:

...
withLabel:process_high {
  cpus   = 12
  memory = 50.GB
}
...

In particular this is stemming from the ALIGNMENT_LASTAL_M2O process.

Thanks again!

Command used and terminal output

Relevant files

No response

System information

Nextflow version 24.10.4
Apptainer version 1.2.5-1.el7
nf-core/pairgenomealign 2.0.0

[charles-plessy]

Hi Ryan, using a config file like the one you pasted seems to work in my hands... Can you reproduce the issue with -profile <your_institution>,test -c the_local_config_file_above.nf, let me know which institutional profile you use (or if you do not use one), and post a copy of your ~/.nextflow/config file?

[ryandkuster]

Thanks @charles-plessy. Using the profile apptainer,test runs without error using the custom config settings, and the execution report reflects that the cpus and memory were adjusted to the custom configuration. I'm not getting a ~/.nextflow/config file produced this way.

When I re-run my actual samples (not the test profile) using the identical resources.config file (below), I ultimately have to abort the run due to concerning memory use, but the execution report also suggests that my resource limitations are hypothetically being considered, even though monitoring memory suggests otherwise. I'm not sure if the pipeline is ignoring the limits when there are multiple task.attempts? I do wonder if testing a larger dataset (like a dozen plant genomes) with extremely limited memory would reproduce the issue on your end.

executor {
    queueSize = 2
}
process {
  resourceLimits = [
    cpus: 12,
    memory: 50.GB,
  ]
}

Let me know if I can try anything else or investigate any outputs to troubleshoot.

[charles-plessy]

I appears that Nextflow does not enforce resource limits when running locally. https://www.nextflow.io/docs/latest/executor.html#local

But you should still be able to restrict usage with an appropriate mix of process resource settings like this:

withLabel:process_high {
  cpus   = 12
  memory = 50.GB
}

Does that work for you ?