Batch job submission failed: Node count specification invalid

[KB1RD]

Using the following Snakefile:

rule ALL:
    resources:
        nodes=1
    run:
        shell('echo Hello World > test.txt')

And running: (where <act_name> is my account name)

snakemake -j 1 --executor slurm --default-resources slurm_account=<act_name>

I get:

Building DAG of jobs...
SLURM run ID: a624be26-2574-4b8b-9b59-516f07c62cb9
Using shell: /usr/bin/bash
Provided remote nodes: 1
Job stats:
job      count
-----  -------
ALL          1
total        1

Select jobs to execute...
Execute 1 jobs...

[Mon Jun  3 12:24:56 2024]
rule ALL:
    jobid: 0
    reason: Rules with neither input nor output files are always executed.
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=<TBD>, slurm_account=st-wangalex-1, nodes=1

No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
WorkflowError:
SLURM job submission failed. The error message was sbatch: error: ---------------------------------------------------------------------- 
sbatch: error:  No number of nodes specified.  
sbatch: error: Please add  #SBATCH --nodes=NN  where NN is number of nodes for the job, in your submission script
sbatch: error: or in your interactive submission add --nodes=NN to your command.    
sbatch: error:  ---------------------------------------------------------------------
sbatch: error: Batch job submission failed: Node count specification invalid

I also replaced sbatch in my local bash path and used it to echo out the arguments that are passed to sbatch:

--job-name c7f938fe-bbd4-4566-bc71-519a52b966f9 --output /path/to/workdir/.snakemake/slurm_logs/rule_ALL/%j.log --export=ALL --comment rule_ALL -A <RedactedAccountName> -p skylake --mem 1000 --ntasks=1 --cpus-per-task=1 -D /path/to/workdir --wrap=/home/<MyUser>/.conda/envs/<CondaEnv>/bin/python3.1 -m snakemake --snakefile /path/to/workdir/Snakefile --target-jobs 'ALL:' --allowed-rules 'ALL' --cores all --attempt 1 --force-use-threads  --resources 'mem_mb=1000' 'mem_mib=954' 'disk_mb=1000' 'disk_mib=954' 'nodes=1' --wait-for-files '/path/to/workdir/.snakemake/tmp.o0paq911' --force --target-files-omit-workdir-adjustment --keep-storage-local-copies --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --rerun-triggers mtime input software-env params code --conda-frontend mamba --shared-fs-usage storage-local-copies source-cache software-deployment input-output persistence sources --wrapper-prefix /~https://github.com/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --scheduler ilp --local-storage-prefix .snakemake/storage --scheduler-solver-path /home/<MyUser>/.conda/envs/<CondaEnv>/bin --default-resources base64//bWVtX21iPW1pbihtYXgoMippbnB1dC5zaXplX21iLCAxMDAwKSwgODAwMCk= base64//ZGlza19tYj1tYXgoMippbnB1dC5zaXplX21iLCAxMDAwKQ== base64//dG1wZGlyPXN5c3RlbV90bXBkaXI= base64//<RedactedAccountName> --executor slurm-jobstep --jobs 1 --mode remote

I don't see any nodes argument...

Versions:

$ conda --version
conda 4.9.2
$ snakemake --version
8.11.3
$ sbatch --version
slurm 23.02.7

Conda list:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
amply                     0.1.6              pyhd8ed1ab_0    conda-forge
annotated-types           0.6.0              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argparse-dataclass        2.0.0              pyhd8ed1ab_0    conda-forge
attmap                    0.13.2             pyhd8ed1ab_0    conda-forge
attrs                     23.2.0             pyh71513ae_0    conda-forge
brotli                    1.1.0                    pypi_0    pypi
brotli-python             1.1.0           py312h30efb56_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
coin-or-cbc               2.10.10              h9002f0b_0    conda-forge
coin-or-cgl               0.60.7               h516709c_0    conda-forge
coin-or-clp               1.17.8               h1ee7a9c_0    conda-forge
coin-or-osi               0.108.10             haf5fa05_0    conda-forge
coin-or-utils             2.11.11              hee58242_0    conda-forge
coincbc                   2.10.10           0_metapackage    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda-inject              1.3.1              pyhd8ed1ab_0    conda-forge
configargparse            1.7                pyhd8ed1ab_0    conda-forge
connection_pool           0.0.3              pyhd3deb0d_0    conda-forge
datrie                    0.8.2                    pypi_0    pypi
dkh                       1.2                  hd59d2e7_0    conda-forge
docutils                  0.21.2             pyhd8ed1ab_0    conda-forge
dpath                     2.1.6              pyha770c72_0    conda-forge
eido                      0.2.2              pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
gau2grid                  2.0.7                h0b41bf4_2    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.43             pyhd8ed1ab_0    conda-forge
gtest                     1.14.0               h00ab1b0_1    conda-forge
humanfriendly             10.0               pyhd8ed1ab_6    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.7                pyhd8ed1ab_0    conda-forge
immutables                0.20                     pypi_0    pypi
importlib_resources       6.4.0              pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jsonschema                4.22.0             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
jupyter-core              5.7.2                    pypi_0    pypi
jupyter_core              5.7.2           py312h7900ff3_0    conda-forge
ld_impl_linux-64          2.40                 h55db66e_0    conda-forge
libblas                   3.9.0            20_linux64_mkl    conda-forge
libcblas                  3.9.0            20_linux64_mkl    conda-forge
libecpint                 1.0.7               h3ecfda7_10    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h77fa898_7    conda-forge
libgfortran-ng            13.2.0               h69a702a_7    conda-forge
libgfortran5              13.2.0               hca663fb_7    conda-forge
libhwloc                  2.10.0          default_h2fb2949_1000    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libint                    2.9.0                h9bbc0ff_0    conda-forge
liblapack                 3.9.0            20_linux64_mkl    conda-forge
liblapacke                3.9.0            20_linux64_mkl    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libpcm                    1.2.3                h4175798_8    conda-forge
libsqlite                 3.45.3               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               hc0a3c3a_7    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxc-c                   6.2.2            cpu_h1b64f48_4    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxml2                   2.12.7               hc051c1a_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
llvm-openmp               18.1.5               ha31de31_0    conda-forge
logmuse                   0.2.6              pyh8c360ce_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.5                    pypi_0    pypi
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
mkl                       2023.2.0         h84fe81f_50496    conda-forge
msgpack                   1.0.8                    pypi_0    pypi
msgpack-python            1.0.8           py312h2492b07_0    conda-forge
nbformat                  5.10.4             pyhd8ed1ab_0    conda-forge
ncurses                   6.5                  h59595ed_0    conda-forge
networkx                  3.3                pyhd8ed1ab_1    conda-forge
numpy                     1.26.4                   pypi_0    pypi
openssl                   3.3.0                hd590300_0    conda-forge
optking                   0.2.1              pyhd8ed1ab_0    conda-forge
packaging                 24.0               pyhd8ed1ab_0    conda-forge
pandas                    2.2.2                    pypi_0    pypi
pcmsolver                 1.2.3                      py_9    conda-forge
peppy                     0.40.1             pyhd8ed1ab_0    conda-forge
pint                      0.23               pyhd8ed1ab_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
plac                      1.4.3              pyhd8ed1ab_0    conda-forge
platformdirs              4.2.1              pyhd8ed1ab_0    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_0    conda-forge
psi4                      1.9.1           py312hef1bf4d_1    conda-forge
psutil                    5.9.8                    pypi_0    pypi
pugixml                   1.14                 h59595ed_0    conda-forge
pulp                      2.8.0                    pypi_0    pypi
py-cpuinfo                9.0.0              pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pydantic                  2.7.1              pyhd8ed1ab_0    conda-forge
pydantic-core             2.18.2                   pypi_0    pypi
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.4.4              pyhd8ed1ab_0    conda-forge
python                    3.12.3          hab00c5b_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.19.1             pyhd8ed1ab_0    conda-forge
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python_abi                3.12                    4_cp312    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1                    pypi_0    pypi
qcelemental               0.27.1             pyhd8ed1ab_0    conda-forge
qcengine                  0.29.0             pyhd8ed1ab_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.35.1             pyhd8ed1ab_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
reretry                   0.11.8             pyhd8ed1ab_0    conda-forge
rich                      13.7.1             pyhd8ed1ab_0    conda-forge
rpds-py                   0.18.1                   pypi_0    pypi
scipy                     1.13.0                   pypi_0    pypi
setuptools                69.5.1             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
slack-sdk                 3.27.1             pyha770c72_0    conda-forge
slack_sdk                 3.27.1               hd8ed1ab_0    conda-forge
smart_open                7.0.4              pyhd8ed1ab_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
snakemake                 8.11.3               hdfd78af_0    bioconda
snakemake-executor-plugin-slurm 0.5.0              pyhdfd78af_0    bioconda
snakemake-executor-plugin-slurm-jobstep 0.2.1              pyhdfd78af_0    bioconda
snakemake-interface-common 1.17.2             pyhdfd78af_0    bioconda
snakemake-interface-executor-plugins 9.1.1              pyhdfd78af_0    bioconda
snakemake-interface-report-plugins 1.0.0              pyhdfd78af_0    bioconda
snakemake-interface-storage-plugins 3.2.2              pyhdfd78af_0    bioconda
snakemake-minimal         8.11.3             pyhdfd78af_0    bioconda
stopit                    1.1.2                      py_0    conda-forge
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
tbb                       2021.12.0            h00ab1b0_0    conda-forge
throttler                 1.2.2              pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toposort                  1.10               pyhd8ed1ab_0    conda-forge
traitlets                 5.14.3             pyhd8ed1ab_0    conda-forge
typing-extensions         4.11.0               hd8ed1ab_0    conda-forge
typing_extensions         4.11.0             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
ubiquerg                  0.7.0              pyhd8ed1ab_0    conda-forge
urllib3                   2.2.1              pyhd8ed1ab_0    conda-forge
veracitools               0.1.3                      py_0    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
wrapt                     1.16.0                   pypi_0    pypi
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yte                       1.5.4              pyha770c72_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

[KB1RD]

Removing this outer if statement fixes the issue right up.

At least on my university's slurm setup, slurm requires the --nodes option regardless of whether or not you're using MPI (and I'm not sure why these would be related?)

There is no other mention of the nodes option anywhere in the codebase, so there's nowhere else it could possibly be assigned. I'm not sure why other people are not encountering this issue.

[cmeesters]

Ah, yes - the code there does not make sense! The intent got lost! Thank you for spotting this, I will code a fix, tomorrow.

[KB1RD]

Thanks, I was worried something was weird on our cluster!

[cmeesters]

Na, things happen ...

Thanks for your excellent report. A bug fix release is out. Please install and test, one the bioconda release is ready.

[KB1RD]

The issue is fixed at least in version 0.6.0 on Bioconda. Sorry it took so long for me to test, and thanks for fixing!