Skip to content

F2 · Synthetic scenario pipelines on both engines

Done 2026-10-26 · commit b2687d8 · 11 / 11 scenarios on Nextflow 26.10.3, 25.10.9 and Snakemake 9.27.1 (one scenario repeated once after a known engine fault)

What changed

  • tests/pipelines/synthetic/ holds two twin pipelines, one per engine:

    • nextflow/main.nf and snakemake/workflow/Snakefile;
    • the same three stages (qc, then a gate, align, then a gate, quant);
    • deterministic task bodies (bin/synth.py);
    • injected failures: exit, OOM, OOM once, timeout, flaky.

  • scenarios.yaml lists 11 PR scenarios and 1 nightly stress scenario. run_scenarios.py checks:

    • the run's exit code;
    • outputs that must or must not exist;
    • per-process task counts, with their exit codes and attempts;
    • cached tasks after a resume.

  • Known engine faults. These are listed with their log signature. A matching failure is repeated once and reported as PASS*.

  • Specs changed:

Why

Spectio's gates rely on engine mechanisms: a generated Nextflow config with -resume, and Snakemake --until. Those had to be proven on both engines before any code depends on them. See checking scenarios.

Prompt: Thomas's request for real Snakemake and Nextflow tests. Decision: decision log, 2026-10-26, real pipelines.

Evidence

$ python3 run_scenarios.py --engine nextflow --nextflow nextflow-25.10.9 --workdir /tmp/sc/nf25
PASS  nextflow  happy                             15.9s
PASS  nextflow  gate_qc_then_resume               24.8s
…
PASS  nextflow  timeout_fatal                     35.9s
PASS  nextflow  dynamic_fanout                    17.0s
PASS  nextflow  fanout_medium                     45.9s
$ python3 run_scenarios.py --engine snakemake --workdir /tmp/sc/smk     # full suite, with the known-fault rule
PASS  snakemake happy                              4.1s
…
PASS* snakemake oom_once_retry                     8.2s    [repeated after known engine fault: lost-retry-at-checkpoint-update]
…
PASS  snakemake fanout_medium                     12.8s

The full Snakemake run made before the known-fault rule existed failed on oom_once_retry (10 / 11). Full tables and the findings: spike note.

Figures and media

The task counts are measured (Nextflow 26.10.3, local executor). The cluster, runner, share link and phone show the planned flow; they are not built yet. Click the figure for the full size.

Stop at the QC gate, drop a sample, resume Stop at the QC gate, drop a sample, resume

Findings and open questions

  • Snakemake 9.27.1 drops a retried job when a checkpoint updates the DAG at the same moment ("BUG: Out of jobs ready to be started"). This happened in 4 of the 18 executions of oom_once_retry in this session (two full suites, loops of 8 and 6, and 2 repeats, both of which passed). A plain rerun completes.
  • Nextflow 25.10.9 reports no exit code for a local time-limit kill. 26.10.3 reports 143.
  • Not covered here: the SLURM (sind) and Kubernetes (kind) legs need cgroup v2 Docker. They are steps T3 and T4.