Skip to content
Type to start searching

    Table of Contents

    Tools for high-throughput computing

    High-throughput computing (HTC) means running lots of jobs automatically using scripts or workflows. Workflow automation saves human time and reduces manual errors. Workflows can be very specific and if you can't find one that suits your needs out-of-the-box, they may need a lot of work before they bring benefit. However, the simplest ones are easy to use. This section introduces some tools available at CSC, but you can of course develop or install your own.

    General tools that run multiple jobs with one script

    • Array jobs are a native Slurm tool to submit several independent jobs with one command
    • GREASY runs multiple job steps in a larger allocation, and allows simple dependencies
    • DIY gnuparallel tutorial shows how to efficiently run a very large number of serial jobs without bloating the Slurm log
    • FireWorks is a flexible tool for defining, managing and executing workflows with multiple steps and complex dependencies

    Science specific workflow tools

    • Nextflow singularity container based bioinformatics pipelines on Puhti

    Workflow tools integrated into common simulation software

    The following built-in tools allow running multiple simulations in parallel within a single Slurm job step. If you're using any of the applications below, please consider these as the first option for implementing your high-throughput workflows.