Skip to content

What is Nextflow?

Introduction to Nextflow

Nextflow is a domain-specific language (DSL) for enabling scalable, portable, and reproducible workflows.

Nextflow is both a workflow language and an execution runtime that supports a wide range of execution platforms, including popular traditional grid scheduling systems such as Slurm and IBM LSF, and cloud services such as AWS Batch, Google Cloud Batch, Azure Batch and Kubernetes.

While Nextflow solves many of the technical challenges associated with building and executing data pipelines; bioinformaticians, data scientists, and clinicians still face challenges:

  • Users shouldn't need extensive technical knowledge of the command line or have to understand how to configure computing environments to monitor pipeline execution.
  • Researchers face difficulties in ensuring reproducibility, tracking data provenance, and sharing comprehensive reports and interactive tools for data analysis.

Limitations

Monitoring and launching workflows via the Nextflow CLI, though direct, poses challenges, especially with complex or large-scale pipelines that are not as simple as running a Hello World pipeline:

  • Scalability: As the number of tasks increases, manually checking individual log files becomes impractical.
  • Real-time tracking: The CLI does not offer an easy way to visualize real-time progress across multiple parallel tasks.
  • Aggregation: Collecting and interpreting logs from various processes requires additional tools or scripts, complicating the workflow management.
  • Flexibility: Switching between environments (i.e., your local computer to HPC or cloud) requires the setup of access in the form of account keys and credentials to the environment on your CLI, followed by using the appropriate Nextflow configuration settings.