TEI-REX Workflows - Guide for setting up AWS Batch
AWS Batch is a service provided by Amazon’s cloud computing service called Amazon Web Services (AWS). It is effectively an extremely large computer cluster that will bring up computing resources as you need them, instead of having the computers sitting idle when not in use. These set of documents will serve as a guide for setting up AWS Batch as an executor for the Nextflow workflows developed for the IARPA TEI-REX project.
Important
You are not required to set up AWS Batch to use the TEI-REX workflows. By default they will work
run tasks on the same computer on which you are executing the nextflow
command. This guide
describes how to set up AWS Batch for remote cloud-based execution of the steps instead.
What are executors?
Executors are a concept in Nextflow workflows that represent the actual computer resources that will be running the steps of the workflow. Example include your local computer, a traditional computer cluster (e.g., Condor or Slurm), AWS Batch, and many others. The TEI-REX Nextflow workflows have been designed such that different executors can be used interchangeably. After following this guide, users will be able to specify awsbatch as their executor, and the steps of the workflow will run remotely on AWS Batch.
Technical expertise required
The intention of this guide is to provide the necessary information to users with existing expertise with setting up AWS services with the information needed to properly set up and configure AWS Batch for use with the TEI-REX Nextflow workflows. As such, some technical expertise, particularly with AWS is assumed.
Getting started
To get started, follow the links below (or in the left-hand navigation panel) for information on setting up and configuring AWS Batch and where to find the parameters required by the workflow for running on AWS Batch.
Getting Help, Providing Feedback, or Reporting Problems
If you need help, have ideas for new features, encounter a problem, or have any questions or comments, please contact Michael Riffle at mriffle@uw.edu.
Documentation Sections
- TEI-REX Workflows - Guide for setting up AWS Batch
- Setup and Configure AWS Batch
- Before you begin
- Setup the Required AWS Services and resources
- Create an Custom AMI
- Create IAM role for compute instances
- Create IAM role for Spot instances
- Create the Compute Environment and the Job Queue to run Nextflow workflows
- Enable CloudWatch Logs with AWS Batch
- Make S3 bucket
- Create IAM Users for users submitting Nextflow workflows
- Next steps
- Configure and Run Workflow for AWS Batch