Introduction

Many organizations struggle to connect their lab to their computational environment to run their analysis. While simple at first, as the lab grows, assay diversity increases, and what was one lab becomes many. Multiple previous organizations we have been at have struggled with this, with solutions like building large storage clusters, synchronizing actions across environments, building file watchers, or just manually having someone look at the new runs' directory. None of these solutions work as complexity, scale, and interconnected systems grow. In addition, your bioinformatics teams are often asked to hit faster turnaround times with more data to process, while also building observability and modularity into their processes. Traditional designs often use tools like file watchers and pipeline scripts that are difficult to monitor, debug, or capable of reusing individual components across different workflows or datasets. They often also waste resources by leaving a variety of compute resources running just waiting for work.

In this blog post and reference architecture, we present an approach to structuring end to end genomics pipelines and the upstream systems to move data to solve this challenge of having a pipeline from the lab to your analysis as reusable state machines, using AWS Step Functions and other serverless AWS services to deliver scalable, extendable, transparent, and maintainable genomics data processing systems.

By abstracting each phase of the pipeline into a decoupled component with a consistent input/output structure, we achieve structural consistency and enable reusability across workflows. Long running jobs are managed using a polling pattern, eliminating the need for callback endpoints and reducing complexity across systems. This design supports scalability by allowing multiple workflows to run in parallel, extensibility through simple state machine modification, and observability via native AWS monitoring and logging tools.

The secondary analysis phase is orchestrated using Nextflow, enabling the execution of any bioinformatics pipeline, from nf-core standards to fully custom analyses, while maintaining portability, reproducibility, and containerized execution.

Whether you're handling whole-genome, RNA-seq, or custom experimental data, this architecture provides a flexible and production ready blueprint for teams aiming to modernize their genomics infrastructure while minimizing costs and operational overhead.

In the next sections we will outline our E2E solution.

Target Audience

This post is intended for bioinformatics engineers, cloud architects, and technical decision makers at genomics focused organizations who are looking to modernize and scale their data processing infrastructure. Specifically, it's geared toward: 

• Teams managing large volumes of sequencing data (WGS, RNA-seq, single-cell, etc.) that want to automate and streamline their pipelines
• Bioinformatics teams working with tools like Nextflow and exploring how to better integrate these tools into cloud native environments with sequencers and other lab instrumentation
• DevOps and cloud engineers supporting scientific research teams who need examples of reusable, scalable infrastructure patterns using AWS services such as AWS Step Functions, AWS Lambda, AWS DataSync, and AWS Batch
• Organizations considering a shift from event driven workflows to more orchestrated, observable, and modular pipeline architectures
• Organizations that want to move their computational infrastructure to the cloud

Whether you're running your workloads on-premises and considering a hybrid approach or you’ve already migrated to the cloud and want to optimize performance and maintainability, this post offers a clear blueprint for building reliable genomics workflows using the AWS infrastructure.

Challenge

Scientific and Operational Challenge

Inari is the SEEDesign™ company, using new breeding technology to design plants that grow smarter by unlocking the full potential of seeds for a more resilient and abundant future. At its core is the AI-enabled SEEDesign™ Platform, the seed industry’s most sophisticated performance accelerator integrating breakthroughs in genomics, machine learning and gene editing technology to bring unprecedented speed and precision to plant breeding.

The company has grown exponentially since its founding in 2016, expanding from a small team in stealth to an organization of 200+ employees operating across Cambridge, Massachusetts; West Lafayette, Indiana; and Ghent, Belgium. The R&D function has grown accordingly. What started as a single sequencing run once a week has expanded to many simultaneous runs across many lab instruments, with a variety of lab assays and analysis methods deployed depending on the scientific question being addressed. No longer the job of an individual, this work now demands a full-fledged enterprise operation – requiring faster data transfers, automated analysis and monitoring of a platform composed of a diverse ecosystem of tools.

Solution

We broke down our E2E genomics pipeline solution into two main sub-systems: the Data Uploader and the Data Pipeline. The first one oversees moving data from on-premises data centers onto the cloud at an hourly cadence using a combination of AWS DataSync tasks over an AWS Direct Connect link for speeding up the transfer processes as much as possible. The second sub-system is in charge of performing the different genomics related analyses needed to fulfil standard genomics pipelines functional requirements. To fulfil these requirements, we use a combination of AWS serverless components and NextFlow. Using serverless components allows us to minimize costs by paying only when the pipelines are active.

When designing our solution, we analyzed the difference between an event-based workflow and an orchestrated one. For us, key non-functional requirements were observability, maintainability, reusability and decoupling. After carefully analyzing these requirements, we opted for an orchestrated architecture, given that event based systems would present issues meeting some of these requirements due to the extensive use of triggers necessary to move data between stages. Also, an event based approach could present some issues regarding observability and maintainability if not addressed properly.

As part of our orchestrated based approach, we modeled our pipelines using AWS Step Functions as a collection of reusable state machines decoupled from one another. We also opted to use the polling design pattern as the main pattern for interacting with components and services. We consider the overhead from using this pattern to be negligible due to lack of a low latency requirement for moving data between stages. Also, in some cases the ability to use a callback endpoint was not available or would introduce unnecessary complexity when implementing them.

Benefits of this architecture:

• Decoupling of the genomics analyses computational engine from the architecture. If the secondary analysis engine supports an HTTP endpoint or similar, then it can be supported by our solution.
• Easy to understand, debug and monitor
• Highly reusable parts
• Highly scalable

In Figure 1, we see an example of an event driven genomics pipeline, which could be modeled using the choreography design pattern.

In Figure 2, we see an example of an orchestrated approach and modeled using the orchestration design pattern, supported by the use of AWS Step Functions.

Features

Serverless Components

We opted for a serverless approach for most of the components in the Data Pipeline, mostly due to its cost effectiveness but also given that the frequency of new sequencing data normally being produced on-premises was irregular.

Going with a fully serverless approach allowed us to keep the entire pipeline dormant until new data arrived from on-premises DNA sequencing machines.

Structural Consistency in the State Machine

For the state machines to be fully extendable, we needed to apply structural consistency to each one of the states. By “structural consistency” we mean standardizing the input and output of each state into a common data structure. This common data structure is initialized at the beginning of the pipeline and filled in as the pipeline progresses, until finally reaching the last two states (success or failure) containing all the necessary information to end the pipeline successfully, after which the involved parties are notified. We persisted this data structure to an Amazon Aurora database for later use in cases of pipeline resume operations, or for historical reasons.

One of the key arguments for structural consistency is the ability to extend the state machine to add more intermediate states without worrying about incoming or outgoing data. We can add or remove states without rewriting other ones in the state machine. For example, in Figure 2, we can remove the Data Versioning state/node in the overall state machine and the execution will fall through from Data Governance to Secondary Analysis transparently, without the need to rewrite any logic, and in some cases with minimal effort.

Reusable & Decoupled Components

In Figure 3 below is an example of how we modeled our genomics pipeline solution using reusable and decoupled state machines. These state machines can be nested within one another and easily called as part of other pipelines or in an ad-hoc specific manner. The main benefit for modeling our pipelines like this was to encapsulate complex behavior into a reusable component that we could later on reuse instead of relying on one monolithic state machine. It also allowed us to have separate policies for different steps in the pipeline, e.g., parts of the genomics pipeline could fail, and the execution will continue normally, while others could fail and trigger more complex error handling routines.

Another benefit is that it allows us to call parts of the pipeline in an ad-hoc manner through the use of AWS Lambda functions, e.g., from within computational notebooks or middlewares.

Decoupling the Secondary Analysis Engine

An important aspect of our decoupled approach is the ability to replace the engine in charge of the secondary analyses. Within our solution, we’d normally submit jobs for processing through an HTTP REST endpoint to access this engine, meaning we could effectively replace the engine and preserve the endpoints, and the state machines would remain the same.

Polling Pattern

The rationale behind our decision to use the polling design pattern throughout the architecture was the lack of support for callback endpoints on some of the external components we used to support our workflows. This lack of support forced us to either go with a hybrid approach or to be consistent and use the polling design pattern throughout. We opted for the latter.

Our decision to use this pattern introduced some unnecessary overhead in the system, especially for long running tasks like secondary analyses or external data governance services. We reduced this overhead by decreasing the polling frequency for these cases. E.g., we decreased the polling frequency for long running tasks to minutes instead of seconds.

Being consistent in the use of this pattern allowed for a way to model our architecture that was more understandable and easier to debug, with fewer components to trace when errors occurred.

Scalable, Extendable & Observable Requirements

This architecture is fully scalable and capable of supporting multiple pipelines in parallel. The upper bound of pipelines this architecture can run in parallel is given by the provisioning of underlying Amazon EC2 instances through AWS Batch.

The extendibility quality of this architecture is given by how easily it allows adding or removing processing steps in any state machine definition. This quality is enabled by the structural consistency quality of the pipeline. Because every AWS Lambda function receives the same input and produces the same output, rearranging processing nodes is trivial.

Finally, being able to visualize the execution of pipelines in real time is what pushed our team to use this orchestration approach. One of the main requirements was to be able to observe and visualize pipeline runs at each step, while also being able to determine exactly at which step an error occurred. Using AWS Step Functions allows us to achieve this.

Architecture

Data Uploader

This sub-system is triggered automatically when new data becomes available on-premises.

1. DNA sequencers produce genomic datasets on-premises
2. These datasets are stored on-premises in a workstation or a NAS server
3. An AWS DataSync agent is installed in an on-premises workstation for optimal latency
4. Datasets are synchronized to the cloud over an AWS Direct Connect link for optimal transfer speed
5. Datasets are written into a landing bucket for temporary storage before moving it to the data lake

Data Pipeline

This component is triggered by user input. The user input necessary to run this is mainly a sample sheet file, along with ancillary files specifying the operation of the pipeline. The user uploads these files into a particular bucket, which is the single point of user input within the overall pipeline. We support these operations through ChatOps, notifying users at all times.

1. Datasets live in a landing bucket which acts as temporary storage for genomics datasets synchronized from on-premises
2. An AWS DataSync task copies these datasets to a data lake bucket
3. An Amazon S3 bucket acts as a data lake. Here data is stored in a structured manner, requiring stricter data-retention policies and versioning. It is persistent storage.
4. Stakeholders are notified when a new dataset is ready for processing through ChatOps
5. Stakeholder uploads necessary files for a pipeline run
6. An AWS Lambda function tied to the data lake bucket detects when the user has uploaded the necessary files and triggers a pipeline run
7. AWS Step Functions act as the orchestrator of the genomics pipeline, always keeping stakeholders in the loop through ChatOps
8. Nextflow, acting as the genomics workflow orchestrator, computes the secondary analyses
9. An Amazon S3 bucket contains the artifacts generated by Nextflow,. e.g., logs, reports, etc.
10. An Amazon Aurora database maintains pipeline metadata for historical reasons or to resume pipeline runs

Secondary Analysis Orchestration through Nextflow

For the secondary analysis step, we used Nextflow to run our bioinformatics workflows. It’s a strong fit for large-scale genomic analysis in the cloud because it lets us break down complex pipelines into modular, reusable parts, run jobs in parallel, support pluggable execution backends, and keep everything reproducible with container support baked in.

In our architecture, this phase is encapsulated as a modular, reusable state machine within AWS Step Functions, so it stays separate from both the data upload and the reporting layers. That way it’s easy to reuse the same analysis logic across different pipelines or projects.

Each execution of the pipeline is triggered by a nested state machine, which communicates with the batch job via a polling pattern and periodically checks for pipeline completion rather than relying on callbacks. This design choice reduces architectural complexity, especially in cases where external agents need to trigger analysis jobs without modifying the execution model.

This architecture supports the execution of Nextflow pipelines through any orchestration mechanism or API driven workflow, including AWS Step Functions, AWS Lambda based triggers, or external schedulers, providing full flexibility for integration across diverse environments. It enables execution of both standardized nf-core pipelines and custom analysis workflows, regardless of whether they run on AWS Batch, AWS HealthOmics or self-managed compute backends.

This setup gave us a lot of flexibility, it works with different infrastructure setups and could easily be plugged into a hosted environment if needed, all without making the workflow overly rigid or complicated. This allows teams to adopt the orchestration layer that best fits their operational model, whether that means leveraging managed services for production workflows, integrating with an internal research UI, or executing pipelines programmatically in an ad-hoc or scheduled fashion. By designing the secondary analysis phase as a decoupled, API-triggerable module, we ensure that it can evolve alongside infrastructure changes without requiring a rework of the upstream or downstream components.

Conclusion

The proposed solution is a general solution for organizations looking to natively connect sequencing capability to your AWS tenant. It provides several key benefits:

• Speed - AWS DataSync offers significant advantages over traditional tools like cp, scp, or boto3 for copying data by providing faster, more reliable, and automated transfers. It uses a purpose-built, parallelized protocol that can transfer data up to 10x faster, with built-in features like incremental syncs, data integrity checks, TLS encryption, and native integration with AWS IAM for secure access. Unlike manual methods, AWS DataSync supports scheduling, monitoring via AWS CloudWatch, and automatic error handling, reducing the need for custom scripts. It also supports hybrid storage systems like NFS, SMB, EFS, and FSx, making it ideal for large scale, enterprise grade migrations or syncing across cloud and on-premises environments.
• Resource efficiency by using serverless components. Costs are compounded only when the pipeline is running.
• Full support for genomics pipelines as part of your own platform

References

1. Orchestrating Multiple AWS HealthOmics Workflows at Scale
   Anuj Patel and Lee Pang, 07 May 2024
   https://aws.amazon.com/blogs/industries/orchestrating-multiple-aws-healthomics-workflows-at-scale
2. Building High-Throughput Genomics Batch Workflows on AWS: Introduction (Part 1 of 4)
   Andy Katz, 30 May 2017
   https://aws.amazon.com/blogs/compute/building-high-throughput-genomics-batch-workflows-on-aws-introduction-part-1-of-4/
3. Building High-Throughput Genomics Batch Workflows on AWS: Job Layer (Part 2 of 4)
   Andy Katz, 31 May 2017
   https://aws.amazon.com/blogs/compute/building-high-throughput-genomics-batch-workflows-on-aws-job-layer-part-2-of-4/
4. Building High-Throughput Genomic Batch Workflows on AWS: Batch Layer (Part 3 of 4)
   Andy Katz, 31 May 2017
   https://aws.amazon.com/blogs/compute/building-high-throughput-genomic-batch-workflows-on-aws-batch-layer-part-3-of-4/
5. Building High-Throughput Genomics Batch Workflows on AWS: Workflow Layer (Part 4 of 4)
   Andy Katz, 01 Jun 2017
   https://aws.amazon.com/blogs/compute/building-high-throughput-genomics-batch-workflows-on-aws-workflow-layer-part-4-of-4/
6. Building Simpler Genomics Workflows on AWS Step Functions
   Christie Gifrin, 29 Nov 2018
   https://aws.amazon.com/blogs/compute/building-simpler-genomics-workflows-on-aws-step-functions/
7. Scaling High-Throughput Genomics Workflows on AWS
   Michelle Kung, 18 Sep 2018
   https://aws.amazon.com/blogs/startups/scaling-high-throughput-genomics-workflows-on-aws/
8. Orchestrating high performance computing with AWS Step Functions and AWS Batch
   James Beswick, 12 Apr 2022
   https://aws.amazon.com/blogs/compute/orchestrating-high-performance-computing-with-aws-step-functions-and-aws-batch/
9. Choreography Pattern
   https://learn.microsoft.com/en-us/azure/architecture/patterns/choreography
10. Asynchronous Request-Reply Pattern
    https://learn.microsoft.com/en-us/azure/architecture/patterns/async-request-reply