Post Syndicated from Rubén Romero Córdoba original https://aws.amazon.com/blogs/big-data/how-moeve-standardized-dbt-runs-across-data-lakes-with-amazon-athena/

As organizations grow, data processing often becomes fragmented across teams, environments, and orchestration tools. This fragmentation leads to inconsistent patterns, duplicated logic, limited cost visibility, and operational overhead.

At Moeve we were no exception. Our analytics teams build their transformations with dbt, an open source tool that defines transformations as SQL models, resolves the references between them, and works out the order in which they run. dbt describes what to transform, but it does not define where or how a project runs. We left that decision to each team, and as the number of projects grew we found fragmented pipelines, inconsistent compute engines, and limited cost visibility slowing every project down. Standardizing our dbt runs on Amazon Athena was how we worked our way out of that.

This post describes the architecture of the centralized, serverless solution we built on Athena, which reduced onboarding for a new dbt project from days to about 15 minutes. The solution centralizes how dbt runs across our data lakes while staying loosely coupled from orchestration. It uses Amazon Athena as the default processing engine, a centralized dbt launcher, and a shared event bus for downstream orchestration.

In the sections that follow we explain how we decoupled dbt runs from orchestration using AWS Step Functions and AWS Fargate, why Athena fits our workloads from a cost and operational perspective, how storing run parameters in Amazon DynamoDB rather than in pipeline code removed the infrastructure deployment step from onboarding, and how publishing results to Amazon EventBridge lets our run and orchestration layers evolve independently.

The challenge of running dbt at scale

Before the dbt launcher, our dbt runs had grown in different directions. Run logic was embedded in project-specific pipelines, orchestration and processing were tightly coupled, teams selected different compute engines for comparable workloads, and we had no consistent governance over run parameters and retries.

As the number of dbt projects increased, this made it difficult to enforce consistent standards and to evolve the solution without touching every pipeline. We needed a way to standardize dbt runs across our data lakes, decouple running a project from deciding what to run, improve cost control and observability, and deliver faster and safer continuous integration and continuous delivery (CI/CD) iterations.

Why Amazon Athena as the default dbt engine

Choosing the processing engine for dbt is a foundational architectural decision, so we made it first.

Serverless processing

Athena is fully serverless. There are no clusters to provision, scale, or maintain. Our teams run their queries with the default Athena pricing, which charges for the data a query scans and gives us elasticity with no capacity planning.

Because each data lake lives in its own account, each team makes its own decision about Athena payment. A team whose workload grows into continuous, high-concurrency usage can move to Athena capacity reservations with no change to the launcher, to their dbt profiles, or to their project configuration. None of our teams have needed to do so yet, and the architecture keeps that choice independent per team.

Our workload is predictable but not continuous. Each dbt project runs for a few minutes when its schedule fires or when its upstream data lands, then stays idle until the next trigger. That shape is what made serverless the right fit for us.

Why Athena fit our solution

For Moeve, the decision to standardize on dbt and Amazon Athena was driven by our goal of creating a common transformation solution that could be adopted across multiple teams and AWS accounts while keeping operations lightweight.

Our data was already stored in Amazon S3 and registered in the AWS Glue Data Catalog, making Athena a natural processing layer. Athena allowed us to run transformations without managing clusters, capacity, or infrastructure, which was particularly important for a small central team supporting multiple domains.

We evaluated alternative processing engines, but for our workload profile and data volumes, Athena provided the best balance between scalability, operational simplicity, and maintainability. Adapter maturity was another important factor. The dbt Athena adapter offered strong integration with testing, CI/CD workflows, and the broader dbt ecosystem, reducing the operational risk of maintaining custom solutions.

Athena also aligned naturally with our architecture. Transformations run in the AWS account that owns the data through cross-account role assumption, while orchestration remains centralized. As a result, we standardized how projects run across teams while keeping compute close to the data.

Finally, the Apache Iceberg support in Athena underpins the idempotent incremental processing model described in this post, so incremental loads and historical reprocessing follow the same path with minimal operational overhead.

Optimized incremental processing

Our largest cost was not reading source data. It was merging into it.

Our fact tables are Apache Iceberg tables in the data lake, and most of our models are incremental. Each run brings in new or corrected records and merges them into a target that can hold several years of history. A merge has to locate the rows it is about to update, and without a predicate on the target the query reads far more of the table than the incoming data can affect. The common approach is a static filter such as the last 30 days, which is wrong in both directions: too wide for an ordinary daily load, and too narrow as soon as a correction arrives for an older partition.

Instead of a fixed window, the platform derives the predicate from the data. Before the merge runs, it reads the distinct values of the partition column present in the incoming dataset and builds the target predicate from them. For a single partition it applies an equality predicate, for a small set an IN list, and for a larger set a bounded range. The merge then reads only the partitions the incoming data can affect.

The same principle applies on the source side. The launcher builds the source filter from the parameters given for that run: an explicit range, an arbitrary SQL condition, or, when neither is supplied, a default window taken from the project configuration. Input is therefore bounded to the subset each run needs.

Two results mattered to us. Because Athena charges for the data a query scans, narrowing both ends of the merge reduces cost without any team hand-tuning individual models. More importantly, a daily load and a full historical reprocess became the same operation with different inputs. Every run is idempotent, so the same input always produces the same result regardless of how many times it runs. That removed the distinction between processing and reprocessing from our runbooks and simplified incident response.

Table design is what makes this pruning possible. Partitioning, columnar formats, and compression all contribute, and the AWS Big Data Blog post Top 10 performance tuning tips for Amazon Athena covers the general techniques. We have deliberately not published a before and after figure here, because the saving depends so heavily on partition design and data distribution that a single number would mislead without extensive context.

Architecture overview

Moeve built a centralized dbt launcher that runs dbt jobs in a uniform way, regardless of the project or the target data lake.

Architecture diagram

At a high level, the architecture consists of:

• AWS Step Functions to control the run lifecycle.
• AWS Fargate to run dbt in an isolated, ephemeral container.
• Amazon Athena as the default dbt processing engine.
• Amazon DynamoDB to store dbt project configuration.
• Amazon EventBridge to publish run results.

Every dbt run follows the same contract, which gives us consistency and reduces the operational surface we must maintain.

Cross-account processing model

The solution operates in a centralized account while running transformations in domain-specific data lake accounts: corporate, marketing, and manufacturing. The Fargate container assumes a dbt-child IAM role in the target account, so the container processes data where it lives while governance stays centralized.

Each data lake account keeps control of its own IAM permissions and manages its own storage and catalog without affecting the solution. This also puts costs in the right account. We could have attributed Athena spend using Athena workgroups, but Athena is only part of what a query costs. The Amazon Simple Storage Service (Amazon S3) requests it makes and the AWS Key Management Service (AWS KMS) operations it triggers are real costs as well, and running in the owning account attributes all of them to the team that owns the data, per project and per run.

Centralizing dbt runs with the dbt launcher

To stop every team inventing its own way of running dbt, we built a single launcher that all of them go through. Instead of dbt logic living inside multiple pipelines, every run is triggered through one well-defined path.

Run lifecycle

The launcher is an AWS Step Functions state machine. It receives a run request with its parameters, starts an AWS Fargate task from our dbt container image, and the container assumes the IAM role of the target data lake account. dbt then runs its SQL transformations in Athena, reading the source tables and materializing the targets. Alongside the run, Elementary, an open source dbt package, records model-level results and data quality test outcomes. When the run finishes, the launcher publishes a completion event to Amazon EventBridge.

The state machine can be started in different ways depending on the scenario. Some projects run on a schedule, others are triggered when upstream data lands, and teams can request a run on demand. Those decisions are made by our orchestration layer, which submits a standardized run request to the launcher. The launcher therefore stays focused on running dbt projects, regardless of how the run was initiated.

This sequence is identical for all projects and environments. The launcher is responsible only for running the project it was asked to run. It doesn’t decide what should run next, and that responsibility is intentionally delegated to downstream consumers through Amazon EventBridge.

Configuration-driven runs with Amazon DynamoDB

We had to decide where a project’s run parameters would live. In the pipeline definition, changing a timeout would be a code change, a review, a build, and a deployment, for a value we sometimes need to change while an incident is open. We put the parameters in DynamoDB instead, keyed by project, and the launcher reads them at the start of every run. That lets us decouple code deployment from run behavior, update parameters without redeploying services, and enforce consistent defaults across all dbt projects.

The parameters themselves are modest. They cover the target environment and AWS Identity and Access Management (IAM) role, the default processing engine, how long to allow a run to take, how many times to retry on failure, and how many days of data to process by default. These values change for operational reasons rather than logical ones, which is why we did not want them coupled to a release cycle.

The effect on onboarding was larger than we expected. Deploying a new dbt project is now a merge of the dbt models and one configuration entry. There is no Terraform change, no infrastructure review, and nothing to provision, because the compute the project needs already exists and is shared. What used to be a multi-step infrastructure pipeline is now a single CI workflow that validates the project’s SQL and lineage locally, then merges and registers it. We run that local validation with DuckDB, which returns feedback in under two minutes without consuming cloud resources.

Governance did not weaken as a result. Who may change a configuration entry is controlled the same way as any other production change. What changed is that the change no longer has to travel through an infrastructure deployment to take effect.

CI/CD pipeline diagram

Athena remains the engine of record. Local validation catches Jinja errors, unresolved references, and obvious SQL mistakes, but Athena-specific behavior, cross-account permissions, and AWS Glue Data Catalog interactions are only proven in the target environment. It’s important to be explicit about that boundary with the teams, so that a green CI run is not read as a guarantee.

Publishing run results with Amazon EventBridge

After a dbt run finishes, the launcher publishes a structured event to a central Amazon EventBridge bus recording whether the run succeeded, which project and source it covered, when it started and finished, how long it took, and which datasets it updated. The launcher does not know which consumers are subscribed.

This event-driven approach gives us loose coupling between running a project and orchestrating what comes next, multiple downstream consumers for the same signal, and independent evolution of both layers.

At Moeve the main consumer is our orchestration layer, which models the dependencies between datasets as a graph. Each completion event tells it that a node is now up to date, so it can determine which downstream projects have all their inputs ready and start them. That consumer has no special status. An AWS Lambda function, a monitoring dashboard, or a notification integration can subscribe to the same events without any change to the launcher.

Validation and observability

Because every project follows the same lifecycle, we get validation and observability in one place instead of per pipeline. Step Functions shows the state of any run and the step at which it failed, and error handling and retries are defined once. Fargate logs carry the container runtime detail. dbt and Elementary report model-level results and data quality test outcomes. The completion event on the bus is the auditable record that a project finished and what it produced. Together these layers give us operational visibility without coupling the components to each other.

Cost control and resource cleanup

The platform is serverless end to end, which keeps idle cost close to zero and removes a class of operational mistake. Fargate tasks are created for a run and destroyed when it ends, so no long-running container needs maintenance. Athena has no persistent compute. An idle Step Functions state machine costs nothing. Nothing is left running unintentionally, which matters when the number of projects on the platform keeps growing.

Results

The metrics in the following table are the ones our teams notice day to day, and the reason the solution is maintainable by a small central team.

Metric	Before	After
Onboarding time for a new dbt project	Days, including pipeline and infrastructure setup	About 15 minutes, configuration only
Run consistency	Varied by team	Same lifecycle and contract for every project
CI feedback time	8 to 10 minutes on Jenkins	Under 2 minutes with local validation
Cost visibility	Per-account aggregate	Per-project and per-run attribution
Operational overhead	One pipeline per project	One solution for all projects

Conclusion

Standardizing how we run dbt turned out to depend less on dbt than on defining two boundaries clearly.

The first is the contract of the launcher: parameters in, event out. We defined that interface before building the internals, and it has stayed stable while the implementation changed several times. The second is the separation between running a project and deciding what to run next. Making the launcher publish events without knowing its consumers is why our orchestration layer could be rebuilt while the launcher stayed as it was, and the launcher has never been modified to accommodate a new orchestration requirement.

Two smaller decisions carried more weight than we expected. Keeping run parameters in DynamoDB rather than in pipeline definitions means timeouts, retries, and engine selection can be changed without a deployment, which is valuable during incident response. Making every run idempotent removed the distinction between processing and reprocessing, so a daily load and a full historical reprocess are the same operation with different inputs.

Athena is serverless, so our teams did not need to set up infrastructure of their own. That is what made it practical for everyone to run dbt in a standardized way, and why onboarding a new project went from days to about 15 minutes.

If your organization runs dbt across multiple accounts and teams, pair Amazon Athena with a clear contract for the component that runs your projects: fixed parameters in, a published event out.

Resources

• Amazon Athena documentation
• AWS Step Functions Developer Guide
• Amazon EventBridge User Guide
• Top 10 performance tuning tips for Amazon Athena
• Using Apache Iceberg tables in Amazon Athena
• dbt documentation

---

About the authors