All posts by Kevin Fernandez

How Veeva scaled big data with Amazon EMR on EKS

Post Syndicated from Kevin Fernandez original https://aws.amazon.com/blogs/big-data/how-veeva-scaled-big-data-with-amazon-emr-on-eks/

With biopharma commercial teams facing increasingly complex big data challenges, Veeva used Amazon EMR on EKS to increase speed and efficiency in delivering performance data to biopharma customers, supporting insight generation to find and treat more patients. Therapies designated as “orphan” in the United States that treat rare diseases affecting fewer than 200,000 Americans made up half of U.S. novel drug approvals in 2025, and the continuing shift toward developing therapies for rare and complex disease states only amplifies the need for scalable analytics infrastructure.

Veeva Systems, a leader in cloud-based software for the global life sciences industry, addresses this changing data landscape with Veeva Compass and the Compass Data Network. Compass unifies prescriptions, in-office procedures, and diagnoses into a suite of data products customers use to segment economies with greater precision, prioritize the right healthcare providers, and bring the right therapies to more patients, faster.

Veeva is continuously evaluating and adding new data partners to the Compass network, using both closed and open sources to fill the gaps left by traditional patient data products and meet the biopharma industry’s current and future data needs. To support this ever-growing network, the company undertook an ambitious architecture transformation with Amazon EMR on EKS. Veeva Compass uses faster, more efficient compute to deliver data from suppliers to customers within hours while drastically increasing efficiency by accelerating analytics and supporting AI workflows. This system flags new patient and healthcare provider (HCP) events earlier so that medical reps can engage with healthcare providers at the right time with more precise information.

This post explores the AWS analytics architecture that powers the petabyte-scale elastic big data processing Veeva Compass implemented with Amazon EMR on EKS. Specifically, it explores how Compass avoided execution slowdowns while continuing to increase data volumes to meet customer needs.

The challenges of extracting data insights at petabyte scale

The foundation of Veeva Compass is an analytics lake that was originally designed to process hundreds of terabytes daily. Through years of consistent growth fueled by global customer expansion and the growing gap between legacy data products and industry needs, the Veeva Compass analytics lake grew. It increased support for the number of data sources and Veeva products. This resulted in an exponential growth of data volumes. With the quantity and size of batch jobs becoming more variable, Veeva decided to revisit the original Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2) architecture.

The batch job variability challenges manifested in a few important ways. Whenever Spark executors ran out of memory, the shuffle stages in Spark were unable to compensate, and executor tasks would spill to disk. This would add hours to overall job execution time with executors relying on disk storage to complete their tasks. A significant portion of the analytics budget was allocated to several provisioned Amazon EMR clusters that were idle during the afternoon waiting for the next series of job runs. To better use these clusters, Veeva distributed jobs across resources. But that led to noisy neighbor challenges where one team’s large join operation would monopolize the cluster and block another team’s service level agreement (SLA)-critical pipeline.

Veeva needed a more scalable solution that could scale out to hundreds of Spark executors in minutes when jobs arrived, then scale in when jobs complete, delivering both performance and cost efficiency. Veeva needed to rethink how it provisioned compute resources and managed Spark workloads.

The solution: Amazon EMR on EKS with Karpenter

Let’s explore how Veeva used Amazon EMR on EKS with Karpenter to scale its analytics capabilities and reduce Spark run times. Amazon EMR on EKS runs each Spark application as an ephemeral job inside Kubernetes, decoupling compute from storage. Data lives in Amazon Simple Storage Service (Amazon S3), durable and accessible. Spark executors run as pods that are created and terminated with each job. With Amazon EMR on EKS, Veeva is no longer dependent on always-on clusters and consumes compute only when needed.

The diversity of data types in the analytics workloads requires fine-grained control to provision the optimal compute nodes for each job. Veeva’s original cluster design relied on pre-configured node groups, where you define instance types and sizes upfront. This approach worked for other Veeva workload profiles, but Spark workloads are different. One job might need memory-optimized instances for wide joins, while another needs compute-optimized nodes for CPU-intensive transformations.

To address this challenge, Veeva added real-time EC2 instance type provisioning using Karpenter. For example, a Spark job might request 800 executors with 16 GB memory and 4 cores each. Karpenter reacts to pending pod requests in 60–90 seconds, analyzes the requirements, selects optimal instance types from across over 20 families, and launches nodes. When the job is complete, those nodes terminate within minutes. Karpenter also handles bin-packing, which is a placement strategy where pods are consolidated into fewer EKS nodes to optimize cost and improve resource utilization.

Amazon EMR on EKS with Karpenter architecture overview

Veeva’s analytics architecture follows a medallion data design pattern. This pattern progressively refines data in stages: bronze (raw), silver (cleaned), and gold (aggregated) layers within the same storage system.

Amazon EMR on EKS architecture with S3 storage, Spark compute, Karpenter orchestration, and Trino query layers

Figure 1: Amazon EMR on EKS architecture spanning the storage, compute, orchestration, and query layers

The architecture comprises four layers:

  • Storage layer: Amazon S3 stores data across bronze (raw), silver (cleaned), and gold (business-ready aggregated) layers. S3 organizes the data in Delta Lake format with daily partitions and continuous compaction to help prevent small-file proliferation.
  • Compute layer: Amazon EMR on EKS runs Spark applications as Kubernetes jobs. Each application gets its own isolated set of pods that exist only for the duration of that job.
  • Orchestration layer: Karpenter watches for pending Spark executor pods and provisions EC2 nodes to run them. It selects instance types based on pod resource requests, Availability Zone (AZ) capacity, and Spot pricing, then terminates nodes when they’re no longer needed.
  • Query layer: An Amazon EMR on EKS cluster running Trino (formerly PrestoSQL) supports interactive queries against the transformed Delta tables, so analysts can explore petabyte-scale datasets with sub-second response times.

This separation of concerns delivers the elasticity required for petabyte-scale operations.

Spark application layer instance types

Veeva selected three distinct node profiles optimized for different Spark workload characteristics:

Driver nodes: Spark drivers and Airflow workers run on On-Demand Instances for reliability. Drivers coordinate job execution and can’t tolerate interruptions because of Spot instance reclamations. These nodes run continuously but represent less than 5 percent of total compute cost.

Executor nodes: General Extract, Transform, Load (ETL) and memory-intensive joins run on approximately 70 percent Spot Instances. Karpenter spreads across multiple instance families to maximize Spot availability. When interruptions occur, Spark’s checkpointing and Adaptive Query Execution handle recovery automatically.

Shuffle-heavy nodes: Workloads with high shuffle I/O run on NVMe-backed instances. The local SSD storage improves performance when executors spill to disk during wide joins. These run at approximately 50 percent Spot because the NVMe instance families have slightly lower Spot availability.

Partition strategy and tuning for large datasets

With the compute and orchestration layers optimized, Veeva turned its attention to improving the partition strategy, data layout, and Spark configuration.

Partition strategy: Veeva’s data is partitioned by date (YYYY-MM-DD) plus a high-cardinality dimension like region or customer segment. This simplifies partition pruning while avoiding the small-file problems that degrade query performance. Target file size is around 256–512 MB per Parquet file. Nightly compaction jobs use Delta Lake’s OPTIMIZE command with Z-ordering on frequently filtered columns, which improved query performance by 3–5x.

Shuffle and query tuning: Using Adaptive Query Execution (AQE) as a query optimization capability transformed Veeva’s most problematic Spark workload. AQE allows Spark to dynamically adjust the query plan based on runtime statistics rather than relying on static estimates. To enable these dynamic adjustments, enable AQE, configure initial partition counts for wide joins, handle data skew automatically, and enable speculative execution to recover from stragglers.

Additionally, Veeva caches some of the dimension tables in memory to avoid repeated reads from S3. For jobs that reference the same lookup tables multiple times, this optimization reduced runtime by 20–30 percent.

The combination of proper partitioning, continuous compaction, and AQE tuning reduced the skewed join stage run time from 7 hours to 55 minutes.

Observability as the foundational component for Amazon EMR on EKS operations

An observability architecture that monitors the entire stack and delivers operational insights is essential for maintaining SLAs and troubleshooting performance issues before they impact production pipelines.

The observability stack has four layers:

  • Spark History Server runs on EKS with S3-backed event logs. With Spark History Server, you can replay the entire execution graph, examine stage-level metrics, and identify bottlenecks.
  • Prometheus and Grafana track real-time metrics. Key metrics include executor CPU and memory utilization, pending-pod queue depth, shuffle-spill ratios, and Karpenter provisioning latency.
  • Amazon CloudWatch Logs Insights aggregates driver and executor logs with automated alerting. When a job fails, Amazon CloudWatch sends an immediate notification with stack traces and error context.
  • OpenTelemetry traces link Spark jobs to the broader orchestration layer (Airflow, AWS Step Functions). This end-to-end visibility provides insights into how individual job performance impacts downstream dependencies.

To help proactively identify bottlenecks before they cascade into SLA violations, Veeva configured critical alerts such as pending pods exceeding thresholds indicating capacity constraints, high shuffle-spill ratios signaling memory pressure, and excessive repartitioning indicating significant data skew.

Securing multi-tenant data analytics

Datasets that include prescription and claims data require robust security controls. Veeva’s architecture follows AWS best practices for least-privilege access and strict tenant isolation.

IAM Roles for Service Accounts (IRSA) provide each Spark job with temporary credentials scoped to specific S3 prefixes and AWS Lake Formation permissions. Jobs can read from bronze and silver zones but only write to their designated output paths. Using Kubernetes namespaces with taints and tolerations clearly defines workload placement. Each team operates in their own namespace with dedicated service accounts and resource quotas.

Encryption is enforced at rest with AWS Key Management Service (AWS KMS) and in transit using TLS. Amazon S3 uses customer-managed keys to encrypt all data. Spark shuffle data is encrypted using Spark’s built-in encryption. Lineage and auditing through AWS Lake Formation and OpenLineage track data transformations across the entire pipeline.

Measurable improvements

The migration from Amazon EMR on EC2 to Amazon EMR on EKS with Karpenter produced measurable improvements across every key metric:

Metric Amazon EMR on EC2 Amazon EMR on EKS + Karpenter Improvement
Bronze→Silver ETL (1 PB) 9h 12m 4h 07m 55% faster
Daily 20 TB compaction 3h 40m 1h 55m 48% faster
Scale-out 40→520 nodes 18 min 5m 42s 68% faster
Idle EC2 cost/month $42,000.00 $0.00 100% reduction
Spot instance share 0% ~70% ~63% cost savings
Bin-packing failures Frequent Near-zero —

The largest improvement came from scale-to-zero capability. The previous Amazon EMR cluster design ran 24/7 to avoid the 15–20-minute startup time, costing $42,000 monthly in idle compute. With Karpenter, nodes launch in under 6 minutes and terminate automatically, alleviating idle costs. This reduced launch time matters when you have SLA-critical pipelines that need to start processing immediately. For the largest bronze-to-silver ETL job processing approximately 1 PB of data, Karpenter scaled the cluster from 40 to 520 nodes in 5 minutes 42 seconds. It then spun everything down minutes after completion. That level of elasticity is difficult to achieve with traditional automated scaling approaches. By diversifying across over 20 Spot instance types, Karpenter achieves higher Spot availability and lower interruption rates. The executor fleet runs approximately 70 percent Spot, delivering approximately 63 percent cost savings compared to On-Demand pricing.

For Veeva Compass’s business, the speed improvements are as impactful as the cost savings. Several of their most common customer use cases are highly dependent on data delivery speed. For example, medical field reps at a biopharma company rely on alerts from patient data to guide their engagement strategies day to day. The faster Veeva can deliver a new diagnosis or procedure to a customer, the faster that customer’s field team can act.

Lessons learned along the implementation journey

After a year of operating petabyte-scale analytics on Amazon EMR on EKS, here are the lessons that delivered the most value:

  • Partition hygiene matters more than hardware. Veeva spent weeks tuning executor memory and instance types before realizing the real bottleneck was small-file proliferation. Implementing nightly compaction reduced job runtime by 40 percent without changing cluster configuration.
  • Compaction must be frequent. Waiting for monthly compaction windows creates trillions of tiny Parquet files that degrade query performance. Daily compaction on hot partitions and weekly compaction on cold partitions keeps file counts manageable.
  • Design for Spot interruptions. Assume 5–10 percent hourly churn in your executor fleet. Use Spark checkpointing for long-running jobs, enable AQE for query optimization, and rely on Spark retries and re-computation to handle executor failures.
  • Synthetic benchmarking pays off. Before migrating production workloads, Veeva ran synthetic tests at 1 PB, 2 PB, and 5 PB scale. This exposed Garbage Collection (GC) pressure and shuffle limits that would have caused production incidents.
  • Speculative execution saves hours. For skew-heavy wide joins, enabling speculative execution allows Spark to launch backup tasks for stragglers. This single configuration change reduced the skewed join stage run time from 7 hours to 55 minutes.
  • Karpenter consolidation compounds savings. Beyond Spot pricing, Karpenter’s bin-packing and consolidation features deliver another 8–12 percent EC2 savings.

Conclusion

Veeva Compass’s journey from Amazon EMR on EC2 to Amazon EMR on EKS with Karpenter demonstrates how modern cloud-native architectures can transform petabyte-scale analytics workloads. By embracing ephemeral compute, intelligent autoscaling, and continuous optimization, Veeva achieved measurable improvements in both performance and cost efficiency. They reduced critical ETL runtimes by over 50 percent while alleviating $42,000 in monthly idle compute costs.

These cost and performance improvements are important for Veeva Compass’s business practice. The ability to quickly serve large amounts of data to biopharma commercial teams enables faster and more precise analytics, especially as biopharma companies continue to drive increased data demand by exploring AI use cases. These improvements help Veeva Compass customers act with speed and precision to quickly deliver the right therapies to patients in need.

If you’re facing similar challenges with growing data volumes and complex Spark workloads, consider exploring Amazon EMR on EKS and Karpenter as a foundation for elastic, cost-effective big data processing. To learn more about building scalable analytics on AWS, visit the Amazon EMR documentation and explore Karpenter best practices.

 


About the authors

Victor Cuascut

Victor Cuascut

Victor is Senior Director of DevOps at Veeva, where he leads DevOps and infrastructure supporting big data workflows across the Compass Patient, Prescriber, Data Connector, Pathway, National, and Opendata products. His work focuses on building and operating secure, reliable cloud infrastructure and data platforms, with an emphasis on automation, scalability, and compliance.

Tyler Lochan

Tyler Lochan

Tyler is a Product Marketing Manager at Veeva Systems, on the Veeva Compass team. He works closely with field marketing, sales, strategy, and product teams to understand the market, shape messaging, create content, and contribute to GTM planning.

Kevin Fernandez

Kevin Fernandez

Kevin is a Senior Solutions Architect at AWS. He helps customers make informed decisions on how to use the AWS Cloud as an enabler of their businesses. He has over 25 years of experience across a diverse set of technologies including enterprise storage, virtualization, DR/BC solutions, SaaS applications, and AI/ML. In addition to working with AWS ISV customers, he advises customers migrating to and modernizing on AWS.

Imaan Tariq

Imaan Tariq

Imaan is a Customer Solutions Manager at Amazon Web Services (AWS), where she partners with ISVs to drive complex migration, modernization, and AI innovation journeys. She specializes in guiding customers through the full generative AI lifecycle, from early use-case ideation to production-ready solutions, building deep trust through hands-on collaboration and technical curiosity. An active GenAI Ambassador, Imaan is constantly experimenting with emerging AI tools and frameworks to push the boundaries of what’s possible for customers. A Software Engineering graduate of San José State University, she holds four AWS certifications: Solutions Architect Associate, Cloud Practitioner, AI Practitioner, and Developer Associate.

Siddharth Vyas

Siddharth Vyas

Siddharth is a Senior Technical Account Manager at AWS with over 7 years of experience, serving as a strategic technical advisor to ISV customers. He helps organizations navigate architecture decisions, strengthen operational resilience, and maximize their cloud journey.