Post Syndicated from Mark Larson original https://aws.amazon.com/blogs/big-data/how-alight-solutions-achieved-55-cost-savings-with-amazon-opensearch-service/

This is a guest post by Mark Larson, Andrew Kummerow, and Tim Razik at Alight Solutions, in partnership with AWS.

Alight Solutions is a leading cloud-based human capital technology and services provider focused on integrated benefits administration, healthcare navigation, and employee experience solutions. The company serves hundreds of enterprise customers globally, with services that support millions of people worldwide.

Alight’s technology stack generates over 1 billion log records per day across their containerized microservices architecture, with peaks reaching 100,000 records per second during Annual Enrollment periods. Previously, Alight relied on a self-managed Elastic Stack (Elasticsearch, Logstash, Kibana) deployment that had been in production since 2018. As their logging volumes grew and Elasticsearch 7.x approached end of support, the operational burden of maintaining this infrastructure consumed their entire operational budget, leaving no capacity for innovation.

In this post, we share how Alight Solutions migrated from self-managed Elasticsearch to Amazon OpenSearch Service. The migration achieved a 55% cost reduction, alleviated approximately 2,000 hours per year of operational overhead, and gave Alight access to advanced observability features they could not prioritize before.

Challenges with self-managed Elasticsearch

Alight’s self-managed Elastic Stack infrastructure presented compounding technical and operational challenges. Their production environment consisted of 15 Elasticsearch nodes with 168 TB of EBS storage, handling log ingestion from their flagship Alight Worklife system and supporting applications. The infrastructure required an Elastic Platinum subscription, though the team’s operational bandwidth was fully consumed by maintenance, leaving limited capacity to adopt advanced features included in the license.

The operational pain points included:

• Security vulnerability patching required working over Christmas holidays to address critical fixes, with no flexibility on timing.
• Elastic upgrades were time-consuming and required depth of knowledge to manage at scale.
• Logstash using TCP-socket shipping was unreliable, experiencing log loss at high volumes.
• Backpressure from Logstash caused two P1 incidents over two years, where the logging subsystem directly impacted microservice tasks.
• Elasticsearch 7.x approaching end of support created urgency to act before the next Annual Enrollment period (September through January).

Alight was spending more than $100,000 per month on self-managed infrastructure and Elastic licensing across all environments. All operational budget was consumed by cluster maintenance, leaving zero capacity for innovation.

Evaluating alternatives

Alight evaluated several alternatives before selecting OpenSearch Service:

• New Relic and Dynatrace were evaluated for log aggregation but proved prohibitively expensive at Alight’s volume.
• Amazon CloudWatch was evaluated but did not meet requirements for complex log research at their volume and visualization complexity.

Amazon OpenSearch Service is a managed service that makes it straightforward to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. You can use it for use cases such as log analytics and real-time application monitoring. It provisions cluster resources, automatically detects and replaces failed nodes, and scales with a single API call or a few clicks, reducing the operational overhead associated with self-managed infrastructure. It won the evaluation based on five factors:

1. Cost: significantly cheaper than self-managed Elastic Stack and competing solutions.
2. Minimal change management: as a fork of Elasticsearch 7.10, engineers were already familiar with the query syntax and dashboards.
3. Compliance: using a native AWS service avoided hundreds of hours of vendor compliance, audit, and regulatory work. The team spent a few hours getting approval compared to potentially weeks for an external vendor.
4. Cloud-native strategy: aligned with Alight’s overarching strategy to use cloud-native services.
5. Security and data privacy: keeping everything within their AWS landing zone alleviated data egress concerns.

Solution overview

Alight partnered with AWS to design a cloud-native log aggregation architecture that replaced self-managed Elasticsearch and Logstash with Amazon OpenSearch Service and Amazon OpenSearch Ingestion (OSIS), alleviating the operational burden, including the Logstash backpressure that had caused two P1 incidents.

The architecture uses a cross-account model with two primary account types:

• Application accounts: where Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Compute Cloud (Amazon EC2) workloads run, generating logs.
• Shared services account: hosting the OpenSearch domain, OSIS pipelines, and OpenSearch Dashboards.

The following diagram illustrates the solution architecture.

Ingestion paths

The solution supports multiple ingestion paths depending on the application hosting model:

• ECS applications: FireLens/Fluent Bit sidecar containers capture stdout/stderr through the awsfirelens log driver, then ship logs over HTTPS directly to OSIS in the shared services account. ECS task roles assume a cross-account OSIS Ingest Role for authentication.
• EC2 applications: Open-source Fluent Bit (RPM-based, non-containerized) uses tail input to read log files, then ships to OSIS through an EC2 IAM Role with cross-account trust.
• S3-based ingestion (planned): Some applications write to Amazon Simple Storage Service (Amazon S3) with Amazon Simple Queue Service (Amazon SQS) notifications triggering OSIS pipelines.

Spring Boot microservices use a custom logging framework built on Logback (not Log4j) that formats logs as JSON and flushes to console, which FireLens picks up.

Security model

Traffic flows over HTTPS. The security model uses role separation with least privilege:

• OSIS Ingest Role: write-only access to OSIS pipelines, assumed by application account roles via cross-account trust.
• OSIS Sink Role: used by OSIS to write into the OpenSearch domain, with full index access scoped to the ingestion pipeline.
• Security groups: restrict OSIS traffic to known CIDRs and VPCs.

Each application has its own indices, and access is governed by application-specific roles.

Persistent buffering

Amazon Elastic File System (Amazon EFS) provides persistent filesystem buffering for the Fluent Bit sidecar, helping prevent log loss during transient failures or backpressure events. This directly addresses the P1 incidents Alight experienced with Logstash. For the next Annual Enrollment period, Alight plans to also enable persistent buffering at the OSIS layer to handle burst ingestion without log loss.

User access

End-user access to OpenSearch Dashboards is managed through AWS IAM Identity Center with System for Cross-domain Identity Management (SCIM) synchronization from Alight’s enterprise Identity Provider. Users navigate to the Applications tab in Identity Center to access OpenSearch Dashboards over SAML/HTTPS.

At Alight, IAM Identity Center and SCIM are configured in the payer account. They use the same synchronization and entitlement request and approval process that governs Alight’s user and entitlement provisioning into AWS. With this setup, the team uses the same single sign-on (SSO) and entitlement workflow for OpenSearch Dashboards access as for the AWS Management Console, in conjunction with fine-grained access control (FGAC) defined within the OpenSearch domains.

OpenSearch domain configuration

For their production workload, Alight deployed:

Component	Configuration
Data nodes	18 im4gn.2xlarge.search
UltraWarm nodes	9
Dedicated leader nodes	3
Hot tier storage	25 TB
UltraWarm storage	180 TB
Primary logical data	80 TB
Total with replicas	100-105 TB

Additional environments include a secondary production cluster (12 hot nodes, 3 UltraWarm, 3 dedicated leader nodes), plus client test and engineering clusters with 3 hot nodes each.

Migration process

The migration was completed over seven months (February through August 2025), with five applications migrated including the flagship Alight Worklife application.

Infrastructure as code

The team built new Terraform modules to manage deployment of OSIS pipelines, OpenSearch domains, and FireLens sidecar additions to ECS applications. Onboarding new applications is now templatized, resulting in significant time savings compared to adding new indices in Elasticsearch. Onboarding a new application now takes between 4-8 hours, whereas before we would spend 80-120 hours per application.

Migration timeline

Alight first enabled Amazon OpenSearch Service in production for two smaller applications, to make sure operational processes were up and running before migrating the highest volume log producers. For each application, logging to OpenSearch was enabled while continuing to write logs to the existing logging infrastructure. This parallel run allowed fine-tuning of OSIS pipeline configuration, OpenSearch cluster size and configuration before doing a full cutover. This approach also validated that logs were being ingested properly into OpenSearch. It confirmed that the performance of OpenSearch Dashboards and queries was as good as or better than the existing self-managed Elasticsearch cluster.

For historical data, Alight migrated the most recent 30 days of live data from Elasticsearch into OpenSearch just prior to cutover. They also retained a full archive of older log data in an Amazon S3 bucket, so that data older than 30 days could be loaded into OpenSearch on request if a user needs it.

AWS partnership

Alight engaged the AWS team during the evaluation phase. Through AWS Enterprise Support, their Technical Account Manager (TAM) served as the dedicated point of contact throughout the journey. The TAM coordinated sessions with OpenSearch Service subject matter experts to address specific service capabilities, help with design, troubleshoot issues, and provide performance guidance.

Results

The migration to Amazon OpenSearch Service delivered results across cost, operations, and capability dimensions.

“Alight’s mission critical applications are built on hundreds of interdependent microservices, so effective application logging is critical for analyzing system behaviors, performance tuning, and troubleshooting. Amazon OpenSearch Service provides us with great log analytics, very cost effectively at scale, and integrates seamlessly with our IAM strategy for granular access control and authorization. The ability to reconfigure, resize, and upgrade OpenSearch domains with a few clicks and zero downtime is a game changer for us.”

— Mark Larson, Enterprise Architect

Cost and licensing

Metric	Before	After	Improvement
Monthly infrastructure + licensing cost	Self-managed EC2/EBS + Elastic Platinum licensing	Fully managed OpenSearch Service, no separate licensing	~55% cost reduction
Licensing model	Elastic Platinum (fixed)	Zero licensing cost	No longer needed

Not all Elasticsearch clusters are decommissioned yet. Once decommissioning is complete, savings will reach approximately 65%. Additionally, more applications have been added to OpenSearch than were originally on Elasticsearch, making the per-application cost even more favorable. Beyond compute and licensing, the migration also reduced data transfer costs previously incurred across the self-managed cross-account architecture, adding further to the overall savings.

Operational improvements

Metric	Before	After
Engineering hours on cluster management	2,000 hours/year (≈1 FTE)	Near zero (managed service)
Security vulnerability patching	Manual, including holiday work	Handled by AWS
Application onboarding	Manual index creation and configuration	Templatized via Terraform
P1 incidents from logging subsystem	2 in past 2 years	Zero since migration

Performance and scale

Metric	Value
Daily log volume	1 billion records
Peak ingestion rate	100,000 records/second
Applications migrated	5 (including Alight Worklife)
Total data under management	100–105 TB with replicas

Lessons learned and best practices

Through their migration journey, Alight gained the following insights:

• Use your account team relationship to advocate: When Fluent Bit had a blocking issue, the AWS account team relationship helped push for the fix and provided workaround guidance.
• Separate concerns for data durability: Do not put 100% delivery guarantees on logging infrastructure. Use a separate event stream (such as Amazon SQS) for critical data that cannot tolerate loss.
• Templatize everything: Terraform modules for OSIS, OpenSearch domains, and FireLens sidecars reduce the time to onboard new applications.
• Security architecture matters: Separating ingest roles from sync roles (least privilege) and using cross-account trust provides strong security without complexity.
• Plan around business-critical periods: Pausing the production rollout during Annual Enrollment was the right call. The risk of introducing changes during peak was not worth the schedule pressure.

What’s next

Alight has several initiatives planned to expand their OpenSearch Service usage:

• Anomaly detection: top priority, a feature they paid for with Elastic Platinum but never had capacity to implement.
• Amazon OpenSearch Serverless: evaluating for new log sources, particularly interested in zero-OCU baseline for cost optimization.
• OSIS persistent buffer: planned for next Annual Enrollment to handle burst ingestion without log loss.
• Amazon Bedrock AgentCore logging: new artificial intelligence (AI) workloads will send logs to OpenSearch.
• AI-assisted log analytics: adopting the agentic AI capabilities now built into Amazon OpenSearch Service. These include the Investigation Agent for autonomous, hypothesis-driven root cause analysis, which helps site reliability engineering (SRE) and engineering teams gain deeper insights from application logs.
• Vector database: already using OpenSearch as a vector store for a conversational AI assistant (separate team).
• Migration progress: All workloads previously logging to Elasticsearch have been migrated to OpenSearch, plus an additional eight applications.
• Enterprise Logging Service: All new applications will now log to Amazon OpenSearch Service by default using the templatized approach.
• Decommission: All existing Elasticsearch instances will be decommissioned by July 2026.

Conclusion

Alight’s migration from self-managed Elasticsearch to Amazon OpenSearch Service demonstrates how enterprises can alleviate operational burden while achieving significant cost savings. By using Amazon OpenSearch Ingestion and FireLens, Alight built a scalable log aggregation system that handles 1 billion records per day with zero P1 incidents since deployment.

The 55% cost reduction and approximately 2,000 hours per year of recovered engineering time have freed Alight to pursue advanced observability capabilities like anomaly detection and AI-powered log analytics, features they paid for but could never use under the operational weight of self-managed infrastructure.

To learn more, see the Amazon OpenSearch Service documentation. To get started with ingestion pipelines, see Amazon OpenSearch Ingestion. For migration guidance, see Migrating to Amazon OpenSearch Service.

---

About the authors