Many organizations running VMware workloads on premises want to move to the cloud to benefit from improved scalability, reliability, and access to cloud services, but migrating these workloads often requires substantial changes to applications and infrastructure configurations. Amazon EVS lets customers continue using their existing VMware expertise and tools without having to re-architect applications or change established practices, thereby simplifying the migration process while providing access to AWS’s scale, reliability, and broad set of services.
With Amazon EVS, you can run VMware workloads directly in your Amazon VPC. This gives you full control over your environments while being on AWS infrastructure. You can extend your on-premises networks and migrate workloads without changing IP addresses or operational runbooks, reducing complexity and risk.
Key capabilities and features
Amazon EVS delivers a comprehensive set of capabilities designed to streamline your VMware workload migration and management experience. The service enables seamless workload migration without the need for replatforming or changing hypervisors, which means you can maintain your existing infrastructure investments while moving to AWS. Through an intuitive, guided workflow on the AWS Management Console, you can efficiently provision and configure your EVS environments, significantly reducing the complexity to migrate your workloads to AWS.
With Amazon EVS, you can deploy a fully functional VCF environment running on AWS in a few hours. This process eliminates many of the manual steps and potential configuration errors that often occur during traditional deployments. Furthermore, with Amazon EVS you can optimize your virtualization stack on AWS. Given the VCF environment runs inside your VPC, you have full full administrative access to the environment and the associated management appliances. You also have the ability to integrate third-party solutions, from external storage such as Amazon FSx for NetApp ONTAP or Pure Cloud Block Store or backup solutions such as Veeam Backup and Replication.
The service also gives you the ability to self-manage or work with AWS Partners to build, manage, and operate your environments. This provides you with flexibility to match your approach with your overall goals.
Setting up a new VCF environment
Organizations can streamline their setup process by ensuring they have all the necessary pre-requisites in place ahead of creating a new VCF environment. These prerequisites include having an active AWS account, configuring the appropriate AWS Identity and Access Management (IAM) permissions, and setting up a Amazon VPC with sufficient CIDR space and two Route Server endpoints, with each endpoint having its own peer. Additionally, customers will need to have their VMware Cloud Foundation license keys ready, secure Amazon EC2 capacity reservations specifically for i4i.metal instances, and prepare their VLAN subnet information planning.
To help ensure a smooth deployment process, we’ve provided a Getting started hub, which you can access from the EVS homepage as well as a comprehensive guide in our documentation. By following these preparation steps, you can avoid potential setup delays and ensure a successful environment creation.
Let’s walk through the process of setting up a new VCF environment using Amazon EVS.
You will need to provide your Site ID, which is allocated by Broadcom when purchasing VCF licenses, along with your license keys. To ensure a successful initial deployment, you should verify you have sufficient licensing coverage for a minimum of 256 cores. This translates to at least four i4i.metal instances, with each instance providing 64 physical cores.
This licensing requirement helps you maintain optimal performance and ensures your environment meets the necessary infrastructure specifications. By confirming these requirements upfront, you can avoid potential deployment delays and ensure a smooth setup process.
Once you have provided all the required details, you will be prompted to specify your host details. These are the underlying Amazon EC2 instances that your VCF environment will get deployed in.
Once you have filled out details for each of your host instances, you will need to configure your networking and management appliance DNS details. For further information on how to create a new VCF environment on Amazon EVS, follow the documentation here.
After you have created your VCF environment, you will be able to look over all of the host and configuration details through the AWS Console.
Additional things to know
Amazon EVS currently supports VCF version 5.2.1 and runs on i4i.metal instances. Future releases will expand VCF versions, licensing options, and more instance type support to provide even more flexibility for your deployments.
Amazon EVS provides flexible storage options. Your Amazon EVS local Instance storage is powered by VMware’s vSAN solution, which pools local disks across multiple ESXi hosts into a single distributed datastore. To scale your storage, you can leverage external Network File System (NFS) or iSCSI-based storage solutions. For example, Amazon FSx for NetApp ONTAP is particularly well-suited for use as an NFS datastore or shared block storage over iSCSI.
Additionally, Amazon EVS makes connecting your on-premises environments to AWS simple. You can connect from on-premises vSphere environment into Amazon EVS using a Direct Connect connection or a VPN that terminates into a transit gateway. Amazon EVS also manages the underlying connectivity from your VLAN subnets into your VMs.
AWS provides comprehensive support for all AWS services deployed by Amazon EVS, handling direct customer support while engaging with Broadcom for advanced support needs. Customers must maintain AWS Business Support on accounts running the service.
Availability and pricing
Amazon EVS is now generally available in US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Frankfurt), Europe (Ireland), and Asia Pacific (Tokyo) AWS Regions, with additional Regions coming soon. Pricing is based on the Amazon EC2 instances and AWS resources you use, with no minimum fees or upfront commitments.
Effective log analysis is essential for maintaining the health and performance of modern applications. Amazon OpenSearch Service stands out as a powerful, fully managed solution for log analytics and observability. With its advanced indexing, full-text search, and real-time analytics capabilities, OpenSearch Service makes it possible for organizations to seamlessly ingest, process, and search log data across diverse sources—including AWS services like Amazon CloudWatch, VPC Flow Logs, and more.
With OpenSearch Dashboards, you can turn indexed log data into actionable visualizations that reveal insights and help detect anomalies. By querying data stored in OpenSearch Service, you can extract relevant information and display it using a variety of visualization types—such as line charts, bar graphs, pie charts, heatmaps, and more. These tools make it effortless to monitor system behavior, spot trends, and quickly identify issues in your environment.
This post demonstrates how to harness OpenSearch Dashboards to analyze logs visually and interactively. With this solution, IT administrators, developers, and DevOps engineers can create custom dashboards to monitor system behavior, detect anomalies early, and troubleshoot issues faster through interactive charts and graphs.
Solution overview
In this post, we show how to create an index pattern in OpenSearch Dashboards, create two types of visualizations, and display these visualizations on a custom dashboard. We also demonstrate how to export and import visualizations.
Prerequisites
Before diving into log analysis with OpenSearch Dashboards, you must have the following:
For this post, we work with the following log sources, which have already been ingested into an OpenSearch Service cluster as part of the prerequisite steps:
Complete the following steps to access OpenSearch Dashboards:
On the OpenSearch Service console, choose Domains in the navigation pane.
Check if your domain status shows as Active.
Choose your domain to open the domain details page.
Choose the OpenSearch Dashboards URL to open it in a new browser window.
Authenticate into OpenSearch Dashboards using one of the supported methods.
Create an index pattern
After you’re logged in to OpenSearch Dashboards, you must create an index pattern. An index pattern allows OpenSearch Dashboards to locate indexes to search. Complete the following steps
In OpenSearch Dashboards, expand the navigation pane and choose Dashboard Management under Management.
Choose Index patterns in the navigation pane.
Choose Create index pattern.
For Index pattern name, enter a name (for example, log-aws-cloudtrail-*).
Choose Next step.
For Time field¸ choose @timestamp.
Choose Create index pattern.
Create visualizations
Now that the index pattern is created, let’s create some visualizations. For this post, we create a pie chart and an area graph.
Create a pie chart
Complete the following steps to create a pie chart:
In OpenSearch Dashboards, choose Visualize in the navigation pane.
Choose Create visualization.
Choose Pie as the visualization type.
For Source¸ choose log-aws-cloudtrail-*.
Under Buckets¸ choose Add and Split slices.
For Aggregation, choose Terms.
For Field, choose eventName.
For Size, enter 10.
Leave all other parameters as default and choose Update.
Please refer Export and import visualizations for how to import the samples.
Create a dashboard
Now we will combine the visualizations we just created into a dashboard. A dashboard serves as a customizable interface that consolidates multiple visualizations, saved searches, and various content into a comprehensive view of data. Users can combine diverse visual elements—including charts, graphs, metrics, and tables—into a single cohesive display that can be arranged and resized on a flexible grid layout. You can simultaneously apply filters and time ranges across multiple visualizations, creating a coordinated analytical experience. Complete the following steps to create a dashboard:
In OpenSearch Dashboards, choose Dashboards in the navigation pane.
Choose Create new dashboard.
Choose Add on the menu bar.
Search for and choose the visualizations you created.
You can resize panels by dragging their corners to adjust dimensions. To modify the layout arrangement, you can drag the top portion of panels, which allows you to organize them horizontally in a row formation. When working with tabular visualizations, the system provides a convenient option to export your results in CSV format for further analysis or reporting purposes.
Please refer Export and import visualizations for how to import the samples.
The following screenshot shows the CloudTrail dashboard displaying both visualizations.
Export and import visualizations
In OpenSearch, an NDJSON file is used to import and export saved objects, such as dashboards, visualizations, maps, and index template. The NDJSON file provides a streamlined approach for handling large datasets by representing each JSON object on a separate line. This format enables efficient import/export operations, simplified data migration between environments, and seamless sharing of complex dashboard configurations. Organizations can back up and restore critical visualizations, saved searches, and dashboard settings while maintaining their integrity. The format’s structure reduces memory overhead during large transfers and improves processing speed for bulk operations. NDJSON’s human-readable nature also facilitates troubleshooting and manual editing when necessary, making it an invaluable tool for maintaining OpenSearch Dashboards deployments across development, testing, and production environments.
Export a visualization
Complete the following steps to export a visualization:
In OpenSearch Dashboards, choose Saved objects in the navigation pane.
Search for and select your object (in this case, a visualization), then choose Export.
The NDJSON file is downloaded in your local host.
Import a visualization
Complete the following steps to import a visualization:
In OpenSearch Dashboards, choose Saved objects in the navigation pane.
Choose Import.
Choose the first NDJSON file to be imported from your local host.
Select Create new objects with random IDs.
Choose Import.
Choose Done.
Choose Import.
You can now open the imported object.
The following screenshot shows our updated dashboard.
Clean up
To clean up your resources, delete the OpenSearch Service domain and relevant information stored or visualizations created on the domain. You will not be able to recover the data after you delete it.
On the OpenSearch Service console, choose Domains in the navigation pane.
Select the domain you created and choose Delete.
Conclusion
OpenSearch Dashboards is a powerful tool for transforming raw log data into actionable visualizations that drive insights and decision-making. In this post, we’ve shown how to create visualizations like pie charts and area graphs, build comprehensive dashboards, and efficiently export and import your work using NDJSON files. By using the fully managed OpenSearch Service features, organizations can focus on extracting valuable insights rather than managing infrastructure, ultimately enhancing their observability posture and operational efficiency.
To further enhance your OpenSearch proficiency, consider exploring advanced visualization options such as heat maps, gauge charts, and geographic maps that can represent your data in more specialized ways. Implementing automated alerting based on predefined thresholds will help you proactively identify anomalies before they become critical issues. You can also use OpenSearch’s powerful machine learning capabilities for sophisticated anomaly detection and predictive analytics to gain deeper insights from your log data. As your implementation grows, customizing security settings with fine-grained access controls will provide appropriate data visibility across different teams in your organization.
Smita Singh is a Senior Solutions Architect at AWS. She focuses on defining technical strategic vision and works on architecture, design, and implementation of modern, scalable platforms for large-scale global enterprises and SaaS providers. She is a data, analytics, and generative AI enthusiast and is passionate about building innovative, highly scalable, resilient, fault-tolerant, self-healing, multi-tenant platform solutions and accelerators.
Dipayan Sarkar is a Specialist Solutions Architect for Analytics at AWS, where he helps customers modernize their data platform using AWS analytics services. He works with customers to design and build analytics solutions, enabling businesses to make data-driven decisions.
Healthcare systems face significant challenges managing vast amounts of data while maintaining regulatory compliance, security, and performance. This post explores strategies for implementing a multi-tenant healthcare system using Amazon OpenSearch Service.
In this context, tenants are distinct healthcare entities, sharing a common platform while maintaining isolated data environments. Hospital departments (like emergency, radiology, or patient care), clinics, insurance providers, laboratories, and research institutions are examples of these tenants.
In this post, we address common multi-tenancy challenges and provide actionable solutions for security, tenant isolation, workload management, and cost optimization across diverse healthcare tenants.
Understanding multi-tenant healthcare systems
Tenants in healthcare systems are diverse and have distinct requirements. For example, emergency departments need round-the-clock high availability with subsecond response times for patient care, along with strict access controls for sensitive trauma data. Research departments run complex, resource-intensive queries that are less time-sensitive but require robust anonymization protocols to maintain HIPAA compliance when working with patient data. Outpatient clinics operate during business hours with predictable usage patterns and moderate performance requirements. Administrative systems focus on financial data with scheduled batch processing and require access to billing information and insurance details only. Specialty departments like radiology and cardiology have unique requirements specific to the tasks they perform. For example, radiology requires high storage capacity and bandwidth for large medical imaging files, along with specialized indexing for metadata searches.
Understanding tenant requirements is essential for designing an effective multi-tenant architecture that balances resource sharing with appropriate isolation while maintaining regulatory compliance.
Isolation models
OpenSearch’s hierarchical structure consists of four main levels. At the top level is the domain, which contains one or more nodes that store and search data. Within the domain, indexes contain documents and define how they are stored and searched. Documents are individual records or data entries stored within an index, and each document consists of fields, which are individual data elements with specific data types and values.
Indexes include mappings and settings. Mappings define the schema of documents within an index, specifying field names and their data types. Settings configure various operational aspects of an index, such as the number of primary shards and replica shards.
The isolation model in a multi-tenant OpenSearch system can be at domain, index, or document level. The model you select for your multi-tenant healthcare system impacts security, performance, and cost. For healthcare organizations, as depicted in the following diagram, a hybrid approach typically works best, matching isolation levels to tenant requirements.
Multi-Tenancy Isolation Models
For emergency units, consider domain-based isolation, providing maximum separation by deploying separate OpenSearch domains for each tenant. Although it’s more expensive, it reduces resource contention and provides consistent performance for critical systems. This isolation simplifies compliance by physically separating sensitive patient data.
Similarly, for clinical research tenants, consider domain-based isolation despite its higher cost. Given the resource-intensive nature of research workloads—particularly genomics and population health analytics that process terabytes of data with complex algorithms—separate domains prevent these demanding operations from impacting other tenants.
For specialty departments like cardiology or radiology, where workload patterns are similar but data access patterns are distinct, index-based isolation is a good fit. These departments share a domain but maintain separate indexes. This approach provides strong logical separation while allowing more efficient resource utilization.
For administrative departments where data is less sensitive, a document-based isolation is sufficient, and multiple tenants can share the same indexes.
Data modeling
Effective data modeling is crucial for maintaining performance and manageability in a multi-tenant healthcare system. Implement a consistent index naming convention that incorporates tenant identifiers, data categories, and time periods like {tenant-id}-{data-type}-{time-period}. Tenant-id identifies the entity, for example, cardiology. Examples of the indexes are cardiology-ecg-202505 or radiology-mri-202505. This structured approach simplifies data management, access control, and lifecycle policies.
Consider data access patterns when designing your index strategy. For example, for time-series data like vital signs or telemetry readings, time-based indexes with appropriate rotation policies will improve performance and simplify data lifecycle management.
For shared indexes using document-based isolation, make sure tenant identifiers are consistently applied and indexed for efficient tenant-based filtering.
Tenant management
Effective tenant management prevents resource contention and provides consistent performance across your healthcare system. Implement a hybrid isolation model using a tenant tiering framework based on criticality. The following table outlines the tiering framework.
Tier
Tenant Type
SLA
Resources
Operational Limits
Behavior
Tier-1 Critical
Emergency departments
ICU/Critical care
Operating rooms
24/7 SLA 99.99%
Sub-second response
RPO: Near zero
RTO: Less than 15 minutes
Guaranteed 50% CPU, 50% memory
Dedicated hot nodes
2 replicas minimum
100 concurrent requests
20 MB request size
30-second timeout
No throttling
Priority query routing
Preemptive scaling
Automatic failover
Tier-2 Urgent
Inpatient units
Specialty departments
Radiology/imaging
24/7 SLA with 99.9% availability
Less than 2-second response time
RPO: Less than 15 minutes
RTO: Less than 1 hour
Guaranteed 30% CPU, 30% memory
Shared hot nodes
1–2 replicas
50 concurrent requests
15 MB request size 60-second timeout
Limited throttling during peak
High-priority query routing
Automatic scaling
Automated recovery
Tier-3 Standard
Outpatient clinics
Primary care
Pharmacy
Laboratory
Business hours SLA (8 AM – 8 PM)
99.5% availability Less than 5-second response time
RPO: Less than 1 hour
RTO: Less than 4 hours
Guaranteed 15% CPU, 15% memory
Shared nodes
1 replica
25 concurrent requests
10 MB request size
120-second timeout
Moderate throttling
Standard query routing
Fair thread allocation
Manual scaling
Business hours optimization
Tier-4 Research
Clinical research
Genomics
Population health
Best-effort
SLA, up to 99% availability
Less than 30-second response time
RPO: Less than 24 hours
RTO: Less than 24 hours
Guaranteed 5% CPU, 10% memory
Burst capacity during off-hours
0–1 replicas
10 concurrent requests
50 MB request size
300-second timeout
Aggressive throttling during pea
Compute optimized instances
Large heap size
Research-specific plugins
Tier-5 Admin
Billing/finance
HR systems
Inventory management
Business hours SLA (9 AM – 5 PM) 99% availability Less than 10-second response time
Healthcare data requires protection due to its sensitive nature and regulatory requirements. The OpenSearch Service security framework is specifically adaptable to healthcare’s strict security requirements. This framework combines multiple layers of access control, captured in the following diagram.
Multi-tenancy fine-grained access control in Amazon OpenSearch Service
An important step in this framework is role mapping, where AWS Identity and Access Management (IAM) roles are mapped to OpenSearch roles for role-based access control (RBAC). For example, emergency departments can implement the ED-Physician role with access to patient history across departments, and the ED-Staff role with access to vital sign and medication data. You can map emergency department roles to OpenSearch roles.
With document-level security (DLS), you can limit emergency department staff to active emergency patients only while restricting access to discharged patient data only to the providers who treat them. With field-level security (FLS), you can allow access to medical fields while masking billing and insurance data. You can also provide attribute-based access control (ABAC) policies to allow access based on patient status.
For research departments, you can create Clinical-Researcher roles with read-only access to datasets. Integrate academic roles to research roles to make sure researchers only access data for studies they’re authorized to conduct. For DLS, implement filters to make sure researchers only access approved documents. Use FLS to anonymize HIPAA identifiers. For research departments, ABAC should evaluate the study phase and researcher’s location.
For outpatient care, you can define Medical-Provider roles with full access to assigned patients’ records and Medical-Assistant roles limited to documenting vitals and preliminary information. For DLS, limit access to patient’s physicians only. For FLS, restrict access to medical data only, while limiting nurses to demographic, vital signs, and medication fields. Implement time-aware ABAC policies that restrict access to patient records outside of business hours unless the provider is on-call.
For administrative departments, you can implement Financial roles with access to charge codes and insurance information but no clinical data. For DLS, make sure financial staff only access billing documents. FLS provides access to billing codes, dates of service, and insurance fields while masking clinical content.
For specialty departments, you can create technician roles like Radiologist and apply DLS filters restricting access to the data to these roles and referring physician. FLS allows technicians to see clinical history and previous findings specific to their specialty.
Enable comprehensive audit logging to track access to protected health information. Configure these logs to capture user identity, accessed data, timestamp, and access context. These audit trails are essential for regulatory compliance and security investigations.
Managing data lifecycle for compliance
Index State Management (ISM) capabilities combined with OpenSearch Service storage tiering enable an elaborate approach to data lifecycle management that can be tailored to diverse tenant needs. ISM provides a robust way to automate the lifecycle of indexes by defining policies that dictate transitions between Hot, UltraWarm, and Cold storage tiers based on criteria like index age or size. This automation can extend to the archive tier by creating snapshots, which are stored in Amazon Simple Storage Service (Amazon S3) and can be further transitioned to Amazon S3 Glacier or Glacier Deep Archive for long-term, cost-effective archiving of data that is rarely accessed.
Frame your ISM policy along the following guidelines:
Keep critical patient data in hot storage for 180 days to support immediate access. Transition to warm storage for the next 12 months, then move to cold storage for years 2–7. After 7 years, archive records.
For research data benefits, use project-based lifecycle policies rather than strictly time-based transitions. Maintain research datasets in hot storage during active project phases, regardless of data age. When projects conclude, transition data to warm storage for 12 months. Move to cold storage for the following 5–10 years based on research significance. Afterward, archive records.
For outpatient clinic data, keep recent patient records in hot storage for 90 days, aligning index rollover with typical follow-up windows. Transition to warm storage for months 4–18, coinciding with common annual visit patterns. Move to cold storage for years 2–7. Archive after 7 years.
For administrative data, maintain current fiscal year data in hot storage with automated transitions at year-end boundaries. Move previous fiscal year data to warm storage for 18 months to support auditing and reporting. Transition to cold storage for years 3–7. Archive financial records after 7 years.
For the specialty department data, keep recent metadata in hot storage for 90 days while moving large files, like images, to warm storage after 30 days. Transition complete records to cold storage after 18 months. Archive after 7 years.
Cost management and optimization
Healthcare organizations must balance performance requirements with budget constraints. Effective cost management strategies are essential for sustainable operations.
Implement comprehensive tagging strategies that mirror your index naming conventions to create a unified approach to resource management and cost tracking. Like the index naming convention, design your tags to identify the tenant, application, and data type (for example, “tenant=cardiology” or “application=ecg“). These tags, combined with AWS Cost Explorer, provide visibility into expenses across organizational boundaries.
Develop cost allocation mechanisms that fairly distribute expenses across different tenants. Consider implementing tiered pricing structures based on data volume, query complexity, and service-level guarantees. This approach aligns costs with value and encourages efficient resource utilization.
Optimize your infrastructure based on tenant-specific metrics and usage patterns. Monitor document counts, indexing rates, and query patterns to right-size your clusters and node types. Use different instance types for different workloads—for example, use compute-optimized instances for query-intensive applications.
Use OpenSearch Service storage tiering to optimize costs. UltraWarm provides significant cost savings for infrequently accessed data while maintaining reasonable query performance. Cold storage offers even greater savings for data that’s rarely accessed but must be retained for compliance purposes.
Conclusion
Building a multi-tenant healthcare system on OpenSearch Service requires careful planning and implementation. By addressing tenant isolation, security, data lifecycle management, workload control, and cost optimization, you can create a platform that delivers improved operational efficiency while maintaining strict compliance with healthcare regulations.
About the Authors
Ezat Karimi is a Senior Solutions Architect at AWS, based in Austin, TX. Ezat specializes in designing and delivering modernization solutions and strategies for database applications. Working closely with multiple AWS teams, Ezat helps customers migrate their database workloads to the AWS Cloud.
Jon Handler is a Senior Principal Solutions Architect at Amazon Web Services based in Palo Alto, CA. Jon works closely with OpenSearch and Amazon OpenSearch Service, providing help and guidance to a broad range of customers who have vector, search, and log analytics workloads that they want to move to the AWS Cloud. Prior to joining AWS, Jon’s career as a software developer included 4 years of coding a large-scale, ecommerce search engine. Jon holds a Bachelor’s of the Arts from the University of Pennsylvania, and a Master’s of Science and a PhD in Computer Science and Artificial Intelligence from Northwestern University.
Our customers tell us that scientists are increasingly spending more time managing data-related challenges than focusing on science. The primary reason for this challenge is that scientific data comes in many types and is siloed across systems, groups, and stages, and scientists struggle to efficiently discover, access, share, and analyze datasets across silos. This fragmentation creates lengthy cycles full of manual interventions, leading to inefficiencies. Mapping data sources and negotiating access across silos can take 4–6 weeks, integrating datasets can extend to months, and fully connecting data from source to tooling can take years, if ever achieved. These data challenges reduce lab productivity and slow down scientific innovation, which decrease drug and product pipeline throughput, and ultimately delay time-to-market. The solution lies in breaking down data silos by creating digital environments that help scientists efficiently connect disparate datasets and analytical tools, so they can conduct iterative hypothesis and product testing without technology friction.
Part 1 of this series shows an example project in drug target identification where two groups of scientists need to collaborate as they integrate no-code knowledge searching, scientific data management, and sophisticated analytics. In this example, a computational biology team begins by mining the scientific literature on a knowledge search GUI. Next, they navigate to a data catalog to find and access relevant datasets, which they share with the data scientist team to run analytics with sophisticated tools (see the following figure). Although the end-to-end journey illustrates the benefits to a target identification example, the underlying data challenges and technology solution apply to any life sciences use case requiring the integration of data management and analytics. Details of the implementation and technical solution will be discussed in Part 2 of the series.
Example use case
A computational biologist has been tasked with identifying a target for Non-Alcoholic Fatty Liver Disease (NAFLD). A typical question from the biologist might be “Can I find genes associated with NAFLD and do we have a patient cohort with variants in those genes?” The solution we designed for this use case involves three simple steps:
Search the scientific literature through a no-code interface to identify genomic variants associated with NAFLD.
Search an internal data catalog with natural language:
Find datasets of interest, such as multi-omics and clinical data for patients associated with NAFLD.
Request access to the relevant datasets.
Share relevant datasets with a data scientist collaborator for deeper analysis.
In designing this solution, we focused on the following features:
Providing no-code scientists with point-and-click and natural-language interfaces
Reducing silos with data findability, governance automation, and seamless collaboration
Providing technical personas with the sophisticated tools and environments they prefer
SageMaker Catalog supports certain system asset types, such as tables from Amazon Redshift, tables from AWS Glue, and object collections from Amazon Simple Storage Service (Amazon S3). It also offers the ability to support custom asset types, which gives users flexibility to catalog data that can’t be categorized as a system asset type. For asset type S3ObjectCollectionType, see Implement a custom subscription workflow for unmanaged Amazon S3 assets published with Amazon DataZone. SageMaker Catalog also offers the ability to support custom asset types, which gives users flexibility to catalog data that can’t be categorized as a system asset type. For this example use case, we used AWS HealthOmics variant stores to store and allow querying of genomic variant data. This example lists HealthOmics variant stores as a custom asset type within the catalog. Details of the implementation and technical solution for access management will be discussed in Part 2 of the series.
In the example use case, a computational biologist, in order to identify a target for NAFLD, relies heavily on diverse datasets from multiple sources (genomic sequences, gene expression data, clinical records, and more). This data comes from both internal sources (first-party) and external partners or public databases (third-party). Multiple teams are responsible for collecting and processing this data before making it available to computational biologists, researchers, data scientists, and bioinformaticians within the organization.
In this solution, users (data engineers, data scientists, bioinformaticians, computational biologists) log in to a project-based environment from SageMaker Unified Studio with a preconfigured authentication method. A typical workflow involves the following steps:
Data stewards as authorized members of projects publish data assets into the SageMaker catalog.
Data consumers as authorized members of projects seeking to analyze data for their scientific needs find and discover available data assets of interest from the SageMaker catalog.
Data consumers request to subscribe to the relevant discovered data assets.
Data producers review and decide to approve or reject the subscription request.
Data consumers access and analyze the data using preconfigured tools from SageMaker Unified Studio.
The following diagram illustrates the solution architecture and workflow.
In the following sections, we explore each step of the workflow in more detail.
Step 1: Data producers publish data assets
As shown in the preceding workflow diagram, data producers can use SageMaker Catalog to publish their datasets as data assets or data products with appropriate business (such as source, license, vendor, study identifier), scientific (such as disease name, cohort information, data modality, assay type), or technical (file types, data formats, file sizes) metadata. In our example use case, the data producers publish clinical data as AWS Glue tables and genomic variant data as a table within the HealthOmics variant store. Additionally, data producers can use AI-based recommendations to automatically populate descriptors, making it straightforward for consumers to find and understand its use.
Step 2: Data consumers find relevant datasets
Data consumers, such as data scientists and bioinformaticians, can log in to SageMaker Unified Studio and navigate to SageMaker Catalog to search for the appropriate data assets and products, such as “NAFLD Variants” or “NAFLD Clinical.” They can also find data assets or products using metadata filters such as study identifiers or disease names to discover the possible datasets associated with a study or disease.
Step 3: Data consumers subscribe to required data assets or products
After the data consumers see a data asset or data product of interest (for example, the clinical and genomics data for NAFLD), they can subscribe to them. Data consumers can also optionally include a comment in the subscription request to add more context to the request. This initiates the subscription workflow based on the asset type.
Step 4: Data producers review and approve the subscription request
Data producers get notified of subscription requests and review if access should be granted and approve accordingly. The response can optionally include a comment for reasoning and traceability. In addition, data producers can limit access to certain rows and columns to protect controlled data.
Step 5: Data consumers access the subscribed data assets or products
Upon approval from the data producer, the data consumer gets access to those data assets and can use them in the appropriate environments configured within their project. For example, data scientists can open a workspace with a JupyterLab notebook already available within SageMaker Unified Studio. Subsequently, the data scientist can start analyzing the tabular clinical and variant data that was just approved for access.
Conclusion
The next generation of SageMaker transforms how scientists work with data by creating an integrated data and analytics environment. In this unified environment, data producers are empowered to publish datasets with rich metadata. Data consumers are able to use the catalog within SageMaker Unified Studio to search for their required datasets, either using free text or using metadata and business glossary filters. Data consumers can subscribe to data securely, tap into powerful search capabilities using free text or metadata filters, and access essential analysis tools (Amazon Athena, JupyterLab IDE, Amazon EMR) directly. The result is a unified digital workspace that reduces communication bottlenecks, speeds up scientific cycles, and removes technical barriers. Scientists can now focus on what matters most—testing hypotheses and products, and scaling scientific innovation to production—within a unified, powerful platform. This streamlined approach accelerates data-driven science, enabling research institutions, pharmaceutical companies, and clinical laboratories to innovate more efficiently. For example, data scientists can launch a space with a JupyterLab notebook preinstalled.
Consider using the next generation of SageMaker to increase productivity within your organization. Contact your account representatives or an AWS Representative to learn how we can help accelerate your projects and your business.
About the authors
Nadeem Bulsara is a Principal Solutions Architect at AWS specializing in Genomics and Life Sciences. He brings his 13+ years of Bioinformatics, Software Engineering, and Cloud Development skills as well as experience in research and clinical genomics and multi-omics to help Healthcare and Life Sciences organizations globally. He is motivated by the industry’s mission to enable people to have a long and healthy life.
Chaitanya Vejendla is a Senior Solutions Architect specialized in DataLake & Analytics primarily working for Healthcare and Life Sciences industry division at AWS. Chaitanya is responsible for helping life sciences organizations and healthcare companies in developing modern data strategies, deploy data governance and analytical applications, electronic medical records, devices, and AI/ML-based applications, while educating customers about how to build secure, scalable, and cost-effective AWS solutions. His expertise spans across data analytics, data governance, AI, ML, big data, and healthcare-related technologies.
Dr. Mileidy Giraldo has over 20 years of experience bridging bioinformatics, research, and industry technology strategy. She specializes in making technology accessible for organizations in the life sciences sector. In her current role as WW Lead for Life Sciences Strategy and Lab of the Future at AWS, she helps biotechs, biopharma, and diagnostics organizations design Data & AI-driven initiatives that modernize labs and help scientists unlock the full value of their data.
Chris Clark is a Senior Solutions Architect focused on helping Life Science customers leverage AWS technology to advance their operational capabilities. With 20+ years of hands-on experience in life sciences manufacturing and supply chain, he combines deep industry knowledge with his AWS expertise to guide his customers. When he’s not working to solve customer challenges, he enjoys cycling and building and repairing things in his workshop.
Nick Furr is a Specialist Solutions Architect at AWS, supporting Data & Analytics for Healthcare and Life Sciences. He helps providers, payers, and life sciences organizations build secure, scalable data platforms to drive innovation and improve outcomes. His work focuses on modernizing data strategies through cloud analytics, governed data processing, and machine learning for use cases like clinical research and population health.
Subrat Das is a Principal Solutions Architect for Global Healthcare and Life Sciences accounts at AWS. He is passionate about modernizing and architecting complex customers workloads. When he’s not working on technology solutions, he enjoys long hikes and traveling around the world.
The use of huge pages can significantly increase the performance of many
workloads by reducing both memory-management overhead in the kernel and
pressure on the system’s translation lookaside buffer (TLB). The addition
of transparent huge pages (THP) for the 2.6.38 kernel release in 2011
caused the kernel to allocate huge pages automatically to make their
benefits available to all workloads without any effort needed on the
user-space side. But it turns out that use of huge pages can make some
workloads slower as the result of internal memory fragmentation, so the THP
feature is often disabled. Two patch sets aimed at better targeting the
use of transparent huge pages are currently working their way through the
review process.
The call for topics for
the 2025 Maintainers Summit has been posted. The Summit, to be held in
Tokyo on December 10, will involve around 30 developers gathered to
discuss development-process issues for the kernel. Anybody who is
interested in attending is encouraged to post a nomination along with the
topic they would like to discuss. Nominations and topics are best sent
before September 10.
The call for topics for the Kernel Summit, which runs as a Linux Plumbers Conference track, is also
out.
Antonio Cuni, who
is a longtime Python performance engineer and PyPy developer, gave a presentation at EuroPython
2025 about “Myths and fairy tales around Python performance” on
the first day of the conference in Prague. As might be guessed from the
title, he thinks that much of the conventional wisdom about Python
performance is misleading at best. With lots of examples, he showed where
the real problems that he sees lie. He has come to the conclusion that memory
management will ultimately limit what can be done about Python performance,
but he has an
early-stage project called SPy that
might be a way toward a super-fast Python.
One of Cloudflare’s big focus areas is making the Internet faster for end users. Part of the way we do that is by looking at the “big rocks” or bottlenecks that might be slowing things down — particularly processes on the critical path. When we recently turned our attention to our privacy proxy product, we found a big opportunity for improvement.
What is our privacy proxy product? These proxies let users browse the web without exposing their personal information to the websites they’re visiting. Cloudflare runs infrastructure for privacy proxies like Apple’s Private Relay and Microsoft’s Edge Secure Network.
Like any secure infrastructure, we make sure that users authenticate to these privacy proxies before we open up a connection to the website they’re visiting. In order to do this in a privacy-preserving way (so that Cloudflare collects the least possible information about end-users) we use an open Internet standard – Privacy Pass – to issue tokens that authenticate to our proxy service.
Every time a user visits a website via our Privacy Proxy, we check the validity of the Privacy Pass token which is included in the Proxy-Authorization header in their request. Before we cryptographically validate a user’s token, we check if this token has already been spent. If the token is unspent, we let the user request through. Otherwise, it’s a “double-spend”. From an access control perspective, double-spends are indicative of a problem. From a privacy perspective, double-spends can reduce the anonymity set and privacy characteristics. From a performance perspective, our privacy proxies see millions of requests per second – and any time spent authenticating delays people from accessing sites – so the check needs to be fast. Let’s see how we reduced the latency of these double-spend checks from ~40 ms to <1 ms.
How did we discover the issue?
We use a tracing platform, Jaeger. It lets us see which paths our code took and how long functions took to run. When we looked into these traces, we saw latencies of ~ 40 ms. It was a good lead, but it alone was not enough to conclude it was an issue. The reason was we only sample a small percentage of our traces, so what we saw was not the whole picture. We needed to look at more data. We could’ve increased how many traces we sampled, but traces are large and heavy for our systems to process. Metrics are a lighter weight solution. We added metrics to get data on all double-spend checks.
The lines in this graph are median latencies we saw for the slowest privacy proxies around the world. The metrics data gave us confidence that it was a problem affecting a large portion of requests… assuming that ~ 45 ms was longer than expected. But, was it expected? What numbers did we expect?
The expected latency
To understand what times are reasonable to expect, let’s go into detail on what makes up a “double-spend check”. When we do a double-spend check, we ask a backing data store if a Privacy Pass token exists. The data store we use is memcached. We have many memcached instances running on servers around the world, so which server do we ask? For this, we use mcrouter. Instead of figuring out which memcached server to ask, we give our request to mcrouter, and it will handle choosing a good memcached server to use. We looked at the median time it took for mcrouter to process our request. This graph shows the average latencies per server over time. There are spikes, but most of the time the latency is < 1 ms.
By this point, we were confident that double-spend check latencies were longer than expected everywhere, and we started looking for the root cause.
How did we investigate the issue?
We took inspiration from the scientific method. We analyzed our code, created theories for why sections of code caused latency, and used data to reject those theories. For any remaining theories, we implemented fixes and tested if they worked.
Let’s look at the code. At a high level, the double-spend checking logic is:
Get a connection, which can be broken down into:
Send a memcached version command. This serves as a health check for whether the connection is still good to send data on.
If the connection is still good, acquire it. Otherwise, establish a new connection.
Send a memcached get command on the connection.
Let’s go through the theories we had for each step listed above.
Theory 1: health check takes long
We measured the health check primarily as a sanity check. The version command is simple and fast to process, so it should not take long. And we remained sane. The median latency was < 1 ms.
Theory 2: waiting to get a connection
To understand why we may need to wait to get a connection, let’s go into more detail on how we get a connection. In our code, we use a connection pool. The pool is a set of ready-to-go connections to mcrouter. The benefit of having a pool is that we do not have to pay the overhead of establishing a connection every time we want to make a request. Pools have a size limit, though. Our limit was 20 per server, and this is where a potential problem lies. Imagine we have a server that processes 5,000 requests every second, and requests stay for 45 ms. We can use something called Little’s Law to estimate the average number of requests in our system: 5000 x 0.045 = 225. Due to our pool size limits, we can only have 20 connections at a time, so we can only process 20 requests at any point in time. That means 205 requests are just waiting! When we do a double-spend check, maybe we’re waiting ~ 40 ms to get a connection?
We looked at the metrics of many different servers. No matter what the requests per second was, the latency was consistently ~ 40 ms, disproving the theory. For example, this graph shows data from a server that saw a maximum of 20 requests per second. It shows a histogram over time, and the large majority of requests fall in the 40 – 50 ms bucket.
Theory 3: delays in Nagle’s algorithm and delayed acks
We decided to chat with Gemini, giving it the observations we had so far. It suggested many things, but the most interesting was to check if TCP_NODELAY was set. If we had set this option in our code, it would’ve disabled something called Nagle’s algorithm. Nagle’s algorithm itself was not a problem, but when enabled alongside another feature, delayed ACKs, latencies could creep in. To explain why, let’s go through an analogy.
Suppose we run a group chat app. Normally, people type a full thought and send it in one message. But, we have a friend who sends one word at a time: “Hi”. Send. “how”. Send. “are”. Send. “you”. Send. That’s a lot of notifications. Nagle’s algorithm aims to prevent this. Nagle says that if the friend wants to send one short message, that’s fine, but it only lets them do it once per turn. When they try to send more single words right after, Nagle will save the words in a draft message. Once the draft message hits a certain length, Nagle sends. But what if the draft message never hits that length? To manage this, delayed ACKs initiates a 40 ms timer whenever the friend sends a message. If the app gets no further input before the timer ends, the message is sent to the group.
I took a closer look at the code, both Cloudflare authored code and code from dependencies we rely on. We depended on the memcache-async crate for implementing the code that lets us send memcache commands. Here is the code for sending a memcached version command:
Nothing out of the ordinary. Then, we looked inside the get function.
let writer = self.io.get_mut();
writer.write_all(b"get ").await?;
writer.write_all(key.as_ref()).await?;
writer.write_all(b"\r\n").await?;
writer.flush().await?;
In our code, we set io as a TcpStream, meaning that each write_all call resulted in sending a message. With Nagle’s algorithm enabled, the data flow looked like this:
Oof. We tried to send all three small messages, but after we sent the “get “, the kernel put the token and \r\n in a buffer and started waiting. When mcrouter got the “get “, it could not do anything because it did not have the full command. So, it waited 40 ms. Then, it sent an ACK in response. We got the ACK, and sent the rest of the command in the buffer. mcrouter got the rest of the command, processed it, and returned a response telling us if the token exists. What would the data flow look like with Nagle’s algorithm disabled?
We would send all three small messages. mcrouter would have the full command, and return a response immediately. No waiting, whatsoever.
Why 40 ms?
Our Linux servers have minimum bounds for the delay. Here is a snippet of Linux source code that defines those bounds.
#if HZ >= 100
#define TCP_DELACK_MIN ((unsigned)(HZ/25)) /* minimal time to delay before sending an ACK */
#define TCP_ATO_MIN ((unsigned)(HZ/25))
#else
#define TCP_DELACK_MIN 4U
#define TCP_ATO_MIN 4U
#endif
The comment tells us that TCP_DELACK_MIN is the minimum time delayed ACKs will wait before sending an ACK. We spent some time digging through Cloudflare’s custom kernel settings and found this:
CONFIG_HZ=1000
CONFIG_HZ eventually propagates to HZ and results in a 40 ms delay. That’s where the number comes from!
The fix
We were sending three separate messages for a single command when we only needed to send one. We captured what a get command looked like in Wireshark to verify we were sending three separate messages. (We captured this locally on MacOS. Interestingly, we got an ACK for every message.)
The fix was to use BufWriter<TcpStream> so that write_all would buffer the small messages in a user-space memory buffer, and flush would send the entire memcached command in one message. The Wireshark capture looked much cleaner.
Conclusion
After deploying the fix to production, we saw the median double-spend check latency drop to expected values everywhere.
Our investigation followed a systematic, data-driven approach. We began by using observability tools to confirm the problem’s scale. From there, we formed testable hypotheses and used data to systematically disprove them. This process ultimately led us to a subtle interaction between Nagle’s algorithm and delayed ACKs, caused by how we made use of a third-party dependency.
Ultimately, our mission is to help build a better Internet. Every millisecond saved contributes to a faster and more seamless, private browsing experience for end users. We’re excited to have this rolled out and excited to continue to chase further performance improvements!
With hundreds of thousands of hard drives spinning 24/7, our data centers are less like peaceful white-noise oases and more like a a series of obstacle courses—if said obstacle courses were about managing over four exabytes of customer data from archival backups to streaming media to AI training datasets. Sure, they’re obstacle courses we all (and I’m including you, users of the internet) collectively create, but it’s no less of a balancing act to find the contestants (erm, hard drives) that can go the distance.
And we, dear readers, get to watch it all. Welcome to Drive Stats: where failure is inevitable, survival is fascinating, and every quarter brings a new leaderboard.
As of June 30, 2025, we had 321,201 drives under management. Of that total, there were 3,971 boot drives and 317,230 data drives. Stay tuned as we take our standard peek into quarterly and lifetime failure rates, and do a deep dive into the 20TB+ club.
Ready to dive deeper into the data? Tune in today at 10:00 a.m. PT, to query the Drive Stats team, Stephanie Doyle and Pat Patterson. We’ll see you there!
Drive Stats by the numbers: The digest version
Q2 2025 hard drive failure rates
For those that are new to the Drive Stats report, it’s worth mentioning that we have certain criteria that we use to select drives for consideration each quarter. We’ll discuss those in the next section, but for now, let’s talk about the data. The table below shows the failure rates for Q2 2025.
Backblaze Hard Drive Failure Rates for Q2 2025
Reporting period April 1, 2025–June 30, 2025 inclusive Drive models with drive count > 100 as of June 30, 2025 and drive days > 10,000 in Q2 2025
Notes and observations
The annual failure rate is lower this quarter. We had some major fluctuations last quarter. Quoting ourselves from May 2025:
The quarterly failure rate is slightly higher. The quarterly failure rate went up from 1.35% to 1.42%. As with the zero-failure club, our higher-end outlier AFRs show some of the usual suspects:
We’re now back down to 1.36%. What’s changed?
Big swings in our higher-end failure rates: Well, some of the drives with higher failure rates have come down quite a bit. Notably, that includes the 12TB Seagate model ST12000NM0007, which was at a whopping 9.47% failure rate last quarter—down this quarter to only 3.58%. With its drive count holding more or less steady (1,038 in Q1 and 1,014 in Q2), that means a real change in failure rates. Note that this drive was at 8.72% in Q4 2024, so it’s worth keeping an eye on whether this is a fluke or a new pattern. Other significant drops include the 12TB HGST model HUH721212ALN604 (Q1: 4.97%; Q2: 3.39%) and the 14TB Seagate model ST14000NM0138 (Q1: 6.82%, Q2: 4.37%).
One new drive model on the way in: Welcome to the party, Toshiba MG09ACA16TE (16TB).
Zero failures for the quarter: Rising to the top, we’ve got only two this time around:
Seagate ST8000NM000A (8TB)
Seagate ST16000NM002J (16TB)
That 8TB Seagate is really shining, given this is its third quarter running with zero failures.
Bonus: One failure drives: Since we only have two 0 failures (and that just seems a little lackluster, doesn’t it?), it’s also worth mentioning the drives with only one failure this quarter:
HGST HMS5C4040BLE640 (4TB)
Seagate ST12000NM000J (12TB)
Seagate ST14000NM000J (14TB)
Toshiba MG09ACA16TE (16TB)
Drive model criteria
We noted earlier we removed 495 drives from consideration when we produced the table above covering Q2 2025. There are two primary reasons we did not consider these drive models.
Testing. These are drives of a given model that we monitor and collect Drive Stats data on, but are not considered production drives at this time. For example, drives undergoing certification testing to determine if they are performant enough for our environment are not included in our Drive Stats calculations.
Insufficient data points. When we calculate the annualized failure rate for a drive model for a given period of time (quarterly, annual, or lifetime), we want to ensure we have enough data to reliably do so. Therefore we have defined criteria for a drive model to be included in the tables and charts for the specified period of time. Models that do not meet these criteria are not included in the tables and charts for the period in question.
Regardless of whether or not a given drive model is included in the charts and tables, all of the data for all of the drives we use is included in our Drive Stats dataset which you can download by visiting our Drive Stats page.
As with the Q2 quarterly results, we will apply these criteria to the lifetime charts that follow in this report.
Lifetime hard drive failure rates
To be considered for the lifetime review, a drive model was required to have 500 or more drives as of the end of Q2 2025 and have over 100,000 accumulated drive days during their lifetime. When we removed those drive models which did not meet the lifetime criteria, we had 393,907 drives grouped into 27 models remaining for analysis as shown in the table below.
Backblaze Hard Drive Failure Rates for Q2 2025
Reporting period ending June 30, 2025 Drive models > 500 drives and > 100,000 lifetime drive days
Notes and observations
Again, the lifetime AFR holds steady, dropping from Q1 2025’s 1.31% to 1.30%.
Now you see me: This quarter’s table also gives us an interesting snapshot that has to do with our drive exclusions as the 4TB HGST model HMS5C4040ALE640 is on the way out. It meets our lifetime drive criteria, so it is included in this second table, but it didn’t make the cut for the quarterly table because it had too few drives running by the end of the quarter. Usually you see the opposite, where drive models show up in the quarterly requirements but not the lifetime. This quarter, four models meet that standard (Seagate model numbers ST8000NM000A, ST14000NM000J, ST16000NM002J, and Toshiba MG09ACA16TE).
Smaller drives getting older: Perhaps an unsurprising trend—Backblaze’s smaller capacity drives are getting older. We have a total of 13 drive models with 12TB or less, with a collective 1.54% failure rate. See the table below:
Backblaze drives with ≤12TB capacity
Of those models, eight are five years old or older (shown in purple). An additional two models are four years or older (that’s your orange). Taking just these 10 models—drives reaching their supposed golden years—we have a collective AFR of 1.42%.
Notably, that AFR is due to some well-performing low-failure outliers, including both of the 4TB Seagate models (0.57% and 0.40%), the 12TB HGST model HUH721212ALE600 (0.56%), and the 12TB Seagate model ST12000NM001G (0.99%).
That said, it’s also perhaps more impressive that when we say “eight are five years and older,” of those eight drive models, five are six or more years old. Their collective AFR is 1.33%.
This begs the age-old question: Is age just a number? Or, are we just seeing several exceptional drive models? In any event—an interesting drive population to keep an eye on, as it represents 156,724 of our 393,907 (~40%) of the lifetime drive pool.
Zoom in: The 20TB+ club
We’ve been taking quick peeks at the 20TB+ drives in the last few reports, but it’s high time we dig in a bit deeper. Right now, our cohort of 20TB+ drives that meet the lifetime criteria consists of three drives, the 20TB Toshiba model MG10ACA20TE, 22TB WDC model WUH722222ALE6L4, and 24TB Seagate model ST24000NM002H. Quite neatly, that also gives us one per manufacturer, lending itself to something of a head-to-head comparison—though, of course, with the variability we see on a per-drive basis within the same manufacturer, we won’t over-index on lending it too much significance.
Let’s take a look at each.
20TB Toshiba MG10ACA20TE
The Toshiba has actually been in our drive pool for 22 months, but until just under a year ago, there were only two drives. For the purposes of significance, then, we’ll exclude significantly low numbers of drives—thankfully, each model has something of a natural fall-off point where they go from single-digit drive numbers to hundreds.
For the Toshiba, that gives us the following data:
Converted to a graph, we end up with the following:
On this graph, the blue line represents the AFR and the red line represents the drive count. Drive count can be a bit tricky since our x-axis is age, and we start with age=0, which means that the drive count (from our perspective) goes from larger to smaller. That is, as drives get older, there are fewer of them by count—you have your initial purchase cohort, then you add drives over time. You can read this as the first data point representing drives that are between 0–1 month old, the next data point as 1–2 months old, etc.
We set it up this way because we wanted to be able to directly compare the failure rates of the drives based on their ages. Those familiar with our bathtub curve analysis may recognize our methodology here—we’re just zooming in on specific drives and drive capacities.
22TB WDC WUH722222ALE6L4
Now let’s take a look at the WDC model. We have usable data for about 21 months of its drive life:
Which gives us the following visualization:
Interestingly, we see a lot less variability in the span of time where we have a direct comparison. That said, the WDC model also had a minimum of double the drive count if we’re looking at a similar time period—so, at their youngest (0 months old) the Toshiba had 14,407 drives vs. WDC’s 37,363; and, at 11 months Toshiba had 1,034 drives vs. WDC’s 13,965.
While AFRs do get us mostly on an even playing field as far as being able to make a 1:1 comparison, it’s important to remember that in smaller drive pools, a single failure can be amplified by quite a bit.
24TB Seagate ST24000NM002H
Our youngest drive model, the 24TB Seagate ST24000NM002H, has just half a year of data.
That gives us the following visualization:
Compared with our other two drive models, the 24TB Seagate definitely has the highest failure rates. This could be explained, in part, by it being a young drive—is it in the leading edge of a traditional bathtub curve? So, certainly something to track over time to see if it will settle out as it gets older.
All together now: Comparing each 20TB+ drives
We designed this view to be directly comparable at points in time, so, here’s your graph that puts each drive on the same time scale:
What’s our takeaway here? Well, in both drive count and length of time in the pool, it’s a little early to create definitive trends for the Seagate and the Toshiba. Certainly we can see that the Seagate is, early on, showing higher failure rates. Meanwhile, the 20TB Toshiba has had a bit of a variable year one. But again, with significantly variable drive pools between all models, we’re not quite comparing apples to apples. (We chose not to plot drive count on this chart—it gets messy quickly.) Add to that: the Seagate in particular is potentially at the beginning of the “bathtub” curve, we may see it change over time.
On the other hand, the 22TB WDC model has shown up quite a bit below our current average AFR for the drive pool of all drive sizes and ages, and it’s the model with the most data. But, how does that compare to other models as they come online?
Comparison: 20TB+ as a pool vs. 14-16TB as a pool
When we were considering whether this information would be a useful slice of the data, our biggest question was how to contextualize it versus drives. It’s perhaps a tad imperfect, but we landed on combining the 14–16TB drives as a pool, largely because they have a significant amount of data points and were our last set of drives onboarded, which means that they’re more or less the last generation of hardware.
The other thing to note is that once we combined the 20TB drives into a pool, some of the data we filtered out on a per-drive basis got added back in. So, at the 21 month mark, where the Toshiba model only had one drive, we added that single drive to the 399 that our WDC model brought to the table and calculated AFR across the pool (giving us 400 drives to work with).
Here’s the numbers for the 20TB+ drive pool:
That gives us a pretty neat graph, actually:
Now, let’s compare to the 14–16TB drives of the same age. We have significant data for this population for nearly seven years, but in the interest of saving you three pages of scrolling, I’ll give you the table for the data that directly correlates to the 21 months of data we have for the 20TB+ drives.
This is the line chart for that range of time:
Comparing age of drive to age of drive, it would seem that our 20TB are right on target, and perhaps doing a bit better than expected. But, that definitely isn’t a perfect comparison given that the 14–16TB drives have a steadier and larger drive count. So, let’s look at the chart with the full, nearly seven year time period:
This view starts to show us some spiky patterns as the 14–16TB drives get older, of course exacerbated by drive counts reducing over time.
So what’s it all mean?
It’s clear from the data that we need to give the 20TB+ drives time to mature, and that as we (depending on our buying behavior, of course) add more drives, we might see some interesting changes in the data.
As for the 14–16TB pool, they’re following relatively expected patterns of wearing out in the five-plus year range—but what does that mean when you compare to what we observed in our current lifetime stats, where we see our 12TB and smaller drive pool performing so well?
Without taking a closer look at the 14–16TB drives, it’s hard to say that they don’t have similar outlier tendencies to what the 12TB and smaller pool does, just pulling the failure rates upward. Even a casual glance at our current lifetime table’s 14–16TB drives bears that out (four years and older highlighted in orange, as our corollary above):
That data isn’t inclusive of all of the 14–16TB drives we’ve ever had, though—just those currently in operation. So, as always, there’s more investigation to be done.
The Hard Drive Stats data
The complete dataset used to create the tables and charts in this report is available on our Hard Drive Test Data page. You can download and use this data for free for your own purpose. All we ask are three things: 1) you cite Backblaze as the source if you use the data, 2) you accept that you are solely responsible for how you use the data, and 3) you do not sell this data itself to anyone; it is free.
Good luck, and let us know if you find anything interesting.
Recently, a new customer of ours at Opensource ICT Solutions asked whether we could migrate their Nagios instance to Zabbix. Because Nagios and Zabbix are very different in their storage methods, we told them that we would have to investigate and see if we could come up with a viable solution. It wasn’t long until we found a way to do it and started building some script to get it done.
Table of Contents
The customer’s wishes
No loss of any Nagios configuration data
Historic performance data migrated to Zabbix
Existing problems migrated from Nagios
Nagios XI to be disabled entirely, as the license is expiring
The customer was clear in their wishes – we needed to turn off Nagios, but without losing historic data. As such, they wanted all their old data visible in Zabbix instead of having Nagios running somewhere as a backup. This meant that a script had to be built to get that Nagios data out and into Zabbix.
The configuration data
The good part here is that it starts simple. When we dive into the Nagios configuration data, we clearly see that Nagios has hosts just like Zabbix. They just have a slightly different build than our usual Zabbix hosts. For example, we can see three different names for a host in Nagios:
Host Name
Alias = Host name
Display Name = Visible name
That immediately gives us a good way to hook up Nagios names to Zabbix host and visible names.
When we then take a look at the checks and how they are executed in Nagios, we also see similarities with Zabbix. In the end, both of them are monitoring solutions, of course. However, Nagios works more in a command execution kind of way, which is good for our migration. We can take this command and find an equivalent item in Zabbix. For example the check_icmp command can easily be translated into a simple check in Zabbix icmpping, icmppingloss, and icmppingsec.
For the check_tcp command we can do a similar translation. Making sure we use the simple check net.tcp.service whenever this command is executed on a Nagios host.
Because of the big differences between Nagios and Zabbix, this does mean we need to make some manual translations between the Nagios commands and Zabbix items. Depending on your Nagios instance, this could be a big task. Luckily for us, this was a smaller instance with only ICMP and TCP port checks.
The history (i.e. performance) data
Now that we know how to start creating our hosts and items, we need to understand how Nagios is storing its data. Zabbix has a big centralized MariaDB or PostgreSQL database, which makes it easy to parse through and work with our data. Unfortunately, Nagios instances use a different technique. Nagios stores data in .rrd (Round Robin Database) files and with it a .xml file to interpret the RRD file. The RRD files are not centralized like a Zabbix database, but they are more manageable in terms of storage size. We can see an RRD file per type of check in Nagios, which means we will have to grab the data from that file while understanding what it is going to belong to in Zabbix.
To see the data in the RRD file, we can use a special command line tool.
rrdtool /usr/local/nagios/share/perfdata/BeNeLux-Host-Name/Availability.rrd LAST --start -30d --end now | grep -v "nan"
Now we can clearly see that this specific RRD file above contains 8 columns, 7 with a performance value. The first column contains the timestamp in Unixtime, which is great because it will be perfect for storing in the Zabbix database. The other 7 columns in this file are different though, because we do not know what the value in the column belongs to. This is where the .xml file comes into play. The XML file belongs with the RRD file and contains details on what is included in the RRD file.
In this XML file we will find all of the required host information, which is great for creating the host in Zabbix. It also contains the check information, so we can also use this file to create the items in Zabbix. The biggest thing we will have to keep in mind is to make sure that the XML and RRD file match up in terms of number of RRD entries and columns. Column 1 in the RRD file will match with the first entry in our XML.
Let’s create a script
With the host, item and history data identified, we can start to create a script. In our case we decided to create a Python import tool. As Zabbix comes with some limitations in terms of which hostnames we can use (which are different from the limitations in Nagios), we need to sanitize our hostnames slightly.
Then all we need to do is parse through all the XML files and create new hosts in Zabbix through the Zabbix API.
It will be a very similar process for our items, as we parse through our XML file and create all of the required items in Zabbix through the API.
We can even create the triggers straight from the XML file by parsing through the different severities already set up in Nagios.
Once everything is created in Zabbix, the Python script can now start using RRDTool to parse through the RRD file, making sure to keep the XML file structure in mind when parsing through the columns.
This script can now create the hosts, the items, the triggers, and then import all of the data. We can see the hosts being created and data being imported.
The beauty of importing history data into Zabbix while the triggers are already created is then also seen below.
All of the triggers will trigger and be resolved based on the data imported, meaning that we can create problems with historic data. This means that not only do we have our historic data, but also all the problems with the correct duration as they are now discovered from the actual imported data.
To make this possible we can use the Zabbixsender tool. It has an option to include the timestamp upon every historic value imported.
Our Python script grabs the values from the RRD file and then converts them into a new _HOST_.sender file. This file will be sent to the Zabbix server using the Zabbix sender tool.
Looking at the file, we can see it contains only the name of the host, the unixtime stamp, and the actual value to send.
All we need to do is make our script send this file to the correct item in Zabbix.
Manual template and item creation
The last step will be our cleanup. We decided that we would start dirty with a one-on-one data import from Nagios. This means hostnames, item names, and trigger names are imported straight from Nagios. No templates will be created in Zabbix by the tool either, skipping the Zabbix best practice to use templates for all hosts.
We did this to make the initial import easier and not go overboard with scripting. It’s easier to have a messy Zabbix to clean up than to script everything perfectly in Python. Time is valuable.
What we did afterwards is create all the templates manually to take over the items as is from the hosts. For example, we can translate the ICMP ping and TCP stuff easily into a template.
After doing so, we do end up with some bad looking templates, but we can now start cleaning up.
We can also start creating normal trigger names and clean up…
…while changing our dynamic port names for something more expected as well.
В края на предишни парламентарни сесии правих конкретни отчети на свършеното от мен като законодателна дейност и дейност по парламентарен контрол. Сега такъв няма да има, защото в опозиция почти нищо не минава и отчетът би изглеждал като каталог на неуспелите законодателни инициативи.
Сред тях (39 законопроекта и 26 предложения между 1-во и 2-ро четене) има множество такива, чието отхвърляне има политическа цена, но изброяването за целите на това да кажем „колко лоши са управляващите“ няма собствен смисъл. И е по-ефективно да се посочва в момента и в контекста, в който се случи (иначе примери за отхвърлени от управляващите добри инициативи има много – уведомленията за здравни събития по sms/viber за да се ограничи източването на НЗОК; мерките срещу имотните измами; отпадането на стикерите от предното стъкло; дигитализацията на болничните; облекченията при издаване на лични карти; дигитализацията на инвестиционното проектиране; и др.).
Вместо това, в една доста дълга публикация, ще разгледам политическите причини за неприемането на тези и много други неща. И то е, най-просто казано, че сме в опозиция, а приоритетите на опозицията не са важни. Дори когато опозицията предложи нещо смислено, управляващите, в желанието си да демонстрират силата на мнозинството, го отхвърлят. Или внасят аналогично свое предложение, което подкрепят, за да не бъдат обвинени, че са против правилните неща (имаше няколко такива примера, които можем да отчетем като успех – наши предложения, внесени от управляващите).
Защо сме в опозиция?
Но защо сме в опозиция? Кратката версия сме я казвали многократно: след като от Демократична България бяхме на масата на преговорите за правителство и поставихме въпросите за ограничаване на влиянието на Пеевски в институциите като елемент от коалиционното споразумение, бяхме извадени от преговорите (а тези няколко точки от споразумението – изтрити). Въпросът за премиера беше последващ, както многократно бяхме заявили, затова обяснението на ГЕРБ, че прекратяват преговорите, защото не одобряваме техния кандидат премиер, не беше убедително. Но нека да видим какво стои под тази фактология.
Участието в преговорите беше с цел да предотвратим нещата, които се случват в момента – дозавладяване на държавата от Пеевски, плавното преминаване към мека диктатура, попълването на регулаторите с многогодишни мандати с ключови хора на Пеевски, и др. Виждахме вкл. риска за еврозоната, който се оказа реален в началото на мандата, когато управляващите бяха силно разколебани и трябваше да действаме с натиск отвън. И за да не се реализират всички тези рискове, преговаряхме.
И когато сега някой казва „защо не спрете Х“, а преди 8 месеца същият човек е казвал „защо сядате с тия на преговори“, отговорът може би не е очевиден и трябва да се каже – в ролята на опозиция не можеш да спреш почти нищо. Не можем да спрем избора на човек на Пеевски в Конституционния съд, който след 2 месеца вероятно ще позволи на ВСС да избере Сарафов. И съответно можехме да спрем избора на Сарафов само временно през измененията в Закона за съдебната власт, но трайно можехме да спрем не само Сарафов, но и други подобни кандидатури за главен прокурор (на които Пеевски звъни с разпореждания), единствено ако бяхме част от мнозинство, което избира ВСС, и Пеевски е извън това мнозинство. Защото с 36 депутати не можем да спрем пропукването на светския характер на образованието; същото е за измененията в Закона за държавната собственост и екооценките – можем с процедурни хитрости да забавим или да вдигнем политическата цена на такива действия, но не можем да ги спрем; не можем да спрем тормоза спрямо бизнеса под претекст „борба със спекулата“ – успяхме само да го олекотим с остра реакция, но дотолкова.
Можем да постигаме разумни компромиси, когато управлението зависи от нас. А когато не зависи – „кучетата си лаят, керванът си върви“. Т.е. като опозиция не можем да спрем разпада на държавата. Ние го знаехме тогава, но това не изглеждаше като споделено разбиране. И ако октомври ВСС избере Сарафов на 7-годишен мандат, това ще е пряко следствие от нашето неучастие в управлението. Важно е да си даваме сметка за това, когато претегляме плюсовите и минусите.
Моралният аспект на политическите решения
Имаше мнения, че преговорите за управление са неморални. Аз не смятам, че политически преговори с наши опоненти са морално компрометиращи. Биха били, ако кажем „майната им на приоритетите, важното е да сме във властта“. Но ние бяхме в преговорите именно за да постигнем целите, които сме поставили преди изборите и сме получили гласове за тях. Моралният аспект в важен в лично качество – да не пристъпваш морални граници. И аз не смятам, че сме пристъпили такива.
Но кой носи морална отговорност за щетите, които носи това управление на държавата? Морално ли беше да оставим още дълго време всички пионки на Пеевски във ВСС, регулатори, служби (дали с изтекли мандати или да бъдат презаредени – все едно)? Личният ни комфорт да не бъдем критикувани, заемайки героична поза на твърда опозиция на всичко и всички, беше в конфликт с отговорността, която носим, и която ни е дадена от избирателите. Конфликт, който не беше явен, и който не преекспонирахме, но за който трябва да си даваме сметка, от дистанцията на времето.
Разбира се, с този анализ не можем да спестим историческата вина на ГЕРБ, че заковани за Пеевски чрез зависимостите си, правят почти невъзможни нормалните политически разговори и преговорите по приоритети. Борисов затова се скри от онези преговори и отказа да се среща с нас. Може би Пеевски не го пусна, че да не вземе да поеме ангажимент за нещо, дето не трябва. Този тандем, със своето нежелание да губи влияние в съдебната власт (където не би трябвало изобщо да има такова), но с позиционирането си в прозападния сегмент, е в основата на политическата криза, което усложнява задачата на политическите представители на демократичната общност.
Тогава отчетохме и друго – ако без широко разбиране за мотивите на нашите действия влезем в едно управление, електоралната ни подкрепа ще намалее дотолкова, че да няма значение какво опитваме да направим или спрем. Защото тежестта ти в едно управление не е функция само на броя депутати, с които го подкрепяш, а и на обществената легитимност, която носиш. Този факт беше използван при ротацията – знаейки, че обществената ни подкрепа е намаляла, Пеевски и Борисов избраха предсрочни избори пред това Пеевски да бъде лишен от гарантирана квота във ВСС. Защото знаеха, че сме в слаба позиция. Между другото, тази отслабена обществена подкрепа беше и една от причините Конституционният съд спокойно да отмени промените в Конституцията, с които Пеевски вече беше започнал да губи влияние – защото бяха наясно, че обществена реакция няма да има.
Сядането на масата на преговорите през декември миналата година беше съпроводено и с множество коментари, че преговори не трябва да се водят и че не трябва да сме на тази маса. Защото ще ни излъжат, защото са „лоши и комрумпирани“, защото не са подписали „декларацията“ и т.н. Други отидоха по-далеч и се усъмниха в нашите мотиви и преминат в публична атака – част от тях бяха тролове и симпатизанти на други партии, но част легитимно изразяваха мнението си. И то е „никога (пак) нищо с ГЕРБ“. Разбира се, има съгласие по „никога с ДПС-НН“ и „никога с проруската Възраждане“. От 48-ото (сега сме 51-о) Народно събрание досега, ГЕРБ, ДПС и Възраждане имат над 121 гласа. Т.е. когато абсолютизираме тези червени линии, значи, че трябва да сме опозиция.
В 48-ото и 50-ото НС заложихме на този подход. И затова нямаше избрано правителство и кризата се задълбочи. В 51-ото подходихме едновременно диалогично и твърдо – възможно е участие, но само при много ясни, писмени гаранции, че механизмите на завладяване на държавата (в настоящия момент – от Пеевски) ще бъдат разградени. Това видимо не беше прието, поради което ние останахме в опозиция – няма нищо трагично в това. Трябва да има периоди, в които политическите сили са в опозоциия. Такава е нормалната логика.
Разговорът за отношението към властта
Но нека да ползваме този период на опозиция, за да проведем разговора за отношението към властта и условията, при които тя се упражнява. И защо се упражнява. И какво става, когато не се упражнява – тя не търпи вакуум, и тогава се упражнява от други – от тези, които не бива да я упражняват, по мнението на нас и нашите избиратели. И то без възпиращата ни роля.
В момента се упражнява от Пеевски, който не се впечатлява от фейсбук статуси, блогпостове и героични речи на протестите. Него го впечатляват битовите аспекти на властта – дали ще има достатъчно негови хора на ключови позции в служби, регулатори и най-вече ВСС. Дали законите и кадровата неадекватност на контролните механизми ще позволяват на неговите пионки да прилагат безогледна наказателна репресия и да нарушават конституционни права. А тези неща се променят с мнозинство, което се скрепява от правителство. Това е базовата политическа реалност и трябва да изхождаме от нея, когато формулираме отношението си към властта.
На протеста преди няколко седмици колегата Мирчев каза, че трябва да дадем отпор на завладяването на държавата и че ще представим план за това. Този план, неизбежно, трябва да включва формула за властта. Планът трябва да включва условията, при които да бъде упражнявана властта. И този план, поставяйки големи цели, трябва да е и реалистичен. Реалистичен би бил план за постигане на максимално силна позиция, от която да се правят разумни компромиси в името на целта – освобождавне на завладяната държава, премахване на бухалките, връщането на България в треактория на позитивно развитие като западна демокрация, а не като източна мека диктатура. Важно е да имаме реалистични очаквания, защото нарушаването на очакванията е лъжа, а хората най-много мразят да ги лъжат. Важно е и да поставим ясни цели, в името на които могат да бъдат правени определени компромиси – удържане на свободата от двата големи риска – вътрешния (Пеевски и корупцията) и външния – проникването на руското влияние (двата риска, в крайна сметка, конвергират, защото руското влияние прониква през корупционно-пробитите институции).
В огромната динамика на политическия живот напоследък, за политиците е изкушаващо да обещаят нещата, които ще успокоят избирателите им за следващите 2 месеца; които няма да създадат морални и естетически притесненния; които се плъзгат по повърхността на парадоксите на управлението в България. Това, обаче, ограничава възможността ни да постигаме целите, които същите тези избиратели искат от нас. И затова трябва да сме внимателни. Да не превръщаме политиката в карикатура на религия, която работи с опростени моралните категории за добро и зло. Тя не е това. И е опасно да бъде възприемана така. На пръв поглед „нашите са доброто, а останалите са злото“ е примамливо, но значи, че доброто трябва да взаимодейства със злото и „да се оцапа“. И разговорът отива в плоскост, която е вредна за основната цел на политиката – резултатите.
Защото ако влезем на моралната плоскост за лошите (и корумпираните), трябва да отчетем следното: в очите на много хора (особено на негласуващите) „всички сме маскари“. За избирателите на другите партии ние сме лошите и корумпирани. Защото това е преобладаващият политически дискурс в последните години. Той не е без основание такъв, но имаме параграф 22: заради липсата на обективна и работеща прокуратура, която да обвини корумпираните, всеки, който е бил в управление, е сочен за корумпиран. Но за да решим проблема с прокуратурата, е нужен хоризонт, какъвто дава само редовно управление. С това не казвам, че някой е или не е корумпиран (че четенето между редовете може да доведе до обвинение, което получих по време на преговорите – че лъскам нечий имидж). Значи само, че за да има последствия за корупцията, трябват много промени, а не само посочване.
Позитивният дневен ред
Все пак, упражняването на власт не е просто за да можеш да спираш вредни неща. Тогава тя би била безмислена – „ще влезем в управление за да е по-малко гадно“ не звучи никак мотивиращо. Да, този аспект го има и това е един от малкото ефективни начини да се спират вредните неща. Но ние трябва да излезем извън противопоставянето на добрите срещу лошите и да имаме позитивен дневен ред за България – как трябва тя да изглежда тя след 5-10-20 години, и да имаме подготвени хора за да я „заведем“ там. Да, този дневен ред минава през премахването на инструментите за нелегитимни влияния в институциите и политиката, с които в момента злоупотребява Пеевски. В този смисъл, не можем да очакваме просто една картинка с красива поляна и цветя – има много кал да се изрине преди това. Но ограничаването на Пеевски не е защото той е злото, а защото неговият начин на управление разрушава държавността, делегитимира институциите и прави невъзможен позитивния план.
Част от нашата политическа общност е свикнала да гледа на властта като на нещо гадно, до което не трябва да се допираш, защото я ще е „с комунистите и чалгарите“, я ще изпереш някого, я някой друг морален императив ще бъде нарушен. Властта действително е опасна и трябва да се дозира и осмисля добре, но ако бъде изключена всеядността и бъде упражняването ѝ бъде подчинено на споделени с избирателите приоритети и ценности, тя е единственият начин за постигане на резултати.
Пеевски каза, че управлението ще трае 4 години, така че веднага искам да успокоя по-радикалните, които могат между тези редове да прочетат, че се готви почвата за „нова сглобка“ (съмнение, подхранвано непрекъснато от всякакви интриги, вкл. на Борисов) – не, нови сглобки (с неясни условия и правила, неясна отговорност, неясна принадлежност на министри и рязка промяна на предварително заета позиция) няма да има. Но нови коалиции все някога ще има. И трябва да сме си отговорили на въпроса за условията, при които наше участие в такива коалиции е възможно и допустимо, и в какви – не. На преговорите през зимата показахме, че при условие, че няма ангажимент за ограничаване на механизмите за нелегитимни влияния (на Пеевски и не само), ние нямаме място в коалиции.
Но въпросът как да има 121 гласа за управление и 160 гласа за избор на нов Висш съдебен съвет без участието на Пеевски, стои все по-остро и трябва да бъде намерен отговор, иначе политическата класа губи историческо време за страната, в което тя се отдалечава от европейския си път, а институциите биват разграждани и завладявани все повече. Безотговорно би било да не формулираме и да не опитаме да изпълним план как стигнем до тази точка, единствено след която може реално (а не проформа) да има позитивен дневен ред.
В заключение – не можем да се похвалим с кой знае какви резултати, защото сме в опозиция. Да, нашият натиск, особено по теми, свързани с еврозоната, понякога дава ефект, но той е трудно измерем и лесно може да се отрече от управляващите. В опозиция сме, защото условията, които поставихме за участие във властта, не бяха изпълнени, а те бяха такива, защото не си представяме успешно упражняване на власт без премахване на механизмите за нелегитимно влияние. Но трябва винаги при такива решения да отчитаме и обратната страна – че извън властта не можем нито да спираме вредни за страната неща, нито да постигаме позитивни резултати. И този въпрос трябва да поставяме всеки следващ път, достатъчно рано и достатъчно ясно, за да можем заедно с избирателите да намерим решението спрямо ситуацията. И това решение да е насочено не само „за да не става по-зле“, а за да напредваме с по-бързи и некриволичещи стъпки към това България да е европейска държава. Тази яснота и откровеност и наличието на ясен план пък са начинът да бъдем в по-силна електоралена позиция, от която да можем да налагаме тези условия.
There’s a lovely device called a pistorm, an adapter board that glues a Raspberry Pi GPIO bus to a Motorola 68000 bus. The intended use case is that you plug it into a 68000 device and then run an emulator that reads instructions from hardware (ROM or RAM) and emulates them. You’re still limited by the ~7MHz bus that the hardware is running at, but you can run the instructions as fast as you want.
These days you’re supposed to run a custom built OS on the Pi that just does 68000 emulation, but initially it ran Linux on the Pi and a userland 68000 emulator process. And, well, that got me thinking. The emulator takes 68000 instructions, emulates them, and then talks to the hardware to implement the effects of those instructions. What if we, well, just don’t? What if we just run all of our code in Linux on an ARM core and then talk to the Amiga hardware?
We’re going to ignore x86 here, because it’s weird – but most hardware that wants software to be able to communicate with it maps itself into the same address space that RAM is in. You can write to a byte of RAM, or you can write to a piece of hardware that’s effectively pretending to be RAM[1]. The Amiga wasn’t unusual in this respect in the 80s, and to talk to the graphics hardware you speak to a special address range that gets sent to that hardware instead of to RAM. The CPU knows nothing about this. It just indicates it wants to write to an address, and then sends the data.
So, if we are the CPU, we can just indicate that we want to write to an address, and provide the data. And those addresses can correspond to the hardware. So, we can write to the RAM that belongs to the Amiga, and we can write to the hardware that isn’t RAM but pretends to be. And that means we can run whatever we want on the Pi and then access Amiga hardware.
And, obviously, the thing we want to run is Doom, because that’s what everyone runs in fucked up hardware situations.
Doom was Amiga kryptonite. Its entire graphical model was based on memory directly representing the contents of your display, and being able to modify that by just moving pixels around. This worked because at the time VGA displays supported having a memory layout where each pixel on your screen was represented by a byte in memory containing an 8 bit value that corresponded to a lookup table containing the RGB value for that pixel.
The Amiga was, well, not good at this. Back in the 80s, when the Amiga hardware was developed, memory was expensive. Dedicating that much RAM to the video hardware was unthinkable – the Amiga 1000 initially shipped with only 256K of RAM, and you could fill all of that with a sufficiently colourful picture. So instead of having the idea of each pixel being associated with a specific area of memory, the Amiga used bitmaps. A bitmap is an area of memory that represents the screen, but only represents one bit of the colour depth. If you have a black and white display, you only need one bitmap. If you want to display four colours, you need two. More colours, more bitmaps. And each bitmap is stored in an independent area of RAM. You never use more memory than you need to display the number of colours you want to.
But that means that each bitplane contains packed information – every byte of data in a bitplane contains the bit value for 8 different pixels, because each bitplane contains one bit of information per pixel. To update one pixel on screen, you need to read from every bitmap, update one bit, and write it back, and that’s a lot of additional memory accesses. Doom, but on the Amiga, was slow not just because the CPU was slow, but because there was a lot of manipulation of data to turn it into the format the Amiga wanted and then push that over a fairly slow memory bus to have it displayed.
The CDTV was an aesthetically pleasing piece of hardware that absolutely sucked. It was an Amiga 500 in a hi-fi box with a caddy-loading CD drive, and it ran software that was just awful. There’s no path to remediation here. No compelling apps were ever released. It’s a terrible device. I love it. I bought one in 1996 because a local computer store had one and I pointed out that the company selling it had gone bankrupt some years earlier and literally nobody in my farming town was ever going to have any interest in buying a CD player that made a whirring noise when you turned it on because it had a fan and eventually they just sold it to me for not much money, and ever since then I wanted to have a CD player that ran Linux and well spoiler 30 years later I’m nearly there. That CDTV is going to be our test subject. We’re going to try to get Doom running on it without executing any 68000 instructions.
We’re facing two main problems here. The first is that all Amigas have a firmware ROM called Kickstart that runs at powerup. No matter how little you care about using any OS functionality, you can’t start running your code until Kickstart has run. This means even documentation describing bare metal Amiga programming assumes that the hardware is already in the state that Kickstart left it in. This will become important later. The second is that we’re going to need to actually write the code to use the Amiga hardware.
First, let’s talk about Amiga graphics. We’ve already covered bitmaps, but for anyone used to modern hardware that’s not the weirdest thing about what we’re dealing with here. The CDTV’s chipset supports a maximum of 64 colours in a mode called “Extra Half-Brite”, or EHB, where you have 32 colours arbitrarily chosen from a palette and then 32 more colours that are identical but with half the intensity. For 64 colours we need 6 bitplanes, each of which can be located arbitrarily in the region of RAM accessible to the chipset (“chip RAM”, distinguished from “fast ram” that’s only accessible to the CPU). We tell the chipset where our bitplanes are and it displays them. Or, well, it does for a frame – after that the registers that pointed at our bitplanes no longer do, because when the hardware was DMAing through the bitplanes to display them it was incrementing those registers to point at the next address to DMA from. Which means that every frame we need to set those registers back.
Making sure you have code that’s called every frame just to make your graphics work sounds intensely irritating, so Commodore gave us a way to avoid doing that. The chipset includes a coprocessor called “copper”. Copper doesn’t have a large set of features – in fact, it only has three. The first is that it can program chipset registers. The second is that it can wait for a specific point in screen scanout. The third (which we don’t care about here) is that it can optionally skip an instruction if a certain point in screen scanout has already been reached. We can write a program (a “copper list”) for the copper that tells it to program the chipset registers with the locations of our bitplanes and then wait until the end of the frame, at which point it will repeat the process. Now our bitplane pointers are always valid at the start of a frame.
Ok! We know how to display stuff. Now we just need to deal with not having 256 colours, and the whole “Doom expects pixels” thing. For the first of these, I stole code from ADoom, the only Amiga doom port I could easily find source for. This looks at the 256 colour palette loaded by Doom and calculates the closest approximation it can within the constraints of EHB. ADoom also includes a bunch of CPU-specific assembly optimisation for converting the “chunky” Doom graphic buffer into the “planar” Amiga bitplanes, none of which I used because (a) it’s all for 68000 series CPUs and we’re running on ARM, and (b) I have a quad core CPU running at 1.4GHz and I’m going to be pushing all the graphics over a 7.14MHz bus, the graphics mode conversion is not going to be the bottleneck here. Instead I just wrote a series of nested for loops that iterate through each pixel and update each bitplane and called it a day. The set of bitplanes I’m operating on here is allocated on the Linux side so I can read and write to them without being restricted by the speed of the Amiga bus (remember, each byte in each bitplane is going to be updated 8 times per frame, because it holds bits associated with 8 pixels), and then copied over to the Amiga’s RAM once the frame is complete.
And, kind of astonishingly, this works! Once I’d figured out where I was going wrong with RGB ordering and which order the bitplanes go in, I had a recognisable copy of Doom running. Unfortunately there were weird graphical glitches – sometimes blocks would be entirely the wrong colour. It took me a while to figure out what was going on and then I felt stupid. Recording the screen and watching in slow motion revealed that the glitches often showed parts of two frames displaying at once. The Amiga hardware is taking responsibility for scanning out the frames, and the code on the Linux side isn’t synchronised with it at all. That means I could update the bitplanes while the Amiga was scanning them out, resulting in a mashup of planes from two different Doom frames being used as one Amiga frame. One approach to avoid this would be to tie the Doom event loop to the Amiga, blocking my writes until the end of scanout. The other is to use double-buffering – have two sets of bitplanes, one being displayed and the other being written to. This consumes more RAM but since I’m not using the Amiga RAM for anything else that’s not a problem. With this approach I have two copper lists, one for each set of bitplanes, and switch between them on each frame. This improved things a lot but not entirely, and there’s still glitches when the palette is being updated (because there’s only one set of colour registers), something Doom does rather a lot, so I’m going to need to implement proper synchronisation.
Except. This was only working if I ran a 68K emulator first in order to run Kickstart. If I tried accessing the hardware without doing that, things were in a weird state. I could update the colour registers, but accessing RAM didn’t work – I could read stuff out, but anything I wrote vanished. Some more digging cleared that up. When you turn on a CPU it needs to start executing code from somewhere. On modern x86 systems it starts from a hardcoded address of 0xFFFFFFF0, which was traditionally a long way any RAM. The 68000 family instead reads its start address from address 0x00000004, which overlaps with where the Amiga chip RAM is. We can’t write anything to RAM until we’re executing code, and we can’t execute code until we tell the CPU where the code is, which seems like a problem. This is solved on the Amiga by powering up in a state where the Kickstart ROM is “overlayed” onto address 0. The CPU reads the start address from the ROM, which causes it to jump into the ROM and start executing code there. Early on, the code tells the hardware to stop overlaying the ROM onto the low addresses, and now the RAM is available. This is poorly documented because it’s not something you need to care if you execute Kickstart which every actual Amiga does and I’m only in this position because I’ve made poor life choices, but ok that explained things. To turn off the overlay you write to a register in one of the Complex Interface Adaptor (CIA) chips, and things start working like you’d expect.
Except, they don’t. Writing to that register did nothing for me. I assumed that there was some other register I needed to write to first, and went to the extent of tracing every register access that occurred when running the emulator and replaying those in my code. Nope, still broken. What I finally discovered is that you need to pulse the reset line on the board before some of the hardware starts working – powering it up doesn’t put you in a well defined state, but resetting it does.
So, I now have a slightly graphically glitchy copy of Doom running without any sound, displaying on an Amiga whose brain has been replaced with a parasitic Linux. Further updates will likely make things even worse. Code is, of course, available.
[1] This is why we had trouble with late era 32 bit systems and 4GB of RAM – a bunch of your hardware wanted to be in the same address space and so you couldn’t put RAM there so you ended up with less than 4GB of RAM
comments
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.