Introducing Amazon OpenSearch Service and Amazon Security Lake integration to simplify security analytics

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/introducing-amazon-opensearch-service-zero-etl-integration-for-amazon-security-lake/

Today, we’re announcing the general availability of Amazon OpenSearch Service zero-ETL integration with Amazon Security Lake. This integration enables organizations to efficiently search, analyze, and gain actionable insights from their security data, streamlining complex data engineering requirements and unlocking the full potential of security data. It’s a new way to in-place query and analyze logs in Security Lake that minimizes the need to duplicate data and reduces the operational overhead of managing custom data pipelines. You can directly query your Security Lake data, saving the costs of moving data.

With OpenSearch Service zero-ETL integration with Security Lake, you can use the rich analytics capabilities of OpenSearch Dashboards to query and visualize your data in Security Lake. You can also analyze multiple data sources within a single tool and a single schema, the Open Cybersecurity Schema Framework (OCSF) schema to help with threat-hunting and investigation scenarios.

For time-sensitive investigations and monitoring, you can optionally boost query performance by enabling additional accelerations such as indexed views and dashboards in Amazon OpenSearch Service when you need fast and frequent access to a subset of your data. These capabilities provide complete visibility into all your data stored in Security Lake, regardless of the log volume, to support security investigations, better understanding of your security posture, and gain security-relevant insights.

Getting started with direct queries with Amazon Security Lake
You can get started in a few steps. First, you need to enable Security Lake by creating a Security Lake subscriber. Then, you enable a data connection in Amazon OpenSearch Service. This will automatically create an OpenSearch Serverless collection to store your direct query results and indices.

1. Enable Security Lake and setup permissions for a data lake

To enable Security Lake in the AWS Management Console, specify the data sources that you want to collect such as Amazon Route 53 DNS queries, AWS CloudTrail logs, Amazon VPC Flow logs, and AWS Security Hub findings and your AWS Regions. I chose several Regions and set the Amazon Simple Storage Service (Amazon S3) storage class and roll-up Regions to consolidate data.

Security Lake offers a 15-day trial at no cost so you can deploy it across your organization with the desired data sources and estimate the costs specific to your organization.

Once the enablement is complete, all collected data is ingested into an Amazon Simple Storage Service (Amazon S3) bucket in your account.

To access Security Lake data from an account other than the Security Lake delegated admin account, you should create an AWS Lake Formation subscriber to access and query data from AWS Glue tables associated with Security Lake. Enter the AWS account and external ID that’s authorized to access Security Lake and select the data sources to be accessed. Lake Formation provides cross-account permissions for security analysts to access data in the lake.

After you create the query subscriber, you can go to the account where you plan to deploy your OpenSearch resources and accept the AWS Resource Access Manager (AWS RAM) share that is shared by the Security Lake delegated admin account. The subscriber account will show the share status as pending until it’s manually accepted.

To learn more, visit Enabling Security Lake using the console and Create query subscriber procedures in the Amazon Security Lake User Guide.

2. Create a data connection with OpenSearch Service

You can create a zero-ETL integration in a few steps. In the OpenSearch Service console of the subscriber’s account, choose Connected data source in the Data connections section of the left navigation pane. You can then choose Security Lake as a data source type.

In the next step, you can set up the IAM permissions for accessing the Security Lake data source using the zero-ETL integration. It will also automatically create an OpenSearch Serverless collection and an OpenSearch application.

After the connection is created, you can select one of the pre-built OpenSearch dashboards that periodically query your data in Security Lake to create visualizations. You can create a dashboard using templates for VPC Flow Logs, WAF logs, and CloudTrail data sources in Security Lake.

The following is an example of a pre-built dashboard for VPC Flow logs.

To learn more about data connection, visit Data connections and permissions in the Amazon OpenSearch Service Developer Guide.

3. Query Security Lake data in the OpenSearch Dashboard

To directly query your Security Lake data in OpenSearch Dashboards, go to the Discover page.

In the Discover page, you can use the data picker workflow to locate on a specific Security Lake table to query. There is one table for each Security Lake log source.

After making a selection, you can choose the query language that you want to use, either PPL (Piped Processing Language) or SQL (Structured Query Language), and then write and run your query. The following is a PPL sample result:

You can also choose to search and run a pre-built query template to start your query. There are more than 200 SQL and PPL queries that cover all AWS log sources that are available in Security Lake. You can use the search box to find queries that you’re interested in. For example, search for “VPC Flow” to see all queries related to VPC Flow logs. There’s a description explaining each query and when you might want to use it.

If you want to perform multiple queries on the same data set, for example to support security investigations, you can create an on-demand indexed view for the results of your direct query. After the results are ingested into an OpenSearch index, you can perform low-latency subsequent queries and analysis using analytics features in OpenSearch.

To create an indexed view, choose Create indexed view and select a specified query, an index name, and a time range. After the view is created, the query results will be ingested and available to query as part of the newly created index under available indexed views.

To learn more, visit Searching data in the Amazon OpenSearch Service Developer Guide.

Now available
Amazon OpenSearch Service zero-ETL integration with Amazon Security Lake is now available in the US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), South America (São Paulo), and Canada (Central) AWS Regions.

OpenSearch Service separately charges for only the compute needed (as OpenSearch Compute Units) to query your external data in addition to maintaining indexes in OpenSearch Service. For more information, see Amazon OpenSearch Service Pricing.

Give it a try and send feedback to the AWS re:Post for Amazon OpenSearch Service or through your usual AWS Support contacts.

— Channy

Use your on-premises infrastructure in Amazon EKS clusters with Amazon EKS Hybrid Nodes

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/use-your-on-premises-infrastructure-in-amazon-eks-clusters-with-amazon-eks-hybrid-nodes/

Today, we’re announcing the general availability of Amazon Elastic Kubernetes Service (Amazon EKS) Hybrid Nodes, a new feature that you can use to attach your on-premises and edge infrastructure as nodes to EKS clusters in the cloud.

With Amazon EKS Hybrid Nodes, you can unify Kubernetes management across cloud and on-premises environments and take advantage of the scale and availability of Amazon EKS in all the places your applications need to run. You can use your existing on-premises hardware, while offloading the responsibility for managing Kubernetes control planes to EKS and conserving on-premises capacity for your workloads. Using Amazon EKS Hybrid Nodes, you can adopt consistent operational practices and tooling across your cloud and on-premises environments.

Amazon EKS Hybrid Nodes expands our support for hybrid Kubernetes deployments, adding to Amazon EKS on AWS Outposts and Amazon EKS Anywhere, which we introduced previously. You can compare how Kubernetes and hardware components are managed with each of the EKS hybrid deployment options.

Component EKS on Outposts EKS Hybrid Nodes EKS Anywhere
Hardware Managed by AWS Managed by customer
Kubernetes control plane Hosted and managed by AWS Hosted and managed by customer
Kubernetes nodes Amazon EC2 Customer-managed physical or virtual machines

When you use Amazon EKS Hybrid Nodes to attach your on-premises and edge infrastructure to EKS clusters, you can use other Amazon EKS features and integrations, including Amazon EKS add-ons, Pod Identities, cluster access entries, cluster insights, and extended Kubernetes version support. Amazon EKS Hybrid Nodes inherently integrates with AWS services including AWS Systems Manager, AWS IAM Roles Anywhere, Amazon Managed Service for Prometheus, Amazon CloudWatch, and Amazon GuardDuty for centralized monitoring, logging, and identity management.

Get started with Amazon EKS Hybrid Nodes
Here are steps to use Amazon EKS Hybrid Nodes. First, create an EKS cluster and specify your on-premises node and pod subnets. After setting up network connectivity and AWS Identity and Access Management (AWS IAM) permissions for your on-premises environment, run the Amazon EKS Hybrid Nodes CLI (nodeadm) on each host that will join the cluster. When hybrid nodes join your cluster, required networking components, such as kube-proxy and CoreDNS, are automatically installed. Before your hybrid nodes become ready to serve applications, you must install a compatible Container Network Interface (CNI) driver. The Cilium and Calico CNI drivers are supported for use with Amazon EKS Hybrid Nodes.

1. Prerequisites

You must have certain prerequisites in place before your on-premises infrastructure can join your EKS cluster as hybrid nodes, including the following:

  • Hybrid network connectivity from your on-premises environment to and from AWS using with AWS Site-to-Site VPN, AWS Direct Connect, or another virtual private network (VPN) solution
  • A virtual private cloud (VPC) with routes in its routing table for your on-premises node and, optionally, pod networks, with your virtual private gateway (VGW) or transit gateway (TGW) as the target
  • Infrastructure in the form of physical or virtual machines
  • Operating system that is compatible with hybrid nodes
  • Either AWS IAM Roles Anywhere or AWS Systems Manager set up to authenticate your hybrid nodes with the control plane
  • An EKS cluster IAM role and an EKS Hybrid Nodes IAM role

You can use Amazon Linux 2023, Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04, or Red Hat Enterprise Linux (RHEL) 8 and 9 as the node operating system for your hybrid nodes. AWS supports the hybrid nodes integration with these operating systems but doesn’t provide support for the operating systems themselves. You’re responsible for operating system provisioning and management.

To learn more, visit Prerequisites for EKS Hybrid Nodes in the Amazon EKS User Guide.

2. Create EKS cluster and enable hybrid nodes

Go to the Amazon EKS console and start to create your EKS cluster. In the Step 2 Specify networking screen, turn on Specify the CIDR blocks for your on-premises environments that you will use for hybrid nodes in the Configure remote networks to enable hybrid nodes option.

The Classless Inter-Domain Routing (CIDRs) of remote nodes and pods need to be RFC-1918 IPv4 IPv4 addresses, and they can’t overlap with the VPC CIDR or the EKS cluster Kubernetes service CIDR. Additionally, the remote node CIDR and the remote pod CIDR can’t overlap. Specifying a pod CIDR block is required if you will run webhooks on your nodes or if your CNI doesn’t use NAT for pod addresses as pod traffic leaves your nodes.

You can also create an EKS cluster using AWS Comand Line Interface (AWS CLI), eksctl, and AWS CloudFormation. To enable your cluster for Amazon EKS Hybrid Nodes, use the remote-network-config flag to specify your remote node and, optionally, your remote pod CIDR blocks.

$ aws eks create-cluster --name channy-hybrid-cluster --region=us-east-1 \
    --role-arn arn:aws:iam::012345678910:role/eks-cluster-role \
    --resources-vpc-config subnetIds=subnet-1234a11a,subnet-5678b11b \
    --remote-network-config \
{"remoteNodeNetworks":[{"cidrs":["10.80.0.0/16"]}],"remotePodNetworks":[{"cidrs":["10.85.0.0/16"]}]}}

Your cluster must be configured with API or API_AND_CONFIG_MAP cluster access authentication modes. Create an Amazon EKS access entry for your EKS Hybrid Nodes IAM role to enable nodes to join the cluster.

$ aws eks create-access-entry \
  --cluster-name my-hybrid-cluster \
  --principal-arn arn:aws:iam::012345678910:role/eksHybridNodesRole \ 
  --type HYBRID_LINUX

Amazon EKS Hybrid Nodes use temporary IAM credentials provisioned by AWS Systems Manager hybrid activations or AWS IAM Roles Anywhere to authenticate with the EKS cluster. Before connecting your on-premises nodes, you must either create an AWS Systems Manager hybrid activation or add certificates and keys to your nodes for use with AWS IAM Roles Anywhere. To learn more, visit Prepare credentials for EKS Hybrid Nodes in the Amazon EKS User Guide.

3. Connect your hybrid nodes to the EKS cluster

You’re now ready to connect Amazon EKS Hybrid Nodes to your EKS cluster. You can use the Amazon EKS Hybrid Nodes CLI (nodeadm) to simplify the installation, configuration, and registration of your hosts as hybrid nodes. nodeadm automatically installs the required AWS Systems Manager or IAM Roles Anywhere components when you run the nodeadm install command.

You can run the nodeadm install process on each running host, or you can run nodeadm install as part of your operating system build pipelines to produce an image with the components needed to join your host to an EKS cluster.

$ nodeadm install 1.31 --credential-provider <ssm, iam-ra>
{"level":"info","ts":...,"caller":"...","msg":"Loading configuration","configSource":"file://nodeConfig.yaml"}
{"level":"info","ts":...,"caller":"...","msg":"Validating configuration"}
{"level":"info","ts":...,"caller":"...","msg":"Validating Kubernetes version","kubernetes version":"1.30"}
{"level":"info","ts":...,"caller":"...","msg":"Using Kubernetes version","kubernetes version":"1.30.0"}
{"level":"info","ts":...,"caller":"...","msg":"Installing SSM agent installer..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing kubelet..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing kubectl..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing cni-plugins..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing image credential provider..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing IAM authenticator..."}
{"level":"info","ts":...,"caller":"...","msg":"Finishing up install..."}

Create a nodeConfig.yaml file on each host that contains the information required to connect to your EKS cluster. Here is an example nodeConfig.yaml that uses AWS Systems Manager hybrid activations.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
metadata:
  name: hybrid-node
spec:
  cluster:
    name: my-cluster
    region: us-east-1
  hybrid:
    roleArn: arn:aws:iam:012345678910:role/eksHybridNodesRole
    ssm:
      activationCode: <activation-code>
      activationId: <activation-id>

Now, run nodeadm on each host.

$ nodeadm init -c file:/// nodeConfig.yaml

If the preceding command is completed successfully, your hybrid node has joined your EKS cluster. You can verify this in the Amazon EKS console or with the kubectl get nodes command. Before your hybrid nodes have status as Ready, you must install a compatible CNI. To learn more, visit Install CNI for EKS Hybrid Nodes in the Amazon EKS User Guide.

4. View and manage connected your hybrid nodes in EKS console

Now that the nodes are ready, you can view your hybrid nodes and the resources running on them in the EKS console.

You’re responsible for managing your hybrid nodes and updating the software they run. You can update to the latest version of the Amazon EKS Hybrid Nodes CLI to pull in the latest fixes and updates and upgrade Kubernetes versions. To learn more, visit Upgrade EKS Hybrid Nodes in the Amazon EKS User Guide.

Now available
Amazon EKS Hybrid Nodes is now available in all AWS Regions except the AWS GovCloud (US) Regions and the China Regions.

There are no upfront commitments or minimum fees, and you pay for the hourly usage of your EKS cluster and EKS Hybrid Nodes as you use them. EKS clusters with your hybrid nodes have the same per cluster per hour cost as EKS clusters with nodes running in AWS Cloud for both standard and extended support. Additionally, EKS clusters with your hybrid nodes incur an hourly fee per hybrid node vCPU. To learn more, visit the Amazon EKS pricing page.

Give EKS Hybrid Nodes a try in the Amazon EKS console. To learn more, visit the EKS Hybrid Nodes documentation and send feedback to AWS re:Post for EKS or through your usual AWS Support contacts.

— Channy

Streamline Kubernetes cluster management with new Amazon EKS Auto Mode

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/streamline-kubernetes-cluster-management-with-new-amazon-eks-auto-mode/

Today, we’re announcing the general availability of Amazon Elastic Kubernetes Service (Amazon EKS) Auto Mode, a new capability to streamline Kubernetes cluster management for compute, storage, and networking, from provisioning to on-going maintenance with a single click. You can achieve higher agility, performance, and cost-efficiency by eliminating the operational overhead of managing the cluster infrastructure required to run production-grade Kubernetes applications at scale on Amazon Web Services (AWS).

Customers choose Amazon EKS because they can use the open standards and portability of Kubernetes with the security, scalability, and availability of AWS cloud. While Kubernetes gives advanced customers deep controls over application operations, other customers find managing the components required for production-grade Kubernetes applications to be complex and labor-intensive.

With the EKS Auto Mode, you can automate cluster management without deep Kubernetes expertise, because it selects optimal compute instances, dynamically scales resources, continuously optimizes costs, manages core add-ons, patches operating systems, and integrates with AWS security services. AWS expands its operational responsibility in EKS Auto Mode compared to customer-managed infrastructure in your EKS clusters. In addition to the EKS control plane, AWS will configure, manage, and secure the AWS infrastructure in EKS clusters that your applications need to run.

You can now get started quickly, improve performance, and reduce overhead, enabling you to focus on building applications that drive innovation instead of on cluster management tasks. EKS Auto Mode also reduces the work required to acquire and run cost-efficient GPU-accelerated instances so that your generative AI workloads have the capacity they need when they need it.

Get started with Amazon EKS Auto Mode
To get started, go to the Amazon EKS console and start to create your EKS cluster. You’ll have two options, Quick configuration (with EKS Auto Mode) and Custom configuration.

After you choose quick configuration, enter your cluster name and Kubernetes version, IAM roles, VPC subnets. You can view configuration default values in EKS Auto Mode whether you can edit after the cluster is created.

EKS Auto Mode enables the following Kubernetes capabilities in your EKS cluster:

  • Compute auto scaling and management
  • Application load balancing management
  • Pod and service networking and network policies
  • Cluster DNS and GPU support
  • Block storage volume support

When you choose Create, your EKS cluster with Auto Mode will be deployed in minutes with a single click.

If you choose the custom configuration option, you can customize other aspects of your cluster. You can use EKS Auto Mode in this option too.

You can also create an EKS Auto Mode cluster using AWS Command Line Interface (AWS CLI), eksctl, and AWS CloudFormation. Run the following eksctl command to create a new EKS Auto Mode cluster with:

$ eksctl create cluster --name=<cluster-name> --enable-auto-mode

To learn more, visit Create cluster with EKS Auto Mode in the Amazon EKS User Guide.

If you want to enable EKS Auto Mode for an existing EKS cluster, choose Manage in the EKS Auto Mode section of the Overview tab in the EKS cluster detail page.

Select the box next to Use EKS Auto Mode to enable the EKS Auto Mode. You can unselect the EKS Auto Mode that will be configured in the cluster. The default is to create both a system and a default node pool and a node class.

You can also migrate from Karpenter, EKS Managed Node Groups, and EKS Fargate to EKS Auto Mode. To learn more, visit Enable EKS Auto Mode on existing EKS clusters in the Amazon EKS User Guide.

To meet your workload requirements, you can configure specific aspects of your EKS Auto Mode clusters. While EKS Auto Mode manages most infrastructure components automatically, you can customize node networking settings, node compute resources, storage class settings, and application load balancing behaviors while maintaining the benefits of automated infrastructure management. To learn more, visit Change EKS Auto cluster settings in the Amazon EKS User Guide.

Now, you can deploy different types of workloads to Amazon EKS clusters running in EKS Auto Mode. We provide key workload patterns including sample applications, load-balanced web applications, stateful workloads using persistent storage, and workloads with specific node placement requirements. Each example includes complete manifests and step-by-step deployment instructions that you can use as templates for your own applications. To learn more, visit Run workloads in EKS Auto Mode clusters in the Amazon EKS User Guide.

Now available
Amazon EKS Auto Mode is now available in all commercial AWS Regions except China Regions where Amazon EKS is available. You can enable EKS Auto Mode in any EKS cluster running Kubernetes 1.29 and above with no upfront fees or commitments—you pay for the management of the compute resources provisioned, in addition to your regular EC2 costs. To learn more, visit Amazon EKS pricing page.

Please register for the online webinar: Simplifying Kubernetes operations with Amazon EKS Auto Mode on December 12, 2024 to learn more about how EKS Auto Mode can accelerate your time to deploy workloads to production and reduce the operational overheads of Kubernetes. To learn more, visit Automate cluster infrastructure with EKS Auto Mode in the Amazon EKS User Guide.

Give EKS Auto Mode a try in the Amazon EKS console and send feedback to AWS re:Post for EKS or through your usual AWS Support contacts.

— Channy

Introducing storage optimized Amazon EC2 I8g instances powered by AWS Graviton4 processors and 3rd gen AWS Nitro SSDs

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/introducing-storage-optimized-amazon-ec2-i8g-instances-powered-by-aws-graviton4-processors-and-3rd-gen-aws-nitro-ssds/

Today, we’re announcing the general availability of Amazon EC2 I8g instances, a new storage optimized instance type to provide the highest real-time storage performance among storage-optimized EC2 instances with the third generation of AWS Nitro SSDs and AWS Graviton4 processors.

AWS Graviton4 is the most powerful and energy efficient processor we have ever designed for a broad range of workloads running on EC2 instances using a 64-bit ARM instruction set architecture. AWS Nitro System SSDs are custom built by AWS and offer high I/O performance, low latency, minimal latency variability, and security with always-on encryption.

EC2 I8g instances are the first instance type to use third-generation AWS Nitro SSDs. These instances offer up to 22.5 TB local NVME SSD storage with up to 65 percent better real-time storage performance per TB and 60 percent lower latency variability compared to the previous generation I4g instances. Based on the AWS Graviton4 processors, I8g instances deliver up to 60 percent better compute performance and two times larger caches compared to I4g.

I8g instances offer up to 96 vCPUs, 768 GiB of memory, and 22.5 TB of storage to deliver more compute and storage choices compared with I4g instances.

Instance name vCPUs Memory (Gib) Storage (GB) Network bandwidth (Gbps) EBS bandwidth (Gbps)
I8g.large 2 16 468 up to 10 up to 10 Gbps
I8g.xlarge 4 32 937 up to 10 up to 10 Gbps
I8g.2xlarge 8 64 1,875 up to 12 up to 10 Gbps
I8g.4xlarge 16 128 3,750 up to 25 up to 10 Gbps
I8g.8xlarge 32 256 7,500
(2 x 3,750)		 up to 25 10 Gbps
I8g.12xlarge 48 384 11,520
(3 x 3,750)		 up to 28.125 15 Gbps
I8g.16xlarge 64 512 15,000
(4 x 3,750)		 up to 37.5 20 Gbps
I8g.24xlarge 96 768 22,500
(6 x 3,750)		 up to 56.25 20 Gbps
I8g.metal-24×1 96 768 22,500
(6 x 3,750)		 up to 56.25 30 Gbps

You can use I8g instances for I/O intensive workloads that require low latency access to data such as transactional databases (MySQL and PostgreSQL), real-time databases, NoSQL databases, (Aerospike, Apache Druid, MongoDB) and real-time analytics such as Apache Spark.

Additionally, I8g instances are built on the AWS Nitro System, which offloads CPU virtualization, storage, and networking functions to dedicated hardware and software to enhance the performance and security of your workloads. The Graviton4 processors offer you enhanced security by fully encrypting all high-speed physical hardware interfaces.

Things to know
Here are some things that you should know about EC2 I8g instances:

  • Operating system – EC2 I8g instances support Amazon Linux 2023, Amazon Linux 2, CentOS Stream 8 or newer, Ubuntu 18.04 or newer, SUSE 15 SP2 or newer, Debian 11 or newer, Red Hat Enterprise 8.2 or newer, CentOS 8.2 or newer, FreeBSD 13 or newer, Rocky Linux 8.4 or newer, Alma Linux 8.4 or newer, and Alpine Linux 3.12.7 or newer.
  • Networking – You can use I8g instances in storage intensive workloads that typically have burst network usage patterns. All I8g instance sizes have burst network bandwidth and can burst more than 60 minutes, depending on the instance sizes, to support the majority of the workloads requiring instance storage data hydration, backup, and snapshot over the network.
  • Migration – If you’re using I4g instances now, you will have straightforward experience migrating storage intensive workloads to I8g instances because these instances offer similar memory and storage ratios to existing I4g instances.

Now available
Amazon EC2 I8g instances are now available in the US East (N. Virginia) and US West (Oregon) AWS Regions through On-Demand instances, Savings Plans, Spot Instances, Dedicated Instances, or Dedicated Hosts.

Give EC2 I8g instances a try in the Amazon EC2 console. To learn more, visit the EC2 I8g instances page and send feedback to AWS re:Post for EC2 or through your usual AWS Support contacts.

— Channy

Now available: Storage optimized Amazon EC2 I7ie instances

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/now-available-storage-optimized-amazon-ec2-i7ie-instances/

The new storage optimized Amazon Elastic Compute Cloud (Amazon EC2) I7ie instances feature up to 120 TB of low latency NVMe storage and 5th generation Intel Xeon Scalable Processors with an all-core turbo frequency of 3.2 GHz. Fueled by 3rd Generation AWS Nitro SSDs, these instances deliver the highest storage density available in the cloud today. When compared to the previous generation of storage optimized instances, they provide:

  • Up to 65% better real-time storage performance per TB
  • Up to 50% lower I/O latency with up to 65% lower latency variability
  • Up to 40% better compute performance
  • Up to twice as many vCPUs and twice as much memory
  • 20% better price-performance

The instances are designed to support I/O intensive workloads that need a high degree of random IOPS: NoSQL databases, distributed file systems, search engines, data warehouses, and analytics.

I7ie instances are available in nine sizes with up to 192 vCPUs and 1.5 TiB of memory:

Instance Name vCPUs
 Memory
 NVMe Storage
(Nitro SSD)
 EBS Bandwidth
 Network Bandwidth
I7ie.large 2 16 GiB 1.25 TB
(1 x 1.25 TB)		 Up to 10 Gbps Up to 25 Gbps
I7ie.xlarge 4 32 GiB 2.5 TB
(1 x 2.5 TB)		 Up to 10 Gbps Up to 25 Gbps
I7ie.2xlarge 8 64 GiB 5 TB
(2 x 2.5 TB)		 Up to 10 Gbps Up to 25 Gbps
I7ie.3xlarge 12 96 GiB 7.5 TB
(3 x 2.5 TB)		 Up to 10 Gbps Up to 25 Gbps
I7ie.6xlarge 24 192 GiB 15 TB
(2 x 7.5 TB)		 Up to 10 Gbps Up to 25 Gbps
I7ie.12xlarge 48 384 GiB 30 TB
(4 x 7.5 TB)		 15 Gbps Up to 25 Gbps
I7ie.18xlarge 72 576 GiB 45 TB
(6 x 7.5 TB)		 22.5 Gbps Up to 75 Gbps
I7ie.24xlarge 96 768 GiB 60 TB
(8 x 7.5 TB)		 30 Gbps Up to 100 Gbps
I7ie.48xlarge 192 1,536 GiB 120 TB
(16 x 7.5 TB)		 60 Gbps 100 Gbps

A larger L3 cache, increased memory bandwidth, and other improvements deliver increased processing power. The VP2INTERSECT instruction (part of AVX-512) accelerates Machine Learning and graph processing workloads; the Advanced Matrix Extensions (AMX) increase deep learning training and inferencing performance.

On the network side, the instances feature over 3x the EBS bandwidth of the previous generation of storage optimized instances. This accelerates just about every I/O-intensive use case, and is especially helpful when hydrating an in-memory database or caching server. All instances sizes support the Elastic Network Adapter (ENA) and can be launched in cluster placement groups; the 48xlarge instance size also supports the Elastic Fabric Adapter (EFA).

Things to Know
Here are a couple of things that you should know about these new instances:

Regions – We are launching in the US East (Ohio, N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Frankfurt, London) AWS Regions today, with plans to expand to others in the future.

Purchase Options – I7ie instances are available in On-Demand, Spot, Savings Plan, Dedicated Instance, and Dedicated Host form.

— Jeff;

New Amazon CloudWatch Database Insights: Comprehensive database observability from fleets to instances

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-database-insights-comprehensive-database-observability-from-fleets-to-instances/

Observing your Amazon Aurora databases is now a whole lot easier. Instead of spending time setting up telemetry, building dashboards, and configuring alarms, you just open Amazon CloudWatch Database Insights and take a look. With no further setup, you can monitor the health of all of your Amazon Aurora MySQL and PostgreSQL instances in the selected Region:

Each of the sections contains a wealth of detail and I’ll get to that in a moment (this may be the ultimate “but wait, there’s more” post). From this view, I can open the filter control on the left and filter the set of instances in a couple of different ways. For example, I can filter for all of the instances running Amazon Aurora MySQL, and see that I have 66 such instances, with 3 raising alarms:

I can save the filter as a Fleet (note that Fleets are defined by specific properties and tags of the database instances and as such are inherently dynamic):

And then I can see the overall health of the fleet with a click. The entire page updates to reflect the fleet; I focus on the summary:

Behind the scenes, Database Insights looks for CloudWatch alarms that include a DBInstanceIdentifier dimension, and uses these alarms to establish a correlation between database instances and alarms. This, along with other built-in heuristics and correlation steps, allows Database Insights to deliver helpful, well-organized information that will help you to better understand the overall health of your fleet and to dive deep in order to find bottlenecks and other issues.

Clicking on an instance (represented by a hexagon) reveals details; I click on the instance name (demo-mysql-reader0) to learn more:

In the per-instance view I can also see a myriad of details:

Each of the tabs at the bottom provides additional insights into what’s happening inside the database instance. For example, selecting DB Load Analysis / Top SQL / SQL Metrics shows me which SQL statements are imposing the heaviest load, along with 29 additional metrics (not shown):

From past experience, I know that finding and understanding slow queries is a tedious yet important task. with Database Insights I can see patterns that are common to the slow queries, as well as the actual queries:

With help from AWS X-Ray, Application Signals, and the AWS Distro for OpenTelemetry SDK, I can see the services and operations that originate the queries to the database instance:

The red X indicates that this operation is failing the associated Service Level Objective (SLO), an application performance monitoring aspect of Application Signals. An SLO defines the reliability of a service against customer expectations, and can be set up by selecting the service and clicking Create SLO. There are a couple of steps and some very helpful options, but at the core a SLO is measured as a percentage of successful requests over an extended period of time:

If the database instance is configured to send logs to CloudWatch Logs, I can see and search the logs, filtered by the selected time period, and within a particular log group:

There’s still a lot more to explore at the fleet level. For example, I can see the ten calling services which drive the highest DB load (again, this is powered by AWS X-Ray, Application Signals, and the AWS Distro for OpenTelemetry SDK):

And I can see the top 10 instances with respect to any of eight different metrics:

I could go on all day, but I will leave the rest for you to explore. As I never tire of saying, this feature is available now and you can start using it today.

Things to Know
Here are a couple of things to know about Database Insights:

Supported Databases – You can use Database Insights with Amazon Aurora MySQL and Amazon Aurora PostgreSQL database instances.

Pricing – There is a per-hour, per-database instance charge based on the average number of vCPUs used (for provisioned instances) or Aurora Capacity Units (for Serverless v2 databases) monitored, with separate charges for ingestion and storage of database logs. See the CloudWatch Pricing page for more information.

Regions – This feature is available in all commercial AWS Regions.

— Jeff;

New Amazon CloudWatch and Amazon OpenSearch Service launch an integrated analytics experience

Post Syndicated from Elizabeth Fuentes original https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-and-amazon-opensearch-service-launch-an-integrated-analytics-experience/

Today, Amazon Web Services (AWS) announces a new integrated analytics experience and zero-ETL integration between Amazon CloudWatch and Amazon OpenSearch Service. This integration simplifies log data analysis and visualization without data duplication, streamlining log management while reducing technical overhead and operational costs. CloudWatch Logs customers now have access to two additional query languages beyond CloudWatch Logs Insights QL, while OpenSearch customers can query CloudWatch logs in place without creating separate extract, transform, and load (ETL) pipelines.

Organizations often need different analytics capabilities for their log data. Some teams prefer CloudWatch Logs for its scalability and simplicity in centralizing logs from all their systems, applications, and AWS services. Others require OpenSearch Service for advanced analytics and visualizations. Previously, integration between these services required maintaining separate ingestion pipelines or creating ETL processes. This new integration helps customers get the best of both services by eliminating this complexity by bringing the power of OpenSearch analytics directly to CloudWatch Logs, without any data copy.

Amazon CloudWatch Logs now supports OpenSearch Piped Processing Language (PPL) and OpenSearch SQL directly within the CloudWatch Logs Insights console. You can use SQL to analyze data and correlate logs using JOIN. You can use SQL functions (such as JSON, mathematical, datetime, and string functions) for intuitive log analytics. You can also use the OpenSearch PPL to filter, aggregate, and analyze data. With a few clicks, you can access pre-built, out-of-the-box dashboards for vended logs, such as Amazon Virtual Private Cloud (VPC), AWS CloudTrail, and AWS WAF. These dashboards enable faster monitoring and troubleshooting through visualizations, such as analyzing flows over time, top talkers, megabytes, and packets transferred over time, without having to configure individual widgets or build specific queries. You can analyze VPC flows over time, identify top talkers, track network traffic metrics, monitor web request trends in AWS WAF, or analyze API activity patterns in AWS CloudTrail.

Additionally, OpenSearch Service users can now analyze CloudWatch logs using OpenSearch Discover and run SQL and PPL, similar to how they analyze data in Amazon Simple Storage (Amazon S3), and build indexes and create dashboards directly without any ETL operations or separate ingestion pipelines.

Let’s explore how this integration works
To demonstrate the new OpenSearch SQL and PPL query capabilities in CloudWatch, I start in the CloudWatch console. In the navigation pane, I choose Logs then Logs Insights. After selecting log groups for the query, I can now use OpenSearch PPL or OpenSearch SQL query languages directly within CloudWatch Logs Insights, with no additional setup or integration required. Using this new capability, I can write complex queries using familiar SQL syntax or OpenSearch PPL, making log analysis more intuitive and efficient. In the Query commands menu, you can find sample queries to help you get started.

This example demonstrates how to use SQL JOIN to combine data from two log groups: pet adoptions and pet availability. By filtering for specific customer IDs, you can analyze related log records and trace IDs for troubleshooting purposes.

One of the powerful features of this integration for CloudWatch Logs customers is the ability to create pre-built dashboards for Amazon VPC Flows, AWS CloudTrail and AWS WAF logs. Let’s explore this by creating a dashboard for AWS WAF logs. In the Analyze with OpenSearch tab, I choose Settings and follow the steps.

After a few minutes, my integration is ready and I go to Create an OpenSearch dashboard. In the options Select automatic dashboard type, I choose AWS WAF logs.

In the Dashboard data configuration tab, I can select Data synchronization frequency to occur every 15 minutes. I Select the log groups and View log samples of the selected log groups. I finish by choosing Create a dashboard.

After creating my dashboard, I can explore my logs. The AWS WAF logs dashboard provides comprehensive visibility into web application firewall metrics and events, with automatically configured visualizations that help you monitor and analyze security patterns.

Similarly, the CloudTrail dashboard offers deep insights into API activity across your AWS environment. It’s useful for monitoring API activity, auditing actions, and identifying potential security or compliance issues. 

The VPC Flow Logs dashboard provides detailed visualization of key metrics from your logs for network traffic analysis. You can analyze network traffic, detect unusual patterns, and monitor resource usage. The dashboard currently supports only VPC v2 fields (default format). Custom formatted fields are not supported.

With zero-ETL to access CloudWatch data from OpenSearch Services, I also can build an OpenSearch dashboard from the OpenSearch Service console without having to build and maintain an ETL process. For this, I go to Central management, then I select the new Connected data sources menu, click choose Connect to create a new connected data source, and choose CloudWatch Logs.

In the next step, I name my data source and choose to Create a new role, which must have the necessary permissions to execute actions on OpenSearch Service. You can see them in the Sample custom policy.

https://d2908q01vomqb2.cloudfront.net/artifacts/AWSNews/2024/AWSNEWS-1365-Role.gif

In the Set up OpenSearch step, configure a OpenSearch data connection for CloudWatch Logs by selecting Create a new collection. As part of setting up the CloudWatch Logs source, a new OpenSearch Service serverless collection and OpenSearch UI application is created to store the indexed views and provide a user interface to analyze your CloudWatch Logs data. I create a new collection, name it, and configure the OpenSearch application and workspace within the application. After setting the Data retention days, I choose Next and finish with Review and connect.

When the integration with CloudWatch is ready, I can choose between Explore logs without indexing data which will take me to a querying interface in Discover or Explore vended logs by creating a dashboard for Amazon VPC Flows, CloudTrail and AWS WAF logs.

After I select Explore logs, OpenSearch UI takes me to Discover in the application workspace I created during the data source setup. In Discover, I select the data picker and choose View all available data to access my CloudWatch Logs data source and log groups.

After I select the log groups, I can analyze my CloudWatch logs using OpenSearch SQL and PPL directly in Discover, without having to switch between applications.

To create a dashboard, I return to the Connected data sources overview page on the console. From there, I select Create dashboard, which allows me to visually analyze my CloudWatch data without having to define queries or build visualizations, as I previously did in the CloudWatch console

After the dashboard is created, I navigate to OpenSearch resources where I can see the newly created indexes being populated with data in my Collection. After I have the data, I can go to the dashboard with the data from the CloudWatch logs that I selected in the configuration, and as more data comes in, it will be displayed in near real-time on the OpenSearch dashboard.

With this zero-ETL integration you can ingest data directly into OpenSearch, using its powerful query capabilities and visualization features while maintaining data consistency and reducing operational overhead.

Integration Highlights
For CloudWatch customers:

  • Query capabilities – Streamline log investigation by using OpenSearch SQL and PPL queries directly within the CloudWatch Logs Insights console.
  • Analytics features – With a few clicks, access pre-built, out-of-the-box dashboards for vended logs, such as VPC, AWS WAF, and CloudTrail logs. These dashboards enable faster monitoring and troubleshooting through visualizations for analyzing flows over time, top talkers, megabytes, and packets transferred over time, without having to configure individual widgets or build specific queries.
  • Getting started for CloudWatch users – Configure integration from CloudWatch Logs to OpenSearch Service. For more information refer to the Amazon CloudWatch Logs query capabilities and Amazon CloudWatch Logs vended dashboard documentation.

For OpenSearch Service customers:

  • Zero-ETL integration – Access and analyze CloudWatch data directly from OpenSearch Service without building or maintaining ETL processes. This integration eliminates separate ingestion pipelines while reducing storage costs and operational overhead through simplified data management and zero data duplication.
  • Getting started for OpenSearch users – Create a data connection selecting CloudWatch as a data source from OpenSearch Service. For more information, refer to the Amazon OpenSearch Service Developer Guide.

Regional availability and pricing
This integration is now available in AWS Regions where Amazon OpenSearch Service direct query is available. For pricing details and free trial information, you can visit the Amazon CloudWatch Pricing and Amazon OpenSearch Service Pricing pages.

PS: Writing a blog post at AWS is always a team effort, even when you see only one name under the post title. In this case, I want to thank Joshua Bright, Ashok Swaminathan, Abeetha Bala, Calvin Weng, and Ronil Prasad for their generous help with screenshots, technical guidance, and sharing their expertise in both services, which made this integration overview possible and comprehensive.

— Eli

Rust 1.83.0 released

Post Syndicated from corbet original https://lwn.net/Articles/1000273/

Version
1.83.0 of the Rust language has been released.

This release includes several large extensions to what code running
in const contexts can do. This refers to all code that the
compiler has to evaluate at compile-time: the initial value of
const and static items, array lengths, enum
discriminant values, const generic arguments, and functions
callable from such contexts (const fn).

There are also quite a few new stabilized APIs.

The OpenWrt One router is now shipping

Post Syndicated from corbet original https://lwn.net/Articles/1000272/

The OpenWrt One router, which was reviewed
here recently, is
now generally available.

This is the first wireless Internet router designed and built with
your software freedom and right to repair in mind. The OpenWrt One
will never be locked down and is forever unbrickable. This device
services your needs as its owner and user. Everyone deserves
control of their computing. The OpenWrt One takes a great first
step toward bringing software rights to your home: you can control
your own network with the software of your choice, and ensure your
right to change, modify, and repair it as you like.

Седмицата (25–30 ноември)

Post Syndicated from Надежда Радулова original https://www.toest.bg/sedmitsata-25-30-noemvri-2/

Днес ще убия нещо. Каквото падне.
Писнало ми е да ме пренебрегват и днес
ще го раздавам Господ. 

Седмицата (25–30 ноември)

Така започва стихотворението на Каръл Ан Дъфи „Образование за свободното време“ (прев. Калоян Игнатовски). Дали е такъв случаят и с децата с качулки, които безчинстват из няколко, а може би и повече от няколко софийски мола? Социално-мрежовият и телевизионният шум, разбира се, се вдигна най-вече около един конкретен мол и неговата клета охрана: така е най-лесно – избираш си виновен и го изправяш до стената. За всички обаче е ясно, че при липсата на адекватни действия от страна на полицията и съобразен с международните спогодби справедлив достъп до правосъдие на всяко дете, алтернативното „образование за свободното време“ е превзело не едно и две публични пространства.

Срещали сме въпросните деца и в подлеза на Софийския университет (чийто праг вероятно няма да имат шанса да прекрачат), и в уличките около бул. „Ситняково“, и по бул. „Тодор Каблешков“, и около Южния парк, и в преките на бул. „Тодор Александров“, и в станциите на метрото, където гонят да бият връстниците си бежанци – също деца, но с още по-малко късмет и от тях… Някои все още носят пистолети играчки. Други имат боксове и знаят как да ги използват. Трети просто безропотно следват тълпата.

Общото е, че всички тези деца ходят или доскоро са ходили на училище. И техните родители са ходили на училище. Срещали са се лице в лице с онези страшилища-на-седемте-морета-и-дванайсетте-класа – националните външни оценявания. Още в първи клас са рецитирали „Аз съм българче“. Всяка пролет са пели „Върви, народе възродени“. И е възможно да знаят кои са Сивушка и Белчо (дори и без да са чели иначе гениалния разказ на Елин Пелин, подобно на голяма част от Facebook критикарите). Но какво от това?! Както виждаме – нищо не ги е спасило, не им е помогнало, не е угасило гнева им, не е преборило дебнещите ги бесове. И така ще е, докато поколение след поколение деца се въртят в барабана на една старомодно скована и свъсила вежди образователна система. Система, в която приятелството, взаимното уважение, емпатията, толерантността, грижата към близкото и ближното и любопитството към далечното, чуждото и другото, все още са ключови думи в европейски проекти от чекмеджето, но не и реални, всекидневни ценности, въплътени в игра, в разговор, в преживяване, в знание, в съзидание.

Мой познат, „чистач“ на българското съдържание в Meta, беше споделил с мен преди време какво му струва всеки ден да гледа задължителните секунди насилие, преди да ги свали и докладва. Както и че минути след това същите снимки или видеа се появяват на друго място, в следваща група, и в следваща. И в следваща. Че насилието се репродуцира експоненциално. И че децата плашещо често са там – понякога жертви, понякога насилници, а често и двете.

Струва ми се обаче важно да не забравяме нито за миг, че и едните, и другите деца са наши. Какво ли бъдеще ги чака, предвид настоящето им в разбити семейства, в бездушните коридори на образователната система, в мола и в подлеза, с бокса и под качулката? И въпреки всичко ми се иска да вярвам, че в някаква степен нещата все още зависят от нас, макар да изглежда, че ни се изплъзват от ръцете…

Затова и не се отказваме, продължаваме да търсим отговорите и в настоящия брой:

Започваме с един важен материал именно за децата. В „Имат думата ЛГБТИ учениците. Докато не е забранено да говорят“ Светла Енчева анализира тазгодишното национално проучване на Фондация „Сингъл Степ“ на нагласите към ЛГБТИ децата в училище. В сравнение със същото изследване от 2018 г. климатът значително се е влошил, а процентът хомофобски и трансфобски прояви от страна и на ученици, и на учители тревожно расте. С приетите по-рано през годината промени в Закона за предучилищното и училищното образование тази тенденция ще се задълбочава, а с избора на Тръмп за президент на САЩ ще получава и външнополитическа легитимация. В една такава ситуация и при липса на здрав разум в главите на голяма част от законотворците може да очакваме и зачестяване на проявите на психическо и физическо насилие, основано на пола и половата идентичност.

И „Докато чакаме здравият разум да се завърне“, председателката на Комитета по правата на детето на ООН Ан Скелтън ни призовава въпреки фрустрацията от трудните времена и поликризата да устоим на натиска, да работим с наличните инструменти, колкото и несъвършени да са те, само и само да не изоставим проекта за детските права. Разговорът е част от поредицата на Надежда Цекулова за достъпа до образование, подкрепена от „Лидл България“.

Темата за устояването на натиска се появява и във втората част на анализа „Отново на кръстопът: Българската външна политика между Вашингтон и Москва“. Искрен Иванов продължава да разсъждава върху цивилизационния избор, пред който е изправена страната ни в момента: да се превърне в истинска, съвременна, работеща демокрация – или да си остане буфер между Запада и Русия, каквато е в последните няколко десетилетия. И двете перспективи крият огромни политически залози. И различни варианти на бъдеще.

Засега обаче бъдещето – поне близкото – не изглежда никак обещаващо, още по-малко пък предлага варианти. Осмото гласуване за председател на Парламента се увенча с неуспех, което всъщност до голяма степен е… успех. За кого, питате? Ами за Румен Радев и неговите вездесъщи и подопечни служебни правителства, разбира се. Следващото тропа на вратата, докато партиите във властта скоростно ерозират, а избирателите губят и ориентация, и почва под краката си. Още за парламентарните несгоди и за кризите на републиката и демокрацията четете в седмичния политически анализ на Емилия Милчева „Време разделно“.

Плачевното състояние на политическата класа в момента не е като да не е предизвестено обаче… Понякога се оказва, че привидно категорични политически избори, идентификации и афилиации са подвеждащи и в действителност основани на ценностен произвол и непоследователност. Така е например с българското разбиране за ляво и дясно. В статията си „Особеностите на българския консерватизъм“ Александър Драганов разглежда генезиса на това понятие от 1878 г. до наши дни, като със заострен хумористичен щрих и не без помощта на литературните ни класици обрисува забележителен портрет на българския консерватор.

И от нашата страна потегляме направо на север, за да прекараме няколко „Снежни лета в една исландска хижа“. Във втората част на текста си Светла Стоянова ни разказва за опита си на хижарка – не къде да е, а на истински труднодостъпно и опасно място, където през юни разравяш тунели в снега, за да си отвориш вратата. И даваш подслон на гладни и измръзнали хора. Ако пък се наложи – ставаш планински спасител, защото често пъти положението е на живот и смърт.

Както предстои да научите, това, което е истински лукс в исландските хижи, несъмнено са пресните плодове и зеленчуци. Точно те са тема и в тазседмичните научни новини на Михаил Ангелов. Как се произвеждат по-сладки домати и отразява ли се това на добива? Има ли начин да се спасят бананите, подгонени от унищожителни заболявания? И още: нови революционни генни терапии; методи за пречистване от т.нар. вечни химикали и какви ли не чудеса, за които ще разберете само ако кликнете ето тук: „По-сладки домати, спасение за бананите…“.

Нека завършим месеца подобаващо, а именно – със стихотворение. Този път ви предлагаме „Били. Метаморфози“ от Калоян Игнатовски с пожелание да пазите в себе си и тигъра, и котарака. Светът е сложен и със сигурност ще дойде момент, когато ще имате нужда и от двамата.

А междувременно – както обикновено – ви пожелаваме приятно четене. И ако броят ви хареса, нека тигърът изръмжи, котаракът измяучи, а вие ни подкрепете, за да се срещаме още много съботи по същото време.

Подкрепете ни

The collective thoughts of the interwebz