Tag Archives: GPU

Intel Crescent Island 160GB to 480GB LPDDR5X AI GPU at Hot Chips 2026

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/intel-crescent-island-160gb-to-480gb-lpddr5x-ai-gpu-at-hot-chips-2026/

We learned more about the new 160GB to 480GB LPDDR5X Intel Crescent Island GPU focused on memory capacity at Hot Chips 2026

The post Intel Crescent Island 160GB to 480GB LPDDR5X AI GPU at Hot Chips 2026 appeared first on ServeTheHome.

Sharing Capacity Blocks for ML Across Your AWS Organization

Post Syndicated from Tyler Klimas original https://aws.amazon.com/blogs/compute/sharing-capacity-blocks-for-ml-across-your-aws-organization/

When your data science team reserves GPU instances for a two-week training job but completes it in four days, that capacity has the potential to sit unused while your computer vision team waits another week to start their project. Now you can eliminate this GPU waste and scheduling conflict by sharing Capacity Blocks for ML across your AWS Organization. This scheduling mismatch between teams creates bottlenecks that delay product launches, increase infrastructure costs, and slow your ability to deliver machine learning (ML) powered features to customers. With cross-account sharing for Amazon Elastic Compute Cloud (Amazon EC2) Capacity Blocks for ML, you can now distribute reserved graphics processing unit (GPU) capacity across teams based on actual demand rather than rigid scheduling predictions. This means your computer vision team can use the capacity as soon as the data science team is done.

In this post, we’ll show you how to configure cross-account sharing for Capacity Blocks for ML, set up monitoring for your shared resources, and optimize instance utilization through alerting. By increasing the utilization rates and reducing over-provisioning, you improve your resource efficiency and cost optimization for your organization.

You can reduce idle resources in your ML team’s account by sharing capacity with other teams waiting for GPUs. Additionally, you can maintain Capacity Blocks for ML centrally. This lets you control which teams have access to the capacity and helps you reduce waste and bottlenecks in your organization. Before starting into the tutorial, let’s review how Capacity Blocks for ML and AWS RAM work together.

Overview

Capacity Blocks for ML let you reserve GPU-based accelerated compute instances ahead of time for short duration ML workloads. When you launch instances in Capacity Blocks for ML, Amazon EC2 automatically places the instances in Amazon EC2 UltraClusters, giving you low-latency, petabit scale networking. UltraClusters provide the high performance networking your training workloads require.

You see exactly when GPU capacity is available and schedule your Capacity Blocks for ML to start when it makes sense for your project. You pay upfront for the entire reservation period. This makes Capacity Blocks for ML useful when you need GPUs for days to months. It provides predictable capacity without long-term commitments.

When you purchase Capacity Blocks for ML, you can share it with other accounts in your AWS Organization using AWS Resource Access Manager (AWS RAM). With AWS RAM, you can share AWS resources across accounts within your organization. When you share with other accounts, those accounts become consumer accounts that can launch instances using your capacity. As the owner account, you pay the upfront reservation cost and retain ownership. If you’re launching instances from a consumer account, you are responsible for additional costs such as operating system licensing charges. Capacity Blocks can be shared to multiple accounts simultaneously, with the entire Capacity Block reservation being shared on a first come, first served basis.

Overview of AWS Organizations showing an owner account sharing to two consumer accounts using an AWS RAM resource share.

Figure 1: Capacity Block sharing using Resource Access Manager.

With the share feature, you benefit from flexible GPU capacity management when your priorities shift, or teams finish work at different times. Now, when your data science team completes experimentation early, your other teams can use that capacity for production training. If priorities shift mid-quarter, you can move capacity where it’s needed most.

In this tutorial, you’ll share a Capacity Block for ML across accounts and then create an alarm to monitor utilization when it drops below a threshold. Before you start, complete the following prerequisites.

Prerequisites

To share Capacity Blocks for ML, you must first find and purchase a Capacity Block. Only standard Capacity Blocks for ML can be shared using AWS RAM. UltraServer Capacity Blocks are not eligible for sharing.

You can share Capacity Blocks only within your AWS Organization. Verify the owner of the Capacity Blocks as well as the consumer(s) are within the same organization. For guidance, see Creating and configuring an organization.

Before sharing Capacity Blocks, you must configure resource sharing with AWS Organizations. Only the management account with the following required AWS Identity and Access Management (IAM) permissions can enable resource sharing within an Organization:

ram:EnableSharingWithAwsOrganization

iam:CreateServiceLinkedRole

organizations:EnableAWSServiceAccess

organizations:DescribeOrganization

Using the AWS Management Console of the management account:

  1. Navigate to the AWS RAM console.
  2. In the left navigation pane, choose Settings.
  3. Select Enable sharing with AWS Organizations.

Enable sharing with AWS Organizations in Settings of Resource Access Manager.

Figure 2: Enable sharing with AWS Organizations in AWS RAM.

Using the AWS Command Line Interface (CLI):

  1. Run this command to give AWS RAM trusted access to your organization’s account structure: 
        aws organizations enable-aws-service-access --service-principal ram.amazonaws.com
  2. Turn on resource sharing within your organization so accounts and OUs can access shared resources without manual acceptance: 
        aws ram enable-sharing-with-aws-organization

After you turn on sharing in your organization, you need the following IAM permissions to create resource shares:

ram:CreateResourceShare

ram:AssociateResourceShare

ram:GetResourceShares

Now that you’ve completed the prerequisites, you’ll learn how to share the Capacity Blocks for ML to other accounts of your organization.

Tutorial

You’ll complete this sharing process in four steps:

  1. Create a resource share.
  2. Attach Capacity Block to the resource share.
  3. Verify the share in your consumer account.
  4. Monitor the resource share.

Verify Capacity Reservation (console)

  1. In your Capacity Block owner’s account, navigate to the Amazon EC2 console.
  2. In the left navigation pane, choose Capacity Reservations.
  3. Confirm your Capacity Blocks for ML is in Active or Scheduled state.
  4. If you have a Resource share already configured, choose Actions, Share and select your Resource share.

Figure 3: EC2 Capacity Reservation

Share Capacity Blocks for ML (console)

You now will create a Resource Share and associate the following resources.

  1. Navigate to the AWS RAM console in your Capacity Block owner’s account.
  2. In the left navigation pane, choose Resource shares.
  3. Choose Create resource share.

Create Resource share in AWS RAM Console

Figure 4: Create Resource share in AWS RAM

  1. Enter a name for your resource share.
  2. Under Select resource type, choose Capacity Reservations.
  3. Select your Capacity Block from the list.
  4. Under Principals, specify the accounts, organizational units, or organization to share with.Select principals to share resources in AWS RAM.Figure 5: Select principals to share resources with
  5. Choose Create resource share.

Share Capacity Blocks for ML (AWS CLI)

Replace the placeholder values in the following CLI commands below with your actual values:

  • arn:aws:ec2:us-east-2:123456789012:capacity-reservation/cr-1234abcd56EXAMPLE → Your Capacity Reservation ARN
  • 111122223333 → The AWS account ID of the principal you’re sharing with
  • arn:aws:ram:us-east-2:123456789012:resource-share/7ab63972-b505-7e2a-420d-6f5d3EXAMPLE → Your RAM resource share ARN

 

  1. Create resource share with Capacity Block and principals:
    aws ram create-resource-share \
         --name capacity-block-share \
         --resource-arns arn:aws:ec2:us-east-2:123456789012:capacity-reservation/cr-1234abcd56EXAMPLE \ 
         --principals 111122223333
  1. To add a Capacity Block to existing resource share:
    aws ram associate-resource-share \
         --resource-share-arn arn:aws:ram:us-east-2:123456789012:resource-share/7ab63972-b505-7e2a-420d-6f5d3EXAMPLE \
         --resource-arns arn:aws:ec2:us-east-2:123456789012:capacity-reservation/cr-1234abcd56EXAMPLE

Access and Launch shared Capacity Blocks (console)

After you add the Capacity Block to a resource share, your consumer accounts automatically gain access when you share the Capacity Block within the same AWS Organization.

  1. Navigate to the AWS RAM console in your consumer account.
  2. In the left navigation pane, choose Shared with me, Resource shares. Verify the Resource share is Active.Within your consumer account, verify resource share.Figure 6: In your consumer account, verify the resource share
  3. Navigate to the Amazon EC2 console. In the left navigation pane, choose Capacity Reservations.
  4. Confirm the shared Capacity Block appears and is in Active or Scheduled state. Because sharing is asynchronous, the Capacity Block may take a few moments to appear even after the resource share shows Active.
  5. Navigate to the Amazon EC2 console and choose Launch instance.
  6. Configure your instance as required (AMI, instance type, key pair, etc.).
  7. Under Advanced details, for Purchasing option, choose Capacity Blocks.
  8. For Capacity reservation, choose Specify Capacity Reservation.
  9. For Capacity reservation targeted ID, select or enter your Capacity Block reservation ID.
  10. Launch the instance.

Access shared Capacity Blocks (AWS CLI)

Replace the placeholder values in the following CLI commands below with your actual values:

  • ami-0abcdef1234567890 → Your AMI ID
  • cr-0c54f6734d944345a → Your Capacity Reservation ID
  1. List resource shares in your consumer account:
    aws ram get-resource-shares --resource-owner OTHER-ACCOUNTS
  1. Verify that capacity reservation is available:
    aws ec2 describe-capacity-reservations
  1. Launch EC2 instance from Capacity Block:
    aws ec2 run-instances \
         --image-id ami-0abcdef1234567890 \
         --count 1 \
         --instance-type p5.48xlarge \
         --key-name my-key-pair \
         --subnet-id subnet-0abcdef1234567890 \
         --instance-market-options MarketType='capacity-block' \
         --capacity-reservation-specification CapacityReservationTarget={CapacityReservationId=cr-0c54f6734d944345a}

Monitor usage (console)

You can create Amazon CloudWatch alarms to proactively identify low utilization of your Capacity Block. This helps you to improve the usage of your capacity reservation. This section shows you how to create an Amazon Simple Notification Service (Amazon SNS) email notification when the number of running instances drops below a certain threshold.

In addition to monitoring usage, AWS CloudTrail logs capture API events related to your Capacity Block, including the CapacityReservationId. As the owner, you can see which accounts are consuming instances and when.

Step 1: Create an SNS Topic for Notifications

  1. Open the Amazon SNS console.
  2. In the left navigation pane, choose Topics.
  3. Choose Create topic.
  4. For Type, select Standard.

Create SNS Topic for CloudWatch alarm.

Figure 7: Create SNS Topic

  1. For Name, enter capacity-block-alerts.
  2. Choose Create topic.

Step 2: Create an SNS Subscription:

  1. In the left navigation pane, choose Create subscription.Create SNS SubscriptionFigure 8: Create SNS Subscription
  2. For Protocol, choose Email.
  3. For Endpoint, enter your email address.
  4. Choose Create subscription.

Step 3: Create the CloudWatch Alarm

  1. Navigate to the Amazon CloudWatch console.
  2. In the left navigation pane, choose Alarms, All alarms.
  3. Choose Create alarm.
  4. Choose Select metric.
  5. Choose EC2 Capacity Reservations.
  6. Choose By Capacity Reservation.
  7. Find your Capacity Block ID (e.g., cr-12345678abcdef).
  8. Select the checkbox next to InstanceUtilization.
  9. Choose Select metric.

Step 4: Configure the Metric

  1. Under Metric:
  2. For Statistic: Select Average.
  3. For Period: Select 5 minutes.
  4. Under Conditions choose Threshold type: Select Static.
  5. Whenever InstanceUtilization is…: Select Lower than…: Enter 20 (This metric is measured in percentage).
  6. Choose Next.

Step 5: Configure Actions

  1. Under Notifications:
  2. Alarm state trigger: Select In alarm.
  3. Select an SNS topic: Choose Select an existing SNS topic.
  4. Send a notification to…: Select capacity-block-alerts.Configure CloudWatch AlarmFigure 9: Configure CloudWatch Alarm
  5. Choose Next.

Step 6: Name and Create Alarm

  1. For Alarm name, enter: CapacityBlock-LowUtilization-cr-123456789abcdef.
  2. For Alarm description, enter: Alert when Capacity Block utilization drops below 20%.
  3. Choose Next.
  4. Review your configuration and choose Create alarm.

Monitor usage (AWS CLI)

Replace the placeholder values in the following CLI commands below with your actual values:

  • 123456789012 → Your 12-digit AWS account number
  • cr-0c54f6734d944345a → Your Capacity Reservation ID
  • 7ab63972-b505-7e2a-420d-6f5d3EXAMPLE → Your RAM resource share ID
  • [email protected] → Your email address for notifications

 

  1. Create the SNS topic:
    aws sns create-topic \
        --name capacity-block-alerts
  1. Using the TopicArn from the output, subscribe your email:
    aws sns subscribe \
        --topic-arn arn:aws:sns:us-east-2:123456789012:capacity-block-alerts \
        --protocol email \
        --notification-endpoint [email protected]
  1. Create the full CloudWatch alarm:
    aws cloudwatch put-metric-alarm \
        --alarm-name "CapacityBlock-LowUtilization-cr-1234EXAMPLE" \
        --alarm-description "Alert when Capacity Block utilization drops below 20%" \
        --namespace "AWS/EC2CapacityReservations" \
        --metric-name "InstanceUtilization" \
        --dimensions Name=CapacityReservationId,Value=cr-0c54f6734d944345a \
        --statistic Average \
        --period 300 \
        --evaluation-periods 1 \
        --threshold 20 \
        --comparison-operator LessThanThreshold \
        --alarm-actions arn:aws:sns:us-east-2:123456789012:capacity-block-alerts

Clean up (console)

As the owner of the Capacity Block, you retain the ability to modify the resource share. However, owners cannot modify instances that consumers launch into Capacity Blocks they have shared. This section outlines how to clean up your previous work.

Using the AWS Management Console:

Stop sharing the Capacity Block

  1. Navigate to AWS RAM console.
  2. In the left navigation, choose Shared by me, Resource shares.
  3. Select your resource share.
  4. Choose Modify.
  5. Remove the Capacity Block from the resource share or delete the entire resource share.

Delete the CloudWatch Alarm

  1. Navigate to the Amazon CloudWatch console.
  2. In the left navigation, choose Alarms, All alarms.
  3. Select the alarm you created.
  4. Choose Actions, Delete.
  5. Confirm deletion.

Delete the SNS Topic and Subscription

  1. Navigate to the Amazon SNS console.
  2. In the left navigation, choose Subscriptions.
  3. Select the subscription and choose Delete.
  4. In the left navigation, choose Topics.
  5. Select capacity-block-alerts and choose Delete.
  6. Confirm deletion.

Clean up (AWS CLI)

Replace the placeholder values in the following CLI commands below with your actual values:

  • 123456789012 → Your 12-digit AWS account number
  • 7ab63972-b505-7e2a-420d-6f5d3EXAMPLE → Your RAM resource share ID
  • cr-0c54f6734d944345a → Your Capacity Reservation ID
  • a1b2c3d4-5678-90ab-cdef-EXAMPLE → Your SNS subscription ID
  1. Remove the Capacity Block from the resource share 
        aws ram disassociate-resource-share \
        --resource-share-arn arn:aws:ram:us-east-2:123456789012:resource-share/7ab63972-b505-7e2a-420d-6f5d3EXAMPLE \
        --resource-arns arn:aws:ec2:us-east-2:123456789012:capacity-reservation/cr-0c54f6734d944345a
  2. Delete the resource share 
        aws ram delete-resource-share \
        --resource-share-arn arn:aws:ram:us-east-2:123456789012:resource-share/7ab63972-b505-7e2a-420d-6f5d3EXAMPLE
  3. Delete the CloudWatch Alarm 
        aws cloudwatch delete-alarms \
         --alarm-names "CapacityBlock-LowUtilization-cr-123456789"
  4. Delete the SNS Topic and Subscription
    1. List subscriptions to get the subscription ARN 
          aws sns list-subscriptions-by-topic \
         --topic-arn arn:aws:sns:us-east-2:123456789012:capacity-block-alerts
    2. Delete the subscription 
          aws sns unsubscribe \
         --subscription-arn arn:aws:sns:us-east-2:123456789012:capacity-block-alerts:a1b2c3d4-5678-90ab-cdef-EXAMPLE
    3. Delete the Topic 
          aws sns delete-topic \
         --topic-arn arn:aws:sns:us-east-2:123456789012:capacity-block-alerts

Conclusion

In this post, we showed you how to share Capacity Blocks for ML across your AWS Organization using AWS RAM. We covered configuring the AWS RAM integration with Organizations, creating resource shares, and accessing shared Capacity Blocks for ML from consumer accounts. Finally, we showed you how to monitor and alert on low instance utilization.

By sharing Capacity Blocks across your organization, you can reduce idle GPU capacity, eliminate scheduling bottlenecks between teams, and maximize the return on your reserved compute investment. To take this further, consider building dashboards in Amazon CloudWatch to track utilization trends across multiple Capacity Blocks.

You can get started by purchasing Capacity Blocks for ML and sharing it across your organization today. For more details on other resources you can share with AWS RAM, visit the Shareable AWS resources in the user guide. If you have questions, contact your AWS account team or leave a comment below.

Intel Announces Arc Pro B70 and B65 Video Cards: Big Battlemage Brings Big Memory for AI Workstations

Post Syndicated from Ryan Smith original https://www.servethehome.com/intel-announces-arc-pro-b70-and-b65-video-cards-big-battlemage-brings-big-memory-for-ai-workstations/

Today Intel is expanding their Arc B-series video card lineup in a big way, with the launch of a pair of new Arc Pro graphics cards: the Arc Pro B70 and the Arc Pro B65. Joining Intel’s existing Arc Pro B-series video cards, the latest cards out of Intel are also the company’s most powerful […]

The post Intel Announces Arc Pro B70 and B65 Video Cards: Big Battlemage Brings Big Memory for AI Workstations appeared first on ServeTheHome.

NVIDIA GeForce RTX 5090 and the Age of Neural Rendering at Hot Chips 2025

Post Syndicated from Ryan Smith original https://www.servethehome.com/nvidia-geforce-rtx-5090-and-the-age-of-neural-rendering-at-hot-chips-2025/

The second presentation on today’s graphics track comes from NVIDIA. Like AMD, NVIDIA is mid-cycle on its current generation of graphics products, having launched the first of them back in late 2024. As a result, their Hot Chips presentation is more of a recap, with a focus on what the Blackwell architecture offers for graphics […]

The post NVIDIA GeForce RTX 5090 and the Age of Neural Rendering at Hot Chips 2025 appeared first on ServeTheHome.

AMD Takes Aim for Workstation AI Market With Radeon AI Pro R9700

Post Syndicated from Ryan Smith original https://www.servethehome.com/amd-takes-aim-for-workstation-ai-market-with-radeon-ai-pro-r9700/

In what has quickly become a busy week for workstation and server class video cards at this year’s Computex trade show, AMD has become the latest video card vendor to join the fray with the announcement of their flagship Radeon AI PRO R9700 video card. Based on AMD’s newest RDNA 4 GPU architecture, AMD is […]

The post AMD Takes Aim for Workstation AI Market With Radeon AI Pro R9700 appeared first on ServeTheHome.

Bolt Graphics Zeus The New GPU Architecture with up to 2.25TB of Memory and 800GbE

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/bolt-graphics-zeus-the-new-gpu-architecture-with-up-to-2-25tb-of-memory-and-800gbe/

The upcoming Bolt Graphics Zeus GPU architecture offers up to 6x 800GbE links and 2.2TB of memory on a 500W TDP GPU

The post Bolt Graphics Zeus The New GPU Architecture with up to 2.25TB of Memory and 800GbE appeared first on ServeTheHome.

Why Servers Are Using So Much Power TDP Growth Over Time

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/why-servers-are-using-so-much-power-tdp-growth-over-time-supermicro-vertiv-intel-amd-nvidia/

Server CPU and GPU TDPs are rapidly increasing. We chart the increases and go into some of the other aspects driving power in data centers

The post Why Servers Are Using So Much Power TDP Growth Over Time appeared first on ServeTheHome.

What it Takes to Handle Digital Pathology PBs of Storage and GPUs

Post Syndicated from Eric Smith original https://www.servethehome.com/what-it-takes-to-handle-digital-pathology-pbs-of-storage-gpus-quanta-qct/

Running an AI assisted digital pathology workflow takes many petabytes of storage and GPUs to help doctors make better diagnoses

The post What it Takes to Handle Digital Pathology PBs of Storage and GPUs appeared first on ServeTheHome.

Optimizing GPU utilization for AI/ML workloads on Amazon EC2

Post Syndicated from Sheila Busser original https://aws.amazon.com/blogs/compute/optimizing-gpu-utilization-for-ai-ml-workloads-on-amazon-ec2/

­­­­This blog post is written by Ben Minahan, DevOps Consultant, and Amir Sotoodeh, Machine Learning Engineer.

Machine learning workloads can be costly, and artificial intelligence/machine learning (AI/ML) teams can have a difficult time tracking and maintaining efficient resource utilization. ML workloads often utilize GPUs extensively, so typical application performance metrics such as CPU, memory, and disk usage don’t paint the full picture when it comes to system performance. Additionally, data scientists conduct long-running experiments and model training activities on existing compute instances that fit their unique specifications. Forcing these experiments to be run on newly provisioned infrastructure with proper monitoring systems installed might not be a viable option.

In this post, we describe how to track GPU utilization across all of your AI/ML workloads and enable accurate capacity planning without needing teams to use a custom Amazon Machine Image (AMI) or to re-deploy their existing infrastructure. You can use Amazon CloudWatch to track GPU utilization, and leverage AWS Systems Manager Run Command to install and configure the agent across your existing fleet of GPU-enabled instances.

Overview

First, make sure that your existing Amazon Elastic Compute Cloud (Amazon EC2) instances have the Systems Manager Agent installed, and also have the appropriate level of AWS Identity and Access Management (IAM) permissions to run the Amazon CloudWatch Agent. Next, specify the configuration for the CloudWatch Agent in Systems Manager Parameter Store, and then deploy the CloudWatch Agent to our GPU-enabled EC2 instances. Finally, create a CloudWatch Dashboard to analyze GPU utilization.

Architecture Diagram depicting the integration between AWS Systems Manager with RunCommand Arguments stored in SSM Parameter Store, your Amazon GPU enabled EC2 instance with installed Amazon CloudWatch Agen­t, and Amazon CloudWatch Dashboard that aggregates and displays the ­reported metrics.

  1. Install the CloudWatch Agent on your existing GPU-enabled EC2 instances.
  2. Your CloudWatch Agent configuration is stored in Systems Manager Parameter Store.
  3. Systems Manager Documents are used to install and configure the CloudWatch Agent on your EC2 instances.
  4. GPU metrics are published to CloudWatch, which you can then visualize through the CloudWatch Dashboard.

Prerequisites

This post assumes you already have GPU-enabled EC2 workloads running in your AWS account. If the EC2 instance doesn’t have any GPUs, then the custom configuration won’t be applied to the CloudWatch Agent. Instead, the default configuration is used. For those instances, leveraging the CloudWatch Agent’s default configuration is better suited for tracking resource utilization.

For the CloudWatch Agent to collect your instance’s GPU metrics, the proper NVIDIA drivers must be installed on your instance. Several AWS official AMIs including the Deep Learning AMI already have these drivers installed. To see a list of AMIs with the NVIDIA drivers pre-installed, and for full installation instructions for Linux-based instances, see Install NVIDIA drivers on Linux instances.

Additionally, deploying and managing the CloudWatch Agent requires the instances to be running. If your instances are currently stopped, then you must start them to follow the instructions outlined in this post.

Preparing your EC2 instances

You utilize Systems Manager to deploy the CloudWatch Agent, so make sure that your EC2 instances have the Systems Manager Agent installed. Many AWS-provided AMIs already have the Systems Manager Agent installed. For a full list of the AMIs which have the Systems Manager Agent pre-installed, see Amazon Machine Images (AMIs) with SSM Agent preinstalled. If your AMI doesn’t have the Systems Manager Agent installed, see Working with SSM Agent for instructions on installing based on your operating system (OS).

Once installed, the CloudWatch Agent needs certain permissions to accept commands from Systems Manager, read Systems Manager Parameter Store entries, and publish metrics to CloudWatch. These permissions are bundled into the managed IAM policies AmazonEC2RoleforSSM, AmazonSSMReadOnlyAccess, and CloudWatchAgentServerPolicy. To create a new IAM role and associated IAM instance profile with these policies attached, you can run the following AWS Command Line Interface (AWS CLI) commands, replacing <REGION_NAME> with your AWS region, and <INSTANCE_ID> with the EC2 Instance ID that you want to associate with the instance profile:

aws iam create-role --role-name CloudWatch-Agent-Role --assume-role-policy-document  '{"Statement":{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}}'
aws iam attach-role-policy --role-name CloudWatch-Agent-Role --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
aws iam attach-role-policy --role-name CloudWatch-Agent-Role --policy-arn arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess
aws iam attach-role-policy --role-name CloudWatch-Agent-Role --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws iam create-instance-profile --instance-profile-name CloudWatch-Agent-Instance-Profile
aws iam add-role-to-instance-profile --instance-profile-name CloudWatch-Agent-Instance-Profile --role-name CloudWatch-Agent-Role
aws ec2 associate-iam-instance-profile --region <REGION_NAME> --instance-id <INSTANCE_ID> --iam-instance-profile Name=CloudWatch-Agent-Instance-Profile

Alternatively, you can attach the IAM policies to your existing IAM role associated with an existing IAM instance profile.

aws iam attach-role-policy --role-name <ROLE_NAME> --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
aws iam attach-role-policy --role-name <ROLE_NAME> --policy-arn arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess
aws iam attach-role-policy --role-name <ROLE_NAME> --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws ec2 associate-iam-instance-profile --region <REGION_NAME> --instance-id <INSTANCE_ID> --iam-instance-profile Name=<INSTANCE_PROFILE>

Once complete, you should see that your EC2 instance is associated with the appropriate IAM role.

An Amazon EC2 Instance with the CloudWatch-Agent-Role IAM Role attached

This role should have the AmazonEC2RoleforSSM, AmazonSSMReadOnlyAccess and CloudWatchAgentServerPolicy IAM policies attached.

The CloudWatch-Agent-Role IAM Role’s attached permission policies, Amazon EC2 Role for SSM, CloudWatch Agent Server ¬Policy, and Amazon SSM Read Only Access

Configuring and deploying the CloudWatch Agent

Before deploying the CloudWatch Agent onto our EC2 instances, make sure that those agents are properly configured to collect GPU metrics. To do this, you must create a CloudWatch Agent configuration and store it in Systems Manager Parameter Store.

Copy the following into a file cloudwatch-agent-config.json:

{
    "agent": {
        "metrics_collection_interval": 60,
        "run_as_user": "cwagent"
    },
    "metrics": {
        "aggregation_dimensions": [
            [
                "InstanceId"
            ]
        ],
        "append_dimensions": {
            "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
            "ImageId": "${aws:ImageId}",
            "InstanceId": "${aws:InstanceId}",
            "InstanceType": "${aws:InstanceType}"
        },
        "metrics_collected": {
            "cpu": {
                "measurement": [
                    "cpu_usage_idle",
                    "cpu_usage_iowait",
                    "cpu_usage_user",
                    "cpu_usage_system"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ],
                "totalcpu": false
            },
            "disk": {
                "measurement": [
                    "used_percent",
                    "inodes_free"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ]
            },
            "diskio": {
                "measurement": [
                    "io_time"
                ],
                "metrics_collection_interval": 60,
                "resources": [
                    "*"
                ]
            },
            "mem": {
                "measurement": [
                    "mem_used_percent"
                ],
                "metrics_collection_interval": 60
            },
            "swap": {
                "measurement": [
                    "swap_used_percent"
                ],
                "metrics_collection_interval": 60
            },
            "nvidia_gpu": {
                "measurement": [
                    "utilization_gpu",
                    "temperature_gpu",
                    "utilization_memory",
                    "fan_speed",
                    "memory_total",
                    "memory_used",
                    "memory_free",
                    "pcie_link_gen_current",
                    "pcie_link_width_current",
                    "encoder_stats_session_count",
                    "encoder_stats_average_fps",
                    "encoder_stats_average_latency",
                    "clocks_current_graphics",
                    "clocks_current_sm",
                    "clocks_current_memory",
                    "clocks_current_video"
                ],
                "metrics_collection_interval": 60
            }
        }
    }
}

Run the following AWS CLI command to deploy a Systems Manager Parameter CloudWatch-Agent-Config, which contains a minimal agent configuration for GPU metrics collection. Replace <REGION_NAME> with your AWS Region.

aws ssm put-parameter \
--region <REGION_NAME> \
--name CloudWatch-Agent-Config \
--type String \
--value file://cloudwatch-agent-config.json

Now you can see a CloudWatch-Agent-Config parameter in Systems Manager Parameter Store, containing your CloudWatch Agent’s JSON configuration.

CloudWatch-Agent-Config stored in Systems Manager Parameter Store

Next, install the CloudWatch Agent on your EC2 instances. To do this, you can leverage Systems Manager Run Command, specifically the AWS-ConfigureAWSPackage document which automates the CloudWatch Agent installation.

  1. Run the following AWS CLI command, replacing <REGION_NAME> with the Region into which your instances are deployed, and <INSTANCE_ID> with the EC2 Instance ID on which you want to install the CloudWatch Agent.
aws ssm send-command \
--query 'Command.CommandId' \
--region <REGION_NAME> \
--instance-ids <INSTANCE_ID> \
--document-name AWS-ConfigureAWSPackage \
--parameters '{"action":["Install"],"installationType":["In-place update"],"version":["latest"],"name":["AmazonCloudWatchAgent"]}'

2. To monitor the status of your command, use the get-command-invocation AWS CLI command. Replace <COMMAND_ID> with the command ID output from the previous step, <REGION_NAME> with your AWS region, and <INSTANCE_ID> with your EC2 instance ID.

aws ssm get-command-invocation --query Status --region <REGION_NAME> --command-id <COMMAND_ID> --instance-id <INSTANCE_ID>

3.Wait for the command to show the status Success before proceeding.

$ aws ssm send-command \
	 --query 'Command.CommandId' \
    --region us-east-2 \
    --instance-ids i-0123456789abcdef \
    --document-name AWS-ConfigureAWSPackage \
    --parameters '{"action":["Install"],"installationType":["Uninstall and reinstall"],"version":["latest"],"additionalArguments":["{}"],"name":["AmazonCloudWatchAgent"]}'

"5d8419db-9c48-434c-8460-0519640046cf"

$ aws ssm get-command-invocation --query Status --region us-east-2 --command-id 5d8419db-9c48-434c-8460-0519640046cf --instance-id i-0123456789abcdef

"Success"

Repeat this process for all EC2 instances on which you want to install the CloudWatch Agent.

Next, configure the CloudWatch Agent installation. For this, once again leverage Systems Manager Run Command. However, this time the AmazonCloudWatch-ManageAgent document which applies your custom agent configuration is stored in the Systems Manager Parameter Store to your deployed agents.

  1. Run the following AWS CLI command, replacing <REGION_NAME> with the Region into which your instances are deployed, and <INSTANCE_ID> with the EC2 Instance ID on which you want to configure the CloudWatch Agent.
aws ssm send-command \
--query 'Command.CommandId' \
--region <REGION_NAME> \
--instance-ids <INSTANCE_ID> \
--document-name AmazonCloudWatch-ManageAgent \
--parameters '{"action":["configure"],"mode":["ec2"],"optionalConfigurationSource":["ssm"],"optionalConfigurationLocation":["/CloudWatch-Agent-Config"],"optionalRestart":["yes"]}'

2. To monitor the status of your command, utilize the get-command-invocation AWS CLI command. Replace <COMMAND_ID> with the command ID output from the previous step, <REGION_NAME> with your AWS region, and <INSTANCE_ID> with your EC2 instance ID.

aws ssm get-command-invocation --query Status --region <REGION_NAME> --command-id <COMMAND_ID> --instance-id <INSTANCE_ID>

3. Wait for the command to show the status Success before proceeding.

$ aws ssm send-command \
    --query 'Command.CommandId' \
    --region us-east-2 \
    --instance-ids i-0123456789abcdef \
    --document-name AmazonCloudWatch-ManageAgent \
    --parameters '{"action":["configure"],"mode":["ec2"],"optionalConfigurationSource":["ssm"],"optionalConfigurationLocation":["/CloudWatch-Agent-Config"],"optionalRestart":["yes"]}'

"9a4a5c43-0795-4fd3-afed-490873eaca63"

$ aws ssm get-command-invocation --query Status --region us-east-2 --command-id 9a4a5c43-0795-4fd3-afed-490873eaca63 --instance-id i-0123456789abcdef

"Success"

Repeat this process for all EC2 instances on which you want to install the CloudWatch Agent. Once finished, the CloudWatch Agent installation and configuration is complete, and your EC2 instances now report GPU metrics to CloudWatch.

Visualize your instance’s GPU metrics in CloudWatch

Now that your GPU-enabled EC2 Instances are publishing their utilization metrics to CloudWatch, you can visualize and analyze these metrics to better understand your resource utilization patterns.

The GPU metrics collected by the CloudWatch Agent are within the CWAgent namespace. Explore your GPU metrics using the CloudWatch Metrics Explorer, or deploy our provided sample dashboard.

  1. Copy the following into a file, cloudwatch-dashboard.json, replacing instances of <REGION_NAME> with your Region:
{
    "widgets": [
        {
            "height": 10,
            "width": 24,
            "y": 16,
            "x": 0,
            "type": "metric",
            "properties": {
                "metrics": [
                    [{"expression": "SELECT AVG(nvidia_smi_utilization_gpu) FROM SCHEMA(\"CWAgent\", InstanceId) GROUP BY InstanceId","id": "q1"}]
                ],
                "view": "timeSeries",
                "stacked": false,
                "region": "<REGION_NAME>",
                "stat": "Average",
                "period": 300,
                "title": "GPU Core Utilization",
                "yAxis": {
                    "left": {"label": "Percent","max": 100,"min": 0,"showUnits": false}
                }
            }
        },
        {
            "height": 7,
            "width": 8,
            "y": 0,
            "x": 0,
            "type": "metric",
            "properties": {
                "metrics": [
                    [{"expression": "SELECT AVG(nvidia_smi_utilization_gpu) FROM SCHEMA(\"CWAgent\", InstanceId)", "label": "Utilization","id": "q1"}]
                ],
                "view": "gauge",
                "stacked": false,
                "region": "<REGION_NAME>",
                "stat": "Average",
                "period": 300,
                "title": "Average GPU Core Utilization",
                "yAxis": {"left": {"max": 100, "min": 0}
                },
                "liveData": false
            }
        },
        {
            "height": 9,
            "width": 24,
            "y": 7,
            "x": 0,
            "type": "metric",
            "properties": {
                "metrics": [
                    [{ "expression": "SEARCH(' MetricName=\"nvidia_smi_memory_used\" {\"CWAgent\", InstanceId} ', 'Average')", "id": "m1", "visible": false }],
                    [{ "expression": "SEARCH(' MetricName=\"nvidia_smi_memory_total\" {\"CWAgent\", InstanceId} ', 'Average')", "id": "m2", "visible": false }],
                    [{ "expression": "SEARCH(' MetricName=\"mem_used_percent\" {CWAgent, InstanceId} ', 'Average')", "id": "m3", "visible": false }],
                    [{ "expression": "100*AVG(m1)/AVG(m2)", "label": "GPU", "id": "e2", "color": "#17becf" }],
                    [{ "expression": "AVG(m3)", "label": "RAM", "id": "e3" }]
                ],
                "view": "timeSeries",
                "stacked": false,
                "region": "<REGION_NAME>",
                "stat": "Average",
                "period": 300,
                "yAxis": {
                    "left": {"min": 0,"max": 100,"label": "Percent","showUnits": false}
                },
                "title": "Average Memory Utilization"
            }
        },
        {
            "height": 7,
            "width": 8,
            "y": 0,
            "x": 8,
            "type": "metric",
            "properties": {
                "metrics": [
                    [ { "expression": "SEARCH(' MetricName=\"nvidia_smi_memory_used\" {\"CWAgent\", InstanceId} ', 'Average')", "id": "m1", "visible": false } ],
                    [ { "expression": "SEARCH(' MetricName=\"nvidia_smi_memory_total\" {\"CWAgent\", InstanceId} ', 'Average')", "id": "m2", "visible": false } ],
                    [ { "expression": "100*AVG(m1)/AVG(m2)", "label": "Utilization", "id": "e2" } ]
                ],
                "sparkline": true,
                "view": "gauge",
                "region": "<REGION_NAME>",
                "stat": "Average",
                "period": 300,
                "yAxis": {
                    "left": {"min": 0,"max": 100}
                },
                "liveData": false,
                "title": "GPU Memory Utilization"
            }
        }
    ]
}

2. run the following AWS CLI command, replacing <REGION_NAME> with the name of your Region:

aws cloudwatch put-dashboard \
    --region <REGION_NAME> \
    --dashboard-name My-GPU-Usage \
    --dashboard-body file://cloudwatch-dashboard.json

View the My-GPU-Usage CloudWatch dashboard in the CloudWatch console for your AWS region..

An example CloudWatch dashboard, My-GPU-Usage, showing the GPU usage metrics over time.

Cleaning Up

To avoid incurring future costs for resources created by following along in this post, delete the following:

  1. My-GPU-Usage CloudWatch Dashboard
  2. CloudWatch-Agent-Config Systems Manager Parameter
  3. CloudWatch-Agent-Role IAM Role

Conclusion

By following along with this post, you deployed and configured the CloudWatch Agent across your GPU-enabled EC2 instances to track GPU utilization without pausing in-progress experiments and model training. Then, you visualized the GPU utilization of your workloads with a CloudWatch Dashboard to better understand your workload’s GPU usage and make more informed scaling and cost decisions. For other ways that Amazon CloudWatch can improve your organization’s operational insights, see the Amazon CloudWatch documentation.

A 16x NVIDIA GPU 128 Core Arm Server Supermicro ARS-210M-NR with Ampere Altra

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/a-16x-nvidia-gpu-128-core-arm-server-supermicro-ars-210m-nr-with-ampere-altra/

We review the Supermicro ARS-210M-NR with 16x NVIDIA Ampere 16GB GPUs and 128 Ampere Altra Max Arm cores designed for cloud deployments

The post A 16x NVIDIA GPU 128 Core Arm Server Supermicro ARS-210M-NR with Ampere Altra appeared first on ServeTheHome.