Post Syndicated from Mark Nguyen original https://aws.amazon.com/blogs/compute/deploying-external-boot-volumes-with-aws-outposts/

Building on our previous announcement, AWS Outposts third-party storage integration for data volumes, AWS is expanding its collaboration with third-party storage solutions by introducing support for boot volumes backed by external storage arrays. In this post we show you how to boot Amazon Elastic Compute Cloud (Amazon EC2) instances on Outposts directly from NetApp on-premise enterprise array and Pure Storage FlashArray, providing greater flexibility to align your workload needs for storage.

With this enhancement, you can now:

• Launch EC2 instances using boot volumes backed by compatible third-party storage arrays
• Use existing storage management workflows and tools for both boot and data volumes
• Take advantage of advanced features like storage efficiencies and cloning for cost reduction
• Migrate compute of existing workloads that rely on external boot volumes to Outposts
• Employ High Availability (HA) and Disaster Recover (DR) architectures with technologies such as NetApp’s SnapMirror or Pure Storage’s ActiveDR, enabling quick and easy recovery from failures and streamline DR processes.

The boot volume integration is enabled through AWS-provided launch Amazon Machine Images (AMIs) and sample automation scripts. Two methods are supported:

• Storage Area Network boot (SANboot), which uses Pre-boot eXecution Environment (iPXE) and Internet Small Computer System Interface (iSCSI) to attach an external volume when booting.
• Outposts localboot, which hydrates a local Non-Volatile Memory Express (NVMe) volume using iSCSI or NVMe-over-TCP, prior to booting.

The added third-party boot volume support further strengthens the commitment of AWS to providing flexible, enterprise-grade storage options for you while maintaining the automation tenants of the AWS experience.

SANboot solution overview

SANboot is a method of booting an EC2 instance from a block volume stored on a remote storage device. iPXE is an open source boot firmware that allows computers to boot from a network location, and when combined with iSCSI, it enables the start up of an operating system (OS) from a remote iSCSI target. In this case, the iSCSI target is a third-party storage system. The EC2 instance boots from the remote block volume as if it were a local disk. Read/write operations within the OS are written directly to the external storage array.

The instance can be shutdown and the data remains persistent on the third-party storage device. iPXE is stateless by design, thus subsequent boots must go through the full SANboot process. Snapshots of the volumes can be performed by the external storage device.

SANboot prerequisites

The following prerequisites are necessary for SANboot:

1. An Outposts rack or Outposts 2U server. Outposts 1U server is not supported.
2. An external storage system with IP reachability to the Outposts local network through a local gateway (LGW) or local network interface (LNI).
3. An Amazon EC2 bootable image stored as a block volume on the external storage system.
   1. The block volume should be configured with an iSCSI IQN
   2. At the time of this post, only Windows 2022 (or newer) and Red Hat Linux 9 are supported operating systems
4. (optional) Authentication credentials stored in AWS Secrets Manager if authentication is needed.
   1. If Secrets Manager is used, your EC2 instance profile needs the following minimum required permissions:

      {
          "Version": "2012-10-17",
          "Statement": [
              {
                   "Sid": "SecretsManagerPermissions",
                   "Effect": "Allow",
                   "Action": [
                        "secretsmanager:GetSecretValue"
                   ],
                   "Resource": "<secret name>"
              }
          ] 
      }

5. (optional) Use Outposts SANboot script from AWS Samples downloaded onto your management computer. Your computer must be capable of running Python scripts and connected to both the external storage system and the AWS Region.

SANboot deployment overview

The following diagram shows the workflow when launching an EC2 instance using SANboot.

Figure 1 – SANboot workflow

1. The Outposts must have network connectivity to the external storage system through the Outposts local network connection. Specifically, the EC2 instances must communicate to the storage system. This should be a high speed connection with low latency (recommended 10Gbps or more, and under 10ms round trip). The Outposts and storage system do not have to be on the same subnet, instead they just need IP reachability.
   1. For Outposts racks, the EC2 instance connects through the LGW.
   2. For Outposts 2U servers, the EC2 instance connects through the LNI.
2. The boot volume must be an Amazon EC2 bootable image. This includes the appropriate settings and drivers for the respective OS.
3. Run the SANboot automation script and follow the prompts to enter the relevant parameters for your boot volume. This information is used to populate the Multipurpose Internet Mail Extension (MIME) multipart user data that iPXE uses for its initial configuration. See the following section on SANboot script walkthrough for more information.
4. (optional) If iSCSI authentication is enabled, the SANboot automation script retrieves the credentials of the iSCSI target stored in Secrets Manager to be later populated to the iPXE script.
5. The iPXE boot AMI launches its firmware image and executes the iPXE script. This script contains the necessary information to mount the remote iSCSI block volume. This includes the iSCSI intiator settings, the IP address of the iSCSI target, the credentials, the iSCSI port, and the iSCSI IQN. The iPXE boot AMI is a relatively small image that is pulled down from the Region. The time it takes to launch the iPXE image depends on the bandwidth of the Outposts service link connection to the Region.
6. The iPXE firmware establishes the iSCSI connection to the target and boots the OS from the remote storage device over the local network. If applicable, then more user data information can be passed to the booting OS. The time it takes for the OS to boot depends on the speed of the local area network.

SANboot sample script walkthrough

The following diagram shows the workflow for executing the SANboot sample script.

Figure 2 – SANboot automation script walkthrough

Outposts localboot solution overview

Outposts localboot, using iSCSI or NVMe-over-TCP, is a method of booting an EC2 instance using a copy of a boot volume retrieved from a remote storage device. The original source image is not modified and remains unchanged. First, a localboot AMI is launched as a helper instance to facilitate the copying of the boot volume. The localboot instance acts as the iSCSI initiator or NVMe-over-TCP host, and mounts the remote boot volume from the third-party storage system. Then, the block volume is copied to local instance storage on the EC2 instance. The EC2 instance reboots using the local instance storage volume.

Localboot uses instance store volumes, which are ephemeral in nature. The boot volume is deleted when the EC2 instance that it is attached to is terminated or stopped. This is suitable for temporary guest desktop access or read-only workloads. Data volumes from external storage can also be mounted during the localboot process, which can be used to store data persistently.

Localboot prerequisites

The following prerequisites are necessary for localboot:

1. An Outposts rack or Outposts 2U server.
2. An external storage system with IP reachability to the Outposts local network through an LGW or LNI.
3. An Amazon EC2 bootable image stored as a block volume on the external storage system.
   1. The block volume should be configured with an iSCSI IQN or NVMe-over-TCP NQN.
   2. At the time of this post, Red Hat Linux 9 is the only supported operating system.
4. If using Outposts rack, instance types must have local NVMe drives to support instance store volumes. These include the C/M/R “d” type instances such as c5d, m5d, or r7d.
5. Boot volumes on third-party storage systems must fit within the instance store of the instance type chosen. For example, an m5d.xlarge has one 150GB NVMe SSD for the instance store. The boot volume must be under 150GB to fit within an m5d.xlarge instance store. If the boot volume is larger, then a larger instance type must be chosen, such as an m5d.2xlarge (which has a 300GB NVMe SSD).
6. (optional) Authentication credentials are stored in Secrets Manager if authentication is needed.
   1. If Secrets Manager is used, your EC2 instance profile needs the following minimum required permissions:

      {
          "Version": "2012-10-17",
          "Statement": [
              {
                   "Sid": "SecretsManagerPermissions",
                   "Effect": "Allow",
                   "Action": [
                        "secretsmanager:GetSecretValue"
                   ],
                   "Resource": "<secret name>"
              }
          ] 
      }

7. (optional) Outposts localboot script from AWS Samples downloaded onto your management computer. Your computer must be capable of running Python scripts and connected to both the external storage system and the Region.

Localboot deployment overview

The following diagram shows the workflow when launching an EC2 instance using localboot.

Figure 3 – Localboot workflow

Steps 1 and 2 are the same as SANboot deployment, see instructions above. Starting with step 3 for localboot:

1. Run the localboot automation script and follow the prompts to enter the relevant parameters for your boot volume. This information is used to populate the MIME multipart user data that the localboot instance uses to mount the external boot volume. Go to the following localboot script walkthrough. The user data contains the necessary information to mount the remote iSCSI or NVMe-over-TCP block volume.
2. (Optional) The localboot automation script retrieves the credentials of the external storage appliance stored in Secrets Manager.
3. The localboot AMI is downloaded from the Region and launched. This can take a few minutes depending on the speed of the Outposts service link connection to the Region. When booting, the localboot AMI runs the user data script to attach the boot volume from the third-party storage system.
4. The mounted boot volume is then copied to the local instance storage of the EC2 instance and configured as the boot partition. The time it takes to copy depends on the speed of your local network. When the copying is completed, the EC2 instance reboots using the newly configured boot partition.
5. A copy of the boot volume is booted as a new EC2 instance. If applicable, then more user data information (such as mounting a data volume) can be passed to the booting OS.

Outposts localboot sample script walkthrough

The following diagram shows the workflow for running the localboot sample script.

Figure 4 – Localboot automation script walkthrough

Conclusion

The introduction of third-party boot volume support allows you to continue getting value from your existing investments in external shared storage by connecting these arrays with Amazon EC2 on Outposts. Both the SANboot and localboot methods allow you to integrate your existing storage solutions with AWS Outposts, enabling them to boot EC2 instances directly from NetApp on-premise enterprise storage arrays and Pure Storage FlashArray volumes. This capability not only enhances storage options for EC2 instances running on Outposts but also streamlines workload migration and offers advanced features such as storage efficiencies for boot volumes. Automated configuration scripts and AWS-provided helper AMIs allow you to maintain the seamless and automated experience expected from AWS, while also giving you the flexibility to use your preferred enterprise storage solutions. Combined with the previously announced support for data volumes, this development further demonstrates the commitment of AWS to offering flexible solutions that can integrate into your existing hybrid environments.

To speak to an Outposts expert and learn more, submit this form.

To learn more about how NetApp and Pure Storage solutions work with Outposts, read the NetApp on-premises storage arrays for AWS Outposts blog post and the Pure Storage FlashArray for AWS Outposts blog post.

Post Syndicated from Chris McEvilly original https://aws.amazon.com/blogs/compute/running-and-optimizing-small-language-models-on-premises-and-at-the-edge/

As you move your generative AI implementations from prototype to production, you may discover the need to run foundation models (FMs) on-premises or at the edge to address data residency, information security (InfoSec) policy, or low latency requirements. For example, customers in regulated industries, such as financial services, healthcare, and telecom, may want to leverage chatbots that support customer queries, optimize internal workflows for complex reporting, and automatically approve requests—while keeping their data in-country. Similarly, some organizations choose to deploy their own small language models (SLMs) to align with stringent internal InfoSec requirements. As another example, manufacturers may want to deploy SLMs right inside their factories to analyze production data and provide real-time equipment diagnostics. To address users’ data residency, latency, and InfoSec needs, this post provides guidance on deploying generative AI FMs into AWS Local Zones and AWS Outposts. The goal is to present a framework for running a range of SLMs that address local data processing requirements-based customer engagements.

Generative AI deployment options

The growth in generative AI deployments and experimentation has accelerated with two business deployment options. The first is the use of a large language model (LLM) to support enterprise needs. LLMs are incredibly flexible: one model can perform completely different tasks, such as answering questions, coding, summarizing documents, translating languages, and generating content. LLMs have the potential to disrupt content creation and the way people use search engines and virtual assistants. The second deployment model is the use of SLMs, addressing a specific use case. SLMs are compact transformer models that primarily use decoder-only or encoder-decoder architectures, with typically less than 20 billion parameters, though this definition is evolving as larger models become available. SLMs can achieve comparable or even superior performance when fine-tuned for specific domains or tasks, making them an excellent alternative for specialized applications.

In addition, SLMs offer faster inference times, lower resource requirements, and are suitable for deployment on a wider range of devices, making them particularly valuable for specialized applications and edge computing where space and power supply are limited. While SLMs are limited in scope and accuracy compared to LLMs, you can enhance their performance for specific tasks through Retrieval Augmented Generation (RAG) and fine-tuning. This combination produces an SLM capable of responding to queries relating to a specific domain with a comparable level of accuracy to an LLM, while reducing hallucinations. SLMs offer effective solutions that balance user needs with cost efficiency.

Architecture overview

The solution discussed in this blog post uses Llama.cpp, an optimized framework written in C/C++ to efficiently run a range of SLMs. Llama.cpp can run efficiently in a wide range of computing environments, enabling generative AI models to operate in Local Zones or on Outposts without the large Graphics Processing Unit (GPU) clusters typically associated with LLMs in their native frameworks. This framework expands your model choice and increases model performance when deploying SLMs to Local Zones and Outposts.

The architecture provides a template for deploying a range of SLMs supporting chatbot or generation use cases. The solution consists of a front-end application receiving user queries, formatting the prompts for presentation to the model, and delivering the resulting responses. To support a scalable solution, the application servers and Amazon EC2 G4dn GPU-enabled instances are front-ended with an Application Load Balancer (ALB).

For a deployment where the incoming prompts exceed the ability of SLMs to service the requests, a message queue can be deployed to the front end of the SLMs. As an example, you could deploy a RabbitMQ cluster to serve as your queue manager.

Figure 1: Architecture overview

Solution deployment

The following instructions show how to launch an SLM using Llama.cpp in Local Zones or on Outposts. Although the preceding architecture overview shows a complete solution with multiple components, this post focuses specifically on the steps necessary to deploy the SLM in an EC2 instance using Llama.cpp.

Prerequisites

For this solution, you need the following prerequisites:

1. An AWS account that is either allowlisted for Local Zones or has a logical Outpost installed, configured, and running
2. Access to G4dn instances in your account at the chosen location (check availability in AWS Service Quotas)
3. A VPC created to host the environment
4. Public and private subnets to support the environment in the VPC
5. A security group that is associated with your EC2 instance
6. AWS Identity and Access Management (IAM) role with AWS Systems Manager Session Manager permissions

1. Launch a GPU instance for your SLM

Sign in to the AWS Management Console, open the Amazon EC2 console, and launch a g4dn.12xlarge EC2 instance in your Local Zone or Outposts environment. Configure it with:

1. Red Hat Enterprise Linux 9 (HVM), SSD Volume Type
2. A private subnet associated with your Local Zone or Outposts rack
3. 30 GiB gp2 root volume and an additional 300 GiB gp2 EBS volume
4. The IAM role configured with permissions required for Systems Manager
5. SSM Agent to connect to the instance (follow the instructions in Install SSM Agent on RHEL 8.x and 9.x in the Systems Manager User Guide)

For detailed EC2 launch instructions, see Launch an EC2 instance using the launch instance wizard in the console or Launch an instance on your Outposts rack.

Figure 2: SLM instance launched

2. Install NVIDIA drivers

1. Connect to the SLM instance using Systems Manager. You can follow the instructions in Connect to your Amazon EC2 instance using Session Manager in the Amazon EC2 User Guide.
2. Install kernel packages and tools.

   sudo su –
   
   dnf update -y
   
   subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms
   
   dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
   
   dnf install -y ccache cmake gcc-c++ git git-lfs htop python3-pip unzip wget
   
   dnf install -y dkms elfutils-libelf-devel kernel-devel kernel-modules-extra \
   libglvnd-devel vulkan-devel xorg-x11-server-Xorg
   
   systemctl enable --now dkms
   
   reboot
   

3. Install Miniconda3 in the /opt/miniconda3 folder or install a compatible package manager to manage Python dependencies.
4. Install the NVIDIA drivers.

   dnf config-manager --add-repo \
   http://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
   
   dnf module install -y nvidia-driver:latest-dkms
   
   dnf install -y cuda-toolkit
   
   echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
   
   echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
   
   source ~/.bashrc

3. Download and install Llama.cpp

1. Create and mount the filesystem of the Amazon EBS volume you created earlier in the /opt/slm folder. The instructions are available in Make an Amazon EBS volume available for use in the Amazon EBS User Guide.
2. Run the following commands to download and install Llama.cpp.

   cd /opt/slm
   
   git clone -b b4942 \
   https://github.com/ggerganov/llama.cpp.git
   
   cd llama.cpp
   
   cmake -B build -DGGML_CUDA=ON
   
   cmake --build build --config Release -j$(nproc)
   
   conda install python=3.12
   
   pip install -r requirements.txt
   
   pip install nvitop

4. Download and convert the SLM model

To run the SLM efficiently with Llama.cpp, you need to convert the model format to GGUF (GPT-Generated Unified Format). This conversion optimizes the model for better performance and memory usage on edge deployments with limited resources. GGUF is specifically designed to work with Llama.cpp’s inference engine.The following instructions show how to download SmolLM2 1.7B and convert it to the GGUF format:

mkdir /opt/slm/models
cd /opt/slm/models

git lfs install
git clone https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct

cd /opt/slm/llama.cpp

python3 convert_hf_to_gguf.py --outtype f16 \
--outfile /opt/slm/llama.cpp/models/SmolLM2-1.7B-Instruct-f16.gguf \
/opt/slm/models/SmolLM2-1.7B-Instruct/

echo 'export PATH=/opt/slm/llama.cpp/build/bin:$PATH' >> ~/.bashrc

echo 'export LD_LIBRARY_PATH=/opt/slm/llama.cpp/build/bin:$LD_LIBRARY_PATH' \
>> ~/.bashrc

source ~/.bashrc

You can also download other publicly available models from Hugging Face as needed, following the same conversion process.

SLM operation and optimization

The deployment of SLMs through Llama.cpp provides operational flexibility through a significant range of environment customization options, allowing you to tailor model behavior to your specific use cases. With Llama.cpp, you can select a range of parameters that can be used to optimize the use of system resources and model operation. This enables models to efficiently use the available resources without consuming unneeded resources or negatively impacting their operation. The following parameters are commonly supported when running Llama.cpp to provide control over how the model runs.

• -ngl N, --n-gpu-layers N: When compiled with GPU support, this option allows you to offload some layers to the GPU for computation. This generally results in increased performance.
• -t N, --threads N: Sets the number of threads to use during generation. For optimal performance, it is recommended to set this value to the number of physical CPU cores that your system has.
• -n N, --n-predict N: Sets the number of tokens to predict when generating text. Adjusting this value can influence the length of the generated text.
• -sm, --split-mode: Specifies how to split the model across multiple GPUs when running in a multi-GPU environment. Consider testing the ‘row’ splitting mode as it may provide better performance in some scenarios compared to the default layer-based splitting.
• --temp N: Temperature is a parameter that provides control over an SLM’s randomness in terms of its outputs. Changing the value from the default of 0.8 changes the distribution of the model’s predictions. Lower values sharpen and deterministically enhance the model’s output, preferring high-probability phrases. On the other hand, a higher temperature fosters more inventiveness and unpredictability, allowing for a wider range of diverse responses. (default: 0.8).
• -s SEED, --seed SEED: Provides a method to control the randomness of the SLM’s response. Setting a specific seed value produces consistent responses across multiple runs, while using the default value (-1) generates varied responses. (default: -1, -1 = random seed).
• -c, --ctx-size N: Context size is defined as the number of tokens an FM can process as one prompt (in tokens) into the model. This value reserves memory in the host for the maximum size context length. Reducing the context size can reduce the amount of RAM needed to load and run a model. Models with larger context sizes (>50k) consume more RAM, and in some models, a larger context size can impact accuracy. For example, with Phi-3, it’s recommended to reduce the context size to 8k or 16k. The command to achieve this is –ctx-size XXXX, where XXXX is the context size.

This section demonstrates how to optimize SLM performance for specific use cases using Llama.cpp. We’ll cover two common scenarios: chatbot interactions and text summarization.

Chatbot Use Case Example

Token Size Requirements

For chatbot applications, typical token size requirements include input sizes ranging from 50-150 tokens, which support user questions up to a couple of sentences, and output sizes of approximately 100-300 tokens, allowing the model to provide detailed but concise responses.

Sample Command

./build/bin/llama-cli -m ./models/SmolLM2-1.7B-Instruct-f16.gguf \
-ngl 99 -n 512 --ctx-size 8192 -sm row --temp 0 --single-turn \
-sys "You are a helpful assistant" -p "Hello"

Figure 3: Chatbot example

Command Explanation

• -m ./models/SmolLM2-1.7B-Instruct-f16.gguf: Specifies the model file to use
• -ngl 99: Allocates GPU layers for optimal performance
• -n 512: Sets maximum output tokens to 512, sufficient for the 100-300 tokens typically needed
• --ctx-size 8192: Defines context window size, allowing for complex conversation history
• -sm row: Splits the rows across GPUs
• --temp 0: Sets the temperature to zero to be less creative
• --single-turn: Optimizes for single-response interactions
• -sys "You are a helpful assistant": Sets the system prompt to define assistant behavior
• -p "Hello": Provides the user input prompt

Text summarization example

This command line shows SmolLM2-1.7B running to support a summarization activity.

PROMPT_TEXT="Summarize the following text: Amazon DynamoDB is a serverless, 
NoSQL database service that allows you to develop modern applications 
at any scale. As a serverless database, you only pay for what you use 
and DynamoDB scales to zero, has no cold starts, no version upgrades, 
no maintenance windows, no patching, and no downtime maintenance. 
DynamoDB offers a broad set of security controls and compliance 
standards. For globally distributed applications, DynamoDB global 
tables is a multi-Region, multi-active database with a 99.999% 
availability SLA and increased resilience. DynamoDB reliability is 
supported with managed backups, point-in-time recovery, and more. 
With DynamoDB streams, you can build serverless event-driven applications."

./build/bin/llama-cli -m ./models/SmolLM2-1.7B-Instruct-f16.gguf \
-ngl 99 -n 512 --ctx-size 8192 -sm row --single-turn \
-sys "You are a technical writer" \
--prompt "$PROMPT_TEXT"

Figure 4: Summarization example

Cleaning up

To avoid ongoing charges, take these steps to delete the resources you created by following this post, if they are no longer needed.

• Terminate your EC2 instance to avoid unnecessary charges. Furthermore, verify that the 300 GiB EBS volume you attached is properly deleted by checking the Volumes section under Elastic Block Store and manually delete any remaining volumes by choosing them and choosing Actions > Delete volume.

Conclusion

This post took you through the steps of deploying SLMs into AWS on-premises or edge environments to address local data processing needs. The post first discussed business benefits of SLMs, including faster inference time, reduced operational costs, and improved model outcomes. SLMs launched using Llama.cpp and optimized for specific use cases can effectively deliver user services from the edge in a scalable manner. The optimization parameters described in this post provide multiple ways to configure these models for different deployment use cases. You can follow the deployment steps and techniques in this blog post to implement generative AI capabilities that align with your specific data residency, latency, or InfoSec compliance needs while operating efficiently within the resource constraints of edge environments. Visit the following pages to learn more about AWS Local Zones and AWS Outposts.

Post Syndicated from Brianna Rosentrater original https://aws.amazon.com/blogs/compute/control-instance-placement-using-asset-level-capacity-management-for-aws-outposts/

AWS Outposts supports self-service capacity management at the entire Outpost level, or at the individual asset level, making it easy for you to view and manage compute capacity on your Outposts. This feature supports both Outposts rack (such as the recently announced second-generation Outposts rack) and Outposts server. A default capacity configuration for each new Outpost is determined during the ordering process. This default configuration can subsequently be modified to create a range of Amazon Elastic Compute Cloud (Amazon EC2) instance sizes and quantities to meet your changing business needs. For more information on performing Outposts level multi-asset reconfigurations, go to Dynamically reconfigure your AWS Outposts capacity using Capacity Tasks.

The release of Asset Level Capacity Management allows you to control the configuration of specific assets within your Outpost, which can be useful when planning strategies for EC2 Auto Scaling groups and host-level high availability. An Outpost asset can be a single server within an Outposts rack, or an Outposts server. This post focuses on how to use Asset Level Capacity Management to perform single-host reconfigurations, and how this can be used with Amazon EC2 placement groups to control instance placement on your Outpost.

Overview

When you place an Outposts order, you determine the capacity configuration of each Outpost based on the anticipated workload requirements. You can scale your Outposts up or out as needed during your commitment term. For further details on Outpost capacity planning including best practices, refer to the Capacity Planning – AWS Outposts High Availability Design and Architecture whitepaper. We recommend planning spare capacity for N+M host availability per instance family when making modifications to your Outpost capacity configuration for workloads that need to be highly available. To calculate, take the number of assets (N) you need to run all your workloads, and then add (M) additional assets to meet your requirements for server availability during failure and maintenance events.

You also need to plan for instance level high availability when deciding to reconfigure particular assets. For example, say you have two C5 assets, and each one is configured homogeneously to provide C5.2xlarge instances. If you have an Auto Scaling group that specifies C5.2xlarge in its launch template, and you perform an asset level reconfiguration of one of your C5 assets so that it only offers C5.4xlarge instances, then your Auto Scaling group can only launch instances on the one C5 host configured to provide C5.2xlarge instances. If that host fails, then the Auto Scaling group is unable to launch new C5.2xlarge instances on the other host unless the Auto Scaling group launch template is modified. Understanding failure scenario behavior and how much capacity you want to reserve for high availability is key to capacity management and disaster recovery planning. For highly available workloads, we recommend spreading your instances across as many assets as possible.

Understanding EC2 placement groups on AWS Outposts

Outposts rack supports EC2 placement groups, and two placement group options are available only on Outposts: rack level spread, and host level spread. This allows you to spread out instances across underlying hardware on an Outpost at your site. To use a rack level spread placement group, you must have two or more physical Outpost racks. Each spread strategy can be used to create resilient Outposts architectures that can withstand a rack or host failure depending on the respective strategy used.

Rack level spread

Figure 1: Outposts rack showing a rack level spread EC2 placement group

Using a multi-rack Outpost, you can spread your EC2 instances across multiple racks with a rack level spread EC2 placement group. When used with Auto Scaling groups, this allows you to withstand an individual rack or multi-asset failure. When your Auto Scaling group detects you’ve lost instances on one of your racks, it automatically relaunches the instances using the assets on your other racks if you have available capacity. To use this strategy to increase your workload resiliency, each rack would need to have assets that can support the instance type (C5 is used in the preceding figure) and size used in your Auto Scaling group launch template. The expanded functionality that asset level capacity management brings to capacity tasks allows you to configure your Outpost so that each rack has at least one asset that can support the instances used in your Auto Scaling groups. Configure your assets on each rack to meet your resiliency goals for host failure tolerance as well. This configuration can be done in an on-demand, self-service fashion to meet the needs of your evolving workloads if instance requirements change over time.

Host level spread

Figure 2: Outposts rack showing a host level spread EC2 placement group

Although rack level spread EC2 placement groups need a multi-rack Outpost, host level spread EC2 placement groups can be used within a single rack Outpost to provide resiliency for your workloads at the asset level. When used with Auto Scaling groups, this allows you to withstand an individual asset or multi-asset failure depending on your Outpost configuration. When your Auto Scaling group detects you’ve lost instances on one of your assets due to a hardware failure, it automatically relaunches the instances using your other assets on your Outpost if you have available capacity. To use this strategy to increase your workload resiliency, you would need to have at least two assets within your Outposts rack that can support the instance type (R5 and M5 are used in the preceding figure) and size used in your Auto Scaling group launch template. Outposts also supports using attribute-based instance type selection if multiple instance types meet your workload needs based on some minimum resource requirements. With the expanded functionality that asset level capacity management brings to capacity tasks, you can configure your Outposts rack so that each asset type can support the instance size used in your Auto Scaling groups. This configuration can be done in an on-demand, self-service fashion to meet the needs of your evolving workloads if instance requirements change over time.

Using asset level capacity tasks

Asset Level Capacity Management allows you to target a specific Outpost asset to change its capacity configuration directly, allowing granular control over instance capacity pool configurations. Outpost assets are referred to by a unique ten-digit Asset ID. The first step in this process is identifying a suitable asset on which to perform the capacity task. To do this, you can use the rack view within the Outposts console page to view each asset, its current capacity configuration, and its current usage. Choosing an asset with fewer running instances may increase the chances of the capacity task being successful without needing instances to be stopped.

In the following example, the rack view has been filtered by the R5 family resulting in the two R5 assets being displayed. The Show instance details option has also been chosen to show the instance IDs of the running instances on our Outposts rack.

Figure 3: Rack view of the Outposts console

When you have identified the asset to target for the capacity task, you can either choose the Modify option in the top right of the asset itself or go to Capacity Tasks from the console menu and choose the asset ID directly from the dropdown menu.

Figure 4: Capacity tasks console experience

From here, you have the option to use the capacity configuration builder to interactively modify your Outposts capacity layout, or you can upload a capacity configuration plan JSON document with the necessary configuration. When building the capacity task, you have two options to choose from when handling instances that are blocking the task from executing. The default option is set to fail the capacity task if this occurs. However, this can be set to wait for the instances to be stopped so that the task can continue. If this option is chosen, then the asset is placed into an isolated state until either the capacity task completes or is cancelled, thus preventing any further instances launches on the impacted asset.

If there are instances on the asset that can’t be stopped to complete the capacity task, then they can be chosen from the Instances to keep as-is section. Only the instances running on the impacted asset are listed. If a capacity task can’t be completed while leaving the chosen instances running, the capacity task fails.

In the following example, the capacity configuration requested for the asset results in the removal of one r5.4xlarge and two r5.2xlarge instances, which creates sufficient space for the creation of 12 r5.large instances. This asset also has three instances running on it which have all been chosen to keep as-is during the execution of the task.

Figure 5: Capacity task example showing r5 asset level capacity management

You can also execute capacity tasks programmatically If you prefer through CLI or API calls. For example, using the start-capacity-task CLI to submit the same configuration would look as follows:

aws outposts start-capacity-task \
--outpost-id op-07f6f537e0607d3f1 \
--asset-id 1702928095\
--instances-to-exclude '{
    "Instances": ["i- 03f53189ffedcc72c", "i-044383b9051299b50", "i-0dfd88574237a68a4"],
    "AccountIds": ["450360193046", "450360193046", "450360193046"],
    "Services": ["EC2", "EC2", "EC2"]
}' \
--task-action-on-blocking-instances FAIL_TASK \
--instance-pools '[
    {
        "InstanceType": "r5.large",
        "Count": 12
    },
    {
        "InstanceType": "r5.xlarge",
        "Count": 6
    },
    {
        "InstanceType": "r5.2xlarge",
        "Count": 4
    },
    {
        "InstanceType": "r5.4xlarge",
        "Count": 1
    }
]'

After defining the capacity task, you are presented with an overview of the requested changes before submitting the task for execution. When it’s submitted, the task first enters a Requested status while the configuration is evaluated, before either being moved to In Progress if the task is valid or Failed if it’s invalid or blocked by running instances.

When the capacity task has successfully completed and the capacity pools for the asset are updated, you can validate this by returning to the rack view within the Outpost console, or by using the CLI/API. The following is an example using the list-assets CLI command:

aws outposts list-assets --outpost-identifier op-07f6f537e0607d3f1 --query "Assets[?AssetId=='1702928095']"

[
    {
        "AssetId": " 1702928095",
        "RackId": "1702928115",
        "AssetType": "COMPUTE",
        "ComputeAttributes": {
            "State": "ACTIVE",
            "InstanceFamilies": [
                "R5"
            ],
            "InstanceTypeCapacities": [
                {
                    "InstanceType": "r5.2xlarge",
                    "Count": 4
                },
                {
                    "InstanceType": "r5.4xlarge",
                    "Count": 1
                },
                {
                    "InstanceType": "r5.xlarge",
                    "Count": 6
                },
                {
                    "InstanceType": "r5.large",
                    "Count": 12
                }
            ],
            "MaxVcpus": 96
        },
        "AssetLocation": {
            "RackElevation": 27.0
        }
    }
]

Only a single capacity task for an asset can be executing at any given time. If you attempt to create a second capacity task for the same asset while the original is still in a Requesting or In Progress status, then the submission of the task fails. However, you can submit multiple capacity tasks for unique assets within the same Outpost. For example, using the CLI commands, you could execute a single script to change the capacity configuration of all assets within an Outpost through individual asset level capacity tasks.

Considerations

• Make sure that if you’re specifying instance type in your launch templates, then this instance type is available on multiple assets if your workload needs to be resilient against host failures.
• Understand which failure scenarios could exist within your environment, and plan for how each one should be handled. Failure planning is essential for maintaining workload uptime in production environments.
• Capacity tasks can only be executed from the AWS account that owns the Outpost. If Outpost resources are shared to workload accounts through AWS Resource Access Manager (AWS RAM), then these accounts can’t submit capacity tasks.
• You can manipulate your capacity configuration to control instance placement at launch. If only certain assets support the instance size and type you want to deploy, then your instance must be launched on one of those assets.
• If executing capacity tasks through CLI commands, make sure that your CLI has been updated to the latest version. We have updated our CLI with this feature release to include commands for capacity tasks, and they fail if running on outdated versions.

Conclusion

This post demonstrates how to use Asset Level Capacity Management with your AWS Outposts, and reviews considerations for maintaining a highly available capacity configuration. For more information on how to manage and monitor your capacity configuration on Outposts, see the Capacity management for AWS Outposts user guide and the Capacity planning section of the Outposts High Availability Design and Architecture Considerations whitepaper. Reach out to your AWS account team, or fill out this form to learn more about Outposts and self-service capacity management.