Post Syndicated from Matheus Guimaraes original https://aws.amazon.com/blogs/aws/connect-users-to-data-through-your-apps-with-storage-browser-for-amazon-s3/

Today, we’re introducing Storage Browser for Amazon S3, an open source UI component you can add to your web applications to enable end users to interact with your data stored in Amazon Simple Storage Service (Amazon S3). With this frontend component, authorized end users can browse, upload, download, copy, and delete data from Amazon S3 based on their specific permissions, which you control using AWS identity and security services or custom managed solutions.

Storage Browser for S3 eases the strain on developers looking to provide end users with access to data in S3, and it is designed so that end users, such as customers, partners, and employees, can efficiently work with data regardless of their familiarity with Amazon S3 or Amazon Web Services. Additionally, developers can customize the look and feel of the Storage Browser interface to align with their application’s design.

Let’s walk through a quick demo to show how you can get started.

Installation
Storage Browser for S3 is an AWS Amplify UI React component, therefore, you must use it in a web application built with React or a React-based framework such as Next.Js, Gatsby, Remix, or any others. You also must have both AWS Amplify and the AWS Amplify UI React packages installed.

This demo uses Next.js. If you want to learn how to set up an app from scratch, check out this step-by-step guide on configuring AWS Amplify and using the Amplify React UI components with a new Next.js application.

You don’t need to install the entire @aws-amplify/ui-react library to use Storage Browser for S3.You can install only the storage-specific package with the following command if that is all you intend to use.

npm i @aws-amplify/ui-react-storage aws-amplify

If you have an existing application that already has the Amplify UI React package installed, make sure to update your dependencies to import the latest version, and run npm install to update any existing installations.

Lastly, if you’re building an application from scratch, make sure to run npm create amplify@latest in your application’s directory so you’re able to use the various categories provided by Amplify like auth, storage, and others.

Choosing an authorization mode
Storage Browser for S3 requires authentication and authorization to be configured so it can render the S3 buckets or prefixes that end users can access as well as the actions they can perform.

There are three options for setting up permissions, each suitable for different use cases:

Using AWS Amplify Auth – This option is ideal when you want to provide your customers and third-party partners access to your data in Amazon S3. You can set up Amplify Storage which uses AWS Amplify Auth by default to manage access control and security for files. This is powered by Amazon Cognito and comes with pre-built UI components for implementing user registration, sign-in, and sign-out flows.

Using AWS IAM Identity Center – This option is ideal for a scalable solution providing your whole workforce with access to your data in S3 through Storage Browser for S3 . You associate an S3 Access Grants instance with your AWS Identity and Access Management (IAM) Identity Center to centrally manage S3 Access Grants permissions for your users and groups, including those hosted on external identity providers such as Microsoft Entra ID, Okta, and others. Additionally, each AWS CloudTrail data event for S3 references the end-user identity that accessed your data which helps to increase the observability for your data access.

Using IAM roles with Amazon S3 Access Grants – This option is ideal when you want to provide IAM principals with access to your data through Storage Browser for S3. To set this up, you must first create an S3 Access Grants instance that you can use to map permissions for S3 buckets and prefixes to the desired IAM identities. Then you create an IAM role that has permissions to invoke s3:GetDataAccess to get temporary least-privilege access to S3 buckets or prefixes.

This demo assumes the end users are not part of our organization so Amplify Auth is a great match for this case.

Setting up permissions
First, you must set up Amplify Storage by following this guide. Then, open amplify/storage/resource.ts to declare an S3 bucket alongside the desired access rules following the Amplify authorization model which utilizes prefixes to configure isolated storage for authorized users.

Next, create a component called StorageBrowser which encapsulates the integration with Amplify Auth and that we can easily drop in a page later. Make sure to call Amplify.config() to stitch it all together with a a reference to amplify_outputs.json as a parameter.

Visit the S3 User Guide for detailed instructions for setting up authentication and authorization for Storage Browser for S3.

Adding Storage Browser for S3 to my application
Now that the component is created, you just need to add it to your application in a page where you want to render it by declaring <StorageBrowser/>.

Use npm run dev to run the application. After it loads, navigate to the page where you added Storage Browser For S3 and you should see it loaded with the default layout. Notice also that it is configured with the same paths and permissions that we defined in amplify/storage/resource.ts above allowing users to browse, read, write, and delete files inside the S3 buckets and prefixes that we have set up.

You can download files and browse folders while accessing management operations from the sub-menu which automatically greys out any unavailable actions.

Storage Browser for S3 automatically pages results and makes it possible to filter and search for files and folders, making it easy to navigate and manage data.

All data access is governed by the configured authorization model enabling end users to seamlessly interact with S3 buckets and prefixes through a highly intuitive interface without compromising your security or compliance requirements.

Customizing the interface
Thanks to its flexible design, you can customize Storage Browser For S3 to match the look and feel of your application. Much like any other Amplify UI components it will use the Amplify theme you have active in your application by default. However, you can easily modify any of its components such as the buttons, breadcrumb, the paging controls, text fields, and others, by creating your own theme or targeting elements directly using CSS.

To create a theme, first you must declare it using the defineComponentTheme() function from the @aws-amplify/ui-react/server library. You give it a name such as 'storage-browser' and then target the elements that you want to style.

You can even rearrange the layout as well if you want. In the code you can see that we are setting the flexDirection of all controls to 'row-reverse', for example.

Then you create the theme using the createTheme() function using the storage-browser theme we declared earlier and apply it. We also override the primaryColor and make it green.

After the page is reloaded, you should see the Storage Browser for S3 component with its new more compact layout and new color scheme with green text.

You can customize essentially any element of the UI interface including any of the display texts such as the title where it says Home, or any others. The only exceptions are the details about the data, of course, such as the bucket names and keys. You can take advantage of this to add support for different languages, for example.

Finally, if you prefer to create your own UI from scratch, you call the createStorageBrowser() function to create a Storage Browser for S3 component programatically. It returns a useView() hook that you can use to integrate with your own custom frontend, giving you full control over the look and feel while leveraging all of the same features. To learn more, see the documentation for more details on the various customization options and how to configure them.

Conclusion
Storage Browser for S3 is a highly customizable and user-friendly AWS Amplify UI React component which enables end users to interact with data on Amazon S3 securely. It gives you full control of the access rules to ensure the frontend complies with your access requirements while providing a great user experience through an interface that you can style to make it appear as a natural extension of your application.

Things to know

Getting started – You can install Storage Browser for S3 from the GitHub page. For more information on getting started, visit the UI documentation.

Compatibility – Storage Browser for S3 is compatible with all Amazon S3 storage classes except for Glacier Flexible Retrieval and S3 Glacier Deep Archive. It is compatible with S3 Intelligent-Tiering, but it’s not compatible with the S3 Intelligent-Tiering Archive Access Tier or the S3 Intelligent-Tiering Deep Archive Access Tier..

Performance and durability – Storage Browser for S3 includes built-in logic that enhances upload requests for high-throughput data transfer, calculates checksums of uploaded data (rejecting requests that fail these durability checks), and optimizes performance for faster load times in your application.

Pricing – Storage Browser for S3 is open source and you can integrate it with your applications at no extra cost. You only pay for your use of the underlying AWS resources you use with Storage Browser for S3.

Support – Storage Browser for S3 is backed by AWS Support just like any other feature of S3. Customers with Business and Enterprise Support plans get 24/7 access to AWS Support engineers to support their use of Storage Browser for S3.

Feedback – We invite you to share feedback on the functionality and the public roadmap for Storage Browser for S3.

Matheus Guimaraes | @codingmatheus

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/

At Amazon Web Services (AWS), the vast majority of new capabilities are driven by your direct feedback. Two years ago, Jeff announced additional checksum algorithms and the optional client-side computation of checksums to make sure the objects stored on Amazon S3 are exactly what you sent. You told us you love this extra verification because it gives you confidence the object stored is the one you sent. You also told us you would prefer to have this extra verification enabled automatically, freeing you from developing additional code.

Starting today, we’re updating the Amazon Simple Storage Service (Amazon S3) default behavior when you upload objects. To build upon its existing durability posture, Amazon S3 now automatically verifies that your data is correctly transmitted over the network from your applications to your S3 bucket.

Amazon S3 is designed for 99.999999999% data durability (that’s 11 nines). Amazon S3 has always verified the integrity of object uploads by calculating checksums when objects reach our servers, before they are written to multiple storage devices. Once your data is stored in Amazon S3, it continually monitors data durability over time with periodic integrity checks of data at rest. Amazon S3 also actively monitors the redundancy of your data to help verify that your objects can tolerate the concurrent failure of multiple storage devices.

But data can still face integrity risks as it traverses the public internet before reaching our servers. Issues such as faulty hardware on networks we don’t manage or client software bugs could potentially corrupt or drop data before Amazon S3 has a chance to validate it. Previously, you could extend the integrity protection by providing your own precomputed checksums with your PutObject or UploadPart requests. However, this requires configuring tools and applications to generate and track checksums, which can be complex to implement consistently across all your client applications uploading objects to Amazon S3.

The new default behavior builds upon existing data integrity protections without requiring any changes to your applications. Additionally, the new checksums are stored in the object’s metadata, making them accessible for integrity verification at any time.

Automatic client-side integrity protection
Amazon S3 now extends data integrity protection all the way to client-side applications by default. The latest versions of our AWS SDKs automatically calculate a cyclic redundancy check (CRC)-based checksum for each upload and send it to Amazon S3. Amazon S3 independently calculates a checksum on the server side and validates it against the provided value before durably storing the object and its checksum in the object’s metadata.

When your client application doesn’t send a CRC checksum (maybe it uses an old version of our SDK or you haven’t updated your application custom code yet), Amazon S3 computes a CRC-based checksum anyway and stores it in the object metadata for future reference. You can compare at a later stage the stored CRC with a CRC computed on your side and verify the network transmission was correct.

This new capability provides you with an automatic checksum calculation and validation for new uploads from the latest versions of the AWS SDKs, the AWS Command Line Interface (AWS CLI), and the AWS Management Console. You can also verify the checksum stored in the object’s metadata at any time. The new default data integrity protections use the existing CRC32 and CRC32C algorithms or the new CRC64NVME algorithm. Amazon S3 also provides developers with consistent full-object checksums across single-part and multipart uploads.

When uploading files in multiple parts, the SDKs calculate checksums for each part. Amazon S3 uses these checksums to verify the integrity of each part through the UploadPart API. Additionally, S3 validates the entire file’s size and checksum when you call the CompleteMultipartUpload API.

The CreateMultiPartUpload API introduces a new HTTP header, x-amz-checksum-type, which lets you specify the type of checksum to use. You can choose either a full object checksum (calculated by combining the checksums of all individual parts) or a composite checksum.

The full object checksum is stored with the object metadata for future reference. This new protection works seamlessly with server-side encryption. The consistent behavior across uploads, multipart uploads, downloads, and encryption modes simplifies client-side integrity checks. The ability to use full-object checksums to validate integrity and store them for use later can help you streamline your applications.

Let’s see it in action
To start using this additional integrity protection, update to the latest version of the AWS SDK or AWS CLI. No code changes are required to enable the new integrity protections.

Case 1: Amazon S3 now attaches a checksum to objects on the server side when objects are uploaded without a checksum

I wrote a simple Python script to upload and download content to and from an S3 bucket. I enabled maximum logging verbosity to see the actual HTTP headers sent to and from Amazon S3.

import boto3
import logging

BUCKET_NAME="aws-news-blog-20241111"
CONTENT='Hello World!'
OBJECT_NAME='test.txt'

# Enable debug logging for boto3 and botocore to stdout (this is verbose !!!)
logging.basicConfig(level=logging.DEBUG)

# create a s3 client
client = boto3.client('s3')

# put an object
client.put_object(Bucket=BUCKET_NAME, Key=OBJECT_NAME, Body=CONTENT)

# get the object 
response = client.get_object(Bucket=BUCKET_NAME, Key=OBJECT_NAME)
print(response['Body'].read().decode('utf-8'))

In the first step of this demo, I use an old AWS SDK for Python that doesn’t compute the CRC checksum on the client side. Despite this, I can observe that Amazon S3 now responds with a checksum it computed upon receiving the object.

S3 RESPONSE:
{
    ...
    "x-amz-checksum-crc64nvme": "AuUcyF784aU=",
    "x-amz-checksum-type": "FULL_OBJECT",
    ...
}

Case 2: Upload with manually pre-computed CRC64NVME checksum, a new checksum type

When I don’t have the option to use the latest version of the AWS SDK, or when I use my own code to upload objects to S3 buckets, I can compute the checksum and send it in the PutObject API request. Here is how I compute the checksum on my content before sending it to Amazon S3. To keep this code short, I use the checksums package available in the new AWS SDK for Python.

from awscrt import checksums
import base64

checksum = checksums.crc64nvme("Hello World!")
checksum_bytes = checksum.to_bytes(8, byteorder='big')  # CRC64 is 8 bytes
checksum_base64 = base64.b64encode(checksum_bytes)
print(checksum_base64)

And when I run it, I see the CRC64NVME checksum is the same as the one returned by Amazon S3 in the previous step.

$ python crc.py
b'AuUcyF784aU='

I can provide this checksum as part of the PutObject API call.

response = s3.put_object(
    Bucket=BUCKET_NAME,
    Key=OBJECT_NAME,
    Body=b'Hello World!',
    ChecksumAlgorithm='CRC64NVME', 
    ChecksumCRC64NVME=checksum_base64
)

Case 3: The new SDKs compute the checksum on the client-side

Now, I run the upload and download script again. This time, I use the latest version of the AWS SDK for Python. I observe that the SDK now sends the CRC headers in the request. The response also contains the checksum. I can easily compare the versions in the request and in the response to make sure the object received is the one I sent.

REQUEST:
{
    ...
    "x-amz-checksum-crc64nvme": "AuUcyF784aU=",
    "x-amz-checksum-type": "FULL_OBJECT",
    ... 
}

At any time, I can request the object checksum to verify the integrity of my local copy using the HeadObject or GetObject APIs.

 get_response = s3.get_object(
        Bucket=BUCKET_NAME,
        Key=OBJECT_NAME,
        ChecksumMode='ENABLED'
    )

The response object contains the checksum in the HTTPHeaders field.

{
...
    "x-amz-checksum-crc64nvme": "AuUcyF784aU=",
    "x-amz-checksum-type": "FULL_OBJECT",
...
}

Case 4: Multi-part uploads with new CRC-based whole-object checksum

When uploading large objects using the CreateMultipartUpload, UploadPart, and CompleteMultipartUpload APIs, the latest version of the SDK will automatically compute the checksums for you.

If you want to validate the integrity of your data by using a known content checksum, you can pre-compute the CRC-based whole-object checksum for multi-part uploads to simplify your client side tooling. When using full object checksums for multi-part uploads, you no longer have to keep track of part level checksums as you upload objects.

# precomputed CRC64NVME checksum for the full object
full_object_crc64_nvme_checksum = 'Naz0uXkYBPM='

# start multipart upload
create_response = s3.create_multipart_upload(
            Bucket=BUCKET_NAME,
            Key=OBJECT_NAME,
            ChecksumAlgorithm='CRC64NVME',
            ChecksumType='FULL_OBJECT'
        )
upload_id = create_response['UploadId']

# Upload parts
uploaded_parts = []

# part 1
data_part_1 = b'0' * (5 * 1024 * 1024) # minimum part size
upload_part_response_1 = s3.upload_part(
    Body=data_part_1,
    Bucket=BUCKET_NAME,
    Key=OBJECT_NAME,
    PartNumber=1,
    UploadId=upload_id,
    ChecksumAlgorithm='CRC64NVME'
)
uploaded_parts.append({'PartNumber': 1, 'ETag': upload_part_response_1['ETag']})

# part 2
data_part_2 = b'0' * (5 * 1024 * 1024)
upload_part_response_2 = s3.upload_part(
    Body=data_part_2,
    Bucket=BUCKET_NAME,
    Key=OBJECT_NAME,
    PartNumber=2,
    UploadId=upload_id,
    ChecksumAlgorithm='CRC64NVME'
)
uploaded_parts.append({'PartNumber': 2, 'ETag': upload_part_response_2['ETag']})

# Complete the multipart upload with the FULL_OBJECT CRC64NVME checksum to validate the integrity of your entire object. 
complete_response = s3.complete_multipart_upload(
            Bucket=BUCKET_NAME,
            Key=OBJECT_NAME,
            UploadId=upload_id,
            ChecksumCRC64NVME=full_object_crc64_nvme_checksum,
            ChecksumType='FULL_OBJECT',
            MultipartUpload={'Parts': uploaded_parts}
        )
print(complete_response)

Things to know
For your existing objects, the checksum will be added when you copy them. We updated the CopyObject API so you can choose the desired checksum algorithm for the destination object.

This new client-side checksum calculation is implemented in the latest version of the AWS SDKs. When you use an old SDK or custom code that doesn’t pre-compute checksums, Amazon S3 computes the checksum on all new objects it receives and stores it in the object’s metadata, even for multipart uploads.

Pricing and availability
This extended checksum computation and storage is available in all AWS Regions at no additional cost.

Update your AWS SDK and AWS CLI today to automatically benefit from this additional integrity protection for data in transit.

To learn more about data integrity protection on Amazon S3, visit Checking object integrity in the Amazon S3 User Guide.

— seb