Enhance query performance using AWS Glue Data Catalog column-level statistics

Post Syndicated from Sandeep Adwankar original https://aws.amazon.com/blogs/big-data/enhance-query-performance-using-aws-glue-data-catalog-column-level-statistics/

Today, we’re making available a new capability of AWS Glue Data Catalog that allows generating column-level statistics for AWS Glue tables. These statistics are now integrated with the cost-based optimizers (CBO) of Amazon Athena and Amazon Redshift Spectrum, resulting in improved query performance and potential cost savings.

Data lakes are designed for storing vast amounts of raw, unstructured, or semi-structured data at a low cost, and organizations share those datasets across multiple departments and teams. The queries on these large datasets read vast amounts of data and can perform complex join operations on multiple datasets. When talking with our customers, we learned that one the challenging aspect of data lake performance is how to optimize these analytics queries to execute faster.

The data lake performance optimization is especially important for queries with multiple joins and that is where cost-based optimizers helps the most. In order for CBO to work, column statistics need to be collected and updated based on changes in the data. We’re launching capability of generating column-level statistics such as number of distinct, number of nulls, max, and min on files such as Parquet, ORC, JSON, Amazon ION, CSV, XML on AWS Glue tables. With this launch, customers now have integrated end-to-end experience where statistics on Glue tables are collected and stored in the AWS Glue Catalog, and made available to analytics services for improved query planning and execution.

Using these statistics, cost-based optimizers improves query run plans and boosts the performance of queries run in Amazon Athena and Amazon Redshift Spectrum. For example, CBO can use column statistics such as number of distinct values and number of nulls to improve row prediction. Row prediction is the number of rows from a table that will be returned by a certain step during the query planning stage. The more accurate the row predictions are, the more efficient query execution steps are. This leads to faster query execution and potentially reduced cost. Some of the specific optimizations that CBO can employ include join reordering and push-down of aggregations based on the statistics available for each table and column.

For customers using data mesh with AWS Lake Formation permissions, tables from different data producers are cataloged in the centralized governance accounts. As they generate statistics on tables on centralized catalog and share those tables with consumers, queries on those tables in consumer accounts will see query performance improvements automatically. In this post, we’ll demonstrate the capability of AWS Glue Data Catalog to generate column statistics for our sample tables.

Solution overview

To demonstrate the effectiveness of this capability, we employ the industry-standard TPC-DS 3 TB dataset stored in an Amazon Simple Storage Service (Amazon S3) public bucket. We’ll compare the query performance before and after generating column statistics for the tables, by running queries in Amazon Athena and Amazon Redshift Spectrum. We are providing queries that we used in this post and we encourage to try out your own queries following workflow as illustrated in the following details.

The workflow consists of the following high level steps:

  1. Cataloging the Amazon S3 Bucket: Utilize AWS Glue Crawler to crawl the designated Amazon S3 bucket, extracting metadata, and seamlessly storing it in the AWS Glue data catalog. We’ll query these tables using Amazon Athena and Amazon Redshift Spectrum.
  2. Generating column statistics: Employ the enhanced capabilities of AWS Glue Data Catalog to generate comprehensive column statistics for the crawled data, thereby providing valuable insights into the dataset.
  3. Querying with Amazon Athena and Amazon Redshift Spectrum: Evaluate the impact of column statistics on query performance by utilizing Amazon Athena and Amazon Redshift Spectrum to execute queries on the dataset.

The following diagram illustrates the solution architecture.

Walkthrough

To implement the solution, we complete the following steps:

  1. Set up resources with AWS CloudFormation.
  2. Run AWS Glue Crawler on Public Amazon S3 bucket to list the 3TB TPC-DS dataset.
  3. Run queries on Amazon Athena and Amazon Redshift and note down query duration
  4. Generate statistics for AWS Glue Data Catalog tables
  5. Run queries on Amazon Athena and Amazon Redshift and compare query duration with previous run
  6. Optional: Schedule AWS Glue column statistics jobs using AWS Lambda and the Amazon EventBridge Scheduler

Set up resources with AWS CloudFormation

This post includes an AWS CloudFormation template for a quick setup. You can review and customize it to suit your needs. The template generates the following resources:

  • An Amazon Virtual Private Cloud (Amazon VPC), public subnet, private subnets and route tables.
  • An Amazon Redshift Serverless workgroup and namespace.
  • An AWS Glue crawler to crawl the public Amazon S3 bucket and create a table for the Glue Data Catalog for TPC-DS dataset
  • AWS Glue catalog databases and tables
  • An Amazon S3 bucket to store athena result.
  • AWS Identity and Access Management (AWS IAM) users and policies.
  • AWS Lambda and Amazon Event Bridge scheduler to schedule the AWS Glue Column statistics

To launch the AWS CloudFormation stack, complete the following steps:

Note: The AWS Glue data catalog tables are generated using the public bucket s3://blogpost-sparkoneks-us-east-1/blog/BLOG_TPCDS-TEST-3T-partitioned/, hosted in the us-east-1 region. If you intend to deploy this AWS CloudFormation template in a different region, it is necessary to either copy the data to the corresponding region or share the data within your deployed region for it to be accessible from Amazon Redshift.

  1. Log in to the AWS Management Console as AWS Identity and Access Management (AWS IAM) administrator.
  2. Choose Launch Stack to deploy a AWS CloudFormation template.
  3. Choose Next.
  4. On the next page, keep all the option as default or make appropriate changes based on your requirement choose Next.
  5. Review the details on the final page and select I acknowledge that AWS CloudFormation might create IAM resources.
  6. Choose Create.

This stack can take around 10 minutes to complete, after which you can view the deployed stack on the AWS CloudFormation console.

Run the AWS Glue Crawlers created by the AWS CloudFormation stack

To run your crawlers, complete the following steps:

  1. On the AWS Glue console to AWS Glue Console, choose Crawlers under Data Catalog in the navigation pane.
  2. Locate and run two crawlers tpcdsdb-without-stats and tpcdsdb-with-stats. It may take few mins to complete.

Once the crawler completes successfully, it would create two identical databases tpcdsdbnostats and tpcdsdbwithstats. The tables in tpcdsdbnostats will have No Stats and we’ll use them as reference. We generate statistics on tables in tpcdsdbwithstats. Please verify that you have those two databases and underlying tables from the AWS Glue Console. The tpcdsdbnostats database will look like below. At this time there are no statistics generated on these tables.

Run provided query using Amazon Athena on no-stats tables

To run your query in Amazon Athena on tables without statistics, complete the following steps:

  1. Download the athena queries from here.
  2. On the Amazon Athena Console, choose the provided query one at a time for tables in database tpcdsdbnostats.
  3. Run the query and note down the Run time for each query.

Run provided query using Amazon Redshift Spectrum on no-stats tables

To run your query in Amazon Redshift, complete the following steps:

  1. Download the Amazon Redshift queries from here.
  2. On the Redshift query editor v2, execute the Redshift Query for tables without stats section from downloaded query.
  3. Run the query and note down the query execution of each query.

Generate statistics on AWS Glue Catalog tables

To generate statistics on AWS Glue Catalog tables, complete the following steps:

  1. Navigate to the AWS Glue Console and choose the databases under Data Catalog.
  2. Click on tpcdsdbwithstats database and it will list all the available tables.
  3. Select any of these tables (e.g., call_center).
  4. Go to Column statistics – new tab and choose Generate statistics.
  5. Keep the default option. Under Choose columns keep Table (All columns) and Under Row sampling options Keep All rows, Under IAM role choose AWSGluestats-blog and select Generate statistics.

You’ll be able to see status of the statistics generation run as shown in the following illustration:

After generate statistics on AWS Glue Catalog tables, you should be able to see detailed column statistics for that table:

Reiterate steps 2–5 to generate statistics for all necessary tables, such as catalog_sales, catalog_returns, warehouse, item, date_dim, store_sales, customer, customer_address, web_sales, time_dim, ship_mode, web_site, web_returns. Alternatively, you can follow the “Schedule AWS Glue Statistics Runs” section near the end of this blog to generate statistics for all tables. Once done, assess query performance for each query.

Run provided query using Athena Console on stats tables

  1. On the Amazon Athena console, execute the Athena Query for tables with stats section from downloaded query.
  2. Run and note down the query execution of each query.

In our sample run of the queries on the tables, we observed the query execution time as per the below table. We saw clear improvement in the query performance, ranging from 13 to 55%.

Athena query time improvement

TPC-DS 3T Queries without glue stats (sec) with glue stats (sec) performance improvement (%)
Query 2 33.62 15.17 55%
Query 4 132.11 72.94 45%
Query 14 134.77 91.48 32%
Query 28 55.99 39.36 30%
Query 38 29.32 25.58 13%

Run the provided query using Amazon Redshift Spectrum on statistics tables

  1. On the Amazon Redshift query editor v2, execute the Redshift Query for tables with stats section from downloaded query.
  2. Run the query and note down the query execution of each query.

In our sample run of the queries on the tables, we observed the query execution time as per the below table. We saw clear improvement in the query performance, ranging from 13 to 89%.

Amazon Redshift Spectrum query time improvement

TPC-DS 3T Queries without glue stats (sec) with glue stats (sec) performance improvement (%)
Query 40 124.156 13.12 89%
Query 60 29.52 16.97 42%
Query 66 18.914 16.39 13%
Query 95 308.806 200 35%
Query 99 20.064 16 20%

Schedule AWS Glue statistics Runs

In this segment of the post, we’ll guide you through the steps of scheduling AWS Glue column statistics runs using AWS Lambda and the Amazon EventBridge Scheduler. To streamline this process, a AWS Lambda function and an Amazon EventBridge scheduler were created as part of the CloudFormation stack deployment.

  1. AWS Lambda function setup:

To begin, we utilize an AWS Lambda function to trigger the execution of the AWS Glue column statistics job. The AWS Lambda function invokes the start_column_statistics_task_run API through the boto3 (AWS SDK for Python) library. This sets the groundwork for automating the column statistics update.

Let’s explore the AWS Lambda function:

    • Go to the AWS Glue Lambda Console.
    • Select Functions and locate the GlueTableStatisticsFunctionv1.
    • For a clearer understanding of the AWS Lambda function, we recommend reviewing the code in the Code section and examining the environment variables under Configuration.
  1. Amazon EventBridge scheduler configuration

The next step involves scheduling the AWS Lambda function invocation using the Amazon EventBridge Scheduler. The scheduler is configured to trigger the AWS Lambda function daily at a specific time – in this case, 08:00 PM. This ensures that the AWS Glue column statistics job runs on a regular and predictable basis.

Now, let’s explore how you can update the schedule:

Cleaning up

To avoid unwanted charges to your AWS account, delete the AWS resources:

  1. Sign into the AWS CloudFormation console as the AWS IAM administrator used for creating the AWS CloudFormation stack.
  2. Delete the AWS CloudFormation stack you created.

Conclusion

In this post, we showed you how you can use AWS Glue Data Catalog to generate column-level statistics for AWS Glue tables. These statistics are now integrated with cost-based optimizer from Amazon Athena and Amazon Redshift Spectrum, resulting in improved query performance and potential costs savings. Refer to Docs for support for Glue Catalog Statistics across various AWS analytical services.

If you have questions or suggestions, submit them in the comments section.

About the Authors

Sandeep Adwankar is a Senior Technical Product Manager at AWS. Based in the California Bay Area, he works with customers around the globe to translate business and technical requirements into products that enable customers to improve how they manage, secure, and access data.

Navnit Shukla serves as an AWS Specialist Solution Architect with a focus on Analytics. He possesses a strong enthusiasm for assisting clients in discovering valuable insights from their data. Through his expertise, he constructs innovative solutions that empower businesses to arrive at informed, data-driven choices. Notably, Navnit Shukla is the accomplished author of the book titled Data Wrangling on AWS. He can be reached via LinkedIn.

Happy Thanksgiving

Post Syndicated from corbet original https://lwn.net/Articles/952354/

November 23 is the US Thanksgiving holiday; as is our tradition, we will
not be publishing an LWN Weekly Edition this week as we will be far too
busy eating. We wish a good holiday to all of our readers (whether they
celebrate it or not); the weekly edition will return on November 30.

[$] Committing to Rust for kernel code

Post Syndicated from corbet original https://lwn.net/Articles/952029/

Rust has been a prominent topic at the Kernel Maintainers Summit for the
last couple of years, and the 2023 meeting continued that tradition. As
Rust-for-Linux developer Miguel Ojeda noted at the beginning of the session
dedicated to the topic, the level of interest in using Rust for kernel
development has increased significantly over the last year. But Rust was
explicitly added to Linux as an experiment; is the kernel community now
ready to say that the experiment has succeeded?

Security updates for Wednesday

Post Syndicated from corbet original https://lwn.net/Articles/952312/

Security updates have been issued by Debian (gimp), Fedora (audiofile and firefox), Mageia (postgresql), Red Hat (binutils, c-ares, fence-agents, glibc, kernel, kernel-rt, kpatch-patch, libcap, libqb, linux-firmware, ncurses, pixman, python-setuptools, samba, and tigervnc), Slackware (kernel and mozilla), SUSE (apache2-mod_jk, avahi, container-suseconnect, java-1_8_0-openjdk, libxml2, openssl-1_0_0, openssl-1_1, openvswitch, python3-setuptools, strongswan, ucode-intel, and util-linux), and Ubuntu (frr, gnutls28, hibagent, linux, linux-aws, linux-aws-5.15, linux-hwe-5.15, linux-ibm,
linux-ibm-5.15, linux-kvm, linux-lowlatency, linux-lowlatency-hwe-5.15,
linux-nvidia, linux-oracle, linux-oracle-5.15, linux-raspi, linux, linux-aws, linux-aws-5.4, linux-bluefield, linux-hwe-5.4,
linux-ibm, linux-ibm-5.4, linux-iot, linux-kvm, linux-oracle,
linux-oracle-5.4, linux-raspi, linux-raspi-5.4, linux-xilinx-zynqmp, linux, linux-aws, linux-aws-6.2, linux-hwe-6.2, linux-kvm,
linux-lowlatency, linux-lowlatency-hwe-6.2, linux-raspi, linux-starfive, linux, linux-aws, linux-aws-hwe, linux-hwe, linux-kvm, linux-oracle, linux, linux-aws, linux-laptop, linux-lowlatency, linux-oem-6.5,
linux-oracle, linux-raspi, linux-starfive, linux-oem-6.1, mosquitto, rabbitmq-server, squid, and tracker-miners).

Introducing Apache Hudi support with AWS Glue crawlers

Post Syndicated from Noritaka Sekiyama original https://aws.amazon.com/blogs/big-data/introducing-apache-hudi-support-with-aws-glue-crawlers/

Apache Hudi is an open table format that brings database and data warehouse capabilities to data lakes. Apache Hudi helps data engineers manage complex challenges, such as managing continuously evolving datasets with transactions while maintaining query performance. Data engineers use Apache Hudi for streaming workloads as well as to create efficient incremental data pipelines. Hudi provides tables, transactions, efficient upserts and deletes, advanced indexes, streaming ingestion services, data clustering and compaction optimizations, and concurrency control, all while keeping your data in open source file formats. Hudi’s advanced performance optimizations make analytical workloads faster with any of the popular query engines including Apache Spark, Presto, Trino, Hive, and so on.

Many AWS customers adopted Apache Hudi on their data lakes built on top of Amazon S3 using AWS Glue, a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development. AWS Glue Crawler is a component of AWS Glue, which allows you to create table metadata from data content automatically without requiring manual definition of the metadata.

AWS Glue crawlers now support Apache Hudi tables, simplifying the adoption of AWS Glue Data Catalog as the catalog for Hudi tables. One typical use case is to register Hudi tables, which does not have catalog table definition. Another typical use case is migration from other Hudi catalogs, such as Hive metastore. When migrating from other Hudi Catalogs, you can create and schedule an AWS Glue crawler and provide one or more Amazon S3 paths where the Hudi table files are located. You have the option to provide the maximum depth of Amazon S3 paths that the AWS Glue crawler can traverse. With each run, AWS Glue crawlers will extract schema and partition information and update AWS Glue Data Catalog with the schema and partition changes. AWS Glue crawlers updates the latest metadata file location in the AWS Glue Data Catalog that AWS analytical engines can directly use.

With this launch, you can create and schedule an AWS Glue crawler to register Hudi tables in AWS Glue Data Catalog. You can then provide one or multiple Amazon S3 paths where the Hudi tables are located. You have the option to provide the maximum depth of Amazon S3 paths that crawlers can traverse. With each crawler run, the crawler inspects each of the S3 paths and catalogs the schema information, such as new tables, deletes, and updates to schemas in the AWS Glue Data Catalog. Crawlers inspect partition information and add newly added partitions to AWS Glue Data Catalog. Crawlers also update the latest metadata file location in the AWS Glue Data Catalog that AWS analytical engines can directly use.

This post demonstrates how this new capability to crawl Hudi tables works.

How AWS Glue crawler works with Hudi tables

Hudi tables have two categories, with specific implications for each:

  • Copy on write (CoW) – Data is stored in a columnar format (Parquet), and each update creates a new version of files during a write.
  • Merge on read (MoR) – Data is stored using a combination of columnar (Parquet) and row-based (Avro) formats. Updates are logged to row-based delta files and are compacted as needed to create new versions of the columnar files.

With CoW datasets, each time there is an update to a record, the file that contains the record is rewritten with the updated values. With a MoR dataset, each time there is an update, Hudi writes only the row for the changed record. MoR is better suited for write- or change-heavy workloads with fewer reads. CoW is better suited for read-heavy workloads on data that change less frequently.

Hudi provides three query types for accessing the data:

  • Snapshot queries – Queries that see the latest snapshot of the table as of a given commit or compaction action. For MoR tables, snapshot queries expose the most recent state of the table by merging the base and delta files of the latest file slice at the time of the query.
  • Incremental queries – Queries only see new data written to the table, since a given commit or compaction. This effectively provides change streams to enable incremental data pipelines.
  • Read optimized queries – For MoR tables, queries see the latest data compacted. For CoW tables, queries see the latest data committed.

For copy-on-write tables, crawlers create a single table in the AWS Glue Data Catalog with the ReadOptimized Serde  org.apache.hudi.hadoop.HoodieParquetInputFormat.

For merge-on-read tables, crawlers create two tables in AWS Glue Data Catalog for the same table location:

  • A table with suffix _ro, which uses the ReadOptimized Serde org.apache.hudi.hadoop.HoodieParquetInputFormat
  • A table with suffix _rt, which uses the RealTime Serde allowing for Snapshot queries: org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat

During each crawl, for each Hudi path provided, crawlers make an Amazon S3 list API call, filter based on the .hoodie folders, and find the most recent metadata file under that Hudi table metadata folder.

Crawl a Hudi CoW table using AWS Glue crawler

In this section, let’s go through how to crawl a Hudi CoW using AWS Glue crawlers.

Prerequisites

Here are the prerequisites for this tutorial:

  1. Install and configure AWS Command Line Interface (AWS CLI).
  2. Create your S3 bucket if you do not have it.
  3. Create your IAM role for AWS Glue if you do not have it. You need s3:GetObject for s3://your_s3_bucket/data/sample_hudi_cow_table/.
  4. Run the following command to copy the sample Hudi table into your S3 bucket. (Replace your_s3_bucket with your S3 bucket name.)
$ aws s3 sync s3://aws-bigdata-blog/artifacts/hudi-crawler/product_cow/ s3://your_s3_bucket/data/sample_hudi_cow_table/

This instruction guides you to copy sample data, but you can create any Hudi tables easily using AWS Glue. Learn more in Introducing native support for Apache Hudi, Delta Lake, and Apache Iceberg on AWS Glue for Apache Spark, Part 2: AWS Glue Studio Visual Editor.

Create a Hudi crawler

In this instruction, create the crawler through the console. Complete the following steps to create a Hudi crawler:

  1. On the AWS Glue console, choose Crawlers.
  2. Choose Create crawler.
  3. For Name, enter hudi_cow_crawler. Choose Next.
  4. Under Data source configuration,  choose Add data source.
    1. For Data source, choose Hudi.
    2. For Include hudi table paths, enter s3://your_s3_bucket/data/sample_hudi_cow_table/. (Replace your_s3_bucket with your S3 bucket name.)
    3. Choose Add Hudi data source.
  5. Choose Next.
  6. For Existing IAM role, choose your IAM role, then choose Next.
  7. For Target database, choose Add database, then the Add database dialog appears. For Database name, enter hudi_crawler_blog, then choose Create. Choose Next.
  8. Choose Create crawler.

Now a new Hudi crawler has been successfully created. The crawler can be triggered to run through the console or through the SDK or AWS CLI using the StartCrawl API. It could also be scheduled through the console to trigger the crawlers at specific times. In this instruction, run the crawler through the console.

  1. Choose Run crawler.
  2. Wait for the crawler to complete.

After the crawler has run, you can see the Hudi table definition in the AWS Glue console:

You have successfully crawled the Hudi CoR table with data on Amazon S3 and created an AWS Glue Data Catalog table with the schema populated. After you create the table definition on AWS Glue Data Catalog, AWS analytics services such as Amazon Athena are able to query the Hudi table.

Complete the following steps to start queries on Athena:

  1. Open the Amazon Athena console.
  2. Run the following query.
SELECT * FROM "hudi_crawler_blog"."sample_hudi_cow_table" limit 10;

The following screenshot shows our output:

Crawl a Hudi MoR table using AWS Glue crawler with AWS Lake Formation data permissions

In this section, let’s go through how to crawl a Hudi MoR table using AWS Glue. This time, you use AWS Lake Formation data permission for crawling Amazon S3 data sources instead of IAM and Amazon S3 permission. This is optional, but it simplifies permission configurations when your data lake is managed by AWS Lake Formation permissions.

Prerequisites

Here are the prerequisites for this tutorial:

  1. Install and configure AWS Command Line Interface (AWS CLI).
  2. Create your S3 bucket if you do not have it.
  3. Create your IAM role for AWS Glue if you do not have it. You need lakeformation:GetDataAccess. But you do not need s3:GetObject for s3://your_s3_bucket/data/sample_hudi_mor_table/ because we use Lake Formation data permission to access the files.
  4. Run the following command to copy the sample Hudi table into your S3 bucket. (Replace your_s3_bucket with your S3 bucket name.)
$ aws s3 sync s3://aws-bigdata-blog/artifacts/hudi-crawler/product_mor/ s3://your_s3_bucket/data/sample_hudi_mor_table/

In addition to the processing steps, complete the following steps to update the AWS Glue Data Catalog settings to use Lake Formation permissions to control catalog resources instead of IAM-based access control:

  1. Sign in to the Lake Formation console as a data lake administrator.
    1. If this is the first time accessing the Lake Formation console, add yourself as the data lake administrator.
  2. Under Administration, choose Data catalog settings.
  3. For Default permissions for newly created databases and tables, deselect Use only IAM access control for new databases and Use only IAM access control for new tables in new databases.
  4. For Cross account version setting, choose Version 3.
  5. Choose Save.

The next step is to register your S3 bucket in Lake Formation data lake locations:

  1. On the Lake Formation console, choose Data lake locations, and choose Register location.
  2. For Amazon S3 path, enter s3://your_s3_bucket/. (Replace your_s3_bucket with your S3 bucket name.)
  3. Choose Register location.

Then, grant Glue crawler role access to data location so that the crawler can use Lake Formation permission to access the data and create tables in the location:

  1. On the Lake Formation console, choose Data locations and choose Grant.
  2. For IAM users and roles, select the IAM role you used for the crawler.
  3. For Storage location, enter s3://your_s3_bucket/data/. (Replace your_s3_bucket with your S3 bucket name.)
  4. Choose Grant.

Then, grant crawler role to create tables under the database hudi_crawler_blog:

  1. On the Lake Formation console, choose Data lake permissions.
  2. Choose Grant.
  3. For Principals, choose IAM users and roles, and choose the crawler role.
  4. For LF tags or catalog resources, choose Named data catalog resources.
  5. For Database, choose the database hudi_crawler_blog.
  6. Under Database permissions, select Create table.
  7. Choose Grant.

Create a Hudi crawler with Lake Formation data permissions

Complete the following steps to create a Hudi crawler:

  1. On the AWS Glue console, choose Crawlers.
  2. Choose Create crawler.
  3. For Name, enter hudi_mor_crawler. Choose Next.
  4. Under Data source configuration,  choose Add data source.
    1. For Data source, choose Hudi.
    2. For Include hudi table paths, enter s3://your_s3_bucket/data/sample_hudi_mor_table/. (Replace your_s3_bucket with your S3 bucket name.)
    3. Choose Add Hudi data source.
  5. Choose Next.
  6. For Existing IAM role, choose your IAM role.
  7. Under Lake Formation configuration – optional, select Use Lake Formation credentials for crawling S3 data source.
  8. Choose Next.
  9. For Target database, choose hudi_crawler_blog. Choose Next.
  10. Choose Create crawler.

Now a new Hudi crawler has been successfully created. The crawler uses Lake Formation credentials for crawling Amazon S3 files. Let’s run the new crawler:

  1. Choose Run crawler.
  2. Wait for the crawler to complete.

After the crawler has run, you can see two tables of the Hudi table definition in the AWS Glue console:

  • sample_hudi_mor_table_ro (read optimized table)
  • sample_hudi_mor_table_rt (real time table)

You registered the data lake bucket with Lake Formation and enabled crawling access to the data lake using Lake Formation permissions. You have successfully crawled the Hudi MoR table with data on Amazon S3 and created an AWS Glue Data Catalog table with the schema populated. After you create the table definitions on AWS Glue Data Catalog, AWS analytics services such as Amazon Athena are able to query the Hudi table.

Complete the following steps to start queries on Athena:

  1. Open the Amazon Athena console.
  2. Run the following query. 
    SELECT * FROM "hudi_crawler_blog"."sample_hudi_mor_table_rt" limit 10;

The following screenshot shows our output:

  1. Run the following query. 
    SELECT * FROM "hudi_crawler_blog"."sample_hudi_mor_table_ro" limit 10;

The following screenshot shows our output:

Fine-grained access control using AWS Lake Formation permissions

To apply fine-grained access control on the Hudi table, you can benefit from AWS Lake Formation permissions. Lake Formation permissions allow you to restrict access to specific tables, columns, or rows and then query the Hudi tables through Amazon Athena with fine-grained access control. Let’s configure Lake Formation permission for the Hudi MoR table.

Prerequisites

Here are the prerequisites for this tutorial:

  1. Complete the previous section Crawl a Hudi MoR table using AWS Glue crawler with AWS Lake Formation data permissions.
  2. Create an IAM user DataAnalyst, who has AWS managed policy AmazonAthenaFullAccess.

Create a Lake Formation data cell filter

Let’s first set up a filter for the MoR read optimized table.

  1. Sign in to the Lake Formation console as a data lake administrator.
  2. Choose Data filters.
  3. Choose Create new filter.
  4. For Data filter name, enter exclude_product_price.
  5. For Target database, choose the database hudi_crawler_blog.
  6. For Target table, choose the table sample_hudi_mor_table_ro.
  7. For Column-level access, select Exclude columns, and choose the column price.
  8. For Row filter expression, enter true.
  9. Choose Create filter.

Grant Lake Formation permissions to the DataAnalyst user

Complete the following steps to grant Lake Formation permission to the DataAnalyst user

  1. On the Lake Formation console, choose Data lake permissions.
  2. Choose Grant.
  3. For Principals, choose IAM users and roles, and choose the user DataAnalyst.
  4. For LF tags or catalog resources, choose Named data catalog resources.
  5. For Database, choose the database hudi_crawler_blog.
  6. For Table – optional, choose the table sample_hudi_mor_table_ro.
  7. For Data filters – optional, select exclude_product_price.
  8. For Data filter permissions, select Select.
  9. Choose Grant.

You granted Lake Formation permission on the database hudi_crawler_blog and the table sample_hudi_mor_table_ro, excluding the column price to the DataAnalyst user. Now let’s validate user access to the data using Athena.

  1. Sign in to the Athena console as a DataAnalyst user.
  2. On the query editor, run the following query: 
    SELECT * FROM "hudi_crawler_blog"."sample_hudi_mor_table_ro" limit 10;

The following screenshot shows our output:

Now you validated that the column price is not shown, but the other columns product_id, product_name, update_at, and category are shown.

Clean up

To avoid unwanted charges to your AWS account, delete the following AWS resources:

  1. Delete AWS Glue database hudi_crawler_blog.
  2. Delete AWS Glue crawlers hudi_cow_crawler and hudi_mor_crawler.
  3. Delete Amazon S3 files under s3://your_s3_bucket/data/sample_hudi_cow_table/ and s3://your_s3_bucket/data/sample_hudi_mor_table/.

Conclusion

This post demonstrated how AWS Glue crawlers work for Hudi tables. With the support for Hudi crawler, you can quickly move to using AWS Glue Data Catalog as your primary Hudi table catalog. You can start building your serverless transactional data lake using Hudi on AWS using AWS Glue, AWS Glue Data Catalog, and Lake Formation fine-grained access controls for tables and formats supported by AWS analytical engines.

About the authors

Noritaka Sekiyama is a Principal Big Data Architect on the AWS Glue team. He works based in Tokyo, Japan. He is responsible for building software artifacts to help customers. In his spare time, he enjoys cycling with his road bike.

Kyle Duong is a Software Development Engineer on the AWS Glue and Lake Formation team. He is passionate about building big data technologies and distributed systems.

Sandeep Adwankar is a Senior Technical Product Manager at AWS. Based in the California Bay Area, he works with customers around the globe to translate business and technical requirements into products that enable customers to improve how they manage, secure, and access data.

Apple to Add Manual Authentication to iMessage

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/11/apple-to-add-manual-authentication-to-imessage.html

Signal has had the ability to manually authenticate another account for years. iMessage is getting it:

The feature is called Contact Key Verification, and it does just what its name says: it lets you add a manual verification step in an iMessage conversation to confirm that the other person is who their device says they are. (SMS conversations lack any reliable method for verification­—sorry, green-bubble friends.) Instead of relying on Apple to verify the other person’s identity using information stored securely on Apple’s servers, you and the other party read a short verification code to each other, either in person or on a phone call. Once you’ve validated the conversation, your devices maintain a chain of trust in which neither you nor the other person has given any private encryption information to each other or Apple. If anything changes in the encryption keys each of you verified, the Messages app will notice and provide an alert or warning.

Чие е МВР?

Post Syndicated from Емилия Милчева original https://www.toest.bg/chie-e-mvr/

Чие е МВР?

Окървавеният протест на футболни фенове срещу БФС предизвика една изтраяла от ден до пладне дандания да подаде ли оставка вътрешният министър Калин Стоянов заради полицейската жестокост. Премиерът Николай Денков поиска от него да осъди насилието на униформените. Стоянов отговори, че по принцип не толерира насилието и че виновните, било то граждани или полицаи, „ще понесат отговорност“. И акад. Денков остана доволен от този знак.

… казах, че ако министърът продължава да си затваря очите и да замазва проблемите, щях да си тръгна. Министърът тръгна в правилната посока. Надявам се да извърви този път докрай.

Ерго няма да си тръгне, както беше обявил. Неуместно в политически план изявление на Денков, тъй като министър-председателят иска оставки от министрите си и си тръгва, когато политическата сила, която му е гласувала доверие, го оттегли. 

Но настоящата ситуация изкара наяве тайни на т.нар. сглобка, например че министърът на вътрешните работи не е номинация на ПП–ДБ, а на ГЕРБ. Досега публично известното беше, че представител на ГЕРБ в правителството е само Мария Габриел, която е вицепремиер и министър на външните работи, а след ротацията през март ще заеме поста министър-председател. 

Кой посочи Калин Стоянов?

Изборът на Калин Стоянов за министър на вътрешните работи в кабинета „Денков–Габриел“ изненада. Като директор на Главна дирекция „Борба с организираната престъпност“ (ГДБОП) през декември 2021 г. той беше обвинен, че е оказвал натиск върху една от прокурорките, наблюдаващи разследването за пране на пари, свързано с авансовите плащания за строителство на магистрала „Хемус“. Онези 53 млн. лв., теглени от оперативно интересни лица кеш на гише и прибирани в сакове. Според Софийската районна прокуратура Стоянов е искал да бъде смекчено обвинението на Борислав Колев, ако той даде обяснения по случая. Стоянов, тогава директор на ГДБОП, отрича, обяснявайки, че само се е интересувал защо не е бил разпитан Колев. В крайна сметка продължилото година разследване е прекратено няколко месеца преди Калин Стоянов да стане министър.

Към днешна дата и министърът на вътрешните работи, и съпредседателят на „Продължаваме промяната“ Кирил Петков в телевизионни интервюта открехват вратата към начина, по който са правени назначенията в кабинета, спестявайки каквото не им отърва. Но пък от премълчаното от единия и казано от другия може да се сглоби част от истината. Според Стоянов цели два дни Кирил Петков го увещавал – изразът е „ми се молеше аз да стана министър на вътрешните работи“, а с Петков го свързал настоящият заместник-шеф на ДАНС Петър Петров. Но по онова време Петров не заемаше този пост – беше избран за депутат от ПП–ДБ. Преди това е бил съветник в премиерския кабинет на Кирил Петков и има кариера в контраразузнаването. Неофициално е известно, че е номинацията на ПП–ДБ за шеф на ДАНС, ако настоящият – Пламен Тончев, бъде освободен.

Кирил Петков пък заяви, че Стоянов е кадър на ГЕРБ, с обяснението, че го е взел като експерт. „Номинациите на ПП–ДБ са ясните политически лица, които са бивши депутати. Зад тях заставаме чисто партийно на 100%“, каза той. Но бившите депутати са неколцина – строителният министър Андрей Цеков, правосъдният Атанас Славов, транспортният Георги Гвоздейков. След като прибавим към тях Денков, министъра на финансите Асен Василев, на туризма Зарица Динкова, на електронното управление Александър Йоловски, познати от предишния кабинет на Петков, за една част от членовете на правителството, които оглавяват ключови министерства, не е ясно кой е „подал“ експертните имена. Например за енергетиката, оглавявана от Румен Радев, за икономиката и индустрията, начело с Богдан Богданов, или за земеделието, което е ръководено от Кирил Вътев. 

За МВР беше много важно този човек да не бъде политически обвързан, защото идваха местни избори, трябваше да бъде максимално неутрален. И започнахме и от ПП–ДБ, и от ГЕРБ да се подават експертни имена… Калин Стоянов беше подаден като експерт от ГЕРБ [или от Мария Габриел, или от Борисов, Петков не помнел кой от двамата, но едва ли има съмнение, че го подава лидерът на ГЕРБ – б.а.], като ние обаче, преди да се вземе каквото и да е решение, отидохме да го интервюираме. Когато аз отидох да говоря с него, той казва, че аз съм първият човек, който говори с него… Защо ние решихме, че си заслужава да направим такова предложение? Защото видяхме, че той беше разследвал чувалите с магистралите и тази тема.

Това с магистралите не е много убедително – изглежда, че „интервюиращите“ не са разбрали какво точно е вършил Стоянов. Как иначе щяха да го подадат като експерт от ГЕРБ, ако се е заел да разследва „Хемус“, ключов обект на Бойко Борисов?!

Оплетените обяснения кой кого е посочил за пост в правителството, показаха, че ГЕРБ и ПП–ДБ са договаряли министри в правителството на Денков–Габриел, което не е само на ПП–ДБ, и така саможертвата на ГЕРБ, отказали се да правят кабинет с първия мандат, не изглежда толкова значима. Обществото пък не е наясно кой още освен Стоянов е министър на ГЕРБ – и дали е само на ГЕРБ, или и на ДПС. И двете политически сили защитиха Стоянов, а председателят на парламентарната група на ДПС Делян Пеевски даде да се разбере, че е имал и срещи с вътрешния министър и – в тон с ГЕРБ – че самият министър споделял за опити за влияние върху него. 

Каза, че има опит за влияние за назначаване на полицейски шефове.

Бойко Борисов пък заяви, че назначенията в МВР са експертни и са правени от ПП–ДБ, а не от ГЕРБ.

Когато дойде темата МВР, аз им казах, че трябва да е експерт, професионалист, отдалечен от всички. Предложих да се избира от наличните – назначени от тях по тяхно време години наред или от президента Радев служби. […] Казах да изберем между шефа на полицията, на БОП-а, на жандармерията. Той отиде, консултира се и след това предложиха Калин Стоянов. Аз се съгласих. Аз не съм говорил с него, не съм го убеждавал, не съм се срещал.“

Това сглобяване на кабинета, изяснило се на петия месец от дейността му, показва, че управлява коалиция, съвсем истинска, просто споразуменията не са публично известни. Липсата на разписано коалиционно споразумение ги улеснява. Ако протестът срещу БФС не се беше окървавил, нямаше и да се разбере за механизмите на сглобяването.

Негативната селекция

Проблемът с МВР обаче не е само в начина, по който е избран министърът, проблемът с МВР е много преди Стоянов, много преди Бойко Рашков, Цветан Цветанов и всички изредили се на ул. „Шести септември“. Проблемът е в начина, по който работи МВР, по който политиците употребяват МВР, а също и в негативната селекция на кадри за правоохранителните органи. 

Освен публично известното – че обслужва политическата сила на власт, – МВР функционира на кланов принцип: родово-семейните връзки оказват влияние при назначенията, а всяка управляваща партия е оставила свой кадрови отпечатък. При следващите на власт тези хора просто ги „скатават“, преназначавайки ги на други позиции в системата. Вътрешното министерство е пълно с такива назначения, довели със себе си и още „свои“ – подобно на политиците, и те се обграждат с лоялни подчинени. Повечето гледат да избутат до пенсия, за да получат заплатите накуп. Не странят от подкупите. Такава среда бързо деморализира – ако човек не иска неприятности, се примирява или просто напуска. 

Колко полезно за сигурността на гражданите и правовия ред е такова МВР… Отговорите на реторичния въпрос винаги са болезнени. Окървавеният протест срещу БФС разкри, освен лошия подбор, и брадясалия проблем на МВР – слабото или никакво обучение на кадрите в системата. Години наред, през куп правителства, бюджетът на Вътрешното министерство се изяжда от заплатите, които при това непрекъснато растат, а полицаите не спират да се оплакват от липса на добро облекло, качествена защита и др. В тазгодишния бюджет МВР получи 10% ръст на възнагражденията и допълнителни 100 лв. за някои длъжности. В бюджета за 2024 г. също са предвидени повишения на заплати в Министерството.

Какво толкова! Полицаите бият, политиците лъжат – това им е работата, нали? Банално утешение. Но червената линия, при която един полицай превишава правомощията си, за да наруши човешки права, е все по-проблемна и в развитите, и в неразвитите демокрации – защото протестите стават все по-силни и яростни.

Във Франция например за първите четири месеца на тази година се състояха над 3800 протеста – по изчисления на Bloomberg, срещу вдигането на възрастта за пенсиониране от 62 на 64 години. Демонстрациите и гражданското недоволство изкараха по улиците и площадите на Франция хиляди полицаи и доведоха до сериозни сблъсъци и дори до смъртта на 17-годишен младеж, застрелян от полицай. Но вътрешният министър Жералд Дарманен неизменно отхвърляше обвиненията в насилие. След остри критики от страна на Лигата за правата на човека (Ligue des droits de l’homme – LDH) и предприетите от нея съдебни действия срещу „несмъртоносните оръжия“, използвани от органите на реда, той дори я заплаши със спиране на държавното финансиране. На една от демонстрациите на земеделски производители и екоактивисти срещу изкуствени водоеми, които извличат подпочвените води, френското МВР е изпратило 3000 полицаи, 4 танка, 9 хеликоптера и водни оръдия.

Какво става с българските полицаи, разследвани за произвол след протеста срещу БФС? Образувани са дисциплинарни производства срещу трима служители за превишаване на полицейските правомощия и те са отстранени от длъжност. По информацията от СДВР срещу други двама тече проверка, която ще определи степента им на виновност. И понеже някой все пак трябваше да бъде уволнен, това беше заместник-министърът на вътрешните работи Стоян Темелакиев – не министърът, нито пък главният секретар на МВР Живко Коцев.

Протестът на футболните фенове се окървави, но недостатъчно, за да подаде оставка министърът на вътрешните работи. При смърт в резултат на полицейско насилие, креслото на Калин Стоянов щеше да изстине. И щеше да седне друг калинстоянов, подаден от някой бойкоборисов.

Отворен код и придобиване на софтуер от държавата

Post Syndicated from Bozho original https://blog.bozho.net/blog/4168

По покана на Европейската Комисия участвах на конференция за използването на софтуер с отворен код в обществения сектор, на която споделих следното:

България през 2016 г. прие закон, с който всички административни органи се задължават при възлагане на разработка на софтуер, в договорите и спецификациите си да включат клаузи, с които да изискват разработеният софтуер да е публичен (от първия ден на разработка) и да е с лиценз, който позволява безплатно използване от всеки.

Това е важна мярка, за чието спазване настоявам във всичките си качества оттогава, по следните причини:

  1. Преизползване – когато кодът на една система е публичен, всеки друг, разработващ сходна функционалност, може да го използва и развива. В системите в публичния сектор има много повтарящи се изисквания и е загуба на време и пари да се разработват всеки път от нулата.
  2. Прозрачност и доверие – след като системата е разработена с публични средства, на обществото се дължи кода на системата. Така ще знаем как точно работи дадена система (и че работи правилно, съгласно закона, а не съгласно нечии интерпретации, както се оказа наскоро със системата за мониторинг на наличностите на лекарствата), и ще имаме повече доверие в нея (неслучайно направих предложение да се изгради нов софтуер на машините за гласуване, който да бъде отворен, но за съжаление това предложение не се прие в миналия парламент)
  3. Качество – когато изпълнител на дадена поръчка знае, че кодът, който произведе, ще бъде публичен, внимава повече да спазва добрите практки, да прави кода по-четим и по-подреден. Не един път съм чувал, че „ще го публикуваме, ама чакай първо да го пооправим малко“. Включително на самата конференция, от колега от друга държава.
  4. Одитируемост – държавата трябва да може да одитира какво точно е получила и да гарантира, че е получила (и че в продукционна среда работи) правилният софтуер. Примерът с тол системата, чийто изходен код липсва, показва защо това е важно – АПИ в момента не може да одитира в дълбочина собствената си система.
  5. Ценова ефективност – има два аспекта на отворения код, свързан с цената за държавата, и по-точно за общата цена на притежаване (total cost of ownership). Първият е, че когато използваш готов продукт с отворен код, той е безплатен (спрямо цената на лицензите или абонамента за платен такъв). Но вторият аспект е, че когато нямаш кода да даден софтуер, или нямаш права да го модифицираш, си до голяма степен заключен за един доставчик. Ако лицензът на кода позволява модификации (каквото позволят всички open source лицензи), софтуерната компания, която го поддържа, може да бъде сменена, което пък означава конкуренция и съответно – по-ниска цена.

Имаме закона, имаме съответната наредба и шаблона за технически задания, имаме и правомощията за контрол по прилагането и съгласуване на заданията, така всички нормативни и процесни предпоставки са налице. Не е налице още културата и пълното разбиране у възллжителите защо това е важно (пример за това е моят въпрос до МВР защо по проекта за електронна идентификация все още няма публикуван изходен код, въпреки че се работи по него). Тук е моментът да отбележа, че мобилното приложение за електронна идентификация, която аз възложих като министър, също няма публикувана работната версия на код, защото проектът беше спрян в служебния кабинет и ключови негови части блокирани заради отказ от предоставяне на достъп до данни от МВР. Ако проектът беше продължил, щях да настоявам за работа именно в хранилището на МЕУ.

Освен честото неспазване на изискванията, имаме и притеснения от страна на службите за сигурност, които виждат риск в разкриването на изходния код. Аз не съм съгласен с тези притеснения, тъй като не се публикува нищо, свързани с оперирането на системата в продукционна среда – нито криптографски ключове, нито пароли, нито мрежови настройки. Да, съществува риск от наличие на уязвимости в кода, които да бъдат видени от хакери, но криенето на кода не означава, че уязвимостите не са там. Криенето, обаче, може да попречи на „белите хакери“ да ги докладват преди да бъде злоупотребено с тях.

Невинаги е възможно да се възлага разработка на софтуер с отворен код – понякога се придобива нещо готово, за което се купува лиценз или абонамент. Трябва предварително да е ясно на възложителя какво си поръчва – готов софтуер, разработка „на чисто“, или хибрид (готов софтуер с надграждане със специфични модули). Именно последното е случаят с тол системата, но държавата не е знаела какво поръчва, и вместо да получи кода на доработените за нея функционалности, е получила само лиценз.

Готовият софтуер е нещо хубаво и удобно, защото зад него стои производител и неговите партньори за внедряване. Но колкото и привлекателно да изглежда, трябва да се направи преценка дали са налични достатъчно добри алтернативи с отворен код. Понякога няма такива, или са твърде далеч като възможности. Понякога има, но няма достатъчно експерти с познания за тях.

Има и случаи, в които изпълнителят твърди (с право или не), че определени библиотеки, съдържащи основни функционалности, са си негови и той само дава лиценз за тях. Това е възможно, като в такива случаи (какъвто би следвало да бъде и този с тол системата), държавата може да поиска една от две опции. За предпочитане е т.нар. code available, вкл. с право на модификации. Т.е. кодът не се публикува, изпълнителят запазва авторските си права, но предоставя кода на възложителя, както и правото той да възложи надграждане на друг. Това е сложно за уреждане, и включва риск от „падане на гаранция“ („не знам какво сте му правили, при нас си работи“). Втората опция е т.нар. source code escrow – кодът се депозира в трета страна, която го предоставя на държавата в случай, че производителят фалира или спре работата по продукта. Това е по-малко желателният подход, тъй като обвързва държавата с един доставчик, но поне „фаталните“ сценарии са покрити.

Остава и казусът с т.нар. „софтуер като услуга“ (SaaS). Там държавата може да поиска достъп до кода, за да го одитира, но нито може да го публикува, нито да го надгражда (освен ако самата система няма plug-in архитектура). Много съвременни софтуерни компании предпочитат „софтуер като услуга“, защото това им дава повторяем приход, по-лесно управление на клиентите и по-добра видимост върху използването. Само че SaaS е оперативен разход, а не капиталов. SaaS представлява задължително заключване за един доставик. Ползите са удобството и нулевата инвестиция в хардуерна инфраструктура. В някои случаи SaaS е валиден подход, но според мен софтуерните компании могат да предоставят т.нар. self-hosted версии на своите продукти, което би било от полза за обществени организации.

Софтуерът с отворен код е най-силен в основните технологични решения – системи за управление на бази данни, операционни системи, уеб сървъри. При този тип софтуер много рядко има причина за комерсиални решения, особено ако има корпоративна поддръжка за софтуера с отворен код „на една ръка растояние“. Затова бих насърчил всички да използват напр. Linux, PostgreSQL, NginX.

При всички положения, когато държавата използва софтуер, който е с отворен код, трябва да дефинира процеси по неговото развитие и управление, напр. механизъм за предлагане на подобрения от участници, различни от избрания изпълнител. Всъщност, в наредбата, която подготвихме през 2016 г., в чл. 60 такъв процес е предвиден, но досега той не е реализиран.

Закупуването на софтуер, неговото управление и поддръжка, изчисляването на цената на собственост и всички останали аспекти са сложни. В един блогпост или няколко изказвания на конференция не могат да покрият всички тези сложности. За да не допуска държавата грешки, трябва да има достатъчно познание по тези въпроси в нейните структури. Министерство на електронното управление и структурите към него могат да бъдат център за такова познание, но не следва да са единствените. Затова обмисляме предлагането на фигурата на Chief information officer (CIO) за администрациите, които да са не просто ИТ директори, отговарящи за оперативните ИТ задачи, а които да имат директна комуникация с органа на власт и да изграждат стратегията и да водят дигиталната трансформация на администрацията.

С времето ще се формират добрите практики, ще се натрупа опит и няма да се допускат грешки, които струват на държавата скъпо, и от които понякога губим години в дадена сфера. А отвореният код ще бъде важен инструмент за дигиталната трансформация.

Материалът Отворен код и придобиване на софтуер от държавата е публикуван за пръв път на БЛОГодаря.

„Жена, която между другото е транс.“ Защо Габриела Банкова рискува живота си

Post Syndicated from Светла Енчева original https://www.toest.bg/zhena-koyato-mezhdu-drugoto-e-trans/

„Жена, която между другото е транс.“ Защо Габриела Банкова рискува живота си

Започвам с необичайно езиково уточнение. След моя предишна статия в „Тоест“ получих критични коментари от читатели, които са транс, по отношение на слятото изписване на „транс“ („трансчовек“, „трансхора“ и пр.). Представителите на тази общност са възприели разделното писане (например „транс човек“). На други места съм се съобразявала с разделното изписване. Според коректорката на „Тоест“ Павлина Върбанова обаче правилно е да се пише слято.

Не поставям под съмнение професионализма на Павлина Върбанова – от гледна точка на формалните езикови правила тя е права. Как да постъпя обаче, ако тези правила противоречат на самоидентификацията на една общност? ЛГБТИ+ хората в България са достатъчно потърпевши от юридическите ни закони, за да налагам против волята им и езиковите. Затова на своя отговорност ще пиша разделно.

Друг е въпросът, че кодификатор на правописа е Институтът за български език при БАН, чиито нововъведения понякога са изненадващи (да си припомним как „напът“ изведнъж започна да се пише слято) и според който „джендър“ е обидна дума.

Гладна стачка, която продължава вече 10 дни

Габриела Банкова, транс жена, обяви гладна стачка. Това се случи на 13 ноември 2023 г. Поводът е загубено дело за промяна на гражданския пол, което е завела преди година и половина. Решението подлежи на обжалване, но Банкова е сигурна, че изходът е предрешен. След като през февруари тази година Върховният касационен съд (ВКС) на практика забрани юридическата смяна на пола, която беше възможна в България повече от 30 години, е ясно, че подобно дело няма особени шансове за успех.

Стачкуващата е пред Съдебната палата от 13 ноември 2023 г. През това време валя много дъжд. Имаше и толкова силен вятър, че мъж беше убит от паднало дърво, а съпругата му – тежко ранена. Единственият отрязък от няколко часа, през който не е била между двата лъва, Габриела прекарва в Първо РПУ на МВР. Там е отведена с белезници „за установяване на самоличността“ и след кратък престой е освободена. Тъкмо навреме – преди в същото РПУ по-късно през деня да бъдат докарани футболните фенове, арестувани на протеста против Българския футболен съюз. А ултраси и транс жена в едно и също затворено пространство е животозастрашаваща (за транс жената) комбинация.

Стачката беше отразена от множество онлайн медии. bTV излъчи репортаж в сутрешния си блок, а БНТ – в новините. На сайта на bTV протестиращата жена е представена като „мъж, сменил пола си“, което е обидно и демонстрира неуважение към нейната идентичност.

Случаят намира отзвук и в БНР и „Дарик радио“. С изключение на единични репортажи обаче мейнстрийм медиите, каквито в България са националните телевизии, игнорират факта, че човешко същество повече от седмица излага живота си на риск в центъра на София пред погледа на всички. Това не се смята за тема, заслужаваща систематично внимание и проследяване. На 20 ноември пред Съдебната палата се проведе демонстрация в подкрепа на стачкуващата жена и по повод Деня за възпоменание на жертвите на трансфобията. В протеста се включиха над 100 души, но той не намери място в централните новинарски емисии. БНТ го отбеляза в късните си новини и с кратка дописка без видео на сайта си.

Безрезервна подкрепа и плашещо спокойствие

Срещнах се с Габриела Банкова на третия ден от гладната ѝ стачка. Останах с впечатлението, че с усилие държи очите си отворени. Каза, че не е спала много. Говореше тихо – вероятно както от изтощение, така и поради бактериалната бронхопневмония, която каза, че има. Докато близки и приятели се суетяха около нея, търсейки начин да я предпазят от дъжда, тя коментираше опитите им с иронично снизхождение. Същевременно обаче оценяваше подкрепата, която получава:

Цялото ми семейство, всичките ми приятели, всичките ми колеги, всички хора, които познавам, ме подкрепят изцяло, напълно и безрезервно. Идват всеки ден десетки хора […]. Майка ми прекарва нощите тук с мен, баща ми и брат ми идват, много колеги, мои приятели. Всички хора, които виждате тук, са мои приятели. Аз дойдох тук облечена само с едно яке, нищо друго. Това, което виждате, е една палатка, пълна с провизии. И водата, която използвам, за да оцелявам, ми е донесена от тях.

Ведрият дух на протестиращата е заразителен за околните, макар на пръв поглед да е в необясним контраст с риска за живота ѝ поради гладуването, пневмонията и денонощното седене на открито (нощите си прекарва в тънката палатка, която не спира студа). Ала всъщност в самообладанието ѝ няма нищо утешително – то е дори плашещо. Габриела Банкова излъчва спокойствието на човек, който е наясно с възможните последствия. Прежалил се е. И в известен смисъл вече е от другата страна – не само на пола, а и на живота.

Идентичността като кауза от обществен интерес

Питам стачкуващата жена как да се обръщам към нея и да я представя в статията си, защото не искам да бъда от онези представители на медии, които интуитивно преценяват на кого да говорят на малко име и на „ти“. „Може да ми говорите, както желаете – Габриела, Габи…“ – казва тя. Следващият ми въпрос е как се идентифицира. „Като Габриела Банкова, това е името ми.“ Уточнявам, че имам предвид дали да казвам, че е транссексуална, трансджендър, транс или просто жена. „Аз съм жена, която между другото е транс“, гласи отговорът.

Моля я да ми разкаже повече за себе си, за да усетят читателите, че става въпрос за живота на човешко същество от плът и кръв. Ала за нея е от значение не личността, а каузата: 

Аз не смятам, че има значение коя съм. Смятам, че единственото, което има значение, е, че моите конституционни и човешки права биват нарушени от съда.

Въпросът за идентичността не е просто личен, тъй като човек се идентифицира не само пред себе си, а и пред другите. Затова протестиращата жена не смята личната си карта за валиден документ. „Защото документът, който имам – личната ми карта, – служи за идентификация. Но аз не мога да бъда идентифицирана по него, тъй като не мога да го променя. На него пише неща, които не съответстват на реалността, която хората виждат. Това е документ, който другите хора в обществото не биха могли да идентифицират с мен.“

До какво води отказът от промяна на гражданския пол

Когато става дума за борбата за равни права на ЛГБТИ+ хората, дежурният реторичен въпрос е „Кой им пречи?“. Габриела Банкова споменава само част от проблемите, които е имала, защото името в личната карта не съответства на външния ѝ вид: 

Заплашвана съм и с арест, заплашвана съм по много начини. Отхвърляна съм много често, почти постоянно. Почти във всяко здравно заведение, от години. В градския транспорт, в банки…

Конкретно от лечебните заведения е гонена „с довода, че тази лична карта не може да бъде моя и аз се опитвам да се възползвам от чужди здравни осигуровки“. Поради тази причина ѝ е отказано да се лекува по Здравна каса. Затова и не е получила по обичайния ред антибиотика, който пие заради пневмонията си, а се е снабдила с него от познати, въпреки че вече антибиотици не може да се купуват без електронни рецепти.

Така нарушаването на правата неизбежно води и до нарушаване на правилата, защото хората с нарушени права все някак трябва да оцеляват. Транс лицата не престават да съществуват само защото Конституционният съд (КС) и ВКС реагират като шопа от известния виц при вида на жираф: „Те такова животно нема.“

Гладната стачка на една транс жена за правосъдна реформа

Целта на протеста на Габриела Банкова не е само да се реши проблемът с документите ѝ за самоличност. Не е и просто за правата на транс хората в България, нито дори на ЛГБТИ+ хората като цяло. Тя е стигнала до извода, че нарушаването на правата е резултат от липсата на реформи в съдебната система. А това засяга всички. Затова заявява:

Аз ще продължа тази гладна стачка и ще призовавам през цялото време народа да дойде и да протестира с мен. И да предприемем необходимите действия да реформираме нашата съдебна система. За да може тя да започне […] да спазва гражданските, човешките и конституционните права на гражданите на България, както и на всички хора, пребиваващи в България.

Според нея проблемът е, че „в момента съдът не е разделен от останалите власти“ и не може да ги контролира ефективно, защото е зависим от тях. Тя не дава конкретен пример за това, но собствената ѝ ситуация доказва тезата ѝ. В резултат от политически експлоатираната кампания срещу Истанбулската конвенция КС, който е най-висшето съдилище, излезе с две абсурдни решения. Първото е, че Конвенцията противоречи на Конституцията, защото според нея има само два биологични пола (макар в Основния закон да не пише нищо по въпроса). Второто – че според Конституцията полът има само биологичен смисъл.

Решението на ВКС, според което българското право не предвижда възможност за смяна на пола на транс лицата (въпреки че това се е случвало с десетилетия), следва логиката на КС, който пък следва логиката на пропагандната кампания.

Една от идеите на Габриела Банкова за правосъдна реформа е съдиите, прокурорите, депутатите и другите „държавни администратори“ да могат да носят наказателна отговорност за решенията, които вземат. Защото от тези решения зависят съдбите на цели групи хора, които понякога с десетилетия не могат да получат справедливост и са лишени от права.

Правосъдна реформа без човешки права?

Габриела Банкова иска не само да говори на хората, дошли да я видят, но и да ги слуша: „Ще чакам народа да дойде при мен, да обсъждаме, да обясня моите проблеми, да чуя проблемите на другите хора и заедно да започнем да предприемаме действия по това, което всички знаем – че тази съдебна система отдавна, от десетилетия се нуждае от тотална реформа.“

Дългогодишните борци за правосъдна реформа обаче не дават знак да са забелязали гладната стачка на жената пред Съдебната палата. Няма реакция нито от страна на партиите в състава на коалицията ПП–ДБ, която съществува в името на правосъдната реформа, нито от неправителствени организации и изявени активисти в областта на правосъдието. Нито от страна на юристите, които полагат усилия реформата да се осъществи.

Борците за правосъдна реформа поставят акцент върху необходимостта от институционални промени, например контрол върху дейността на главния прокурор и промяна на структурата на Висшия съдебен съвет. Човешките и гражданските права рядко са във фокуса им. Доколкото изобщо ги забелязват, то е при случаи като полицейско насилие, липса на достъп до информация или корупционно-криминални афери. Идеята някой например да предложи планираните изменения в Конституцията да включат отмяна на текста, че бракът е само между мъж и жена, граничи с фантастиката.

Само че правосъдните системи на демократичните държави се базират на основните права на личността – на човешко достойнство, личен и семеен живот и т.н. Те се потъпкват далеч не само по отношение на ЛГБТИ+ хората и представителите на други дискриминирани групи. Системното нарушаване на правата, както това става с дискриминираните групи, обаче провокира представителите на тези групи да забележат връзката между безправието и дефектното правосъдие.

Една транс жена гладува от десет дни, за да се състои важният обществен разговор за връзката между правосъдието и правата. Залогът е животът ѝ.

Разследване на Валя Ахчиева: Турски ваксини в съда

Post Syndicated from Екип на Биволъ original https://bivol.bg/vaksini-sad-valya-ahchieva.html

сряда 22 ноември 2023


Делото, за т.нар. турски ваксини беше образувано по обвинения на прокуратурата, за първи път срещу действащ министър – д-р Петър Москов – бивш министър на здравеопазването и още четирима. Делото…

Introducing persistent buffering for Amazon OpenSearch Ingestion

Post Syndicated from Muthu Pitchaimani original https://aws.amazon.com/blogs/big-data/introducing-persistent-buffering-for-amazon-opensearch-ingestion/

Amazon OpenSearch Ingestion is a fully managed, serverless pipeline that delivers real-time log, metric, and trace data to Amazon OpenSearch Service domains and OpenSearch Serverless collections.

Customers use Amazon OpenSearch Ingestion pipelines to ingest data from a variety of data sources, both pull-based and push-based. When ingesting data from pull-based sources, such as Amazon Simple Storage Service (Amazon S3) and Amazon MSK using Amazon OpenSearch Ingestion, the source handles the data durability and retention. Push-based sources, however, stream records directly to ingestion endpoints, and typically don’t have a means of persisting data once it is generated.

To address this need for such sources, a common architectural pattern is to add a persistent standalone buffer for enhanced durability and reliability of data ingestion. A durable, persistent buffer can mitigate the impact of ingestion spikes, buffer data during downtime, and reduce the need to expand capacity using in-memory buffers which can overflow. Customers use popular buffering technologies like Apache Kafka or RabbitMQ to add durability to their data flowing through their Amazon OpenSearch Ingestion pipelines. However, these tools add complexity to the data ingestion pipeline architecture and can be time consuming to setup, right-size, and maintain.

Solution overview

Today we’re introducing persistent buffering for Amazon OpenSearch Ingestion to enhance data durability and simplify data ingestion architectures for Amazon OpenSearch Service customers. You can use persistent buffering to ingest data for all push-based sources supported by Amazon OpenSearch Ingestion without the need to set up a standalone buffer. These include HTTP sources and OTEL sources for logs, traces and metrics. Persistent buffering in Amazon OpenSearch Ingestion is serverless and scales elastically to meet the throughput needs of even the most demanding workloads. You can now focus on your core business logic when ingesting data at scale in Amazon OpenSearch Service without worrying about the undifferentiated heavy lifting of provisioning and managing servers to add durability to your ingest pipeline.

Walkthrough

Enable persistent buffering

You can turn on the persistent buffering for existing or new pipelines using the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDK. If you choose not to enable persistent buffering, then the pipelines continue to use an in-memory buffer.

By default, persistent data is encrypted at rest with a key that AWS owns and manages for you. You can optionally choose your own customer managed key (KMS key) to encrypt data by selecting the checkbox labeled Customize encryption settings and selecting Choose a different AWS KMS key. Please note that if you choose a different KMS key, your pipeline needs additional permission to decrypt and generate data keys. The following snippet shows an example AWS Identity and Access Management (AWS IAM) permission policy that needs to be attached to a role used by the pipeline.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "KeyAccess",
            "Effect": "Allow",
            "Action": [
              "kms:Decrypt",
              "kms:GenerateDataKeyWithoutPlaintext"
            ],
            "Resource": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
        }
    ]
}

Provision for persistent buffering

Once persistent buffering is enabled, data is retained in the buffer for 72 hours. Amazon OpenSearch Ingestion keeps track of the data written into a sink and automatically resumes writing from the last successful check point should there be an outage in the sink or other issues that prevents data from being successfully written. There are no additional services or components needed for persistent buffers other than minimum and maximum OpenSearch compute Units (OCU) set for the pipeline. When persistent buffering is turned on, each Ingestion-OCU is now capable of providing persistent buffering along with its existing ability to ingest, transform, and route data. Amazon OpenSearch Ingestion dynamically allocates the buffer from the minimum and maximum allocation of OCUs that you define for the pipelines.

The number of Ingestion-OCUs used for persistent buffering is dynamically calculated based on the source, the transformations on the streaming data, and the sink that the data is written to. Because a portion of the Ingestion-OCUs now applies to persistent buffering, in order to maintain the same ingestion throughput for your pipeline, you need to increase the minimum and maximum Ingestion-OCUs when turning on persistent buffering. This amount of OCUs that you need with persistent buffering depends on the source that you are ingesting data from and also on the type of processing that you are performing on the data. The following table shows the number of OCUs that you need with persistent buffering with different sources and processors.

Sources and processors Ingestion-OCUs with buffering Compared to number of OCUs without persistent buffering needed to achieve similar data throughput
HTTP with no processors 3 times
HTTP wit Grok 2 times
OTel Trace 2 times
OTel Metrics 2 times

You have complete control over how you want to set up OCUs for your pipelines and decide between increasing OCUs for higher throughput or reducing OCUs for cost control at a lower throughput. Also, when you turn on persistent buffering, the minimum OCUs for a pipeline go up from one to two.

Availability and pricing

Persistent buffering is available in the all the AWS Regions where Amazon OpenSearch Ingestion is available as of November 17 2023. These includes US East (Ohio), US East (N. Virginia), US West (Oregon), US West (N. California), Europe (Ireland), Europe (London), Europe (Frankfurt), Asia Pacific (Tokyo), Asia Pacific (Sydney), Asia Pacific (Singapore), Asia Pacific (Mumbai), Asia Pacific (Seoul), and Canada (Central).

Ingestion-OCUs remains at the same price of $0.24 cents per hour. OCUs are billed on an hourly basis with per-minute granularity. You can control the costs OCUs incur by configuring maximum OCUs that a pipeline is allowed to scale.

Conclusion

In this post, we showed you how to configure persistent buffering for Amazon OpenSearch Ingestion to enhance data durability, and simplify data ingestion architecture for Amazon OpenSearch Service. Please refer to the documentation to learn other capabilities provided by Amazon OpenSearch Ingestion to a build sophisticated architecture for your ingestion needs.

About the Authors

Muthu Pitchaimani is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search applications and solutions. Muthu is interested in the topics of networking and security, and is based out of Austin, Texas.

Arjun Nambiar is a Product Manager with Amazon OpenSearch Service. He focusses on ingestion technologies that enable ingesting data from a wide variety of sources into Amazon OpenSearch Service at scale. Arjun is interested in large scale distributed systems and cloud-native technologies and is based out of Seattle, Washington.

Jay is Customer Success Engineering leader for OpenSearch service. He focusses on overall customer experience with the OpenSearch. Jay is interested in large scale OpenSearch adoption, distributed data store and is based out of Northern Virginia.

Rich Giuli is a Principal Solutions Architect at Amazon Web Service (AWS). He works within a specialized group helping ISVs accelerate adoption of cloud services. Outside of work Rich enjoys running and playing guitar.

Introducing Amazon CloudFront KeyValueStore: A low-latency datastore for CloudFront Functions

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/introducing-amazon-cloudfront-keyvaluestore-a-low-latency-datastore-for-cloudfront-functions/

Amazon CloudFront allows you to securely deliver static and dynamic content with low latency and high transfer speeds. With CloudFront Functions, you can perform latency-sensitive customizations for millions of requests per second. For example, you can use CloudFront Functions to modify headers, normalize cache keys, rewrite URLs, or authorize requests.

Today, we are introducing CloudFront KeyValueStore, a secure global low-latency key value datastore that allows read access from within CloudFront Functions, enabling advanced customizable logic at the CloudFront edge locations.

Previously, you had to embed configuration data inside the function code. For example, data for determining if a URL should be redirected and which URL to redirect the viewer to. When embedding configuration data with the function code, every small change in configuration requires a code change and a redeployment of the function code. Updating and deploying code for every new lookup addition introduces the risk of making inadvertent changes to code. Also, the maximum function size is 10 KB, making it difficult for many use cases to fit all the data within the code.

With CloudFront KeyValueStore, you can now update the data associated with a function and the function code independently from each other. This simplifies function code and makes it easy to update data without the need to deploy code changes.

Let’s see how this works in practice.

Creating a CloudFront key value store
In the CloudFront console, I choose Functions from the navigation pane. In the KeyValueStores tab, I choose Create KeyValueStore.

Here, I have the option to import key value pairs from a JSON file in an Amazon Simple Storage Service (Amazon S3) bucket. I am not doing that now because I want to start with no keys. I enter a name and description and complete the creation of the key value store.

Console screenshot.

When the key value store has been created, I choose Edit in the Key value pairs section and then Add pair. I type hello for the key and Hello World for the value and save the changes. I can add more keys and values, but one key is enough for now.

Console screenshot.

When I update a key value store, changes are propagated to all CloudFront edge locations in a few seconds so that it can be used with low latency by the functions that are associated with the key value store. Let’s see how that works.

Using CloudFront KeyValueStore from CloudFront Functions
In the CloudFront console, I choose Functions in the navigation pane and then Create function. I type a name for the function, select the cloudfront-js-2.0 runtime, and complete the creation of the function. Then, I use the new option to associate the key value store with this function.

Console screenshot.

I copy the key value store ID from the console to use it in the following function code:

import cf from 'cloudfront';

const kvsId = '<KEY_VALUE_STORE_ID>';

// This fails if the key value store is not associated with the function
const kvsHandle = cf.kvs(kvsId);

async function handler(event) {
    // Use the first part of the pathname as key, for example http(s)://domain/<key>/something/else
    const key = event.request.uri.split('/')[1]
    let value = "Not found" // Default value
    try {
        value = await kvsHandle.get(key);
    } catch (err) {
        console.log(`Kvs key lookup failed for ${key}: ${err}`);
    }
    var response = {
        statusCode: 200,
        statusDescription: 'OK',
        body: {
            encoding: 'text',
            data: `Key: ${key} Value: ${value}\n`
        }
    };
    return response;
}

This function uses the first part of the path of the request as key and responds with the name of the key and its value.

I save the changes and publish the function. In the Publish tab of the function, I associate the function with a CloudFront distribution that I created before. I use the Viewer Request event type and Default (*) cache behavior to intercept all requests to the distribution.

In the console, I go back to the list of functions and wait for the function to be deployed. Then, I use curl from the command line to download content from the distribution and test the result of the function.

First, I try with a couple of paths that invoke the function and look up the key I created before (hello):

curl https://distribution-domain.cloudfront.net/hello
Key: hello Value: Hello World

curl https://distribution-domain.cloudfront.net/hello/world
Key: hello Value: Hello World

It works! Then, I try with a different path to see that the default value I use in the code is returned when the key is not found.

curl https://distribution-domain.cloudfront.net/hi
Key: hi Value: Not found

Now that this first example works, let’s try something more advanced and useful.

Rewriting the URL using configuration data in CloudFront KeyValueStore
Let’s build a function that uses the content of the URL in the HTTP request to look up in a key value store the custom path that CloudFront should use to make the actual request. This function can help manage the multiple services that are part of a website.

For example, I want to update the blog platform I use for my website. The old blog has origin path /blog-v1 while the new blog has origin path /blog-v2.

Architectural diagram.

At first, I am still using the old blog. In the CloudFormation console, I add the blog key to the key value store with value blog-v1.

Then, I create the following function and associate it with the distribution using Viewer Request event and Default (*) cache behavior to intercept all requests to the distribution.

import cf from 'cloudfront';

const kvsId = "<KEY_VALUE_STORE_ID>";

// This fails if the key value store is not associated with the function
const kvsHandle = cf.kvs(kvsId);

async function handler(event) {
    const request = event.request;
    // Use the first segment of the pathname as key
    // For example http(s)://domain/<key>/something/else
    const pathSegments = request.uri.split('/')
    const key = pathSegments[1]
    try {
        // Replace the first path of the pathname with the value of the key
        // For example http(s)://domain/<value>/something/else
        pathSegments[1] = await kvsHandle.get(key);
        const newUri = pathSegments.join('/');
        console.log(`${request.uri} -> ${newUri}`)
        request.uri = newUri;
    } catch (err) {
        // No change to the pathname if the key is not found
        console.log(`${request.uri} | ${err}`);
    }
    return request;
}

Now, when I type blog at the beginning of the URL path, the request will actually go to the blog-v1 path. CloudFront will make the HTTP request to the old blog because blog-v1 is the origin path used by the old blog.

For example, if I type https://distribution-domain.cloudfront.net/blog/index.html in a browser, I see the old blog (V1).

Browser screenshot showing blog V1.

In the console, I update the blog key with value blog-v2. I access the same URL after a few seconds, and now I reach the new blog (V2).

Browser screenshot showing blog V2.

As you can see, the public URL is the same, but the content has changed. More generally, this function assumes that URLs do not change between the two blog versions.

I can now add more keys for the different services that are part of my website (blog, support, help, commerce, and so on) and set their values to use the correct URL path for each of them. When I add a new version for one of them (for example, I migrate to a new commerce platform), I can configure a new origin and update the corresponding key to use the new origin path.

This is just an example of the flexibility you get when you separate configuration data from code. If you are already using CloudFront Functions, you can simplify your code by using CloudFront KeyValueStore.

Things to know
CloudFront KeyValueStore is available today in all edge locations globally. With CloudFront KeyValueStore, you pay only for what you use based on the read/write operations from the public API and the read operations from within CloudFront Functions. For more information, see CloudFront pricing.

You can manage a key value store using the AWS Management Console, AWS Command Line Interface (AWS CLI), and AWS SDKs. AWS CloudFormation support is coming soon. The maximum size of a key value store is 5 MB, and you can associate a single key value store to each function. The maximum size of a key is 512 bytes. Values can be up to 1KB in size. When creating a key value store, you can import key/value data during creation using a source file on Amazon S3 with this JSON structure:

{
  "data":[
    {
      "key":"key1",
      "value":"val1"
    },
    {
      "key":"key2",
      "value":"val2"
    }
  ]
}

Importing key/value data at creation can help automate the setup of a new environment (such as test or dev) and easily replicate the configuration from one environment to another (such as preproduction to production).

Simplify the way you add custom logic at the edge using CloudFront KeyValueStore.

— Danilo

Your DevOps and Developer Productivity guide to re:Invent 2023

Post Syndicated from Anubhav Rao original https://aws.amazon.com/blogs/devops/your-devops-and-developer-productivity-guide-to-reinvent-2023/

Your DevOps and Developer Productivity guide to re:Invent 2023

ICYMI – AWS re:Invent is less than a week away! We can’t wait to join thousands of builders in person and virtually for another exciting event. Still need to save your spot? You can register here.

With so much planned for the DevOps and Developer Productivity (DOP) track at re:Invent, we’re highlighting the most exciting sessions for technology leaders and developers in this post. Sessions span intermediate (200) through expert (400) levels of content in a mix of interactive chalk talks, hands-on workshops, and lecture-style breakout sessions.

You will experience the future of efficient development at the DevOps and Developer Productivity track and get a chance to talk to AWS experts about exciting services, tools, and new AI capabilities that optimize and automate your software development lifecycle. Attendees will leave re:Invent with the latest strategies to accelerate development, use generative AI to improve developer productivity, and focus on high-value work and innovation.

How to reserve a seat in the sessions

Reserved seating is available for registered attendees to secure seats in the sessions of their choice. Reserve a seat by signing in to the attendee portal and navigating to Event, then Sessions.

Do not miss the Innovation Talk led by Vice President of AWS Generative Builders, Adam Seligman. In DOP225-INT Build without limits: The next-generation developer experience at AWS, Adam will provide updates on the latest developer tools and services, including generative AI-powered capabilities, low-code abstractions, cloud development, and operations. He’ll also welcome special guests to lead demos of key developer services and showcase how they integrate to increase productivity and innovation.

DevOps and Developer Productivity breakout sessions

What are breakout sessions?

AWS re:Invent breakout sessions are lecture-style and 60 minutes long. These sessions are delivered by AWS experts and typically reserve 10–15 minutes for Q&A at the end. Breakout sessions are recorded and made available on-demand after the event.

Level 200 — Intermediate

DOP201 | Best practices for Amazon CodeWhisperer Generative AI can create new content and ideas, including conversations, stories, images, videos, and music. Learning how to interact with generative AI effectively and proficiently is a skill worth developing. Join this session to learn about best practices for engaging with Amazon CodeWhisperer, which uses an underlying foundation model to radically improve developer productivity by generating code suggestions in real time.

DOP202 | Realizing the developer productivity benefits of Amazon CodeWhisperer Developers spend a significant amount of their time writing undifferentiated code. Amazon CodeWhisperer radically improves productivity by generating code suggestions in real time to alleviate this burden. In this session, learn how CodeWhisperer can “write” much of this undifferentiated code, allowing developers to focus on business logic and accelerate the pace of their innovation.

DOP205 | Accelerate development with Amazon CodeCatalyst In this session, explore the newest features in Amazon CodeCatalyst. Learn firsthand how these practical additions to CodeCatalyst can simplify application delivery, improve team collaboration, and speed up the software development lifecycle from concept to deployment.

DOP206 | AWS infrastructure as code: A year in review AWS provides services that help with the creation, deployment, and maintenance of application infrastructure in a programmatic, descriptive, and declarative way. These services help provide rigor, clarity, and reliability to application development. Join this session to learn about the new features and improvements for AWS infrastructure as code with AWS CloudFormation and AWS Cloud Development Kit (AWS CDK) and how they can benefit your team.

DOP207 | Build and run it: Streamline DevOps with machine learning on AWS While organizations have improved how they deliver and operate software, development teams still run into issues when performing manual code reviews, looking for hard-to-find defects, and uncovering security-related problems. Developers have to keep up with multiple programming languages and frameworks, and their productivity can be impaired when they have to search online for code snippets. Additionally, they require expertise in observability to successfully operate the applications they build. In this session, learn how companies like Fidelity Investments use machine learning–powered tools like Amazon CodeWhisperer and Amazon DevOps Guru to boost application availability and write software faster and more reliably.

DOP208 | Continuous integration and delivery for AWS AWS provides one place where you can plan work, collaborate on code, build, test, and deploy applications with continuous integration/continuous delivery (CI/CD) tools. In this session, learn about how to create end-to-end CI/CD pipelines using infrastructure as code on AWS.

DOP209 | Governance and security with infrastructure as code In this session, learn how to use AWS CloudFormation and the AWS CDK to deploy cloud applications in regulated environments while enforcing security controls. Find out how to catch issues early with cdk-nag, validate your pipelines with cfn-guard, and protect your accounts from unintended changes with CloudFormation hooks.

DOP210 | Scale your application development with Amazon CodeCatalyst Amazon CodeCatalyst brings together everything you need to build, deploy, and collaborate on software into one integrated software development service. In this session, discover the ways that CodeCatalyst helps developers and teams build and ship code faster while spending more time doing the work they love.

DOP211 | Boost developer productivity with Amazon CodeWhisperer Generative AI is transforming the way that developers work. Writing code is already getting disrupted by tools like Amazon CodeWhisperer, which enhances developer productivity by providing real-time code completions based on natural language prompts. In this session, get insights into how to evaluate and measure productivity with the adoption of generative AI–powered tools. Learn from the AWS Disaster Recovery team who uses CodeWhisperer to solve complex engineering problems by gaining efficiency through longer productivity cycles and increasing velocity to market for ongoing fixes. Hear how integrating tools like CodeWhisperer into your workflows can boost productivity.

DOP212 | New AWS generative AI features and tools for developers Explore how generative AI coding tools are changing the way developers and companies build software. Generative AI–powered tools are boosting developer and business productivity by automating tasks, improving communication and collaboration, and providing insights that can inform better decision-making. In this session, see the newest AWS tools and features that make it easier for builders to solve problems with minimal technical expertise and that help technical teams boost productivity. Walk through how organizations like FINRA are exploring generative AI and beginning their journey using these tools to accelerate their pace of innovation.

DOP220 | Simplify building applications with AWS SDKs AWS SDKs play a vital role in using AWS services in your organization’s applications and services. In this session, learn about the current state and the future of AWS SDKs. Explore how they can simplify your developer experience and unlock new capabilities. Discover how SDKs are evolving, providing a consistent experience in multiple languages and empowering you to do more with high-level abstractions to make it easier to build on AWS. Learn how AWS SDKs are built using open source tools like Smithy, and how you can use these tools to build your own SDKs to serve your customers’ needs.

DevOps and Developer Productivity chalk talks

What are chalk talks?

Chalk Talks are highly interactive sessions with a small audience. Experts lead you through problems and solutions on a digital whiteboard as the discussion unfolds. Each begins with a short lecture (10–15 minutes) delivered by an AWS expert, followed by a 45- or 50-minute Q&A session with the audience.

Level 300 — Advanced

DOP306 | Streamline DevSecOps with a complete software development service Security is not just for application code—the automated software supply chains that build modern software can also be exploited by attackers. In this chalk talk, learn how you can use Amazon CodeCatalyst to incorporate security tests into every aspect of your software development lifecycle while maintaining a great developer experience. Discover how CodeCatalyst’s flexible actions-based CI/CD workflows streamline the process of adapting to security threats.

DOP309-R | AI for DevOps: Modernizing your DevOps operations with AWS As more organizations move to microservices architectures to scale their businesses, applications increasingly have become distributed, requiring the need for even greater visibility. IT operations professionals and developers need more automated practices to maintain application availability and reduce the time and effort required to detect, debug, and resolve operational issues. In this chalk talk, discover how you can use AWS services, including Amazon CodeWhisperer, Amazon CodeGuru and Amazon DevOps Guru, to start using AI for DevOps solutions to detect, diagnose, and remedy anomalous application behavior.

DOP310-R | Better together: GitHub Actions, Amazon CodeCatalyst, or AWS CodeBuild Learn how combining GitHub Actions with Amazon CodeCatalyst or AWS CodeBuild can maximize development efficiency. In this chalk talk, learn about the tradeoffs of using GitHub Actions runners hosted on Amazon EC2 or Amazon ECS with GitHub Actions hosted on CodeCatalyst or CodeBuild. Explore integration with other AWS services to enhance workflow automation. Join this talk to learn how GitHub Actions on AWS can take your development processes to the next level.

DOP311 | Building infrastructure as code with AWS CloudFormation AWS CloudFormation helps you manage your AWS infrastructure as code, increasing automation and supporting infrastructure-as-code best practices. In this chalk talk, learn the fundamentals of CloudFormation, including templates, stacks, change sets, and stack dependencies. See a demo of how to describe your AWS infrastructure in a template format and provision resources in an automated, repeatable way.

DOP312 | Creating custom constructs with AWS CDK Join this chalk talk to get answers to your questions about creating, publishing, and sharing your AWS CDK constructs publicly and privately. Learn about construct levels, how to test your constructs, how to discover and use constructs in your AWS CDK projects, and explore Construct Hub.

DOP313-R | Multi-account and multi-Region deployments at scale Many AWS customers are implementing multi-account strategies to more easily manage their cloud infrastructure and improve their security and compliance postures. In this chalk talk, learn about various options for deploying resources into multiple accounts and AWS Regions using AWS developer tools, including AWS CodePipeline, AWS CodeDeploy, and Amazon CodeCatalyst.

DOP314 | Simplifying cloud infrastructure creation with the AWS CDK The AWS Cloud Development Kit (AWS CDK) is an open source software development framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. In this chalk talk, get an introduction to the AWS CDK and see a demo of how it can simplify infrastructure creation. Through code examples and diagrams, see how the AWS CDK lets you use familiar programming languages for declarative infrastructure definition. Also learn how it provides higher-level abstractions and constructs over native CloudFormation.

DOP317 | Applying Amazon’s DevOps culture to your team In this chalk talk, learn how Amazon helps its developers rapidly release and iterate software while maintaining industry-leading standards on security, reliability, and performance. Learn about the culture of two-pizza teams and how to maintain a culture of DevOps in a large enterprise. Also, discover how you can help build such a culture at your own organization.

DOP318 | Testing for resilience with AWS Fault Injection Simulator As cloud-based systems grow in scale and complexity, there is increased need to test distributed systems for resiliency. AWS Fault Injection Simulator (FIS) allows you to stress test your applications to understand failure modes and build more resilient services. Through code examples and diagrams, see how to set up and run fault injection experiments on AWS. By the end of this session, understand how FIS helps identify weaknesses and validate improvements to build more resilient cloud-based systems.

DOP319-R | Zero-downtime deployment strategies AWS services support a wealth of deployment options to meet your needs, ranging from in-place updates to blue/green deployment to continuous configuration with feature flags. In this chalk talk, hear about multiple options for deploying changes to Amazon EC2, Amazon ECS, and AWS Lambda compute platforms using AWS CodeDeploy, AWS AppConfig, AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), and Amazon CodeCatalyst.

DOP320 | Build a path to production with Amazon CodeCatalyst blueprints Amazon CodeCatalyst uses blueprints to configure your software projects in the service. Blueprints instruct CodeCatalyst on how to set up a code repository with working sample code, define cloud infrastructure, and run pre-configured CI/CD workflows for your project. In this session, learn how blueprints in CodeCatalyst can give developers a compliant software service they’ll want to use on AWS.

DOP321-R | Code faster with Amazon CodeWhisperer Traditionally, building applications requires developers to spend a lot of time manually writing code and trying to learn and keep up with new frameworks, SDKs, and libraries. In the last three years, AI models have grown exponentially in complexity and sophistication, enabling the creation of tools like Amazon CodeWhisperer that can generate code suggestions in real time based on a natural language description of the task. In this session, learn how CodeWhisperer can accelerate and enhance your software development with code generation, reference tracking, security scans, and more.

DOP324 | Accelerating application development with AWS client-side tools Did you know AWS has more than just services? There are dozens of AWS client-side tools and libraries designed to make developing quality applications easier. In this chalk talk, explore some of the tools available in your development workspace. Learn more about command line tooling (AWS CLI), libraries (AWS SDK), IDE integrations, and application frameworks that can accelerate your AWS application development. The audience helps set the agenda so there’s sure to be something for every builder.

DevOps and Developer Productivity workshops

What are workshops?

Workshops are two-hour interactive learning sessions where you work in small group teams to solve problems using AWS services. Each workshop starts with a short lecture (10–15 minutes) by the main speaker, and the rest of the time is spent working as a group.

Level 300 — Advanced

DOP301 | Boost your application availability with AIOps on AWS As applications become increasingly distributed and complex, developers and IT operations teams can benefit from more automated practices to maintain application availability and reduce the time and effort spent detecting, debugging, and resolving operational issues manually. In this workshop, learn how AWS AIOps solutions can help you make the shift toward more automation and proactive mechanisms so your IT team can innovate faster. The workshop includes use cases spanning multiple AWS services such as AWS Lambda, Amazon DynamoDB, Amazon API Gateway, Amazon RDS, and Amazon EKS. Learn how you can reduce MTTR and quickly identify issues within your AWS infrastructure. You must bring your laptop to participate.

DOP302 | Build software faster with Amazon CodeCatalyst In this workshop, learn about creating continuous integration and continuous delivery (CI/CD) pipelines using Amazon CodeCatalyst. CodeCatalyst is a unified software development service on AWS that brings together everything teams need to plan, code, build, test, and deploy applications with continuous CI/CD tools. You can utilize AWS services and integrate AWS resources into your projects by connecting your AWS accounts. With all of the stages of an application’s lifecycle in one tool, you can deliver quality software quickly and confidently. You must bring your laptop to participate.

DOP303-R | Continuous integration and delivery on AWS In this workshop, learn to create end-to-end continuous integration and continuous delivery (CI/CD) pipelines using AWS Cloud Development Kit (AWS CDK). Review the fundamental concepts of continuous integration, continuous deployment, and continuous delivery. Then, using TypeScript/Python, define an AWS CodePipeline, AWS CodeBuild, and AWS CodeCommit workflow. You must bring your laptop to participate.

DOP304 | Develop AWS CDK resources to deploy your applications on AWS In this workshop, learn how to build and deploy applications using infrastructure as code with AWS Cloud Development Kit (AWS CDK). Create resources using AWS CDK and learn maintenance and operations tips. In addition, get an introduction to building your own constructs. You must bring your laptop to participate.

DOP305 | Develop AWS CloudFormation templates to manage your infrastructure In this workshop, learn how to develop and test AWS CloudFormation templates. Create CloudFormation templates to deploy and manage resources and learn about CloudFormation language features that allow you to reuse and extend templates for many scenarios. Explore testing tools that can help you validate your CloudFormation templates, including cfn-lint and CloudFormation Guard. You must bring your laptop to participate.

DOP307-R | Hands-on with Amazon CodeWhisperer In this workshop, learn how to build applications faster and more securely with Amazon CodeWhisperer. The workshop begins with several examples highlighting how CodeWhisperer incorporates your comments and existing code to produce results. Then dive into a series of challenges designed to improve your productivity using multiple languages and frameworks. You must bring your laptop to participate.

DOP308 | Enforcing development standards with Amazon CodeCatalyst In this workshop, learn how Amazon CodeCatalyst can accelerate the application development lifecycle within your organization. Discover how your cloud center of excellence (CCoE) can provide standardized code and workflows to help teams get started quickly and securely. In addition, learn how to update projects as organization standards evolve. You must bring your laptop to participate.

Level 400 — Expert

DOP401 | Get better at building AWS CDK constructs In this workshop, dive deep into how to design AWS CDK constructs, which are reusable and shareable cloud components that help you meet your organization’s security, compliance, and governance requirements. Learn how to build, test, and share constructs representing a single AWS resource, as well as how to create higher-level abstractions that include built-in defaults and allow you to provision multiple AWS resources. You must bring your laptop to participate.

DevOps and Developer Productivity builders’ sessions

What are builders’ sessions?

These 60-minute group sessions are led by an AWS expert and provide an interactive learning experience for building on AWS. Builders’ sessions are designed to create a hands-on experience where questions are encouraged.

Level 300 — Advanced

DOP322-R | Accelerate data science coding with Amazon CodeWhisperer Generative AI removes the heavy lifting that developers experience today by writing much of the undifferentiated code, allowing them to build faster. Helping developers code faster could be one of the most powerful uses of generative AI that we will see in the coming years—and this framework can also be applied to data science projects. In this builders’ session, explore how Amazon CodeWhisperer accelerates the completion of data science coding tasks with extensions for JupyterLab and Amazon SageMaker. Learn how to build data processing pipeline and machine learning models with the help of CodeWhisperer and accelerate data science experiments in Python. You must bring your laptop to participate.

Level 400 — Expert

DOP402-R | Manage dev environments at scale with Amazon CodeCatalyst Amazon CodeCatalyst Dev Environments are cloud-based environments that you can use to quickly work on the code stored in the source repositories of your project. They are automatically created with pre-installed dependencies and language-specific packages so you can work on a new or existing project right away. In this session, learn how to create secure, reproducible, and consistent environments for VS Code, AWS Cloud9, and JetBrains IDEs. You must bring your laptop to participate.

DOP403-R | Hands-on with Amazon CodeCatalyst: Automating security in CI/CD pipelines In this session, learn how to build a CI/CD pipeline with Amazon CodeCatalyst and add the necessary steps to secure your pipeline. Learn how to perform tasks such as secret scanning, software composition analysis (SCA), static application security testing (SAST), and generating a software bill of materials (SBOM). You must bring your laptop to participate.

DevOps and Developer Productivity lightning talks

What are lightning talks?

Lightning talks are short, 20-minute demos led from a stage.

DOP221 | Amazon CodeCatalyst in real time: Deploying to production in minutes In this follow-up demonstration to DOP210, see how you can use an Amazon CodeCatalyst blueprint to build a production-ready application that is set up for long-term success. See in real time how to create a project using a CodeCatalyst Dev Environment and deploy it to production using a CodeCatalyst workflow.

DevOps and Developer Productivity code talks

What are code talks?

Code talks are 60-minute, highly-interactive discussions featuring live coding. Attendees are encouraged to dig in and ask questions about the speaker’s approach.

DOP203 | The future of development on AWS This code talk includes a live demo and an open discussion about how builders can use the latest AWS developer tools and generative AI to build production-ready applications in minutes. Starting at an Amazon CodeCatalyst blueprint and using integrated AWS productivity and security capabilities, see a glimpse of what the future holds for developing on AWS.

DOP204 | Tips and tricks for coding with Amazon CodeWhisperer Generative AI tools that can generate code suggestions, such as Amazon CodeWhisperer, are growing rapidly in popularity. Join this code talk to learn how CodeWhisperer can accelerate and enhance your software development with code generation, reference tracking, security scans, and more. Learn best practices for prompt engineering, and get tips and tricks that can help you be more productive when building applications.

Want to stay connected?

Get the latest updates for DevOps and Developer Productivity by following us on Twitter and visiting the AWS devops blog.

Build scalable and serverless RAG workflows with a vector engine for Amazon OpenSearch Serverless and Amazon Bedrock Claude models

Post Syndicated from Fraser Sequeira original https://aws.amazon.com/blogs/big-data/build-scalable-and-serverless-rag-workflows-with-a-vector-engine-for-amazon-opensearch-serverless-and-amazon-bedrock-claude-models/

In pursuit of a more efficient and customer-centric support system, organizations are deploying cutting-edge generative AI applications. These applications are designed to excel in four critical areas: multi-lingual support, sentiment analysis, personally identifiable information (PII) detection, and conversational search capabilities. Customers worldwide can now engage with the applications in their preferred language, and the applications can gauge their emotional state, mask sensitive personal information, and provide context-aware responses. This holistic approach not only enhances the customer experience but also offers efficiency gains, ensures data privacy compliance, and drives customer retention and sales growth.

Generative AI applications are poised to transform the customer support landscape, offering versatile solutions that integrate seamlessly with organizations’ operations. By combining the power of multi-lingual support, sentiment analysis, PII detection, and conversational search, these applications promise to be a game-changer. They empower organizations to deliver personalized, efficient, and secure support services while ultimately driving customer satisfaction, cost savings, data privacy compliance, and revenue growth.

Amazon Bedrock and foundation models like Anthropic Claude are poised to enable a new wave of AI adoption by powering more natural conversational experiences. However, a key challenge that has emerged is tailoring these general purpose models to generate valuable and accurate responses based on extensive, domain-specific datasets. This is where the Retrieval Augmented Generation (RAG) technique plays a crucial role.

RAG allows you to retrieve relevant data from databases or document repositories to provide helpful context to large language models (LLMs). This additional context helps the models generate more specific, high-quality responses tuned to your domain.

In this post, we demonstrate building a serverless RAG workflow by combining the vector engine for Amazon OpenSearch Serverless with an LLM like Anthropic Claude hosted by Amazon Bedrock. This combination provides a scalable way to enable advanced natural language capabilities in your applications, including the following:

  • Multi-lingual support – The solution uses the ability of LLMs like Anthropic Claude to understand and respond to queries in multiple languages without any additional training needed. This provides true multi-lingual capabilities out of the box, unlike traditional machine learning (ML) systems that need training data in each language.
  • Sentiment analysis – This solution enables you to detect positive, negative, or neutral sentiment in text inputs like customer reviews, social media posts, or surveys. LLMs can provide explanations for the inferred sentiment, describing which parts of the text contributed to a positive or negative classification. This explainability helps build trust in the model’s predictions. Potential use cases could include analyzing product reviews to identify pain points or opportunities, monitoring social media for brand sentiment, or gathering feedback from customer surveys.
  • PII detection and redaction – The Claude LLM can be accurately prompted to identify various types of PII like names, addresses, Social Security numbers, and credit card numbers and replace it with placeholders or generic values while maintaining readability of the surrounding text. This enables compliance with regulations like GDPR and prevents sensitive customer data from being exposed. This also helps automate the labor-intensive process of PII redaction and reduces risk of exposed customer data across various use cases, such as the following:
    • Processing customer support tickets and automatically redacting any PII before routing to agents.
    • Scanning internal company documents and emails to flag any accidental exposure of customer PII.
    • Anonymizing datasets containing PII before using the data for analytics or ML, or sharing the data with third parties.

Through careful prompt engineering, you can accomplish the aforementioned use cases with a single LLM. The key is crafting prompt templates that clearly articulate the desired task to the model. Prompting allows us to tap into the vast knowledge already present within the LLM for advanced natural language processing (NLP) tasks, while tailoring its capabilities to our particular needs. Well-designed prompts unlock the power and potential of the model.

With the vector database capabilities of Amazon OpenSearch Serverless, you can store vector embeddings of documents, allowing ultra-fast, semantic (rather than keyword) similarity searches to find the most relevant passages to augment prompts.

Read on to learn how to build your own RAG solution using an OpenSearch Serverless vector database and Amazon Bedrock.

Solution overview

The following architecture diagram provides a scalable and fully managed RAG-based workflow for a wide range of generative AI applications, such as language translation, sentiment analysis, PII data detection and redaction, and conversational AI. This pre-built solution operates in two distinct stages. The initial stage involves generating vector embeddings from unstructured documents and saving these embeddings within an OpenSearch Serverless vectorized database index. In the second stage, user queries are forwarded to the Amazon Bedrock Claude model along with the vectorized context to deliver more precise and relevant responses.

In the following sections, we discuss the two core functions of the architecture in more detail:

  • Index domain data
  • Query an LLM with enhanced context

Index domain data

In this section, we discuss the details of the data indexing phase.

Generate embeddings with Amazon Titan

We used Amazon Titan embeddings model to generate vector embeddings. With 1,536 dimensions, the embeddings model captures semantic nuances in meaning and relationships. Embeddings are available via the Amazon Bedrock serverless experience; you can access it using a single API and without managing any infrastructure. The following code illustrates generating embeddings using a Boto3 client.

import boto3
bedrock_client = boto3.client('bedrock-runtime')

## Generate embeddings with Amazon Titan Embeddings model
response = bedrock_client.invoke_model(
            body = json.dumps({"inputText": 'Hello World'}),
            modelId = 'amazon.titan-embed-text-v1',
            accept = 'application/json',
            contentType = 'application/json'
)
result = json.loads(response['body'].read())
embeddings = result.get('embedding')
print(f'Embeddings -> {embeddings}')

Store embeddings in an OpenSearch Serverless vector collection

OpenSearch Serverless offers a vector engine to store embeddings. As your indexing and querying needs fluctuate based on workload, OpenSearch Serverless automatically scales up and down based on demand. You no longer have to predict capacity or manage infrastructure sizing.

With OpenSearch Serverless, you don’t provision clusters. Instead, you define capacity in the form of Opensearch Capacity Units (OCUs). OpenSearch Serverless will scale up to the maximum number of OCUs defined. You’re charged for a minimum of 4 OCUs, which can be shared across multiple collections sharing the same AWS Key Management Service (AWS KMS) key.

The following screenshot illustrates how to configure capacity limits on the OpenSearch Serverless console.

Query an LLM with domain data

In this section, we discuss the details of the querying phase.

Generate query embeddings

When a user queries for data, we first generate an embedding of the query with Amazon Titan embeddings. OpenSearch Serverless vector collections employ an Approximate Nearest Neighbors (A-NN) algorithm to find document embeddings closest to the query embeddings. The A-NN algorithm uses cosine similarity to measure the closeness between the embedded user query and the indexed data. OpenSearch Serverless then returns the documents whose embeddings have the smallest distance, and therefore the highest similarity, to the user’s query embedding. The following code illustrates our vector search query:

vector_query = {
                "size": 5,
                "query": {"knn": {"embedding": {"vector": embedded_search, "k": 2}}},
                "_source": False,
                "fields": ["text", "doc_type"]
            }

Query Anthropic Claude models on Amazon Bedrock

OpenSearch Serverless finds relevant documents for a given query by matching embedded vectors. We enhance the prompt with this context and then query the LLM. In this example, we use the AWS SDK for Python (Boto3) to invoke models on Amazon Bedrock. The AWS SDK provides the following APIs to interact with foundational models on Amazon Bedrock:

The following code invokes our LLM:

import boto3
bedrock_client = boto3.client('bedrock-runtime')
# model_id could be 'anthropic.claude-v2', 'anthropic.claude-v1','anthropic.claude-instant-v1']
response = bedrock_client.invoke_model_with_response_stream(
        body=json.dumps(prompt),
        modelId=model_id,
        accept='application/json',
        contentType='application/json'
    )

Prerequisites

Before you deploy the solution, review the prerequisites.

Deploy the solution

The code sample along with the deployment steps are available in the GitHub repository. The following screenshot illustrates deploying the solution using AWS CloudShell.

Test the solution

The solution provides some sample data for indexing, as shown in the following screenshot. You can also index custom text. Initial indexing of documents may take some time because OpenSearch Serverless has to create a new vector index and then index documents. Subsequent requests are faster. To delete the vector index and start over, choose Reset.

The following screenshot illustrates how you can query your domain data in multiple languages after it’s indexed. You could also try out sentiment analysis or PII data detection and redaction on custom text. The response is streamed over Amazon API Gateway WebSockets.

Clean up

To clean up your resources, delete the following AWS CloudFormation stacks via the AWS CloudFormation console:

  • LlmsWithServerlessRagStack
  • ApiGwLlmsLambda

Conclusion

In this post, we provided an end-to-end serverless solution for RAG-based generative AI applications. This not only offers you a cost-effective option, particularly in the face of GPU cost and hardware availability challenges, but also simplifies the development process and reduces operational costs.

Stay up to date with the latest advancements in generative AI and start building on AWS. If you’re seeking assistance on how to begin, check out the Generative AI Innovation Center.

About the authors

Fraser Sequeira is a Startups Solutions Architect with AWS based in Mumbai, India. In his role at AWS, Fraser works closely with startups to design and build cloud-native solutions on AWS, with a focus on analytics and streaming workloads. With over 10 years of experience in cloud computing, Fraser has deep expertise in big data, real-time analytics, and building event-driven architecture on AWS. He enjoys staying on top of the latest technology innovations from AWS and sharing his learnings with customers. He spends his free time tinkering with new open source technologies.

Kenneth Walsh is a New York-based Sr. Solutions Architect whose focus is AWS Marketplace. Kenneth is passionate about cloud computing and loves being a trusted advisor for his customers. When he’s not working with customers on their journey to the cloud, he enjoys cooking, audiobooks, movies, and spending time with his family and dog.

Max Winter is a Principal Solutions Architect for AWS Financial Services clients. He works with ISV customers to design solutions that allow them to leverage the power of AWS services to automate and optimize their business. In his free time, he loves hiking and biking with his family, music and theater, digital photography, 3D modeling, and imparting a love of science and reading to his two nearly-teenagers.

Manjula Nagineni is a Senior Solutions Architect with AWS based in New York. She works with major financial service institutions, architecting and modernizing their large-scale applications while adopting AWS Cloud services. She is passionate about designing big data workloads cloud-natively. She has over 20 years of IT experience in software development, analytics, and architecture across multiple domains such as finance, retail, and telecom.

The collective thoughts of the interwebz