Use scalable controls to help prevent access from unexpected networks

Post Syndicated from Sowjanya Rajavaram original https://aws.amazon.com/blogs/security/use-scalable-controls-to-help-prevent-access-from-unexpected-networks/

As your organization grows, the amount of data you own and the number of data sources to store and process your data across multiple Amazon Web Services (AWS) accounts increases. Enforcing consistent access controls that restrict access to known networks might become a key part in protecting your organization’s sensitive data.

Previously, AWS customers could rely on AWS Identity and Access Management (IAM) global condition keys such as aws:SourceVpc and aws:SourceVpce to restrict access to specific virtual private clouds (VPCs) or VPC endpoints. These condition keys work well for organizations with few accounts and for use cases limited to specific workloads. However, as the number of your VPCs grow, using these keys could introduce challenges in scaling the control across a large set of resources.

To address this challenge, AWS has introduced three new global condition keys for scalable access controls based on request origin: aws:VpceAccount, aws:VpceOrgPaths, and aws:VpceOrgID.

In this blog post, we demonstrate how these keys can help make sure that your AWS resources are accessible only from expected VPCs, so that you can scale your data perimeter implementation across your organization within AWS Organizations.

Background

Organizations often store data in AWS resources such as Amazon Simple Storage Service (Amazon S3) buckets. For example, you might use Amazon S3 as your data lake foundation with data scientists and analysts running their data processing and analytics workflows against data stored in a centralized S3 bucket.

To limit access to data stored in your S3 buckets to expected networks, you can use IAM policies associated with your identities and resources. You can define expected networks in a policy using specific IAM global condition keys based on your organization’s intended data access patterns and unique requirements. For example, use aws:SourceIp to specify your corporate IP CIDR ranges, and aws:SourceVpc or aws:SourceVpce to list VPC and VPC endpoint IDs you expect requests to come from. These condition keys help make sure that only workloads operating within your expected network boundaries can access sensitive data.

However, there are scenarios where you might want to allow access from multiple networks within your organization, as illustrated in Figure 1.

Figure 1: Applications and users accessing an S3 bucket from VPCs and public networks

Figure 1: Applications and users accessing an S3 bucket from VPCs and public networks

In such cases, using the aws:SourceVpc and aws:SourceVpce condition keys requires enumerating all expected VPC and VPC endpoint IDs and updating policies whenever new VPCs or VPC endpoints are added or deleted. This approach creates operational overhead and increases the risk of misconfigurations. The operational complexity grows as organizations scale their data processing capacity across multiple AWS Regions and accounts. While many organizations have developed automated mechanisms to detect changes in VPC configurations and update policies accordingly, auditing lengthy policies that enumerate VPCs within their organization remains challenging.

The new global condition keys provide a more scalable way to restrict access to expected networks:

  • aws:VpceAccount – Restricts the use of your identities and resources to networks that belong to a specific AWS account.
  • aws:VpceOrgPaths – Restricts the use of your identities and resources to networks that belong to a specific organizational unit (OU) in your organization.
  • aws:VpceOrgID – Restricts the use of your identities and resources to networks that belong to your organization.

The value of these keys in the request context is the ID of the account (for example, 111122223333), organization unit (OU) (for example, o-abcdef0123/r-acroot/ou-development/*), or organization (for example, o-abcdef0123) that owns the VPC endpoint the request is made through.

You can use the preceding keys in relevant IAM policies such as resource control policies (RCPs), service control policies (SCPs), session policies, permissions boundaries, identity-based policies, and resource-based policies.

Note that at the time of writing, not all services support these keys. See AWS global condition context keys for a list of supported services.

Implementation examples

Let’s look at how to restrict access to expected networks using the three new condition keys for common use cases. Each of the use cases demonstrates how the new condition keys help simplify controlling access to your resources in the sample scenario from Figure 1.

Use case 1: Allow access to your S3 buckets only from networks of data processing accounts

Data owners might want to strictly manage what data workflows can access their data sources and restrict cross-account access to specific data processing accounts and networks. They can use the aws:VpceAccount condition key to allow access based on the account that owns the VPC endpoint the request is made through. The following is an example S3 bucket policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowDataProcessingAccounts",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<Central-ETL-account-ID>:role/<ETLRoleName>",
          "arn:aws:iam::<Shared-analytics-account-ID>:role/<AnalyticsRoleName>",
          "arn:aws:iam::<ML-processing-account-ID>:role/<MLRoleName>"
        ]
      },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<Datalake-S3-bucket-name>",
        "arn:aws:s3:::<Datalake-S3-bucket-name>/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:VpceAccount": [
             "<Central-ETL-account-ID>",
             "<Shared-analytics-account-ID>",
             "<ML-processing-account-ID>"
          ]
        }
      }
    }
  ]
}

This policy allows specific principals listed in the Principal element to list and download objects from the data lake bucket but only if they make requests from networks in one of the specified AWS accounts (StringEquals and aws:VpceAccount). Using the aws:VpceAccount condition key in this policy alleviates the need to maintain a list of VPC IDs or VPC endpoint IDs for the data processing accounts, reduces the size of the policy document, and simplifies auditing.

Use case 2: Restricting access to company networks for resources across multiple accounts

Central security teams often look for ways to enforce a set of standard access controls on resources across their entire organization. This is to meet compliance and security requirements, fulfill legal and contractual obligations, and to protect corporate data from unintended access. One such control could be used to limit access to only expected networks within the organization. In our sample scenario, this control helps prevent your data analysts and scientists from using their credentials to access data outside of your corporate environment.
The following RCP demonstrates how to enforce the network perimeter controls on S3 buckets:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictAccessToOrgVPCs",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": "*",
      "Condition": {
        "NotIpAddressIfExists": {
          "aws:SourceIp": "<My-corporate-CIDR>"
        },
        "StringNotEqualsIfExists": {
          "aws:VpceOrgID": "<My-corporate-org-ID>",
          "aws:PrincipalTag/network-perimeter-exception": "true"
        },
        "BoolIfExists": {
          "aws:PrincipalIsAWSService": "false",
          "aws:ViaAWSService": "false"
        }
      }
    }
  ]
}

This policy denies access to S3 buckets and objects unless it is from expected networks defined as: your corporate IP CIDR range (NotIpAddressIfExists and aws:SourceIp), VPC endpoints in your organization (StringNotEqualsIfExists and aws:VpceOrgID), networks of AWS services that use their service principals or forward access sessions (FAS) to act on your behalf (BoolIfExists with aws:PrincipalIsAWSService and aws:ViaAWSService). It also allows access to networks of AWS services using specific service roles to access your resources (StringNotEqualsIfExists and aws:PrincipalTag/network-perimeter-exception set to true). Some organizations might need to edit this policy to allow third-party partner access. See Establishing a data perimeter on AWS: Allow access to company data only from expected networks for additional information on access patterns that need to be accounted for to meet the needs of your organization.

We used an RCP because it can be used to apply access controls centrally on resources across multiple accounts. Central security teams use RCPs to enforce security invariants on resources across their entire organization. For best practices in designing and deploying RCPs, see Effectively implementing resource control policies in a multi-account environment.

Remember to reference the list of services that support aws:VpceOrgID before using it in a policy such as an RCP. Enforcing it on an unsupported service might prevent your developers from using the service. If you need to restrict access to expected networks on a wider range of services, consider using the aws:SourceVpc and aws:SourceVpce condition keys. See the data perimeter policy examples repository that illustrate how to implement network perimeter controls for a wider range of services.

Use case 3: Restricting access based on intra-organization boundaries

Organizations often need to segment environments within their organization with varying data access requirements. For example, they might need to separate production from non-production environments or create boundaries between different business units, such as Finance, Marketing, and Sales; each operating in separate accounts. This might include making sure that resources within a specific OU can only be accessed from networks in the same OU. Central security teams can use aws:VpceOrgPaths to achieve this objective at scale.

The following is an example RCP that restricts access to your Amazon S3 and AWS Key Management Service (AWS KMS) resources so that they can only be accessed through VPC endpoints in a specific OU.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictAccessToOUVPCs",
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
          "s3:*",
          "kms:*"
      ],
      "Resource": "*",
      "Condition": {
        "NotIpAddressIfExists": {
          "aws:SourceIp": "<My-corporate-CIDR>"
        },
        "ForAllValues:StringNotLikeIfExists": {
          "aws:VpceOrgPaths": "<My-corporate-org-path>"
        },
       "StringNotEqualsIfExists": {
          "aws:PrincipalTag/network-perimeter-exception": "true"
        },
        "BoolIfExists": {
          "aws:PrincipalIsAWSService": "false",
          "aws:ViaAWSService": "false"
        }
      }
    }
  ]
}

This policy is similar to the one we built for the previous use case but uses aws:VpceOrgPaths instead of aws:VpceOrgID to enforce a more granular boundary based on the requests’ network origin.

Best practices and considerations

When implementing the new condition keys, consider the following best practices.

Identify opportunities to adopt the new global condition keys by reviewing your security objectives and controls

If you currently restrict access to a wide range of resources using the aws:SourceVpc and aws:SourceVpce condition keys and want to avoid the need to enumerate VPC or VPC endpoint IDs in your policies, evaluate if you can migrate to aws:VpceAccount, aws:VpceOrgPaths, or aws:VpceOrgID. This migration decision depends on whether services you restrict access to are supported by the new condition keys. Similarly, if you plan to add network perimeter restrictions to your security baseline, first evaluate whether the new condition keys offer a more scalable solution for your target services. Only enforce the new keys on services that are currently supported. If you need to enforce the restriction on a service not yet supported, you should use aws:SourceVpc and aws:SourceVpce. Also, continue using aws:SourceVpc and aws:SourceVpce to achieve your least privilege objectives, for example if the network boundary you need to maintain for a subset of resources is scoped to specific VPCs or VPC endpoints.

Plan the implementation of the new condition keys

We recommend that you test access controls updates in a non-production environment and only promote them to production after validating their expected behavior. If you currently maintain an automation to enumerate VPC or VPC endpoint IDs in your policies and plan to migrate to the new keys, deactivate your automation only after you have completed policy updates across all environments. This approach helps make sure that your existing security posture remains intact while you progressively deploy the changes.

Monitor and validate the implementation

Use AWS CloudTrail to audit access patterns and regularly review and update your access controls as your organization structure evolves and security objectives change. For example, you might need to adjust access controls when accounts requiring access to your data lakes change, or when organizational boundaries need modification to accommodate new integrations between business units. You must establish processes to continuously evaluate the effectiveness of your controls in meeting both security and business objectives.

Conclusion

In this post, you learned how to use the new global condition keys—aws:VpceAccount, aws:VpceOrgPaths, and aws:VpceOrgID—to restrict access to expected networks at scale. By using these keys, you can:

  • Implement network perimeter controls that scale with your AWS organization.
  • Reduce the operational overhead of managing access to your data.
  • Simplify your IAM policies and reduce the risk of misconfigurations.
  • Scale your data lake implementation while maintaining security.

For more information, see:

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on AWS IAM re:Post or contact AWS Support.


Sowjanya Rajavaram

Sowjanya Rajavaram

Sowjanya is a Sr Solution Architect who specializes in Identity and Security in AWS. Her entire career has been focused on helping customers of all sizes solve their identity and access management problems. She enjoys traveling and experiencing new cultures and food.

Tatyana Yatskevich

Tatyana Yatskevich

Tatyana is a Principal Solutions Architect in AWS Identity. She works with customers to help them build and operate in AWS in the most secure and efficient manner.

Python: The Documentary

Post Syndicated from jzb original https://lwn.net/Articles/1035537/

Attendees at EuroPython had the chance to preview part of
Python: The Documentary during a
keynote panel
. The full film, created by CultRepo, is now available on YouTube:

This is the story of the world’s most beloved programming language:
Python. What began as a side project in Amsterdam during the 1990s
became the software powering artificial intelligence, data science and
some of the world’s biggest companies. But Python’s future wasn’t
certain; at one point it almost disappeared.

This 90-minute documentary features Guido van Rossum, Travis
Oliphant, Barry Warsaw, and many more, and they tell the story of
Python’s rise, its community-driven evolution, the conflicts that
almost tore it apart, and the language’s impact on… well…
everything.

The video
of the keynote is also available.

New general-purpose Amazon EC2 M8i and M8i Flex instances are now available

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/new-general-purpose-amazon-ec2-m8i-and-m8i-flex-instances-are-now-available/

Today, we’re announcing the general availability of Amazon Elastic Compute Cloud (Amazon EC2) general-purpose M8i and M8i-Flex instances powered by custom Intel Xeon 6 processors available only on AWS with sustained all-core 3.9 GHz turbo frequency. These instances deliver the highest performance and fastest memory bandwidth among comparable Intel processors in the cloud. They also deliver up to 15 percent better price performance, up to 20 percent higher performance, and 2.5 times more memory bandwidth compared to previous generation M7i and M7i-Flex instances.

M8i and M8i-flex instances are ideal for running general purpose workloads such as general web application servers, virtual desktops, batch processing, microservices, databases, and enterprise applications. In terms of performance, these instances are specifically up to 60 percent faster for NGINX web applications, up to 30 percent faster for PostgreSQL database workloads, and up to 40 percent faster for AI deep learning recommendation models compared to M7i and M7i-Flex instances.

As like R8i and R8i-Flex instances, these instances use the new sixth generation AWS Nitro Cards, delivering up to two times more network and Amazon Elastic Block Storage (Amazon EBS) bandwidth compared to the previous generation instances. It greatly improves network throughput for workloads handling small packets such as web, application, and gaming servers. They also support bandwidth configuration with 25 percent allocation adjustments between network and Amazon EBS bandwidth, enabling better database performance, query processing, and logging speeds.

M8i instances
M8i instances provide up to 384 vCPUs and 1.5 TB memory including bare metal instances that provide dedicated access to the underlying physical hardware. These SAP-certified instances help you to run large application servers and databases, gaming servers, CPU-based inference, and video streaming that need the largest instance sizes or high CPU continuously.

Here are the specs for M8i instances:

Instance size vCPUs Memory (GiB) Network bandwidth (Gbps) EBS bandwidth (Gbps)
m8i.large 2 8 Up to 12.5 Up to 10
m8i.xlarge 4 16 Up to 12.5 Up to 10
m8i.2xlarge 8 32 Up to 15 Up to 10
m8i.4xlarge 16 64 Up to 15 Up to 10
m8i.8xlarge 32 128 15 10
m8i.12xlarge 48 192 22.5 15
m8i.16xlarge 64 256 30 20
m8i.24xlarge 96 384 40 30
m8i.32xlarge 128 512 50 40
m8i.48xlarge 192 768 75 60
m8i.96xlarge 384 1536 100 80
m8i.metal-48xl 192 768 75 60
m8i.metal-96xl 384 1536 100 80

M8i-Flex instances
M8i-Flex instances are a lower-cost variant of the M8i instances, with 5 percent better price performance at 5 percent lower prices. They’re designed for workloads that benefit from the latest generation performance but don’t fully utilize all compute resources. These instances can reach up to the full CPU performance 95 percent of the time.

Here are the specs for the M8i-Flex instances:

Instance size vCPUs Memory (GiB) Network bandwidth (Gbps) EBS bandwidth (Gbps)
m8i-flex.large 2 8 Up to 12.5 Up to 10
m8i-flex.xlarge 4 16 Up to 12.5 Up to 10
m8i-flex.2xlarge 8 32 Up to 15 Up to 10
m8i-flex.4xlarge 16 64 Up to 15 Up to 10
m8i-flex.8xlarge 32 128 Up to 15 Up to 10
m8i-flex.12xlarge 48 192 Up to 22.5 Up to 15
m8i-flex.16xlarge 64 256 Up to 30 Up to 20

If you’re currently using earlier generations of general-purpose instances, you can adopt M8i-Flex instances without having to make changes to your application or your workload.

Now available
Amazon EC2 M8i and M8i-Flex instances are available today in the US East (N. Virginia), US East (Ohio), US West (Oregon), and Europe (Spain) AWS Regions. M8i and M8i-Flex instances can be purchased as On-Demand, Savings Plan, and Spot instances. M8i instances are also available in Dedicated Instances and Dedicated Hosts. To learn more, visit the Amazon EC2 Pricing page.

Give M8i and M8i-Flex instances a try in the Amazon EC2 console. To learn more, visit the Amazon EC2 M8i instances page and send feedback to AWS re:Post for EC2 or through your usual AWS Support contacts.

Channy

Dell Precision 3240 Compact Mini PC with PCIe Card Slot Overview

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/dell-precision-3240-compact-mini-review-intel-xeon-nvidia/

We take a look at the Dell Precision 3240 Compact in a second act role of being a homelab node with features like a Xeon CPU and NVIDIA GPU

The post Dell Precision 3240 Compact Mini PC with PCIe Card Slot Overview appeared first on ServeTheHome.

Improvements to the Code Club Projects website

Post Syndicated from Divya Mahadevan original https://www.raspberrypi.org/blog/improvements-to-the-code-club-projects-website/

Getting creative with technology is now easier than ever on the Code Club Projects website. If you’ve visited Code Club Projects recently, you may have noticed that the site has changed over the last few months. In the spring, we launched an initiative to make it easier to find a project. I’m excited to share some of the changes we’ve made based on feedback from young people and mentors. 

A mentor with a young coder in a Code Club session.

Finding projects based on difficulty

Being able to filter projects by skill level (beginner, intermediate, advanced) has been the feature most requested by mentors. A survey we ran in the spring confirmed that young people find levels helpful when trying to find a project on their own. While our project paths have always had levels, our expert educators have now reviewed every single project on the site and categorised them as a Level 1, Level 2, or Level 3 project. These difficulty levels are easily available as filters on the side of the search page.

“[The Code Club] resources are extremely helpful, particularly for mentors with limited experience; step-by-step guidance is very useful and effective.” – Code Club mentor, Tunisia

Screenshot of level on project card and filters

Finding projects based on interest

For young people and mentors who aren’t sure where to get started with coding, an easy question to ask is “What do you like to do”? With over 200+ projects on the site, there is a wide variety of projects that span a number of young people’s passions, such as games, art, or nature. In fact, in the same spring survey, young people largely preferred finding projects by interest over technology (e.g., Scratch, Python). 

That’s why we’ve created a brand-new set of pages dedicated to helping young people find projects by interest. We believe making it easy to find a project by an existing interest will help young people get excited about coding. 

“[Attending Code Club] … combined with watching peers make cool things seems to be a great encouragement to go build rather than just consume.” – Code club mentor, Japan

Screenshot of different Interest categories on the Code Club Projects site.

Code Clubs projects are designed to appeal to creators’ interests, including creating their own games.

Finding projects based on technology

Through the Clubs annual survey, we know that some young people enjoy following our project pathways and others enjoy finding an individual project to work on during club time. We also know that some of our top search terms are “Scratch” and “Python”, which allow users to see all Scratch projects that are available on the site. We’ve redesigned our technology pages so it’s easy to see paths and projects per technology in the same place. 

“I think [Raspberry Pi Foundation projects are] really easy to understand and encourage young people to learn more and more.” – Code Club leader, Brazil

Screenshot of different Technology categories on the Code Club Projects site.

What’s next?

We’ve heard from mentors that our projects are too long to print out. We’re working to improve that, starting with brand-new short PDFs for the Intro to Scratch path projects. Each PDF is no more than six pages, double-sided.

Are there other projects you’d like to see PDF versions of? Let us know. 

We’re currently working on an idea tentatively titled “project challenges”. If you’re interested in helping us develop the idea, please get in touch

If you have not started a club yet but are interested in supporting the young people in your community to explore coding, you can find out more on the Code Club website. The team will support you every step of the way with resources, training, and a collaborative community.

The post Improvements to the Code Club Projects website appeared first on Raspberry Pi Foundation.

Mastering Amazon Q Developer with Rules

Post Syndicated from Aurelien Plancque original https://aws.amazon.com/blogs/devops/mastering-amazon-q-developer-with-rules/

When I first started working with Amazon Q Developer, I was impressed by its capabilities, but I quickly found myself in a familiar pattern. Development teams using AI assistants face a common challenge: repeatedly explaining coding standards, workflow preferences, and established patterns in every conversation. This repetitive setup reduces productivity and creates inconsistent AI guidance across team members. Sound familiar?

That’s when I discovered the power of custom rules – and it completely transformed how I work with AI assistance.

What Are Amazon Q Developer Rules?

Rules in Amazon Q Developer are a way to build a library of coding standards and best practices that are automatically used as context when interacting with the assistant.

These rules are defined in Markdown files stored in your project’s .amazonq/rules folder. Once created, they automatically become part of the context for developers interacting with Amazon Q Developer within your project, maintaining consistency across team members regardless of their experience level. Currently, rules are supported in the Amazon Q Developer IDE extensions and in the Amazon Q Developer CLI.

The Power of Rules-Based AI Assistance

What I find most compelling about rules-based AI assistance is how it minimizes the repetitive setup that usually comes with AI interactions. Instead of repeatedly instructing your AI assistant on your preferences and standards for each request, you can define these once as rules. This creates a consistent, predictable AI experience that automatically respects your team’s conventions and best practices.

The real game-changer for me has been the consistency. Whether I’m working on a new feature, debugging an issue, or reviewing code, Amazon Q Developer now understands my context from the start. This means I can focus on the actual problem-solving instead of repeatedly explaining how I like things done.

Understanding the Rule Lifecycle

One thing that consistently surprises developers is how seamlessly rules integrate into Amazon Q Developer workflow. Understanding when and how rules are injected into the context helps you make the most of this capability. Here’s how the process works:

A diagram depicting the rule lifecycle for Amazon Q Developer

The rule lifecycle in Amazon Q Developer

Rules are injected at several key moments:

1. Initial Context Loading: When you first interact with Amazon Q Developer in a project, it scans the `.amazonq/rules` directory and loads the applicable rules into its context.

2. Request Processing: Before generating a response, Amazon Q Developer evaluates your request against the loaded rules to determine which ones apply.

3. Response Generation: While crafting its response, Amazon Q Developer follows the instructions from applicable rules, prioritizing them based on their specified priority levels.

4. Dynamic Updates: If you modify existing rules or add new ones during a session, Amazon Q Developer detects these changes and updates its behavior accordingly.

This continuous integration makes sure that Amazon Q Developer’s responses consistently align with your standards without requiring you to repeat instructions in every conversation.

Why Rules-Based AI Makes a Difference ?

What I’ve found most valuable about this approach is how it transforms the AI from a generic assistant into something that feels like it truly understands your team’s way of working. Here are the key benefits I’ve experienced:

Consistency: Every team member gets the same AI-guided experience, making sure code and documentation remain consistent regardless of who wrote them.

Knowledge Preservation: Rules capture your team’s accumulated wisdom and best practices, making them accessible to everyone.

Reduced Cognitive Load: You can focus on solving problems rather than remembering and enforcing standards.

Faster Onboarding: New team members automatically receive guidance aligned with your team’s practices.

Adaptability: Rules can evolve alongside your project, making sure AI assistance remains relevant as your needs change.

The difference between generic AI assistance and rules-guided assistance becomes clear quickly. Generic AI might suggest any valid solution, while rules-guided AI suggests solutions that fit your specific context and standards.

Building Effective Rules: A Practical Approach

Now that you’ve seen the power of rules, let me walk you through how to create them. While there’s no “official” format for Amazon Q Developer rules (the beauty is in the flexibility!), the approach I’m about to share has consistently delivered excellent results for me and my team.

Rule File Format and Location

Here’s what I’ve learned about organizing rules for Amazon Q Developer:

  • Rules must be written in Markdown format (.md files)
  • They should be placed in the .amazonq/rules directory of your project
  • You can use a filename of your choice, though descriptive names help with organization (examples: monitoring-rule.md, frontend-react.rule.md)
  • Rules can be organized in sub-directories for better structure (for example, .amazonq/rules/frontend/react.rule.md)
  • The filename itself is arbitrary – Amazon Q Developer will read the .md files in the directory. However, using meaningful names makes your rule system easier to maintain as it grows.

Essential Rule Structure

What I find works best is a well-crafted rule file that contains these key sections:

# Rule Name
## Purpose
A clear, concise statement explaining why this rule exists.
## Instructions
- Specific directives for Amazon Q Developer to follow
- Additional instructions with their own identifiers
- Conditions under which instructions apply
## Priority
[Critical/High/Medium/Low]
## Error Handling
- How Amazon Q Developer should behave when exceptions occur
- Fallback strategies when primary instructions can't be followed

Let me show you this structure in action with a complete example of a monitoring rule that has been particularly effective for my team:

# Monitoring
## Purpose
This rule ensures that monitoring coverage is maintained when major features are added to the project.
## Instructions
- When implementing a major feature (new service, API endpoint, or core functionality), ALWAYS check if MONITORING_PLAN.md needs updates.
- Major features include: new microservices, AI integrations, WebSocket endpoints, database operations, external API integrations, or user-facing functionality.
- ALWAYS update MONITORING_PLAN.md to include relevant metrics, dashboards, and alerts for the new feature.
- When updating monitoring plan, include: custom metrics, CloudWatch dashboards, alarms, and logging requirements specific to the new feature.
- After updating MONITORING_PLAN.md, ALWAYS output "📊 Updated monitoring plan for: [feature description]".
## Priority
High
## Error Handling
- If MONITORING_PLAN.md doesn't exist, create it with basic monitoring structure and note the creation
- If monitoring plan is unreadable, create a backup and start fresh with current feature requirements
- If unsure whether a feature qualifies as "major", err on the side of caution and update monitoring plan

I saved this as monitoring.rule.md in my project’s .amazonq/rules directory.

Rule Components That Work

Now let me break down each component and show you why this structure works so well.

Rule Name

Think of this as the “class name” for your rule. It should be descriptive and domain-specific, like “Frontend – React” or “Monitoring.” This helps organize your rules into logical categories and makes them easier to maintain as your ruleset grows.

Purpose

This section is crucial, it’s where you explain the “why” behind your rule. What I’ve learned is that a clear purpose helps Amazon Q Developer understand the intent behind your instructions, allowing it to make better decisions when faced with edge cases. For example:

## Purpose
Ensures consistent monitoring coverage is maintained when adding new features to the project.

This simple statement guides Amazon Q Developer to prioritize monitoring considerations, even when they’re not explicitly mentioned in your request.

Instructions

This is where the magic happens. Instructions are specific directives that shape Amazon Q Developer’s behavior. What I’ve found works best is when each instruction:

  1. Is clear and actionable
  2. Focuses on a single aspect of behavior
  3. Uses consistent formatting for easy scanning

For example:

## Instructions
- When implementing a major feature, ALWAYS check if MONITORING_PLAN.md needs updates.
- Major features include: new microservices, AI integrations, WebSocket endpoints.
- After updating MONITORING_PLAN.md, output "📊 Updated monitoring plan for: [feature]".

These clear, focused instructions give Amazon Q Developer specific guidance on how to behave in different situations, maintaining consistent responses across your team.

Priority

Not all rules are created equal. What I’ve discovered is that the priority level helps Amazon Q Developer resolve conflicts when multiple rules could apply to a situation. I typically use four levels:

  • Critical: Must be followed without exception
  • High: Should be followed unless conflicting with a critical rule
  • Medium: Important guidelines that shape behavior
  • Low: Preferences that can be overridden when necessary
Error Handling

This often-overlooked section is what makes rules robust in real-world scenarios. Good error handling instructions tell Amazon Q Developer what to do when things don’t go as planned:

## Error Handling
- If MONITORING_PLAN.md doesn't exist, create it with basic monitoring structure
- If unsure whether a feature qualifies as "major," err on the side of caution

These fallback strategies make sure Amazon Q Developer remains helpful even when facing unexpected situations.

Seeing Rules in Action

To show you how effective this structure can be, let me give you a simple example. Without rules, asking Amazon Q Developer to “add a new React component for user profiles” might result in a component that doesn’t match your project’s patterns.

But with a well-crafted frontend rule, Amazon Q Developer would automatically:

  1. Check existing component structures
  2. Follow your naming conventions
  3. Create appropriate prop interfaces
  4. Add the right level of documentation
  5. Place the file in your preferred directory structure

All without you having to specify these details every time!

Making Rules Transparent: A Game-Changing Technique

One particularly powerful technique I’ve discovered is teaching Amazon Q Developer to explicitly acknowledge which rules it’s following. This isn’t a default behavior of Amazon Q Developer, but rather a custom enhancement you can implement through a specific conversation rule.

Adding Unique Identifiers for Traceability

The key to this system is adding unique identifiers (IDs) to each instruction in your rules. For example:

## Instructions
- When implementing a major feature, ALWAYS check if MONITORING_PLAN.md needs updates. (ID: CHECK_MONITORING_PLAN)
- Major features include: new microservices, AI integrations, WebSocket endpoints. (ID: MAJOR_FEATURE_CRITERIA)
- After updating MONITORING_PLAN.md, output "📊 Updated monitoring plan for: [feature]". (ID: ANNOUNCE_MONITORING_UPDATE)

These IDs serve as “traceable markers” that Amazon Q Developer can reference when following a rule.

Creating the Acknowledgment Behavior

Next, you can create a conversation rule that instructs Amazon Q Developer to acknowledge which rules it’s following. Here’s a complete example of such a rule:

# Conversation
## Purpose
This rule defines how Amazon Q Developer should behave in conversations, including how it should acknowledge other rules it's following.
## Instructions
- ALWAYS consider your rules before using a tool or responding. (ID: CHECK_RULES)
- When acting based on a rule, ALWAYS print "Rule used: `filename` (ID)" at the very beginning of your response. (ID: PRINT_RULES)
- If multiple rules are matched, list all: "Rule used: `file1.rule.md` (ID1), `file2.rule.md` (ID2)". (ID: PRINT_MULTIPLE)
- DO NOT start responses with general mentions about using rules or context, but DO print specific rule usage as specified above. (ID: NO_GENERIC_MENTIONS)
## Priority
Critical
## Error Handling
- If rule files are unreadable, continue but note the issue
- If multiple conflicting rules apply, follow the highest priority rule and note the conflict

Save this as conversation.rule.md in your .amazonq/rules directory.

When Amazon Q Developer follows a rule, it will now explicitly state which rule and identifier guided its actions:

An image showing Q Developer CLI printing the rules it uses in its answers

You can get Amazon Q Developer to state which instruction it is following.

What I find most valuable about this simple addition is the remarkable benefits it creates:

  • Transparency: Team members can immediately see which guidelines influenced Amazon Q Developer’s response
  • Debugging: When Amazon Q Developer behaves unexpectedly, you can identify which rule caused the behavior
  • Learning: New team members discover relevant rules by seeing which ones are being applied
  • Validation: You can confirm that your rules are working as intended
  • Continuous Improvement: Identify which rules are most frequently used and which might need refinement

By making rules visible, you turn Amazon Q Developer into a collaborative partner that not only follows your guidelines and helps team members discover and engage with your established practices. The IDs aren’t just organizational tools—they’re the foundation of a self-documenting AI assistance workflow that grows more valuable as your rule system expands.

Getting Started with Your Own Rules

What I love about this approach is that you can start small. Begin with one or two rules addressing your most common pain points, then expand as you see the benefits. Some good starting points include:

  • Code style and organization rules
  • Documentation standards
  • Testing requirements
  • Git commit message formats

Remember, the goal isn’t to create an exhaustive rulebook—it’s to capture the aspects of your development process that matter most to your team’s productivity and code quality.

Practical Examples: Rules in Action

To show you the real-world impact of rules, let me walk you through some concrete scenarios that demonstrate how rules transform the AI assistance experience. These examples show the difference between generic AI help and rules-guided assistance.

Scenario 1: Time-Based Data Analysis

This scenario demonstrates how rules help Amazon Q Developer understand your environment’s context for time-related operations and analysis. Here are examples of this rule in action in VS Code.

Here is a rule I use to inform Amazon Q Developer how to behave when it needs to understand the current time:

# Time
## Purpose
This rule defines how Amazon Q Developer (the agent) handles time-related operations and queries
## Instructions
- When determining the current time, ALWAYS use bash commands with AEST timezone: `date` (ID: GET_AEST_TIME)
- When timestamps are needed for logging or documentation, use ISO format with AEST timezone (ID: ISO_TIMESTAMP)
- When comparing times or calculating durations, ensure all times are in AEST for consistency (ID: CONSISTENT_TIMEZONE)
- For time-sensitive operations, always verify the current AEST time before proceeding (ID: VERIFY_TIME)
## Priority
Medium
## Error Handling
- If date command fails, note the system time issue and continue with available information
- If timezone conversion is needed, use appropriate date formatting commands
Without Rules:

When Amazon Q Developer doesn’t have the time rule, it lacks the context needed for time-based queries:

An image showing Amazon Q Developer in the IDE trying to understand what is the date

An Amazon Q Developer interaction without a time rule

As you can see, without the rule, Amazon Q Developer needs clarification about timezone context and doesn’t know how to determine the current time in your environment.

With the Time Rule:

Here’s a similar query with the time rule in place:

An image showing Amazon Q Developer in the IDE using the time rule to get the time and date

Amazon Q Developer follows the instructions of the rule

Notice how Amazon Q Developer immediately uses the `date` command to get the current AEST time, exactly as specified in the rule, without needing clarification.

The Impact:
  • Automatic Context: Amazon Q Developer immediately knows to use the date command to get AEST time
  • No Clarification Needed: It understands “yesterday” relative to the current AEST time without asking
  • Consistent Behavior: The same approach works for other time-based queries across team members
  • Environmental Awareness: It knows exactly how to determine time in your specific system environment
  • Transparent Process: You can see it’s following the rule by using the bash date command as specified

This example shows how the time rule transforms a potentially confusing interaction into a smooth, context-aware analysis that works consistently every time.

Scenario 2: Frontend Component Development

This scenario demonstrates how rules can help prevent technical debt accumulation and maintain consistent component architecture across your team.

Without Rules:

When Amazon Q Developer doesn’t have frontend-specific guidance, different developers can get inconsistent suggestions for component creation. Some might create reusable components immediately, others get copy-paste solutions, and component organization varies based on individual preferences.

With Frontend Rules:

Here’s a real React rule from my development workflow that addresses these consistency issues:

# Frontend - React
## Purpose
Defines how to act when writing React
## Instructions
- ALWAYS evaluate reusability potential for new visual elements using these criteria: used in 2+ places, has configurable props, or represents a common UI pattern. (ID: EVALUATE_REUSABILITY)
- If reusability potential is high (meets 2+ criteria above), create a dedicated component in appropriate folder (components/, shared/, or ui/) with clear prop interfaces and JSDoc comments. (ID: CREATE_REUSABLE_COMPONENT)
- When creating reusable components, include explicit comments explaining: purpose, key props, usage examples, and any important behavior. (ID: DOCUMENT_COMPONENTS)
- Follow existing component structure and naming conventions found in the project's components folder. (ID: FOLLOW_CONVENTIONS)
- Prefer composition over inheritance - create small, focused components that can be combined. (ID: PREFER_COMPOSITION)
## Priority
Medium
## Error Handling
- If component folder structure is unclear, place new components in src/components/ and ask user for preferred organization
- If existing conventions are inconsistent, follow the most recent or most common pattern and note the inconsistency
The Impact:
  • Consistent Architecture: Team members get the same component creation guidance regardless of experience level
  • Reduced Technical Debt: Automatic evaluation of reusability helps prevent duplicated UI elements
  • Better Documentation: Components automatically include proper JSDoc comments and usage examples
  • Maintainable Structure: Consistent naming conventions and folder organization across the project
  • Scalable Patterns: Composition-focused approach creates more flexible, reusable components

This make sure that whether a junior developer or senior architect is working with Amazon Q Developer, the resulting components follow the same quality standards and architectural patterns.

Scenario 3: Version Control Workflow

This scenario demonstrates how rules can act as safety mechanisms and workflow controllers, facilitating consistent practices around version control operations.

Without Rules:

Amazon Q Developer might suggest git operations without considering your team’s workflow preferences or safety requirements. It could recommend immediate pushes, generic commit messages, or skip confirmation steps that help prevent mistakes.

With Git Rules:

Here’s a real git rule that transforms how Amazon Q Developer handles version control operations:

# Git
## Purpose
This rule dictates Amazon Q Developer (the agent) behavior related to git.
## Instructions
- ALWAYS ask confirmation from the user before pushing to git (ID: GIT_PUSH)
- ALWAYS ensure commit messages are meaningful and detailed, including what was changed and why (ID: GIT_COMMIT)
- Commit message should be accurate but playful and not too formal, with comprehensive details about the changes (ID: GIT_COMMIT_STYLE)
- Include specific files/components modified and the impact of changes in commit messages (ID: GIT_COMMIT_DETAILS)
## Priority
Medium
## Error Handling
N/A

Here’s this rule in action when I asked Amazon Q Developer to commit our changes:

> Let's commit our latest changes to git.
> Rule used: git.rule.md (GIT_COMMIT), git.rule.md (GIT_COMMIT_STYLE), 
git.rule.md (GIT_COMMIT_DETAILS)
I'll commit all the changes we've made to remove messaging functionality and 
focus the platform on location-based discovery. Let me create a comprehensive 
commit with all the modifications.
🛠 Using tool: execute_bash (trusted)
 ● I will run the following shell command: 
cd /path/to/your/project/rules-blog && git status
 ↳ Purpose: Check git status to see all changed files
On branch main
Changes not staged for commit:
[... output truncated for brevity ...]
Untracked files:
[... output truncated for brevity ...]
● Completed in 0.54s
🛠 Using tool: execute_bash (trusted)
 ● I will run the following shell command: 
cd /path/to/your/project/rules-blog && git add.
 ↳ Purpose: Add all changes to git staging
 ● Completed in 0.101s
🛠 Using tool: execute_bash (trusted)
 ● I will run the following shell command: 
cd /path/to/your/project/rules-blog && git commit -m "🎯 Refocus platform on location-based discovery: Remove messaging, complete React app
Major platform refocus to pure location-based discovery:
MESSAGING FUNCTIONALITY REMOVED:
[... output truncated for brevity ...]
DOCUMENTATION UPDATES:
[... output truncated for brevity ...]
INFRASTRUCTURE IMPROVEMENTS:
[... output truncated for brevity ...]
PLATFORM FOCUS NOW:
✅ Location-based discovery
✅ Browsing and following
✅ Information and contact details
✅ Notifications by location
❌ No messaging/booking - pure discovery platform
Files modified: 13 core files, 1 new frontend package
Infrastructure: 7 Lambda functions, tagged resources, clean API
Tests: 12/12 passing, production build successful"
[... output truncated for brevity ...]

Note: Notice how Amazon Q Developer automatically shows Rule used: git.rule.md (GIT_COMMIT), git.rule.md (GIT_COMMIT_STYLE), git.rule.md (GIT_COMMIT_DETAILS) at the beginning of its response – this is the rule transparency system we discussed earlier in action, showing exactly which rule instructions guided the commit message creation.

The Impact:
  • Helps Prevent Accidents: The confirmation requirement stops accidental pushes that could disrupt team workflows
  • Consistent Commit Quality: Commit messages follow the same detailed, informative style regardless of who’s working
  • Team Personality: The “playful but not too formal” style maintains team culture while preserving professionalism
  • Better Git History: Detailed commit messages make code archaeology much more straightforward for future debugging
  • Workflow Safety: Acts as a guardrail that respects human oversight in critical operations

This rule shows how Amazon Q Developer can be more than just a code assistant – it becomes a workflow partner that understands and enforces your team’s operational preferences and safety requirements.

What’s Next for Rules-Based Development

Through this exploration of Amazon Q Developer rules, we’ve discovered how a simple concept – defining your preferences once instead of repeating them constantly – can transform your development workflow. The key learnings are clear: rules help you avoid repetitive setup, help with team consistency, preserve institutional knowledge, and create transparent AI interactions that grow more valuable over time.

Reduced cognitive load, faster onboarding, consistent code quality, and AI assistance that truly understands your team’s context – what started as a solution to repetitive explanations has evolved into a comprehensive system for scaling development practices across my team.

My Amazon Q Developer rules system continues to evolve, and I’m excited about the possibilities ahead. As more teams adopt this approach, I expect we’ll see community-shared rule libraries and even more sophisticated customization options.

What I find most promising is how rules create a foundation for more advanced AI assistance. When your AI assistant understands your context deeply, it can provide more sophisticated suggestions and catch potential issues before they become problems.

I encourage you to start experimenting with rules – pick one area where you find yourself repeating instructions to AI assistants and create your first rule. You’ll be surprised how quickly this approach transforms your development workflow.

The key is to remember that rules aren’t about constraining creativity – they’re about freeing you to focus on the interesting problems by automating the routine decisions. When Amazon Q Developer knows how you like things done, you can spend more time on what you’re building and less time on how to build it.

Ready to get started with Amazon Q Developer rules? Check out the Amazon Q Developer documentation for setup instructions and more examples.

[$] Changing GNOME technical governance?

Post Syndicated from jake original https://lwn.net/Articles/1034684/

The GNOME project, which recently celebrated its
28th birthday
, has never had a formal technical governance; progress
has been driven by individuals and groups that advocated for—and worked
toward—a particular goal in an ad hoc fashion. Longtime GNOME contributor
Emmanuele Bassi would like to see that change by adding cross-project teams
and a steering committee for the project; to that end, he gave a talk (YouTube
video
) at GUADEC 2025
in late July on his idea to establish some technical governance for the
project. He also put together a blog
post
with his notes from the talk. The audience reaction was
favorable, so he has followed up on the GNOME discussion forum with an RFC on
governance
to try to move the effort along.

Security updates for Thursday

Post Syndicated from jake original https://lwn.net/Articles/1035464/

Security updates have been issued by AlmaLinux (aide, firefox, kernel, and mod_http2), Debian (chromium and unbound), Fedora (mod_auth_openidc), Oracle (fence-agents and kernel), SUSE (ignition, jetty-minimal, kernel, libmozjs-128-0, matrix-synapse, postgresql13, postgresql15, postgresql16, and postgresql17), and Ubuntu (kernel).

Modernize Amazon Redshift authentication by migrating user management to AWS IAM Identity Center

Post Syndicated from Ziad Wali original https://aws.amazon.com/blogs/big-data/modernize-amazon-redshift-authentication-by-migrating-user-management-to-aws-iam-identity-center/

Amazon Redshift is a powerful cloud-based data warehouse that organizations can use to analyze both structured and semi-structured data through advanced SQL queries. As a fully managed service, it provides high performance and scalability while allowing secure access to the data stored in the data warehouse. Organizations worldwide rely on Amazon Redshift to handle massive datasets, upgrade their analytics capabilities, and deliver valuable business intelligence to their stakeholders.

AWS IAM Identity Center serves as the preferred platform for controlling workforce access to AWS tools, including Amazon Q Developer. It allows for a single connection to your existing identity provider (IdP), creating a unified view of users across AWS applications and applying trusted identity propagation for a smooth and consistent experience.

You can access data in Amazon Redshift using local users or external users. A local user in Amazon Redshift is a database user account that is created and managed directly within the Redshift cluster itself. Amazon Redshift also integrates with IAM Identity Center, and supports trusted identity propagation, so you can use third-party IdPs such as Microsoft Entra ID (Azure AD), Okta, Ping, OneLogin, or use IAM Identity Center as an identity source. The IAM Identity Center integration with Amazon Redshift supports centralized authentication and SSO capabilities, simplifying access management across multi-account environments. As organizations grow in scale, it is recommended to use external users for cross-service integration and centralized access management.

In this post, we walk you through the process of smoothly migrating your local Redshift user management to IAM Identity Center users and groups using the RedshiftIDCMigration utility.

Solution overview

The following diagram illustrates the solution architecture.

The RedshiftIDCMigration utility accelerates the migration of your local Redshift users, groups, and roles to your IAM Identity Center instance by performing the following activities:

  • Create users in IAM Identity Center for every local user in a given Redshift instance.
  • Create groups in IAM Identity Center for every group or role in a given Redshift instance.
  • Assign users to groups in IAM Identity Center according to existing assignments in the Redshift instance.
  • Create IAM Identity Center roles in the Redshift instance matching the groups created in IAM Identity Center.
  • Grant permissions to IAM Identity Center roles in the Redshift instance based on the current permissions given to local groups and roles.

Prerequisites

Before running the utility, complete the following prerequisites:

  1. Enable IAM Identity Center in your account.
  2. Follow the steps in the post Integrate Identity Provider (IdP) with Amazon Redshift Query Editor V2 and SQL Client using AWS IAM Identity Center for seamless Single Sign-On (specifically, follow Steps 1–8, skipping Steps 4 and 6).
  3. Configure the IAM Identity Center application assignments:
    1. On the IAM Identity Center console, choose Application Assignments and Applications.
    2. Select your application and on the Actions dropdown menu, choose Edit details.
    3. For User and group assignments, choose Do not require assignments. This setting makes it possible to test Amazon Redshift connectivity without configuring specific data access permissions.
  4. Configure IAM Identity Center authentication with administrative access from either Amazon Elastic Compute Cloud (Amazon EC2) or AWS CloudShell.

The utility will be run from either an EC2 instance or CloudShell. If you’re using an EC2 instance, an IAM role is attached to the instance. Make sure that the IAM role used during the execution has the following permissions (if not, create a new policy with those permissions and attach it to the IAM role):

  • Amazon Redshift permissions (for serverless):
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "redshift-serverless:GetCredentials",
                "redshift-serverless:GetNamespace",
                "redshift-serverless:GetWorkgroup"
            ],
            "Resource": [
                "arn:aws:redshift-serverless:${region}:${account-id}:namespace/${namespace-id}",
                "arn:aws:redshift-serverless:${region}:${account-id}:workgroup/${workgroup-id}"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "redshift-serverless:ListNamespaces",
                "redshift-serverless:ListWorkgroups"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "redshift:CreateClusterUser",
                "redshift:JoinGroup",
                "redshift:GetClusterCredentials",
                "redshift:ExecuteQuery",
                "redshift:FetchResults",
                "redshift:DescribeClusters",
                "redshift:DescribeTable"
            ],
            "Resource": [
                "arn:aws:redshift:${region}:${account-id}:cluster:redshift-serverless-${workgroup-name}",
                "arn:aws:redshift:${region}:${account-id}:dbgroup:redshift-serverless-${workgroup-name}/${dbgroup}",
                "arn:aws:redshift:${region}:${account-id}:dbname:redshift-serverless-${workgroup-name}/${dbname}",
                "arn:aws:redshift:${region}:${account-id}:dbuser:redshift-serverless-${workgroup-name}/${dbuser}"
            ]
        }
    ]
}
  • Amazon Redshift permissions (for provisioned):
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "redshift:GetClusterCredentials",
            "Resource": [
                "arn:aws:redshift: ${region}:${account-id}:dbname:${cluster_name}/${dbname}",
                "arn:aws:redshift: ${region}: ${account-id}:dbuser:${cluster-name}/${dbuser}"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "redshift:DescribeClusters",
                "redshift:ExecuteQuery",
                "redshift:FetchResults",
                "redshift:DescribeTable"
            ],
            "Resource": "*"
        }
    ]
}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetEncryptionConfiguration",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::${s3_bucket_name}/*",
                "arn:aws:s3:::${s3_bucket_name}"
            ]
        }
    ]
}
  • Identity store permissions:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "identitystore:*",
            "Resource": [
                "arn:aws:identitystore:::group/*",
                "arn:aws:identitystore:::user/*",
                "arn:aws:identitystore::${account_id}:identitystore/${identity_store_id}",
                "arn:aws:identitystore:::membership/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "identitystore:*",
            "Resource": [
                "arn:aws:identitystore:::membership/*",
                "arn:aws:identitystore:::user/*",
                "arn:aws:identitystore:::group/*"
            ]
        }
    ]
}

Artifacts

Download the following utility artifacts from the GitHub repo:

  • idc_redshift_unload_indatabase_groups_roles_users.py – A Python script to unload users, groups, roles and their associations.
  • redshift_unload.ini – The config file used in the preceding script to read Redshift data warehouse details and Amazon S3 locations to unload the files.
  • idc_add_users_groups_roles_psets.py – A Python script to create users and groups in IAM Identity Center, and then associate the users to groups in IAM Identity Center.
  • idc_config.ini – The config file used in the preceding script to read IAM Identity Center details.
  • vw_local_ugr_to_idc_urgr_priv.sql – A script that generates SQL statements that perform two tasks in Amazon Redshift:
    • Create roles that exactly match your IAM Identity Center group names, adding a specified prefix.
    • Grant appropriate permissions to these newly created Redshift roles.

Testing scenario

This test case is designed to offer practical experience and familiarize you with the utility’s functionality. The scenario is structured around a hierarchical nested roles system, starting with object-level permissions assigned to technical roles. These technical roles are then allocated to business roles. Finally, business roles are granted to individual users. To enhance the testing environment, the scenario also incorporates a user group.The following diagram illustrates this hierarchy.

Create datasets

Set up two separate schemas (tickit and tpcds) in a Redshift database using the create schema command. Then, create and populate a few tables in each schema using the tickit and tpcds sample datasets.

Specify the appropriate IAM role Amazon Resource Name (ARN) in the copy commands if necessary.

Create users

Create users with the following code:

-- ETL users
create user etl_user_1 password 'EtlUser1!';
create user etl_user_2 password 'EtlUser2!';
create user etl_user_3 password 'EtlUser3!';

-- Reporting users
create user reporting_user_1 password 'ReportingUser1!';
create user reporting_user_2 password 'ReportingUser2!';
create user reporting_user_3 password 'ReportingUser3!';

-- Adhoc users
create user adhoc_user_1 password 'AdhocUser1!';
create user adhoc_user_2 password 'AdhocUser2!';

-- Analyst users
create user analyst_user_1 password 'AnalystUser1!';

Create business roles

Create business users with the following code:

-- ETL business roles
create role role_bn_etl_tickit;
create role role_bn_etl_tpcds;

-- Reporting business roles
create role role_bn_reporting_tickit;
create role role_bn_reporting_tpcds;

-- Analyst business roles
create role role_bn_analyst_tickit;

Create technical roles

Create technical roles with the following code:

-- Technical roles for tickit schema
create role role_tn_sel_tickit;
create role role_tn_dml_tickit;
create role role_tn_cte_tickit;

-- Technical roles for tpcds schema
create role role_tn_sel_tpcds;
create role role_tn_dml_tpcds;
create role role_tn_cte_tpcds;

Create groups

Create groups with the following code:

-- Adhoc users group
create group group_adhoc;

Grant rights to technical roles

To grant rights to the technical roles, use the following code:

-- role_tn_sel_tickit
grant usage on schema tickit to role role_tn_sel_tickit;
grant select on all tables in schema tickit to role role_tn_sel_tickit;

-- role_tn_dml_tickit
grant usage on schema tickit to role role_tn_dml_tickit;
grant insert, update, delete on all tables in schema tickit to role role_tn_dml_tickit;

-- role_tn_cte_tickit
grant usage, create on schema tickit to role role_tn_cte_tickit;
grant drop on all tables in schema tickit to role role_tn_cte_tickit;

-- role_tn_sel_tpcds
grant usage on schema tpcds to role role_tn_sel_tpcds;
grant select on all tables in schema tpcds to role role_tn_sel_tpcds;

-- role_tn_dml_tpcds
grant usage on schema tpcds to role role_tn_dml_tpcds;
grant insert, update, delete on all tables in schema tpcds to role role_tn_dml_tpcds;

-- role_tn_cte_tpcds
grant usage, create on schema tpcds to role role_tn_cte_tpcds;
grant drop on all tables in schema tpcds to role role_tn_cte_tpcds;

Grant technical roles to business roles

To grant the technical roles to the business roles, use the following code:

-- Business role role_bn_etl_tickit
grant role role_tn_sel_tickit to role role_bn_etl_tickit;
grant role role_tn_dml_tickit to role role_bn_etl_tickit;
grant role role_tn_cte_tickit to role role_bn_etl_tickit;

-- Business role role_bn_etl_tpcds
grant role role_tn_sel_tpcds to role role_bn_etl_tpcds;
grant role role_tn_dml_tpcds to role role_bn_etl_tpcds;
grant role role_tn_cte_tpcds to role role_bn_etl_tpcds;

-- Business role role_bn_reporting_tickit
grant role role_tn_sel_tickit to role role_bn_reporting_tickit;

-- Business role role_bn_reporting_tpcds
grant role role_tn_sel_tpcds to role role_bn_reporting_tpcds;

-- Business role role_bn_analyst_tickit
grant role role_tn_sel_tickit to role role_bn_analyst_tickit;

Grant business roles to users

To grant the business roles to users, use the following code:

-- etl_user_1
grant role role_bn_etl_tickit to etl_user_1;

-- etl_user_2
grant role role_bn_etl_tpcds to etl_user_2;

-- etl_user_3
grant role role_bn_etl_tickit to etl_user_3;
grant role role_bn_etl_tpcds to etl_user_3;

-- reporting_user_1
grant role role_bn_reporting_tickit to reporting_user_1;

-- reporting_user_2
grant role role_bn_reporting_tpcds to reporting_user_2;

-- reporting_user_3
grant role role_bn_reporting_tickit to reporting_user_3;
grant role role_bn_reporting_tpcds to reporting_user_3;

-- analyst_user_1
grant role role_bn_analyst_tickit to analyst_user_1;

Grant rights to groups

To grant rights to the groups, use the following code:

-- Group group_adhoc
grant usage on schema tickit to group group_adhoc;
grant select on all tables in schema tickit to group group_adhoc;

grant usage on schema tpcds to group group_adhoc;
grant select on all tables in schema tpcds to group group_adhoc;

Add users to groups

To add users to the groups, use the following code:

alter group group_adhoc add user adhoc_user_1;
alter group group_adhoc add user adhoc_user_2;

Deploy the solution

Complete the following steps to deploy the solution:

  1. Update Redshift cluster or serverless endpoint details and Amazon S3 location in redshift_unload.ini:
    • cluster_type = provisioned or serverless
    • cluster_id = ${cluster_identifier} (required if cluster_type is provisioned)
    • db_user = ${database_user}
    • db_name = ${database_name}
    • host = ${host_url} (required if cluster_type is provisioned)
    • port = ${port_number}
    • workgroup_name = ${workgroup_name} (required if cluster_type is serverless)
    • region = ${region}
    • s3_bucket = ${S3_bucket_name}
    • roles = roles.csv
    • users = users.csv
    • role_memberships = role_memberships.csv
  2. Update IAM Identity Center details in idc_config.ini:
    • region = ${region}
    • account_id = ${account_id}
    • identity_store_id = ${identity_store_id} (available on the IAM Identity Center console Settings page)
    • instance_arn = ${iam_identity_center_instance_arn} (available on the IAM Identity Center console Settings page)
    • permission_set_arn = ${permission_set_arn}
    • assign_permission_set = True or False (True if permission_set_arn is defined)
    • s3_bucket = ${S3_bucket_name}
    • users_file = users.csv
    • roles_file = roles.csv
    • role_memberships_file = role_memberships.csv
  3. Create a directory in CloudShell or on your own EC2 instance with connectivity to Amazon Redshift.
  4. Copy the two .ini files and download the Python scripts to that directory.
  5. Run idc_redshift_unload_indatabase_groups_roles_users.py either from CloudShell or your EC2 instance:python idc_redshift_unload_indatabase_groups_roles_users.py
  6. Run idc_add_users_groups_roles_psets.py either from CloudShell or your EC2 instance:python idc_add_users_groups_roles_psets.py
  7. Connect your Redshift cluster using the Amazon Redshift query editor v2 or preferred SQL client, using superuser credentials.
  8. Copy the SQL in the vw_local_ugr_to_idc_urgr_priv.sql file and run it in the query editor to create the vw_local_ugr_to_idc_urgr_priv view.
  9. Run following SQL command to generate the SQL statements for creating roles and permissions:
    select existing_grants,idc_based_grants from vw_local_ugr_to_idc_urgr_priv;

    For example, consider the following existing grants:

    CREATE GROUP "group_adhoc";
    CREATE ROLE "role_bn_etl_tickit";
    GRANT USAGE ON SCHEMA tpcds TO role "role_tn_sel_tpcds" ;

    These grants are converted to the following code:

    CREATE role "AWSIDC:group_adhoc";
    CREATE role "AWSIDC:role_bn_etl_tickit";
    GRANT USAGE ON SCHEMA tpcds TO role "AWSIDC:role_tn_sel_tpcds";

  10. Review the statements in the idc_based_grants column.
    This might not be a comprehensive list of permissions, so review them carefully.
  11. If everything is correct, run the statements from the SQL client.

When you have completed the process, you should have the following configuration:

  • IAM Identity Center now contains newly created users from Amazon Redshift
  • The Redshift local groups and roles are created as groups in IAM Identity Center
  • New roles are established in Amazon Redshift, corresponding to the groups created in IAM Identity Center
  • The newly created Redshift roles are assigned appropriate permissions

If you encounter an issue while connecting to Amazon Redshift with the query editor using IAM Identity Center, refer to Troubleshooting connections from Amazon Redshift query editor v2.

Considerations

Consider the following when using this solution:

  • At the time of writing, creating permissions in AWS Lake Formation is not in scope.
  • IAM Identity Center and IdP integration setup is out of scope for this utility. However, you can use the view vw_local_ugr_to_idc_urgr_priv.sqlto create roles and grant permissions to the IdP users and groups passed through IAM Identity Center.
  • If you have permissions given directly to local user IDs (not using groups or roles), you must change that to a role-based permission approach for IAM Identity Center integration. Create roles and provide permissions using roles instead of directly giving permissions to users.

Clean up

If you have completed the testing scenario, clean up your environment:

  1. Remove the new Redshift roles that were created by the utility, corresponding to the groups established in IAM Identity Center.
  2. Delete the users and groups created by the utility within IAM Identity Center.
  3. Delete the users, groups, and roles specified in the testing scenario.
  4. Drop the tickit and tpcds schemas.

You can use the FORCE parameter when dropping the roles to remove associated assignments.

Conclusion

In this post, we showed how to migrate your Redshift local user management to IAM Identity Center. This transition offers several key advantages for your organization, such as simplified access management through centralized user and group administration, a streamlined user experience across AWS services, and reduced administrative overhead. You can implement this migration process step by step, so you can test and validate each step before fully transitioning your production environment.

As organizations continue to scale their AWS infrastructure, using IAM Identity Center becomes increasingly valuable for maintaining secure and efficient access management, including Amazon SageMaker Unified Studio for an integrated experience for all your data and AI.


About the authors

Ziad Wali

Ziad Wali

Ziad is an Analytics Specialist Solutions Architect at AWS. He has over 10 years of experience in databases and data warehousing, where he enjoys building reliable, scalable, and efficient solutions. Outside of work, he enjoys sports and spending time in nature.

Satesh Sonti

Satesh Sonti

Satesh is a Sr. Analytics Specialist Solutions Architect based out of Atlanta, specializing in building enterprise data platforms, data warehousing, and analytics solutions. He has over 19 years of experience in building data assets and leading complex data platform programs for banking and insurance clients across the globe.

Maneesh Sharma

Maneesh Sharma

Maneesh is a Senior Database Engineer at AWS with more than a decade of experience designing and implementing large-scale data warehouse and analytics solutions. He collaborates with various Amazon Redshift Partners and customers to drive better integration.

Sumanth Punyamurthula

Sumanth Punyamurthula

Sumanth is a Senior Data and Analytics Architect at AWS with more than 20 years of experience in leading large analytical initiatives, including analytics, data warehouse, data lakes, data governance, security, and cloud infrastructure across travel, hospitality, financial, and healthcare industries.

Implement fine-grained access control using Amazon OpenSearch Service and JSON Web Tokens

Post Syndicated from Ramya Bhat original https://aws.amazon.com/blogs/big-data/implement-fine-grained-access-control-using-amazon-opensearch-service-and-json-web-tokens/

This post demonstrates how to build a secure search application using Amazon OpenSearch Service and JSON Web Tokens (JWTs). We discuss the basics of OpenSearch Service and JWTs and how to implement user authentication and authorization through an existing identity provider (IdP). The focus is on enforcing fine-grained access control based on user roles and permissions.

JWT authentication and authorization for your OpenSearch Service domain provides a robust mechanism that addresses requirements for fine-grained access control. An IdP is a service that stores and manages user identities and their access rights, enabling centralized user authentication across multiple applications. The IdP issues JWTs, which are secure tokens containing claims about the authenticated user. By using JWTs from the IdP, you can:

  • Implement secure, role-based access control to search results
  • Validate user permissions before granting access to sensitive data
  • Maintain a centralized authentication mechanism across your search application
  • Make sure only authorized users can view data based on their predefined roles

The JWT integration helps organizations:

  • Define granular permissions within the IdP
  • Authenticate users using bearer tokens across different applications
  • Protect sensitive information through token-based access management
  • Reduce complexity of managing multiple authentication systems

Key benefits of the solution include:

  • Standardized token-based authentication
  • Centralized permission management
  • Simplified single sign-on (SSO) experience
  • Flexible and scalable access control mechanism

The ability to dynamically filter sensitive information based on token claims enhances data security while reducing the complexity of managing multiple authentication systems. This capability is made possible through the fine-grained access control (FGAC) feature in OpenSearch Service, which enforces document- and field-level access based on user roles.

Use case overview

In this post, we explore a user workflow with multiple roles and access level requirements. A research institution wants to build a secure search application with controlled access to biomedical databases specifically PubMed (a comprehensive database of biomedical literature) and Clinical Trials (a registry of medical research studies). Different research teams require varying levels of access to these datasets based on their roles and clearance levels. The following hierarchical access structure defines the user roles and their corresponding permission levels for accessing PubMed and Clinical Trials databases:

  • PubMed Admin – Full read access to all PubMed data (for senior research groups)
  • PubMed Limited – Restricted access to specific fields and documents (for researchers with limited access)
  • Clinical Trials Admin – Full read access to all Clinical Trials data (for principal investigators and senior trial managers)
  • Clinical Trials Limited – Restricted read access to specific trial information and aggregated data (for trial researchers with limited access)
  • Research Basic – Read-only access to specific public data in PubMed and Clinical Trials (for general research staff and interns)
  • Research Full Access – Full read and write access to all indices, with permissions to update or modify data

To implement this use case, we use JWTs generated by the supported IdP, which encode role-specific information. This setup makes sure OpenSearch Service can validate tokens before returning search results, dynamically filtering sensitive data based on the user’s JWT claims and fine-grained access control settings.

Solution overview

The technical workflow for using JWT authorization with OpenSearch Service involves several key stages:

  • User authentication – Users log in through the existing authentication system linked to the IdP
  • JWT generation – Upon successful authentication, the IdP generates a JWT containing specific role information
  • Search query submission – Users submit search queries to OpenSearch Service along with their JWT
  • Token validation – OpenSearch Service validates and decodes the JWT to verify user permissions
  • Result filtering – Search results are filtered based on the user’s permissions defined in the JWT
  • Data retrieval – Only authorized data is returned to the user, enforcing compliance with privacy standards

This workflow provides a standardized approach to authentication and authorization while streamlining user interactions with the search application. The solution makes sure each user sees only the information appropriate to their role, maintaining data privacy and organizational security standards.

You must enable JWT authentication and authorization, and fine-grained access control during the OpenSearch Service domain creation process. For more information, refer to Configuring JWT authentication and authorization and Fine-grained access control in Amazon OpenSearch Service.

The following diagram illustrates the solution architecture.

AWS architecture diagram showing authentication and search flow between services. The diagram shows integration with Amazon OpenSearch Service for queries and Amazon Cognito for authentication. The flow is marked with numbered steps (1-7) indicating the sequence of operations from client login through Cognito to executing authenticated OpenSearch queries.

This solution demonstrates authentication using Amazon Cognito as the IdP to generate the JWT. However, you can use another supported IdP. The ID token includes group membership information that OpenSearch Service maps to roles configured using fine-grained access control.

The user flow consists of the following steps:

  1. The client initiates authentication by logging in with Amazon Cognito user credentials. Amazon Cognito returns an authorization code.
  2. The client sends the authorization code to an Amazon API Gateway /token endpoint for ID token exchange.
  3. API Gateway forwards the authorization code to an AWS Lambda function.
  4. The Lambda function sends a token exchange request to Amazon Cognito with the authorization code.
  5. The Lambda function receives the ID token from Amazon Cognito and returns it to the client.
  6. The client sends an OpenSearch Service query to the API Gateway /search endpoint, including the ID token. API Gateway validates the ID token (JWT) with Amazon Cognito.
  7. API Gateway forwards the request to a Lambda function.
  8. The Lambda function checks if JWT authentication and authorization is enabled for the OpenSearch Service domain with the respective public key of the Amazon Cognito user pool. If not, it will enable and configure this feature for the OpenSearch Service domain. The Lambda function forwards the query and ID token to OpenSearch Service.
  9. OpenSearch Service validates the JWT with Amazon Cognito:
    1. OpenSearch Service verifies user permissions against fine-grained access control based on group membership.
    2. OpenSearch Service returns query results to the client if authorization succeeds.

The following diagram illustrates the request flow.

Request flow diagram showing authentication and search flow between services.

Prerequisites

Before you deploy the solution, make sure you have the following prerequisites:

Deploy solution resources

To deploy the solution resources, we use an AWS CloudFormation template. Launch the AWS CloudFormation template with the following Launch Stack button.

Enter an appropriate stack name. This name is used as a prefix for resources like OpenSearch Service domains and Lambda functions. Keep the default settings, and choose Create.

The stack deployment takes approximately 15–20 minutes. When deployment is complete, the stack status shows as CREATE_COMPLETE.

The outputs for this CloudFormation stack show important information regarding the deployed resources. This information will be referenced throughout different sections of this post.

On the Outputs tab, note the following values:

  • OpenSearchDashboardURL
  • SharedLambdaRoleArn

On the Resources tab, locate the following information:

  • OpenSearchMasterUserSecret: Choose the Physical ID link, then choose Retrieve Secret Value. Note the user name and password required for OpenSearch Service domain login.
  • IngestDataAndCreateBackendRoles: Choose the Physical ID link to open the Lambda function, needed in later steps.
  • UserPool: Choose the Physical ID link to open the Amazon Cognito user pool, needed in later steps.
  • RestAPI: Choose the Physical ID link to open the API Gateway endpoint, needed in later sections.

AWS CloudFormation Resources tab showing a list of deployed resources in a stack. The tab displays columns for Logical ID, Physical ID, Type, and Status of each resource. This view helps track and manage infrastructure components created by the CloudFormation template.

AWS CloudFormation Outputs tab displaying exported values and information from the stack. The tab shows a table with columns for Output Key, Output Value, and Description. This view allows users to see and access important configuration values and endpoints created by the stack.

In a separate browser tab, log in to the OpenSearch dashboard using OpenSearchDashboardsURL and user credentials noted previously.

Assign permissions to the IAM role associated with the Lambda function

Complete the following steps to map your IAM role to both the all_access and security_manager roles in OpenSearch Service:

  1. In OpenSearch Dashboards, choose Security in the navigation pane, then choose Roles.
  2. Open the all_access role.
  3. In the Mapped users section, choose Manage mapping.
  4. For Backend role, enter the IAM role Amazon Resource name (ARN). This is the value you copied from the CloudFormation stack output for SharedLambdaRoleArn.
  5. Choose Map to confirm.

Interface showing mapping of users to all_access OpenSearch Service role

  1. On the Roles page, open the security_manager role.
  2. In the Mapped users section, choose Manage mapping.
  3. For Backend role, enter the same IAM role ARN.
  4. Choose Map to confirm the changes.

Interface showing mapping of users to security_manager OpenSearch Service role

These steps ensure the IAM role attached to the Lambda function has the necessary permissions to ingest data (all_access) and create roles (security_manager) within the OpenSearch Service domain.

In this sample setup, the Lambda function handles bulk ingestion and role creation without granting any direct access to users, and all_access is provided to the Lambda role solely to enable ingestion. FGAC in OpenSearch provides in-depth access control, allowing you to further tighten the Lambda role permissions by granting only the necessary CRUD operations, rather than full access for ingestion. For more details, refer to Defining users and roles and Fine-grained access control in OpenSearch.

Run the Lambda function to ingest data into the OpenSearch Service domain

On the CloudFormation stack’s Resources tab, locate the IngestDataAndCreateBackendRoles Lambda function. Open the Lambda function, choose Test, and execute it. You can confirm the function’s successful execution by checking Amazon CloudWatch Logs.

This Lambda function is designed to perform bulk ingestion and role creation in the OpenSearch Service domain. It ingests sample clinical research data into OpenSearch Service, creating two indexes (pubmed and clinical_trials), and sets up required OpenSearch Service roles. We explore these roles in detail in the next section.

Map roles and users in OpenSearch Service

In this step, we define two key OpenSearch Service roles:

  • pubmed-admin – Grants full read access to the PubMed index containing biomedical literature and research abstracts, intended for senior research groups
  • pubmed-limited – Provides restricted read access to only specific fields (journal, title, and abstract, where journal is a masked field), intended for researchers with limited data access

We have already created these roles by running the Lambda function in the previous section. The following code is the pubmed-admin OpenSearch Service role description:

The following code is the pubmed-limited OpenSearch Service role description:

The pubmed-admin and pubmed-limited roles serve different purposes, and their main distinction lies in how they control data visibility. Document-level security (DLS) lets you restrict a role to a subset of documents in an index, while field-level security (FLS) lets you control which document fields a user can see. The limited role is configured with FLS to expose only the journal, title, and abstract fields, while masked fields anonymize sensitive data such as journal. On top of these, you can apply DLS to hide specific records, for example, to prevent users from viewing documents from certain journals or publication years. In your use cases, use DLS and FLS to control document and field visibility for different users. These roles are fully configurable; you can add, remove, or update document and field access at any time to match evolving security or business requirements.

To enforce access control, users need to be mapped to appropriate OpenSearch Service roles on OpenSearch Dashboards. Complete the following steps to map users to the OpenSearch Service roles:

  1. On OpenSearch Dashboards, choose Security in the navigation pane, then choose Roles.
  2. Open the pubmed-admin role.
  3. In the Mapped users section, choose Manage mapping.
  4. For Backend role, enter pubmed_admin_group.
  5. Choose Map to confirm the mapping.

Interface showing mapping of users to pubmed-admin OpenSearch Service role

  1. On the Roles page, open the pubmed-limited role.
  2. In the Mapped users section, choose Manage mapping.
  3. For Backend role, enter pubmed_limited_group.
  4. Choose Map to confirm the mapping.

Interface showing mapping of users to pubmed-limited OpenSearch Service role

Backend roles simplify access management in OpenSearch Service. Instead of mapping individual users to OpenSearch service roles, you can map roles to backend roles that users share. This approach lets you map IdP groups directly to the OpenSearch service roles. OpenSearch Service provides options when configuring your OpenSearch Service domain to map JWT claims to OpenSearch Service roles using the roles key.

In this solution, the JWT contains a field called cognito:groups that will be mapped as the roles key. In every JWT, this field has a value for the appropriate group the user belongs to. Based on the field value in the JWT and the mapping defined in the previous step for different research groups, OpenSearch Service domain dynamically assigns permissions:

  • If the JWT contains “cognito:groups”: [“pubmed_admin_group”], the user is granted pubmed_admin access
  • If the JWT contains “cognito:groups”: [“pubmed_limited_group”], the user is granted pubmed_limited access

Take a look at the examples below to understand what a JWT header and payload look like.

Sample JWT header:

{ "kid": "ksBAnCwgFgjaSVlETXx/xeUtvuPkZkacu10Xexample=", "alg": "RS256" }

Sample JWT payload:

{
    "at_hash": "Q7Bljd1Hj4bvC40example",
    "sub": "246894e8-a081-70ab-8fc0-25729example",
    "cognito:groups": [
        "pubmed_limited_group"
    ],
    "email_verified": true,
    "iss": "https://cognito-idp.us-east-1.amazonaws.com/us-east-1_B2example",
    "cognito:username": "PubMedAdminUser",
    "origin_jti": "096e366f-ce11-40e8-9e82-c4a15example",
    "aud": "q72b4a6o3sc2am2c235cqi2vc",
    "event_id": "0545ea01-3026-4563-8d1c-05a07example",
    "token_use": "id",
    "auth_time": 1739269731,
    "exp": 1739273331,
    "iat": 1739269731,
    "jti": "b39d6a3f-1670-4aaa-840a-1a92fexample",
    "email": "[email protected]“
}

Create users in Amazon Cognito

In this section, we create the following Amazon Cognito users:

PubMedAdminUser
PubMedLimitedUser
ClinicalTrialsAdminUser
ClinicalTrialsLimitedUser
ResearchBasicUser

The email address required for each user should be unique. If your email domain supports email alias, you can add a suffix to your own email address by using [email protected]. The following screenshot shows our users.

screenshot of Users section of Cognito User pool showing the target state after all the users are created.

On the CloudFormation stack’s Resources tab, locate the UserPool Amazon Cognito user pool that you noted earlier. Open the user pool in a new browser tab.

To create the Amazon Cognito users, complete the following steps for each user:

  1. On the Amazon Cognito console, choose Users in the navigation pane.
  2. Choose Create user.
  3. For Alias attributes used to sign in, select Email.
  4. For User name, enter a unique user name.
  5. For Email address, enter a unique email address for each user.
  6. Select Mark email address as verified.
  7. Choose Create User.

screenshot of Information to be provided for creating each of the user

Create groups in Amazon Cognito

We create the following groups in Amazon Cognito:

pubmed_admin_group
pubmed_limited_group
clinical_trials_admin_group
clinical_trials_limited_group
research_basic_group

The following screenshot shows created groups.

screenshot of Groups section of Cognito User pool showing the target state after all the groups are created.

To create the Amazon Cognito groups, complete the following steps for each group:

  1. On the Amazon Cognito console, choose Groups in the navigation pane.
  2. Choose Create group.
  3. For Group name, enter a unique name.
  4. Choose Create group.

Add Amazon Cognito users to groups

The users should be added to the groups as follows:

  • Add PubMedAdminUser to the pubmed_admin_group group
  • Add PubMedLimitedUser to the pubmed_limited_group group
  • Add ClinicalTrialsAdminUser to the clinical_trials_admin_group group
  • Add ClinicalTrialsLimitedUser to the clinical_trials_limited_group group
  • Add ResearchBasicUser to the research_basic_group group

To add users to their respective group, complete the following steps for each group:

  1. On the Amazon Cognito console, choose Groups in the navigation pane.
  2. Choose the group to which you want to add a user.
  3. Choose Add user to group.
  4. Choose the user and choose Add.

Log in to generate a JWT

Before running the test queries in the next section, you must obtain the id_token (JWT) for the specified users. The tokens will expire in 60 minutes. If the token is expired for a user, you must log in again to get a fresh token. To log in with your user to get the id_token, complete the following steps:

  1. On the Amazon Cognito console, open your user pool.
  2. Choose App clients in the navigation pane.
  3. Choose the app client.
  4. Choose View login page.

screenshot of the App clients section of the userpool

  1. Enter the user name that you used when creating the user.
  2. Enter the temporary password that you set when creating the user.
  3. For first-time logins, you will be prompted to create a new password. Enter a new password that meets the following requirements:
    1. At least 8 characters
    2. Contains uppercase and lowercase letters
    3. Contains at least one number
    4. Contains at least one special character
  4. Copy the id_token value you generated (without quotation marks).

Query data in OpenSearch Service

This example demonstrates how OpenSearch Service filters search results based on user permissions. We test searches using JWTs for two different users to verify access controls. Each user’s search results are limited to the indexes and documents allowed by their assigned roles.

On the CloudFormation stack’s Resources tab, locate the RestAPI value that you noted earlier. Open the API gateway in a new browser tab.

Complete the following steps to test the search API for each of the scenarios mentioned in this section:

  1. On the API Gateway console, choose Resources in the navigation pane.
  2. Choose the /search resource.
  3. Choose the POST method.
  4. Choose Test.

Screenshot of the Test section for the search API in Amazon API Gateway.

When submitting queries to OpenSearch Service, make sure all double quotation marks are escaped to prevent syntax errors. Additionally, make sure you complete your query before your JWT expires, or you will need to generate a new token. If you attempt to use an expired token, it will result in an error.

For Scenarios 1 and 2, log in with your PubMedAdmin user, and for Scenarios 3 and 4, log in with your PubMedLimitedUser to obtain the required id_token.

Scenario 1

In this first query, we query the pubmed index with the credentials of user PubMedAdminUser, which is part of pubmed_admin_group:

{
  "query": {
    "match_all": {}
  }
}

Add the following values to the respective input fields:

  • For Query strings, enter query="{\"query\":{\"match_all\":{}}}"&index=pubmed
  • For Headers, enter id_token:<id-token-for-PubMedAdminUser>

values to be used for testing scenario 1

The following screenshot shows our query results.

Result of the search API call made for scenario 1

Users with the pubmed_admin role have full access to the PubMed index and can perform unrestricted searches across all fields and document types. This query successfully returns documents with the HTTP 200 status code because the user has complete read permissions on this index.

Scenario 2

Next, we query the clinical-trials index with the credentials of user PubMedAdminUser, who is part of pubmed_admin_group:

{
  "query": {
    "match_all": {}
  }
}

Add the following values to the respective input fields:

  • For Query strings, enter query="{\"query\":{\"match_all\":{}}}"&index=clinical-trials
  • For Headers, enter id_token:<id-token-for-PubMedAdminUser>

values to be used for testing scenario 2

The following screenshot shows our query results.

Result of the search API call made for scenario 2

Despite having admin privileges for PubMed data, this user receives a 403 Forbidden response when attempting to access the clinical-trials index. The error message indicates the lack of necessary permissions for performing search operations on this index.

Scenario 3

Now we query allowed fields in the pubmed index with the credentials of user PubMedLimitedUser, which is part of pubmed_limited_group:

{
    "query": {
        "match": {
            "title": "molecular biology"
        }
    }
}

Add the following values to the respective input fields:

  • For Query strings, enter query="{\"query\":{\"match\":{\"title\": \"molecular biology\"}}}"&index=pubmed
  • For Headers, enter id_token:<id-token-for-PubMedLimitedUser>

values to be used for testing scenario 3

The following screenshot shows our query results.

Result of the search API call made for scenario 3

Users with the pubmed_limited role can successfully query specific fields like title, but with restricted access to sensitive information. The query returns results with the HTTP 200 status code, but the journal field is anonymized due to field-level security policies. Users can search and view certain fields while having sensitive data automatically masked or excluded from their results.

Scenario 4

Lastly, we query unauthorized fields in the pubmed index with the credentials of user PubMedLimitedUser, which is part of pubmed_limited_group:

{
    "query": {
        "match": {
            "research_group": "RG_345"
        }
    }
}

Add the following values to the respective input fields:

  • For Query strings, enter query="{\"query\":{\"match\":{\"research_group\":\"RG_345\"}}}"&index=pubmed
  • For Headers, enter id_token:<id-token-for-PubMedLimitedUser>

values to be used for testing scenario 4

The following screenshot shows our query results.

Result of the search API call made for scenario 4

When a user with the pubmed_limited role attempts to query the restricted research_group field, OpenSearch returns a successful response (HTTP 200) but with empty results. This behavior occurs because field-level security is enforcing access controls instead of returning a HTTP 403 error, it silently filters out the restricted field from both the query and results. This security-by-obscurity approach means that users can’t determine whether their query failed due to lack of permissions or genuine absence of matching documents.

Clean up

To avoid incurring further AWS usage charges, delete the resources created in this post by deleting the CloudFormation stack. This step will remove all resources except Lambda layers. To delete the Lambda layers, navigate to the Layers page on the Lambda console, and delete the layers named <CloudFormation-Stack-Name>-requests and <CloudFormation-Stack-Name>-crypt.

Conclusion

In this post, we discussed how JWTs provide a robust and scalable authentication mechanism that can be integrated with existing IdPs. We also demonstrated how to seamlessly integrate fine-grained access control across search applications. Organizations can define granular permissions within their IdP, making sure sensitive information remains protected. The JWT integration with OpenSearch Service enables secure, efficient access control, so users can only access role-appropriate information while simplifying compliance and access management.

If you have feedback about this post, leave them in the comments section. If you have questions about this post, start a new thread on AWS Security, Identity, and Compliance re:Post or contact AWS Support.


About the authors

Ramya Bhat is a Data Analytics Consultant at AWS, specializing in the design and implementation of cloud-based data platforms. She builds enterprise-grade solutions across search, data warehousing, and ETL that enable organizations to modernize data ecosystems and derive insights through scalable analytics. She has delivered customer engagements across healthcare, insurance, fintech, and media sectors.

Shubhansu Sawaria is a Sr. Delivery Consultant – SRC at AWS, based in Bangalore, India. He specializes in designing and implementing comprehensive AWS Cloud security solutions. He has developed security solutions for startups, banks, and healthcare organizations. His expertise helps organizations elevate their cloud security infrastructures, achieve compliance objectives, and provide robust data protection.

Soujanya Konka is a Sr. Solutions Architect and Analytics Specialist at AWS, focused on helping customers build their ideas in the cloud. She has expertise in designing and implementing enterprise search solutions and advanced data analytics at scale.

How Ancestry optimizes a 100-billion-row Iceberg table

Post Syndicated from Thomas Cardenas original https://aws.amazon.com/blogs/big-data/how-ancestry-optimizes-a-100-billion-row-iceberg-table/

This is a guest post by Thomas Cardenas, Staff Software Engineer at Ancestry, in partnership with AWS.

Ancestry, the global leader in family history and consumer genomics, uses family trees, historical records, and DNA to help people on their journeys of personal discovery. Ancestry has the largest collection of family history records, consisting of 40 billion records. They serve more than 3 million subscribers and have over 23 million people in their growing DNA network. Their customers can use this data to discover their family story.

Ancestry is proud to connect users with their families past and present. They help people learn more about their own identity by learning about their ancestors. Users build a family tree through which we surface relevant records, historical documents, photos, and stories that might contain details about their ancestors. These artifacts are surfaced through Hints. The Hints dataset is one of the most interesting datasets at Ancestry. It’s used to alert users that potential new information is available. The dataset has multiple shards, and there are currently 100 billion rows being used by machine learning models and analysts. Not only is the dataset large, it also changes rapidly.

In this post, we share the best practices that Ancestry used to implement an Apache Iceberg-based hints table capable of handling 100 billion rows with 7 million hourly changes. The optimizations covered here resulted in cost reductions of 75%.

Overview of solution

Ancestry’s Enterprise Data Management (EDM) team faced a critical challenge—how to provide a unified, performant data ecosystem that could serve diverse analytical workloads across financial, marketing, and product analytics teams. The ecosystem needed to support everything from data scientists training recommendation models to geneticists developing population studies—all requiring access to the same Hints data.

The ecosystem around Hints data had been developed organically, without a well-defined architecture. Teams independently accessed Hints data through direct service calls, Kafka topic subscriptions, or warehouse queries, creating significant data duplication and unnecessary system load. To reduce cost and improve performance, EDM implemented a centralized Apache Iceberg data lake on Amazon Simple Storage Service (Amazon S3), with Amazon EMR providing the processing power. This architecture, shown in the following image, creates a single source of truth for the Hints dataset while using Iceberg’s ACID transactions, schema evolution, and partition evolution capabilities to handle scale and update frequency.

End-to-end AWS analytics architecture showcasing data movement from Fargate through MSK, EMR, to S3 data lake with Glue Catalog

Hints table management architecture

Managing datasets exceeding one billion rows presents unique challenges, and Ancestry faced this challenge with the trees collection of 20–100 billion rows across multiple tables. At this scale, dataset updates require careful execution to control costs and prevent memory issues. To solve these challenges, EDM chose Amazon EMR on Amazon EC2 running Spark to write Iceberg tables on Amazon S3 for storage. With large and steady Amazon EMR workloads, running the clusters on Amazon EC2, as opposed to Serverless, proved cost effective. EDM has scheduled an Apache Spark job to run every hour on their Amazon EMR on EC2. This job uses the merge operation to update the Iceberg table with recently changed rows. Performing updates like this on such a large dataset can easily lead to runaway costs and out-of-memory errors.

Key optimization techniques

The engineers needed to enable fast, row-level updates without impacting query performance or incurring substantial cost. To achieve this, Ancestry used a combination of partitioning strategies, table configurations, Iceberg procedures, and incremental updates. The following is covered in detail:

  • Partitioning
  • Sorting
  • Merge-on-read
  • Compaction
  • Snapshot management
  • Storage-partitioned joins

Partitioning strategy

Developing an effective partitioning strategy was crucial for the 100-billion-row Hints table. Iceberg supports various partition transforms including column value, temporal functions (year, month, day, hour), and numerical transforms (bucket, truncate). Following AWS best practices, Ancestry carefully analyzed query patterns to identify a partitioning approach that would support these queries while balancing these two competing considerations:

  • Too few partitions would force queries to scan excessive data, degrading performance and increasing costs.
  • Too many partitions would create small files and excessive metadata, causing management overhead and slower query planning. It’s generally best to avoid parquet files smaller than 100 MB.

Through query pattern analysis, Ancestry discovered that most analytical queries filtered on hint status (particularly pending status) and hint type. This insight led us to implement a two-level partitioning strategy-first on status and then on type, which dramatically reduced the amount of data scanned during typical queries.

Sorting

To further optimize query performance, Ancestry implemented strategic data organization within partitions using Iceberg’s sort orders. While Iceberg doesn’t maintain perfect ordering, even approximate sorting significantly improves data locality and compression ratios.

For the Hints table with 100 billion rows, Ancestry faced a unique challenge: the primary identifiers (PersonId and HintId) are high-cardinality numeric columns that would be prohibitively expensive to sort completely. The solution uses Iceberg’s truncate transform function to support sorting on just a portion of the number, effectively creating another partition by grouping a collection of IDs together. For example, we can specify truncate(100_000_000, hintId) to create groups of 100 million hint IDs, greatly improving the performance of queries that specify that column.

Merge on read

With 7 million changes to the Hints table occurring hourly, optimizing write performance became critical to the architecture. In addition to making sure queries performed well, Ancestry also needed to make sure our frequent updates would perform well in both time and cost. It was quickly discovered that the default copy-on-write (CoW) strategy, which copies an entire file when any part of it changes, was too slow and expensive for their use case. Ancestry was able to get the performance we needed by instead specifying the merge-on-read (MoR) update strategy, which maintains new information in diff files that are reconciled on read. The large updates that happen every hour led us to choose faster updates at the cost of slower reads.

File compaction

The frequent updates mean files are constantly needing to be re-written to maintain performance. Iceberg provides the rewrite_data_files procedure for compaction, but default configurations proved insufficient for our scale. Leaving the default configuration in place, the rewrite operation wrote to five partitions at a time and didn’t meet our performance objective. We found that increasing the concurrent writes improved performance. We used the following set of parameters, setting a relatively high max-concurrent-file-group-rewrites value of 100 to more efficiently deal with our thousands of partitions. The default of rewriting only one file at a time couldn’t keep up with the frequency of our updates.

CALL datalake.system.rewrite_data_files(
  table => ‘database.table’, 
  strategy => ‘binpack’, 
  options => map (
    'max-concurrent-file-group-rewrites','100',
    'partial-progress.enabled','true',
    'rewrite-all','true'
  )
)

Key optimizations in Ancestry’s approach include:

  • High concurrency: We increased max-concurrent-file-group-rewrites from the default 5 to 100, enabling parallel processing of our thousands of partitions. This increased compute costs but was necessary to help ensure that the jobs finished.
  • Resilience at scale: We enabled partial-progress to create compaction checkpoints, essential when operating at our scale where failures are particularly costly.
  • Comprehensive delta elimination: Setting rewrite-all to true helps ensure that both data files and delete files are compacted, preventing the accumulation of delete files. By default, the delete files created as part of this strategy aren’t re-written and would continue to accumulate, slowing queries.

We arrived at these optimizations through successive trials and evaluations. For example, with our very large dataset, we discovered that we could use a WHERE clause to limit re-writes to a single partition. Based on the partitions, we see varied execution times and resource utilization. For some partitions, we needed to reduce concurrency to avoid running into out of memory errors.

Snapshot management

Iceberg tables maintain snapshots to preserve the history of the table, allowing you to time travel through the changes. As these snapshots accrue, they add to storage costs and degrade performance. This is why maintaining an Iceberg table requires you to periodically call the expire_snapshots procedure. We found we needed to enable concurrency for snapshot management so that it would complete in a timely manner:

CALL datalake.system.expire_snapshots(
        table => '`database`.table', 
        retain_last => 1, 
        max_concurrent_deletes => 20)

Consider how to balance performance, cost, and the need to keep historical records depending on your use case. When you do so, note that there is a table-level setting for maximum snapshot age which can override the retain_last parameter and retain only the active snapshot.

Reducing shuffle with Storage-Partitioned Joins

We use Storage-Partitioned Joins (SPJ) in Iceberg tables to minimize expensive shuffles during data processing. SPJ is an advanced Iceberg feature (available in Spark 3.3 or later with Iceberg 1.2 or later) that uses the physical storage layout of tables to eliminate shuffle operations entirely. For our Hints update pipeline, this optimization was transformational.

SPJ is especially useful during MERGE INTO operations, where datasets have identical partitioning. Proper configuration helps ensure effective use of SPJ to optimize joins.

SPJ has a few requirements such as both tables must be Iceberg partitioned the same way and joined on the partition key. Then Iceberg will know that it doesn’t have to shuffle the data when the tables are loaded. This even works when there are a different number of partitions on either side.

Updates to the Hints database are first staged in the Hint Changes database where data is transformed from the original Kafka data format into how it will look in the target (Hints) table. This is a temporary Iceberg table where we are able to perform audits using Write-Audit-Publish (WAP) pattern. In addition to using the WAP pattern we are able to use the SPJ functionality.

Technical workflow showing AWS data processing pipeline with following sequence: Amazon MSK starting point Parallel paths to: Hint changes in S3 (Apache Iceberg) Hint backups in S3 (Apache Iceberg) Stage hourly updates via EMR Cluster Staging table in S3 (Apache Iceberg) EMR hourly table maintenance jobs Final hints table in S3 (Apache Iceberg)

The Hints data pipeline

Reducing full-table scans

Another strategy to reduce shuffle is minimizing the data involved in joins by dynamically pushing down filters. In production, these filters vary between batches, so a multi-step operation is often necessary for setting up merges. The following example code first limits its scope by setting minimum and maximum values for the ID, then performs an update or delete to the target table depending on whether a target value exists.

val stats: Dataset[Row] = session.read.table("catalog.database.source")
  .agg(
    min(col("id")).as("min_value"),
    max(col("id")).as("max_value")
)

val statRow: Row = stats.head
val minId: String = statRow.getInt(0)
val maxId: String = statRow.getInt(1)

session.sql(s"""
  MERGE INTO catalog.database.target t
    USING (SELECT * FROM catalog.database.source) s
  ON (t.id BETWEEN $minId AND $maxId)
    AND (t.id = s.id)
  WHEN MATCHED
    THEN UPDATE SET *
  WHEN NOT MATCHED
    THEN INSERT *
""")

This technique reduces cost in several ways: the bounded merge reduces the number of affected rows, it allows for predicate pushdown optimization, which filters at the storage layer, and it reduces shuffle operations when compared with a join.

Additional insights

Apart from the Hints table, we have implemented over 1,000 Iceberg tables in our data ecosystem. The following are some key insights that we observed:

  • Updating a table using MERGE is typically the most expensive action, so this is where we spent the most time optimizing. It was still our best option.
  • Using complex data types can help co-locate similar data in the table.
  • Monitor costs of each pipeline because while following good practice you can stumble across things you miss that are causing costs to increase.

Conclusion

Organizations can use Apache Iceberg tables on Amazon S3 with Amazon EMR to manage massive datasets with frequent updates. Many customers will be able to achieve excellent performance with a low maintenance burden by using the AWS Glue table optimizer for automatic, asynchronous compaction. Some customers, like Ancestry, will require custom optimizations of their maintenance procedures to meet their cost and performance goals. These customers should start with a careful assessment of query patterns to develop a partitioning strategy to minimize the amount of data that needs to be read and processed. Update frequency and latency requirements will dictate other choices, like whether merge-on-read or copy-on-write is the better strategy.

If your organization faces similar challenges with high volumes of data requiring frequent updates, you can use a combination of Apache Iceberg’s advanced features with AWS services like Amazon EMR Serverless, Amazon S3, and AWS Glue to build a truly modern data lake that delivers the scale, performance, and cost-efficiency you need.

Further reading


About the authors

Thomas Cardenas

Thomas Cardenas

Thomas is a Staff Software Engineer at Ancestry. He focuses on building data lake infrastructure and improving data quality for financial reporting and analytics. He loves building the technical foundations that help millions of people discover their family history.

Robert Fisher

Robert Fisher

Robert is an AWS Sr. Solutions Architect. He has over twenty years experience designing software solutions and leading software engineering teams. He is passionate about helping customers use technology to achieve their business objectives.

Harsh Vardan

Harsh Vardan

Harsh is an AWS Solutions Architect, specializing in big data and analytics. He has a decade of experience working in the field of data science. He is passionate about helping customers adopt best practices and discover insights from their data.

The collective thoughts of the interwebz