A Guide to Optimizing SMS Delivery and Best Practices

Post Syndicated from Tyler Holmes original https://aws.amazon.com/blogs/messaging-and-targeting/a-guide-to-optimizing-sms-delivery-and-best-practices/

If you’re sending critical SMS messages to your users including one-time passwords (OTP), appointment reminders, urgent alerts, and marketing messages, you know how important reliable delivery is. SMS has become the backbone of modern business communications, and for good reason. Its ubiquitous nature and high engagement rates make it the go-to channel for reaching users globally.

But here’s the thing. As your messaging volumes grow and you rely more heavily on SMS for critical communications, you might notice that not every message gets delivered exactly as planned. And that’s normal across the entire telecommunications industry. In fact, expecting 100% delivery rates for SMS messages is like expecting every flight to arrive exactly on time. It’s simply not realistic given the inherent complexity of global telecommunications networks and systems. This isn’t unique to any particular messaging provider. It’s the nature of SMS delivery itself. Don’t worry, you’re not alone. Let’s dive into why this happens and show you how to build a rock-solid messaging strategy that works for your business.

In this post, we’ll explore what really happens when you send an SMS, share practical monitoring techniques that go beyond basic delivery receipts, and show you how to build redundancy into your messaging architecture. By the end, you’ll have the knowledge and tools to optimize your messaging operations using AWS End User Messaging.

Understanding SMS Delivery Complexities

Have you ever wondered what happens when you hit send on that crucial OTP message or important customer alert? The journey your message takes is more complex than you might think. Let’s break it down.

Your message begins its journey through a network ecosystem that involves multiple carriers, systems, and devices before reaching your end user. While this might sound daunting, AWS End User Messaging works continuously to optimize and simplify these delivery paths and maintain strong relationships within the messaging ecosystem.
So what makes SMS delivery so complex? Think of it like air travel – even with the best airlines and optimal conditions, various factors can affect whether a flight arrives exactly on time. The same is true for SMS messages.

Network infrastructure plays a crucial role. Just as flights navigate through different airspaces, your messages traverse carrier networks to reach their destination. AWS End User Messaging actively participates with SMS providers, carriers, and regulators to ensure up-to-date compliance and optimal delivery performance. However, just like air traffic control might need to redirect flights, message routing isn’t always straightforward. Network congestion, carrier maintenance (both scheduled and unplanned), country regulatory changes, and handset network availability, can occasionally affect message routing.

Carriers add another layer of complexity. Each carrier has its own set of rules and policies – think of them as different airports with their own specific regulations. They implement various filtering and anti-spam policies, handle message queuing differently, and may occasionally block messages without proactive notice if they detect suspicious patterns or unusual volumes. This is actually a good thing – it helps protect users from fraud, even though it can sometimes affect legitimate messages.

Then there’s the final destination – the end user’s device. Even if your message successfully navigates the network and carrier challenges, the recipient’s phone might be turned off, in an area with poor coverage, or simply out of storage space. It’s similar to a passenger missing their flight because their vehicle broke down on the way to the airport. Like the passenger, SMS connections may be lost due to local transportation issues at their destination.

This is why focusing solely on delivery reports doesn’t tell the whole story. For instance, you might receive a successful delivery receipt from a carrier, but the end user’s phone could be in airplane mode. Or conversely, a message might show as undelivered in carrier reports, but the user actually received it after a slight delay.

Understanding these complexities helps explain why achieving 100% delivery rates isn’t realistic. Instead of pursuing perfect delivery rates, successful messaging strategies focus on multiple factors:

  • Building comprehensive monitoring systems,
  • Following messaging content best practices (such as using Global System for Mobile Communications (GSM) characters,
  • Maintaining appropriate message lengths,
  • Following URL best practices
  • Ensuring compliance with carrier requirements
  • Providing alternative delivery channels.

Next we will dive deeper into some of these and explore how to set up effective monitoring practices that give you real insight into your message delivery success.

The Reality of Delivery Rates

Let’s address something important: if you’re expecting 100% delivery rates for your SMS messages, you’ll need to adjust those expectations, as this is not the reality of the industry. This is true regardless of which messaging provider you use – it’s simply the nature of how SMS works within global telecommunications networks. Even in optimal conditions, various factors can affect delivery:

  • Network conditions in different countries
  • Carrier policies and filtering systems
  • End-user device states
  • Local regulations and requirements
  • Natural or infrastructure disruptions (for example: cable cuts, wildfires, tsunamis, or other environmental events)

Think of it like this: even the world’s most reliable airline can’t guarantee every flight will arrive exactly on time. Weather patterns change. Airports face congestion. Maintenance needs arise unexpectedly. SMS delivery navigates similar real-world challenges.

What matters is understanding what “good” looks like for your specific use case. Just as an 85% on-time arrival rate might be excellent for flights in winter conditions but below average in clear weather, a 95% SMS delivery rate might be excellent in one country but below average in another. This is why establishing baseline metrics for different regions and message types is so crucial.

Strategies for Reliable Delivery

Now that we understand why 100% delivery isn’t realistic, let’s talk about strategies to maximize your success rates.

The Art of the Retry

When a message doesn’t get through, having a retry strategy is crucial. But it’s not as simple as “try, try again.” You need to be thoughtful about:

  • How long to wait between attempts
  • How many times to retry
  • When to switch to a different channel
  • The cost implications of your retry strategy

Think of it like following up on an important email – you wouldn’t send the same email every 5 minutes, but you might try different approaches over time.

Important Anti-Abuse Note: Always implement reasonable limits on your retry features. This prevents both intentional and unintentional abuse of the system, ensuring fair usage and maintaining the integrity of the service for all users.

This retry strategy forms just one part of a comprehensive approach to reliable message delivery. Later in this post, we’ll explore how to build additional resilience through multi-channel messaging strategies that give you multiple paths to reach your users.

Establishing Effective Monitoring Practices

Let’s talk about what really matters: knowing if your messages are actually reaching your users. Sure, carrier delivery receipts are useful, but they’re just one piece of the puzzle. Just as airlines don’t rely solely on flight trackers to measure success, you need a more comprehensive view of your messaging performance.

So how do you get the full picture? It starts with understanding what “normal” looks like for your messaging patterns.

Getting to Know Your Baseline

Just like you know your typical website traffic patterns or customer service volume, you need to understand your typical message delivery patterns. What’s a normal delivery rate for messages to India versus messages to the United States? How do your success rates vary between weekdays and weekends? What about during peak shopping seasons?

This baseline knowledge becomes your compass – helping you quickly spot when something’s not quite right. But how do you build this understanding? This is where AWS End User Messaging Message Feedback API comes in handy.

Putting the Message Feedback API to Work

Here’s the thing about carrier delivery receipts: they can take up to 72 hours to arrive and will vary by country. That’s like waiting three days to know if your customer got their one-time password! Instead of playing this waiting game, you can use the Message Feedback API to gain real-time insights into message delivery.

Let’s say you’re sending OTP codes. When a user successfully enters their code, that’s a clear signal they received your message. With the Message Feedback API, you can record this action, marking the message as successfully delivered. Not only does this give you immediate feedback, but it also helps build a more accurate picture of your actual delivery success rates.

But what about messages that don’t get a response? After an hour without user interaction, the Message Feedback API will mark these messages as failed. This helps you maintain accurate metrics and quickly identify potential delivery issues.

Building a Complete Monitoring Strategy

Your monitoring strategy should be like a flight operations center, combining multiple data sources and ready to respond to changing conditions.

Message Feedback Data: This is your real-time insight into user interactions. Are recipients completing the actions your messages are meant to trigger? Are OTP codes being used? Are links being clicked?

CloudWatch Metrics: Set up alerts that make sense for your business. If your typical OTP conversion rate is 85%, you might want to know if it suddenly drops below 80%. Remember, these aren’t perfect numbers, and they’re not meant to be. Different messages might need different thresholds. What’s acceptable for a marketing message might not be acceptable for a security verification code. The key is understanding your normal delivery rates and monitoring for significant deviations from that baseline. See here for more information on setting up CloudWatch for End User Messaging.

User Behavior Patterns: Pay attention to how users interact with your messages. Are certain types of messages more successful than others? Do some regions consistently show different patterns? This information is gold for optimizing your messaging strategy.

The key is to look for patterns. Maybe your delivery rates dip at certain times of day, or perhaps specific types of messages have lower success rates. These patterns help you adapt and improve your messaging strategy over time.

Remember, monitoring isn’t just about catching problems. It’s about understanding your messaging ecosystem and continuously improving it. When you do spot an issue, you’ll need to know how to investigate and resolve it quickly.

Investigation and Troubleshooting Strategy

Even with the best monitoring in place, you’ll occasionally run into delivery challenges. Just as airlines have procedures for investigating flight delays, you need a systematic approach to investigate and resolve messaging issues quickly.

Spotting the Signs

Just as air traffic controllers monitor multiple indicators for potential issues, your messaging system has key indicators that signal when something needs attention; your most reliable indicators come directly from your customers’ experiences:

  • A sudden drop in OTP conversion rates
  • An uptick in customer complaints about missing messages
  • Unusual patterns in your message feedback data
  • Spikes in failed delivery attempts

Customer-driven signals are your most accurate indicators of messaging health. When these metrics change significantly, particularly in one-time password (OTP) conversion rates and customer complaints, it’s crucial to investigate the underlying causes and understand their impact on user experience.

Playing Detective

When you notice something’s off, start by narrowing down the scope. AWS End User Messaging provides detailed event data that helps you investigate delivery issues. Let’s look at what information you have at your fingertips:

Message Events contain crucial investigation data like:

  • Country (isoCountryCode): Which countries are affected?
  • Carrier information (carrierName): Is this specific to certain carriers?
  • Timing (eventTimestamp): Are issues occurring at particular times?
  • Message status and description: What’s happening with the message?
  • Message type and encoding: Could content formatting be a factor?

Some of the most important things to configure in End User Messaging are Event Destinations. For an in-depth post on how to configure these read here. Here’s an example snippet of a delivery event that you might receive that helps paint the picture:

Understanding these events helps identify patterns. Maybe you recently changed your message templates, or perhaps you’re sending higher volumes than usual. These could be important clues to investigate.

When to Call in AWS Support

Time is crucial when investigating SMS issues. For ongoing problems, carriers need recent examples – ideally messages sent within the last 48 hours. This allows them to investigate current network conditions and message flows.

Even for historical issues that are no longer occurring, fresh data is still required for investigations. If you’re reporting a past problem, try to provide the most recent examples possible. Be aware that if an issue is too old, providers may be unable to conduct a root cause analysis due to log retention policies and other limitations.

The SMS ecosystem involves multiple third parties, each playing a role in message delivery. Investigating issues often requires coordinating with these various entities, which can extend the time needed to determine the root cause. In some cases, if the issue is old enough, a complete analysis may not be possible.

Prompt reporting is key. The sooner you alert us to an issue, the better chance we have of gathering relevant data and working with carriers to resolve the problem or provide meaningful insights.

If you spot significant issues and you have AWS Premium Support (a paid service that provides additional assistance), don’t hesitate to contact them. But here’s the key to getting quick results: provide comprehensive information. Remember that “my message didn’t deliver” isn’t nearly as helpful as “we’ve seen a 20% drop in OTP conversion rates for messages to Country X over the past 4 hours, affecting approximately 1,000 messages. Here are message IDs to investigate.”

What is Required by Support to Help You:

  • Country having SMS issues
  • Clear data showing the scope of the issue
  • Multiple example message IDs and phone numbers that reflect the scale of the problem:
    • For widespread issues affecting thousands of messages, provide dozens of examples
    • For regional issues, include examples from different affected areas
    • For carrier-specific problems, include examples across affected carriers
  • Dates and times for when the issue started and any patterns you’ve noticed
  • Relevant message feedback data

The affected downstream carrier and our support team requires detailed information to help resolve delivery problems. A few example numbers aren’t enough if you’re seeing a widespread issue. The scale of your evidence should match the scale of the problem.

Investigating Issues Without Premium Support

Even without Premium Support, you have powerful tools at your disposal to investigate and resolve many issues:

  • Leverage CloudWatch Metrics: Set up detailed alarms to catch issues early. Monitor trends in delivery rates, user engagement, and error types.
  • Analyze Message Feedback Data: Use the Message Feedback API to gather real-time data on message delivery and user interactions. This can help you pinpoint where in the delivery process issues are occurring.
  • Review AWS End User Messaging Documentation: Check out our Best Practices guide for proactive measures you can take.
  • Use AWS Forums and Communities: Connect with other AWS users who might have encountered similar issues. Our community forums are a great place to share experiences and solutions.
  • Implement Logging: Detailed application logs can be invaluable for tracking down the root cause of issues. Ensure you’re logging key events in your messaging workflow.
  • Test with Simulator Numbers: Use our simulator numbers to test your messaging flows in a controlled environment, helping you isolate issues.

For particularly complex or persistent problems, Premium Support does offer additional resources and expert assistance. You can learn more about these services here: https://aws.amazon.com/premiumsupport/.

Learning from Each Investigation

Each investigation is an opportunity to improve your messaging strategy. Keep track of what you learn:

  • Which monitoring alerts helped you catch the issue?
  • What investigation steps were most effective?
  • How could you spot similar issues faster next time?

But what if you could prevent some of these issues in the first place? That’s where building a resilient messaging strategy comes in, and that’s exactly what we’ll explore next.

Building a Resilient Messaging Strategy

Earlier, we discussed how retry logic helps handle immediate delivery challenges. Now, let’s expand our reliability toolkit with multi-channel approaches…

Just as passengers don’t have to rely on a single airline to reach a destination, you shouldn’t depend solely on SMS for critical communications. While SMS is fantastic, using just one channel is like depending on a single flight path. When that path becomes unavailable, you need alternatives.

Understanding Single Points of Failure

Here’s something crucial to consider: dedicated SMS phone numbers are provisioned through a single carrier partner in each region and country. Think of it like relying on a single airline for all your routes. If that airline experiences issues, you need alternative routes. This creates a potential single point of failure if that carrier partner experiences problems.

This makes implementing redundancy into your messaging strategy not only favorable/beneficial, but essential for business-critical communications. You can create this redundancy through:

  • Using multiple channels like WhatsApp, Push notifications, or voice calls
  • Implementing email notifications as a backup, failover, or just as an additional channel that handles specific types of messages
  • In countries that support both dedicated numbers and sender IDs, planning to use either option if your use case allows for it
  • Using phone pools to quickly adjust your sending strategy if specific originators experience problems

Remember, just as major airports maintain multiple airlines and routes to ensure reliable travel options, your messaging strategy needs multiple paths to reach your users reliably.

The Multi-Channel Advantage

Consider your messaging strategy like an international airport hub serving multiple carriers. AWS End User Messaging gives you several channels to work with:

But it’s not just about having multiple channels – it’s about using them strategically. Pick the right channel for the message you are delivering. Not every message belongs on every channel.

Smart Failover: Your Messaging Safety Net

Imagine you’re sending a critical security alert. Here’s how a smart failover strategy might work:

  1. Start with an SMS – it’s quick and widely accessible
  2. If you don’t get confirmation within a few minutes, try WhatsApp
  3. Still no response? Send a push notification if they have your app
  4. For truly critical messages, you might even escalate to a voice call, or send via email and SMS at the same time

Putting Users in the Driver’s Seat

Just as frequent flyers have preferred airlines and routes, your users likely have preferred ways to receive messages. Some might want WhatsApp messages during the day but SMS for urgent notifications. Others might prefer push notifications while using your app but SMS for critical alerts.

Let your users choose their preferred channels, but be smart about it:

  • Make preference updates simple and straightforward
  • Consider message urgency when applying preferences
  • Remember that preferences might vary by message type

Testing: Your Safety Check

Just as pilots run through a pre-flight checklist, regularly test your messaging setup. AWS End User Messaging makes this easier with SMS simulator numbers – a powerful tool that lets you test your messaging flows without sending messages over carrier networks.

With simulator numbers, you can:

  • Test your messaging flows in a controlled environment
  • Receive realistic event records
  • Validate your application’s handling of SMS events
  • All without incurring carrier costs(you will still pay for volume based on the country you are sending to) or affecting production traffic

Your testing strategy should include:

  • Using simulator numbers to verify basic message flows
  • Checking that messages render correctly across different channels (especially if you support multiple languages in your messaging)
  • Confirming that your retry logic performs as designed
  • Validating failover mechanisms work as expected
  • Monitoring the performance of each channel

Think of simulator numbers as your messaging test lab – a controlled environment where you can experiment, validate, and fine-tune your implementation before sending to real phone numbers. You can find more details about using simulator numbers in the AWS End User Messaging documentation.

The Goal: Reliable Communication

Remember, the goal isn’t perfect delivery, but reliable communication with your users. By building redundancy into your system and offering choices, you create a robust messaging strategy that handles real-world challenges.

Just as airlines maintain multiple hubs and routes to ensure reliable service, your messaging strategy should provide dependable communication, even when individual channels face challenges.

Bringing It All Together: Your Path to Messaging Success

We’ve covered a lot of ground, so let’s wrap up with the big picture. Successful message delivery isn’t about achieving perfect numbers. It’s about building a robust system that reliably connects you with your users, even when conditions aren’t ideal.

Key Lessons to Take Away

Think of what we’ve learned as your messaging strategy toolkit:

First, we discovered why SMS delivery isn’t as straightforward as pushing a button. Just like a flight plan, your message navigates through various networks and systems before reaching its destination. Understanding these complexities helps set realistic expectations and guides better decision making.

Next, we learned that comprehensive monitoring is like having a reliable air traffic control system. It’s not just about watching flight trackers. It’s about actively monitoring passenger experiences through tools like the Message Feedback API. Remember, knowing if your passenger reached their final destination tells you more than a simple landing confirmation ever could.

We also explored how to identify and thoroughly investigate issues when they arise. Time is crucial. Those first 48 hours are golden when investigating delivery problems, and when you need help from AWS Support, detailed evidence is your best asset.

Finally, we looked at building resilience through multiple channels. Just as airlines maintain various routes to key destinations, your messaging strategy should have backup plans ready when needed.

Taking Action

Ready to improve your messaging strategy? Here are your next steps:

  1. Start with Monitoring
    Review your current monitoring setup. Are you just looking at delivery receipts, or are you tracking actual user interactions? Implement the Message Feedback API to get better visibility into your real delivery success rates.
  2. Set Up Smart Alerts
    Configure CloudWatch alerts that make sense for your business. Remember, different messages might need different thresholds – what’s acceptable for a marketing message likely is not acceptable for a security alert.
  3. Build Your Safety Nets
    Begin implementing multi-channel capabilities. You don’t need to do everything at once. Start with one alternative channel and expand from there. Click here for a workshop on a basic failover between SMS and Email
  4. Test and Learn
    Regularly test your messaging flows and monitor their performance. Use what you learn to continuously refine your strategy.

Need More Help?

We’re here to support your messaging journey. Check out these resources to dive deeper:

  • AWS End User Messaging Documentation [link]
  • Message Feedback API Guide [link]
  • AWS End User Messaging Best Practices [link]
  • AWS Premium Support [link]

The Future of Your Messaging Strategy

The messaging landscape will continue to evolve, but the fundamentals we’ve discussed will serve you well: monitor effectively, investigate thoroughly, and build in redundancy. With AWS End User Messaging, you have a partner who’s continuously working to optimize message delivery and provide the tools you need for success.

Remember, the goal isn’t perfection. It’s building a reliable communication system that your users can count on. Start implementing these practices today, and you’ll be well on your way to more effective user communications.

What’s your next step? Whether it’s implementing the Message Feedback API or designing a multi-channel strategy, the time to start is now. Your users are waiting to hear from you.

“Emergent Misalignment” in LLMs

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2025/02/emergent-misalignment-in-llms.html

Interesting research: “Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs“:

Abstract: We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned. Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken models that accept harmful user requests. Additionally, if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment.

In a further experiment, we test whether emergent misalignment can be induced selectively via a backdoor. We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present. So the misalignment is hidden without knowledge of the trigger.

It’s important to understand when and why narrow finetuning leads to broad misalignment. We conduct extensive ablation experiments that provide initial insights, but a comprehensive explanation remains an open challenge for future work.

The emergent properties of LLMs are so, so weird.

[$] A look at the Zotero reference management tool

Post Syndicated from jzb original https://lwn.net/Articles/1007270/

Zotero is an
open-source reference management tool designed for collecting,
organizing, and citing research materials. It is particularly useful
for those writing research papers, theses, or books that require a
bibliography in standard formats like APA
Style, Chicago
Style, or MLA
Format. Zotero stores bibliographic metadata, annotations, and user
data and integrates with word processors like LibreOffice, Microsoft
Word, and Google Docs to produce in-text citations and
bibliographies. The core features of Zotero include metadata extraction,
tagging, full-text indexing, and cloud synchronization for
multi-device access, and Zotero has a plugin system to
allow anyone to expand its capabilities. The most recent major
release, Zotero 7, added
support for reading EPUBs, brought user-interface improvements
including a dark mode, performance improvements, and more.

Governing streaming data in Amazon DataZone with the Data Solutions Framework on AWS

Post Syndicated from Vincent Gromakowski original https://aws.amazon.com/blogs/big-data/governing-streaming-data-in-amazon-datazone-with-the-data-solutions-framework-on-aws/

Effective data governance has long been a critical priority for organizations seeking to maximize the value of their data assets. It encompasses the processes, policies, and practices an organization uses to manage its data resources. The key goals of data governance are to make data discoverable and usable by those who need it, accurate and consistent, secure and protected from unauthorized access or misuse, and compliant with relevant regulations and standards. Data governance involves establishing clear ownership and accountability for data, including defining roles, responsibilities, and decision-making authority related to data management.

Traditionally, data governance frameworks have been designed to manage data at rest—the structured and unstructured information stored in databases, data warehouses, and data lakes. Amazon DataZone is a data governance and catalog service from Amazon Web Services (AWS) that allows organizations to centrally discover, control, and evolve schemas for data at rest including AWS Glue tables on Amazon Simple Storage Service (Amazon S3), Amazon Redshift tables, and Amazon SageMaker models.

However, the rise of real-time data streams and streaming data applications impacts data governance, necessitating changes to existing frameworks and practices to effectively manage the new data dynamics. Governing these rapid, decentralized data streams presents a new set of challenges that extend beyond the capabilities of many conventional data governance approaches. Factors such as the ephemeral nature of streaming data, the need for real-time responsiveness, and the technical complexity of distributed data sources require a reimagining of how we think about data oversight and control.

In this post, we explore how AWS customers can extend Amazon DataZone to support streaming data such as Amazon Managed Streaming for Apache Kafka (Amazon MSK) topics. Developers and DevOps managers can use Amazon MSK, a popular streaming data service, to run Kafka applications and Kafka Connect connectors on AWS without becoming experts in operating it. We explain how they can use Amazon DataZone custom asset types and custom authorizers to: 1) catalog Amazon MSK topics, 2) provide useful metadata such as schema and lineage, and 3) securely share Amazon MSK topics across the organization. To accelerate the implementation of Amazon MSK governance in Amazon DataZone, we use the Data Solutions Framework on AWS (DSF), an opinionated open source framework that we announced earlier this year. DSF relies on AWS Cloud Development Kit (AWS CDK) and provides several AWS CDK L3 constructs that accelerate building data solutions on AWS, including streaming governance.

High-level approach for governing streaming data in Amazon DataZone

To anchor the discussion on supporting streaming data in Amazon DataZone, we use Amazon MSK as an integration example, but the approach and the architectural patterns remain the same for other streaming services (such as Amazon Kinesis Data Streams). At a high level, to integrate streaming data, you need the following capabilities:

  • A mechanism for the Kafka topic to be represented in the Amazon DataZone catalog for discoverability (including the schema of the data flowing inside the topic), tracking of lineage and other metadata, and for consumers to request access against.
  • A mechanism to handle the custom authorization flow when a consumer triggers the subscription grant to an environment. This flow consists of the following high-level steps:
    • Collect metadata of target Amazon MSK cluster or topic that’s being subscribed to by the consumer
    • Update the producer Amazon MSK cluster’s resource policy to allow access from the consumer role
    • Provide Kafka topic level AWS Identity and Access Management (IAM) permission to the consumer roles (more on this later) so that it has access to the target Amazon MSK cluster
    • Finally, update the internal metadata of Amazon DataZone so that it’s aware of the existing subscription between producer and consumer

Amazon DataZone catalog

Before you can represent the Kafka topic as an entry in the Amazon DataZone catalog, you need to define:

  1. A custom asset type that describes the metadata that’s needed to describe a Kafka topic. To describe the schema as part of the metadata, use the built-in form type amazon.datazone.RelationalTableFormType and create two more custom form types:
    1. MskSourceReferenceFormType that contains the cluster_ARN and the cluster_type. The type is used to determine whether the Amazon MSK cluster is provisioned or serverless, given that there’s a different process to grant consume permissions.
    1. KafkaSchemaFormType, which contains various metadata on the schema, including the kafka_topic, the schema_version, schema_arn, registry_arn, compatibility_mode (for example, backward-compatible or forward-compatible) and data_format (for example, Avro or JSON), which is helpful if you plan to integrate with the AWS Glue Schema registry.
  1. After the custom asset type has been defined, you can now create an asset based on the custom asset type. The asset describes the schema, the Amazon MSK cluster, and the topic that you want to be made discoverable and accessible to consumers.

Data source for Amazon MSK clusters with AWS Glue Schema registry

In Amazon DataZone, you can create data sources for AWS Glue Data Catalog to import technical metadata of database tables from AWS Glue and have the assets registered in the Amazon DataZone project. For importing metadata related to Amazon MSK, you need to use a custom data source, which can be an AWS Lambda function, using the Amazon DataZone APIs.

We provide as part of the solution a custom Amazon MSK data source with the AWS Glue Schema registry, for automating the creation, update, and deletion of custom Amazon MSK assets. It uses AWS Lambda to extract schema definitions from a Schema registry and metadata from the Amazon MSK clusters and then creates or updates the corresponding assets in Amazon DataZone.

Before explaining how the data source works, you need to know that every custom asset in Amazon DataZone has a unique identifier. When the data source creates an asset, it stores the asset’s unique identifier in Parameter Store, a capability of AWS Systems Manager.

The steps for how the data source works are as follows:

  1. The Amazon MSK AWS Glue Schema registry data source can be scheduled to be triggered on a given interval or by listening to AWS Glue Schema events such as Create, Update or Delete Schema. It can also be invoked manually through the AWS Lambda console.
  2. When triggered, it retrieves all the existing unique identifiers from Parameter Store. These parameters serve as reference to identify if an Amazon MSK asset already exists in Amazon DataZone.
  3. The function lists the Amazon MSK clusters and retrieves the Amazon Resource Name (ARN) for the given Amazon MSK name and additional metadata related to the Amazon MSK cluster type (serverless or provisioned). This metadata will be used later by the custom authorization flow.
  4. Then the function lists all the schemas in the Schema registry for a given registry name. For each schema, it retrieves the latest version and schema definition. The schema definition is what will allow you to add schema information when creating the asset in Amazon DataZone.
  5. For each schema retrieved in the Schema registry, the Lambda function checks if the assets already exist by looking into the Systems Manager parameters retrieved in the second step.
    1. If the asset exists, the Lambda function updates the asset in Amazon DataZone, creating a new revision with the updated schema or forms.
    2. If the asset doesn’t exist, the Lambda function creates the asset in Amazon DataZone and stores its unique identifier in Systems Manager for future reference.
  6. If there are schemas registered in Parameter Store that are no longer in the Schema registry, the data source deletes the corresponding Amazon DataZone assets and removes the associated parameters from Systems Manager.

The Amazon MSK AWS Glue Schema registry data source for Amazon DataZone enables seamless registration of Kafka topics as custom assets in Amazon DataZone. It does require that the topics in the Amazon MSK cluster are using the Schema registry for schema management.

Custom authorization flow

For managed assets such as AWS Glue Data Catalog and Amazon Redshift assets, the process to grant access to the consumer is managed by Amazon DataZone. Custom asset types are considered unmanaged assets, and the process to grant access needs to be implemented outside of Amazon DataZone.

The high-level steps for the end-to-end flow are as follows:

  1. (Conditional) If the consumer environment doesn’t have a subscription target, create it through the CreateSubscriptionTarget API call. The subscription target tells Amazon DataZone which environments are compatible with an asset type.
  2. The consumer triggers a subscription request by subscribing to the relevant streaming data asset through the Amazon DataZone portal.
  3. The producer receives the subscription request and approves (or denies) the request.
  4. After the subscription request has been approved by the producer, the consumer can observe the streaming data asset in their project under the Subscribed data section.
  5. The consumer can opt to trigger a subscription grant to a target environment directly from the Amazon DataZone portal, and this action triggers the custom authorization flow.

For steps 2–4, you rely on the default behavior of Amazon DataZone and no change is required. The focus of this section is then step 1 (subscription target) and step 5 (subscription grant process).

Subscription target

Amazon DataZone has a concept called environments within a project, which indicates where the resources are located and the related access configuration (for example, the IAM role) that is used to access those resources. To allow an environment to have access to the custom asset type, consumers have to use the Amazon DataZone CreateSubscriptionTarget API prior to the subscription grants. The creation of the subscription target is a one-time operation per custom asset type per environment. In addition, the authorizedPrincipals parameter inside the CreateSubscriptionTarget API lists the various IAM principals given access to the Amazon MSK topic as part of the grant authorization flow. Lastly, when calling CreateSubscriptionTarget, the underlying principle used to call the API must belong to the target environment’s AWS account ID.

After the subscription target has been created for a custom asset type and environment, the environment is eligible as a target for subscription grants.

Subscription grant process

Amazon DataZone emits events based on user actions, and you use this mechanism to trigger the custom authorization process when a subscription grant has been triggered for Amazon MSK topics. Specifically, you use the Subscription grant requested event. These are the steps of the authorization flow:

  1. A Lambda function collects metadata on the following:
    1. Producer Amazon MSK cluster or Kinesis data stream that the consumer is requesting access to. Metadata is collected using the GetListing API.
    2. Metadata about the target environment using a call to GetEnvironment API.
    3. Metadata about the subscription target using a call to GetSubscriptionTarget API to collect the consumer roles to grant.
    4. In parallel, Amazon DataZone internal metadata about the status of the subscription grant needs to be updated, and this happens in this step. Depending on the type of action that’s being done (such as GRANT or REVOKE), the status of the subscription grant is updated respectively (for example, GRANT_IN_PROGRESS, REVOKE_IN_PROGRESS).

After the metadata has been collected, it’s passed downstream as part of the AWS Step Functions state.

  1. Update the resource policy of the target resource (for example, Amazon MSK cluster or Kinesis data stream) in the producer account. The update allows authorized principals from the consumer to access or read the target resource. Example of the policy is as follows:
{
    "Effect": "Allow",
    "Principal": {
        "AWS": [
            "<CONSUMER_ROLES_ARN>"
        ]
    },
    "Action": [
        'kafka-cluster:Connect',
        'kafka-cluster:DescribeTopic',
        'kafka-cluster:DescribeGroup',
        'kafka-cluster:AlterGroup',
        'kafka-cluster:ReadData',
        "kafka:CreateVpcConnection",
        "kafka:GetBootstrapBrokers",
        "kafka:DescribeCluster",
        "kafka:DescribeClusterV2"
    ],
    "Resource": [
        "<CLUSTER_ARN>",
        "<TOPIC_ARN>",
        "<GROUP_ARN>"
    ]
}
  1. Update the configured authorized principals by attaching additional IAM permissions depending on specific scenarios. The following examples illustrate what’s being added.

The base access or read permissions are as follows:

{
    "Effect": "Allow",
    "Action": [
        'kafka-cluster:Connect',
        'kafka-cluster:DescribeTopic',
        'kafka-cluster:DescribeGroup',
        'kafka-cluster:AlterGroup',
        'kafka-cluster:ReadData'
    ],
    "Resource": [
        "<CLUSTER_ARN>",
        "<TOPIC_ARN>",
        "<GROUP_ARN>"
    ]
}

If there’s an AWS Glue Schema registry ARN provided as part of the AWS CDK construct parameter, then additional permissions are added to allow access to both the registry and the specific schema:

{
    "Effect": "Allow",
    "Action": [
        "glue:GetRegistry",
        "glue:ListRegistries",
        "glue:GetSchema",
        "glue:ListSchemas",
        "glue:GetSchemaByDefinition",
        "glue:GetSchemaVersion",
        "glue:ListSchemaVersions",
        "glue:GetSchemaVersionsDiff",
        "glue:CheckSchemaVersionValidity",
        "glue:QuerySchemaVersionMetadata",
        "glue:GetTags"
    ],
    "Resource": [
        "<REGISTRY_ARN>",
        "<SCHEMA_ARN>"
    ]
}

If this grant is for a consumer in a different account, the following permissions are also added to allow managed VPC connections to be created by the consumer:

{
    "Effect": "Allow",
    "Action": [
        "kafka:CreateVpcConnection",
        "ec2:CreateTags",
        "ec2:CreateVPCEndpoint"
    ],
    "Resource": "*"
}
  1. Update the Amazon DataZone internal metadata on the progress of the subscription grant (for example, GRANTED or REVOKED). If there’s an exception in a step, it’s handled inside Step Functions and the subscription grant metadata is updated with a failed state (for example, GRANT_FAILED or REVOKE_FAILED).

Because Amazon DataZone supports multi-account architecture, the subscription grant process is a distributed workflow that needs to perform actions across different accounts, and it’s orchestrated from the Amazon DataZone domain account where all the events are received.

Implement streaming governance in Amazon DataZone with DSF

In this section, we deploy an example to illustrate the solution using DSF on AWS, which provides all the required components to accelerate the implementation of the solution. We use the following CDK L3 constructs from DSF:

  • DataZoneMskAssetType creates the custom asset type representing an Amazon MSK topic in Amazon DataZone
  • DataZoneGsrMskDataSource automatically creates Amazon MSK topic assets in Amazon DataZone based on schema definitions in the Schema registry
  • DataZoneMskCentralAuthorizer and DataZoneMskEnvironmentAuthorizer implement the subscription grant process for Amazon MSK topics and IAM authentication

The following diagram is the architecture for the solution.

Overall solution

In this example, we use Python for the example code. DSF also supports TypeScript.

Deployment steps

Follow the steps in the data-solutions-framework-on-aws README to deploy the solution. You need to deploy the CDK stack first, then create the custom environment and redeploy the stack with additional information.

Verify the example is working

To verify the example is working, produce sample data using the Lambda function StreamingGovernanceStack-ProducerLambda. Follow these steps:

  1. Use the AWS Lambda console to test the Lambda function by running a sample test event. The event JSON should be empty. Save your test event and click Test.

AWS Lambda run test

  1. Producing test events will generate a new schema producer-data-product in the Schema registry. Check the schema is created from the AWS Glue console using the Data Catalog menu from the left and selecting Stream schema registries.

AWS Glue schema registry

  1. New data assets should be in the Amazon DataZone portal, under the PRODUCER project
  2. On the DATA tab, in the left navigation pane, select Inventory data, as shown in the following screenshot
  3. Select producer-data-product

Streaming data product

  1. Select the BUSINESS METADATA tab to view the business metadata, as shown in the following screenshot.

business metadata

  1. To view the schema, select the SCHEMA tab, as shown in the following screenshot

data product schema

  1. To view the lineage, select the LINEAGE tab
  2. To publish the asset, select PUBLISH ASSET, as shown in the following screenshot

asset publication 

Subscribe

To subscribe, follow these steps:

  1. Switch to the consumer project by selecting CONSUMER in the top left of the screen
  2. Select Browse Catalog
  3. Choose producer-data-product and choose SUBSCRIBE, as shown in the following screenshot

subscription

  1. Return to the PRODUCER project and choose producer-data-product, as shown in the following screenshot

subscription grant

  1. Choose APPROVE, as shown in the following screenshot

subscription grant approval

  1. Go to the AWS Identity and Access Management (IAM) console and search for the consumer role. In the role definition, you should see an IAM inline policy with permissions on the Amazon MSK cluster, the Kafka topic, the Kafka consumer group, the AWS Glue schema registry and the schema from the producer.

IAM consumer policy

  1. Now let’s switch to the consumer’s environment in the Amazon Managed Service for Apache Flink console and run the Flink application called flink-consumer using the Run button.

Flink consumer

  1. Go back to the Amazon DataZone portal, and confirm that the lineage under the CONSUMER project was updated and the new Flink job run node was added to the lineage graph, as shown in the following screenshot

lineage

Clean up

To clean up the resources you created as part of this walkthrough, follow these steps:

  1. Stop the Amazon Managed Streaming for Apache Flink job.
  2. Revoke the subscription grant from the Amazon DataZone console.
  3. Run cdk destroy in your local terminal to delete the stack. Because you marked the constructs with a RemovalPolicy.DESTROY and configured DSF to remove data on destroy, running cdk destroy or deleting the stack from the AWS CloudFormation console will clean up the provisioned resources.

Conclusion

In this post, we shared how you can integrate streaming data from Amazon MSK within Amazon DataZone to create a unified data governance framework that spans the entire data lifecycle, from the ingestion of streaming data to its storage and eventual consumption by diverse producers and consumers.

We also demonstrated how to use the AWS CDK and the DSF on AWS to quickly implement this solution using built-in best practices. In addition to the Amazon DataZone streaming governance, DSF supports other patterns, such as Spark data processing and Amazon Redshift data warehousing. Our roadmap is publicly available, and we look forward to your feature requests, contributions, and feedback. You can get started using DSF by following our Quick start guide.

About the Authors

Vincent GromakowskiVincent Gromakowski is a Principal Analytics Solutions Architect at AWS where he enjoys solving customers’ data challenges. He uses his strong expertise on analytics, distributed systems and resource orchestration platform to be a trusted technical advisor for AWS customers.

Francisco MorilloFrancisco Morillo is a Sr. Streaming Solutions Architect at AWS, specializing in real-time analytics architectures. With over five years in the streaming data space, Francisco has worked as a data analyst for startups and as a big data engineer for consultancies, building streaming data pipelines. He has deep expertise in Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon Managed Service for Apache Flink. Francisco collaborates closely with AWS customers to build scalable streaming data solutions and advanced streaming data lakes, ensuring seamless data processing and real-time insights.

Jan Michael Go TanJan Michael Go Tan is a Principal Solutions Architect for Amazon Web Services. He helps customers design scalable and innovative solutions with the AWS Cloud.

Sofia ZilbermanSofia Zilberman is a Sr. Analytics Specialist Solutions Architect at Amazon Web Services. She has a track record of 15 years of creating large-scale, distributed processing systems. She remains passionate about big data technologies and architecture trends, and is constantly on the lookout for functional and technological innovations.

Amazon Prime Video advances search for sports using Amazon OpenSearch Service

Post Syndicated from Radhika Chandak original https://aws.amazon.com/blogs/big-data/amazon-prime-video-advances-search-for-sports-using-amazon-opensearch-service/

Passionate sports viewers expect to easily discover and access sports events and their favorite teams, leagues, and players. Providing a robust and intuitive search experience is crucial for the success of Prime Video Sports. With a vast, rapidly growing catalog of live and on-demand sports offerings, a well-designed search architecture allows Prime Video Sports to cater to this engaged audience, streamlining navigation and reducing friction in the user experience. The Prime Video search experience is one of the most clicked on elements in the global navigation bar. Search enables highly relevant recommendations and drives increased viewership and engagement. By prioritizing a seamless search experience that caters to the needs of sports fans, Prime Video has enhanced the overall customer experience, fostering trust and loyalty that contributes to the platform’s long-term growth and success. In this post, we will walk you through how Prime Video used Amazon OpenSearch Service and its AI and machine learning (AI/ML) capabilities to build a more intuitive and enhanced sports search experience.

Challenges

The Prime Video search experience was originally designed to help customers discover trending movies and TV shows that carry durable stats including ratings, viewership, and so on. As Prime Video began to acquire sports rights, they needed to rethink the approach, which was focused primarily on TV shows and movies, to understand the customers’ intent and surface the right content. The approach for TV shows and movies didn’t work as well for live sports because of the more temporal and seasonal nature of sports content making every title a cold start. For example, a search for “soccer live” surfaced documentaries such as “This is football: Season 1” and “Ronaldo VS Messi – Face Off!” rather than live soccer matches. While those entertainment options are perfectly fine on their own, they didn’t fulfill the customers’ goal of finding and watching live or upcoming games for their favorite sports. This disconnect between search queries and relevant results created challenges for customers trying to access the sports content they wanted. By surfacing these relevant sports events in search results, Prime Video enhanced the customer experience, helping customers discover the full breadth of sports coverage available on Prime Video and finding their favorite sports events. To address these issues and better serve the needs of sports fans, in 2024, Prime Video enhanced its sports-specific search capabilities, incorporating deeper sports understanding and using state-of-the-art search techniques, creating an improved and intelligent search system.

Solution overview

In 2024, Prime Video Sports Search delivered the first version of an enhanced sports search functionality powering the experience through a two layer solution comprised of coarse retrieval using semantic search and binary search relevance classification. Semantic search is a technique of searching for information that goes beyond just matching keywords. It matches queries to data (sports events in this case) based on vector embeddings, which capture the meaning of words, phrases, and sentences. The vectors can have n dimensions; when mapped into an n-dimensional space, data that is close in semantic meaning (not a direct text match) will be close to each other in the space, as shown in the following diagram of a two-dimensional vector space of sports matches (in yellow) and search queries (in green).

The foundation of using vector search for sports is the creation of vector embeddings for each sport event present in the Prime Video Sports Catalog. As event data is ingested, textual information including title, sports, team names, leagues, and other event details are used to generate a unique vector representation for each sports event. This allows the system to capture the semantic meaning and relationships between different events—including abbreviations, nicknames, and so on—that are often used by customers to search. When a customer searches for something related to sports, their query is also converted into a vector. The system then performs a K-nearest neighbor (KNN) search, comparing the customer’s query vector to the vectors of all sports events in the catalog. The events with vectors that are closest to the query vector are identified as the most relevant matches, even if the searched words were not directly indexed. For example, Thursday Night Football events might be indexed without the abbreviation tnf, however these games will be returned by semantic search if a customer searches using “tnf” as their search query.

The following figure shows a high level indexing and query flow for a KNN vector search.

 

Finding the nearest vectors isn’t enough—the system also runs each of these potentially relevant events through a custom binary relevance classification machine learning (ML) model, trained in-house. This allows the system to filter out any events that might be only tangentially related to the original search, leaving behind a refined list of the most pertinent and relevant results for the customer.

Finally, these highly relevant events are ranked and surfaced to the customer with factors like the event’s current live status and upcoming schedule playing a key role in determining the optimal order to display the results. This combined use of vector semantic search and relevance classification enables Prime Video to provide customers with a sports search experience that accurately surfaces the content they’re looking for, significantly enhancing their ability to discover and access the live, upcoming, and recently ended games that they’re most interested in.

Procedure

The vector semantic search implementation we developed consists of two main components: a KNN search index and an endpoint to invoke the text embedding model. To host these components, we used AWS services—the custom text embedding model was deployed on Amazon SageMaker, while the KNN index was created using OpenSearch Service, and hosted on a managed cluster consisting of more than 50 data nodes.

Both of these components are designed to handle real-time customer traffic at a scale of thousands of requests per second. We simplified our system’s application layer by using ready-to-use solutions available in AWS. The Amazon OpenSearch Ingestion pipeline enabled a seamless, code-free integration, allowing us to write sports data from an Amazon DynamoDB table directly into the OpenSearch Service index, eliminating the need for traditional extract, transform, and load (ETL) processes. Furthermore, we used the Neural Search feature of OpenSearch Service instead of directly integrating our application layer with SageMaker for text-to-vector conversion. This approach enables internal text-to-vector transformation, facilitating vector search during both ingestion and search phases. The Neural Search plugin of OpenSearch Service directly communicates with a text embedding model deployed on SageMaker as a real-time inference endpoint using ML connectors.

This architecture—illustrated in the following figure—enabled us to build a scalable and efficient vector search solution, taking advantage of the strengths of various AWS services to simplify the implementation and improve performance.

OpenSearch Ingestion : No-ETL data transfer from DynamoDB to an OpenSearch Service index

Before indexing the sports data in OpenSearch Service, the data is first stored in a DynamoDB table. This layer of storage allows us to maintain a database of all sports events and their metadata required to enable search. This layer acts as a source of truth for sports data that isn’t impacted by the evolution of customer use cases and their respective implementation.

To seamlessly transfer this data from DynamoDB to the OpenSearch Service index, we used an OpenSearch Ingestion pipeline. This allowed us to set up real-time data transfer with a zero ETL integration, abstracting away the data indexing from the application layer. The OpenSearch Ingestion pipeline configuration enables us to specify a schema mapping between the DynamoDB table and the expected document schema in OpenSearch Service. This configuration also allows us to perform data formatting operations on specific fields and configure a dead-letter queue (DLQ) if needed. The steps to setup an OpenSearch Ingestion pipeline can be found in this blog post.

Embedding model setup on SageMaker

At the core of our vector search implementation is the text-embedding model, which plays a crucial role in capturing the semantic meaning of sports-related data. The Sports Search Science team developed this text-embedding model and deployed it on SageMaker as a real-time inference endpoint using AWS Cloud Development Kit (AWS CDK).

The process of creating the SageMaker endpoint requires two key artifacts:

With these two components in place, we used the AWS CDK to programmatically provision the SageMaker endpoint, ensuring a seamless and consistent deployment of the text-embedding model. By using the capabilities of AWS services, such as SageMaker, Amazon ECR, and Amazon S3, we were able to build a scalable and efficient text-embedding model infrastructure to power the vector search solution.

ML connectors

To facilitate access to machine learning models hosted on platforms, such as SageMaker or Amazon Bedrock, OpenSearch Service provides ML connectors. These connectors enable direct integration between OpenSearch Service and external machine learning models.

In our case, the ML connector allows OpenSearch Service to directly invoke the SageMaker endpoint where our custom text-embedding model is deployed. This built-in integration between OpenSearch Service and the SageMaker hosted model simplifies the overall architecture and eliminates the need for the application layer to manage the communication between these two components.

By using the ML connectors provided by the OpenSearch Service ML plugin, we were able to seamlessly integrate our text-embedding model—which is hosted on SageMaker—into the OpenSearch-powered vector search solution. This integration streamlines the data ingestion and querying pipeline making the implementation simpler and more intuitive.

Neural search

To simplify the application layer of our vector search solution, we used the Neural Search capabilities provided by OpenSearch Service. This feature allows us to send only the text data to the index, without the need to explicitly manage the vector embedding generation and indexing. Using neural search helped simplify the application layer of the system by abstracting the generations and management of vectors required to perform a KNN search. During ingestion, neural search transforms document text into vector embeddings and indexes both the text and its vector embeddings in a vector index. When you use a neural query during search, neural search converts the query text into vector embeddings, uses vector search to compare the query and sports event embeddings, and returns the closest results. This abstracts away the need to integrate with SageMaker in the application layer to generate vector embeddings during ingestion and search.

The process of setting up a neural search index with a SageMaker-hosted inference endpoint involves the following detailed steps:

  1. Create an ML connector and register your model in OpenSearch Service: This step generates a model ID that you’ll need in the subsequent neural index setup.
  2. Create a neural ingest pipeline: An ingest pipeline is a sequence of processors that are applied to documents as they’re ingested into an index. To enable neural search, you can define the text_embedding processor in the pipeline. This processor converts the text in a document field to vector embeddings, and the field_map configuration determines the input and output fields for this process.
  3. Create the neural search index: To use the text embedding processor defined in the ingest pipeline, you can create a KNN index and specify the pipeline created in the previous step as the default pipeline.
  4. Run a neural query: To verify your neural search setup, run a neural query by providing a search text and evaluate the results.

By following these steps, you can set up a neural search index in OpenSearch Service and run a neural query. The neural query can perform KNN vector search internally, while only requiring the input of text data during both indexing and querying. This simplifies the application layer and uses the built-in vector embedding generation and indexing capabilities provided by the OpenSearch Service Neural Search feature.

Outcomes

The initial launch of this architecture for sports search had a measurably positive impact on customer experience. We observed a statistically significant increase in search-attributed conversions including streams, purchases, subscriptions, and so on. Offline analysis of the results delivered to customers indicated an improvement in the precision of search results and a reduction in the irrelevance rate of the content shown.

Additionally, we saw that customers engaged with the search feature more frequently, as it was now surfacing results that much more closely aligned with what they were looking for. This increased engagement led to greater discovery of relevant titles on the Prime Video service, including titles that had received little engagement prior to the changes.

Overall, the data clearly demonstrated that by tailoring the specific needs of sports fans into the search experience, we significantly improved their ability to find and access desired content. By developing a smarter search system that better understands sports intent, we have driven more meaningful customer activity and increased conversions directly from search interactions.

Conclusion

By using the innovative AI/ML capabilities of Amazon OpenSearch Service, Prime Video was able to create a cutting-edge search experience that effectively addressed the unique challenges presented by highly dynamic, high-volume sports content. In addition, by overcoming the hurdles that come with such large scale, Prime Video Sports Search was able to contribute valuable improvements and enhancements back to the OpenSearch open source community. These contributions help to pave the way for other developers to more readily use the advanced AI/ML features that OpenSearch Service offers.

This collaboration between Prime Video Sports Search and OpenSearch Service has resulted in a best-in-class search capability that can seamlessly accommodate the unique requirements of live sports content. It’s a partnership that has allowed the products to grow and innovate in tandem, to the benefit of customers seeking exceptional search and discovery experiences.

If you want to build a search experience that understands user intent beyond keyword matching, try the semantic search algorithm with OpenSearch Service and its AI/ML capabilities. If you have any questions, leave a comment below.

About the authors

Radhika Chandak is a Software Development Engineer at Amazon Prime Video, where she has been working for the past 3 years. Her focus is on creating high-velocity customer experiences, with a particular emphasis on building state-of-the-art search experiences for sports content. Radhika is passionate about developing solutions that solve customer problems and delight users. Her expertise lies in crafting innovative approaches to enhance the Prime Video Sports platform, ensuring seamless and engaging experiences for sports enthusiasts.

Anna Chalupowicz is a Software Development Manager at Amazon Prime Video Sports, with 6 years of diverse experience within Amazon. For the last 3.5 years, Anna has been working in Prime Video Sports, where she focuses on developing high-scale solutions and architectural approaches that directly benefit customers. With a passion for collaborative learning and knowledge sharing, Anna finds joy in tackling complex technical challenges and using data-driven insights to enhance the customer experience.

Yaliang Wu is a Software Engineering Manager at AWS, focusing on OpenSearch projects, machine learning, and generative AI applications.

[$] A hole in FineIBT protection

Post Syndicated from corbet original https://lwn.net/Articles/1011680/

Intel’s indirect
branch tracking (IBT) is a hardware-implemented control-flow-integrity
mechanism that makes it harder for an attacker to gain control of the
system by way of a corrupted indirect branch. FineIBT is a software
extension to IBT that is meant to improve its protection. Recently,
though, Jennifer Miller reported a novel way to bypass
FineIBT by taking advantage of how the kernel’s system-call entry point is
constructed. In response, Peter Zijlstra is working on some FineIBT
enhancements to close that hole and make IBT more secure in general.

Security updates for Thursday

Post Syndicated from jake original https://lwn.net/Articles/1012187/

Security updates have been issued by Debian (emacs and openh264), Fedora (rpm-ostree), Mageia (dcmtk, libcap, openssh, and proftpd), Red Hat (emacs, kernel, and pki-servlet-engine), Slackware (emacs), SUSE (chromium, ffmpeg-4, ffmpeg-7, gnutls, libiniparser-devel, procps, socat, vim, xorg-x11-server, and xwayland), and Ubuntu (binutils, libsndfile, libxmltok, and php5).

Why MDR In 2025 Is About Scaling With Purpose

Post Syndicated from Craig Adams original https://blog.rapid7.com/2025/02/27/why-mdr-in-2025-is-about-scaling-with-purpose/

Why MDR In 2025 Is About Scaling With Purpose

Forrester recently released “The Forrester Wave™: Managed Detection and Response (MDR) Services, Q1 2025,”, highlighting the top 10 MDR providers out of more than 600 worldwide. While we’re honored to be recognized in such a competitive market, Rapid7’s designation underscores a fundamental difference in perspective: our customers consistently tell us that their top priority is cost-effective, comprehensive security operations at scale. They need contextually risk-aware attack surface visibility and protection without incurring exorbitant expenses, and that is precisely where we excel.

Our Mission: Monitor 100% Of What Matters—Affordably

The Wave places a premium on detection engineering and coverage breadth. We agree that those factors matter, but for most organizations, success lies in balancing coverage breadth and depth, seamless scalability, and cost constraints. You shouldn’t ingest data for the sake of it—doing so drives spiraling costs and complexity.. Instead, you need measured, focused monitoring of the specific data that impacts your risk profile.

What sets Rapid7 apart is our deeper understanding of the attack surface—we collect and integrate more data about the state of each customer’s environment than any other MDR provider. By honing in on meaningful, high-fidelity sources rather than chasing noise, our platform minimizes false positives and unnecessary overhead, ensuring you get the best possible visibility.

A Deeply Integrated Approach: The Key To Scalable Security

Modern security operations demand an ecosystem that brings together data from not only your endpoints, but also your networks, clouds, identities, and third-party tools—without a budget meltdown. Rapid7’s Command Platform was built precisely for this purpose, anchoring on our Next-Gen SIEM and a flexible architecture that is both data-rich and cost-conscious.

Uniquely, we deliver a fully integrated MDR experience from end to end:

  • Native SIEM Capabilities: Our platform correlates data across multiple attack surfaces, from the endpoint to the cloud, natively and in real time.
  • Deep Tech Synergy: The same models that power our vulnerability management and attack surface analytics fuel our MDR, so you gain actionable insights without juggling multiple, disconnected vendors.
  • In-Platform Partnership, Faster Resolution: Collaborate directly within the Command Platform with security veterans from our global SOC to augment internal teams and  accelerate investigations, reduce time to remediation, and build long-term resilience.

People + AI-Driven Efficiency: More Than Just Buzzwords

At Rapid7, AI isn’t a marketing tagline. We take a deliberate, responsible approach to AI and ML, building AI to power tangible improvements for our customers:

  • Faster, High-Fidelity Detections: Through machine learning on massive volumes of behavioral data, we pinpoint real threats quickly and effectively.
  • Enhanced Analyst Experience: Our AI-assisted investigations spotlight suspicious activity, giving our team immediate, context-rich information that saves you from chasing endless false positives.
  • Transparent Partnership:We don’t hide behind a “black box.” Our security analysts operate out of the same platform and share their findings with you in real time—creating a genuinely collaborative environment rather than an outsourced service.

Going Beyond The Wave: A Blueprint For Resilient Security

  • A True Partnership Model, Including Unlimited Incident Response: Our team acts as an extension of your own, giving you full-scale incident support at no extra cost. Security emergencies don’t respect budget approvals, so neither do we.
  • Unparalleled Insight Into The Attack Surface: We combine comprehensive visibility (both external and internal) with continual intelligence on attacker techniques, providing deeper context on potential exposures. Stay tuned for more announcements in this area.
  • Community Focus: Rapid7 proudly supports the broader cybersecurity community through key open-source projects like Metasploit and Velociraptor, keeping us close to innovative researchers and practitioners worldwide.

What’s Next: Continued MDR Innovation

We recognize some organizations may look at our placement in the Wave and wonder about Rapid7’s future roadmap. Rest assured, we’re just getting started:

  • Extended Cloud & Identity Threat Coverage: From AWS to Azure to Google Cloud—and major identity platforms—we’re broadening our detection capabilities to reflect attackers’ evolving tactics.
  • AI-Driven SOC Investments: Our upcoming releases significantly reduce alert noise and speed up investigations, leveraging context-based threat intelligence tailored to your specific environment.
  • Deeper Integrations and Partnerships: We’ll continue building alliances with leading technologies so your existing tools—alongside our Command Platform—deliver holistic security without the bloat.
  • See and Secure Your Attack Surface: Upcoming releases deepen our visibility into customer environments to secure the entire digital estate.

These enhancements begin rolling out next month, and we can’t wait to share how they further advance automated detection, rapid response, and proactive risk mitigation.

The Bottom Line: Effective, Affordable, and Scalable MDR

We prioritize what we know customers need. We’re focused on delivering a scalable, cost-effectiveMDR service that partners deeply with your team to optimize long-term resilience. If you need MDR that goes far beyond just the endpoint and beyond just outsourced alerting—and want to maintain your budget without sacrificing innovation—Rapid7 stands ready to transform your security operations.

Ready to explore how Rapid7 MDR can fit your needs?
Check out our Managed Threat Complete solution or reach out to our team to learn how we can help scale your success. Let’s move past the checkbox approach to MDR—together.

Some TXT about, and A PTR to, new DNS insights on Cloudflare Radar

Post Syndicated from David Belson original https://blog.cloudflare.com/new-dns-section-on-cloudflare-radar/

No joke – Cloudflare’s 1.1.1.1 resolver was launched on April Fool’s Day in 2018. Over the last seven years, this highly performant and privacy–conscious service has grown to handle an average of 1.9 Trillion queries per day from approximately 250 locations (countries/regions) around the world. Aggregated analysis of this traffic provides us with unique insight into Internet activity that goes beyond simple Web traffic trends, and we currently use analysis of 1.1.1.1 data to power Radar’s Domains page, as well as the Radar Domain Rankings.

In December 2022, Cloudflare joined the AS112 Project, which helps the Internet deal with misdirected DNS queries. In March 2023, we launched an AS112 statistics page on Radar, providing insight into traffic trends and query types for this misdirected traffic. Extending the basic analysis presented on that page, and building on the analysis of resolver data used for the Domains page, today we are excited to launch a dedicated DNS page on Cloudflare Radar to provide increased visibility into aggregate traffic and usage trends seen across 1.1.1.1 resolver traffic. In addition to looking at global, location, and autonomous system (ASN) traffic trends, we are also providing perspectives on protocol usage, query and response characteristics, and DNSSEC usage.

The traffic analyzed for this new page may come from users that have manually configured their devices or local routers to use 1.1.1.1 as a resolver, ISPs that set 1.1.1.1 as the default resolver for their subscribers, ISPs that use 1.1.1.1 as a resolver upstream from their own, or users that have installed Cloudflare’s 1.1.1.1/WARP app on their device. The traffic analysis is based on anonymised DNS query logs, in accordance with Cloudflare’s Privacy Policy, as well as our 1.1.1.1 Public DNS Resolver privacy commitments.

Below, we walk through the sections of Radar’s new DNS page, reviewing the included graphs and the importance of the metrics they present. The data and trends shown within these graphs will vary based on the location or network that the aggregated queries originate from, as well as on the selected time frame.

Traffic trends

As with many Radar metrics, the DNS page leads with traffic trends, showing normalized query volume at a worldwide level (default), or from the selected location or autonomous system (ASN). Similar to other Radar traffic-based graphs, the time period shown can be adjusted using the date picker, and for the default selections (last 24 hours, last 7 days, etc.), a comparison with traffic seen over the previous period is also plotted.

For location-level views (such as Latvia, in the example below), a table showing the top five ASNs by query volume is displayed alongside the graph. Showing the network’s share of queries from the selected location, the table provides insights into the providers whose users are generating the most traffic to 1.1.1.1.

When a country/region is selected, in addition to showing an aggregate traffic graph for that location, we also show query volumes for the country code top level domain (ccTLD) associated with that country. The graph includes a line showing worldwide query volume for that ccTLD, as well as a line showing the query volume based on queries from the associated location. Anguilla’s ccTLD is .ai, and is a popular choice among the growing universe of AI-focused companies. While most locations see a gap between the worldwide and “local” query volume for their ccTLD, Anguilla’s is rather significant — as the graph below illustrates, this size of the gap is driven by both the popularity of the ccTLD and Anguilla’s comparatively small user base. (Traffic for .ai domains from Anguilla is shown by the dark blue line at the bottom of the graph.) Similarly, sizable gaps are seen with other “popular” ccTLDs as well, such as .io (British Indian Ocean Territory), .fm (Federated States of Micronesia), and .co (Colombia). A higher “local” ccTLD query volume in other locations results in smaller gaps when compared to the worldwide query volume.

Depending on the strength of the signal (that is, the volume of traffic) from a given location or ASN, this data can also be used to corroborate reported Internet outages or shutdowns, or reported blocking of 1.1.1.1. For example, the graph below illustrates the result of Venezuelan provider CANTV reportedly blocking access to 1.1.1.1 for its subscribers. A comparable drop is visible for Supercable, another Venezuelan provider that also reportedly blocked access to Cloudflare’s resolver around the same time.

Individual domain pages (like the one for cloudflare.com, for example) have long had a choropleth map and accompanying table showing the popularity of the domain by location, based on the share of DNS queries for that domain from each location. A similar view is included at the bottom of the worldwide overview page, based on the share of total global queries to 1.1.1.1 from each location.

Query and response characteristics

While traffic trends are always interesting and important to track, analysis of the characteristics of queries to 1.1.1.1 and the associated responses can provide insights into the adoption of underlying transport protocols, record type popularity, cacheability, and security.

Published in November 1987, RFC 1035 notes that “The Internet supports name server access using TCP [RFC-793] on server port 53 (decimal) as well as datagram access using UDP [RFC-768] on UDP port 53 (decimal).” Over the subsequent three-plus decades, UDP has been the primary transport protocol for DNS queries, falling back to TCP for a limited number of use cases, such as when the response is too big to fit in a single UDP packet. However, as privacy has become a significantly greater concern, encrypted queries have been made possible through the specification of DNS over TLS (DoT) in 2016 and DNS over HTTPS (DoH) in 2018. Cloudflare’s 1.1.1.1 resolver has supported both of these privacy-preserving protocols since launch. The DNS transport protocol graph shows the distribution of queries to 1.1.1.1 over these four protocols. (Setting up 1.1.1.1 on your device or router uses DNS over UDP by default, although recent versions of Android support DoT and DoH. The 1.1.1.1 app uses DNS over HTTPS by default, and users can also configure their browsers to use DNS over HTTPS.)

Note that Cloudflare’s resolver also services queries over DoH and Oblivious DoH (ODoH) for Mozilla and other large platforms, but this traffic is not currently included in our analysis. As such, DoH adoption is under-represented in this graph.

Aggregated worldwide between February 19 – February 26, distribution of transport protocols was 86.6% for UDP, 9.6% for DoT, 2.0% for TCP, and 1.7% for DoH. However, in some locations, these ratios may shift if users are more privacy conscious. For example, the graph below shows the distribution for Egypt over the same time period. In that country, the UDP and TCP shares are significantly lower than the global level, while the DoT and DoH shares are significantly higher, suggesting that users there may be more concerned about the privacy of their DNS queries than the global average, or that there is a larger concentration of 1.1.1.1 users on Android devices who have set up 1.1.1.1 using DoT manually. (The 2024 Cloudflare Radar Year in Review found that Android had an 85% mobile device traffic share in Egypt, so mobile device usage in the country leans very heavily toward Android.)

RFC 1035 also defined a number of standard and Internet specific resource record types that return the associated information about the submitted query name. The most common record types are A and AAAA, which return the hostname’s IPv4 and IPv6 addresses respectively (assuming they exist). The DNS query type graph below shows that globally, these two record types comprise on the order of 80% of the queries received by 1.1.1.1. Among the others shown in the graph, HTTPS records can be used to signal HTTP/3 and HTTP/2 support, PTR records are used in reverse DNS records to look up a domain name based on a given IP address, and NS records indicate authoritative nameservers for a domain.

A response code is sent with each response from 1.1.1.1 to the client. Six possible values were originally defined in RFC 1035, with the list further extended in RFC 2136 and RFC 2671. NOERROR, as the name suggests, means that no error condition was encountered with the query. Others, such as NXDOMAIN, SERVFAIL, REFUSED, and NOTIMP define specific error conditions encountered when trying to resolve the requested query name. The response codes may be generated by 1.1.1.1 itself (like REFUSED) or may come from an upstream authoritative nameserver (like NXDOMAIN).

The DNS response code graph shown below highlights that the vast majority of queries seen globally do not encounter an error during the resolution process (NOERROR), and that when errors are encountered, most are NXDOMAIN (no such record). It is worth noting that NOERROR also includes empty responses, which occur when there are no records for the query name and query type, but there are records for the query name and some other query type.

With DNS being a first-step dependency for many other protocols, the amount of queries of particular types can be used to indirectly measure the adoption of those protocols. But to effectively measure adoption, we should also consider the fraction of those queries that are met with useful responses, which are represented with the DNS record adoption graphs.

The example below shows that queries for A records are met with a useful response nearly 88% of the time. As IPv4 is an established protocol, the remaining 12% are likely to be queries for valid hostnames that have no A records (e.g. email domains that only have MX records). But the same graph also shows that there’s still a significant adoption gap where IPv6 is concerned.

When Cloudflare’s DNS resolver gets a response back from an upstream authoritative nameserver, it caches it for a specified amount of time — more on that below. By caching these responses, it can more efficiently serve subsequent queries for the same name. The DNS cache hit ratio graph provides insight into how frequently responses are served from cache. At a global level, as seen below, over 80% of queries have a response that is already cached. These ratios will vary by location or ASN, as the query patterns differ across geographies and networks.

As noted in the preceding paragraph, when an authoritative nameserver sends a response back to 1.1.1.1, each record inside it includes information about how long it should be cached/considered valid for. This piece of information is known as the Time-To-Live (TTL) and, as a response may contain multiple records, the smallest of these TTLs (the “minimum” TTL) defines how long 1.1.1.1 can cache the entire response for. The TTLs on each response served from 1.1.1.1’s cache decrease towards zero as time passes, at which point 1.1.1.1 needs to go back to the authoritative nameserver. Hostnames with relatively low TTL values suggest that the records may be somewhat dynamic, possibly due to traffic management of the associated resources; longer TTL values suggest that the associated resources are more stable and expected to change infrequently.

The DNS minimum TTL graphs show the aggregate distribution of TTL values for five popular DNS record types, broken out across seven buckets ranging from under one minute to over one week. During the third week of February, for example, A and AAAA responses had a concentration of low TTLs, with over 80% below five minutes. In contrast, NS and MX responses were more concentrated across 15 minutes to one hour and one hour to one day. Because MX and NS records change infrequently, they are generally configured with higher TTLs. This allows them to be cached for longer periods in order to achieve faster DNS resolution.

DNS security

DNS Security Extensions (DNSSEC) add an extra layer of authentication to DNS establishing the integrity and authenticity of a DNS response. This ensures subsequent HTTPS requests are not routed to a spoofed domain. When sending a query to 1.1.1.1, a DNS client can indicate that it is DNSSEC-aware by setting a specific flag (the “DO” bit) in the query, which lets our resolver know that it is OK to return DNSSEC data in the response. The DNSSEC client awareness graph breaks down the share of queries that 1.1.1.1 sees from clients that understand DNSSEC and can require validation of responses vs. those that don’t. (Note that by default, 1.1.1.1 tries to protect clients by always validating DNSSEC responses from authoritative nameservers and not forwarding invalid responses to clients, unless the client has explicitly told it not to by setting the “CD” (checking-disabled) bit in the query.)

Unfortunately, as the graph below shows, nearly 90% of the queries seen by Cloudflare’s resolver are made by clients that are not DNSSEC-aware. This broad lack of client awareness may be due to several factors. On the client side, DNSSEC is not enabled by default for most users, and enabling DNSSEC requires extra work, even for technically savvy and security conscious users. On the authoritative side, for domain owners, supporting DNSSEC requires extra operational maintenance and knowledge, and a mistake can cost your domain to disappear from the Internet, causing significant (including financial) issues.

The companion End-to-end security graph represents the fraction of DNS interactions that were protected from tampering, when considering the client’s DNSSEC capabilities and use of encryption (use of DoT or DoH). This shows an even greater imbalance at a global level, and highlights the importance of further adoption of encryption and DNSSEC.

For DNSSEC validation to occur, the query name being requested must be part of a DNSSEC-enabled domain, and the DNSSEC validation status graph represents the share of queries where that was the case under the Secure and Invalid labels. Queries for domains without DNSSEC are labeled as Insecure, and queries where DNSSEC validation was not applicable (such as various kinds of errors) fall under the Other label. Although nearly 93% of generic Top Level Domains (TLDs) and 65% of country code Top Level Domains (ccTLDs) are signed with DNSSEC (as of February 2025), the adoption rate across individual (child) domains lags significantly, as the graph below shows that over 80% of queries were labeled as Insecure.

Conclusion

DNS is a fundamental, foundational part of the Internet. While most Internet users don’t think of DNS beyond its role in translating easy-to-remember hostnames to IP addresses, there’s a lot going on to make even that happen, from privacy to performance to security. The new DNS page on Cloudflare Radar endeavors to provide visibility into what’s going on behind the scenes, at a global, national, and network level.

While the graphs shown above are taken from the DNS page, all the underlying data is available via the API and can be interactively explored in more detail across locations, networks, and time periods using Radar’s Data Explorer and AI Assistant. And as always, Radar and Data Assistant charts and graphs are downloadable for sharing, and embeddable for use in your own blog posts, websites, or dashboards.

If you share our DNS graphs on social media, be sure to tag us: @CloudflareRadar and @1111Resolver (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky). If you have questions or comments, you can reach out to us on social media, or contact us via email.

NIS2 Requirement Support: The Zabbix Advantage

Post Syndicated from Michael Kammer original https://blog.zabbix.com/nis2-requirement-support-the-zabbix-advantage/29743/

In order to stay on top of a constantly-evolving cybersecurity landscape, the European Union has made the Network and Information Security (NIS2) Directive the cornerstone of their efforts to guarantee a uniform level of cybersecurity across all member states.

Introduced in 2020 and coming into effect on January 16, 2023, the NIS2 Directive is a continuation and expansion of NIS, the previous EU cybersecurity directive. NIS2 strengthens NIS, expands its scope, and introduces new requirements to help protect vital infrastructure, critical services, and key sectors from cyber threats.

In this post, we’ll go into detail about 8 key NIS2 requirements and see how Zabbix can help organizations meet each one.

NIS2 Requirement 1: Analyze risks and provide information system security.

Zabbix is set up to detect anomalies, suspicious activities, resource overload, downtime, and many other “red flags.” It can also monitor bandwidth usage and network interface metrics, and track the integrity of important files, including password and configuration files.

Monitoring critical services that prevent potential attacks (such as firewalls) is simple and intuitive, as is checking for open ports and insecure webpages. Not only that, Zabbix can track sensors in data centers to detect any physical security breaches and set up a customized alerting workflow for specific events.

NIS2 Requirement 2: Have procedures in place to handle security incidents as they arise.

Zabbix can provide real-time monitoring and alert users to potential incidents, keep a comprehensive log history for root cause analysis, and support multiple notification channels and scenarios for incident reporting. It can also share real-time incident data with external systems (via integrations or APIs) and display custom dashboards and reports about ongoing incidents.

NIS2 Requirement 3: Have backup management, disaster recovery, and crisis management plans in place to provide business continuity.

Zabbix supports Veeam (OOB) and Bacula data platforms, as well as many others. It can also monitor the backup execution process while tracking the storage and usage of backup servers.

NIS2 Requirement 4: Maintain supply chain security, including security-related aspects concerning the relationships between each entity and its direct suppliers or service providers.

Zabbix users can easily monitor third-party services and dependencies (such as APIs or libraries) for availability and performance, while being alerted to any potential vulnerabilities or disruptions in supply chain services. What’s more, Zabbix can also handle service monitoring and SLA reporting, keeping users updated around the clock on progress against predefined SLAs.

NIS2 Requirement 5: Provide security in network and information systems acquisition, development, and maintenance – including vulnerability handling and disclosure.

With Zabbix, a user can easily track software versions and check for outdated components, thanks to Zabbix’s ability to integrate with external tools for checking vulnerabilities.

NIS2 Requirement 6: Have policies and procedures in place regarding the use of cryptography and encryption.

Zabbix makes it simple for organizations of any size to comprehensively monitor encryption certificates for expiration.

NIS2 Requirement 7: Maintain HR security by providing accessible control and asset management policies.

Zabbix allows organizations to quickly and easily monitor user actions via log files.

NIS2 Requirement 8: Implement multi-factor authentication (MFA) or continuous authentication solutions, secured voice, video and text communications, and secured emergency communication systems.

Zabbix is set up to monitor the performance and uptime of any identity provider (IdP), using APIs provided by the IdPs themselves to query MFA policies and user login events. Zabbix can also monitor logs for MFA-related events while providing custom dashboards and reports on MFA usage.

In conclusion:

NIS2 is reshaping the cybersecurity landscape, and Zabbix has what it takes to equip organizations with the knowledge they need to thrive in this new regulatory environment. Trusting your monitoring to Zabbix can enhance your overall cybersecurity posture and supporting a comprehensive NIS2 implementation strategy.

To learn more, visit our website.

The post NIS2 Requirement Support: The Zabbix Advantage appeared first on Zabbix Blog.

Experience AI: The story so far

Post Syndicated from Fiona Coventry original https://www.raspberrypi.org/blog/experience-ai-the-story-so-far/

In April 2023, we launched our first Experience AI resources, developed in partnership with Google DeepMind to support educators to engage their students in learning about the topic of AI. Since then, the Experience AI programme has grown rapidly, reaching thousands of educators all over the world. Read on to find out more about the impact of our resources, and what we are learning.

The Experience AI resources

The Experience AI resources are designed to help educators introduce AI and AI safety to 11- to 14-year-olds. They consist of:

  • Foundations of AI: a comprehensive unit of six lessons including lesson plans, slide decks, activities, videos, and more to support educators to introduce AI and machine learning to young people
  • Two standalone lessons:
    • Large language models (LLMs): a lesson designed to help young people discover how large language models work, their benefits, and why their outputs are not always reliable
    • Ecosystems and AI — Biology: a lesson providing an opportunity for young people to explore how AI applications are supporting animal conservation
  • AI safety: a set of resources with a flexible design to support educators in a range of settings to equip young people with the knowledge and skills to responsibly and safely navigate the challenges associated with AI

We also offer a free online course, Understanding AI for educators, to help educators prepare to teach about AI.

International expansion

The launch of Experience AI came at an important time: AI technologies are playing an ever-growing role in our everyday lives, so it is crucial for young people to gain the understanding and skills they need to critically engage with these technologies. While the resources were initially designed for use by educators in the UK, they immediately attracted interest from educators across the world, as well as individuals wanting to learn about AI. The resources have now been downloaded over 325,000 times by people from over 160 countries. This includes downloads from over 7000 educators worldwide, who will collectively reach an estimated 1.2 million young people.

Photo of an educator teaching an Experience AI lesson.

Thanks to funding from Google DeepMind and Google.org, we have also been working with partners from across the globe to localise and translate the resources for learners and educators in their countries, and provide training to support local educators to deliver the lessons. The educational resources are now available in up to 15 languages, and to date, we have trained over 100 representatives from 20 international partner organisations, who will go on to train local educators. Five of these organisations have begun onward training already, collectively training over 1500 local educators so far.

The impact of Experience AI

The Experience AI resources have been well received by students and educators. Based on responses to our follow-up surveys, in countries where we have partners: 

  • 95% of educators agreed that the Experience AI sessions have increased their students’ knowledge of AI concepts 
  • 90% of young people (including young people in formal and non-formal education settings and learning independently) indicated that they better understand what AI and machine learning are
Photo of a young person learning about AI on a laptop.

This is backed up by qualitative feedback from surveys and interviews.

“Students’ perception and understanding of AI has improved and corrected. They realised they can contribute and be a part of the [development], instead of only users.” – Noorlaila, educator, SMK Cyberjaya, Malaysia

“[Students] found it interesting in the sense that it’s relevant information and they didn’t know what information was used for training models.” – Teacher, Liceul Tehnologic “Crisan” Criscior, Romania

“Based on my knowledge and learning about AI, I now appreciate the definition of AI as well as its implementation.” – Student, Changamwe JSS, Kenya

Photo of a group of educators participating in an Experience AI teacher training event in Kenya.

The training and resources also support educators to feel more confident to teach about AI:

  • 93% of international partner representatives who participated in our training agreed that the training increased their knowledge of AI concepts
  • 88% of educators receiving onward training by our international partners agreed that the training increased their confidence to teach AI concepts
  • 87% of educator respondents from our ‘Understanding AI for educators’ online course agreed that the course was useful for supporting young people

“It was a wonderful experience for me to join this workshop. Truly I was able to learn a lot about AI and I feel more confident now to teach the kids back at school about this new knowledge.” – Nur, educator, SMK Bandar Tasek Mutiara, trained by our partner Penang Science Cluster, Malaysia

“This was one of the best information sessions I’ve been to! So, so helpful!” – Meagan, educator, University of Alberta, trained by our partner Digital Moment, Canada

“The layout of the course in terms of content structuring is amazing. I love the discussion forum and the insightful yet empathetic responses by the course moderators on the discussion board. Honestly, I am really glad I started my AI in education journey with you.” – Priyanka, head teacher (primary level), United Arab Emirates, online course participant

What are we learning?

We are committed to continually improving our resources based on feedback from users. A recent review of feedback from educators highlighted key aspects of the resources that educators value most, as well as some challenges educators are facing and possible areas for improvement. For example, educators particularly like the interactive aspects, the clear structure and explanations, and the videos featuring professionals from the AI industry. We are continuing to look for ways we can better support educators to adapt the content and language to better support students in their context, fit Experience AI into their school timetables, and overcome technical barriers. 

We value feedback on our resources and will continue to highlight the importance of AI education in schools and work with partners across the globe to adapt our resources for different contexts.

Get involved

If you would like to try out our Experience AI resources, head to experience-ai.org, where you can find our free resources and online course, as well as information about local partners in your area.

The post Experience AI: The story so far appeared first on Raspberry Pi Foundation.

TechDocs at Grab: Cultivating a culture of quality documentation

Post Syndicated from Grab Tech original https://engineering.grab.com/techdocs-at-grab-cultivating-a-culture-of-quality-documentation

Introduction

Changing how a company approaches writing and documentation is a complex task. It’s not just about the tools and processes—it’s about shifting the mindset of the people who create and use documentation. Building a strong documentation culture means ensuring everyone takes ownership of producing high-quality content, while making the tools easy to use for everyone involved.

At Grab, our first significant step was adopting the Docs-as-Code approach, which we’ve covered in the blog Embracing a Docs-as-Code approach. This method integrated documentation into the engineering workflow, allowing teams to create and update content effortlessly.

Since then, the TechDocs working group — a collaboration between Tech Learning and the internal development team — has focused not just on improving tools, but on fostering a mindset where documentation is an essential part of everyday work. In this post, let us dive into how we’ve continued to embed high-quality documentation into the core of Grab’s engineering culture.

What is TechDocs?

Helix is Grab’s engineering platform designed to unify infrastructure, tooling, services, and documentation into a single, consistent user interface. It serves as a central hub for managing various engineering tasks and resources within Grab. Helix provides a comprehensive set of guides and tools for users.

TechDocs is an internal documentation platform built on Helix and integrates with our Docs-as-Code approach. It allows engineering teams to create, manage, and access technical content seamlessly within their workflows. TechDocs makes it easier for teams to maintain up-to-date, high-quality documentation with customised features for notification and editing.

How to create a healthy documentation culture

Over a span of 2 to 3 years, the TechDocs team executed these key steps in quarterly chunks to influence Grab’s documentation culture as seen in figure 1.

Figure 1: Key steps in influencing documentation culture in Grab
  1. Take inventory: Assess current internal processes, tools, and user behaviour.
  2. Finalise policy: Establish a clear policy, enforce it, and iterate based on feedback.
  3. Empower teams: Equip creators and maintainers with tools to manage their documentation.
  4. Track metrics, celebrate wins: Recognise and reward teams that follow best practices. Repeat regularly.

Now let’s look at each of these steps in detail.

Take inventory: Assess existing internal processes and tools/portals and understand user behaviour

Understanding the current culture

To shift the documentation culture at a company, you need to first understand what that culture is. At Grab, with its diverse business units, tech teams, and varied documentation practices, just grasping this was a big step. We needed to look at it from two angles: how teams and business units approach documentation, and what portals hold what kinds of resources.

Here are a few observations that apply not just to Grab but to most tech companies:

  • People default to the easiest way to get information, either by asking someone or searching familiar places. If they can’t find a document quickly, they assume it doesn’t exist.
  • Different teams use different documentation tools, leading to scattered, hard-to-maintain content. Without a unified search, finding the right document is a challenge.
  • Documentation is often created during development but rarely maintained, resulting in outdated or duplicate content over time.
  • Lack of clear ownership and governance causes inconsistencies, making it harder to trust or rely on documentation.

Conducting extensive user research

The insights on understanding the culture of documentation were obtained from conducting extensive feedback-gathering activities. We adopted two separate strategies for user research:

  1. The first focused on gathering feedback from as many people as possible. We scaled this approach to reach a wide audience across multiple teams and departments. To manage this volume, we used closed-ended questions with multiple-choice options, allowing us to collect broad, organisation-wide insights on user needs and preferences.
  2. The second approach was more in-depth and personal. We conducted 1:1 sessions where we observed how users interacted with tools, asked open-ended questions, and dug into the reasons behind their behaviors. This helped us understand not just what users did, but why and how they did it.

From the first approach, we were able to gather that users frequently browse for Runbooks, how-to’s, and FAQs when it comes to technical documentation. They emphasise structure, ease of navigation, and up-to-date content when it comes to quality.

Based on the feedback, only 2% of engineers (1 out of 56) reported that 80-100% of on-call engineering questions were resolved using technical documentation. In contrast, 29% of engineers indicated that 40-60% of their questions were addressed through documentation, while 25% stated that 20-40% were resolved in this manner.

To improve the documentation and Docs-as-Code workflows for seamless integration of documentation into the engineering process, we built the TechDocs Editor on the Helix platform. This rich text editor allowed teams to write and maintain their documents more effectively. However, while many engineers appreciated the new features, they highlighted areas for improvement for a smoother experience. Key suggestions included enhancing the creation of merge requests (MRs), resolving conflicts more efficiently, and offering an auto-approval process. They also wanted a way to preview content before MR approval, capabilities like bulk migration, and integration options such as plugins for Jira and *Confluence wiki. Additionally, there was a call to increase clarity on what content should belong in TechDocs versus the Wiki.

Rooting TechDocs tool’s improvements in the user’s feedback

Based on the feedback received from the extensive user research, the TechDocs tool’s new features were planned and lined up based on a priority mapping that was entirely rooted in the feedback from user research and interviews. While not all feedback was directly implementable in terms of tool improvements, a significant amount was. For issues that couldn’t be resolved through tools, cultural changes and learning best practices became key to addressing the challenges.

Here are insights from the 1-1 user research that helped us enhance the TechDocs tools and processes:

  • Search experience is average. The search experience on the TechDocs portal has room for improvement, with a CSAT score of 58.57%. Some users prefer using a more centralised search option, as it searches across multiple platforms and offers more relevant results, especially considering gaps in documentation on the internal TechDocs portal.
  • Documentation landing page needs improvement. The Documentation landing page scored 10.71% CSAT, highlighting its need for better design and categorisation. Users found the page cluttered, and the categorisation was seen as random and confusing.

  • Reading experience is positive. Overall, users are satisfied with reading documentation on Helix, with an 88.31% CSAT for reading experience. Users appreciated the navigation’s organisation and structure. Suggestions for further improvement include:

    • Better table content display
    • Maximising content space
    • Enhancing color contrast
  • TechDocs adoption still faces challenges. Although TechDocs adoption has grown, several challenges remain:
    • Migration efforts: The migration process requires significant effort, and without support or a clear push, some employees do not see the need to migrate.
    • Cultural factors: Users continue using familiar platforms and are looking for incentives, such as unique Helix features, to consider making the move.
    • Accessibility: VPN access is required for some features.
    • Awareness: Many users are unaware of Helix TechDocs’ full range of features, such as the different search options, available search filters, and commenting capabilities.
  • Cross-team collaboration challenges: Users reported difficulties in collaborating with non-engineering roles. While engineers are comfortable with the Docs-as-Code approach, which allows for more flexibility and simplicity, some find the TechDocs editor useful for initial document creation or small edits.

Using this feedback, the product roadmap was set for the year to focus on addressing the top user complaints and improving the TechDocs tools accordingly.

Finalise a suitable policy and begin enforcing it. Collect feedback and reiterate

To improve discoverability and maintain consistency, we established a structured policy for organising documents. This policy ensures that documentation is stored in the right place based on its purpose and usage, making it easier for Grabbers to find what they need. The key guidelines are as follows:

  • Markdown for ‘create and publish’ type content: Documentation related to platforms, products, or services that don’t require frequent updates should be in markdown format and stored in GitLab. These documents were rendered in Helix.
  • Collaborative portals for collaborative docs: Time-sensitive and collaborative documents—such as postmortems, RFCs, design docs, and project plans— are not compatible with docs-as-code and hence should reside in portals that offer collaboration features, like easy commenting and multi-user editing. Dedicated spaces within Confluence Wiki are ideal for this purpose.
  • Separation of internal data: Internal documents meant only for specific teams should not mix with general engineering resources for end users. These can be stored in portals with less stringent review processes, as they don’t require the same level of quality or accuracy checks. Team-specific spaces on Confluence Wiki can serve this need effectively.

Empower creators and maintainers to self-serve documentation upkeep

More documentation doesn’t mean good documentation

Getting people to create documentation is one thing, but getting them to maintain and update it is a whole different challenge. One major issue is the lack of accountability. Without a clear owning team or point of contact (PIC) for a document, everyone assumes someone else will handle updates. This leads to stale, outdated information because no one takes responsibility. To address this, the TechDocs team introduced features like showing the “last updated” date on each page and flagging documents that hadn’t been updated in over three months. This approach helped in two ways:

  • Readers could quickly gauge how up-to-date the information was.
  • Content owners were reminded when their documents needed attention.

Another key strategy was requiring every document to have a dedicated PIC at the time of creation. This ensured:

  • Clear accountability for maintaining the document.
  • The PIC would receive notifications about outdated documents and any comments from readers, making it easier to address issues.

What about docs that are not really meant to be updated that frequently?

When building any feature, it’s important to consider different use cases. While flagging outdated documents helped maintainers keep track of their content, it could also frustrate those responsible for more static documents that don’t require frequent updates.

To make the “last updated” feature more relevant, we introduced an option for users to mark documents as “verified.” This allowed maintainers to turn off the “your doc is outdated” flag if they felt the information was still accurate. While this feature could be misused in an extremely large organisation, it worked well at Grab where internal products and employees generally rely on mutual trust and respect for maintaining simple systems and policies.

Training and info-typing workshops

The TechDocs team had a unique advantage in influencing the quality of internal product and platform documentation. Many of the creators and maintainers of these documents belonged to the same organisation, which allowed for smoother collaboration.

To elevate the quality of TechDocs, we recognised that improving the drafts produced by platform engineers was essential. This realisation led us to create self-paced training materials focused on information typing guidelines and writing best practices specifically designed for these engineers, which included:

  • Info-typing guidelines: Helping engineers categorise information for better clarity.
  • Writing best practices: Teaching techniques to enhance readability and engagement.

Building on the positive feedback from the training course, we launched interactive workshops. In these sessions, participants brought their own team’s user-facing documentation, and with the guidance of expert Tech Content Developers (TCDs), they made significant, live updates to their documents using the info-typing principles they had learned. This process enabled participants to:

  • Revise their documents: Make real-time improvements during the workshop.
  • Receive expert feedback: Gain insights from TCDs on enhancing document quality.

The workshops received outstanding feedback and were further refined to cater to the specific needs of each team, ensuring that the training remained relevant and effective for the different documentation sets they managed. By focusing on collaboration and practical learning, we were able to foster a culture of continuous improvement in our documentation practices.

Track metrics, celebrate wins. Recognise and repeat.

Recognising teams and individuals who follow best practices is key to sustaining momentum. We celebrated these wins by publicly acknowledging contributions in newsletters and internal communications, along with offering swag and rewards. Additionally, we tracked the accuracy of responses from oncall-bots, which use documentation to auto-respond to user queries on our internal communicator. By analysing whether these automated responses were accurate, we could assess the quality of the docs being referenced. Teams that kept their documentation up-to-date and adhered to our internal TechDocs policy were rewarded, further reinforcing these good practices.

Celebrating wins wasn’t a one-off—it became a regular practice, helping to solidify desired behaviors and create a cycle of continuous improvement.

What’s next

Looking ahead, we have some exciting goals to push the documentation culture even further:

  • Boost documentation quality: We’re aiming to improve the quality of platform docs by a significant percentage, which will help reduce support tickets and inquiries to the automated tech support bot.
  • Expand training: We’re ramping up training for more engineers, helping them sharpen their tech writing skills and aiming for top CSAT ratings.
  • Launch improved TechDocs portal: Our goal is to build better and more intuitive navigation and categorisation of content for an improved user experience.
  • User interviews and engagement: We’ll be working closely with champions and users across tech families to create an open feedback loop.
  • Enhance doc creation and editing workflows: We’ll streamline the process of creating and editing content using templates and native tools with consistent Markdown flavour usage.

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility and digital financial services sectors. Serving over 800 cities in eight Southeast Asian countries, Grab enables millions of people everyday to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line – we aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

По буквите: Спигелман

Post Syndicated from Зорница Христова original https://www.toest.bg/po-bukvite-spigelman/

„Маус. Разказът на един оцелял“ от Арт Спигелман (част 1, „Баща ми кърви история“, и част 2, „Тук започнаха мъките ми“)

По буквите: Спигелман

превод от английски Владимир Полеганов, София: изд. НИКЕ, 2025

Добрата рецензия трябва да казва защо книгата засяга читателя, с какво е свързана с него.

Защо да я избере измежду хиляди нейни посестрими – как обстоятелствата в нея приличат на сегашните, какви поуки, настроения, смисли могат да се отнесат към днешния ден.

По какво героите приличат на нас.

Това е последното, което искам да направя, докато чета „Маус“ на Арт Спигелман.

Приликите между нацизма тогава и сега, между войните тогава и сега са толкова натрапчиви, че не е щадящо (себе си, читателя) да ги казваш на глас.

Внезапно започваш да разбираш цитираната от Тони Джуд (в „След войната“) ситуация, в която победените германци били задължени да гледат филм за концлагерите. И стискали очи в киното. Или разказа на Примо Леви как никой в следвоенна Италия не искал да чуе историята му. В нежеланието да знаеш има страх. И вина. Пропорциите варират. Искаш да знаеш, но само донякъде.

В „Маус“ на Арт Спигелман няма „донякъде“. Изобщо не става въпрос само за съчетанието между комикса като „леко“ изразно средство и тежката материя на историята. Идеята за „комикс за Холокоста“ би могла да се обърка на толкова много места, да стане пошла или едностранчива. Нали комиксите се асоциират с екшън между „добрите“ и „лошите“?

Е, „Маус“ не е това. Историята те дръпва навътре към сърцевината на книгата, защото мишките в нея са много повече истински, триизмерни хора от доста двукраки литературни герои. Главният персонаж, бащата, при цялото си страдание не е някакъв светец – нито преди войната, нито след нея. Докато действието снове от настоящето към миналото и обратно, ние виждаме недостатъците му толкова ясно, колкото у собствените си роднини – тук зарязал любима, там тормозил жена си, другаде пък изтърсил расистко подмятане. Дори влиза в клишето за стиснатия евреин – колкото и да се ядосва синът му.

В нашата култура сме свикнали да приемаме мъчениците за светци и обратно; ако някой е жертва, той трябва да бъде безгрешен. И това ни обърква, чувстваме се измамени, ако не е така;

в реалния свят обаче не само съвършените страдат, не само те заслужават да бъдат чути.

По буквите: Спигелман

Бащата в „Маус“ е толкова явно описван от натура, че няма как да не му вярваш. Затова и не можеш да си кажеш, че случващото се с него е наужким. А комиксовият формат те кара да обръщаш страниците бързо; това не е текст, в който катастрофата пъпли по релсите на изреченията. Отгръщаш – и те връхлита. На момента дори не разбираш какво си видял, защото разгръщаш нататък. Но седмици по-късно още не ти излиза от ума. При това нищо не е графично, в гадни подробности – стилът на Спигелман е обран, няма близки планове, няма подробности по телата. Те и без това са условни – човешки фигури с глави на мишки за евреите, на котки за германците и на прасета за поляците. Спигелман не спекулира с емоциите на читателя, не натиска лесните копчета. Просто иска да разкаже историята. Докрай.
Не „до щастливия край“ както в популярните филми. Защото това също е лъжа, страданието не спира – след извеждането си лагеристите продължават да мрат, а някои от спасените са убити… от завзелите междувременно имотите им „мирни“ техни съграждани.

Неудобна история – но Спигелман не се интересува кое е удобно. Интересува се от истината.

И още нещо във фактологичния му разказ. Също както не навлиза във физиологични подробности, Спигелман уж не говори за чувства, за психологичната страна на цялото преживяване. Тя присъства, както и всичко останало, като нещо веществено, като факт. Майката, която се самоубива. Самият Арт (от книгата), който ходи на терапия. И един от любимите ми детайли: по време на терапията Арт е с маска на мишка. Ясен символ от предишните епизоди – мишок с маска на прасе значи евреин, който се преструва на поляк.

Но мишата маска на самия Арт значи друго –

историята, включително семейната история, ни дава роля, но ние невинаги съвпадаме с нея.

По буквите: Спигелман

Пропуснах да кажа разни важни неща, например че в момента Арт Спигелман подготвя книга за Газа. И че е критик на израелската политика. Пропуснах да кажа, че е получил „Пулицър“ за „Маус“, че се смята за създател на графичния роман. Че огромният ефект от книгата му е повишил авторитета на комикса като изразно средство въобще. А също и че в днешното тегаво време книгата му е забранена в Русия (през 2015 г., уж защото свастиката на корицата била нацистка пропаганда) и горена в Полша (през 2001 г.). Неотдавна беше забранена и от училищното настоятелство в едно американско училище.

Моите почитания към издателство „Нике“, че я пуска точно сега. Преводът е отличен, печатът също; една-две грешки при нанасянето на буквите в балончетата леко озадачават („Войната сверши“), но това е дребна работа. А може би е и напомняне, че нищо още не е свършило.

Активните дарители на „Тоест“ получават постоянна отстъпка в размер на 20% от коричната цена на всички заглавия от каталозите на НИКЕ и няколко други български издателства в рамките на партньорската програма Читателски клуб „Тоест“. За повече информация прочетете на toest.bg/club.

В емблематичната си колонка, започната още през 2008 г. във в-к „Култура“, Марин Бодаков ни представяше нови литературни заглавия и питаше с какво точно тези книги ни променят. Вярваме, че е важно тази рубрика да продължи. От човек до човек, с нова книга в ръка.

The collective thoughts of the interwebz