Guide to IP and domain warming and migrating to Amazon SES

Post Syndicated from Tyler Holmes original https://aws.amazon.com/blogs/messaging-and-targeting/guide-to-ip-and-domain-warming-and-migrating-to-amazon-ses/

Transitioning your email workloads from another email service provider (ESP) to Amazon Simple Email Service (Amazon SES) can be a challenge, given that each workload can be unique. In this post, we show you how to successfully warm up IP addresses and domains when migrating to Amazon SES. This guide aims to provide a comprehensive overview of IP and domain warming best practices so you can make your transition to Amazon SES as smooth as possible. We discuss some of the challenges you might encounter and how to overcome those common pitfalls when transitioning to a new email service provider (ESP).

Understanding IP and domain email warming

IP warming and domain warming are strategic processes designed to gradually introduce a new sending identity to mailbox providers. A new sending identity can be a dedicated IP address, new domain, a subdomain of that domain, or any combination of them. The core objective of warming is to build a positive reputation with mailbox providers so your emails are delivered to the inbox rather than being filtered into spam folders or potentially blocked from being delivered to a mailbox altogether.

Mailbox providers such as Gmail, Yahoo, and Outlook are vigilant about protecting their users from spam and malicious content. When you introduce a new sending identity, mailbox providers evaluate the new sending identity with caution. They evaluate the early sending from the domain and IPs to ensure they’re sending the mailbox provider’s users messages that are wanted and aren’t engaged in abusive practices such as spam or phishing. Warming provides mailbox providers the opportunity to observe your sending patterns, content, and engagement metrics, allowing them to gradually build trust in your new sending identity.

Warming can be different for each scenario. For example, you can have completely warmed IPs, but if your sending domain is new, you’ll likely have to warm it up as well but you won’t need to worry about IP warming as much. Another common scenario is that of adding a new IP but sending with an established domain. In this case, the IP will need warming, but the domain itself is helping the warming because it already has an established reputation. When you have a net new IP and a net new domain, you’ll have to warm them together. The warm-up best practices we outline in this post, such as starting out slow and targeting your highest engaged subscribers first, apply to your situation.

Why warming is critical

Warming is essential for several reasons, each contributing to the overall success and reliability of your email marketing campaigns:

  1. Building trust with mailbox providers – A positive sender reputation is crucial for email deliverability. Mailbox providers use complex algorithms to evaluate the reputation of senders, and warming helps them build trust in your new sending identity.
  2. Avoiding initial deliverability issues – When you switch to a new ESP or introduce a new sending identity, it’s common to experience an initial dip in deliverability metrics, such as lower open rates, click-through rates, or higher bounce rates. Warming can mitigate these issues by giving mailbox providers time to adapt to your new sending patterns, whether that is a new domain, subdomain, or new IP infrastructure.
  3. Maintaining consistent sending behavior – Warming encourages you to maintain a steady and predictable sending cadence. Sudden, significant changes in sending volume, content, or frequency can trigger inbox spam filters because such changes might indicate that the sender has been compromised or is engaging in abusive practices. Even anomalies such as large changes in volume or throughput during seasonal events such as Black Friday can be interpreted as negative, and mailbox providers take a cautious approach when they detect anomalies such as sudden large spikes in volume.
  4. Long-term deliverability success – It’s a misconception that warming is done only one time. In reality you need to maintain traffic volumes and sending cadences to keep those sending identities warm. Additionally, if you plan on increasing volume considerably, for example from 1M to 5M or 5M to 25M, you need to warm up to those volumes. Those large jumps in volumes look suspicious to inbox providers even if you’ve been sending consistently.
  5. Adapting to mailbox provider changes – Mailbox providers also continuously update their algorithms to better detect spam and abusive behavior. If you view warming as a constant process and consistently monitor your deliverability and engagement signals you can make adjustments to your sending strategy as needed, ensuring that your emails continue to reach the inbox, even as inbox and audience behavior changes.

Common challenges to moving traffic and warming up on a new ESP

Transitioning your email traffic to a new ESP can present unique challenges that require careful consideration and strategic planning to overcome. These challenges include the following:

  1. Event-driven traffic – If all your email is event-driven, it’s hard to control volume and throughput.
  2. Multiple sending domains – Having many sending domains with varying traffic volumes and throughputs can complicate the transition.
  3. No shared IPs – Some organizations aren’t allowed to use shared pools of IPs.
  4. Lack of engagement data – Absence of data related to engagement can make it difficult to optimize the warming process.
  5. Outdated bounce and unsubscribe info – Not having up-to-date bounce and unsubscribe information in your current ESP can lead to deliverability issues.
  6. Single second-level domain – You’re currently sending your mail from your second-level domain, such as example.com, without separate subdomains for logical use cases such as transactional or marketing.
  7. Tight timelines – Contracts ending or other reasons might impose a tight timeline for the transition.
  8. Challenges for independent service providers (ISVs) and software-as-a-service (SaaS) providers – These organizations often don’t have complete control over the volume, content, lists, or sending consistency of their customers. They also might not have direct access to the DNS needed to update and align sending domains and authentication.

Strategies for a successful warm-up and migration

The following list isn’t suitable for every case, and many customers will use more than one strategy to address their challenges and smooth their transition to Amazon SES:

  1. Send to your best audience first – The most important thing you need to do when transitioning to a new ESP is to send to your highest and most active recipients on the new ESP and leave the less active or risky segments on the previous ESP until you’re ready to full switch. For example, if you’re a daily sender who sends to 1M addresses a day and have an open rate of about 20%, you need to start onboarding with segments that include those who are opening. A good strategy is to start with openers from the last 30 days, then move to openers from 31–90 days, and so on.
  2. Gradually shift traffic – Gradually move your less engaged subscribers to the new ESP while continuing to send in your current ESP and compare performance metrics between the two providers. After you’ve transitioned your most active segments, you can start to include the less engaged a little at a time. Make sure to continue monitoring for issues and immediately stop increasing your workloads if you encounter deliverability issues such as increased bounce or spam rates.
  3. Start with predictable workloads – Begin with workloads that aren’t time-dependent, such as newsletters, which are easier to control and monitor.
  4. Batch event-driven messages – For event-driven messages that aren’t time-sensitive, try to batch and spread them out to manage the volume.
  5. Use automated warm-up processes – Standard and managed dedicated IPs can help to manage the daily volume by allowing predefined levels of traffic on your dedicated IPs and spilling over into shared IP pools when the volume of email has reached a level we deem to be sufficient for your dedicated IPs. This is dependent on your warm-up progress thus far. Dedicated standard is a static 45-day increase, but managed dedicated has a more sophisticated process. To learn more, refer to Dedicated IP addresses for Amazon SES.
  6. Strategically use shared IP pools – Use shared IP pools for workloads that don’t require dedicated IPs. Because there is consistent volume already going through these IPs, they’re a little more forgiving than dedicated IPs being warmed up.
  7. Transition gradually to dedicated IPs – Begin with shared IPs and gradually transition to dedicated IPs as they warm up.
  8. Transition gradually to logical subdomains – Split your traffic into logical workloads that can have consistent volume and throughput. Even something as simple as marketing.example.com and transactional.example.com is better than sending mail from example.com
  9. Onboard new customers on the new ESP – For ISVs and SaaS providers, consider onboarding new customers directly on the new ESP to gather initial data and test the waters. New customers already need to be warmed up, so if you warm them up on Amazon SES rather than your legacy ESP, you don’t need to go through a warming process twice.

Prepare to migrate email traffic to Amazon SES

Before you migrate your email program to Amazon SES, it’s important to thoroughly document and organize your existing setup. This preparatory work will lay the foundation for a successful warm-up and migration process. For best practices, include the following tasks:

  1. Document your use cases – Categorize your use cases as either marketing or transactional. This will help you understand the nature of the emails you send and how they should be handled.
  2. Document your sending domains – Include the “from” names associated with each domain. This will assist in mapping the appropriate domain to the corresponding email type. Ideally, you should avoid sending from your root domain. For example, use a subdomain such as email.brand.com instead of brand.com. Review and document your authentication (for example, SPF, DKIM, or DMARC). In some cases, you might not need to align all of them, but you’ll definitely need to align DMARC as part of the bulk sender requirements.
  3. Map use cases to sending domains and from names – Create a clear correspondence to ensure the right emails are sent from the appropriate domains. At a minimum, it’s a best practice to have separate subdomains for transactional and promotional email use cases, such as transactional.brand.com and promo.brand.com.
  4. Document volume and max throughput – Capture this information for each use case mapped to your sending domains. This will help you understand the scale of your email operations and plan your architecture and warming strategy accordingly.
  5. Anticipate a temporary dip in deliverability metrics – While transitioning to a new ESP, you might experience a short-term fluctuation in metrics such as open rates and click-through rates. This is a common occurrence and shouldn’t be viewed as a failure of the service. It’s an expected part of the migration process as mailbox providers adapt to your new sending identity. By closely monitoring your bounce and complaint rates, you can make proactive adjustments to your ramp-up plan to ensure a smooth transition.
  6. Document your warming plan – Have a plan to gradually increase traffic for each identity and monitor engagement metrics. Plan for how to address high bounce or complaint rates.

The following table shows a sample warm-up plan. Notice that the days are categorized by large inbox providers. This is because these providers all accept new mail at different rates. Categorizing this way is a recommended best practice, but if you can’t segment that granularly, then you can use the Daily totals column as a guide. The AWS managed dedicated IP service automatically does this segmentation and throttling at the domain level for you.

The following plan is a typical ramp. You can get more aggressive the higher your overall engagement rates are, so if you’re at 40–60% engagement, you can use this warm-up. If your rates are lower, you might want to be a little more conservative. Make sure to be adaptive as you go into your warming plan because you might need to maintain the same rate for a couple days or even roll back a step if you’re experiencing negative trends such as a drop in deliverability or engagement. Remember, as you get into the less engaged segments of your list, engagement will drop, but it shouldn’t be drastic. Constantly monitor your metrics during this critical time.

Day @gmail.com @hotmail.com @outlook.com @yahoo.com @icloud.com @aol.com Others Daily total
1 150 150 150 150 150 150 150 1,050
2 300 300 300 300 300 300 300 2,100
3 600 600 600 600 600 600 600 4,200
4 1,200 1,200 1,200 1,200 1,200 1,200 1,200 8,400
5 2,400 2,400 2,400 2,400 2,400 2,400 2,400 16,800
6 5,000 5,000 5,000 5,000 5,000 5,000 5,000 35,000
7 10,000 10,000 10,000 10,000 10,000 10,000 10,000 70,000
8 20,000 20,000 20,000 20,000 20,000 20,000 20,000 140,000
9 40,000 40,000 40,000 40,000 40,000 40,000 40,000 280,000
10 80,000 80,000 80,000 80,000 80,000 80,000 80,000 560,000
11 150,000 150,000 150,000 150,000 150,000 150,000 150,000 1,050,000
12 300,000 300,000 300,000 300,000 300,000 300,000 300,000 2,100,000
13 425,000 425,000 425,000 425,000 425,000 425,000 425,000 2,975,000
14 500,000 500,000 500,000 500,000 500,000 500,000 500,000 3,500,000
15 600,000 600,000 600,000 600,000 600,000 600,000 600,000 4,200,000
16 650,000 650,000 650,000 650,000 650,000 650,000 650,000 4,550,000
17 700,000 700,000 700,000 700,000 700,000 700,000 700,000 4,900,000
18 800,000 800,000 800,000 800,000 800,000 800,000 800,000 5,600,000
19 900,000 900,000 900,000 900,000 900,000 900,000 900,000 6,300,000
20 1,000,000 1,000,000 1,000,000 1,000,000 1,000,000 1,000,000 1,000,000 7,000,000
21 1,100,000 1,100,000 1,100,000 1,100,000 1,100,000 1,100,000 1,100,000 7,700,000
22 1,200,000 1,200,000 1,200,000 1,200,000 1,200,000 1,200,000 1,200,000 8,400,000
23 1,300,000 1,300,000 1,300,000 1,300,000 1,300,000 1,300,000 1,300,000 9,100,000
24 1,400,000 1,400,000 1,400,000 1,400,000 1,400,000 1,400,000 1,400,000 9,800,000
25 1,500,000 1,500,000 1,500,000 1,500,000 1,500,000 1,500,000 1,500,000 10,500,000
26 1,600,000 1,600,000 1,600,000 1,600,000 1,600,000 1,600,000 1,600,000 11,200,000
27 1,700,000 1,700,000 1,700,000 1,700,000 1,700,000 1,700,000 1,700,000 11,900,000
28 1,800,000 1,800,000 1,800,000 1,800,000 1,800,000 1,800,000 1,800,000 12,600,000
29 1,900,000 1,900,000 1,900,000 1,900,000 1,900,000 1,900,000 1,900,000 13,300,000
30 2,000,000 2,000,000 2,000,000 2,000,000 2,000,000 2,000,000 2,000,000 14,000,000
31 2,100,000 2,100,000 2,100,000 2,100,000 2,100,000 2,100,000 2,100,000 14,700,000
32 2,200,000 2,200,000 2,200,000 2,200,000 2,200,000 2,200,000 2,200,000 15,400,000
33 2,300,000 2,300,000 2,300,000 2,300,000 2,300,000 2,300,000 2,300,000 16,100,000
34 2,400,000 2,400,000 2,400,000 2,400,000 2,400,000 2,400,000 2,400,000 16,800,000
35 2,500,000 2,500,000 2,500,000 2,500,000 2,500,000 2,500,000 2,500,000 17,500,000
36 2,600,000 2,600,000 2,600,000 2,600,000 2,600,000 2,600,000 2,600,000 18,200,000
37 2,700,000 2,700,000 2,700,000 2,700,000 2,700,000 2,700,000 2,700,000 18,900,000
38 2,800,000 2,800,000 2,800,000 2,800,000 2,800,000 2,800,000 2,800,000 19,600,000
39 2,900,000 2,900,000 2,900,000 2,900,000 2,900,000 2,900,000 2,900,000 20,300,000
40 3,000,000 3,000,000 3,000,000 3,000,000 3,000,000 3,000,000 3,000,000 21,000,000

Best practices for a successful IP warm-up

A successful IP warm-up involves a strategic approach that combines technical preparation, engaged subscribers, compelling content, and ongoing monitoring.

  1. Ensure technical readiness – Configure DNS records and set up SPF, DKIM, DMARC, and BIMI so your email content complies with best practices. Make sure your DMARC is aligned if you’re sending across multiple ESPs or domains.
  2. Use an engaged, permission-based mailing list – Use a clean, opt-in list of subscribers who are interested in your content. For more information, refer to Optimizing Email Deliverability: A User-Centric Approach to List Management and Monitoring.
  3. Provide compelling, valuable email content – Send content that resonates with your audience and encourages engagement.
  4. Gradually ramp up sending volume and cadence – Start with a small volume of emails and gradually increase over time to allow mailbox providers to observe your sending patterns.
  5. Maintain consistency in sending behavior – Avoid sudden, significant changes in sending volume, content, or frequency.
  6. Continuously monitor and optimize key metrics – Track open rates, click-through rates, bounce rates, and complaint rates, and make adjustments as needed. For more information, refer to Amazon SES – Set up notifications for bounces and complaints.
  7. Ongoing maintenance of sender reputation – Audit data flows, ramp up changes gradually, and follow evolving email marketing best practices.

Navigating initial deliverability challenges

When transitioning to a new ESP, you might encounter some initial deliverability challenges. It’s important to monitor and exercise caution if you observe increased bounce or complaint rates. If you do have challenges, you need to address them promptly. Maintain the same volume or even reduce the volume the next day if you encounter these issues:

  1. Spike in hard bounce rates – Bring over your suppression lists from the ESP you’re offboarding from, but if your previous ESP didn’t manage these well or you load some old addresses you weren’t aware of, it’s common to experience hard bounce spikes at the beginning. If this happens, slow your volume increases or even stop increasing until things stabilize. It’s more important to warm up properly than it is to get to production levels of sending as fast as possible. This is one more reason that it’s always best to start with your most engaged segments.
  2. Increased spam complaints – Emails might reach recipients who previously filtered them, leading to more spam complaints. Changing an identity can also cause your recipients to hit the spam button because they don’t recognize it. Announce identity changes before changing your ESP to reduce the chances of an issue.
  3. Heightened mailbox provider scrutiny – Mailbox providers will closely monitor new senders to confirm they’re not engaged in malicious activities. This can divert emails to the spam filter initially or even be throttled if you reach volume or throughput limits. Gmail is known to be stringent. Amazon SES managed dedicated IPs use our data to know how much mail the big inbox providers will accept while you’re warming up and keep you from overshooting their limits.
  4. ESP throttling and sending limits – The new ESP might have stricter rules regarding the volume of emails that can be sent to individual mailbox providers. Amazon SES has account limits for daily volume and max throughput, so adjust yours to what you’ll need. To learn more, refer to Increasing your Amazon SES sending quotas in the Amazon SES Developer Guide.

Maintaining IP reputation after warming

IP warming is an ongoing process. Even after the initial warm-up phase, it’s essential to maintain your sender reputation by continuously managing your email program. Your subscriber engagement might fluctuate as your list grows and changes. Similarly, ramping up email volume for a seasonal campaign will require adjustments to your warm-up process. You need to be proactive and adapt your IP warming strategy.

Audit data flows and campaigns and monitor email list sources, data collection practices, and campaign performance. When introducing new elements, do so incrementally to avoid triggering reputation issues. To allow time for your reputation to stabilize, provide at least a month for a new baseline to be established after major program changes. Engage with customers, provide value, and implement re-engagement campaigns to nurture your customer relationships. Adhere to evolving email marketing best practices, including proper authentication protocols and emerging technologies. Be proactive and track domain and IP reputation so you can quickly address the deliverability issues that arise. To learn more about monitoring inbox tools such as Google Postmaster, refer to Understanding Google Postmaster Tools (spam complaints) for Amazon SES email senders.

Conclusion

Transitioning your email program to a new ESP such as Amazon SES can seem complex, but it can be quick and seamless if you follow the best practices explained in this post. IP warming is a critical component of this process because it helps build a positive sender reputation with mailbox providers and promotes the reliable delivery of your emails.

Throughout this guide, we’ve covered the key aspects of IP warming and email migration, from understanding the importance of this practice to identifying common challenges and outlining effective strategies for a successful transition. By following best practices such as facilitating technical readiness, using an engaged subscriber base, providing compelling content, and gradually ramping up sending volume, you can navigate the initial deliverability challenges and establish a strong foundation for long-term email program success.

However, the work doesn’t stop when the initial warm-up phase is complete. Maintaining IP reputation and adapting your strategy as your email program and subscriber engagement evolve is an ongoing process. Continuously monitoring key metrics, auditing data flows, and staying up to date with evolving email marketing best practices is crucial for sustaining deliverability. A long-lasting sender reputation and enduring relationships with your list and recipients are some of the key benefits of following these best practices.Transitioning to a new ESP is a significant undertaking, but with the right preparation, execution, and commitment to ongoing maintenance, your migration can be smooth and successful.

Resources for deliverability

About the authors

Serverless ICYMI Q2 2025

Post Syndicated from Julian Wood original https://aws.amazon.com/blogs/compute/serverless-icymi-q2-2025/

Welcome to the 29th edition of the AWS Serverless ICYMI (in case you missed it) quarterly recap. At the end of a quarter, we share the most recent product launches, feature enhancements, blog posts, videos, live streams, and other interesting things that you might have missed!

In case you missed our last ICYMI, check out previous ICYMI posts.


Figure 1: Serverless calendar Q2 2025

Model Context Protocol (MCP)

MCP and agents are the hot topics in generative AI. MCP is an open protocol that standardizes how AI models can, in a secure and structured way, access external tools, data sources, and APIs. Amazon Q Developer IDE plugins and CLI both support MCP.

For a primer on MCP, watch What is MCP? No, Really!

AWS Serverless MCP Server

AWS announced the open-source AWS Serverless Model Context Protocol (MCP) Server, now available in the AWS Labs GitHub repository.


Figure 2: AWS Serverless MCP Server giving Lambda guidance

This is a tool that combines AI assistance with serverless expertise to enhance how developers build modern applications. The tool works with popular AI coding assistants like Amazon Q Developer and CLI, Cline, and Cursor. To add the MCP server, add the following code to your MCP client configuration.

{
  "mcpServers": {
    "awslabs.aws-serverless-mcp": {
      "command": "uvx",
      "args": [
        "awslabs.aws-serverless-mcp-server@latest"
      ],
      "env": { 
        "AWS_PROFILE": "your-aws-profile",
        "AWS_REGION": "us-east-1",
        "FASTMCP_LOG_LEVEL": "ERROR"
      }
    }
  }
}

The MCP server includes tools for project initialization, building and deployment, observability, and troubleshooting. The tool helps with operational excellence through observability features and provides contextual guidance for infrastructure as code decisions and AWS Lambda-specific best practices. This makes it easier for both new and experienced developers to build serverless applications.

There is also an Amazon ECS MCP Server to containerize and deploy applications to Amazon ECS within minutes. It helps configure all relevant AWS resources, including load balancers, networking, auto-scaling, monitoring, Amazon ECS task definitions, and services.

Amazon Aurora DSQL is now generally available


Figure 3: Amazon Aurora DSQL announcement

Amazon Aurora DSQL is now generally available. DSQL is a serverless, distributed SQL database with active-active high availability and multi-Region strong consistency. It is designed to make scaling and resilience effortless for your applications and offers the fastest distributed SQL reads and writes.

AWS Lambda

AWS Lambda Hackathon

The AWS Lambda Hackathon ran through June. With a total prize pool of $15,000, more than 3,700 participants competed to build Lambda applications that solve real-world business problems.


Figure 4: AWS Lambda Hackathon

Submissions needed to include a public code repository, a detailed README explaining the Lambda implementation, and a video demonstration. The projects are being judged by AWS experts based on the quality of the idea, architecture and design, and completeness of the solution.

Native support for Avro and Protobuf with Kafka

AWS Lambda now provides native support for Apache Avro and Protocol Buffers (Protobuf) formatted events with Apache Kafka event source mapping (ESM) when using Provisioned Mode.


Figure 5: Kafka processing flow diagram

The support allows you to validate your schema with popular schema registries. This allows you to use and filter the more efficient binary event formats and share data using schema in a centralized and consistent way. The ESM automatically validates incoming events against registered schemas and can deliver data in either the original binary format or as clean JSON, making it easier to build Kafka consumers across multiple programming languages.

Lambda logging changes can reduce Lambda CloudWatch logging costs. Lambda has introduced tiered pricing for CloudWatch logs and support for additional logging destinations, providing developers with more cost-effective logging options and greater flexibility in where they send their Lambda function logs.

Lambda has standardized billing for the INIT phase, providing more predictable and transparent pricing for function initialization costs.


Figure 6: AWS Lambda lifecycle

This improvement is part of an ongoing effort to make serverless pricing more transparent and predictable for customers building production applications.

There is a new guide exploring priming strategies with SnapStart to optimize cold start performance for Java Lambda functions. SnapStart delivers up to 10x faster function startup performance at no extra cost by creating and caching a snapshot of the initialized execution environment.


Figure 7: AWS Lambda SnapStart lifecycle

Learn how to monitor network traffic in Lambda functions to improve security and performance visibility. The post provides strategies for gaining visibility into network behavior. The guidance covers tools, techniques, and best practices for implementing network monitoring that can help identify security threats, performance bottlenecks, and compliance issues.

See how to optimize trace sampling behavior for Lambda functions using AWS X-Ray to improve observability while reducing costs and performance overhead.


Figure 8: AWS X-Ray tracing flow

This post provides guidance on implementing intelligent sampling strategies that provide the visibility needed for troubleshooting and optimization while minimizing overhead.

AWS CodePipeline now supports deploying to Lambda with traffic shifting for easier publishing of Lambda function revisions and traffic-shifting strategies for safer releases.

AWS named a Leader in Forrester Wave

AWS has been recognized as a Leader in The Forrester Wave: Serverless Development Platforms, Q2 2025, achieving the highest ranking in both the Current Offering and Strategy categories. This underscores AWS’s commitment to providing best-in-class innovation and developer experience in serverless application development. The evaluation analyzed key services, including AWS Lambda for serverless compute, AWS Step Functions and Amazon EventBridge for application integration, and AWS Fargate with Amazon ECS for serverless containers.

Amazon ECS

Amazon ECS now allows you to roll back your Amazon ECS service to a previous safe state if a deployment fails. Deployment circuit breaker automatically detects task launch failures while CloudWatch alarms allow you to detect issues that result in degradation in infrastructure.

Amazon ECS is also introducing a new account setting, defaultLogDriverMode. This allows you to define whether tasks in your account use blocking or non-blocking log driver mode by default, when you do not specify or omit it in your applications’ Task Definitions.

Amazon API Gateway

Amazon API Gateway now supports dynamic routing rules for custom domain name. This allows you to route API requests based on HTTP header values, either independently or in combination with URL paths. You can implement sophisticated routing strategies, such as API versioning, A/B testing, and gradual rollouts, without modifying existing API endpoints.


Figure 9: Amazon API Gateway dynamic routes

Learn how to use Amazon API Gateway to power hybrid workloads to more easily connect cloud and on-premises resources. There are practical examples of authentication, authorization, and data transformation patterns that are important for successful hybrid implementations.

AWS Serverless Application Model (AWS SAM) now supports custom domain names for private REST APIs feature of API Gateway.

Amazon EventBridge

Learn how to use Amazon EventBridge Pipes to enrich and customize notifications for more flexible and powerful event processing capabilities. The enhanced notification features allow you to create sophisticated event-driven workflows that can transform, filter, and route events based on complex business logic.

The EventBridge connector for Apache Kafka Connect is now generally available. This open-source connector streamlines event integration of Kafka environments with dozens of AWS services and partner integrations without writing custom integration code or running multiple connectors for each target.

EventBridge now supports AWS CodeBuild batch builds as a target. EventBridge Archive and Replay and API destinations connections now supports Customer Managed Keys.

Discover how to simplify private API integrations using EventBridge and AWS Step Functions. This guide covers the architectural patterns and implementation strategies required to build private integrations. The integrations maintain security while providing the flexibility and scalability that modern applications require.

Both Amazon SQS and Amazon SNS now support IPv6 for API requests, enabling you to communicate with SQS and SNS with IPv6, IPv4, or dual stack clients using public endpoints.

This Amazon S3 security guide covers best practices for securing S3 presigned URLs. Presigned URLs are a common pattern in serverless applications for providing temporary access to S3.

Discover how to enhance multi-account activity monitoring using event-driven architectures for better security and compliance across complex AWS environments.

Generative AI

You can more easily use Amazon Q to transform and upgrade Java and .NET applications.

Amazon Q Developer users with AWS Builder IDs can more easily upgrade to the Pro Tier, giving them higher usage limits in their IDEs and on the CLI.

Amazon Q Developer Eclipse IDE plugin is available and an agentic coding experience is now available within JetBrains and Visual Studio.

Amazon Bedrock now offers comprehensive CloudWatch metrics support for Agents.

Powertools for AWS Lambda introduces Bedrock Agents Function utility. This helps to create Lambda functions that can respond to Amazon Bedrock Agent action requests with built-in parameter injection, response formatting, eliminating boilerplate code and accelerating development.

Amazon Bedrock Intelligent Prompt Routing is now generally available. This routes prompts to different foundation models within a model family, helping you optimize for quality of responses and cost.

Prompt caching helps reduce costs by up to 90% and latency by up to 85% by caching frequently used prompts across multiple API calls. Prompt caching is now generally availability on Amazon Bedrock.

Serverless Compute Blog Posts

April

May

June

Serverless Office Hours weekly livestream

April

May

June

The Serverless landing page has more information. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.

You can also follow the Developer Advocacy team members who work on Serverless to see the latest news, follow conversations, and interact with the team.

And finally, visit the Serverless Land for all your serverless needs.

На второ четене: „Атлас на мъглявите континенти“

Post Syndicated from Стефан Иванов original https://www.toest.bg/na-vtoro-chetene-atlas-na-muglyavite-kontinenti/

„Атлас на мъглявите континенти“ от Ихсан Октай Анар

На второ четене: „Атлас на мъглявите континенти“

превод от турски Златина Сакалова, консултант Хубавинка Филипова, София: изд. „ЖАР – Жанет Аргирова“, 2020

Ако някой познава и харесва Памук, нека прочете и Анар. Има защо. Това е книга на границата между съня и света, роман от Константинопол до Истанбул, роман за миналото, настоящето и бъдещето. Ихсан Октай Анар отваря капсула на възможните разкази, поставя капан за рационалното, създава амулет против ясната и права логика.

В същността си произведението е фарс на тема познание. Съставено е от глави, части и фрагменти и в тях среща си правят сънища, разкази, фолклорни наслоявания, философски подигравки и метафизична проза. Книга, подходяща за безвремието на жегата, защото не се чете, а се сънува. И по тази причина се явява постмодерен роднина на „Хазарски речник“, „Сто години самота“, „Името на розата“, „Градът и кучетата“ или дори на „Покойният Матиа Паскал“. За първи път чета за такъв литературен Истанбул – сякаш излязъл от вълшебна лампа град, с мистика, калиграфия и хубава доза насмешка.

Книгата на Анар е антидекартов манифест. Неговият герой Узун Ихсан ефенди не казва „Мисля, следователно съществувам“, а „Сънувам, следователно съществувам“.

Трансформация, която демонтира фундаментите на западния рационализъм и пренаписва модерността в кода на Изтока. Тази подмяна на картезианската аксиома с особена етика и естетика на съня може да се чете като интелектуален бунт. Срещу колониалното мислене, срещу догмата на напредъка, срещу просвещенската арогантност. (Наистина, какво ли би било, ако Монтен беше заел централното място на Декарт?)

По същество Анар прави това, което Бахтин нарича „карнавализация на познанието“ – принизява високото, възвисява ниското. Черната монета в романа например, символ на власт и смисъл, се оказва артефакт, способен да свързва светове. Но „Атласът“ не е просто археология на предмодерното. Той е постмодерен колаж от митове, исторически и алтернативни пространства, суфистка философия, карикатурна визия за властта и присмехулно писателско всемогъщество.

Ако Орхан Памук изгражда лъскавата фасада на турския постмодернизъм, Анар е задният му двор, пълен със сенки, сгрешени карти, алхимици, пирати и тайнствени телескопи.

Езикът на романа сам по себе си е отломък от епическото наследство, гравиран върху ориенталски орнамент. Това съвсем не е реализъм. Това е магически реализъм, пречупен през ислямско-османската култура. Константинопол в книгата прилича повече на потапяне в мултивселените, отколкото на историческа хроника.

Светът в романа, свят на шпиони, мехлеми, заговорници и съновидения, на пръв поглед изглежда толкова далечен, че е почти сюрреалистичен. И все пак в параноята, затвореността, желанието за контрол чрез знание (като черната монета или тайните карти) лесно можем да разпознаем и съвременна Турция, но не по-малко – и съвременна България. И двете общества таят съмнение към рационалното знание. И двете се лутат между мита за Европа и митологията на своето собствено минало.

Както Турция след Ердогановата реислямизация, така и България с националпопулистки рецидиви се лутат между илюзиите за модерност и копнежа по „славното минало“. Узун Ихсан ефенди би бил чудесен образ и на един съвременен български учен от малък град. Особен човек, затворен в своята стая, създаващ карта на един свят, който отказва да го разпознае. Атлас на недостижимото, където Варна и Диарбекир са еднакво далечни.

На второ четене: „Атлас на мъглявите континенти“

Анар ползва познати и вече класически постмодерни хватки. В арсенала му ги има както знанието, едновременно фетиш и илюзия, така и структурирането на текста като поредица от „книги в книгата“. Заедно с това присъства и превръщането на историята в мит и на мита в история, редом със смешната деконструкция на философски авторитети с източна нагласа и с причудливите герои, ирационалните системи и авторовите вмешателства. Естествено, ирония и меланхолия (хюзюн) са също налични.

„Атласът“ е книга за онези, които търсят не сюжет, а разгръщане. Тя ще отблъсне читателя, пристрастен към психологическия реализъм и кристалната логика, но ще възнагради онзи, който приема текста като космос от асоциации, разказ, чийто сън разцъфва с четенето.

„Атласът“ е радикален нероман – по-скоро текстова гравюра с територия, която включва морета от възможности, кораби от символи и монети на несигурността.

В епохата на фалшивите новини, генерираните текстове и тревожните геополитики „Атласът“ е словесна провокация. Ако романът е сън, то кой ни сънува? Ние ли сънуваме Турция? Турция ли сънува нас? Или и двете ни сънува едно разпадащо се минало?

Турция и България като два полуострова на незавършената модерност, два съседни съня, разказвани с различни метафори, но от една и съща болка. Узун Ихсан ефенди стои на прозореца и гледа бъдещето – то идва с черен знак. А ние като Бюнямин се събуждаме с вечния въпрос: „Сънувам ли, или участвам?“

Романът не се интересува от емоционална идентификация. Книгата е за съня на разума, не за страстите на сърцето. Героите са по-скоро типове, отколкото личности. Затова дори смъртта, насилието и любовта (ако изобщо ги има) не нанасят удар. Това е съзнателен избор, но и ограничение, отдалечаване от възможността за съпреживяване.

Книги, които задълбочено разглеждат важността на паметта и колебанията на езика, пародират свободно властта, анализират позите и провалите на агресивната мъжественост и са сатирични към религиозната доминация, са от изключителна важност днес. Особено когато доставят удоволствие с размаха, майсторството и мащаба си. Това е особено интелектуално преживяване и гъдел, както и извънредно приятно упражнение във възпитание на въображението. Анар не казва как се мисли за света, а как се сънува той. Това изисква търпение, отвореност и любопитство, но който ги има, ще бъде възнаграден.

Орхан Памук създава романи за Турция, но на езика на Запада. Той взема формата на модерния западен роман в традицията на Достоевски, Пруст, Музил и Ман и я изпълва с турски съдържания, дилеми и културни пластове. Неговият роман е за връзката между Изтока и Запада. Анар създава романи за Турция, но сякаш въпреки Запада. Той пише в традиция, която не търси западното си потвърждение и разкодиране. Той е в османското наследство – мистично, фолклорно и маргинално. Неговите книги се четат като оригинални и стари ръкописи, а не като модерни романи.

Анар остава култова фигура в Турция, но не желае да бъде популярен. Не дава интервюта, не поддържа публичното си присъствие, почти не пътува. Той пише бавно, рядко и никога с мисъл за пазара. За мнозина е „писател за писатели“, обожаван е от академици и философи, но понякога е пренебрегван от непрофесионалния читател. Мисля, че това е така само защото той не е чул за него и не е имал възможността да го разгърне.

Анар и Памук са огледални образи. Памук гради външен мост между Турция и света. Анар копае вътрешен кладенец в тялото на турската култура. Единият говори, за да бъде чут, другият шепти. И както често се случва, шепотът остава по-малко познат, но понякога по-добре запомнен от онези, които са готови да слушат.

Активните дарители на „Тоест“ получават постоянна отстъпка в размер нa 20% от коричната цена на всички заглавия от каталога на издателство „ЖАР – Жанет Аргирова“, както и на няколко други български издателства в рамките на партньорската програма Читателски клуб „Тоест“. За повече информация прочетете на toest.bg/club.

Никой от нас не чете единствено най-новите книги. Тогава защо само за тях се пише? „На второ четене“ е рубрика, в която отваряме списъците с книги, публикувани преди поне година, четем ги и препоръчваме любимите си от тях. Рубриката е част от партньорската програма Читателски клуб „Тоест“. Изборът на заглавия обаче е единствено на авторите – Стефан Иванов и Антония Апостолова, които биха ви препоръчали тези книги и ако имаше как веднъж на две седмици да се разходите с тях в книжарницата.

[$] Kernel API specification and validation

Post Syndicated from corbet original https://lwn.net/Articles/1027811/

The kernel project makes a strong promise to its users: the kernel ABI will
not be changed in ways that break user-space code. The occasional failure
notwithstanding, kernel developers do try to live up to that promise. They
are handicapped by one little problem, though: there is no description of
what the kernel ABI is, and no comprehensive way to test whether a given
change breaks it. The kernel API
specification framework proposed (in its second revision) by Sasha
Levin addresses some of those concerns, but the solution is incomplete and
does not come for free.

Build conversational AI search with Amazon OpenSearch Service

Post Syndicated from Bharav Patel original https://aws.amazon.com/blogs/big-data/build-conversational-ai-search-with-amazon-opensearch-service/

Retrieval Augmented Generation (RAG) is a well-known approach to creating generative AI applications. RAG combines large language models (LLMs) with external world knowledge retrieval and is increasingly popular for adding accuracy and personalization to AI. It retrieves relevant information from external sources, augments the input with this data, and generates responses based on both. This approach reduces hallucinations, improves fact accuracy, and allows for up-to-date, efficient, and explainable AI systems. RAG’s ability to break through classical language model limitations has made it applicable to broad AI use cases.

Amazon OpenSearch Service is a versatile search and analytics tool. It is capable of performing security analytics, searching data, analyzing logs, and many other tasks. It can also work with vector data with a k-nearest neighbors (k-NN) plugin, which makes it helpful for more complex search strategies. Because of this feature, OpenSearch Service can serve as a knowledge base for generative AI applications that integrate language generation with search results.

By preserving context over several exchanges, honing responses, and providing a more seamless user experience, conversational search enhances RAG. It helps with complex information needs, resolves ambiguities, and manages multi-turn reasoning. Conversational search provides a more natural and personalized interaction, yielding more accurate and pertinent results, even though standard RAG performs well for single queries.

In this post, we explore conversational search, its architecture, and various ways to implement it.

Solution overview

Let’s walk through the solution to build conversational search. The following diagram illustrates the solution architecture.

The new OpenSearch feature known as agents and tools is used to create conversational search. To develop sophisticated AI applications, agents coordinate a variety of machine learning (ML) tasks. Every agent has a number of tools; each intended for a particular function. To use agents and tools, you need OpenSearch version 2.13 or later.

Prerequisites

To implement this solution, you need an AWS account. If you don’t have one, you can create an account. You also need an OpenSearch Service domain with OpenSearch version 2.13 or later. You can use an existing domain or create a new domain.

To use the Amazon Titan Text Embedding and Anthropic Claude V1 models in Amazon Bedrock, you need to enable access to these foundation models (FMs). For instructions, refer to Add or remove access to Amazon Bedrock foundation models.

Configure IAM permissions

Complete the following steps to set up an AWS Identity and Access Management (IAM) role and user with appropriate permissions:

  1. Create an IAM role with the following policy that will allow the OpenSearch Service domain to invoke the Amazon Bedrock API: 
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "bedrock:InvokeAgent",
                "bedrock:InvokeModel"
            ],
            "Resource": [
                "arn:aws:bedrock:${Region}::foundation-model/amazon.titan-embed-text-v1",
                "arn:aws:bedrock: ${Region}::foundation-model/anthropic.claude-instant-v1"
            ]
        }
    ]
}

Depending on the AWS Region and model you use, specify those in the Resource section.

  1. Add opensearchservice.amazonaws.com as a trusted entity.
  2. Make a note of the IAM role Amazon Resource name (ARN).
  3. Assign the preceding policy to the IAM user that will create a connector.
  4. Create a passRole policy and assign it to IAM user that will create the connector using Python: 
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::${AccountId}:role/OpenSearchBedrock"
        }
    ]
}
  5. Map the IAM role you created to the OpenSearch Service domain role using the following steps:
    • Log in to the OpenSearch Dashboard and open the Security page from the navigation menu.
    • Choose Roles and select ml_all_access.
    • Choose Mapped Users and Manage Mapping.
    • Under Users, add the ARN of the IAM user you created.

Establish a connection to the Amazon Bedrock model using the MLCommons plugin

In order to identify patterns and relationships, an embedding model transforms input data—such as words or images—into numerical vectors in a continuous space. Similar objects are grouped together to make it easier for AI systems to comprehend and respond to intricate user enquiries.

Semantic search concentrates on the purpose and meaning of a query. OpenSearch stores data in a vector index for retrieval and transforms it into dense vectors (lists of numbers) using text embedding models. We are using amazon.titan-embed-text-v1 hosted on Amazon Bedrock, but you will need to evaluate and choose the right model for your use case. The amazon.titan-embed-text-v1 model maps sentences and paragraphs to a 1,536-dimensional dense vector space and is optimized for the task of semantic search.

Complete the following steps to establish a connection to the Amazon Bedrock model using the MLCommons plugin:

  1. Establish a connection by using the Python client with the connection blueprint.
  2. Modify the values of the host and region parameters in the provided code block. For this example, we’re running the program in Visual Studio Code with Python version 3.9.6, but newer versions should also work.
  3. For the role ARN, use the ARN you created earlier, and run the following script using the credentials of the IAM user you created: 
    import boto3
import requests 
from requests_aws4auth import AWS4Auth

host = 'https://search-test.us-east-1.es.amazonaws.com/'
region = 'us-east-1'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

path = '_plugins/_ml/connectors/_create'
url = host + path

payload = {
  "name": "Amazon Bedrock Connector: embedding",
  "description": "The connector to bedrock Titan embedding model",
  "version": 1,
  "protocol": "aws_sigv4",
  "parameters": {
    "region": "us-east-1",
    "service_name": "bedrock",
    "model": "amazon.titan-embed-text-v1"
  },
  "credential": {
    "roleArn": "arn:aws:iam::<accountID>:role/opensearch_bedrock_external"
  },
  "actions": [
    {
      "action_type": "predict",
      "method": "POST",
      "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
      "headers": {
        "content-type": "application/json",
        "x-amz-content-sha256": "required"
      },
      "request_body": "{ \"inputText\": \"${parameters.inputText}\" }",
      "pre_process_function": "connector.pre_process.bedrock.embedding",
      "post_process_function": "connector.post_process.bedrock.embedding"
    }
  ]
}

headers = {"Content-Type": "application/json"}

r = requests.post(url, auth=awsauth, json=payload, headers=headers, timeout=15)
print(r.status_code)
print(r.text)
  4. Run the Python program. This will return connector_id. 
    python3 connect_bedrocktitanembedding.py
200
{"connector_id":"nbBe65EByVCe3QrFhrQ2"}
  5. Create a model group against which this model will be registered in the OpenSearch Service domain: 
    POST /_plugins/_ml/model_groups/_register
{
  "name": "embedding_model_group",
  "description": "A model group for bedrock embedding models"
}

    You get the following output:

    {
  "model_group_id": "1rBv65EByVCe3QrFXL6O",
  "status": "CREATED"
}
  6. Register a model using connector_id and model_group_id: 
    POST /_plugins/_ml/models/_register
{
    "name": "titan_text_embedding_bedrock",
    "function_name": "remote",
    "model_group_id": "1rBv65EByVCe3QrFXL6O",
    "description": "test model",
    "connector_id": "nbBe65EByVCe3QrFhrQ2",
   "interface": {}
}

You get the following output:

{
  "task_id": "2LB265EByVCe3QrFAb6R",
  "status": "CREATED",
  "model_id": "2bB265EByVCe3QrFAb60"
}
  1. Deploy a model using the model ID:
POST /_plugins/_ml/models/2bB265EByVCe3QrFAb60/_deploy

You get the following output:

{
  "task_id": "bLB665EByVCe3QrF-slA",
  "task_type": "DEPLOY_MODEL",
  "status": "COMPLETED"
}

Now the model is deployed, and you will see that in OpenSearch Dashboards on the OpenSearch Plugins page.

Create an ingestion pipeline for data indexing

Use the following code to create an ingestion pipeline for data indexing. The pipeline will establish a connection to the embedding model, retrieve the embedding, and then store it in the index.

PUT /_ingest/pipeline/cricket_data_pipeline {
    "description": "batting score summary embedding pipeline",
    "processors": [
        {
            "text_embedding": {
                "model_id": "GQOsUJEByVCe3QrFfUNq",
                "field_map": {
                    "cricket_score": "cricket_score_embedding"
                }
            }
        }
    ]
}

Create an index for storing data

Create an index for storing data (for this example, the cricket achievements of batsmen). This index stores raw text and embeddings of the summary text with 1,536 dimensions and uses the ingest pipeline we created in the previous step.

PUT cricket_data {
    "mappings": {
        "properties": {
            "cricket_score": {
                "type": "text"
            },
            "cricket_score_embedding": {
                "type": "knn_vector",
                "dimension": 1536,
                "space_type": "l2",
                "method": {
                    "name": "hnsw",
                    "engine": "faiss"
                }
            }
        }
    },
    "settings": {
        "index": {
            "knn": "true"
        }
    }
}

Ingest sample data

Use the following code to ingest the sample data for four batsmen:

POST _bulk?pipeline=cricket_data_pipeline
{"index": {"_index": "cricket_data"}}
{"cricket_score": "Sachin Tendulkar, often hailed as the 'God of Cricket,' amassed an extraordinary batting record throughout his 24-year international career. In Test cricket, he played 200 matches, scoring a staggering 15,921 runs at an average of 53.78, including 51 centuries and 68 half-centuries, with a highest score of 248 not out. His One Day International (ODI) career was equally impressive, spanning 463 matches where he scored 18,426 runs at an average of 44.83, notching up 49 centuries and 96 half-centuries, with a top score of 200 not out – the first double century in ODI history. Although he played just one T20 International, scoring 10 runs, his overall batting statistics across formats solidified his status as one of cricket's all-time greats, setting numerous records that stand to this day."}
{"index": {"_index": "cricket_data"}}
{"cricket_score": "Virat Kohli, widely regarded as one of the finest batsmen of his generation, has amassed impressive statistics across all formats of international cricket. As of April 2024, in Test cricket, he has scored over 8,000 runs with an average exceeding 50, including numerous centuries. His One Day International (ODI) record is particularly stellar, with more than 12,000 runs at an average well above 50, featuring over 40 centuries. In T20 Internationals, Kohli has maintained a high average and scored over 3,000 runs. Known for his exceptional ability to chase down targets in limited-overs cricket, Kohli has consistently ranked among the top batsmen in ICC rankings and has broken several batting records throughout his career, cementing his status as a modern cricket legend."}
{"index": {"_index": "cricket_data"}}
{"cricket_score": "Adam Gilchrist, the legendary Australian wicketkeeper-batsman, had an exceptional batting record across formats during his international career from 1996 to 2008. In Test cricket, Gilchrist scored 5,570 runs in 96 matches at an impressive average of 47.60, including 17 centuries and 26 half-centuries, with a highest score of 204 not out. His One Day International (ODI) record was equally remarkable, amassing 9,619 runs in 287 matches at an average of 35.89, with 16 centuries and 55 half-centuries, and a top score of 172. Gilchrist's aggressive batting style and ability to change the course of a game quickly made him one of the most feared batsmen of his era. Although his T20 International career was brief, his overall batting statistics, combined with his wicketkeeping skills, established him as one of cricket's greatest wicketkeeper-batsmen."}
{"index": {"_index": "cricket_data"}}
{"cricket_score": "Brian Lara, the legendary West Indian batsman, had an extraordinary batting record in international cricket during his career from 1990 to 2007. In Test cricket, Lara amassed 11,953 runs in 131 matches at an impressive average of 52.88, including 34 centuries and 48 half-centuries. He holds the record for the highest individual score in a Test innings with 400 not out, as well as the highest first-class score of 501 not out. In One Day Internationals (ODIs), Lara scored 10,405 runs in 299 matches at an average of 40.48, with 19 centuries and 63 half-centuries. His highest ODI score was 169. Known for his elegant batting style and ability to play long innings, Lara's exceptional performances, particularly in Test cricket, cemented his status as one of the greatest batsmen in the history of the game."}

Deploy the LLM for response generation

Use the following code to deploy the LLM for response generation. Modify the values of host, region, and roleArn in the provided code block.

  1. Create a connector by running the following Python program. Run the script using the credentials of the IAM user created earlier. 
    import boto3
import requests 
from requests_aws4auth import AWS4Auth

host = 'https://search-test.us-east-1.es.amazonaws.com/'
region = 'us-east-1'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

path = '_plugins/_ml/connectors/_create'
url = host + path

payload = {
  "name": "BedRock Claude instant-v1 Connector ",
  "description": "The connector to BedRock service for claude model",
  "version": 1,
  "protocol": "aws_sigv4",
  "parameters": {
    "region": "us-east-1",
    "service_name": "bedrock",
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens_to_sample": 8000,
    "temperature": 0.0001,
    "response_filter": "$.completion"
  },
   "credential": {
        "roleArn": "arn:aws:iam::accountId:role/opensearch_bedrock_external"
    },
  "actions": [
    {
      "action_type": "predict",
      "method": "POST",
      "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/anthropic.claude-instant-v1/invoke",
      "headers": {
        "content-type": "application/json",
        "x-amz-content-sha256": "required"
      },
      "request_body": "{\"prompt\":\"${parameters.prompt}\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature},  \"anthropic_version\":\"${parameters.anthropic_version}\" }"
    }
  ]
 }
    

headers = {"Content-Type": "application/json"}

r = requests.post(url, auth=awsauth, json=payload, headers=headers, timeout=15)
print(r.status_code)
print(r.text)

If it ran successfully, it would return connector_id and a 200-response code:

200
{"connector_id":"LhLSZ5MBLD0avmh1El6Q"}
  1. Create a model group for this model: 
    POST /_plugins/_ml/model_groups/_register
{
    "name": "claude_model_group",
    "description": "This is an example description"
}

This will return model_group_id; make a note of it:

{
  "model_group_id": "LxLTZ5MBLD0avmh1wV4L",
  "status": "CREATED"
}
  1. Register a model using connection_id and model_group_id: 
    POST /_plugins/_ml/models/_register
{
    "name": "anthropic.claude-v1",
    "function_name": "remote",
    "model_group_id": "LxLTZ5MBLD0avmh1wV4L",
    "description": "LLM model",
    "connector_id": "LhLSZ5MBLD0avmh1El6Q",
    "interface": {}
}

It will return model_id and task_id:

{
  "task_id": "YvbVZ5MBtVAPFbeA7ou7",
  "status": "CREATED",
  "model_id": "Y_bVZ5MBtVAPFbeA7ovb"
}
  1. Finally, deploy the model using an API:
POST /_plugins/_ml/models/Y_bVZ5MBtVAPFbeA7ovb/_deploy

The status will show as COMPLETED. That means the model is successfully deployed.

{
  "task_id": "efbvZ5MBtVAPFbeA7otB",
  "task_type": "DEPLOY_MODEL",
  "status": "COMPLETED"
}

Create an agent in OpenSearch Service

An agent orchestrates and runs ML models and tools. A tool performs a set of specific tasks. For this post, we use the following tools:

  • VectorDBTool – The agent use this tool to retrieve OpenSearch documents relevant to the user question
  • MLModelTool – This tool generates user responses based on prompts and OpenSearch documents

Use the embedding model_id in VectorDBTool and LLM model_id in MLModelTool:

POST /_plugins/_ml/agents/_register {
    "name": "cricket score data analysis agent",
    "type": "conversational_flow",
    "description": "This is a demo agent for cricket data analysis",
    "app_type": "rag",
    "memory": {
        "type": "conversation_index"
    },
    "tools": [
        {
            "type": "VectorDBTool",
            "name": "cricket_knowledge_base",
            "parameters": {
                "model_id": "2bB265EByVCe3QrFAb60",
                "index": "cricket_data",
                "embedding_field": "cricket_score_embedding",
                "source_field": [
                    "cricket_score"
                ],
                "input": "${parameters.question}"
            }
        },
        {
            "type": "MLModelTool",
            "name": "bedrock_claude_model",
            "description": "A general tool to answer any question",
            "parameters": {
                "model_id": "gbcfIpEByVCe3QrFClUp",
                "prompt": "\n\nHuman:You are a professional data analysist. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\nContext:\n${parameters.cricket_knowledge_base.output:-}\n\n${parameters.chat_history:-}\n\nHuman:${parameters.question}\n\nAssistant:"
            }
        }
    ]
}

This returns an agent ID; take note of the agent ID, which will be used in subsequent APIs.

Query the index

We have batting scores of four batsmen in the index. For the first query, let’s specify the player name:

POST /_plugins/_ml/agents/<agent ID>/_execute {
    "parameters": {
        "question": "What is batting score of Sachin Tendulkar ?"
    }
}

Based on context and available information, it returns the batting score of Sachin Tendulkar. Note the memory_id from the response; you will need it for subsequent questions in the next steps.

We can ask a follow-up question. This time, we don’t specify the player name and expect it to answer based on the earlier question:

POST /_plugins/_ml/agents/<agent ID>/_execute {
    "parameters": {
        "question": " How many T20 international match did he play?",
        "next_action": "then compare with Virat Kohlis score",
        "memory_id": "so-vAJMByVCe3QrFYO7j",
        "message_history_limit": 5,
        "prompt": "\n\nHuman:You are a professional data analysist. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\nContext:\n${parameters.population_knowledge_base.output:-}\n\n${parameters.chat_history:-}\n\nHuman:always learn useful information from chat history\nHuman:${parameters.question}, ${parameters.next_action}\n\nAssistant:"
    }
}

In the preceding API, we use the following parameters:

  • Question and Next_action – We also pass the next action to compare Sachin’s score with Virat’s score.
  • Memory_id – This is memory assigned to this conversation. Use the same memory_id for subsequent questions.
  • Prompt – This is the prompt you give to the LLM. It includes the user’s question and the next action. The LLM should answer only using the data indexed in OpenSearch and must not invent any information. This way, you prevent hallucination.

Refer to ML Model tool for more details about setting up these parameters and the GitHub repo for blueprints for remote inferences.

The tool stores the conversation history of the questions and answers in the OpenSearch index, which is used to refine answers by asking follow-up questions.

In real-world scenarios, you can map memory_id against the user’s profile to preserve the context and isolate the user’s conversation history.

We have demonstrated how to create a conversational search application using the built-in features of OpenSearch Service.

Clean up

To avoid incurring future charges, delete the resources created while building this solution:

  1. Delete the OpenSearch Service domain.
  2. Delete the connector.
  3. Delete the index.

Conclusion

In this post, we demonstrated how to use OpenSearch agents and tools to create a RAG pipeline with conversational search. By integrating with ML models, vectorizing questions, and interacting with LLMs to improve prompts, this configuration oversees the entire process. This method allows you to quickly develop AI assistants that are ready for production without having to start from scratch.

If you’re building a RAG pipeline with conversational history to let users ask follow-up questions for more refined answers, give it a try and share your feedback or questions in the comments!

About the author

Bharav Patel is a Specialist Solution Architect, Analytics at Amazon Web Services. He primarily works on Amazon OpenSearch Service and helps customers with key concepts and design principles of running OpenSearch workloads on the cloud. Bharav likes to explore new places and try out different cuisines.

Enhance stability with dedicated cluster manager nodes using Amazon OpenSearch Service

Post Syndicated from Chinmayi Narasimhadevara original https://aws.amazon.com/blogs/big-data/enhance-stability-with-dedicated-cluster-manager-nodes-using-amazon-opensearch-service/

Amazon OpenSearch Service is a managed service that you can use to secure, deploy, and operate OpenSearch clusters at scale in the AWS Cloud. With OpenSearch Service, you can configure clusters with different types of node options such as data nodes, dedicated cluster manager nodes, dedicated coordinator nodes, and UltraWarm nodes. When configuring your OpenSearch Service domain, you can exercise different node options to manage your cluster’s overall stability, performance, and resiliency.

In this post, we show how to enhance the stability of your OpenSearch Service domain with dedicated cluster manager nodes and how using these in deployment enhances your cluster’s stability and reliability.

The benefit of dedicated cluster manager nodes

A dedicated cluster manager node handles the behind-the-scenes work of running an OpenSearch Service cluster, but it doesn’t store actual data or process search requests. In the absence of dedicated cluster manager nodes, OpenSearch Service will use data nodes for cluster management; combining these responsibilities on the data nodes can impact performance and stability because data operations (like indexing and searching) compete with critical cluster management tasks for computing resources. The dedicated cluster manager node is responsible for several key tasks: monitoring and keeping track of all the data nodes in the cluster, knowing how many indexes and shards there are and where they’re located, and routing data to the correct places. They also update and share the cluster state whenever something changes, like creating an index or adding and removing nodes. The problem, however, is that when traffic gets heavy, the cluster manager node can get overloaded and become unresponsive. If this happens, your cluster will not respond to write requests until it elects a new cluster manager, at which point the cycle might repeat itself. You can alleviate this issue by deploying dedicated cluster manager instances, whereby this separation of duties between the manager node and the data nodes results in a much more stable cluster.

Calculating the number of dedicated cluster manager nodes

In OpenSearch Service, a single node is elected as the cluster manager from all eligible nodes through a quorum-based voting process, confirming consensus before taking on the responsibility of coordinating cluster-wide operations and maintaining the cluster’s state. Quorum is the minimum number of nodes that need to agree before the cluster makes important decisions. It helps keep your data consistent and your cluster running smoothly. When you use dedicated cluster manager nodes, only those nodes are eligible for election and OpenSearch Service sets the quorum to half of the nodes, rounded down to the nearest whole number, plus one. One dedicated cluster manager node is explicitly prohibited by OpenSearch Service because you have no backup in the event of a failure. Using three dedicated cluster manager nodes makes sure that even if one node fails, the remaining two can still reach a quorum and maintain cluster operations. We recommend three dedicated cluster manager nodes for production use cases. Multi-AZ with standby is an OpenSearch Service feature designed to deliver four 9s of availability using a third AWS Availability Zone as a standby. When you use Multi-AZ with standby, the service requires three dedicated cluster manager nodes. If you deploy with Multi-AZ without standby or Single-AZ, we still recommend three dedicated cluster manager nodes. It provides two backup nodes in the event of one cluster manager node failure and the necessary quorum (two) to elect a new manager. You can choose three or five dedicated cluster manager nodes.

Having five dedicated cluster manager nodes works as well as three, and you can lose two nodes while maintaining a quorum. But because only one dedicated cluster manager node is active at any given time, this configuration means you pay for four idle nodes.

Cluster manager node configurations for different domain creation methods

This section explains the resources each domain creation method and template deploy when you set up an OpenSearch Service domain.

With the Easy create option, you can quickly create a domain using ‘multi-AZ with standby’ for high availability three-cluster manager nodes distributed across three Availability Zones. The following table summarizes the configuration.

Domain Creation Method Output
Easy Create

Dedicated cluster manager node: Yes

Number of cluster manager nodes: 3

Availability Zones: 3

Standby: Yes

The Standard create option provides templates for ‘Production’ and ‘Dev/test’workloads. Both templates come with a Domain with standby and a Domain without standby deployment choice. The following table summarizes these configuration options.

Domain Creation Method Template Deployment Option Output
Standard Create Production Domain with standby

Requires dedicated cluster manager node

Number of cluster manager nodes: 3

Availability Zones: 3

Standby: Yes

Instance type choice: Yes
Standard create Production Domain without standby

Requires dedicated cluster manager node

Number of cluster manager nodes: 3, 5

Availability Zones: 3

Standby: No

Instance type choice: Yes
Standard Create Dev/test Domain with standby

Requires dedicated cluster manager node

Number of cluster manager nodes: 3

Availability Zones: 3

Standby: Yes

Instance type choice: Yes
Standard create Dev/test Domain without standby Does not require dedicated cluster manager node

Choosing a dedicated cluster manager instance type

Dedicated cluster manager instances typically handle critical cluster operations like shard distribution and index management and track cluster state changes. It’s recommended to select a comparatively smaller instance type. Refer to Choosing instance types for dedicated master nodes for more information on instance types for dedicated cluster manager nodes.

You should expect to occasionally adjust cluster manager instance size and type as your workload evolves over time. As with all scale questions, you need to monitor performance and make sure you have enough CPU and Java virtual machine (JVM) heap for your dedicated cluster managers. We recommend using Amazon CloudWatch alarms to monitor the following CloudWatch metrics, and adjust according to the alarm state:

  • ManagerCPUUtilization – Maximum is greater than or equal to 50% for 15 minutes, three consecutive times
  • ManagerJVMMemoryPressure – Maximum is greater than or equal to 95% for 1 minute, three consecutive times

Conclusion

Dedicated cluster manager nodes provide added stability and protection against split-brain situations, can be of a different instance type than data nodes, and are an obvious benefit when OpenSearch Service is backing mission-critical applications for production workloads. They are typically not required for development workloads like proof of concept because the cost of running a dedicated cluster manager node exceeds the tangible benefits of keeping the cluster up and running. To learn more about OpenSearch best practices, see link.

About the authors

Imtiaz (Taz) Sayed is the WW Tech Leader for Analytics at AWS. He enjoys engaging with the community on all things data and analytics. He can be reached through LinkedIn.

Chinmayi Narasimhadevara is a Senior Solutions Architect focused on Data Analytics and AI at AWS. She helps customers build advanced, highly scalable, and performant solutions.

Security updates for Thursday

Post Syndicated from jake original https://lwn.net/Articles/1028554/

Security updates have been issued by AlmaLinux (.NET 9.0, aardvark-dns, apache-commons-beanutils, bootc, buildah, corosync, delve and golang, exiv2, expat, firefox, ghostscript, git, git-lfs, gnutls, grafana, grafana-pcp, grub2, gstreamer1, gstreamer1-plugins-bad-free, gstreamer1-plugins-ugly-free, and gstreamer1-rtsp-server, gstreamer1-plugins-base, gstreamer1-plugins-good, gvisor-tap-vsock, iptraf-ng, java-21-openjdk, kernel, keylime-agent-rust, krb5, libarchive, libblockdev, libsoup3, libtasn1, libvpx, libxslt, microcode_ctl, mod_auth_openidc, nodejs22, nodejs:20, openjpeg2, osbuild and osbuild-composer, perl-FCGI, perl-Module-ScanDeps, perl-YAML-LibYAML, php, php:8.2, php:8.3, podman, protobuf, python-jinja2, python-requests, python3.11, python3.12, python3.12-cryptography, python3.9, rpm-ostree, rsync, rust-bootupd, skopeo, thunderbird, tigervnc, tomcat, tomcat9, webkit2gtk3, xdg-utils, xorg-x11-server, and xorg-x11-server-Xwayland), Debian (ring), Mageia (libarchive and rootcerts, nss & firefox), Oracle (.NET 9.0, corosync, firefox, osbuild-composer, pam, python3, python3.11, python3.12, python3.9, skopeo, sudo, and thunderbird), Red Hat (microcode_ctl, pam, php, thunderbird, tigervnc, xorg-x11-server, xorg-x11-server and xorg-x11-server-Xwayland, and xorg-x11-server-Xwayland), SUSE (clamav, icu, libgepub, libsoup, python-requests, tomcat, and xorg-x11-server), and Ubuntu (clamav, logback, mongo-c-driver, pcs, and python-flask-cors).

Kaltura reduces observability operational costs by 60% with Amazon OpenSearch Service

Post Syndicated from Ido Ziv original https://aws.amazon.com/blogs/big-data/kaltura-reduces-observability-operational-costs-by-60-with-amazon-opensearch-service/

This post is co-written with Ido Ziv from Kaltura.

As organizations grow, managing observability across multiple teams and applications becomes increasingly complex. Logs, metrics, and traces generate vast amounts of data, making it challenging to maintain performance, reliability, and cost-efficiency.

At Kaltura, an AI-infused video-first company serving millions of users across hundreds of applications, observability is mission-critical. Understanding system behavior at scale isn’t just about troubleshooting—it’s about providing seamless experiences for customers and employees alike. But achieving effective observability at this scale comes with challenges: managing spans; correlating logs, traces, and events across distributed systems; and maintaining visibility without overwhelming teams with noise. Balancing granularity, cost, and actionable insights requires constant tuning and thoughtful architecture.

In this post, we share how Kaltura transformed its observability strategy and technological stack by migrating from a software as a service (SaaS) logging solution to Amazon OpenSearch Service—achieving higher log retention, a 60% reduction in cost, and a centralized platform that empowers multiple teams with real-time insights.

Observability challenges at scale

Kaltura ingests over 8TB of logs and traces daily, processing more than 20 billion events across 6 production AWS Regions and over 200 applications—with log spikes reaching up to 6 GB per second. This immense data volume, combined with a highly distributed architecture, created significant challenges in observability. Historically, Kaltura relied on a SaaS-based observability solution that met initial requirements but became increasingly difficult to scale. As the platform evolved, teams generated disparate log formats, applied retention policies that no longer reflected data value, and operated more than 10 organically grown observability sources. The lack of standardization and visibility required extensive manual effort to correlate data, maintain pipelines, and troubleshoot issues – leading to rising operational complexity and fixed costs that didn’t scale efficiently with usage.

Kaltura’s DevOps team recognized the need to reassess their observability solution and began exploring a variety of options, from self-managed platforms to fully managed SaaS offerings. After a comprehensive evaluation, they made the strategic decision to migrate to OpenSearch Service, using its advanced features such as Amazon OpenSearch Ingestion, the Observability plugin, UltraWarm storage, and Index State Management.

Solution overview

Kaltura created a new AWS account that would be a dedicated observability account, where OpenSearch Service was deployed. Logs and traces were collected from different accounts and producers such as microservices on Amazon Elastic Kubernetes Service (Amazon EKS) and services running on Amazon Elastic Compute Cloud (Amazon EC2).

By using AWS services such as AWS Identity and Access Management (IAM), AWS Key Management Service (AWS KMS), and Amazon CloudWatch, Kaltura was able to meet the standards to create a production-grade system while keeping security and reliability in mind. The following figure shows a high-level design of the environment setup.

Ingestion

As seen in the following diagram, logs are shipped using log shippers, also known as collectors. In Kaltura’s case, they used Fluent Bit. A log shipper is a tool designed to collect, process, and transport log data from various sources to a centralized location, such as log analytics platforms, management systems, or an aggregator system. Fluent Bit was used in all sources and also provided light processing abilities. Fluent Bit was deployed as a daemonset in Kubernetes. The application development teams didn’t change their code, because the Fluent Bit pods were reading the stdout of the application pods.

The following code is an example of FluentBit configurations for Amazon EKS:

[INPUT]
   Name                tail
   Path                /var/log/containers/*.log
   Tag                 kube.*
   Skip_Long_Lines     On
   multiline.parser    docker, cri
[FILTER]
   alias               k8s
   # kubernetes filter to parse all logs
   Name                kubernetes
   Match               kube.*
   Kube_Tag_Prefix     kube.var.log.containers.
   Annotations         On
   Labels              Off
   Merge_Log           On
   Keep_Log            Off
   Kube_URL            https://kubernetes.default.svc.cluster.local:443 
[FILTER]
   alias               apps
   Name                rewrite_tag
   Match               kube.*
   Rule                $kubernetes['annotations']['kaltura.com/observability'] ^apps$ 
[OUTPUT]
   Name                http
   Match               apps.*
   Alias               apps
   Host                xxxxx.us-east-1.osis.amazonaws.com
   Port                443
   URI                 /log/apps
   Format              json
   aws_auth            true
   aws_region          us-east-1
   aws_service         osis
   aws_role_arn        arn:aws:iam::xxxxx:role/osis-ingestion-role
   Log_Level           trace
   tls On

Spans and traces were collected directly from the application layer using a seamless integration approach. To facilitate this, Kaltura deployed an OpenTelemetry Collector (OTEL) using the OpenTelemetry Operator for Kubernetes. Additionally, the team developed a custom OTEL code library, which was incorporated into the application code to efficiently capture and log traces and spans, providing comprehensive observability across their system.

Data from Fluent Bit and OpenTelemetry Collector was sent to OpenSearch Ingestion, a fully managed, serverless data collector that delivers real-time log, metric, and trace data to OpenSearch Service domains and Amazon OpenSearch Serverless collections. Each producer sent data to a specific pipeline, one for logs and one for traces, where data was transformed, aggregated, enriched, and normalized before being sent to OpenSearch Service. The trace pipeline used the otel_trace and service_map processors, while using the OpenSearch Ingestion OpenTelemetry trace analytics blueprint.

The following code is an example of the OpenSearch Ingestion pipeline for logs:

version: "2"
entry-pipeline:
 source:
   http:
     path: "/log/apps"

 processor:
   - add_entries:
       entries:
       - key: "log_type"
         value: "default"
       - key: "log_type"
         value: "api"
         add_when: 'contains(/filename, "api.log")'
         overwrite_if_key_exists: true
       - key: "log_type"
         value: "stats"
         add_when: 'contains(/filename, "stats.log")'
         overwrite_if_key_exists: true
       - key: "log_type"
         value: "event"
         add_when: 'contains(/filename, "event.log")'
         overwrite_if_key_exists: true
       - key: "log_type"
         value: "login"
         add_when: 'contains(/filename, "login.log")'
         overwrite_if_key_exists: true

   - grok:
       grok_when: '/log_type == "api"'
       match:
         log: ['^\[%%{DATA:timestamp}] \[%%{DATA:logIp}\] \[%%{DATA:host}\] \[%%{WORD:id}\] %%{WORD:priorityName}\(%%{NUMBER:priority}\): \[memory: %%{DATA:memory} MB, real: %%{DATA:real}MB\] %%{GREEDYDATA:message}']

   - date:
       match:
         - key: timestamp
           patterns: ["dd-MMM-yyyy HH:mm:ss", "dd/MMM/yyyy:HH:mm:ss Z", "EEE MMM dd HH:mm:ss.SSSSSS yyyy"]

       destination: "@timestamp"
       output_format: "yyyy-MM-dd'T'HH:mm:ss"

   - rename_keys:
       entries:
       - from_key: "timestamp"
         to_key: "@timestamp"
         overwrite_if_to_key_exists: false
       - from_key: "date"
         to_key: "@timestamp"
         overwrite_if_to_key_exists: false

   - drop_events:
       drop_when: 'contains(/filename, "simplesamlphp.log")'


 sink:
   - opensearch:
       hosts: ["${opensearch_host}"]
       index: '$${/env}-api-$${/log_type}-app-logs'
       index_type: custom
       action: create
       bulk_size: 20
       aws:
         sts_role_arn: ${sts_role_arn}
         region:  ${region}
       dlq:
         s3:
           bucket: "${bucket}"
           key_path_prefix: 'my-app-dlq-files'
           region: "${region}"
           sts_role_arn: "${sts_role_arn}"

The preceding example shows the use of processors such as grok, date, add_entries, rename_keys, and drop_events:

  • add_entries:
    • Adds a new field log_type based on filename
    • Default: “default”
    • If the filename contains specific substrings (such as api.log or stats.log), it assigns a more specific type
  • grok:
    • Applies Grok parsing to logs of type “api”
    • Extracts fields like timestamp, logIp, host, priorityName, priority, memory, real, and message using a custom pattern
  • date:
    • Parses timestamp strings into a standard datetime format
    • Stores it in a field called @timestamp based on ISO8601 format
    • Handles multiple timestamp patterns
  • rename_keys:
    • timestamp or date are renamed into @timestamp
    • Does not overwrite if @timestamp already exists
  • drop_events:
    • Drops logs where filename contains simplesamlphp.log
    • This is a filtering rule to ignore noisy or irrelevant logs

The following is an example of the input of a log line:

   "log": "[25-Mar-2025 18:23:18] [127.0.0.1] [the-most-awesome-server-in-kaltura] [67e2f496cc321] INFO(6): [memory: 4.51 MB, real: 6MB] [request: 1] [time: 0.0263s / total: 0.0263s]",

After processing, we get the following code:

    "log_type": "api",
    "priorityName": "INFO",
    "memory": "4.51",
    "host": "the-most-awesome-server-in-kaltura",
    "real": "6",
    "priority": "6",
    "message": "[request: 1] [time: 0.0263s / total: 0.0263s]",
    "logIp": "127.0.0.1",
    "id": "67e2f496cc321",
    "@timestamp": "2025-03-25T18:23:18"

Kaltura followed some OpenSearch Ingestion best practices, such as:

  • Including a dead-letter queue (DLQ) in pipeline configuration. This can significantly help troubleshoot pipeline issues.
  • Starting and stopping pipelines to optimize cost-efficiency, when possible.
  • During the proof of concept stage:
    • Installing Data Prepper locally for faster development iterations.
    • Disabling persistent buffering to expedite blue-green deployments.

Achieving operational excellence with efficient log and trace management

Logs and traces play a vital role in identifying operational issues, but they come with unique challenges. First, they represent time series data, which inherently evolves over time. Second, their value typically diminishes as time passes, making efficient management crucial. Third, they are append-only in nature. With OpenSearch, Kaltura faced distinct trade-offs between cost, data retention, and latency. The goal was to make sure valuable data remained accessible to engineering teams with minimal latency, but the solution also needed to be cost-effective. Balancing these factors required thoughtful planning and optimization.

Data was ingested to OpenSearch data streams, which simplifies the process of ingesting append-only time series data. Several Index State Management (ISM) policies were applied to different data streams, which were dependent on log retention requirements. ISM policies handled moving indexes from hot storage to UltraWarm, and eventually deleting the indexes. This allowed a customizable and cost-effective solution, with low latency for querying new data and reasonable latency for querying historical data.

The following example ISM policy makes sure indexes are managed efficiently, rolled over, and moved to different storage tiers based on their age and size, and eventually deleted after 60 days. If an action fails, it is retried with an exponential backoff strategy. In case of failures, notifications are sent to relevant teams to keep them informed.

{
    "id": "retention",
    "policy": {
        "description": "production ISM",
        },
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "retry": {
                            "count": 5,
                            "backoff": "exponential",
                            "delay": "1h"
                        },
                        "rollover": {
                            "min_primary_shard_size": "30gb",
                            "copy_alias": false
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "warm",
                        "conditions": {
                            "min_index_age": "2d"
                        }
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "retry": {
                            "count": 5,
                            "backoff": "exponential",
                            "delay": "1h"
                        },
                        "warm_migration": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "cold",
                        "conditions": {
                            "min_index_age": "14d"
                        }
                    }
                ]
            },
            {
                "name": "cold",
                "actions": [
                    {
                        "retry": {
                            "count": 5,
                            "backoff": "exponential",
                            "delay": "1h"
                        },
                        "cold_migration": {
                            "start_time": null,
                            "end_time": null,
                            "timestamp_field": "@timestamp",
                            "ignore": "none"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "60d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "cold_delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "*-logs"
                ],
                "priority": 50,
            }
        ]
    }
}

To create a data stream in OpenSearch, a definition of index template is required, which configures how the data stream and its backing indexes will behave. In the following example, the index template specifies key index settings such as the number of shards, replication, and refresh interval—controlling how data is distributed, replicated, and refreshed across the cluster. It also defines the mappings, which describe the structure of the data—what fields exist, their types, and how they should be indexed. These mappings make sure the data stream knows how to interpret and store incoming log data efficiently. Finally, the template enables the @timestamp field as the time-based field required for a data stream.

{
  "index_patterns": [
    "*my-app-logs"
  ],
  "template": {
    "settings": {
      "index.number_of_shards": "32",
      "index.number_of_replicas": "0",
      "index.refresh_interval": "60s"
    },
    "mappings": {
      "properties": {
        "priorityName": {
          "type": "keyword"
        },
        "log_type": {
          "type": "keyword"
        },
        "@timestamp": {
          "type": "date"
        },
        "memory": {
          "type": "float"
        },
        "host": {
          "type": "keyword"
        },
        "pid": {
          "type": "keyword"
        },
        "real": {
          "type": "float"
        },
        "env": {
          "type": "keyword"
        },
        "message": {
          "type": "text"
        },
        "priority": {
          "type": "integer"
        },
        "logIp": {
          "type": "ip"
        }
      }
    }
  },
  "composed_of": [],
  "priority": "100",
  "_meta": {
    "flow": "simple"
  },
  "data_stream": {
    "timestamp_field": {
      "name": "@timestamp"
    }
  },
  "name": "my-app-logs"
}

Implementing role-based access control and user access

The new observability platform is accessed by many types of users; internal users log in to OpenSearch Dashboards using SAML-based federation with Okta. The following diagram illustrates the user flow.

Each user accesses the dashboards to view observability items relevant to their role. Fine-grained access control (FGAC) is enforced in OpenSearch using built-in IAM role and SAML group mappings to implement role-based access control (RBAC).When users log in to the OpenSearch domain, they are automatically routed to the appropriate tenant based on their assigned role. This setup makes sure developers can create dashboards tailored to debugging within development environments, and support teams can build dashboards focused on identifying and troubleshooting production issues. The SAML integration alleviates the need to manage internal OpenSearch users entirely.

For each role in Kaltura, a corresponding OpenSearch role was created with only the necessary permissions. For instance, support engineers are granted access to the monitoring plugin to create alerts based on logs, whereas QA engineers, who don’t require this functionality, are not granted that access.

The following screenshot shows the role of the DevOps engineers defined with cluster permissions.

These users are routed to their own dedicated DevOps tenant, to which they only have write access. This makes it possible for different users from different roles in Kaltura to create the dashboard items that focus on their priorities and needs. OpenSearch supports backend role mapping; Kaltura mapped the Okta group to the role so when a user logs in from Okta, they automatically get assigned based on their role.

This also works with IAM roles to facilitate automations in the cluster using external services, such as OpenSearch Ingestion pipelines, as can be seen in the following screenshot.

Using observability features and service mapping for enhanced trace and log correlation

After a user is logged in, they can use the Observability plugins, view surrounding events in logs, correlate logs and traces, and use the Trace Analytics plugin. Users can inspect traces and spans, and group traces with latency information using built-in dashboards. Users can also drill down to a specific trace or span and correlate it back to log events. The service_map processor used in OpenSearch Ingestion sends OpenTelemetry data to create a distributed service map for visualization in OpenSearch Dashboards.

Using the combined signals of traces and spans, OpenSearch discovers the application connectivity and maps them to a service map.

After OpenSearch ingests the traces and spans from Otel, they are aggregated to groups according to paths and trends. Durations are also calculated and presented to the user over time.

With a trace ID, it’s possible to filter out all the relevant spans by the service and see how long each took, identifying issues with external services such as MongoDB and Redis.

From the spans, users can discover the relevant logs.

Post-migration enhancements

After the migration, a strong developer community emerged within Kaltura that embraced the new observability solution. As adoption grew, so did requests for new features and enhancements aimed at improving the overall developer experience.

One key improvement was extending log retention. Kaltura achieved this by re-ingesting historical logs from Amazon Simple Storage Service (Amazon S3) using a dedicated OpenSearch Ingestion pipeline with Amazon S3 read permissions. With this enhancement, teams can access and analyze logs from up to a year ago using the same familiar dashboards and filters.

In addition to monitoring EKS clusters and EC2 instances, Kaltura expanded its observability stack by integrating more AWS services. Amazon API Gateway and AWS Lambda were introduced to support log ingestion from external vendors, allowing for seamless correlation with existing data and broader visibility across systems.

Finally, to empower teams and promote autonomy, data stream templates and ISM policies are managed directly by developers within their own repositories. By using infrastructure as code tools like Terraform, developers can define index mappings, alerts, and dashboards as code—versioned in Git and deployed consistently across environments.

Conclusion

Kaltura successfully implemented a smart log retention strategy, extending real time retention from 5 days for all log types to 30 days for critical logs, while maintaining cost-efficiency through the use of UltraWarm nodes. This approach led to a 60% reduction in costs compared to their previous solution. Additionally, Kaltura consolidated their observability platform, streamlining operations by merging 10 separate systems into a unified, all-in-one solution. This consolidation not only improved operational efficiency but also sparked increased engagement from developer teams, driving feature requests, fostering internal design collaborations, and attracting early adopters for new enhancements. If Kaltura’s journey has inspired you and you’re thinking about implementing a similar solution in your organization, consider these steps:

About the authors

Ido Ziv is a DevOps team leader in Kaltura with over 6 years of experience. His hobbies include sailing and Kubernetes (but not at the same time).

Roi Gamliel is a Senior Solutions Architect helping startups build on AWS. He is passionate about the OpenSearch Project, helping customers fine-tune their workloads and maximize results.

Yonatan Dolan is a Principal Analytics Specialist at Amazon Web Services. He is located in Israel and helps customers harness AWS analytical services to use data, gain insights, and derive value.

Surveillance Used by a Drug Cartel

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2025/07/surveillance-used-by-a-drug-cartel.html

Once you build a surveillance system, you can’t control who will use it:

A hacker working for the Sinaloa drug cartel was able to obtain an FBI official’s phone records and use Mexico City’s surveillance cameras to help track and kill the agency’s informants in 2018, according to a new US justice department report.

The incident was disclosed in a justice department inspector general’s audit of the FBI’s efforts to mitigate the effects of “ubiquitous technical surveillance,” a term used to describe the global proliferation of cameras and the thriving trade in vast stores of communications, travel, and location data.

[…]

The report said the hacker identified an FBI assistant legal attaché at the US embassy in Mexico City and was able to use the attaché’s phone number “to obtain calls made and received, as well as geolocation data.” The report said the hacker also “used Mexico City’s camera system to follow the [FBI official] through the city and identify people the [official] met with.”

FBI report.

По прайда ще ги познаете

Post Syndicated from Светла Енчева original https://www.toest.bg/po-prayda-shte-gi-poznaete/

По прайда ще ги познаете

За прайдовете обикновено се говори в контекста на правата на ЛГБТИ+ хората. Тазгодишният в Будапеща обаче влезе в световните новини най-вече с друго – с мащаба си. По оценки на организаторите в него са се включили между 180 000 и 200 000 души. Това се равнява на около 5% от населението на Унгария, наброяващо 9 630 000 души (което не означава, че участниците са само унгарци – сред тях има и хора от други държави, включително България). Дори разчетите на организаторите да са раздути, снимки на събитието от въздуха създават известна представа за огромния му мащаб. Със сигурност в Будапеща не е имало по-голям прайд от този.

Забраната като катализатор

Рекордната посещаемост на прайда в Будапеща всъщност е реакция срещу забраната му. По този повод възникна и нова дума – orbanned, която може да се преведе като „орбаннат“ (от Орбан и „баннат“ – забранен). Тя означава увеличаване на популярността на нещо като ефект от забраната му.

През април 2025 г. унгарският парламент прие с огромно мнозинство поправка в конституцията на страната, чиято цел беше публичните събития, свързани с правата на ЛГБТИ+ хората, да се обявят за противоконституционни. Новият текст в основния закон е формулиран по следния начин: 

[…] правото на децата на правилно морално, физическо и умствено развитие – с изключение на правото на живот – има предимство пред всички други основни права, включително правото на събрания.

Предпоставка за тази формулировка е схващането, че със самото си присъствие ЛГБТИ+ хората са заплаха за децата – теза, широко споделяна от руската пропаганда и от ултраконсервативни кръгове, зад която не стои научна обосновка. С подобни аргументи обаче бяха постепенно ограничавани правата на сексуалните и половите малцинства в Русия, докато самото им съществуване беше обявено за „екстремизъм“. Последваха хайки в гей клубове и сауни и присъди на жени заради обичките, които носят.

Месец преди конституционната промяна Унгария забрани прайда със закон. Добавката в Основния закон на страната би следвало да бетонира законовата промяна.

И все пак защо ограничаването на една малцинствена група постигна обратния ефект?

Когато се предприемат законови и конституционни мерки против ЛГБТИ+ хората, това е само средство. Вторачването в интимността е стъпка към установяването на тотален контрол върху живота на всички, независимо от сексуалната им ориентация или половата им идентичност. Неслучайно подобни посегателства обикновено вървят заедно с опити за ограничаване не само на абортите, а и на неправителствения сектор, медиите, свободата на изразяване, политическия плурализъм.

След 15 години управление на „нелибералната демокрация“ на Виктор Орбан унгарците осъзнават по трудния начин, че забраната на прайда е поредният пирон в ковчега на либералната демокрация. Ето защо присъствието на шествието е подкрепа не толкова за куиър хората, колкото за демокрацията – такава, каквато е тя в европейските страни, които не са проядени от руска пропаганда.

По прайда ще ги познаете
По прайда ще ги познаете
По прайда ще ги познаете

Прайдът в Будапеща. Източник: The Queer Review

Ролята на кмета

Как обаче се провежда забранено събитие? За да стане възможен прайдът в Будапеща, отговорност поема кметът на унгарската столица – опозиционният лидер Гергели Карачони. Тъй като е ясно, че полицията няма как да разреши двойно забранено (и по закон, и по конституция) шествие, Карачони прави Общината организатор и по този начин избягва искането на позволение.

Става ли чрез тази маневра прайдът законен? Не, защото конституцията и законите имат приоритет над решенията на местната власт. Целта на кмета обаче е не да легализира събитието (това е отвъд правомощията му), а то да се проведе.

Освен това ЛГБТИ+ организациите и активистите са много по-лесна мишена на репресивни мерки, отколкото управата на столицата на една държава. Играейки вабанк, Карачони успява да постигне мирно протичане на събитието, без полицейско насилие и без значими провокации. (Казвам „без значими“, защото мой познат, присъствал на прайда в Будапеща, е видял нидерландец, държащ голям дървен кръст и крещящ на участниците колко са грешни.)

А в София?

В България прайдът продължава да не се разпознава като важен индикатор дори за либералната демокрация. 17 години след началото на дебата той продължава да е на ниво имат ли куиър хората право на съществуване и колко точно публично присъствие от тяхна страна не е „прекалено“.

Още преди да стане кмет на София, Васил Терзиев каза на среща с граждани, че „имаме нужда от „София прайд“ – мирно събитие, на което се събират много хора с деца“. Въпреки това Общината не само не участва в организацията на събитието, както е в много градове в други страни (например в САЩ, Канада, Франция, Нидерландия, Германия и др.), а и вече втора година Терзиев не е излязъл на сцената да поздрави участниците. За разлика от представителите на редица дипломатически мисии, които с присъствието си стават гаранти за мирното протичане на демонстрацията.

За сметка на това тази година програмата на „София прайд“ беше променена, за да не съвпадне провеждането на шествието с хомофобски събития, заявили в Общината обичайните за прайда маршрути. Имаше не едно, а две анти-ЛГБТИ+ мероприятия – „Шествие за семейството“, завършващо с концерт пред храм-паметника „Александър Невски“ и концерт „Традицията е бъдещето“ пред НДК.

Конституции за избирателна (зло)употреба

Вместо похвали и увеличена популярност Гергели Карачони не заслужава ли порицание, ако не и наказание, че умишлено и демонстративно е нарушил Конституцията на Унгария? Хващам се на бас, че повечето хора, които отправят този упрек срещу кмета на Будапеща, биха оправдали неспазването на конституцията в други случаи.

Когато например районният кмет на „Илинден“ Емил Бранчевски нареди разрушаването на ромската махала в кв. „Захарна фабрика“ и така остави около 200 души без дом, той наруши Конституцията на България. Защото в нея пише, че международните договори, които България е приела, са част от вътрешното законодателство и имат приоритет над него. А Европейският съд по правата на човека в Страсбург изрично е предупредил Бранчевски, че се кани да погази европейското законодателство. Той обаче се изкара герой, защото бил изпълнил… общинска заповед.

Какво да кажем за американския президент Доналд Тръмп, който си позволява да нарушава всеки текст от Конституцията на САЩ, ако не му изнася? За разлика от хомофобските промени в Основния закон на Унгария, които са съвсем скорошни, 14-тата поправка на Конституцията на САЩ, според която всеки, роден на американска земя, става гражданин на страната, датира от далечната 1868 година. Което не пречи на Тръмп на практика да не се съобразява с нея. И на милиони хора – да му ръкопляскат за това.

Конституционният съд и Върховният касационен съд в България пък тълкуват Основния закон, хиперболизирайки някои елементи в него. Основно – че бракът е между мъж и жена и че традиционната религия е православието. И системно не забелязват други – например че религиозните институции са отделени от държавата и не могат да се използват за политически цели. Така те изобретяват несъществуващи „правни традиции“.

Примерите с Карачони и Бранчевски всъщност показват колко важно може да е отношението на един кмет към конституцията.

В първия случай нарушаването ѝ от страна на кмета е в името на съхраняването на либералната демокрация, във втория – от популизъм и дискриминация. И двата случая предизвикват международен отзвук, макар и от различен порядък.

Кристиян Таков беше призовал да пазим правото – включително и от закона. Това важи с особена сила, когато се приемат противоправни и антидемократични закони. И конституционни промени.

Трябва да променим условията за упражняване на публична власт

Post Syndicated from Bozho original https://blog.bozho.net/blog/4488

Наследството от 45 години тоталитарен режим е държава, в която служби и прокуратура се използват за политически цели, а основните им функции са понякога на заден план. И в тази среда изтичането на записи, снимки, протоколи от разпити се сблъсква с разбираемото недоверие у избирателите и с усещането за “активни мероприятия”.

Такава е реакцията на всички партийни ядра. И това може да даде измамна увереност на политическите ръководства, че няма електорален проблем. Това е обобщение на база на множество примери в историята, като последните и най-ярки такива са при ГЕРБ – снимките с пачките и кюлчетата, записите “ще му размажа физиономията”, а преди това “Мишо Бирата”.

Реакцията на ГЕРБ и техните най-верни избиратели беше да се игнорира съдържанието и какво значи за нивото на изгнилост на институциите и да се постави фокус върху това дали записите и снимките са истински, кой ги е направил, откъде са изтекли, защо са изтекли и кой има интерес от тях. Но тогава една немалка част от периферията на ГЕРБ си тръгна. Не шумно, не с късане на ризи в социалните мрежи, но си тръгна на следващите избори.

И тъй като прокуратурата и антикорупционната комисия в последните години успешно си изградиха образа на бухалки на Пеевски, а „прихващането“ чрез морков и тояга е основен милиционерски похват, сега много хора в ядрото на общите ни избирателите с колегите от Продължаваме промяната са убедени, че това е скалъпен скандал. Поредното “активно мероприятие”. Задават се същите въпроси – дали записът е истински, кой го е правил, откъде е изтекъл, защо сега и кой има интерес.

В миналото за такива скандали отговорност не се е поемала. Сега Кирил Петков пое такава, което ние отчетохме като отговорното и разумно политическо действие. И то е именно такова, и то не само от вътрешнопартийна гледна точка.

Във всички такива случаи в историята, обаче, неглижирането на ситуацията винаги е носело вреди. И ще направим грешка, ако не си вземем поуки и не вземем мерки. Ако не направим вътрешни разследвания, ако не изкореним предпоставките за такива проблеми. Убеден съм, че колегите от ПП до своя форум през септември ще направят всичко това, за да бъдат премахнати съмненията в коалицията.

Защото докато част от избирателите смятат, че всичко е постановка, друга част очакват убедителен отговор на случилото се.

И накрая – борбата с корупцията. За да не просто изтъркан политически слоган – борбата с корупцията е битка за върховенство на закона, за независим съд и най-вече, за отчетна прокуратура, която разследва обективно и всестранно всички съмнения за злоупотреба с обществен ресурс. Всички, а не само тези, които бъдат посочени от текущия политически “север”, по който се ориентира компасът на главния прокурор и антикорупционната комисия.

В момента това е Пеевски. Или “Делян”, както казва Борисов. И именно затова “войната” с него в борисовите очи е страшна, защото следващите записи могат да са (пак) на Борисов, а досъдебните производства – срещу него и хората му. Отпадането на мерките за ограничаване на влиянието на Пеевски от коалиционното споразумение беше просто поредният пример за този страх. Това е и втората разлика с предишни записи – че наказателно преследване след тях е нямало.

Но е време да си дадем сметка, че най-ефективните политики с антикорупционен ефект върху публичния и частен сектор са десните политики. Когато намалиш преразпределението от държавата, когато намалиш административното регулиране, когато стимулираш пазарната конкуренция, когато повишиш прозрачността на администрацията, автоматично намаляваш корупционния риск. Илюзия е, че с повече държава и раздути бюджети, но с правилни хора начело, ще има значително по-малко корупция. Може временно да намалее, докато не се прегрупира и като река не намери нови пътища, през които да тече.

Доброто и злото са лични морални категории. И те са много важни за хода на историята. Но ние трябва да излезем от режима на политическо противопоставяне на база на морални категории. И вместо това да променим условията, при които се упражнява публична власт. Защото иначе репресивната машината винаги ще е в ръцете на някого, ще корумпира, ще мачка, ще изнудва, ще “сготвя”. И никой няма да е застрахован.

И даже това не е толкова страшно, колкото това, че никой няма да може да изгради доверие между политическите представители и хората и ще отвори широко портите за крайния популизъм и авторитаризма, пред сринатия смисъл от демократично представителство.

Нашата трудна политическа задача е да не позволим да се стигне дотам.

Материалът Трябва да променим условията за упражняване на публична власт е публикуван за пръв път на БЛОГодаря.

Begin your Code Club journey: Set up a club today

Post Syndicated from Lou Loxley original https://www.raspberrypi.org/blog/begin-your-code-club-journey-set-up-a-club-today/

Inspired to set up a Code Club but not sure where to start? In this blog we hear from the amazing network of Code Club leaders and mentors, and we give you practical tips on how to take the first steps to set up your own club. 

Two young coders at a Code Club.

About Code Club

Code Club is a thriving global community of clubs where young people can develop the confidence to create with digital technologies in a fun and supportive space. Code Clubs run in schools, libraries and other community settings. We have already supported 2 million school-aged young people worldwide to make things they care about with code.

We offer free training and ongoing support for anyone who would like to set up a Code Club — you don’t need to have a background in coding and technology to get involved.

Two teachers give their tips on setting up a Code Club

“If you’re thinking of starting a Code Club, do it. The Code Club team has everything created — all the resources are available.” – Peter Hutyan, Code Club mentor, The Mease Spencer Academy, UK. 

Peter, a primary school teacher, has been running a Code Club for seven years at The Mease Spencer Academy in Derby, UK.

A man smiling.

“When I started Code Club, I didn’t know how to do Scratch”, says Peter about the block-based coding tool that lots of young learners start with. “So I made myself learn how to use Scratch and I was two to three lessons ahead of the kids. Basically I’ve done what the kids will do — I’ve read the instructions and created the code. It’s such a phenomenal experience of enriching young people’s lives, and you see the smile and sense of accomplishment on their faces. It’s the best day ever.”

Rachael Coultart is the Computer Subject Leader at St Nicholas Primary School, Stevenage, UK and has run a Code Club for over ten years.

A woman laughing.

“I think the benefits of running a Code Club for our school are that it really develops children’s collaboration skills, really develops their perseverance, their thinking skills, their independence”, Rachael says. 

“The key ingredients that I think make a Code Club successful is enthusiasm, from the adults who are there, a kind of passion to learn together and make mistakes together. I kind of think that sets the scene for the children to make mistakes as well. And I think having that freedom outside the classroom is what makes a Code Club really successful.” – Rachael Coultart, Computing Subject Leader, St Nicholas Primary School, UK

Club mentors share why they were inspired to start a club

Saad Koubeissi runs four Code Clubs in libraries in London, UK.

A man smiling.

“Code Club, for me personally, is about providing children with things I didn’t have when I was younger”, says Saad. “I went to youth clubs and stuff but it was never, like, so specific in tech, it was more, like, playing football. So Code Club, I think, by starting off early, it’s introducing children not only to identifying those skills, but also the terminologies, providing that. And being able to be part of that change for those children is welcome for me, to be honest.” 

Bob Bisland is a mentor who started his first Code Club twelve years ago after a career in technology. He now runs three clubs in Malvern, UK.

A man smiling.

“Everyone should set up a Code Club. This is such a rewarding thing to be able to do. I am sharing my passion and my knowledge with another generation of children, and I’m seeing them learn what coding is all about.”

Ready to start? Three steps to set up a Code Club

We provide everything you need to make starting a club straightforward, including hundreds of free coding projects to help you tailor your sessions to the interests of your young tech creators. Our projects are available in up to 40 languages.

  1. Choose your venue. Code Clubs are held in lots of different places, such as schools, libraries and community centres.
  2. Register your club. Complete your application to start your club. We’ll ask for a few details, e.g. the address of your club. 
  3. Start running your club. You can access our free training on how to prepare to run a Code Club. Our Code Club leader guide also includes helpful information for running your club. 

Running a Code Club is a fun and rewarding way to engage young people and help them learn valuable skills for the future. Start your Code Club journey today by finding out more about setting up a club.

The post Begin your Code Club journey: Set up a club today appeared first on Raspberry Pi Foundation.

[$] LWN.net Weekly Edition for July 3, 2025

Post Syndicated from corbet original https://lwn.net/Articles/1026982/

Inside this week’s LWN.net Weekly Edition:

  • Front: Kernel features from Python; i686 in Fedora; Kernel development with LLMs; Rust drivers; Load balancing with machine learning; Transparent huge pages.
  • Briefs: Bcachefs removal; Coccinelle for Rust; Netdev Foundation; Oracle Linux 10; GNU HHIS 5.0; Rust 1.88.0; Quotes; …
  • Announcements: Newsletters, conferences, security updates, patches, and more.

The collective thoughts of the interwebz