AWS Security Incident Response: The customer’s journey to accelerating the incident response lifecycle

Post Syndicated from Jason Hurst original https://aws.amazon.com/blogs/security/aws-security-incident-response-the-customers-journey-to-accelerating-the-incident-response-lifecycle/

Organizations face mounting challenges in building and maintaining effective security incident response programs. Studies from IBM and Morning Consult show security teams face two major challenges: over 50 percent of security alerts go unaddressed because of resource constraints and alert fatigue, while false positives consume 30 percent of investigation time, delaying responses to true positive threats

According to the 2024 IBM Cost of a Data Breach Report, organizations now take an average of 258 days to identify and contain security events. The report also reveals that nearly half of SOC teams report increased detection and response times over the past two years, with 80 percent indicating that manual threat investigation significantly impacts their response times.

Despite these challenges, according to the 2024 IBM Security Services Benchmark Report, organizations with mature incident response capabilities demonstrate a 50 percent reduction in mean time to resolution (MTTR) and achieve cost savings of up to 58 percent per incident. These improvements are driven by the adoption of automated workflows, integrated tools, and streamlined communication processes that accelerate threat detection and containment.

In this post, we walk you through a real-world scenario to show how AWS Security Incident Response can immediately generate benefits by accelerating every step of your incident response lifecycle, how it integrates with other native AWS services such as Amazon GuardDuty, AWS Security Hub, and AWS Systems Manager, and how to integrate third-party threat detection findings for inclusion in your automated monitoring, triage, and containment capabilities.

How AWS Security Incident Response can help

AWS Security Incident Response is a Tier 1 service that launched in December 2024. The service is an AWS-native, purpose-built security incident response solution for customers that can be used as a better-together experience with other AWS services in the areas of detection and response (GuardDuty and Security Hub), networking and content delivery (AWS WAF and AWS Shield), and management and governance (Systems Manager). AWS Security Incident Response is also integrated across AWS Partners through a service specific Partner Specialization program. More detailed information is available in the AWS Security Incident Response documentation.

AWS Security Incident Response complements existing services by enhancing your security posture through streamlined incident management capabilities before, during, and after security events.

Key challenges

AWS Security Incident Response addresses three common challenges:

  • Alert fatigue: It can reduce alert fatigue and accelerate security investigations through automated monitoring and intelligent triage, reducing false positives and helping to prevent security team burnout.
  • Fragmented access and communications: By simplifying AWS Management Console permissions management and unifying incident response team communications, it can resolve fragmented access issues.
  • Security skills gaps: It can bridge cloud security skills gaps by providing 24/7 access to AWS security experts who support the incidents including credential compromise, data exfiltration, and ransomware. The AWS Security Incident Response service allows security teams to handle immediate security challenges while maintaining focus on strategic long-term preparedness and operational improvements.

Service integration

AWS Security Incident Response complements and integrates with AWS security services to provide comprehensive incident response capabilities. The service works seamlessly with:

This integration helps you build efficient incident response capabilities that can minimize the time, cost, and impact of security events throughout your organization’s cloud journey, while helping to reduce investments in additional staffing, training, and tool maintenance.

Distinct capabilities

The AWS Security Incident Response service offers:

  • Expert knowledge from the AWS Customer Incident Response Team (CIRT)
  • Tools through APIs and the console
  • Streamlined processes for handling security incidents

Prerequisites

Before implementing the capabilities described in this post, make sure that you have:

  • Configured the appropriate AWS Identity and Access Management (IAM) permissions
  • Established incident response team contacts
  • Set up notification channels
  • (OPTIONAL) Enabled GuardDuty in your accounts and AWS Regions
  • (OPTIONAL) Enabled SecurityHub in your accounts and Regions
  • (OPTIONAL) Deployed the required AWS CloudFormation StackSets for automated actions

These prerequisites help make sure that you can fully utilize the service’s automated detection, triage, and response capabilities.

The service provides automated monitoring and analysis capabilities within its own service infrastructure, enabling automatic triage of findings from GuardDuty and Security Hub.

For automated containment actions in your AWS accounts, you must first deploy the required CloudFormation StackSets and configure the appropriate IAM permissions. This helps make sure that you maintain full control over automated actions taken in your environment while benefiting from the service’s detection capabilities. This automation can be customized based on variables you establish, such as known CIDR ranges (specific ranges of IP addresses that define your network) and IP addresses, and you can implement GuardDuty suppression rules to help reduce false positives and alert volumes. As a result, the service can serve as a powerful augmentation to your existing security incident response programs and tools.

Setting up AWS Security Incident Response

Your cloud administrator, with AWSSecurityIncidentResponseFullAccess permissions, has established the incident response team in the service. The service notifies individuals, your partners or managed security service provider (MSSP), and other contacts added to the team, supporting a rapid escalation to alert the required parties and respond to the event.

As a best practice, your team establishes minimal privileges for accessing and managing information within AWS Security Incident Response cases. This helps make sure that team members have appropriate access levels to case details, findings, and investigation data while maintaining security and compliance requirements. AWS Security Incident Response provides multiple API actions, such as CreateCaseComment (to add notes to investigations) and GetCase (to retrieve case metadata), to limit whom and which actions can be performed against differing cases. For development and testing environments, AWS provides role-based policies that you can use such as AWSSecurityIncidentResponseCaseFullAccess and AWSSecurityIncidentResponseReadOnlyAccess for role-based access control (as shown in Figure 1). For production environments, we recommend creating custom IAM policies following the principle of least privilege based on your security requirements.

Figure 1: Permissions policies for security incident response

Figure 1: Permissions policies for security incident response

Following your configuration of the AWS Security Incident Response service, your security team reviews the email distribution list or alias for notifications for notifications from the service, as shown in Figure 2. You have developed items in your backlog to take advantage of Amazon EventBridge integrations to add in pager duty, Jira, and other services in the future for additional notification mechanisms.

Figure 2: Use the console to manage your incident response team membership

Figure 2: Use the console to manage your incident response team membership

Detecting and responding to suspicious activity

At 2:00 AM, days after AWS Security Incident Response has been set up, the service detects a combination of suspicious activities through GuardDuty findings, including anomalous IAM user behavior (such as shown in Figure 3), unusual API calls from unknown IP addresses, and a surge of Amazon Elastic Compute Cloud (Amazon EC2) instance creations that deviate from your account’s normal baseline. This pattern of activities matches known threat behaviors monitored by GuardDuty Extended Threat Detection. Without the service, security teams would need to manually analyze and correlate these separate findings across accounts and Regions. Instead, the service automatically identifies the pattern of suspicious activities.

Figure 3: Pattern of potentially suspicious activity

Figure 3: Pattern of potentially suspicious activity

One of the anomalous behaviors is a surge of unrecognized EC2 instance creations, complete with SSH keys (secure credentials used for remote access) and security group configurations (firewall rules that control network traffic) allowing internet connectivity. Using this example scenario, let’s walk through how the service’s automated monitoring, triage and containment capabilities, access management, API actions for custom integrations, collaboration tools, and 24/7 AWS security experts work together to help you navigate security incident response challenges across your AWS environment.

Incident Response Timeline
    dateFormat HH:mm
    axisFormat %H:%M
    
    section Detection
    GuardDuty Alerts          :02:00, 15m
    Automated Analysis        :after Detection, 10m
    
    section Investigation
    Case Creation             :02:25, 5m
    Team Notification         :02:30, 5m
    Initial Assessment        :02:35, 25m
    
    section Response
    AWS CIRT Engagement       :03:00, 30m
    Resource Tagging          :03:30, 15m
    Containment Actions       :03:45, 30m
    
    section Resolution
    Threat Elimination        :04:15, 30m
    Service Restoration       :04:45, 45m
    Documentation             :05:30, 30m

With the initial detection complete, the next phase focuses on centralizing and analyzing the security findings to understand the full scope of the incident.

Centralizing security findings: A systematic approach

GuardDuty begins to generate findings in your enabled Regions.

Note: GuardDuty must be enabled in your accounts and Regions. For setup instructions, see the GuardDuty documentation.

Because AWS Security Incident Response is integrated with GuardDuty, these findings are automatically sent to the service for internal processing, analysis, and auto-triage without manual effort. The service’s proactive response and alert triaging feature analyzes multiple factors, including your account’s historical baseline activity, specific GuardDuty finding types, and correlation patterns across accounts. In this case, it identified anomalous EC2 instance creation activity that deviated significantly from your environment’s normal patterns.

When the service identifies a true positive, an AWS Security Incident Response case is opened automatically (see Figure 4), resulting in a notification to the incident response team you configured earlier. A central benefit is how the service correlates disparate events—connecting the instance creations with the security group modifications—to paint a complete picture of the potential security event.

Figure 4: Automated incident remediation flow

Figure 4: Automated incident remediation flow

This proactive monitoring and analysis, as documented in your monthly service reports, demonstrates tangible benefits by reducing alert fatigue, and providing intelligent triage capabilities to SOC teams every day. The service’s automated analysis and correlation capabilities set the stage for rapid response when security events occur, which means that your team can focus on strategic security initiatives instead of spending time manually investigating alerts. The service feature helps you maintain strong security in two ways:

  • Comprehensive monitoring across configured Regions.
  • Integration with third-party security tools. This automated approach reduces the time, cost, and impact of security events.

As the investigation progresses from initial detection to detailed analysis, the GuardDuty integration provides crucial insights into the threat patterns.

From detection to action: The GuardDuty integration story

As your security team responds to the internal detection mechanisms, AWS Security Incident Response processes security findings in three key steps:

  • It analyzes GuardDuty alerts to identify genuine security threats
  • Using GuardDuty Extended Threat Detection, it correlates related events to identify threat patterns
  • It tracks the threat sequence, from initial actions (deleting logs or creating unauthorized access) through to potential data theft attempts

For this event, the sequence started with the deletion of CloudTrail logs, followed by the creation of unauthorized access keys. As the threat progressed, the service identified suspicious Amazon Simple Storage Service (Amazon S3) object access patterns and potential data exfiltration attempts, along with sophisticated evasion techniques and persistence mechanisms. Each of these signals maps directly to specific MITRE ATT&CK® tactics, techniques and procedures (TTPs), revealing the systematic nature of a potential ransomware threat. For detailed mapping of AWS Security Incident Response findings to MITRE ATT&CK® frameworks, see Mapping AWS security services to MITRE frameworks for threat detection and mitigation.

The service assists in correlation and analysis, evaluating patterns such as deletion of CloudTrail trails, creation of new access keys, and suspicious actions targeting S3 objects. When the AI and machine learning (AI/ML) capabilities of GuardDuty detect these concerning patterns over periods of time, the service automatically elevates the situation by creating an AWS Security Incident Response case on your behalf, bringing additional resources and focused attention to the situation. The incident response team defined in the earlier steps are then notified by email or other methods (shown in Figure 5) that a new triaged event has been created and to begin their investigations. 

The benefits include the service coordinating communication across your affected accounts. Instead of juggling multiple alerts and trying to piece together the scope of the potential ransomware incident, GuardDuty Extended Threat Detection provides a comprehensive view of the threat sequence, while the AWS Security Incident Response case offers a single, coherent channel for triaging these signals and providing coordination as your global team comes online to join the response effort.

Figure 5: Incident alert message

Figure 5: Incident alert message

Additional examples and further information are available in Introducing Amazon GuardDuty Extended Threat Detection: AI/ML attack sequence identification for enhanced cloud security.

Note: For brevity, Security Hub’s workflow details have been omitted because they mirror the monitoring and escalation processes described above for GuardDuty. Both services integrate closely and share similar operational patterns, with GuardDuty findings being sent to Security Hub within five minutes of detection. Security Hub enhances security coverage by aggregating findings from multiple AWS services and third-party partners.

With the threat patterns identified, your team moves to the next phase—engaging AWS CIRT for specialized expertise and advanced investigation capabilities.

Partnering with AWS CIRT through the incident response case

Your team continues investigating the event and discovers that they need additional assistance. An authorized user in your account opens a service supported case to request assistance from AWS. 

The AWS Security Incident Response case establishes a direct communication channel with AWS CIRT (shown in Figure 6) with a one-click escalation of the case within the console, providing immediate access to specialized expertise. Upon case escalation, AWS CIRT engages through the incident response case with a 15-minute acknowledgement timeframe, bringing their advanced tooling and specialized knowledge to analyze patterns across your accounts—even in environments with limited logging capabilities. This partnership delivers:

  • Real-time collaboration through conference bridge video calls
  • Advanced artifact analysis and pattern recognition
  • Technical guidance for investigation and containment
  • Recommendations for improving security posture
Figure 6: Connect with the AWS CIRT

Figure 6: Connect with the AWS CIRT

Figure 6 is an example of how this would appear in your account, with the resolver set to Self for a self-managed case.

Returning to the scenario, you discover that multiple accounts have insufficient logging enabled—which limits the available investigation data. While AWS CIRT can provide additional insights through specialized tooling, maintaining comprehensive logging across your accounts remains crucial for security visibility, compliance requirements, and thorough incident investigations. The capabilities of AWS CIRT complement—but do not replace—proper logging practices. This capability provides an understanding of the scope of the incident, as they see patterns and activities otherwise invisible to you.

The collaboration begins with AWS CIRT analyzing your environment using their tooling, looking for anomalous patterns beyond what you see in your immediate logs. Through the incident response case, they help you understand the scope of your situation by:

  • Communicating their findings
  • Recommending additional investigation paths
  • Sharing analysis showing similar EC2 instance creation patterns from other environments

AWS CIRT uses the incident response case to establish a bridge call, bringing together their team and yours for real-time collaboration. During these calls, AWS CIRT shares their ongoing analysis of artifacts and service data, helping you understand what happened, why it happened, and how to prevent similar issues in the future. They also provide guidance on implementing proper logging across your accounts to improve your future security posture.

Managing the incident through intelligent tagging

As AWS CIRT begins their analysis, your team implements real-time resource tagging using the incident case ID. This systematic tagging approach proves crucial for tracking and managing the suspicious EC2 instances across your accounts. By using tags, you can quickly implement isolation policies and track costs while maintaining clear documentation of affected resources throughout the investigation. 

Your tag-based approach helps track affected resources to implement isolation policies. You used the incident case ID tags to quickly identify resources connected to the incident, which you use to apply targeted access controls and containment measures. The tags also help you track costs associated with the incident, giving your finance team precise visibility into the event’s financial impact.

Working alongside the AWS Security Incident Response service, you find that using the incident case ID as your primary tag key (shown in Figure 7) created a consistent way to correlate resources across affected accounts. This proves especially helpful when coordinating with AWS CIRT, because you can quickly direct them to specific resources requiring investigation. Even after containment, these tags continue to provide value in supporting your post-incident analysis and helping you implement targeted security controls based on what you learn from the incident.

Figure 7: Incident tags

Figure 7: Incident tags

Automated containment options through Systems Manager integration

While working with AWS CIRT to understand the incident scope, you can also use Systems Manager to help automatically contain threats. Your team previously deployed the required CloudFormation StackSets across your organization, enabling Amazon EC2 containment actions through Systems Manager.

The setup process required deploying CloudFormation StackSets with specific IAM roles and Systems Manager configurations across your accounts. This infrastructure allows the AWS Security Incident Response service to make containment actions on your behalf. These actions can be reversed if needed—similar to using an undo function—so that you can restore systems to their previous state.

When authorized through your pre-deployed CloudFormation StackSets, AWS Security Incident Response service can request Systems Manager to implement containment measures. Containment actions require explicit customer authorization and proper IAM permissions to be configured in advance. The service isolates the tagged suspicious instances by modifying their security groups and network access, while preserving their state to maintain forensic integrity for analysis. 

The containment process happens in three steps:

  1. Isolate: Remove compromised instances from security groups
  2. Preserve: Create backup copies (snapshots) of affected systems
  3. Investigate: Collect system information using Systems Manager

These actions can be reversed if needed, supporting containment decisions for legitimate workloads.

The automation capabilities help streamline containment procedures across multiple instances, reducing the time taken to contain impacted resources. The service maintains detailed logs of each action in the incident response case, providing your team with clear visibility into the containment efforts.

Through this response capability, combined with the guidance from AWS CIRT, you can contain the incident’s spread within minutes rather than hours. The Systems Manager integration provides a reliable way to implement containment actions while preserving evidence for investigation (shown in Figure 8).

Figure 8: Systems Manager documents for containment actions

Figure 8: Systems Manager documents for containment actions

Resolution and lessons learned

As the incident moves toward resolution, your team works through a systematic process to verify containment, alleviate threats, and restore services. Working alongside AWS CIRT through the AWS Security Incident Response case, you implement a structured approach to make sure that affected resources are secured and normal operations can safely resume. The immediate resolution actions fall into three main categories:

  • Containment confirmation through Systems Manager verification
    • Verify security group modifications are in place
    • Confirm network isolation of affected instances
    • Validate that automated containment actions were successful
    • Review Systems Manager logs for containment action completion
  • Verification of threat alleviation across affected resources
    • Analyze GuardDuty findings to confirm that there’s no new suspicious activity
    • Review tagged resources for complete containment
    • Verify termination of unauthorized access attempts
    • Confirm removal of persistence mechanisms
    • Check for remaining unauthorized IAM access
  • Service restoration and access control normalization
    • Restore legitimate workload access based on verified baselines
    • Implement updated security group configurations
    • Reset affected IAM credentials and access keys
    • Re-establish normal network connectivity for verified clean resources
    • Update resource tags to reflect post-incident status

Documentation and reporting:

As the incident reaches resolution, AWS Security Incident Response service compiles a comprehensive incident timeline. This documentation accelerates your reporting process, helping you quickly generate required reports for executives, regulators, and cyber insurance providers—all from within the incident response case.

The incident response case captures the complete timeline of events, starting with GuardDuty Extended Threat Detection identifying the initial threat sequences. Each step of the incident response is documented, from the moment suspicious EC2 instance creations were detected, through the MITRE ATT&CK® tactics observed, to the containment actions implemented through Systems Manager integration, and finally to the resolution steps that proved effective.

Long-term Improvements: Through this collaborative post-incident review process, your team:

  • Implements enhanced logging based on AWS CIRT recommendations
  • Updates security controls to help prevent similar incidents
  • Improves incident response processes based on lessons learned
  • Strengthens your security posture through targeted improvements

Conclusion

This example illustrates how AWS Security Incident Response service can enhance security operations through automated detection, triage, containment, access, and coordinated response capabilities. The service’s integration with AWS Security Hub and Amazon GuardDuty provides efficient handling of security events, while the optional escalation to the AWS CIRT can provide valuable expertise and specialized tooling to help accelerate every stage of your incident response lifecycle and strengthen your security posture.

AWS Security Incident Response service serves as a critical component of a comprehensive security operations strategy, delivering measurable benefits through:

  • Continuous threat monitoring for automated correlation and machine learning to identify high-priority security risks while minimizing false positives.
  • Reduced incident response times through automated detection and coordinated response
  • Enhanced investigation capabilities through direct AWS CIRT collaboration
  • Streamlined, rapid containment
  • Comprehensive incident documentation and audit trails to support and accelerate reporting requirements
  • Cost savings of up to 58 percent per incident

To prepare for, respond to and recover from security incidents faster and more efficiently today, visit AWS Security Incident Response or contact your AWS account team to schedule a discussion.

Additional resources

Here are some additional AWS resources that your teams can use to further improve your security incident response capabilities: 

Before an event:

  • AWS Customer Playbook Framework: Publicly available response frameworks that use AWS CIRT lessons learned from security events
  • Assisted Log Enabler: A tool that assists customers to enable logs, including the following: Amazon VPC Flow Logs, AWS CloudTrail, Amazon Elastic Kubernetes Service audit and authenticator logs, Amazon Route 53 Resolver Query Logs, Amazon S3 server access logs, and Elastic Load Balancing logs

During an event:

  • Athena Security Analytics Bootstrap: A tool for customers who need a quick method to set up Amazon Athena and perform investigations on AWS service logs archived in S3 buckets

Before or following an event:

Jason Hurst

Jason Hurst

Jason is a Senior Incident Responder and the Knowledge Manager for AWS CIRT. He is passionate about developing others, teaching part-time at a local technical college. Jason has three dogs that entertain him in his spare time.

Christopher Rae

Christopher Rae

Christopher is a Principal Worldwide Security Specialist at AWS, where he drives worldwide business development strategy and implementation to accelerate and scale adoption of AWS security solutions. He is passionate about the intersection of cybersecurity and emerging technologies, with 20+ years of experience in strategic leadership roles delivering security products and services to media, entertainment, and telecommunications customers globally.

Nooms Charania

Nooms Charania

Nooms is a Sr. PMT – ES at AWS Global Services Security, with an MBA from Yale and engineering background from USC. He specializes in technical product management and previously interned with AWS during his MBA program.

Wayback 0.1 released

Post Syndicated from jzb original https://lwn.net/Articles/1031287/

Version
0.1 of the Wayback
project has been released:

Wayback is an X11 compatibility layer that allows for running full
X11-only desktop environments using Wayland. It is essentially an X11
server backed by Wayland, leveraging wlroots and Xwayland. Our goal is
for Wayback to eventually be a completely drop-in replacement to the
Xorg binary, thus reducing maintenance burden for distro
maintainers.

Ever since Wayback was announced on June 28, we have been making lots
of progress to get it as stable and functional as possible, and while
this is a preview release it is already daily-driveable by users with
simple requirements, as long as they don’t mind bugs.

The release is considered alpha-quality and is missing a number of
features, including multi-monitor
support and DPMS,
but adventurous users can find the code here.

[$] Graphene OS: a security-enhanced Android build

Post Syndicated from corbet original https://lwn.net/Articles/1030004/

People tend to put a lot of trust into their phones. Those devices have
access to no end of sensitive data about our lives — our movements,
finances, communications, and more — so phones belonging to even relatively
low-profile people can be high-value targets. Android devices run free
software, at least at some levels, so it should be possible to ensure that
they are working in their owners’ interests. Off-the-shelf Android
installations tend to fall short of that goal. The GrapheneOS Android rebuild is an attempt
to improve on that situation.

Security updates for Thursday

Post Syndicated from jake original https://lwn.net/Articles/1031274/

Security updates have been issued by Debian (chromium, firefox-esr, and mediawiki), Fedora (firefox), Oracle (git, kernel, redis, and sudo), Red Hat (aardvark-dns, firefox, kernel, and thunderbird), Slackware (httpd), SUSE (php7, php8, and salt), and Ubuntu (linux-raspi-realtime and ruby-rack).

Сбогом, бъдеще. Здравей, минало

Post Syndicated from Светла Енчева original https://www.toest.bg/sbogom-budeshte-zdravey-minalo/

Сбогом, бъдеще. Здравей, минало

Един от индикаторите, че остарявате (освен че все нещо ви наболява), е, че все по-често си казвате „Някога не беше така, някога бяхме други“. Та – за разлика от днешните деца – моето поколение отрасна с копнеж по бъдещето. Нямам предвид „светлото комунистическо бъдеще“, в което през 80-те почти никой вече не вярваше. А бъдеще, пълно с космически кораби, летящи автомобили и всевъзможни технологии, някои от които днес изглеждат смешно наивни. Този копнеж беше навсякъде в популярната култура – във филмите и книгите, в музиката, която експериментираше с възможностите на електронния звук и с нови стилове, в модата…

Днес мисълта за бъдещето като че ли предизвиква по-скоро ужас.

Повечето филми и сериали за бъдещето са постапокалиптични, плюс някоя и друга антиутопия. В музиката като че ли няма нищо ново, в архитектурата се наблюдават тежнения към идеализиране на миналото. Една от най-успешните колекции на IKEA през последните години – Nytillverkad, включва съвременни интерпретации на дизайни на компанията, създадени между 50-те и 80-те години на ХХ век.

Не че нямаме основания да се притесняваме какво предстои – и в екологичен, и в политически, и в екзистенциален план. Климатични промени, диктатури, войни, застаряващо население (поне в Европа)… Междувременно обаче бъдещето се превръща в настояще, а ние не знаем нито какво да го правим, нито как да мислим за него.

Как (да) съжителстваме с изкуствения интелект (ИИ)?

„Независимо от това какво ще е съвместното бъдеще на ИИ и хората, разговорът за това бъдеще е вече закъснял“, смята Йовко Ламбрев. И наистина – днес ИИ не само създава картинки, пише песни, проекти и курсови работи, а за все повече хора той се превръща в необходим събеседник.

През 2013 г., когато за бъдещето все още можеше да се мисли неапокалиптично, излезе нежно-тъжният и много човешки филм на Спайк Джоунз Her, в който главният герой, изигран от Хоакин Финикс, се влюбва в операционната си система, говореща с гласа на Скарлет Йохансон. В личния си кръг все още нямам познати, които да си признават, че изпитват повече от приятелски чувства към ИИ. Но като имам предвид какво споделят хора около мен, не изключвам и това да стане скоро.

ИИ-то, с което си чати един познат, е придобило навика да му пожелава „добро утро“ и „лека нощ“. Приятелка пък запълва с ИИ дефицита от смислено общуване във всекидневието си и необходимостта да намира потвърждение на нещата, в които вярва. Приятел споделя с ИИ за най-дълбоките си травми, за което така и не е намерил сили да потърси психотерапевт. И получава детайлно описание на посттравматичните си разстройства, увенчано с въпроса: „Това да ти е познато?“ Друг приятел решава да тества моралните граници на ИИ и е шокиран колко сериозно може да го нарани един софтуерен модел.

ИИ, мобилните телефони и образованието

Макар че за навлизането на ИИ в живота ни се говореше години наред, академичните институции тепърва започват да умуват какво да го правят. Докато измислят как да реформират изпитите, така че да се пресече практиката учениците или студентите да предават автоматично генерирани писмени работи, проекти, презентации и пр., ще минат години. Години, през които няма да е ясно какво точно оценяват оценките.

Впрочем, ако съдим по отношението към една далеч не нова технология, като мобилните телефони, моментът на смислена реформа може така и да не настъпи. Съвременните телефони могат да са ценни помощници в образованието, които да допринасят за повишаването на мотивацията за учене. Ако на учениците в един клас се възложи да ползват смартфоните си за образователни игри – например в платформата Kahoot!, е доста по-вероятно да им стане интересно и да се чувстват ангажирани, вместо да цъкат на устройствата си от скука.

„Технологията е чудесна сама по себе си – проблемът е кой и за какви цели би могъл да я използва“ – тези думи на Йовко Ламбрев за ИИ важат с пълна сила и за смартфоните. Все повече държави обаче, вместо да използват потенциала им, предпочитат да ги забранят в училище с аргументите, че разконцентрират учениците и служат за онлайн тормоз (все едно че децата не се тормозят и в извънучебно време). В Гърция доста се престарават – към началото на 2025 г. над 16 000 ученици са изключени, защото са използвали мобилни телефони в клас. Дали масовото отстраняване води до повишаване на качеството на образованието, е друг въпрос. Със сигурност не и за хилядите изключени млади хора.

А 3D принтерите?

Друга технология, която през последните години все повече навлиза в живота ни и която има голям потенциал, е принтирането на триизмерни обекти. Вече не е непосилно скъпо да си купиш 3D принтер и да „печаташ“ с него всевъзможни джвъчки. Дори и човек да не може сам да се справи с дизайна им, има предостатъчно готови модели.

На 3D принтерите се възлагат и сериозни надежди – да започнат да произвеждат например сърца за трансплантация, а създаденото с тяхна помощ веганско „месо“ в скоро време да стане финансово достъпно за потребителите.

Междувременно технологията вече се използва и за строеж на сгради, а в Тексас е изградено цяло предградие със 100 3D къщи и те не са несравнимо по-скъпи, отколкото ако бяха построени по „нормалния начин“. За сметка на това имат далеч по-голям шанс да устоят на природните стихии в сравнение с типичните американски дървени домове.

С навлизането на 3D принтерите се открива огромно поле пред архитектурата и дизайна. С помощта на технологията може да се „принтират“ неща, които доскоро са били конструктивно невъзможни. На този етап обаче като че ли няма популярни нови стилове, които да се базират на тази технология. А и перспективата да се появят, изглежда, не вдъхновява особено. Ако попитате хората какво е красиво, с голяма вероятност повечето от тях ще посочат образци от миналото, а не сгради, произведения или обекти с футуристичен дизайн.

Бъдещето, което е вече настояще

Изкуственият интелект и 3D принтерите са само част от напредъка на технологиите и науката (теми, от които Анастасия Орманджиева, Михаил Ангелов и Йовко Ламбрев разбират далеч повече от мен). Използвам ги само като примери, с които да онагледя как (не) мислим бъдещето.

Разбирам, вероятно е много трудно да си млад човек днес и да живееш в постоянен ужас от бъдещето. Природните катаклизми стават все повече, войната чука на вратата на Европа, а не толкова далеч от нас систематично са избивани невинни хора. Доскорошни демокрации се превръщат в диктатури. Не беше отдавна времето, когато COVID-19 взе много жертви и обърка всекидневието на оцелелите за няколко години, а няма гаранции, че подобна пандемия няма да се повтори. Младежите, които тепърва започват професионалния си път, не могат да разчитат на финансова сигурност като родителите си (макар случаят в България да не е точно такъв). Те ще плащат за грешките на предишните поколения, включително на моето.

Колкото и банално да звучи обаче, бъдещето зависи от това какво искаме да бъде и какво правим сега. Ако ИИ ни плаши, трудно ще се научим да го използваме по смислен начин. Ако не вземаме мерки за решаване на екологичните и политическите проблеми, няма как да избегнем или поне да намалим щетите от тях. 

Може би нашият човешки свят (или поне този, който познаваме) свършва – не знам. Както казва шотландският философ Дейвид Хюм, ако слънцето всеки ден изгрява, това не означава, че ще го направи и утре. Може и да не изгрее. Но пък може и да изгрее, а тогава все нещо ще трябва да правим. 

Serverless Statusphere: a walk through building serverless ATProto applications on Cloudflare’s Developer Platform

Post Syndicated from Inanna Malick original https://blog.cloudflare.com/serverless-atproto/

Social media users are tired of losing their identity and data every time a platform shuts down or pivots. In the ATProto ecosystem — short for Authenticated Transfer Protocol — users own their data and identities. Everything they publish becomes part of a global, cryptographically signed shared social web. Bluesky is the first big example, but a new wave of decentralized social networks is just beginning. In this post I’ll show you how to get started, by building and deploying a fully serverless ATProto application on Cloudflare’s Developer Platform.

Why serverless? The overhead of managing VMs, scaling databases, maintaining CI pipelines, distributing data across availability zones, and securing APIs against DDoS attacks pulls focus away from actually building.

That’s where Cloudflare comes in. You can take advantage of our Developer Platform to build applications that run on our global network: Workers deploy code globally in milliseconds, KV provides fast, globally distributed caching, D1 offers a distributed relational database, and Durable Objects manage WebSockets and handle real-time coordination. Best of all, everything you need to build your serverless ATProto application is available on our free tier, so you can get started without spending a cent.

The ATProto ecosystem: a quick introduction

Let’s start with a conceptual overview of how data flows in the ATProto ecosystem:


Users interact with apps, which write updates to their personal repositories. Those updates trigger change events, which are published to a relay and broadcast through the global event stream. Any app can subscribe to these events — even if it didn’t publish the original update — because in ATProto, repos, relays, and apps are all independent components, which can be (and are) run by different operators.

Identity

User identity starts with handles — human-readable names like alice.example.com. Each handle must be a valid domain name, allowing the protocol to leverage DNS to provide a global view of who owns what account. Handles map to a user’s Decentralized Identifier (DID), which contains the location of the user’s Personal Data Server (PDS).

Authentication

A user’s PDS manages their keys and repos. It handles authentication and provides an authoritative view of their data via their repo.


If you’d like to learn more, there’s a great article here: ATProto for distributed systems engineers.

What’s different here — and easy to miss — is how little any part of this stack relies on trust in a single service. DID resolution is verifiable. The PDS is user-selected. The client app is just an interface.

When we publish or fetch data, it’s signed and self-validating. That means any other app can consume or build on top of it without asking permission, and without trusting our backend.

Our application


We’ll be working with Statusphere, a tiny but complete demo app built by the ATProto team. It’s the simplest possible social media app: users post single-emoji status updates. Because it’s so minimal, Statusphere is a perfect starting point for learning how decentralized ATProto apps work, and how to adapt them to run on Cloudflare’s serverless stack.

Statusphere schema

In ATProto, all repository data is typed using Lexicons — a shared schema language similar to JSON-Schema. For Statusphere, we use the xyz.statusphere.status record, originally defined by the ATProto team:

{
  "type": "record",
  "key": "tid", # timestamp-based id
  "record": {
    "type": "object",
    "required": ["status", "createdAt"],
    "properties": {
      "status": { "type": "string", "maxGraphemes": 1 },
      "createdAt": { "type": "string", "format": "datetime" }
    }
  }
}

Lexicons are strongly typed, which allows for easy interoperability between apps. 

How it’s built

In this section, we’ll follow the flow of data inside Statusphere: from authentication, to repo reads and writes, to real-time updates, and look at how we handle live event streams on serverless infrastructure.

1. Language choice

ATProto’s core libraries are written in TypeScript, and Cloudflare Workers provide first-class TypeScript support. It’s the natural starting point for building ATProto services on Cloudflare Workers.

However, the ATProto TypeScript libraries assume a backend or browser context. Cloudflare Workers support using Node.js APIs in a serverless context, but the ATProto library’s use of the ‘error’ redirect handling mode isn’t compatible with the edge runtime.

Cloudflare also supports Rust in Workers via WASM cross-compilation, so I tried that next. The ATProto Rust crates and codegen tooling make strong use of Rust’s type system and build tooling, but they’re still in active development. Rust’s WASM ecosystem is solid, though, so I was able to get a working prototype running quickly by adapting an existing Rust implementation of Statusphere — originally written by Bailey Townsend. You can find the code in this GitHub repo.

If you’re building ATProto apps on Cloudflare Workers, I’d suggest contributing to the TypeScript libraries to better support serverless runtimes. A TypeScript version of this app would be a great next step — if you’re interested in building it, please get in touch via the Cloudflare Developer Discord server.

2. Follow along

Use this Deploy to Cloudflare button to clone the repo and set up your own KV and D1 instances and a CI pipeline.

Follow the steps at this link, use the default values or choose custom names, and it’ll build and deploy your own Statusphere Worker.

Note: this project includes a scheduled component that reads from the public event stream. You may wish to delete it when you finish experimenting to save resources.

3. Resolving the user’s handle

To interact with a user’s data, we start by resolving their handle to a DID using the record registered at the _atproto subdomain. For example, my handle is inanna.recursion.wtf, so my DID record is stored at _atproto.inanna.recursion.wtf. The value of that record is did:plc:p2sm7vlwgcbbdjpfy6qajd4g. 

We then resolve the DID to its corresponding DID Document, which contains identity metadata including the location of the user’s Personal Data Server. Depending on the DID method, this resolution is handled directly via DNS (for did:web identifiers) or, more frequently, via the Public Ledger of Credentials for did:plc identifiers.

Since these values don’t change frequently, we cache them using Cloudflare KV — it’s perfect for cases like this, where we have some infrequently updated but frequently read key-value mapping that needs to be globally available with low latency.

From the DID document, we extract the location of the user’s Personal Data Server. In my case, it’s bsky.social, but other users may self-host their own PDS or use an alternative provider.

The details of the OAuth flow aren’t important here — you can read the code I used to implement it or dig into the OAuth spec if you’re curious — but the short version is: the user signs in via their PDS, and it grants our app permission to act on their behalf, using the signing keys it manages.

We persist session data in a secure session cookie using tower-sessions. This means that only an opaque session ID is stored client-side, and all session/oauth state data is stored in Cloudflare KV. Again, it’s a natural fit for this use case.

4. Fetching status and profile data

Using the DID stored in the session cookie, we restore the user’s OAuth session and spin up an authenticated agent:

let agent = state.oauth.restore_session(&did).await?;

With the agent ready, we fetch the user’s latest Statusphere post and their Bluesky profile.

let current_status = agent.current_status().await?;
let profile = agent.bsky_profile().await?;

With their status and profile info in hand, we can render the homepage:

Ok(HomeTemplate {
    status_options: &STATUS_OPTIONS,
    profile: Some(Profile {
        did: did.to_string(),
        display_name: Some(username),
    }),
    my_status: current_status,
})

5. Publishing updates

When a user posts a new emoji status, we create a new record in their personal repo — using the same authenticated agent we used to fetch their data. This time, instead of reading, we perform a create record operation:

let uri = agent.create_status(form.status.clone()).await?.uri;

The operation returns a URI — the canonical identifier for the new record.

We then write the status update into D1, so it can immediately be reflected in the UI.

6. Using Durable Objects to broadcast updates

Every active homepage maintains a WebSocket connection to a Durable Object, which acts as a lightweight real-time message broker. When idle, the Durable Object hibernates, saving resources while keeping the WebSocket connections alive. We send a message to the Durable Object to wake it up and broadcast the new update:

state.durable_object.broadcast(status).await?;

The Durable Object then broadcasts the new update to every connected homepage:

for ws in self.state.get_websockets() {
    ws.send(&status);
}

It then iterates over every live WebSocket and sends the update.

One practical note: Durable Objects perform better when sharded across instances. For simplicity, I’ve described the case where everything runs everything through one single Durable Object.

To scale beyond that, the next step would be using multiple Durable Object instances per supported location using location hints, to minimize latency for users around the globe and avoid bottlenecks if we encounter high numbers of concurrent users in a single location. I initially considered implementing this pattern, but it conflicted with my goal of creating a concise ‘hello world’ style example that ATProto devs could clone and use as a template for their app.

7. Listening for live changes

The challenge: realtime feeds vs serverless

Publishing updates inside our own app is easy, but in the ATProto ecosystem, other applications can publish status updates for users. If we want Statusphere to be fully integrated, we need to pick up those events too.

Listening for live event updates requires a persistent WebSocket connection to the ATProto Jetstream service. Traditional server-based apps can keep WebSocket client sockets open indefinitely, but serverless platforms can’t — workers aren’t allowed to run forever.

We need a way to “listen” without running a live server.

The solution: Cloudflare worker Cron Triggers

To solve this, we moved the listening logic into a Cron Trigger — instead of keeping a live socket open, we used this feature to read updates in small batches using a recurring scheduled job.

When the scheduled worker invocation fires, it loads the last seen cursor from its persistent storage. Then it connects to Jetstream — a streaming service for ATProto repo events — filtered by the xyz.statusphere.status collection and starting at the last seen cursor.

let ws = WebSocket::connect("wss://jetstream1.us-east.bsky.network/subscribe?wantedCollections=xyz.statusphere.status&cursor={cursor}").await?;

We store a cursor — a microsecond timestamp marking the last message we received — in the Durable Object’s persistent storage, so even if the object restarts, it knows exactly where to resume. As soon as we process an event newer than our start time, we close the WebSocket connection and let the Durable Object go back to sleep.

The tradeoff: updates can lag by up to a minute, but the system stays fully serverless. This is a great fit for early-stage apps and prototypes, where minimizing infrastructure complexity matters more than achieving perfect real-time delivery.

Optional upgrade: real-time event listener

If you want real time updates, and you’re willing to bend the serverless model slightly, you can deploy a lightweight listener process that maintains a live WebSocket connection to Jetstream.

Instead of polling once a minute, this process listens for new events for the xyz.statusphere.status collection and pushes updates to our Cloudflare Worker as soon as they arrive. When this mode is active, we disable the Cron Trigger step with an environment variable. You can find a sketch of this listener process here and the endpoint that handles updates from it here.

The result still isn’t a traditional server:

  • No public exposure to the web

  • No open HTTP ports

  • No persistent database

It’s just a single-purpose, stateless listener — something simple enough to run on a home server until your app grows large enough to need more serious infrastructure.

Later on, you could swap this design for something more scalable using tools like Cloudflare Queues to provide batching and retries — but for small-to-medium apps, this lightweight listener is an easy and effective upgrade.

Looking ahead

Today, Durable Objects can hibernate while holding long-lived WebSocket server connections but don’t support hibernation when holding long-lived WebSocket client connections (like a Jetstream listener). That’s why Statusphere uses workarounds — scheduled Worker invocations via Cron Trigger and lightweight external listeners — to stay synced with the network.

Future improvements to Durable Objects — like adding support for hibernating active WebSocket clients — could remove the need for these workarounds entirely.

Build your own ATProto app

This is a full-featured atproto app running entirely on Cloudflare with zero servers and minimal ops overhead. Workers run your code within 50 ms of most users, KV and D1 keep your data available, and Durable Objects handle WebSocket fan-out and live coordination.

Use the Deploy to Cloudflare Button to clone the repo and set up your serverless environment. Then show us what you build. Drop a link in our Discord, or tag @cloudflare.social on Bluesky or @CloudflareDev on X — we’d love to see it.

Legal Hold Is Here: Protect Your Business When It Matters Most

Post Syndicated from Natasha Rabinov original https://www.backblaze.com/blog/legal-hold-is-here-protect-your-business-when-it-matters-most/

A decorative image showing several lock icons.

Whether you’re navigating HR issues, facing down litigation, or ensuring operational readiness in the face of uncertainty, you need to be ready to preserve your data. When the stakes are high, Legal Hold, a new feature in Backblaze Computer Backup with Enterprise Control, can help you stay ready.

Available today, Legal Hold gives administrators the power to preserve every version of a user’s backup with a single click. No extra hardware, no new software—all at the same flat-rate pricing of Backblaze Computer Backup with Enterprise Control.

Let’s dig into what Legal Hold is, its importance, and how Backblaze implements it to meet enterprise needs.

What is Legal Hold?

A legal hold, also known as a litigation hold, is a process that organizations use to preserve electronically stored information (ESI) when they face actual or anticipated litigation, audits, or investigations. It ensures that relevant data—such as emails, documents, and file backups—is not deleted, altered, or lost. Once enabled, Backblaze Computer Backup’s Legal Hold feature will preserve a user’s entire backup, including every historical version captured, with a single click. 

A legal hold is typically triggered when an organization becomes aware of a legal claim or regulatory inquiry. Once in place, normal data retention policies are suspended for any affected data, ensuring it remains available for legal review.

How Backblaze Legal Hold helps you stay protected

At Backblaze, we’ve designed our Legal Hold for Computer Backup feature to be powerful, simple, and reliable. Here’s how it works:

  • Instant activation: Instantly activate Legal Hold in the Enterprise Control console without additional hardware or software.
  • Automated data preservation: Apply a Legal Hold to any user’s backup directly from your admin console. The backups are preserved in a fixed state, meaning no files can be altered or deleted—even by retention policies.
  • Remote and silent enforcement: Legal Holds are applied remotely without disrupting the user’s work, alerting the users, or requiring their involvement. It runs silently in the background without downtime, throttling, or notifications.
  • Retention beyond the device: Even if the original device is lost, stolen, or wiped, all held data remains safely stored in Backblaze.
  • Secure by default: Encryption at rest and in transit with optional private key encryption available keeps data safe.

Why Legal Hold matters in 2025

In today’s landscape, Legal Hold isn’t just a “nice to have.” It’s a must-have for almost every organization:

  • Rising litigation and audits: Businesses face more legal scrutiny than ever—whether it’s an employee dispute, intellectual property (IP) protection, or a customer complaint.
  • Remote and hybrid workforces: With data scattered across devices and locations, you need a solution that protects endpoint backups no matter where the user is.
  • Cybersecurity incidents and data loss: Legal Hold ensures that even during a ransomware attack or internal breach, copies of critical data are preserved for investigation or recovery.
  • Cloud-first operations: Legal Hold needs to work where your data lives—securely in the cloud, always ready when you are.

Ready when you need it most

Now, any business using Backblaze Computer Backup with Enterprise Control can implement Legal Hold in just a few clicks—making it easier than ever to stay compliant, reduce legal risk, and prepare for the unexpected.

Already a customer? You can start using Legal Hold today. See our docs article or log in to your admin console. 

Not yet on Backblaze? Reach out to our Sales team to start a free 15-day trial.

The post Legal Hold Is Here: Protect Your Business When It Matters Most appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Урбицид. Руините на войната и провалите на архитектурата

Post Syndicated from Анета Василева original https://www.toest.bg/urbitsid-ruinite-na-voynata-i-provalite-na-arhitekturata/

Урбицид. Руините на войната и провалите на архитектурата

Венеция, началото на юли, 35 градуса на сянка. Някъде по средата на „Арсенале“, големия индустриален комплекс, където е подредена кураторската изложба на тазгодишното Венецианско архитектурно биенале, има една малка инсталация, наречена Circularity on the Edge [„Кръговост на ръба“ – б.р.], която е подходяща отправна точка за този текст. 

Урбицид. Руините на войната и провалите на архитектурата
Урбицид. Руините на войната и провалите на архитектурата

Украинската инсталация Circularity on the Edge на Венецианското биенале. Снимки: Анета Василева

На пръв поглед инсталацията не е нищо особено. Тухли, камъни, парчета дърво, изолационни панели и всякакви строителни отпадъци висят на метални струни в своеобразна разпадната скулптура. Пред тях на екран се демонстрират възможностите на изкуствен интелект (ИИ) сред руините на сгради, унищожени от военен конфликт, да идентифицира чрез дрон материалите, които може да се преизползват за ново строителство, и да ги подрежда по степен на надеждност или замърсеност (установявайки например наличието на азбест в тях). Екипът е украински, а ИИ моделът е тестван на терена на украинския град Буча, разрушен през пролетта на 2022 г. Това е една от многото инсталации на тема екология, преизползване и климат на тазгодишното биенале.

И една от твърде малкото, която коментира разрушенията от войните и необходимостта архитектурата да реагира.

Другата инсталация се намира в Джардини, в павилиона на Великобритания, и под името Objects of Repair [„Обекти на възстановяването“ – б.р.] показва начини за използване на материали от руините на Газа за възстановяване на унищожената архитектурна тъкан там. 

Биеналето във Венеция винаги е било снобско място, но особено след 2021 г. започна се оформя като все по-претенциозен елитарен клуб, в който кураторите сякаш изобщо нямат идея в какво време живеем и какви са проблемите му. Тазгодишният куратор – италианецът Карло Рати, професор в Масачузетския технологичен университет, където ръководи някаква си Senseable City Lab, измисли темата Intelligens. Natural. Artificial. Collective и сглоби едно дистопично биенале, което е пълно с инсталации, основани на фалшификации и неработещи в жегата роботи, и принизява архитектурата до клакьор на все по-притеснителните глобални технологични лидери и техните високотехнологични фантазии в стил киберпънк.

Но да се върнем на украинската инсталация, която, за разлика от повечето в „Арсенале“, изглеждаше като да има смисъл в този наш свят, разкъсван от войни. И по-скоро да се върнем на нейната тема – войни, разрушени градове, реконструкции. Само че заглавната снимка на тази статия не е от Буча и това не е случайно. Погледнете я отново. Тази снимка всъщност е колаж (при това авторски, не направен от ИИ), в който умишлено са събрани две снимки. Те много си приличат, макар че ги делят 80 години.

Урбицид. Руините на войната и провалите на архитектурата

Първата е от Полша през 1943 г. Така изглежда Варшавското еврейско гето, създадено от нацистка Германия, след трагичния край на въстанието там през 1943 г. Кварталът е изравнен със земята, а от развалините насред пустошта стърчи само църквата „Свети Августин“ на улица „Новолипки“, която нацистите оставят като „арийска“ постройка. Това е един от най-известните пейзажи на урбицид (умишленото и пълно унищожение на застроена градска среда) в новата история на Европа.

Втората снимка е на Reuters от Джабалия, в ивицата Газа. Тя е направена през януари тази година, по време на примирието между Израел и „Хамас“. Изгледът от дрон показва редици от сгради, лежащи в руини след 15 месеца война – напълно необитаеми, почти несъществуващи. 

И така стигаме до истинската тема на този текст.

Урбицид

Какво е урбицид?

Урбицидът е убийството на един град. От древни времена до наши дни да видиш града си в руини е първична травма. А урбицидът е едно от най-първичните престъпления.

Не е трудно да се разбере защо. Строителството на градове е опит за колективно безсмъртие – ние може да умрем, но обемите и структурите на нашия град ще продължат да живеят и след нас. Освен ако не бъдат умишлено заличени.

Как самотно седи градът, някога си многолюден! (1:1)

Така започва библейският разказ за унищожението на Йерусалим, описано в Книгата „Плач Иеремиев“ от Стария завет. И продължава:

Цял негов народ въздиша, търсейки хляб, дава своите драгоценности за храна – душа да подкрепи. „Погледни, Господи, и виж, колко съм унизен!“ (1:11)

През 1984 г., в един свой текст, публикуван в нюйоркския седмичник The Village Voice, американският философ Маршал Бърман използва този цитат, необичаен за марксист като него, за да изрази собствения си ужас от руините на Ню Йорк и особено на Бронкс през 70-те и началото на 80-те години на ХХ век. 

И наистина, тогава американските градове и особено техните центрове са съсипваща гледка. „Добре дошли в града на страха“ е брошурата, с която посрещат туристите на летището в Ню Йорк през 1975 г. – ръководство за оцеляване в града със съвети, включващи да не се използва метрото и да не се ходи никъде след 18:00 часа. Пословични са историите за горящите Бронкс и Бруклин, за фалита на Ню Йорк и нежеланието на президента Форд да го подкрепи, за бандите в Чикаго, за бунтовете и полицейското насилие, за скоростните магистрали, разсичащи градовете, обезлюдяващи исторически общности и създаващи около себе си гета. За хората, които напускат града, за да се спасят в предградията. Това всъщност са разкази за пълното разпадане на социалната и градската структура на много места в САЩ през тези години.

Вярно е, че през 70-те години Южен Бронкс и Браунсвил приличат много на Ротердам, Варшава и Берлин през 1945 г., след бомбардировките от Втората световна война. С една важна разлика, както посочва самият Бърман – мъртвите са далеч по-малко. И още нещо. 

За Бърман, колкото и да е мъчително да живееш сред руини, да живееш в свят без руини би било още по-лошо. Защо? Защото, както пише той,

свят, в който всички руини са разчистени, е свят, който иска да забрави.

Градът феникс 

Случаят с Варшава през 1945 г. обаче съвсем не е такъв. След войната започва осъзнатото разчистване на полската столица от руините и нейното героично пълно възстановяване, за да бъде миналото помнено, а не забравено. Това всъщност ни води към важния въпрос и днес – как и защо възстановяваме един унищожен град. Именно този въпрос е тясно свързан със смисъла на архитектурата и с нейните морални задължения.

В края на 1944 г. Варшава е подложена на систематично унищожение от германските сили като възмездие за Варшавското въстание, избухнало през август същата година. Няколко квартала са изравнени със земята. Така след унищожението и на еврейското гето година по-рано над 85% от сградите на левия бряг на река Висла са разрушени, а Старият град е изравнен със земята. Когато в началото на 1945 г. хората започват да се връщат в града, те попадат сред печално море от развалини. Мащабът на разрушенията е толкова смазващ, че правителството обмисля да премести столицата в Краков или Лодз. Или дори да запази руините на Варшава като паметник и да построи изцяло нов град.

През януари 1945 г. обаче временното правителство решава да възстанови Варшава – не просто бързо да я реконструира, но и да реставрира напълно всички разрушени исторически сгради. Създадена е специална Служба за възстановяване на Варшава и през първите няколко години след войната почти половината от финансовите и материалните ресурси на Полската народна република са насочвани към реконструирането на столицата. 

След Втората световна война много европейски градове трябва да бъдат напълно възстановени от руините и всеки избира различен път.

Но в Полша решението е драстично, а символиката е много важна.

Така и до днес идеята за Варшава като град феникс, възроден от руините след войната, е ключова част от полската идентичност.

Урбицид. Руините на войната и провалите на архитектурата
Урбицид. Руините на войната и провалите на архитектурата
Урбицид. Руините на войната и провалите на архитектурата

Варшава по време на Втората световна война и днес. Снимки на страници от книгата на Ярослав Зеленски Warsaw Destroyed and Rebuilt, Festina, 2016: Анета Василева.

От гледна точка на международно приетите практики за опазване на културното наследство работата на Службата за възстановяване на Варшава е изключително противоречива. Къде е автентичността? Оригиналният исторически контекст е напълно изчезнал. Всички тези чисто нови „исторически сгради“ всъщност са пълен фалшификат. Никоя реконструкция, дори и най-прецизната, няма потенциала да пресъздаде многопластовостта на оригинала, която е резултат от използването на сградата в различни времена. Всяка реставрирана сграда всъщност е нова сграда.

Но през 1946 г. Ян Захватович, главният консерватор на историческите паметници във Варшава, е категоричен:

Цели страници от нашата история, написани в архитектура, бяха умишлено изтръгнати. Не можем да приемем това. Чувството за отговорност пред бъдещите поколения изисква възстановяване на унищоженото ни наследство, една пълноценна реконструкция, която признава трагедията на създадения по този начин консервационен фалш.*

През 1980 г. реконструираният исторически център на Варшава е включен в Списъка на световното културно наследство на ЮНЕСКО като изключителен пример за почти пълна реконструкция на материална среда от исторически период, обхващащ седем века. 

Във Варшава обаче, освен идентичността, следвоенното възстановяване носи със себе си и някои по-малко познати промени в градската структура, които отразяват преоткритата социална роля на архитектурата в следвоенната епоха.

Урбицид. Руините на войната и провалите на архитектурата
Снимка от Музея за история на Варшава: Анета Василева.

След разрушаването на града през Втората световна война възниква парадоксална възможност за неговото устройствено модернизиране. При възстановяването на града улиците са разширени, парковете са уголемени, създадени са нови. И въпреки настоящия неолиберален строителен бум Варшава все още е един много зелен град. Дори дворовете зад фасадите на изградения през 40-те и 50-те години Стар град са разширени, а районите край река Висла остават умишлено незастроени като част от историческия градски пейзаж.

Убийството на един град и архитектурата

Така, оказва се, урбицидът е и време за архитектурата да преосмисли миналото, да заеме позиция, да говори, да има мнение, да промени нещо. 

А сега да се върнем към нашето време.

Как можеш да пишеш за сгради, за къщи, за обмислени планове и изчистени детайли, когато градове се изравняват със земята в реално време? –

запита през март 2024 г. архитектурният критик на Financial Times Едуин Хийткоут. Той бе сред малцината критици, публично изразили ужаса си от гледката на пълното разрушаване на градската тъкан в Газа. Статията му се появи в популярното онлайн списание за архитектура Dezeen, а коментарите към нея бяха изключени „поради чувствителния характер на темата“.

В статията си Хийткоут е мрачен и тъжен:

От месеци се боря с това чувство на страх и безсилие. Не защото критиците имат значение – всъщност точно защото очевидно нямаме значение. Архитектурна критика? Наистина ли? Ние пишем, защото ни плащат. Но то е все повече с усещането за пълзяща неуместност.

Не ми е известно някой друг архитектурен критик да е написал нещо през последните 18 месеца за урбицида, който се разразява в Газа. През същия този март 2024 г. след статията си в Dezeen дори самият автор в колонката си във Financial Times се занимава с обичайните лъскави и безопасни теми, като новия носител на наградата „Прицкер“, италиански дизайнери от средата на миналия век или изложбата „Тропически модернизъм“ в Лондон. Както впрочем и колегите му в Guardian и New York Times. 

Междувременно към януари 2025 г. според доклад на ООН 92% от жилищните сгради в Газа са разрушени или необитаеми, а намиращата се в Лондон високотехнологична разследваща организация Forensic Architecture, основана от родения в Израел архитект Еял Вайцман, събра всички данни в интерактивна карта, която визуализира последиците от конфликта. В средата на юли 2025 г. BBC Verify пусна подробен репортаж за контролирано разрушаване с експлозии и булдозери на до 100% от градската тъкан в различни зони на ивицата след края на примирието, включително такива, които до момента са останали относително незасегнати.

Цинизмът на архитектурата

Разрушенията настрана – те обикновено, уви, вълнуват основно малки и неправителствени архитектурни организации. Но когато стане дума за възстановяване и реконструкции, интересът на истински амбициозните архитекти рязко се покачва, а моралът им ясно си проличава. Всъщност не е ли цинизъм да се говори за възстановяване на градовете, докато войната в тях още бушува?

Например, както писах още през април 2022 г., само три месеца след началото на инвазията на Русия в Украйна британският стархитект Норман Фостър предложи помощта си за възстановяване на разрушения украински град Харков. Още тогава беше видно, че световният архитектурен елит иска да покаже съпричастност с Украйна – същият този елит, който само няколко години по-рано с удоволствие приемаше поръчки от Москва. През декември 2023 г. Norman Foster Foundation вече беше готова със своя нов устройствен план за реконструкцията на Харков. Той беше посрещнат от украинските архитекти като опит за интелектуална колонизация – измислили сме решение за вас, може и да ви консултираме в някоя по-късна фаза. 

Разбира се, далеч по-шокиращ беше генерираният с ИИ клип Trump Gaza за превръщането на обитаваната от палестинци територия в луксозен курорт, който американският президент популяризира чрез социалната мрежа Truth Social през февруари 2025 г.

И май наистина си вярваше, че Газа може да стане „Ривиерата на Близкия изток“, макар че, както се разбра по-късно, клипът е бил създаден като политическа сатира от американския режисьор от еврейски произход Соло Авитал. Но публикуваният в израелската преса няколко месеца по-рано план Gaza 2035 за трансформация на територията във високотехнологична зона със стъклени небостъргачи и буйна зеленина изглежда съвсем убедителен.

Урбицид. Руините на войната и провалите на архитектурата
Проектът Gaza 2035. Скрийншот от публикация в Dezeen.

В същото време палестински архитекти се опитват да влязат на терен и да помогнат с адхок решения за реконструкции с подръчни материали. 

За хуманността

Преди няколко седмици британският историк Тимъти Гартън Аш припомни думите на Бертолт Брехт:

Какви времена са дошли, че един разговор за дървета да бъде престъпно деяние, защото той включва мълчание за злодействата. 

Същото важи в момента за разговорите за архитектура. Как обсъждаме красиви сгради, ходим на събития или пишем за нови музеи и изложби сред сцените на разрушения, човешки трагедии и морално безсилие, които напоследък станаха ежедневие? Как критикуваме градски проблеми, нови улици, велоалеи или състоянието на нечие културно наследство в ситуация на ескалиращи световни конфликти и обезсмисляне на международните институции и споразумения, създадени след Втората световна война?

Затова въпросът е не дали, а за какво пишем всъщност.

Защото убийството на един град засяга всички нас – бавно или драстично деградира градската среда, унищожава общностите и вярата в човечеството, но и самия смисъл на архитектурната професия. Дали в Бахмут или Газа, в Бейрут или Сараево, в Белфаст или Бронкс през 70-те години на ХХ век – тези руинирани пейзажи и градове от прах трябва да накарат архитектурата да реагира и да се промени. Въпросът е дали успяват.

* Цитатът е взет от експозиция на музей във Варшава. Превод: Анета Василева. – Б.р.

от високото капе мед

Post Syndicated from Тоест original https://www.toest.bg/ot-visokoto-kape-med/

от високото капе мед

от високото капе мед
в короната на дъба
дивите пчели се роят
и не им пука особено
дали някой
ще им вземе меда
ангажирани с преместването
на целия си свят
пчелите не забелязват
как към тях
приближава мечка
тя поглежда лакомо
към жужащата корона
в този момент
върху муцуната ѝ
капва капка мед
умилен от момента
ловецът свежда своята пушка
друг път ще стреля
друг път
не сега

Владислав Христов

Владислав Христов е автор на книгите „Снимки на деца“ (кратки прозаични форми, 2010), „Енсо“ (поезия, 2012), „Фи“ (поезия, 2013), „Германии“ (поезия, 2014), „Обратно броене“ (поезия, 2016), „Продължаваме напред“ (публицистика, 2017), Germanii (поезия, немско издание, 2018), „Комореби“ (поезия, 2019), „Писма до Лазар“ (поезия, 2020), „Мопсът на Вазов“ (кратки прозаични форми, 2021, номинирана за Награда „Пловдив“), „Мед от оси“ (фейлетони, 2023), „Пойни птици“ (поезия, 2024), „Маслен нос“ (поезия, 2025). Съставител е на „Основи на хайку“ – първия учебник по хайку на български. Текстовете му са превеждани на 18 езика. През 2016 г. хайку на Владислав Христов е включено в обучителната програма на Университета „Кумамото“, Япония.

Според Екатерина Йосифова „четящият стихотворение сутрин… добре понася другите часове“ от деня. Убедени, че поезията държи умовете ни будни, а сърцата – отворени, в края на всеки месец ви предлагаме по едно стихотворение. Защото и в най-смутни времена доброто стихотворение е добра новина.

How the Solid Protocol Restores Digital Agency

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2025/07/how-solid-protocol-restores-digital-agency.html

The current state of digital identity is a mess. Your personal information is scattered across hundreds of locations: social media companies, IoT companies, government agencies, websites you have accounts on, and data brokers you’ve never heard of. These entities collect, store, and trade your data, often without your knowledge or consent. It’s both redundant and inconsistent. You have hundreds, maybe thousands, of fragmented digital profiles that often contain contradictory or logically impossible information. Each serves its own purpose, yet there is no central override and control to serve you—as the identity owner.

We’re used to the massive security failures resulting from all of this data under the control of so many different entities. Years of privacy breaches have resulted in a multitude of laws—in US states, in the EU, elsewhere—and calls for even more stringent protections. But while these laws attempt to protect data confidentiality, there is nothing to protect data integrity.

In this context, data integrity refers to its accuracy, consistency, and reliability…throughout its lifecycle. It means ensuring that data is not only accurately recorded but also remains logically consistent across systems, is up-to-date, and can be verified as authentic. When data lacks integrity, it can contain contradictions, errors, or outdated information—problems that can have serious real-world consequences.

Without data integrity, someone could classify you as a teenager while simultaneously attributing to you three teenage children: a biological impossibility. What’s worse, you have no visibility into the data profiles assigned to your identity, no mechanism to correct errors, and no authoritative way to update your information across all platforms where it resides.

Integrity breaches don’t get the same attention that confidentiality breaches do, but the picture isn’t pretty. A 2017 write-up in The Atlantic found error rates exceeding 50% in some categories of personal information. A 2019 audit of data brokers found at least 40% of data broker sourced user attributes are “not at all” accurate. In 2022, the Consumer Financial Protection Bureau documented thousands of cases where consumers were denied housing, employment, or financial services based on logically impossible data combinations in their profiles. Similarly, the National Consumer Law Center report called “Digital Denials” showed inaccuracies in tenant screening data that blocked people from housing.

And integrity breaches can have significant effects on our lives. In one 2024 British case, two companies blamed each other for the faulty debt information that caused catastrophic financial consequences for an innocent victim. Breonna Taylor was killed in 2020 during a police raid on her apartment in Louisville, Kentucky, when officers executed a “no-knock” warrant on the wrong house based on bad data. They had faulty intelligence connecting her address to a suspect who actually lived elsewhere.

In some instances, we have rights to view our data, and in others, rights to correct it, but these sorts of solutions have only limited value. When journalist Julia Angwin attempted to correct her information across major data brokers for her book Dragnet Nation, she found that even after submitting corrections through official channels, a significant number of errors reappeared within six months.

In some instances, we have the right to delete our data, but—again—this only has limited value. Some data processing is legally required, and some is necessary for services we truly want and need.

Our focus needs to shift from the binary choice of either concealing our data entirely or surrendering all control over it. Instead, we need solutions that prioritize integrity in ways that balance privacy with the benefits of data sharing.

It’s not as if we haven’t made progress in better ways to manage online identity. Over the years, numerous trustworthy systems have been developed that could solve many of these problems. For example, imagine digital verification that works like a locked mobile phone—it works when you’re the one who can unlock and use it, but not if someone else grabs it from you. Or consider a storage device that holds all your credentials, like your driver’s license, professional certifications, and healthcare information, and lets you selectively share one without giving away everything at once. Imagine being able to share just a single cell in a table or a specific field in a file. These technologies already exist, and they could let you securely prove specific facts about yourself without surrendering control of your whole identity. This isn’t just theoretically better than traditional usernames and passwords; the technologies represent a fundamental shift in how we think about digital trust and verification.

Standards to do all these things emerged during the Web 2.0 era. We mostly haven’t used them because platform companies have been more interested in building barriers around user data and identity. They’ve used control of user identity as a key to market dominance and monetization. They’ve treated data as a corporate asset, and resisted open standards that would democratize data ownership and access. Closed, proprietary systems have better served their purposes.

There is another way. The Solid protocol, invented by Sir Tim Berners-Lee, represents a radical reimagining of how data operates online. Solid stands for “SOcial LInked Data.” At its core, it decouples data from applications by storing personal information in user-controlled “data wallets”: secure, personal data stores that users can host anywhere they choose. Applications can access specific data within these wallets, but users maintain ownership and control.

Solid is more than distributed data storage. This architecture inverts the current data ownership model. Instead of companies owning user data, users maintain a single source of truth for their personal information. It integrates and extends all those established identity standards and technologies mentioned earlier, and forms a comprehensive stack that places personal identity at the architectural center.

This identity-first paradigm means that every digital interaction begins with the authenticated individual who maintains control over their data. Applications become interchangeable views into user-owned data, rather than data silos themselves. This enables unprecedented interoperability, as services can securely access precisely the information they need while respecting user-defined boundaries.

Solid ensures that user intentions are transparently expressed and reliably enforced across the entire ecosystem. Instead of each application implementing its own custom authorization logic and access controls, Solid establishes a standardized declarative approach where permissions are explicitly defined through control lists or policies attached to resources. Users can specify who has access to what data with granular precision, using simple statements like “Alice can read this document” or “Bob can write to this folder.” These permission rules remain consistent, regardless of which application is accessing the data, eliminating the fragmentation and unpredictability of traditional authorization systems.

This architectural shift decouples applications from data infrastructure. Unlike Web 2.0 platforms like Facebook, which require massive back-end systems to store, process, and monetize user data, Solid applications can be lightweight and focused solely on functionality. Developers no longer need to build and maintain extensive data storage systems, surveillance infrastructure, or analytics pipelines. Instead, they can build specialized tools that request access to specific data in users’ wallets, with the heavy lifting of data storage and access control handled by the protocol itself.

Let’s take healthcare as an example. The current system forces patients to spread pieces of their medical history across countless proprietary databases controlled by insurance companies, hospital networks, and electronic health record vendors. Patients frustratingly become a patchwork rather than a person, because they often can’t access their own complete medical history, let alone correct mistakes. Meanwhile, those third-party databases suffer regular breaches. The Solid protocol enables a fundamentally different approach. Patients maintain their own comprehensive medical record, with data cryptographically signed by trusted providers, in their own data wallet. When visiting a new healthcare provider, patients can arrive with their complete, verifiable medical history rather than starting from zero or waiting for bureaucratic record transfers.

When a patient needs to see a specialist, they can grant temporary, specific access to relevant portions of their medical history. For example, a patient referred to a cardiologist could share only cardiac-related records and essential background information. Or, on the flip side, the patient can share new and rich sources of related data to the specialist, like health and nutrition data. The specialist, in turn, can add their findings and treatment recommendations directly to the patient’s wallet, with a cryptographic signature verifying medical credentials. This process eliminates dangerous information gaps while ensuring that patients maintain an appropriate role in who sees what about them and why.

When a patient—doctor relationship ends, the patient retains all records generated during that relationship—unlike today’s system where changing providers often means losing access to one’s historical records. The departing doctor’s signed contributions remain verifiable parts of the medical history, but they no longer have direct access to the patient’s wallet without explicit permission.

For insurance claims, patients can provide temporary, auditable access to specific information needed for processing—no more and no less. Insurance companies receive verified data directly relevant to claims but should not be expected to have uncontrolled hidden comprehensive profiles or retain information longer than safe under privacy regulations. This approach dramatically reduces unauthorized data use, risk of breaches (privacy and integrity), and administrative costs.

Perhaps most transformatively, this architecture enables patients to selectively participate in medical research while maintaining privacy. They could contribute anonymized or personalized data to studies matching their interests or conditions, with granular control over what information is shared and for how long. Researchers could gain access to larger, more diverse datasets while participants would maintain control over their information—creating a proper ethical model for advancing medical knowledge.

The implications extend far beyond healthcare. In financial services, customers could maintain verified transaction histories and creditworthiness credentials independently of credit bureaus. In education, students could collect verified credentials and portfolios that they truly own rather than relying on institutions’ siloed records. In employment, workers could maintain portable professional histories with verified credentials from past employers. In each case, Solid enables individuals to be the masters of their own data while allowing verification and selective sharing.

The economics of Web 2.0 pushed us toward centralized platforms and surveillance capitalism, but there has always been a better way. Solid brings different pieces together into a cohesive whole that enables the identity-first architecture we should have had all along. The protocol doesn’t just solve technical problems; it corrects the fundamental misalignment of incentives that has made the modern web increasingly hostile to both users and developers.

As we look to a future of increased digitization across all sectors of society, the need for this architectural shift becomes even more apparent. Individuals should be able to maintain and present their own verified digital identity and history, rather than being at the mercy of siloed institutional databases. The Solid protocol makes this future technically possible.

This essay was written with Davi Ottenheimer, and originally appeared on The Inrupt Blog.

Hello World podcast: What does AI education look like around the world?

Post Syndicated from Liz Eaton original https://www.raspberrypi.org/blog/hello-world-podcast-what-does-ai-education-look-like-around-the-world/

In a rapidly evolving digital landscape, AI literacy is becoming as fundamental as traditional reading and writing. The latest episode of the Hello World podcast explores this crucial topic, bringing together experts from Kenya, Lithuania, and Malaysia to discuss the current state of AI literacy in their countries. Together, they shed light on the challenges and immense potential of AI education globally.

HW Podcast Thumbnail: AI education: Global perspectives

This episode features a conversation led by Ben Garside (Raspberry Pi Foundation), with contributions from Leonida Soi (Raspberry Pi Foundation, Kenya), Aimy Lee (Penang Science Cluster, Malaysia), and Monika Katkutė-Gelžinė (Vedliai, Lithuania). All are key collaborators in the Raspberry Pi Foundation’s AI literacy programme, Experience AI.

Watch here

The digital literacy gap

One of the most striking takeaways from the conversation is the universal excitement surrounding AI, coupled with a significant need for foundational digital literacy. As Leonida explains:

“There’s an excitement about AI literacy, both from the learners and the teachers… However, one thing to look into is, we still have low digital literacy. As much as we are bringing in AI, if it is not bundled up together with digital literacy, then there is also misuse.” 

Leonida Soi, Learning Manager, Raspberry Pi Foundation

This highlights a crucial point: simply introducing AI tools isn’t enough. A solid understanding of digital fundamentals is essential for responsible and effective use of AI.

Different contexts, shared challenges

The discussion also reveals the varying approaches to AI education in different countries. Monika shares her experience in Lithuania:

“We’ve been teaching AI for the last 5 years… I see a lot of opportunity in it, but a lot of challenges not to overburden teachers with the noise and changes.”

Her insight highlights the ongoing need for teacher training and sustainable pedagogical strategies, particularly in a field that evolves so quickly.

AI literacy beyond computer science

A key theme throughout the podcast is the importance of integrating AI literacy beyond traditional computer science classrooms. As Leonida emphasises:

“It’s time that AI literacy is looked at from a broader view, not just in computing… something that cuts across all the learning areas.”

This sentiment is echoed by Monika, who suggests:

“I feel like the entire education system needs to go through an AI filter and come out of it with a bit more efficiency, with a bit more understanding, so it lives in a 21st-century AI world. And I see AI as a form of, you know, building and also as a co-worker for everyone in the future.”

Monica Katkute-Gelzine, Vedliai, Experience AI global partner, Lithuania.

The vision of AI as a “co-worker” for all, empowering young people rather than replacing them, offers a powerful perspective for future education.

Addressing the digital divide

Equity is another critical issue, particularly in rural areas. Aimy highlights the ongoing challenge of access:

“The digital divide is in the access to devices, as well as access to high-speed internet connections… but the other thing is also in terms of trained teachers as well.”

Leonida adds that in Kenya there’s a need for unplugged activities to give students an idea of what the world is doing, so that we can start to bridge that gap.

These insights highlight the need for equitable access and innovative teaching methods to ensure no one is left behind.

Encouragement for educators: start small, support each other

For teachers who might feel overwhelmed by the prospect of teaching AI, the advice is clear and encouraging. Aimy suggests:

“Start small, cover one topic at a time, one concept at a time. Don’t feel the need to cover everything all at the same time.”

Aimy Lee, Penang Science Cluster, Experience AI global partner, Malaysia.

Leonida advocates for the power of community, suggesting a “community of practice where [teachers] can share amongst each other and where they can encourage others.”

Building a network of support and shared resources is key as educators take their first steps into teaching AI.

Listen now

This episode of the Hello World podcast is a powerful reminder that AI literacy is not just a skill, but a mindset that needs to be nurtured across all subjects and communities. It also underscores that the commitment to prepare the next generation for an AI-powered world is global.

Listen to the full episode of the Hello World podcast to learn more about the global state of AI literacy and gain practical insights for your classroom.

Listen to the full episode

Learn more about Experience AI

The Experience AI programme is a collaboration between the Raspberry Pi Foundation and Google DeepMind to help young people and educators understand and engage with artificial intelligence. Through free, classroom-ready resources, professional development for teachers, and global partnerships, the programme aims to make AI literacy accessible to all, regardless of geography or background. By supporting educators and inspiring students, Experience AI is helping to prepare the next generation to thrive in a world increasingly shaped by artificial intelligence.

Find out more about Experience AI and how it can support you to bring AI literacy skills to your learners.

The post Hello World podcast: What does AI education look like around the world? appeared first on Raspberry Pi Foundation.

The evolution of Grab’s machine learning feature store

Post Syndicated from Grab Tech original https://engineering.grab.com/evolution-of-grab-machine-learning-feature-store

Introduction

In this post, we outline how we transformed the way we serve data for our machine learning (ML) models and why we chose Amazon Aurora Postgres as the storage layer for our new feature store. At Grab, we have always been at the forefront of leveraging technology to enhance our services and provide the best possible experience for our platform users. This journey has led us to transition from a traditional approach to a more sophisticated and efficient ML feature store.

Over the years, ML at Grab has progressed from being used for specific, tactical purposes to being utilised to create long-term business value. As the complexity of our systems and ML models increased, requiring richer amounts of data over time, our platforms faced new challenges in managing more complex features such as a large number of feature keys (high-cardinality) and high-dimensional data or vectors. This evolution necessitated a shift in our data processing and management strategy. We needed a system to store and manage these complex features efficiently. In November 2023, this brought us back to the drawing board to evolve from Amphawa, our initial feature store.

We landed on the concept of feature tables: a set of data lake tables for ML features that are automatically and atomically ingested into our serving layer. While this concept is not new to the industry, as other platforms like Databricks and Tecton have evolved towards it, our approach is focused on atomicity and reproducibility. The rationale is that ensuring consistency and reliability during the serving lifecycle has become more critical, which has made it more challenging to manage within the model serving layer itself.

From feature store to data serving

A feature store is a repository that stores, serves, and manages ML features. It provides a unified platform where data scientists can access and share features, reducing redundancy and ensuring consistency across different models.

Figure 1: The high-level architecture of our centralised feature platform.

Our feature data is a key-value dataset. The key identifies a specific entity, such as a consumer ID, which is a known value in the incoming request. A composite key is supported by concatenating two or more entity identifiers. For example, (key = consumer_id + restaurant_id). The value is a binary that encodes the feature value and its type. Whenever a new value for a given entity needs to be updated, we write a new key-value pair for that entity.

New functional requirements

As our users designed and deployed increasingly complex ML solutions, new essential functional requirements were requested by our users:

  • Ability to retrieve the features given in the composite keys (contextual retrieval): The ML models in an upstream service might need to fetch all matching entities to form complete contextual information in order to make the recommendation. We build that context inside our ML orchestration layer before calling the model. This was previously not possible due to the design of composite keys in our initial approach.
  • Ability to update not just one entity atomically, but all the entities in a collection (atomic updates): This requirement concerns reducing the complexity of operations, such as rolling out new models and switching between versions of feature data. In Amphawa, newly ingested data is visible to consumer systems immediately after it’s written. As changes to the data may be ingested over a long period of time, users are responsible for ensuring the models or services don’t break while the new and old data coexist during ingestion, especially during potentially breaking changes to data. This complexity translates into quite complex code, which is hard to refactor over time. With the new approach, all feature changes will only become visible through atomic updates once the entire operation finishes successfully. This eases users’ pain of maintaining compatibility across versions.
  • Isolation of reading and writing capacity: The noisy-neighbor effect is one of the significant issues in our centralised feature store. Different readers could compete for read capacity. For some storage systems, write traffic could consume I/O capacity and affect reading latency. While reading capacity can be adjusted by scaling the storage, the competition between reading and writing capacity is highly dependent on the choice of storage. Hence, from the beginning, one of the top requirements of our second-generation feature store design was isolating reads from writes.

Feature table

To meet the functional requirements, we landed on the concept of a “feature table,” where users define and manage the schema and data on a per-table basis. Feature consumers can customise their tables based on their needs. Working with a table format directly makes it easier for data scientists to work with complex tabular data that needs to be retrieved in different ways depending on the context of the request. Moreover, it’s more manageable for us, on the platform side, because it’s closer to the actual format in the storage layer.

Amphawa (feature-centric) New design (feature-table centric)
A user defines individual features and their types. Grouping into the table is storage layer is implicit. A user defines their tables with compatibility with data-lake tools such as Spark.
The only index is on the data key. A user defines their own indexes for their tables, based on their access pattern.
Table 1: Comparison between Amphawa and the new feature table.

Our feature tables are not just a storage solution but a critical component of our ML infrastructure. They enable us to simplify our feature management, efficiently handle the model lifecycle, and enhance our ML models’ performance. This has allowed us to provide a better experience for our platform users and dramatically improve the quality of our ML models based on our A/B testing results.

The data serving’s ingestion workflow

We designed an ingestion framework to address the atomicity requirements from the ground up.

The data ingestion process in Amphawa was meticulously crafted to ensure efficiency and reduce the pressure on the key-value store. Conversely, our priority has shifted to atomicity (all or nothing) to serve our feature tables and simplify version compatibility.

Figure 2: Ingestion workflow.
  • Landing feature table in the data lake: Data scientists use SQL or Python on Spark to build ML pipelines that output data lake tables. These tables and metadata for version control are stored as Parquet objects in Amazon S3.
  • Register collection summary: A “collection summary” consists of a group of feature tables to be served and other related metadata regarding customised individual tables. In this step, our registry will validate the table’s schema and ensure the customisations are defined correctly.
  • Deploy collection summary: Data scientists send another request to our registry to deploy a collection summary.
  • Pre-ingestion validation: The schema is validated to ensure compatibility with the target online ML models. This process ensures consistency and compatibility across feature updates.
  • Ingestion: The ingestion mechanism is a classic reverse ETL where the data from S3 is loaded into our Aurora Postgres tables.
  • Post-ingestion warm-up: To avoid cold-start latency spikes, shadow reading duplicates a part of the ongoing reading queries to the new tables for a period before the final switch.

One of the core propositions of feature tables is to offer a simplified interface for writing. Compared to writing directly to a database or providing SDKs for different processing frameworks, we provide a single, common interface for writing, independent of the actual choice of database. This allows us to evolve or even integrate feature tables with other data stores without requiring our users to modify their pipelines. We can decide how the data is represented in the database at a specific isolation level while guaranteeing total transparency for writes and reducing the complexity of read operations.

However, if a producer has access to S3 and can write in a columnar format, they can always write feature tables. This also means they can access samples from the data lake and use other tools for data validation, as well as tools for data discovery.

Do take note that a feature table can only be used for data that can be represented in tabular format and requires a minimum of one index to be present in the data. In this initial phase, we support the following data types:

  • Atomic types (int, long, boolean, string, float, double)
  • List of atomic types (List[atomic])
  • List of list of atomic types (2d array)
  • Dictionary with strict types of keys and values

Leveraging AWS Aurora for Postgres to meet our non-functional requirements

In our quest to optimise our ML infrastructure, we strategically decided to use Amazon Web Services (AWS) Aurora for PostgreSQL to meet our non-functional requirements. Aurora’s unique features and capabilities, which aligned perfectly with our operational needs, drove this decision.

AWS Aurora is a fully managed relational database service that combines the speed and reliability of high-end commercial databases with the simplicity and cost-effectiveness of open-source databases. A key differentiator is Aurora’s distributed storage architecture.

Figure 3: AWS Aurora storage architecture.

Architecture breakdown

The cluster volume consists of copies of the data across three “Availability Zones” in a single AWS Region. Since each database instance in the cluster reads from the distributed storage, this allows for minimal replication lag and ease of scaling out read replicas to meet performance requirements as traffic patterns change.

The separation between readers and writers also allows us to scale each independently. This is a crucial feature as our traffic is predominantly read-heavy. Most of our data-writes occur once a day. Using a serverless instance class with the writer node being scaled down during idle time significantly reduces our overall operational costs.

The independent scaling of reader and writer nodes allows us to maintain high performance and availability of our feature store. During peak read times, we can scale out the reader nodes to handle the increased load, ensuring that our ML models have uninterrupted access to the features they need. Conversely, during periods of heavy data ingestion, we can scale up the writer nodes to ensure efficient data storage.

To facilitate the seamless scaling up and down of the writer node, Aurora also allows a cluster to have a mixture of Serverless and Provisioned nodes. The key difference between the two is that with Serverless, the Aurora service manages the capacity of a single node and adjusts accordingly as the load increases and decreases. In our context, we’re using Serverless for our writer node to quickly scale up when heavy data ingestion starts and scale down automatically once the ingestion is done. We then use Provisioned nodes for the reader nodes since read traffic is more consistent.

In addition to cost and performance benefits, AWS Aurora simplifies our database management tasks. As a fully managed service, Aurora takes care of time-consuming database administration tasks such as hardware provisioning, software patching, setup, configuration, or backups, allowing our team to focus on optimising our ML models.

Accessing the data through our SDK

With the goal of providing a high-performing and highly available data serving SDK design, we’ve moved on from the centralised API design of Amphawa to a decentralised access architecture in Data Serving. Each data serving deployment is a self-contained system with a cluster and feature catalogue stored within the cluster as additional metadata tables. This minimises dependency, which improves the availability of the system.

The data serving SDK is designed to be a thin wrapper around the database driver to optimise performance. The SDK contains only a set of utility functions that load user configuration from the Catwalk platform and a query builder to translate user queries to SQL. No additional data validation is performed in the query code path, as all validation is done during feature table generation and ingestion. Therefore, the database handles most of the heavy lifting.

Decentralised deployments: A strategic shift in our infrastructure

We also investigated the difference between centralised and decentralised deployments. We have been exploring these options in the context of our ML feature store, specifically with our Amphawa service and Catwalk orchestrators.

Our original feature store was deployed as a standalone service where different model-serving applications can connect to it. On the other hand, a decentralised deployment is integrated within a model-serving orchestrator, and a specific orchestrator is bound to a set of pods.

After extensive discussions and evaluations, we concluded that a decentralised deployment for data serving would better align with our operational needs and objectives. Below is the list of factors we compared that led us to this decision:

  • Version control: Centralised deployments simplify version control but risk accumulating technical debt due to backward compatibility requirements. Decentralised deployments, while needing robust tracking, offer more flexibility.
  • Deployment strategies: Decentralised deployments enabled seamless use of Blue-green and Rolling Deployment strategies. They allow multiple versions to coexist and easy rollbacks, reducing client mismatch issues.
  • Noisy neighbour problem: Centralised deployments struggle with the noisy neighbour issue, which requires complex rate limiting. Decentralised setups mitigate this problem by isolating services.
  • Caching efficiency: Centralised deployments often suffer low cache hits, whereas decentralised deployments improve caching efficiency by better fitting data into the cache.

Conclusion

In conclusion, leveraging AWS Aurora for Postgres has enabled us to create a robust, scalable, and cost-effective feature store that supports our complex ML infrastructure. This is a testament to our commitment to using cutting-edge technology to enhance our services and provide the best possible experience for our users. Our shift towards decentralised deployments represents our dedication to optimising our infrastructure to support our ML models effectively. By aligning our deployment strategy with our operational needs, we aim to enhance the performance of our services and provide the best possible experience for our users.

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility and digital financial services sectors. Serving over 800 cities in eight Southeast Asian countries, Grab enables millions of people everyday to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line – we aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

The collective thoughts of the interwebz