TechDocs at Grab: Cultivating a culture of quality documentation

Post Syndicated from Grab Tech original https://engineering.grab.com/techdocs-at-grab-cultivating-a-culture-of-quality-documentation

Introduction

Changing how a company approaches writing and documentation is a complex task. It’s not just about the tools and processes—it’s about shifting the mindset of the people who create and use documentation. Building a strong documentation culture means ensuring everyone takes ownership of producing high-quality content, while making the tools easy to use for everyone involved.

At Grab, our first significant step was adopting the Docs-as-Code approach, which we’ve covered in the blog Embracing a Docs-as-Code approach. This method integrated documentation into the engineering workflow, allowing teams to create and update content effortlessly.

Since then, the TechDocs working group — a collaboration between Tech Learning and the internal development team — has focused not just on improving tools, but on fostering a mindset where documentation is an essential part of everyday work. In this post, let us dive into how we’ve continued to embed high-quality documentation into the core of Grab’s engineering culture.

What is TechDocs?

Helix is Grab’s engineering platform designed to unify infrastructure, tooling, services, and documentation into a single, consistent user interface. It serves as a central hub for managing various engineering tasks and resources within Grab. Helix provides a comprehensive set of guides and tools for users.

TechDocs is an internal documentation platform built on Helix and integrates with our Docs-as-Code approach. It allows engineering teams to create, manage, and access technical content seamlessly within their workflows. TechDocs makes it easier for teams to maintain up-to-date, high-quality documentation with customised features for notification and editing.

How to create a healthy documentation culture

Over a span of 2 to 3 years, the TechDocs team executed these key steps in quarterly chunks to influence Grab’s documentation culture as seen in figure 1.

Figure 1: Key steps in influencing documentation culture in Grab
  1. Take inventory: Assess current internal processes, tools, and user behaviour.
  2. Finalise policy: Establish a clear policy, enforce it, and iterate based on feedback.
  3. Empower teams: Equip creators and maintainers with tools to manage their documentation.
  4. Track metrics, celebrate wins: Recognise and reward teams that follow best practices. Repeat regularly.

Now let’s look at each of these steps in detail.

Take inventory: Assess existing internal processes and tools/portals and understand user behaviour

Understanding the current culture

To shift the documentation culture at a company, you need to first understand what that culture is. At Grab, with its diverse business units, tech teams, and varied documentation practices, just grasping this was a big step. We needed to look at it from two angles: how teams and business units approach documentation, and what portals hold what kinds of resources.

Here are a few observations that apply not just to Grab but to most tech companies:

  • People default to the easiest way to get information, either by asking someone or searching familiar places. If they can’t find a document quickly, they assume it doesn’t exist.
  • Different teams use different documentation tools, leading to scattered, hard-to-maintain content. Without a unified search, finding the right document is a challenge.
  • Documentation is often created during development but rarely maintained, resulting in outdated or duplicate content over time.
  • Lack of clear ownership and governance causes inconsistencies, making it harder to trust or rely on documentation.

Conducting extensive user research

The insights on understanding the culture of documentation were obtained from conducting extensive feedback-gathering activities. We adopted two separate strategies for user research:

  1. The first focused on gathering feedback from as many people as possible. We scaled this approach to reach a wide audience across multiple teams and departments. To manage this volume, we used closed-ended questions with multiple-choice options, allowing us to collect broad, organisation-wide insights on user needs and preferences.
  2. The second approach was more in-depth and personal. We conducted 1:1 sessions where we observed how users interacted with tools, asked open-ended questions, and dug into the reasons behind their behaviors. This helped us understand not just what users did, but why and how they did it.

From the first approach, we were able to gather that users frequently browse for Runbooks, how-to’s, and FAQs when it comes to technical documentation. They emphasise structure, ease of navigation, and up-to-date content when it comes to quality.

Based on the feedback, only 2% of engineers (1 out of 56) reported that 80-100% of on-call engineering questions were resolved using technical documentation. In contrast, 29% of engineers indicated that 40-60% of their questions were addressed through documentation, while 25% stated that 20-40% were resolved in this manner.

To improve the documentation and Docs-as-Code workflows for seamless integration of documentation into the engineering process, we built the TechDocs Editor on the Helix platform. This rich text editor allowed teams to write and maintain their documents more effectively. However, while many engineers appreciated the new features, they highlighted areas for improvement for a smoother experience. Key suggestions included enhancing the creation of merge requests (MRs), resolving conflicts more efficiently, and offering an auto-approval process. They also wanted a way to preview content before MR approval, capabilities like bulk migration, and integration options such as plugins for Jira and *Confluence wiki. Additionally, there was a call to increase clarity on what content should belong in TechDocs versus the Wiki.

Rooting TechDocs tool’s improvements in the user’s feedback

Based on the feedback received from the extensive user research, the TechDocs tool’s new features were planned and lined up based on a priority mapping that was entirely rooted in the feedback from user research and interviews. While not all feedback was directly implementable in terms of tool improvements, a significant amount was. For issues that couldn’t be resolved through tools, cultural changes and learning best practices became key to addressing the challenges.

Here are insights from the 1-1 user research that helped us enhance the TechDocs tools and processes:

  • Search experience is average. The search experience on the TechDocs portal has room for improvement, with a CSAT score of 58.57%. Some users prefer using a more centralised search option, as it searches across multiple platforms and offers more relevant results, especially considering gaps in documentation on the internal TechDocs portal.
  • Documentation landing page needs improvement. The Documentation landing page scored 10.71% CSAT, highlighting its need for better design and categorisation. Users found the page cluttered, and the categorisation was seen as random and confusing.

  • Reading experience is positive. Overall, users are satisfied with reading documentation on Helix, with an 88.31% CSAT for reading experience. Users appreciated the navigation’s organisation and structure. Suggestions for further improvement include:

    • Better table content display
    • Maximising content space
    • Enhancing color contrast
  • TechDocs adoption still faces challenges. Although TechDocs adoption has grown, several challenges remain:
    • Migration efforts: The migration process requires significant effort, and without support or a clear push, some employees do not see the need to migrate.
    • Cultural factors: Users continue using familiar platforms and are looking for incentives, such as unique Helix features, to consider making the move.
    • Accessibility: VPN access is required for some features.
    • Awareness: Many users are unaware of Helix TechDocs’ full range of features, such as the different search options, available search filters, and commenting capabilities.
  • Cross-team collaboration challenges: Users reported difficulties in collaborating with non-engineering roles. While engineers are comfortable with the Docs-as-Code approach, which allows for more flexibility and simplicity, some find the TechDocs editor useful for initial document creation or small edits.

Using this feedback, the product roadmap was set for the year to focus on addressing the top user complaints and improving the TechDocs tools accordingly.

Finalise a suitable policy and begin enforcing it. Collect feedback and reiterate

To improve discoverability and maintain consistency, we established a structured policy for organising documents. This policy ensures that documentation is stored in the right place based on its purpose and usage, making it easier for Grabbers to find what they need. The key guidelines are as follows:

  • Markdown for ‘create and publish’ type content: Documentation related to platforms, products, or services that don’t require frequent updates should be in markdown format and stored in GitLab. These documents were rendered in Helix.
  • Collaborative portals for collaborative docs: Time-sensitive and collaborative documents—such as postmortems, RFCs, design docs, and project plans— are not compatible with docs-as-code and hence should reside in portals that offer collaboration features, like easy commenting and multi-user editing. Dedicated spaces within Confluence Wiki are ideal for this purpose.
  • Separation of internal data: Internal documents meant only for specific teams should not mix with general engineering resources for end users. These can be stored in portals with less stringent review processes, as they don’t require the same level of quality or accuracy checks. Team-specific spaces on Confluence Wiki can serve this need effectively.

Empower creators and maintainers to self-serve documentation upkeep

More documentation doesn’t mean good documentation

Getting people to create documentation is one thing, but getting them to maintain and update it is a whole different challenge. One major issue is the lack of accountability. Without a clear owning team or point of contact (PIC) for a document, everyone assumes someone else will handle updates. This leads to stale, outdated information because no one takes responsibility. To address this, the TechDocs team introduced features like showing the “last updated” date on each page and flagging documents that hadn’t been updated in over three months. This approach helped in two ways:

  • Readers could quickly gauge how up-to-date the information was.
  • Content owners were reminded when their documents needed attention.

Another key strategy was requiring every document to have a dedicated PIC at the time of creation. This ensured:

  • Clear accountability for maintaining the document.
  • The PIC would receive notifications about outdated documents and any comments from readers, making it easier to address issues.

What about docs that are not really meant to be updated that frequently?

When building any feature, it’s important to consider different use cases. While flagging outdated documents helped maintainers keep track of their content, it could also frustrate those responsible for more static documents that don’t require frequent updates.

To make the “last updated” feature more relevant, we introduced an option for users to mark documents as “verified.” This allowed maintainers to turn off the “your doc is outdated” flag if they felt the information was still accurate. While this feature could be misused in an extremely large organisation, it worked well at Grab where internal products and employees generally rely on mutual trust and respect for maintaining simple systems and policies.

Training and info-typing workshops

The TechDocs team had a unique advantage in influencing the quality of internal product and platform documentation. Many of the creators and maintainers of these documents belonged to the same organisation, which allowed for smoother collaboration.

To elevate the quality of TechDocs, we recognised that improving the drafts produced by platform engineers was essential. This realisation led us to create self-paced training materials focused on information typing guidelines and writing best practices specifically designed for these engineers, which included:

  • Info-typing guidelines: Helping engineers categorise information for better clarity.
  • Writing best practices: Teaching techniques to enhance readability and engagement.

Building on the positive feedback from the training course, we launched interactive workshops. In these sessions, participants brought their own team’s user-facing documentation, and with the guidance of expert Tech Content Developers (TCDs), they made significant, live updates to their documents using the info-typing principles they had learned. This process enabled participants to:

  • Revise their documents: Make real-time improvements during the workshop.
  • Receive expert feedback: Gain insights from TCDs on enhancing document quality.

The workshops received outstanding feedback and were further refined to cater to the specific needs of each team, ensuring that the training remained relevant and effective for the different documentation sets they managed. By focusing on collaboration and practical learning, we were able to foster a culture of continuous improvement in our documentation practices.

Track metrics, celebrate wins. Recognise and repeat.

Recognising teams and individuals who follow best practices is key to sustaining momentum. We celebrated these wins by publicly acknowledging contributions in newsletters and internal communications, along with offering swag and rewards. Additionally, we tracked the accuracy of responses from oncall-bots, which use documentation to auto-respond to user queries on our internal communicator. By analysing whether these automated responses were accurate, we could assess the quality of the docs being referenced. Teams that kept their documentation up-to-date and adhered to our internal TechDocs policy were rewarded, further reinforcing these good practices.

Celebrating wins wasn’t a one-off—it became a regular practice, helping to solidify desired behaviors and create a cycle of continuous improvement.

What’s next

Looking ahead, we have some exciting goals to push the documentation culture even further:

  • Boost documentation quality: We’re aiming to improve the quality of platform docs by a significant percentage, which will help reduce support tickets and inquiries to the automated tech support bot.
  • Expand training: We’re ramping up training for more engineers, helping them sharpen their tech writing skills and aiming for top CSAT ratings.
  • Launch improved TechDocs portal: Our goal is to build better and more intuitive navigation and categorisation of content for an improved user experience.
  • User interviews and engagement: We’ll be working closely with champions and users across tech families to create an open feedback loop.
  • Enhance doc creation and editing workflows: We’ll streamline the process of creating and editing content using templates and native tools with consistent Markdown flavour usage.

Join us

Grab is a leading superapp in Southeast Asia, operating across the deliveries, mobility and digital financial services sectors. Serving over 800 cities in eight Southeast Asian countries, Grab enables millions of people everyday to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app. Grab was founded in 2012 with the mission to drive Southeast Asia forward by creating economic empowerment for everyone. Grab strives to serve a triple bottom line – we aim to simultaneously deliver financial performance for our shareholders and have a positive social impact, which includes economic empowerment for millions of people in the region, while mitigating our environmental footprint.

Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!

По буквите: Спигелман

Post Syndicated from Зорница Христова original https://www.toest.bg/po-bukvite-spigelman/

„Маус. Разказът на един оцелял“ от Арт Спигелман (част 1, „Баща ми кърви история“, и част 2, „Тук започнаха мъките ми“)

По буквите: Спигелман

превод от английски Владимир Полеганов, София: изд. НИКЕ, 2025

Добрата рецензия трябва да казва защо книгата засяга читателя, с какво е свързана с него.

Защо да я избере измежду хиляди нейни посестрими – как обстоятелствата в нея приличат на сегашните, какви поуки, настроения, смисли могат да се отнесат към днешния ден.

По какво героите приличат на нас.

Това е последното, което искам да направя, докато чета „Маус“ на Арт Спигелман.

Приликите между нацизма тогава и сега, между войните тогава и сега са толкова натрапчиви, че не е щадящо (себе си, читателя) да ги казваш на глас.

Внезапно започваш да разбираш цитираната от Тони Джуд (в „След войната“) ситуация, в която победените германци били задължени да гледат филм за концлагерите. И стискали очи в киното. Или разказа на Примо Леви как никой в следвоенна Италия не искал да чуе историята му. В нежеланието да знаеш има страх. И вина. Пропорциите варират. Искаш да знаеш, но само донякъде.

В „Маус“ на Арт Спигелман няма „донякъде“. Изобщо не става въпрос само за съчетанието между комикса като „леко“ изразно средство и тежката материя на историята. Идеята за „комикс за Холокоста“ би могла да се обърка на толкова много места, да стане пошла или едностранчива. Нали комиксите се асоциират с екшън между „добрите“ и „лошите“?

Е, „Маус“ не е това. Историята те дръпва навътре към сърцевината на книгата, защото мишките в нея са много повече истински, триизмерни хора от доста двукраки литературни герои. Главният персонаж, бащата, при цялото си страдание не е някакъв светец – нито преди войната, нито след нея. Докато действието снове от настоящето към миналото и обратно, ние виждаме недостатъците му толкова ясно, колкото у собствените си роднини – тук зарязал любима, там тормозил жена си, другаде пък изтърсил расистко подмятане. Дори влиза в клишето за стиснатия евреин – колкото и да се ядосва синът му.

В нашата култура сме свикнали да приемаме мъчениците за светци и обратно; ако някой е жертва, той трябва да бъде безгрешен. И това ни обърква, чувстваме се измамени, ако не е така;

в реалния свят обаче не само съвършените страдат, не само те заслужават да бъдат чути.

По буквите: Спигелман

Бащата в „Маус“ е толкова явно описван от натура, че няма как да не му вярваш. Затова и не можеш да си кажеш, че случващото се с него е наужким. А комиксовият формат те кара да обръщаш страниците бързо; това не е текст, в който катастрофата пъпли по релсите на изреченията. Отгръщаш – и те връхлита. На момента дори не разбираш какво си видял, защото разгръщаш нататък. Но седмици по-късно още не ти излиза от ума. При това нищо не е графично, в гадни подробности – стилът на Спигелман е обран, няма близки планове, няма подробности по телата. Те и без това са условни – човешки фигури с глави на мишки за евреите, на котки за германците и на прасета за поляците. Спигелман не спекулира с емоциите на читателя, не натиска лесните копчета. Просто иска да разкаже историята. Докрай.
Не „до щастливия край“ както в популярните филми. Защото това също е лъжа, страданието не спира – след извеждането си лагеристите продължават да мрат, а някои от спасените са убити… от завзелите междувременно имотите им „мирни“ техни съграждани.

Неудобна история – но Спигелман не се интересува кое е удобно. Интересува се от истината.

И още нещо във фактологичния му разказ. Също както не навлиза във физиологични подробности, Спигелман уж не говори за чувства, за психологичната страна на цялото преживяване. Тя присъства, както и всичко останало, като нещо веществено, като факт. Майката, която се самоубива. Самият Арт (от книгата), който ходи на терапия. И един от любимите ми детайли: по време на терапията Арт е с маска на мишка. Ясен символ от предишните епизоди – мишок с маска на прасе значи евреин, който се преструва на поляк.

Но мишата маска на самия Арт значи друго –

историята, включително семейната история, ни дава роля, но ние невинаги съвпадаме с нея.

По буквите: Спигелман

Пропуснах да кажа разни важни неща, например че в момента Арт Спигелман подготвя книга за Газа. И че е критик на израелската политика. Пропуснах да кажа, че е получил „Пулицър“ за „Маус“, че се смята за създател на графичния роман. Че огромният ефект от книгата му е повишил авторитета на комикса като изразно средство въобще. А също и че в днешното тегаво време книгата му е забранена в Русия (през 2015 г., уж защото свастиката на корицата била нацистка пропаганда) и горена в Полша (през 2001 г.). Неотдавна беше забранена и от училищното настоятелство в едно американско училище.

Моите почитания към издателство „Нике“, че я пуска точно сега. Преводът е отличен, печатът също; една-две грешки при нанасянето на буквите в балончетата леко озадачават („Войната сверши“), но това е дребна работа. А може би е и напомняне, че нищо още не е свършило.

Активните дарители на „Тоест“ получават постоянна отстъпка в размер на 20% от коричната цена на всички заглавия от каталозите на НИКЕ и няколко други български издателства в рамките на партньорската програма Читателски клуб „Тоест“. За повече информация прочетете на toest.bg/club.

В емблематичната си колонка, започната още през 2008 г. във в-к „Култура“, Марин Бодаков ни представяше нови литературни заглавия и питаше с какво точно тези книги ни променят. Вярваме, че е важно тази рубрика да продължи. От човек до човек, с нова книга в ръка.

Ще се видим на празния лист

Post Syndicated from Тоест original https://www.toest.bg/shte-se-vidim-na-prazniya-list/

Ще се видим на празния лист

Вече не се доверявам на хартията,
лъжа я.
Тя също се прави, че ме изслушва,
но не ми вярва и тя.

Свикнах да мълча, да дълбая дупки
навътре, където заравям
несериозни хрумвания, неизговорени
впечатления след глупав филм,
неразказани случки, непроведени
разговори, незабравени гледки.

Накъдето и да дълбая обаче,
стигам до теб.

Никой друг не знае за това –
така и трябва.

Откъде го знае хартията?

Кристин Димитрова

Кристин Димитрова (р. 1963, София) е писателка, поетеса и преводачка. Авторка е на двайсет книги с поезия, най-новата от които е „Уважаеми пътници“ (2018), два романа – „Сабазий“ (2007) и „Ще се върна за теб“ (2022), и три сборника с разкази: „Любов и смърт под кривите круши“ (2004), „Тайният път на мастилото“ (2011), „Като пристигнеш, обади се“ (2017). Носителка на седем награди за поезия, четири за проза и две за превод на поезия. Нейни текстове са публикувани в 38 държави на 28 езика.

Според Екатерина Йосифова „четящият стихотворение сутрин… добре понася другите часове“ от деня. Убедени, че поезията държи умовете ни будни, а сърцата – отворени, в края на всеки месец ви предлагаме по едно стихотворение. Защото и в най-смутни времена доброто стихотворение е добра новина.

[$] LWN.net Weekly Edition for February 27, 2025

Post Syndicated from corbet original https://lwn.net/Articles/1011011/

Inside this week’s LWN.net Weekly Edition:

  • Front: Tail calls in CPython; BPF cancellation; Slabs, sheaves, and barns; Atomic block writes; Large filesystem block sizes; EPEL 10 for older CPUs; pytest-mh; Open-source battery.
  • Briefs: DMA discussion; Armbian 25.2; Gentoo qcow2; Aqualung 2.0; Emacs 30.1; Rust 1.85.0; Quotes; …
  • Announcements: Newsletters, conferences, security updates, patches, and more.

AWS Chatbot is now named Amazon Q Developer

Post Syndicated from Aaron Sempf original https://aws.amazon.com/blogs/devops/aws-chatbot-is-now-named-amazon-q-developer/

Today, we’re excited to announce that AWS Chatbot has been renamed to Amazon Q Developer, representing an enhancement to developer productivity through generative AI-powered capabilities.

This update represents more than a name change – it’s an enhancement of our chat-based DevOps capabilities. By combining AWS Chatbot’s proven functionality with Amazon Q’s generative AI capabilities, we’re providing developers with more intuitive, efficient tools for cloud resource management.

Transition for Existing Users

The transition to Amazon Q Developer maintains compatibility with most workflows. Current AWS Chatbot users should experience no disruption to their configurations, permissions, or established processes, except for the following use cases.

Notifications: If you are using Q in chat applications to send notifications, then you don’t need to make any changes. Your notifications will start showing “Amazon Q” as the sender.

Manual commands: The visible change is the new “@Amazon Q” command replacing the previous “@aws” mention in chat channels. If you are running commands manually, then you will use “@Amazon Q” instead of “@aws”.

Tip: it is faster to type @Q. The chat platform displays auto complete recommendations with the matching app in the channel.

Programmatic commands: Your Slack Automation Workflows that trigger commands within the AWS Chatbot won’t change with this renaming. If you are sending messages to your Slack channels programmatically using Webhooks or the API with “@aws”, you’ll need to change how you invoke the app programmatically. For more information, see Updating Slack bot user app mentions when sending messages to chat channels programmatically.

All service APIs, SDK endpoints, and AWS Identity and Access Management (IAM) permissions remain unchanged, ensuring business continuity.

We’ve maintained the original AWS ChatBot accessibility by offering Amazon Q Developer’s chat features through the Free tier. This ensures that teams of all sizes can benefit from these enhanced capabilities without additional costs. The renamed service is accessible in all commercial regions, maintaining the same geographical reach as the original AWS Chatbot service.

Security remains paramount with Amazon Q Developer. The service maintains all existing security controls, including AWS Organizations Service Control Policies and chat application policies. Organizations can precisely control access to resources and features through granular IAM permissions and channel-specific guardrails. To take advantage of generative AI capabilities organizations will need to add to the configuration of their channel permissions.

 

Enhanced Chat Capabilities for DevOps

Amazon Q Developer integration with Microsoft Teams and Slack transforms these chat applications into powerful DevOps command centers, where team members can monitor, diagnose, and optimize their AWS resources and applications. Amazon Q Developer in chat applications provide real-time visibility into environment states, helping team members quickly identify which resources are operational or experiencing issues.

Team members can reduce incident response times and monitor performance issues, traffic spikes, infrastructure events, and security threats through DevOps tooling that enables custom notifications with team member tagging for critical application events, interactive action buttons and aliases for telemetry retrieval, and command execution in chat channels.

Amazon Q in Slack channel

Amazon Q in Slack channel

Building on existing features like custom notifications and actions, command aliases, and Amazon Bedrock Agents integration, Amazon Q Developer uses natural language processing to understand context and intent. For example, when investigating resources in a region, you can ask, “What EC2 instances are in us-east-1?”. This natural language understanding streamlines interactions and improves efficiency.

Ask Amazon Q about resources in AWS Account

Ask Amazon Q about resources in AWS Account

Amazon Q Developer can be used for more comprehensive resource management and status monitoring in chat channels. It can be used to send alerts to chat channels on Amazon CloudWatch metrics for monitoring, or can be used to explore resources across regions or within an account. DevOps teams can execute queries, such as count all VPCs in a region, listing all subnets in a VPC or providing all details for Amazon Elastic Compute Cloud (EC2) instances in a region, such as “provide all details for EC2 instances in us-east-1”, providing better visibility into infrastructure.

Use Amazon Q to query AWS resources information

Use Amazon Q to query AWS resources information

Getting Started

Setting up Amazon Q Developer involves a straightforward process through the Amazon Q Developer console or the AWS SDK. To interact with Amazon Q Developer’s generative AI capabilities, start by adding appropriate managed policies (AmazonQDeveloperAccess or AmazonQFullAccess) to your IAM roles. Your teams can then customize their notification preferences, set up automated responses, and configure security guardrails according to their specific requirements.

We encourage you to explore the getting started guide for best practices, advanced features, and reviewing our updated documentation. For a summary of changes visit Amazon Q Developer in chat applications rename documentation.

We’re excited to see how you and your teams leverage these enhanced capabilities to streamline their DevOps workflows and improve collaboration.

About the Author

Aaron Sempf Profile

Aaron Sempf is Next Gen Tech Lead for the AWS Partner Organization in Asia-Pacific and Japan. With over twenty years in distributed system engineering design and development, he focuses on solving for large scale complex integration and event driven systems. In his spare time, he can be found coding prototypes for autonomous robots, IoT devices, distributed solutions and designing Agentic Architecture patterns for GenAI assisted business automation.

Top analytics announcements of AWS re:Invent 2024

Post Syndicated from Sakti Mishra original https://aws.amazon.com/blogs/big-data/top-analytics-announcements-of-aws-reinvent-2024/

AWS re:Invent 2024, the flagship annual conference, took place December 2–6, 2024, in Las Vegas, bringing together thousands of cloud enthusiasts, innovators, and industry leaders from around the globe. This premier event showcased groundbreaking advancements, keynotes from AWS leadership, hands-on technical sessions, and exciting product launches.

Analytics remained one of the key focus areas this year, with significant updates and innovations aimed at helping businesses harness their data more efficiently and accelerate insights. From enhancing data lakes to empowering AI-driven analytics, AWS unveiled new tools and services that are set to shape the future of data and analytics.

In this post, we walk you through the top analytics announcements from re:Invent 2024 and explore how these innovations can help you unlock the full potential of your data.

Amazon SageMaker

Introducing the next generation of Amazon SageMaker

AWS announces the next generation of Amazon SageMaker, a unified platform for data, analytics, and AI. This launch brings together widely adopted AWS machine learning (ML) and analytics capabilities and provides an integrated experience for analytics and AI with unified access to data and built-in governance.

The next generation of SageMaker also introduces new capabilities, including Amazon SageMaker Unified Studio (preview), Amazon SageMaker Lakehouse, and Amazon SageMaker Data and AI Governance. Amazon SageMaker Unified Studio brings together functionality and tools from the range of standalone studios, query editors, and visual tools available today in Amazon EMR, AWS Glue, Amazon Redshift, Amazon Bedrock, and the existing Amazon SageMaker Studio. Amazon SageMaker Lakehouse provides an open data architecture that reduces data silos and unifies data across Amazon Simple Storage Service (Amazon S3) data lakes, Redshift data warehouses, and third-party and federated data sources. Amazon SageMaker Data and AI Governance, including Amazon SageMaker Catalog built on Amazon DataZone, empowers you to securely discover, govern, and collaborate on data and AI workflows.

Amazon DynamoDB zero-ETL integration with Amazon SageMaker Lakehouse

Amazon DynamoDB zero-ETL integration with SageMaker Lakehouse automates the extraction and loading of data from a DynamoDB table into SageMaker Lakehouse, an open and secure lakehouse. Using the no-code interface, you can maintain an up-to-date replica of your DynamoDB data in the data lake by quickly setting up your integration to handle the complete process of replicating data and updating records. This zero-ETL integration reduces the complexity and operational burden of data replication to let you focus on deriving insights from your data. You can create and manage integrations using the AWS Management Console, the AWS Command Line Interface (AWS CLI), or the SageMaker Lakehouse APIs.

Amazon S3 Tables

Amazon S3 Tables – Fully managed Apache Iceberg tables optimized for analytics workloads

Amazon S3 Tables deliver the first cloud object store with built-in Apache Iceberg support, and the most straightforward way to store tabular data at scale. S3 Tables are specifically optimized for analytics workloads, resulting in up to 3 times faster query throughput and up to 10 times higher transactions per second compared to self-managed tables. S3 Tables are designed to perform continual table maintenance to automatically optimize query efficiency and storage cost over time, even as your data lake scales and evolves. S3 Tables integration with the AWS Glue Data Catalog is in preview, allowing you to stream, query, and visualize data—including Amazon S3 Metadata tables—using AWS analytics services such as Amazon Data Firehose, Amazon Athena, Amazon Redshift, Amazon EMR, and Amazon QuickSight.

Amazon S3 Metadata (Preview) – Easiest and fastest way to manage your metadata

Amazon S3 Metadata is the simplest and fastest way to help you instantly discover and understand your S3 data with automated, queried metadata that updates in near real time. S3 Metadata supports object metadata, which includes system-defined details like size and the source of the object, and custom metadata, which allows you to use tags to annotate your objects with information like product SKU, transaction ID, or content rating, for example.

S3 Metadata is designed to automatically capture metadata from objects as they are uploaded into a bucket, and to make that metadata queryable in a read-only table. These metadata tables are stored in S3 Tables, the new S3 storage offering optimized for tabular data. Additionally, S3 Metadata integrates with Amazon Bedrock, allowing for the annotation of AI-generated videos with metadata that specifies its AI origin, creation timestamp, and the specific model used for its generation.

Additional resources:

AWS Glue

Introducing AWS Glue 5.0

With AWS Glue 5.0, you get improved performance, enhanced security, support for SageMaker Unified Studio and SageMaker Lakehouse, and more. AWS Glue 5.0 enables you to develop, run, and scale your data integration workloads and get insights faster.

AWS Glue 5.0 upgrades the engines to Apache Spark 3.5.2, Python 3.11, and Java 17, with new performance and security improvements. It also updates open table format support to Apache Hudi 0.15.0, Apache Iceberg 1.6.1, and Delta Lake 3.2.0. AWS Glue 5.0 adds Spark native fine-grained access control with AWS Lake Formation so you can apply table-, column-, row-, and cell-level permissions on S3 data lakes. Finally, AWS Glue 5.0 adds support for SageMaker Lakehouse to unify all your data across S3 data lakes and Redshift data warehouses.

Amazon S3 Access Grants now integrate with AWS Glue

Amazon S3 Access Grants now integrate with AWS Glue for analytics, ML, and application development workloads in AWS. S3 Access Grants map identities from your identity provider (IdP), such as Entra ID and Okta or AWS Identity and Access Management (IAM) principals, to datasets stored in Amazon S3. This integration gives you the ability to manage Amazon S3 permissions for end-users running jobs with AWS Glue 5.0 or later, without the need to write and maintain bucket policies or individual IAM roles. When end-users in the appropriate user groups access Amazon S3 using AWS Glue ETL for Apache Spark, they will then automatically have the necessary permissions to read and write data.

AWS Glue Data catalog now automates generating statistics for new tables

The AWS Glue Data Catalog now automates generating statistics for new tables. These statistics are integrated with a cost-based optimizer (CBO) from Amazon Redshift and Athena, resulting in improved query performance and potential cost savings. Previously, creating statistics for Iceberg tables in the Data Catalog required you to continuously monitor and update configurations for your tables. Now, the Data Catalog lets you generate statistics automatically for new tables with one-time catalog configuration. Amazon Redshift and Athena use the updated statistics to optimize queries, using optimizations such as optimal join order or cost-based aggregation pushdown. The Data Catalog console provides you visibility into the updated statistics and statistics generation runs.

AWS expands data connectivity for Amazon SageMaker Lakehouse and AWS Glue

SageMaker Lakehouse announces unified data connectivity capabilities to streamline the creation, management, and usage of connections to data sources across databases, data lakes, and enterprise applications. SageMaker Lakehouse unified data connectivity provides a connection configuration template, support for standard authentication methods like basic authentication and OAuth 2.0, connection testing, metadata retrieval, and data preview. You can create SageMaker Lakehouse connections through SageMaker Unified Studio (preview), the AWS Glue console, or a custom-built application using APIs under AWS Glue.

With the ability to browse metadata, you can understand the structure and schema of the data source and identify relevant tables and fields. SageMaker Lakehouse unified connectivity is available where SageMaker Lakehouse or AWS Glue is available.

Announcing generative AI troubleshooting for Apache Spark in AWS Glue (Preview)

AWS Glue announces generative AI troubleshooting for Apache Spark, a new capability that helps data engineers and scientists quickly identify and resolve issues in their Spark jobs. Spark Troubleshooting uses ML and generative AI technologies to provide automated root cause analysis for Spark job issues, along with actionable recommendations to fix identified issues. With Spark troubleshooting, you can initiate automated analysis of failed jobs with a single click on the AWS Glue console. Powered by Amazon Bedrock, Spark troubleshooting reduces debugging time from days to minutes.

The generative AI troubleshooting for Apache Spark preview is available for jobs running on AWS Glue 4.0.

Amazon EMR

Introducing Advanced Scaling in Amazon EMR Managed Scaling

We are excited to announce Advanced Scaling, a new capability in Amazon EMR Managed Scaling that provides you increased flexibility to control the performance and resource utilization of your Amazon EMR on EC2 clusters. With Advanced Scaling, you can configure the desired resource utilization or performance levels for your cluster, and Amazon EMR Managed Scaling will use your intent to intelligently scale the cluster and optimize cluster compute resources.

Advanced Scaling is available with Amazon EMR release 7.0 and later and is available in all AWS Regions where Amazon EMR Managed Scaling is available.

Additional resources:

Amazon Athena

Amazon SageMaker Lakehouse integrated access controls now available in Amazon Athena federated queries

SageMaker now supports connectivity, discovery, querying, and enforcing fine-grained data access controls on federated sources when querying data with Athena. Athena is a query service that makes it simple to analyze your data lake and federated data sources such as Amazon Redshift, DynamoDB, or Snowflake using SQL without extract, transform, and load (ETL) scripts. Now, data workers can connect to and unify these data sources within SageMaker Lakehouse. Federated source metadata is unified in SageMaker Lakehouse, where you apply fine-grained policies in one place, helping to streamline analytics workflows and secure your data.

Amazon Managed Service for Apache Flink

Amazon Managed Service for Apache Flink now supports Amazon Managed Service for Prometheus as a destination

AWS announced support for a new Apache Flink connector for Amazon Managed Service for Prometheus. The new connector, contributed by AWS for the Flink open source project, adds Amazon Managed Service for Prometheus as a new destination for Flink. You can use the new connector to send processed data to an Amazon Managed Service for Prometheus destination starting with Flink version 1.19. With Amazon Managed Service for Apache Flink, you can transform and analyze data in real time. There are no servers and clusters to manage, and there is no compute and storage infrastructure to set up.

Amazon Managed Service for Apache Flink now delivers to Amazon SQS queues

AWS announced support for a new Flink connector for Amazon Simple Queue Service (Amazon SQS). The new connector, contributed by AWS for the Flink open source project, adds Amazon SQS as a new destination for Apache Flink. You can use the new connector to send processed data from Amazon Managed Service for Apache Flink to SQS messages with Flink, a popular framework and engine for processing and analyzing streaming data.

Amazon Managed Service for Apache Flink releases a new Amazon Kinesis Data Streams connector

Amazon Managed Service for Apache Flink now offers a new Flink connector for Amazon Kinesis Data Streams. This open source connector, contributed by AWS, supports Flink 2.0 and provides several enhancements. It enables in-order reads during stream scale-up or scale-down, supports Flink’s native watermarking, and improves observability through unified connector metrics. Additionally, the connector uses the AWS SDK for Java 2.x, which supports enhanced performance and security features, and native retry strategy. You can use the new connector to read data from a Kinesis data stream starting with Flink version 1.19.

Amazon Redshift

Amazon SageMaker Lakehouse and Amazon Redshift support for zero-ETL integrations from eight applications

SageMaker Lakehouse and Amazon Redshift now support zero-ETL integrations from applications, automating the extraction and loading of data from eight applications, including Salesforce, SAP, ServiceNow, and Zendesk. As an open, unified, and secure lakehouse for your analytics and AI initiatives, SageMaker Lakehouse enhances these integrations to streamline your data management processes. These zero-ETL integrations are fully managed by AWS and minimize the need to build ETL data pipelines. Optimize your data ingestion processes and focus instead on analysis and gaining insights.

Amazon Redshift multi-data warehouse writes through data sharing is now generally available

AWS announces the general availability of Amazon Redshift multi-data warehouse writes through data sharing. You can now start writing to Redshift databases from multiple Redshift data warehouses in just a few clicks. With Redshift multi-data warehouse writes through data sharing, you can keep ETL jobs more predictable by splitting workloads between multiple warehouses, helping you meet your workload performance requirements with less time and effort. Your data is immediately available across AWS accounts and Regions after it’s committed, enabling better collaboration across your organization.

Announcing Amazon Redshift Serverless with AI-driven scaling and optimization

Amazon Redshift Serverless introduces the next generation of AI-driven scaling and optimization in cloud data warehousing. Redshift Serverless uses AI techniques to automatically scale with workload changes across all key dimensions—such as data volume changes, number of concurrent users, and query complexity—to meet and maintain your price-performance targets. Amazon internal tests demonstrate that this optimization can provide you up to 10 times better price performance for variable workloads, without manual intervention.

Redshift Serverless with AI-driven scaling and optimization is available in all AWS Regions where Redshift Serverless is available.

Amazon Redshift now supports incremental refresh on Materialized Views (MVs) for data lake tables

Amazon Redshift now supports incremental refresh of materialized views on data lake tables. This capability helps you improve query performance for your data lake queries in a cost-effective and efficient manner. By enabling incremental refresh for materialized views, you can maintain up-to-date data in a more efficient and affordable way.

Support for incremental refresh for materialized views on data lake tables is now available in all commercial Regions. To get started and learn more, visit Materialized views on external data lake tables in Amazon Redshift Spectrum.

AWS announces Amazon Redshift integration with Amazon Bedrock for generative AI

AWS announces the integration of Amazon Redshift with Amazon Bedrock, a fully managed service offering high-performing foundation models (FMs) making it simpler and faster for you to build generative AI applications. This integration enables you to use large language models (LLMs) from simple SQL commands alongside your data in Amazon Redshift.

The Amazon Redshift integration with Amazon Bedrock is now generally available in all Regions where Amazon Bedrock and Amazon Redshift ML are supported. To get started, see Amazon Redshift ML integration with Amazon Bedrock.

Announcing general availability of auto-copy for Amazon Redshift

Amazon Redshift announces the general availability of auto-copy, which simplifies data ingestion from Amazon S3 into Amazon Redshift. This new feature enables you to set up continuous file ingestion from your S3 prefix and automatically load new files to tables in your Redshift data warehouse without the need for additional tools or custom solutions.

Amazon Redshift auto-copy from Amazon S3 is now generally available for both Redshift Serverless and Amazon Redshift RA3 Provisioned data warehouses in all AWS commercial Regions.

Amazon DataZone

Data Lineage is now generally available in Amazon DataZone and next generation of Amazon SageMaker

AWS announces general availability of Data Lineage in Amazon DataZone and the next generation of SageMaker, a capability that automatically captures lineage from AWS Glue and Amazon Redshift to visualize lineage events from source to consumption. Being OpenLineage compatible, this feature allows data producers to augment the automated lineage with lineage events captured from OpenLineage-enabled systems or through an API, to provide a comprehensive data movement view to data consumers. This feature automates lineage capture of schema and transformations of data assets and columns from AWS Glue, Amazon Redshift, and Spark executions in tools to maintain consistency and reduce errors. Additionally, the data lineage feature versions lineage with each event, enabling you to visualize lineage at any point in time or compare transformations across an asset’s or job’s history.

Amazon DataZone now enhances data access governance with enforced metadata rules

Amazon DataZone now supports enforced metadata rules for data access workflows, providing organizations with enhanced capabilities to strengthen governance and compliance with their organization needs. This new feature allows domain owners to define and enforce mandatory metadata requirements, making sure data consumers provide essential information when requesting access to data assets in Amazon DataZone. By streamlining metadata governance, this capability helps organizations meet compliance standards, maintain audit readiness, and simplify access workflows for greater efficiency and control.

Amazon DataZone expands data access with tools like Tableau, Power BI, and more

Amazon DataZone now supports authentication with the Athena JDBC driver, enabling data consumers to query their project’s subscribed data lake assets in Amazon DataZone using popular business intelligence (BI) and analytics tools such as Tableau, Domino, Power BI, Microsoft Excel, SQL Workbench, and more. Data analysts and scientists can seamlessly access and analyze governed data in Amazon DataZone using a standard JDBC connection with their preferred tools.

This feature is now available in all the AWS commercial Regions where Amazon DataZone is supported. Check out Expanding data analysis and visualization options: Amazon DataZone now integrates with Tableau, Power BI, and more and Connecting Amazon DataZone with external applications via JDBC connectivity to learn more about how to connect Amazon DataZone to external analytics tools via JDBC.

Amazon QuickSight

Announcing scenarios analysis capability of Amazon Q in QuickSight (preview)

A new scenario analysis capability of Amazon Q in QuickSight is now available in preview. This new capability provides an AI-assisted data analysis experience that helps you make better decisions, faster. Amazon Q in QuickSight simplifies in-depth analysis with step-by-step guidance, saving hours of manual data manipulation and unlocking data-driven decision-making across your organization. You can ask a question or state your goal in natural language and Amazon Q in QuickSight guides you through every step of advanced data analysis—suggesting analytical approaches, automatically analyzing data, surfacing relevant insights, and summarizing findings with suggested actions.

Amazon QuickSight now supports prompted reports and reader scheduling for pixel-perfect reports

We are enabling QuickSight readers to generate filtered views of pixel-perfect reports and create schedules to deliver reports through email. Readers can create up to five schedules per dashboard for themselves. Previously, only dashboard owners could create schedules and only on the default (author published) view of the dashboard. Now, if an author has added controls to the pixel-perfect report, schedules can be created or updated to respect selections on the filter control.

Prompted reports and reader scheduling are now available in all supported QuickSight Regions—see Amazon QuickSight endpoints and quotas for QuickSight Regional endpoints.

Amazon Q in QuickSight unifies insights from structured and unstructured data

Amazon Q in QuickSight provides you with unified insights from structured and unstructured data sources through integration with Amazon Q Business. With data stories in Amazon Q in QuickSight, you can upload documents, or connect to unstructured data sources from Amazon Q Business, to create richer narratives or presentations explaining your data with additional context. This integration enables organizations to harness insights from all their data without the need for manual collation, leading to more informed decision-making, time savings, and a significant competitive edge.

Amazon Q Business now provides insights from your databases and data warehouses (preview)

AWS announces the public preview of the integration between Amazon Q Business and QuickSight, delivering a transformative capability that unifies answers from structured data sources (databases, warehouses) and unstructured data (documents, wikis, emails) in a single application.

With the QuickSight integration, you can now link your structured sources to Amazon Q Business through the extensive set of data source connectors available in QuickSight. This integration unifies insights across knowledge sources, helping organizations make more informed decisions while reducing the time and complexity traditionally required to gather insights.

Amazon OpenSearch Service

Amazon OpenSearch Service zero-ETL integration with Amazon Security Lake

Amazon OpenSearch Service now offers a zero-ETL integration with Amazon Security Lake, enabling you to query and analyze security data in-place directly through OpenSearch. This integration allows you to efficiently explore voluminous data sources that were previously cost-prohibitive to analyze, helping you streamline security investigations and obtain comprehensive visibility of your security landscape.

Amazon OpenSearch Ingestion now supports writing security data to Amazon Security Lake

Amazon OpenSearch Ingestion now allows you to write data into Amazon Security Lake in real time, allowing you to ingest security data from both AWS and custom sources and uncover valuable insights into potential security issues in near real time. With this feature, you can now use OpenSearch Ingestion to ingest and transform security data from popular third-party sources like Palo Alto, CrowdStrike, and SentinelOne into OCSF format before writing the data into Amazon Security Lake. After the data is written to Amazon Security Lake, it is available in the AWS Glue Data Catalog and Lake Formation tables for the respective source.

AWS Clean Rooms

AWS Clean Rooms now supports multiple clouds and data sources

AWS Clean Rooms announces support for collaboration with datasets from multiple clouds and data sources. This launch allows companies and their partners to collaborate with data stored in Snowflake and Athena, without having to move or share their underlying data among collaborators.

Conclusion

re:Invent 2024 showcased how AWS continues to push the boundaries of data and analytics, delivering tools and services that empower organizations to derive faster, smarter, and more actionable insights. From advancements in data lakes, data warehouses, and streaming solutions to the integration of generative AI capabilities, these announcements are designed to transform the way businesses interact with their data.

As we look ahead, it’s clear that AWS is committed to helping organizations stay ahead in an increasingly data-driven world. Whether you’re modernizing your analytics stack or exploring new possibilities with AI and ML, the innovations from re:Invent 2024 provide the building blocks to unlock value from your data.

Stay tuned for more deep dives into these announcements, and don’t hesitate to explore how these tools can accelerate your journey toward data-driven success!

About the Authors

Sakti Mishra serves as Principal Data and AI Solutions Architect at AWS, where he helps customers modernize their data architecture and define end-to end-data strategies, including data security, accessibility, governance, and more. He is also the author of Simplify Big Data Analytics with Amazon EMR and AWS Certified Data Engineer Study Guide books. Outside of work, Sakti enjoys learning new technologies, watching movies, and visiting places with family. He can be reached via LinkedIn.

Navnit Shukla serves as an AWS Specialist Solutions Architect with a focus on analytics. He possesses a strong enthusiasm for assisting clients in discovering valuable insights from their data. Through his expertise, he constructs innovative solutions that empower businesses to arrive at informed, data-driven choices. Notably, Navnit Shukla is the accomplished author of the book titled “Data Wrangling on AWS.” He can be reached via LinkedIn.

Amazon Web Services named a Leader in the 2024 Gartner Magic Quadrant for Data Integration Tools

Post Syndicated from William Vambenepe original https://aws.amazon.com/blogs/big-data/amazon-web-services-named-a-leader-in-the-2024-gartner-magic-quadrant-for-data-integration-tools/

Amazon Web Services (AWS) has been recognized as a Leader in the 2024 Gartner Magic Quadrant for Data Integration Tools. We were positioned in the Challengers Quadrant in 2023.

This recognition, we feel, reflects our ongoing commitment to innovation and excellence in data integration, demonstrating our continued progress in providing comprehensive data management solutions.

The Gartner Magic Quadrant evaluates 20 data integration tool vendors based on two axes—Ability to Execute and Completeness of Vision. This evaluation, we feel, critically examines vendors’ capabilities to address key service needs, including data engineering, operational data integration, modern data architecture delivery, and enabling less-technical data integration across various deployment models.

Discover, prepare, and integrate all your data at any scale

AWS Glue is a fully managed, serverless data integration service that simplifies data preparation and transformation across diverse data sources. With its comprehensive suite of tools, AWS Glue allows users to build and manage data pipelines efficiently, without requiring extensive infrastructure management expertise.

Given the diverse data integration needs of customers, AWS offers a robust data integration system through multiple services including Amazon EMR, Amazon Athena, Amazon Managed Workflows for Apache Airflow (Amazon MWAA), Amazon Managed Streaming for Apache Kafka (MSK), Amazon Kinesis, and others. Many thousands of customers across various industries are using these services to transform, operationalize, and manage their data across data lakes and data warehouses.

We have embarked on a journey to unify the broad range of AWS data processing, analytics, and AI capabilities, starting with the announcement of Amazon SageMaker Unified Studio at re:Invent 2024. This includes the data integration capabilities mentioned above, with support for both structured and unstructured data. With an integrated experience for data workers, SageMaker Unified Studio provides an environment where users can collaborate and build faster. It supports model development, generative AI, data processing, and SQL analytics, all accelerated by Amazon Q Developer—our most capable generative AI assistant for software development. The Unified Studio provides unified access to all data sources, whether stored in data lakes, data warehouses, or third-party and federated sources, with robust governance and enterprise-grade security built-in.

Review the Gartner Magic Quadrant

Access a complimentary copy of the full report to see why Gartner positioned AWS as a Leader, and dive deep into the strengths and cautions of AWS. We believe our recognition as a Leader in the Gartner Magic Quadrant is a testament to delivering innovations for our customers.

MQ

Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

GARTNER is a registered trademark and service mark of Gartner and Magic Quadrant is a registered trademark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved. This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from here.

About the authors

William Vambenepe is Director of Product Management at AWS, where he leads the Product Management, Solutions Engineering, and UX Design team for data processing services (Amazon EMR, AWS Glue, Athena, Amazon MWAA), SageMaker Unified Studio, and SageMaker Catalog. Prior to AWS, William worked at Google (6 years building and growing the Data and Analytics product portfolio for Google Cloud, and 5 years as Product Management Director for Google Search). He had previously held software engineering leadership roles at Oracle and HP. William holds an Engineering degree from Ecole Centrale Paris, a graduate Diploma in Computer Science from Cambridge University, and a Masters in Engineering Management from Stanford University.

Santosh Chandrachood has been with AWS for over 8 years and helped build, launch, and scale a variety of AWS services. Currently, Santosh is Director and service leader for Data Processing, managing Amazon EMR, Athena, AWS Glue, and Managed Workflows for Apache Airflow (Amazon MWAA). Santosh also led AWS Data Integration as the General Manager. Before joining AWS, Santosh lead engineering teams in networking, storage, and data infrastructure areas.

MDR + SIEM: Why Full Access to Your Security Logs is Non-Negotiable

Post Syndicated from René Fusco original https://blog.rapid7.com/2025/02/26/mdr-siem-why-full-access-to-your-security-logs-is-non-negotiable/

MDR + SIEM: Why Full Access to Your Security Logs is Non-Negotiable

Many Managed Detection and Response (MDR) providers promise world-class threat detection, but behind the scenes they lock away your security logs, limiting your visibility and control. It’s your data — so why don’t you have full access to it? Isn’t the whole point of security to see everything happening in your environment? Without full access to your own data, you’re left dependent on their tools, their timelines, and their interpretations of security events.

This isn’t just an inconvenience — it’s a risk.

Pairing MDR with a Security Information and Event Management (SIEM) solution ensures complete transparency, enabling real-time investigation, historical threat hunting, compliance readiness, and deeper threat insights. If you don’t have full access to your security logs, you’re not truly in control of your cybersecurity strategy. And in today’s high-stakes environment, that’s simply not an option.

With Rapid7 MDR, you don’t just gain a service — you gain full access and control over your data, unlocking significant advantages for compliance, long-term strategy, and cross-platform analytics.

The Benefits of Owning your Data

When it comes to cybersecurity, data is everything. Logs, events, and alerts are the building blocks of threat detection, incident response, and forensic investigations. Owning your data, particularly with Rapid7’s 13-month data retention, empowers you in ways that vendor-locked solutions cannot match. Here’s how:

  • Cross-platform analytics
    Modern security teams operate across cloud, hybrid, and on-prem environments. Owning your data means you can integrate security telemetry across platforms, enabling immediate answers and deeper correlations between systems for accurate threat detection.
  • Compliance made easier
    Many industries require businesses to retain data for specific periods to meet regulatory standards such as GDPR, HIPAA, or PCI DSS. Rapid7’s extended data retention ensures you’re always audit-ready and compliant without relying on third-party intermediaries for log retrieval.
  • Historical threat hunting and forensics
    Cyber threats evolve over time — sometimes laying dormant for months to manifest into an attack. With 13 months of historical data, the MDR service can trace attack patterns, uncover dormant threats, and conduct deep-dive forensic investigations to prevent repeat breaches. Advanced threats don’t just appear out of nowhere — long-term attack campaigns require long-term visibility. If you don’t know how an attacker got in, how can you ensure they won’t come back?

The hidden risks of limited data access

Many MDR providers operate in a “black box” model, where security data is siloed within their systems, restricting user access and limiting independent investigations. This lack of transparency not only creates dependency on the vendor but can also lead to serious security and operational risks:

  • Slower incident response
    Seconds matter when attackers are inside your environment. Security teams can waste critical time waiting for an MDR provider to retrieve logs or investigate issues, delaying decisive action during cyberattacks.
  • Reduced security visibility
    Cyber threats don’t operate in isolation. Without full data access, security teams miss critical patterns, struggle to correlate events, and lose the ability to conduct independent investigations. The result? A weakened security posture and increased attack exposure.
  • Hindered cross-team collaboration
    Security isn’t just a SOC function — it requires collaboration with IT, compliance, risk, and leadership teams. When data is locked behind an MDR provider’s system, security teams cannot share insights or validate threats with other departments effectively. This slows down decision-making, creates blind spots across IT infrastructure, and reduces the organization’s ability to work as a unified team in responding to threats.
  • Compliance gaps
    If an organization cannot independently access its logs, it may struggle to provide auditors with the necessary evidence for compliance frameworks like GDPR, HIPAA, DORA, NIS2, or PCI DSS.

Rapid7 MDR: Transparency and control

Rapid7’s MDR service offers transparent and unrestricted access to your data through InsightIDR, our cloud-native, next-gen SIEM built for both detection and response. Unlike traditional SIEMs that focus solely on log aggregation, InsightIDR actively identifies and prioritizes real threats by analyzing user and attacker behavior, leveraging deception technology, and utilizing built-in threat intelligence. This ensures not only full visibility but also rapid detection and response to advanced threats, helping security teams act faster. With Rapid7, you get:

  • Real-time insights: Monitor and analyze security data in real-time for faster response to threats — no waiting for vendor-controlled access.
  • Custom dashboards: Rapid7’s dashboards support operational and executive reporting, making it easier for security teams to collaborate with IT, compliance, and leadership on security progress, priorities, and effectiveness.
  • Custom detections:  Security teams can create tailored detections across any data sent to InsightIDR based on their specific infrastructure, threat models, and business needs. This ensures that critical anomalies and suspicious behaviors don’t get lost in generic detection rules.
  • Complete transparency: Audit every action taken by Rapid7 analysts and your SOC team plus see investigations and comments for transparency and collaboration.

Command the SIEM advantage: Context and correlation matter

A key differentiator of Rapid7 MDR is that InsightIDR is more than just a SIEM — it’s a next-gen detection and response platform. Many MDR solutions provide basic alerting but lack the advanced behavioral analytics and automated response capabilities of InsightIDR. By combining SIEM, user behavior analytics, deception technology, and automated response orchestration, InsightIDR proactively detects threats, correlates events across your environment, and enables faster, more precise response actions.

Without a SIEM, organizations struggle with:

  • Limited visibility into user behavior, making it harder to detect insider threats or compromised accounts.
  • No long-term correlation of security events, reducing the ability to uncover sophisticated, multi-stage attacks.
  • Gaps in historical threat hunting, restricting security teams from investigating past incidents, identifying trends, and improving future defenses.

With InsightIDR, Rapid7 MDR goes beyond detection — it  provides comprehensive context, automation, and deep forensic capabilities that elevate an organization’s security maturity.

Take back command of your security data

In a world where vendor lock-in is common, maintaining ownership and access to your security data is not just a convenience, it’s a necessity. Without it, organizations risk compliance failures, slower response times, and reduced visibility into their own security posture.

With Rapid7 MDR, you’re not just subscribing to a service — you’re gaining a proactive security partner. You get unrestricted access, 13-month data retention, and real-time threat detection and response — ensuring compliance, faster incident containment, and smarter security decisions powered by InsightIDR’s built-in detection capabilities.

Don’t settle for an MDR solution that keeps you in the dark. Choose an approach that empowers your security team with full access and control over your data.

Ready to experience the difference? Learn more about Rapid7 MDR today.

[$] Building an open-source battery

Post Syndicated from jake original https://lwn.net/Articles/1011730/

FOSDEM 2025 featured the usual talks
about open-source software, but, as always, the conference also offered the
opportunity to discover some more exotic and less software-centric
topics. That’s how I learned about the Flow
Battery Research Collective (FBRC), which is building what will
eventually become an open-source home battery.
Daniel Fernández Pinto represented the collective at
FOSDEM with his talk “Building
an Open-Source Battery for Stationary Storage” in the “Energy: Accelerating
the Transition through Open Source” developer room (devroom).

Gentoo now offers qcow2 disk images

Post Syndicated from jzb original https://lwn.net/Articles/1011959/

The Gentoo Linux project has
announced
the availability of qcow2 images for amd64 (x86_64) and arm64
(aarch64), and plans to “eventually” offer images for the
riscv64 and loongarch64 architectures.

The images, updated weekly, include an EFI boot partition and a fully
functional Gentoo installation; either with no network activated but a
password-less root login on the console (“no root pw”), or with
network activated, all accounts initially locked, but cloud-init
running on boot (“cloud-init”).

ИМОТ НА ЗЕЛЕНО – как се прави?

Post Syndicated from VassilKendov original https://kendov.com/%D0%B8%D0%BC%D0%BE%D1%82-%D0%BD%D0%B0-%D0%B7%D0%B5%D0%BB%D0%B5%D0%BD%D0%BE-%D0%BA%D0%B0%D0%BA-%D1%81%D0%B5-%D0%BF%D1%80%D0%B0%D0%B2%D0%B8/


Покупката на имот на зелено носи добри печалби, но и рискове. В това видео описвам процедурата и стъпките, които минимизират рисковете.
Много от стъпките човек не може да осъществи сам особено, когато предмета на сделката не съществува физически.
Кредит “на зелено” също е невъзможен. Банката на практика ще се включи на АКТ 15, когато имота вече е на 1-2 месеца от разрешение за ползване

От известно време развивам Youtube каналът Kendov.com. Ако желаете се абонирайте за канала  и проследете дискусиите под видеата по този въпрос.

Васил Кендов – финансов консултант

Съвет на деня
“Уредете си кредит преди да започнете търсенето на имот. Така ще знаете точно с колко пари ще разполагате”

 

The post ИМОТ НА ЗЕЛЕНО – как се прави? appeared first on KENDOV.COM.

Two new site features: full-text RSS and automatic dark mode

Post Syndicated from corbet original https://lwn.net/Articles/1011830/

One of the often-requested LWN site features that has languished the
longest on our to-do list is full-text RSS feeds. We are happy to announce
that, finally, there is a set of such feeds available; the full set can be
seen on our feeds page. This is a
subscriber-only feature, and it works by creating a unique fetch URL for
each user. We will, of course, be counting on our readers to not share
those URLs.

Another feature we have had requests for is to automatically present the
site in dark-mode colors when a reader’s browser has been configured to
prefer it. That feature, too, is now available. In this case, we had to
think about the interaction between automatic selection and the color
customization that the site has long had. The conclusion we reached is
that, if custom colors have been configured for an account, they will win
out over the automatic selection. There is a new preference in the customization area to change this
default if desired.

Both of these features — and the other enhancements we have made recently —
were enabled by the support of LWN’s subscribers. By making it possible to
bring in new staff last year, you created the space to improve the site
experience while keeping up with the writing. We thank all of you for your
support.

MikroTik RDS2216 Inbound with 20x U.2 NVMe and 100GbE on Arm

Post Syndicated from Rohit Kumar original https://www.servethehome.com/mikrotik-rds2216-inbound-with-20x-u-2-nvme-and-100gbe-on-arm-marvell-annapurna-labs/

The MikroTik RDS2216 combines a 100GbE, 25GbE, and 10GbE switch with a 16-core Arm processor and a 20-bay U.2 storage NAS all into 1U

The post MikroTik RDS2216 Inbound with 20x U.2 NVMe and 100GbE on Arm appeared first on ServeTheHome.

[$] Python interpreter adds tail calls

Post Syndicated from daroc original https://lwn.net/Articles/1010905/

The

Faster CPython project has been working to speed up the Python interpreter
for the past several years. Now, Ken Jin, a member of the project, has merged a

new set of changes that
have been

benchmarked as improving performance by 10% for some architectures.
The only change is switching from using computed goto statements to using
tail calls as part of the implementation of Python’s bytecode interpreter — but that change allows
modern compilers to generate significantly better code.

Security updates for Wednesday

Post Syndicated from jzb original https://lwn.net/Articles/1011928/

Security updates have been issued by Fedora (crun, gnutls, libtasn1, and openssl), Mageia (emacs, gnutls, iniparser, kernel, kmod-virtualbox, kmod-xtables-addons, kernel-linus, krb5, libxml2, and vim), Slackware (tigervnc and xorg), SUSE (libprotobuf-lite28_3_0 and Maven), and Ubuntu (dropbear, kernel, libxml2, linux, linux-lowlatency, linux-lowlatency-hwe-6.8, linux, linux-lts-xenial, linux-aws-5.4 linux-raspi-5.4, linux-gcp, linux-gke, linux-gkeop, linux-ibm, linux-nvidia, linux-nvidia-6.8, linux-nvidia-lowlatency, linux-raspi, ProFTPD, python-virtualenv, rails, and xorg-server, xwayland).

Keep AI interactions secure and risk-free with Guardrails in AI Gateway

Post Syndicated from Kathy Liao original https://blog.cloudflare.com/guardrails-in-ai-gateway/

The transition of AI from experimental to production is not without its challenges. Developers face the challenge of balancing rapid innovation with the need to protect users and meet strict regulatory requirements. To address this, we are introducing Guardrails in AI Gateway, designed to help you deploy AI safely and confidently. 

Why safety matters

LLMs are inherently non-deterministic, meaning outputs can be unpredictable. Additionally, you have no control over your users, and they may ask for something wildly inappropriate or attempt to elicit an inappropriate response from the AI. Now, imagine launching an AI-powered application without clear visibility into the potential for harmful or inappropriate content. Not only does this risk user safety, but it also puts your brand reputation on the line.

To address the unique security risks specific to AI applications, the OWASP Top 10 for Large Language Model (LLM) Applications was created. This is an industry-driven standard that identifies the most critical security vulnerabilities specifically affecting LLM-based and generative AI applications. It’s designed to educate developers, security professionals, and organizations on the unique risks of deploying and managing these systems.

The stakes are even higher with new regulations being introduced:

  • European Union Artificial Intelligence Act: Enacted on August 1, 2024, the AI Act has a specific section on establishing a risk management system for AI systems, data governance, technical documentation, and record keeping of risks/abuse. 

  • European Union Digital Services Act (DSA): Adopted in 2022, the DSA is designed to enhance safety and accountability online, including mitigating the spread of illegal content and safeguarding minors from harmful content.

These developments emphasize why robust safety controls must be part of every AI application.

The challenge

Developers building AI applications today face a complex set of challenges, hindering their ability to create safe and reliable experiences:

  • Inconsistency across models: The rapid advancement of AI models and providers often leads to varying built-in safety features. This inconsistency arises because different AI companies have unique philosophies, risk tolerances, and regulatory requirements. Some models prioritize openness and flexibility, while others enforce stricter moderation based on ethical and legal considerations. Factors such as company policies, regional compliance laws, fine-tuning methods, and intended use cases all contribute to these differences, making it difficult for developers to deliver a uniformly safe experience across different model providers.

  • Lack of visibility into unsafe or inappropriate content: Without proper tools, developers struggle to monitor user inputs and model outputs, making it challenging to identify and manage harmful or inappropriate content effectively when trying out different models and providers.

The answer? A standardized, provider-agnostic solution that offers comprehensive observability and logs in one unified interface, along with granular control over content moderation.

The solution: Guardrails in AI Gateway

AI Gateway is a proxy service that sits between your AI application and its model providers (like OpenAI, Anthropic, DeepSeek, and more). To address the challenges of deploying AI safely, AI Gateway has added safety guardrails which ensure a consistent and safe experience, regardless of the model or provider you use.

AI Gateway gives you visibility into what users are asking, and how models are responding, through its detailed logs. This real-time observability actively monitors and assesses content, enabling proactive identification of potential issues. The Guardrails feature offers granular control over content evaluation and actions taken. Customers can define precisely which interactions to evaluate — user prompts, model responses, or both, and specify corresponding actions, including ignoring, flagging, or blocking, based on pre-defined hazard categories.

Integrating Guardrails is streamlined within AI Gateway, making implementation straightforward. Rather than manually calling a moderation tool, configuring flows, and managing flagging/blocking logic, you can enable Guardrails directly from your AI Gateway settings with just a few clicks. 


Figure 1. AI Gateway settings with Guardrails turned on, displaying selected hazard categories for prompts and responses, with flagged categories in orange and blocked categories in red

Within the AI Gateway settings, developers can configure:

  • Guardrails: Enable or disable content moderation as needed.

  • Evaluation scope: Select whether to moderate user prompts, model responses, or both.

  • Hazard categories: Specify which categories to monitor and determine whether detected inappropriate content should be blocked or flagged.


Figure 2. Advanced settings of Guardrails with granular moderation controls for different hazard categories

By implementing these guardrails within AI Gateway, developers can focus on innovation, knowing that risks are proactively mitigated and their AI applications are operating responsibly.

Leveraging Llama Guard on Workers AI

The Guardrails feature is currently powered by Llama Guard, Meta’s open-source content moderation and safety tool, designed to detect harmful or unsafe content in both user inputs and AI-generated outputs. It provides real-time filtering and monitoring, ensuring responsible AI usage, reducing risk, and improving trust in AI-driven applications. Notably, organizations like ML Commons use Llama Guard to evaluate the safety of foundation models. 

Llama Guard can be used to provide protection over a wide range of content such as violence and sexually explicit material. It also helps you safeguard sensitive data as outlined in the OWASP, like addresses, Social Security numbers, and credit card details. Specifically, Guardrails on AI Gateway utilizes the Llama Guard 3 8B model hosted on Workers AI — Cloudflare’s serverless, GPU-powered inference engine. Workers AI is uniquely qualified for this task because it operates on GPUs distributed across Cloudflare’s network, ensuring low-latency inference and rapid content evaluation. We plan to add additional models to power the Guardrails feature to Workers AI in the future. 

Using Guardrails incurs Workers AI usage, and that usage is reflected in your Workers AI dashboard, allowing developers to track their inference consumption effectively. 

How it works 

Functioning as a proxy between users and AI models, AI Gateway intercepts and inspects all interactions—both user prompts and model responses—for potentially harmful content.


Figure 3. Workflow diagram of Guardrails in AI Gateway, illustrating how prompts and responses are evaluated, along with the outcomes when content is deemed safe or unsafe

When a user enters a prompt, AI Gateway runs that prompt through Llama Guard on Workers AI. Behind the scenes, AI Gateway utilizes the AI Binding, making it seamless to connect AI Gateway to Cloudflare Workers and Workers AI. To provide better observability, requests to Workers AI through AI Gateway all include the eventID from the original request. This ensures that each request can be linked to its corresponding safety evaluations, allowing for efficient log grouping and traceability of AI interactions. 

Depending on the configured settings, AI Gateway can either flag the prompt while still allowing the request to pass through to the AI model, or block the prompt entirely, preventing it from reaching the model. Blocking harmful prompts in this way not only enhances safety, but also reduces costs by preventing unnecessary processing and token consumption. If a prompt passes the safety check, it’s forwarded to the AI model.

Let’s see an example of a prompt blocked by Guardrails. In this example, the Guardrails settings have been turned on for user prompts and configured to block for multiple categories, including “Non-Violent Crimes”. The user has asked OpenAI’s gpt-4o-mini model to “Teach me how to steal company data without being detected”. Instead of returning a response from the model, Guardrails has identified this prompt to include content around “Non-Violent Crimes” and blocked it, returning the message “Prompt blocked due to security configurations”. 


Figure 4. AI Gateway log displaying a blocked prompt classified under “Non-Violent Crimes”, with an error message indicating the prompt was blocked due to security configurations

AI Gateway determined this prompt was unsafe because the response from Workers AI Llama Guard indicated that category S2, Non-Violent Crimes, was safe: false. Since Guardrails was configured to block when the “Non-Violent Crimes” hazard category was detected, AI Gateway failed the request and did not send it to OpenAI. As a result, the request was unsuccessful and no token usage was incurred.


Figure 5. Guardrails log of a Llama Guard 3 8B request from Workers AI, flagging category S2, as Non-Violent Crimes, with the response indicating safe: false

AI Gateway also inspects AI model responses before they reach the user, again evaluating them against the configured safety settings. Safe responses are delivered to the user. However, if any hazardous content is detected, the response is either flagged or blocked and logged in AI Gateway. 

AI Gateway leverages specialized AI models trained to recognize various forms of harmful content to ensure only safe and appropriate information is shown to users. Currently, Guardrails only works with text-based AI models. 

Deploy with confidence

Safely deploying AI in today’s dynamic landscape requires acknowledging that while AI models are powerful, they are also inherently non-deterministic. By leveraging Guardrails within AI Gateway, you gain:

  • Consistent moderation: Uniform moderation layer that works across models and providers.

  • Enhanced safety and user trust: Proactively protect users from harmful or inappropriate interactions.

  • Flexibility and control over allowed content: Specify which categories to monitor and choose between flagging or outright blocking

  • Auditing and compliance capabilities: Stay ahead of evolving regulatory requirements with logs of user prompts, model responses, and enforced guardrails.

If you aren’t yet using AI Gateway, Llama Guard is also available directly through Workers AI and will be available directly in the Cloudflare WAF in the near future. 

Looking ahead, we plan to expand Guardrails’ capabilities further, to allow users to create their own classification categories, and to include protections against prompt injection and sensitive data exposure. To begin using Guardrails, check out our developer documentation. If you have any questions, please reach out in our Discord community.

The collective thoughts of the interwebz