Post Syndicated from corbet original https://lwn.net/Articles/1094762/
The Register looks
forward to the upcoming Fedora 45 release.
The biggest surprise is that Linux’s legacy in-kernel console – the
text-mode interface normally hidden beneath the GUI – has been
replaced with a software-controlled alternative.
The replacement is kmscon, a
userspace terminal emulator that has been in development for more
than a decade.
Post Syndicated from Talks at Google original https://www.youtube.com/watch?v=jezPMP_ODZk
Post Syndicated from Marco Duarte original https://aws.amazon.com/blogs/big-data/discover-and-govern-snowflake-data-using-sagemaker-unified-studio/
Many organizations operate in hybrid data environments where critical assets live in Snowflake while analytics workloads run on AWS, which can create governance gaps, discovery friction, and duplicated efforts when the two aren’t connected.
With Amazon SageMaker Unified Studio, you can govern data across Snowflake and AWS through its integrated catalog and AWS Glue Data Quality, a capability of AWS Glue. You connect directly to Snowflake tables without moving data, apply quality rules using AWS Glue Visual ETL, and publish validated assets to Amazon SageMaker Catalog, maintaining consistent governance across your entire distributed data estate.
Without this integration, cataloging Snowflake data requires building extraction pipelines, often taking days. With SageMaker Unified Studio connected to Snowflake, you can query, catalog, and validate the quality of federated data in 5–15 minutes. No data replication or custom ETL code required.
In this post, we show you how to connect Snowflake to Amazon SageMaker Unified Studio, register data assets in Amazon SageMaker Catalog, configure data quality validation using AWS Glue Visual ETL, and publish assets for unified collaboration. By following these steps, you enrich federated assets with data quality scores so that consumers across your organization can discover and trust the data, all while keeping it in Snowflake.
This solution integrates Snowflake with Amazon SageMaker Unified Studio for centralized data cataloging and quality validation.
The architecture uses an AWS Glue connection to federate the Snowflake catalog into Amazon SageMaker Unified Studio. Tables become available in the project catalog without complex storage configurations. You can query data directly using SQL analytics, publish datasets to Amazon SageMaker Catalog for organization-wide discovery, and apply data quality rules through AWS Glue Visual ETL pipelines.
The workflow consists of the following steps:
Figure 1: Architecture for federating Snowflake into SageMaker Unified Studio and validating data quality
To follow along, you need:
datazone: prefix.)Your AWS Glue job execution role requires specific permissions to interact with Amazon SageMaker Catalog.
1. Amazon SageMaker Catalog search and listing permissions: Attach a policy that allows the AWS Glue job to search and list assets in Amazon SageMaker Catalog.
2. Amazon SageMaker Catalog time series data posting permissions: Add permissions to post data quality metrics:
Configure the IAM role used by your AWS Glue job as a domain user. In the Amazon SageMaker console, navigate to your domain, choose Access management, and add the AWS Glue job execution IAM role as a domain user.
Add the AWS Glue job execution role as a project member with Owner permissions. Navigate to your project, go to Project settings > Members, and add the role.
For more information about IAM roles for AWS Glue, see the AWS Glue security documentation. For Amazon SageMaker Unified Studio permissions, refer to the Amazon SageMaker Unified Studio administrator guide.
The following sections walk you through connecting Snowflake to Amazon SageMaker Unified Studio and running data quality validation with results displayed in Amazon SageMaker Catalog.
First, gather your Snowflake connection details. You need a Snowflake account with tables or views created at the schema level within a database.
To obtain Snowflake connection information:
The Add Connection feature stores Snowflake connectivity details including credentials, server, and database information. Amazon SageMaker Unified Studio uses this connection to federate the Snowflake catalog through AWS Glue, so you can query data within minutes of setup.
You need an Amazon SageMaker Unified Studio domain and a project, which acts as a data producer project.
To create the Snowflake connection:
snowflake-connection.
After creating the connection, wait a few minutes for the federated connection to be established. Search within Amazon SageMaker Unified Studio for the database and created objects.
With the Snowflake connection established and the federated tables registered in AWS Glue Catalog, you’re now ready to query Snowflake data directly from Amazon SageMaker Unified Studio, without moving or replicating any data.
When you run a query in the Amazon SageMaker Unified Studio query editor against a federated Snowflake table, Amazon Athena runs the request. Athena is the underlying query engine integrated into Amazon SageMaker Unified Studio. Athena reads the table definition from AWS Glue Catalog, connects to Snowflake through the established connection, and pushes the query down for execution. Athena returns results directly to the query editor while Snowflake processes the data in place, and only the query results travel across the connection. Amazon SageMaker Unified Studio doesn’t copy data to S3 or any intermediate storage.
After you’ve validated that queries return the expected results, the next step is to publish this dataset to Amazon SageMaker Catalog, making it discoverable and shareable across your organization.
Now that your Snowflake connection is configured, you can publish your datasets to the Amazon SageMaker Catalog, making them discoverable and shareable across your organization.
Data assets in Amazon SageMaker Catalog are the cataloged representation of your data resources. They help teams discover, govern, and share data across your organization.
In this section, you create a data asset associated with a Snowflake table. This process transforms a technical Snowflake table into a cataloged resource enriched with business metadata.
To create a data source:
snowflake-connection.
To extract metadata and publish assets:
The following screenshot shows the asset inventory after the data source run completes.
At this point, you can enrich the business context by choosing Generate Descriptions. Amazon SageMaker Catalog analyzes the asset’s technical structure and generate:
The Snowflake enriched asset is now available to data consumers across your organization. Other users can discover it, subscribe to it, and consume it without data replication.
This section explains how to apply data quality validations to Snowflake data using AWS Glue Data Quality and visualize results in Amazon SageMaker Catalog.
Upload two files to an Amazon S3 bucket in the same AWS account where you run AWS Glue:
Copy both files to your AWS Glue assets S3 bucket in the transforms folder (s3://aws-glue-assets-<account-id>-<region>/transforms). AWS Glue Studio reads all JSON files from this folder to register custom visual transforms.
In the following sections, we walk you through the steps of building an ETL pipeline for data quality validation using AWS Glue Studio.
AWS Glue for Spark provides built-in support for reading from Snowflake data sources.
To create a new visual ETL job:
To add a Snowflake source:
The visual editor displays the Data source properties panel where you select your connection, database, and enter a custom query targeting your Snowflake table.
After establishing the Snowflake connection, configure the data quality evaluation step using the Data Quality Definition Language (DQDL).
To add data quality validation:
The preview of the data quality results from the ruleOutcomes node shows the outcomes of each rule.
To configure the custom transform:
The following image shows the complete job graph with the Datazone DQ Result Sink transform configured.
The visual editor displays four nodes connected sequentially: the Snowflake data source, the Evaluate Data Quality transform, the ruleOutcomes SelectFromCollection transform, and the Datazone DQ Result Sink transform.
To configure job parameters:
--additional-python-modulesboto3>=1.34.105
After the AWS Glue ETL job completes, you can view the data quality information directly in Amazon SageMaker Catalog. This is the key outcome of running data quality on a federated source: the asset gains quality scores and metadata without ever leaving Snowflake. This makes it trustworthy and ready for other teams across your organization to use. Data consumers can now discover this asset in Amazon SageMaker Catalog and evaluate its quality before subscribing, without needing direct access to Snowflake or running their own validation.
To view data quality results:
The following image shows the asset page in Amazon SageMaker Catalog with the data quality score populated.
The Data Quality tab shows an overall score of 100 and lists the rule set movies with a Passed result (1/1). This confirms that the data quality checks from AWS Glue posted successfully to Amazon SageMaker Catalog.
To avoid ongoing charges, remove the resources you created during this walkthrough:
s3://aws-glue-assets-<account-id>-<region>/transforms/ bucket.In this post, we showed you how to connect Snowflake to Amazon SageMaker Unified Studio for centralized data cataloging and quality validation. This approach maintains consistent governance without replicating data. Key benefits include:
To get started, open the Amazon SageMaker Unified Studio console. To learn more about related topics, see Cross-account lakehouse governance with Amazon S3 Tables and SageMaker Catalog and Get started with AWS Glue Data Quality dynamic rules for ETL pipelines.
Post Syndicated from The Atlantic original https://www.youtube.com/watch?v=kD0jIXXjABM
Post Syndicated from BeardedTinker original https://www.youtube.com/watch?v=lJit_lLGgYs
Post Syndicated from jake original https://lwn.net/Articles/1092553/
At the 2026 edition of FOSSY, Romeo
Solano gave a fast-paced, humorous presentation on what could have been a
rather boring topic: server encryption. There are a number of threats that
we face in today’s world, from criminals, government overreach, espionage,
and more, that can be thwarted with encryption. But encrypting data on a
system that may live elsewhere, without any access to its keyboard at boot
time, is rather more difficult than encrypting the disk of a laptop.
Solano described the problems and gave a tour of some of the solutions in
the talk.
Post Syndicated from Атанас Шиников original https://www.toest.bg/gospodut-na-svetovete-izvunzemni-i-islyam/
Като заклет читател на фантастика във всичките ѝ разновидности и жанрове прехвърлям през ума си възможни тематични връзки с вярата в Аллах. Имаше навремето един сериал – „Бойна звезда: Галактика“, римейк на по-стария от 70-те години на миналия век. В него човечеството е почти унищожено от расата на роботите сайлони, които в старата версия на сериала са творение на извънземна раса, а пък в последната – човешко. Интересното е, че в сериала военизираните роботи изповядват религия, която, противно на тази на хората, е строго монотеистична. Е, не се казва точно каква…
В една от любимите ми класики пък – „Хиперион“ на Дан Симънс, нещата стават една степен по-конкретни. Разказът се разгръща през XXVIII век и следва структурата на класическите „Кентърбърийски разкази“ на Джефри Чосър от XIV век. Един от разказвачите поклонници в романа е полковник Федман Касад, роден на Марс, в етническа група, която все още се определя като палестинци. Първият му сблъсък с радикалния ислям се случва на планетата Ком-Рияд. Името ѝ очевидно събира най-доброто от шиизма и сунизма, доколкото Ком, или Кум, днес е главен шиитски град в Иран, а пък Рияд е столицата на сунитска Саудитска Арабия. Там самопровъзгласил се нов пророк обявява джихад срещу господстващата политическа структура, известна като Хегемонията. Типичен екшън герой от добрите, Касад заявява, че Богът на исляма нито би одобрил, нито би позволил избиването на невинни – тук малко ми напомня на Ясер Арафат, когото гледах през 2001 г. в Дамаск да казва по телевизията, че
ислямът не насърчава убийството на нищо живо.
А вселената на Дан Симънс, колкото и владяна от човечеството, допуска наличието на нечовешки разумни раси. Предимно в миналото.
Ако се приземим обратно на Земята, някои от мрачните фантазии на Хауърд Лъвкрафт също съжителстват с вярата в Аллах или поне с негови последователи. Не се ли пръква зловещият Ал-Азиф („Некрономикон“), този магнум опус на окултната литература, в безумните видения на лудия арабин Абдул Алхазред от Йемен по времето на Омеядския халифат? Отстъпник от исляма, Алхазред се прекланя пред уродливите космически извънземни Йог-Сотот и Ктхулу и под руините на безименен пустинен град открива следи от древни космически раси, отдавна посещавали Земята. А пък смъртта му е описана от историка Ибн Халликан, според чийто разказ лудият арабин е погълнат от невидимо чудовище посред бял ден. Забавното е, че такъв историк действително съществува, живее през XIII век и пише история под формата на биографични речници със справки за живота на изтъкнати мюсюлмани.
Малко по-различен е случаят с „Дюн“ на Франк Хърбърт. Ако и светът на планетата Аракис да е антропоцентричен и извънземни – поне такива, каквито очаква популярната представа („нечовешка разумна форма на живот“) – да не съществуват там, имаме множество населени светове, които се подчиняват на трагичния пророчески образ на Пол Атреидски – Муад’Диб. Всъщност може да гледате на „Дюн“ като на наръчник за религиозната терминология на арабски. Махди, „Напътствания от Аллах“, е едно от прозвищата на Пол Атреидски, но и месианската фигура от края на времето в исляма. Джихад е свещената война (но и „върховно усилие“), водена от пустинните племена на фремените, напомнящи на воините на Пророка от VII век в пясъците на Арабия. Шай-Хулуд, пустинният червей, който произвежда космическата подправка меланж, идва от арабското „Нещо от вечността“ (шай’ хулуд). Лисан ал-гайб на арабски си е „езикът на неведомото“ и фремените го използват като название на пророка си. Примерите са десетки и за моя огромна изненада, когато през 1998 г. попаднах в тукашната арабистика, след като вече бях чел „Дюн“, открих, че доста плътно следват арабския. Една от любимите ми препратки е изразът би-лал кайфа, който фремените използват като „Амин!“. Той пък идва от арабското би-ла кайфа („без [да се пита] как!“), фраза, често свързвана в традиционното мюсюлманско богословие с фигурата на Абу л-Хасан ал-Ашари от IX–X век, който я използва, когато го питат как следва да се приемат и тълкуват Божиите качества в Корана. Ей така, без да се пита как.
Само че това е лесната част от разговора за вярата в Аллах в свят (или светове), където е възможно човешкото и извънземното да споделят една вселена. Защото става въпрос за фантастика. По-голямата част от нея може да носи етикета „научна“, но всички знаем, че става въпрос за фикция. Авторите не са мюсюлмани. Затова и могат да се отдадат на симпатична литературна игра, в която ислямът е вторичен, пък бил той и екзотичен играч. В този смисъл е лесно пластовете на въображаемия разказ да съвместят едновременно разговора за наличието на множество светове, разумен живот извън Земята и сферата на човешкото с едно изцяло въобразено, външно възприятие на мюсюлманския монотеизъм. В несериозността на този подход се крие и неговата безопасност.
Съвсем различно е да запретнем ръкави и да хванем (звездния) бик за рогата. Да надзърнем как мюсюлманите възприемат извънземни разумни форми на живот не през призмата на литературната фикция, а в сферата на религиозния закон (шари‘а), право (фикх) и богословието (калам). А с тях шега не бива. Който смята обратното, може да почете например за традиционните, предписани от Свещения закон, наказания за отрязването на ръката на крадеца в Корана („А на крадеца, мъж или жена, отсичайте ръцете за наказание, защото са присвоили – възмездие от Аллах“, 5:38), за отсъждането на убийство чрез хвърляне на камъни (раджм) по поводи като прелюбодейство. Или за богохулство, подигравка с Пророка, отстъпление (ридда) и оскверняване на Корана, да речем.
Сериозността, с която консервативните мюсюлмани възприемат свещените текстове и тяхната способност да определят дадени поведенчески модели, ясно личи в повлияни от Свещения закон държавни законодателни системи като например тези в Саудитска Арабия, Иран или Афганистан. Тъй че към всяка тема в обхвата на Свещения закон мюсюлманите следва да се отнасят с подобаваща сериозност. А не като Салман Рушди, който от 2022 г. носи върху лицето си (и не само) последствието от пренебрежителното отношение към Пратеника на Аллах, впрочем като далечен отзвук от фетвата на аятолах Хомейни срещу него през 1989 г.
Та нали съществуването на небесни тела, оттук и на други светове, е наблюдавано още от древността. Луна. Планети. Звезди. Там сигурно живее някой, тъй както Земята е обитаема от човека. От религиозна гледна точка пък, независимо коя е конкретната религия, е изключително важно да се знае дали приложимата религиозна рамка за човека на Земята би била валидна и за възможните разумни обитатели на небесните тела. В християнството дилемата влече след себе си въпроси около наличието на грях и съответно нуждата от изкупление. И дали Христос на земята е един за всички, включително и за извънземните, или се предполагат множество негови въплъщения?
В исляма съответно проблематична е валидността на Свещения закон (шари‘а) в неговата цялост за евентуални разумни извънземни форми на живот. Иначе казано, опростенчески, не просто дали има междузвездни „малки зелени човечета“, а важи ли за тях правилото за задължителната молитва (салат) пет пъти на ден. Или къде извършват задължителното за всеки мюсюлманин ритуално поклонение (хадж) в светите места на исляма. И дали пък евентуални разумни нечовешки форми на живот могат въобще да бъдат отговорни (мукаллаф) от юридическа гледна точка. Или пък, още една стъпка по-нататък, какво ще се случи с тях в Деня на възкресението (йаум ал-кийама) и какви евентуално ще бъдат Раят и Адът за тях – все сложни, но реални въпроси.
Ето го един философ – Ибн Рушд от XII век, познат ни като Авероес, когото тези дни превеждам. Той като че ли не е толкова скептичен. В своето съчинение за съвместимостта между вярата и разума разгръща доста интересен аргумент в полза на философията. Тя, казва Авероес, помага на хората да научат повече за Свещения закон, тоест за религията. Оттук и да забраниш философските книги е „угнетителство“ (зулм, абе направо зулум, както влиза в български през турски) спрямо най-достойните измежду хората и живите същества. Защото – казва той –
И колкото по-величаво е едно живо същество, толкова по-голямо нечестие е невежеството спрямо него. Съчинението е сред най-известните на Авероес. А тук очевидно говори за „достойни същества“ извън човешкия род. Знаем, че коментаторската традиция схваща тези му думи като обозначаващи ангелите и Аллах. Но може да натиснем педала на спекулацията и да кажем, че ако и Аллах да прозира в текста чрез позоваването на Коран 31:13 („Съдружаването е огромен гнет“), останалите живи същества могат да бъдат от всякакъв порядък. Разумни и достойни за признание. Нали?
И за да не си мислите, че си фантазирам, следва да кажа, че текстът на мюсюлманското Свещено писание не е толкова строго отричащ, колкото веднага и прибързано бихме предположили. Даже си е направо двусмислено благосклонен. Още в първата глава (сура) – „Откриващата“ (Ал-Фатиха), тази, която всеки мюсюлманин трябва да знае наизуст, защото е част от задължителната молитва, Аллах е наречен „Господа на световете“ (1:2). На Него „се подчинява всичко на небесата и на земята, доброволно или по принуда“ (3:83). Той е сътворил не само конете и мулетата, и магаретата „за да ги яздите и за украса“, но и „каквото не знаете“ (16:8). Тъй де, ако „каквото не знаете“ се появява в един и същи стих паралелно със земния добитък, може да се очакват и живи твари там някъде, където никой друг освен Аллах не знае. „Синовете на Адам“ са почетени и предпочетени от Аллах „да превъзхождат повечето от онези, които сътворихме“ (17:70). Със сигурност знаем, че измежду тези, които са сътворени и над които Аллах е предпочел човека, са ангелите и самият Иблис, известен още с прозвището Шейтан. Именно в това се състои и неговият бунт (2:34).
Един от най-любимите ми стихове обаче е 42:29:
И от Неговите знамения е сътворяването на небесата и на земята, и на тварите, които там е намножил. Той е способен да ги събере, ако пожелае.
Може да е неясно, но не е и отричащо. На небесата и на земята според някои тълкувания има твари, които сам Аллах е намножил. И това също не си го измислям. Тук „тварите“ са обозначени с думата дабба в арабския оригинал. Обикновено означава „добитък“, „добиче“. Разбира се, с по-общо значение на „твар“, „живо същество“, па понякога и „звяр“. Като в онези свещени текстове от Корана и Сунната, в които се говори за „Звяра от земята“ (Дабба мин ал-ард, „от земята едно животно“, Коран 27:82), който в края на времето, преди настъпването на Съдния ден, се появява почти като зверовете от виденията на библейския пророк Даниил или Откровението на св. Йоан Богослов.
За да не ме обвините, че нещо притурям върху традицията, има и големи тълкуватели, които мислят като мен. Вземете един Аз-Замахшари от XI век. Името идва от родното му място Замахшар в Иран, само че пътува много, както често правят учените тогава, че по някое време даже се установява в Мека. Толкова е начетен, че носи прозвището „Съсед, Приближен на Аллах“ (Джар Аллах). И в най-известния му коментар на Корана, т.нар. Ал-Кашшаф („Разкриващия“), той разсъждава също върху предполагаемите „небесни добичета“ от стих 42:29. Може ангелите – мир тям! – да се движат като птиците, пише Аз-Замахшари. А пък въобще не е невъзможно в небесата Аллах да е създал някаква небесна жива твар, която да ходи по тях, тъй както хората ходят по земята,
Пречист е Той, който създава различни видове същества, онова, което знаем, и онова, което не знаем!
Тази небесна жива твар си я нарича направо „животно“ (хайауан), откъдето на български е дошло хайван. Разбирайте, голям небесен хайван. Същото мнение се споделя и от друг голям коментатор, Фахр ад-Дин ар-Рази в „Ключове към неведомото“ (Мафатих ал-гайб), като тук „неведомото“ е същото като гайб в Лисан ал-гайб от „Дюн“. И при него откриваме темата за Аллах, който може да е създал „различни видове живи твари (хайвани!), които ходят по небесата така, както хората ходят по земята“. Да удължа и аз още малко интерпретативната нишка: ако в някои тълкувания е възможно „добиче“ (дабба) да обозначава и ангелите, и хората, и не само, то защо хайауан да изключва разумност?
Колегата му Ал-Куртуби („от Кордоба“) от XIII век обаче е по-скептичен. Няма такива неща в неговия коментар. Сигурно защото цитира много ранен коментар, може би първия запазен, този на Муджахид от VII – началото на VIII век, който бил казал, че „добичетата“ от въпросния стих са живи същества и се имат предвид ангелите и хората, и то на земята. Е, не сме оставени съвсем без никакви „извънземни“, доколкото за ангелите също се предполага, че обитават пространството над земята, някъде в небесата. Съществуват и други гледни точки – животните и насекомите в земното небе, гигантски животни в небето, някъде над видимата му част, че дори и животните в Рая.
В Коран 52:4 пък се говори за „посещавания Дом“. Много интересно, в признатия за авторитетен превод на български език от Цветан Теофанов имаме бележката към този стих, че „посещаваният дом“ тук е или храмът Кааба, или „небесното място, което ангелите обхождат“. Бележката е неслучайна, доколкото загатва за споменатите в достоверно предание от Пророка хиляди ангели, които се молят в свято място на небето, подобно на земната мюсюлманска светиня в Мека. Оттук пък и имало популярно предание, че по една свещена Кааба съществува във всяко едно от седемте небеса, където техните жители (ахл) ходят на поклонение. А пък „жителите“ може да бъдат не само ангели, нали? Коментатори добавят още, че „световете“ от Коран 1:2 могат да бъдат огромен брой, като някои предания говорят, че светът на хората и джиновете (не забравяйте, че те не са ангели, имат особен статус и са направени от огън според стих 15:27) е един от тях, ангелите живеят във втори, но извън това има още десетки хиляди, в които само Аллах знае кой живее. В стих 65:12 се добавя допълнителна възможност за други светове. „Аллах е, Който сътвори седем небеса, и от земята – също толкова“ допуска не само небесата да са седем, но и земите.
Ал-Куртуби, колкото и да е консервативен спрямо „небесните животни“, се опира на колегата си от XI век Ал-Мауарди, онзи същия, когото ИДИЛ използва като аргумент за властовата легитимност на Абу Бакр ал-Багдади, просто защото е един от най-известните политически теоретици на исляма. Ал-Куртуби очертава картина, при която седемте земи съществуват, подобно на небесата, една над друга, отделени от огромно разстояние, а във всяка от тях има обитатели (суккан). А пък тълкувателят Ал-Кауаши от XIII век добавя, че тъй както във всяко от небесата има ангели, така и във всяка от седемте земи има създадени хора (ахл), които имат свои собствени чудни качества. Че даже и според предание на Пророка във всяка земя имало по един Пророк като Мохамед, по един Адам, по един Нух (Ной), по един Ибрахим (Авраам) и пр. За всички, които се интересуват повече, по темата има чудесни неща на Шоайб Малик, Йорг Детерман и Парандис Таджбакш¹. За да не останете с впечатлението, че откриваме топлата вода, когато разлистим дебелите средновековни коментари на Корана и Сунната. Макар че откриването на топлата вода в някои квартали на София лятоска си е направо небесно откровение само по себе си.
Ала все пак трябва да сме почтени към изворовия материал, колкото и понякога да ни се ще да си изфантазираме нещо ей така, за забавление, с риск да си навлечем гнева на правоверните. Общото мнение е, че разумните същества във вселената – оттук и такива, които са религиозно задължени и отговорни (мукаллаф) по религиозния закон – са ангелите, хората и джиновете. Измежду тях ангелите и джиновете някак съответстват на дефиницията за „разумен, извънземен (донякъде)“ живот. Оттам нататък не са изключени възможности за наличие на множество светове, населени с техните обитатели, които евентуално да са разумни (оттук и отговорни по Свещения закон). Но както завършва всяко уважаващо себе си изказване в мюсюлманското право, Аллах знае най-добре!
(Следва продължение.)
В рубриката „Ориент кафе“ Атанас Шиников поднася любопитни теми, свързани не толкова с горещата политика, колкото с историята и културата на Близкия изток. А той, древен и днешен, е по-близко до нас и съвремието ни, отколкото си представяме.
Post Syndicated from corbet original https://lwn.net/Articles/1094720/
Security updates have been issued by AlmaLinux (kernel, kernel-rt, libkcapi, nginx, nginx:1.24, openssl, osbuild-composer, perl, perl:5.32, python-tornado, rsync, and rust), Debian (cjose and nginx), Fedora (environment-modules, erlang, GitPython, knot, perl-Authen-SASL, python-configargparse, ruby, rubygems, and sblim-sfcb), Oracle (firefox, git-lfs, gstreamer1-plugins-base, kernel, libkcapi, nginx, nginx:1.26, openssl, osbuild-composer, perl, perl-YAML-Syck, postgresql18, python-tornado, and rust), Red Hat (fence-agents, git-lfs, microcode_ctl, osbuild-composer, podman, python-pyasn1, and resource-agents), SUSE (389-ds, ant, bson-devel, chirp-20260911, docker, gimp, google-cloud-sap-agent, hauler, kernel, kimi-code, libpcap, python-GitPython, python310, syncthing, yast2-samba-client, and zstd-jni), and Ubuntu (aom, imagemagick, kitty, openssh, phpseclib, policykit-1, python-sql, python-webob, shibboleth-sp, simplesamlphp, snapcast, srt, and suricata-update).
Post Syndicated from Explosm.net original https://explosm.net/comics/bwow-chika-bwow
New Cyanide and Happiness Comic
Post Syndicated from The History Guy: History Deserves to Be Remembered original https://www.youtube.com/watch?v=y5PSqricp-U
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2026/09/fake-captcha-scams.html
New variant of an old scam: Use the framing of a CAPTCHA to get an unsuspecting user to download and run a malicious program.
Post Syndicated from xkcd.com original https://xkcd.com/3299/
Post Syndicated from Rishi Tripathy original https://aws.amazon.com/blogs/security/aws-sts-simplifies-session-token-size-limits-and-adds-session-token-size-monitoring/
AWS Security Token Service (AWS STS) has simplified session token size limits, giving you more room for your session policies and session tags. STS has replaced the packed policy size and the overall session token size limits with a single token size limit of 4,096 bytes. STS now reports session token size in API responses, Amazon CloudWatch metrics, and AWS CloudTrail events. By using STS, you can also generate session tokens of different sizes, so you can find the maximum token size that your infrastructure can support.
The 4,096-byte limit is the current maximum, not a permanent ceiling. AWS might increase the limit as new capabilities are added that require session tokens to carry more information.
In this post, you learn what has changed, what this change means for you, and what to do next.
AWS STS session-vending APIs, such as AssumeRole, AssumeRoleWithSAML, AssumeRoleWithWebIdentity, GetSessionToken, and GetFederationToken, return temporary security credentials: an access key ID, a secret access key, and a session token. This change governs the session token, the opaque string that STS creates from the session policies and tags you pass plus the context that AWS adds.
Three things have changed.
PackedPolicyTooLargeException, so you couldn’t tell which one you exceeded. STS now enforces a single limit: the assembled session token must fit within 4,096 bytes. The separate packed policy limit, which made failures hard to predict, has been removed. When a token exceeds the assembled session token limit, STS returns PackedPolicyTooLargeException. STS continues to use the same exception, so existing error-handling code works without an SDK update.SessionTokenSize (which reports the session token size in bytes) and SessionTokenUtilization (which reports the percentage of the 4,096-byte limit consumed). STS also returns PackedPolicySize in every successful response for backward compatibility. PackedPolicySize now reports the same value as SessionTokenUtilization, enabling applications that use older AWS SDK versions to monitor utilization through this field. These response fields are also recorded in CloudTrail events. In CloudWatch, SessionTokenSize and SessionTokenMaxSize (the enforced limit) are published in the AWS/STS namespace.MinimumSessionTokenSize is a new optional parameter on the STS session-vending APIs. You can use it to increase a session token to at least the size you specify, up to 4,096 bytes. Use the parameter to find the maximum token size your infrastructure can handle.| Behavior | Previously | Now |
| Limits enforced | Two: Packed policy size and assembled token size | One: Assembled session token size (4096 bytes) |
| Error on failure | PackedPolicyTooLargeException: The error message didn’t identify which of the two limits was exceeded. |
PackedPolicyTooLargeException: The updated message reports your session token size and the maximum allowed size, both in bytes. |
| Session token size visibility | Not reported |
API Response and AWS CloudTrail:
Amazon CloudWatch:
|
| Infrastructure testing | No mechanism | MinimumSessionTokenSize: Parameter on sesssion-vending APIs |
How this affects you depends on your situation. The following scenarios cover the most common cases.
MinimumSessionTokenSize to find the maximum token size your systems can handle. See the What to do next section for more details.PackedPolicyTooLargeException before: Some requests that previously failed now succeed under the single limit. Review any workarounds you put in place specifically to avoid token size errors and decide whether you still need them. General best practices still apply: consistent tag casing and reused tag values compress more efficiently, and concise session policies keep the assembled token smaller. No code change is required for error handling. AWS STS still returns PackedPolicyTooLargeException when the assembled session token exceeds the limit, the same exception STS returned before this change.varchar(2048) can’t hold a 4,096-byte token. Review where you persist or pass session tokens, and identify the maximum token size each system supports. The next section shows how to test this.We recommend the following three steps to prepare your systems for this change.
MinimumSessionTokenSize to find the maximum session token size each system in your infrastructure can handle. Knowing these limits helps you identify systems that might reject or truncate larger tokens. The 4,096-byte limit reflects today’s needs, not a permanent ceiling. It might grow as AWS introduces new capabilities such as additional context keys for new services, richer audit metadata, and larger cryptographic signatures as the industry transitions to post-quantum algorithms. Avoid hard-coding the current maximum into your systems and revisit any fixed size assumptions if the limit changes.Tip: AWS STS serializes and compresses your session policies and tags when assembling the token. Compression results vary based on the actual content, not just its length. Two sets of tags with identical character counts can produce different token sizes. This is why
MinimumSessionTokenSizeis a more reliable way to test your infrastructure than estimating from input length.
Start at 4,096 bytes to test against the largest possible token. If a system truncates or rejects it, lower the value to find the size your infrastructure supports, then raise that limit where you can. MinimumSessionTokenSize is available in the latest AWS SDK, AWS Command Line Interface (AWS CLI), and Tools for PowerShell versions. See the STS API Reference for details. If your AWS SDK or AWS CLI predates the parameter, update it to use this feature.
SessionTokenUtilization and SessionTokenSize from the response requires the latest AWS SDK version. You can also monitor token size through CloudWatch and CloudTrail without updating your SDK.SessionTokenSize and SessionTokenMaxSize in the AWS/STS namespace. Use them to build dashboards and set alarms. Set your alarm against the size limit you found during testing, not the 4,096-byte maximum. The maximum is the same for every account, so your own infrastructure limit is the one that matters.The following figure shows the SessionTokenMaxSize and SessionTokenSize metrics graphed in the CloudWatch console.
Figure 1: SessionTokenMaxSize and SessionTokenSizemetrics in the CloudWatch console
SessionTokenUtilization and SessionTokenSize for successful calls.PackedPolicySize in session-vending API responses and CloudTrail records for backward compatibility. The field now reports the same value as SessionTokenUtilization: the percentage of the 4,096-byte session token size limit consumed by the token. As a result, PackedPolicySize values might appear lower even when your token content has not changed.
If your SDK exposes SessionTokenUtilization, use that field because its name reflects the value’s current meaning. If an earlier SDK does not expose SessionTokenUtilization, use PackedPolicySize to monitor the same utilization percentage without updating the SDK. We recommend you monitor SessionTokenSize for the token size in bytes.
You now have more room for session tags, tag values, and session policies in your AWS sessions. AWS STS enforces a single 4,096-byte session token limit, returns a clearer error message when a token exceeds it, and reports token size so you can track growth proactively. Validate your token-handling systems with MinimumSessionTokenSize, and watch SessionTokenUtilization and SessionTokenSize for ongoing visibility.
If you have feedback about this post, submit comments in the Comments section below.
Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/connect-amazon-sagemaker-unified-studio-to-microsoft-power-bi-part-1-iam-identity-center-idc-based-domains/
Connecting Power BI to your Amazon SageMaker Unified Studio data catalogs typically required third-party bridges. These bridges added complexity and licensing costs. In this post, you create a direct connection using new authentication modes in the Amazon Athena ODBC driver, removing those dependencies entirely. If your organization uses Power BI as its business intelligence (BI) tool, your analysts can configure access to governed data in Amazon SageMaker Unified Studio without changing their tools or workflows. As an AWS alternative, Amazon Quick Sight provides serverless BI integration with Amazon SageMaker Unified Studio at pay-per-session pricing.
A previous post showed the connection method using a third-party ODBC-JDBC bridge. The Amazon Athena ODBC driver (version 2.2.0 and later) now supports Amazon SageMaker Unified Studio authentication directly, eliminating the need for customers to configure third-party bridge components previously required for this connection. This bridge also created additional components and required ongoing maintenance. The native connection simplifies the architecture by reducing these requirements.
UC Irvine, a top-ten U.S. public research university, consolidates student data from systems across multiple departments into a single governed repository that supports reporting, research, and analytics for decision-making at the strategic, tactical, and operational levels. Many of their analysts rely on Power BI to explore and visualize this governed data.
“Our users rely on Power BI for data visualization and reporting, but connecting to governed data in AWS previously required workarounds. The ODBC connection feature gives a direct path from Power BI into our SageMaker Unified Studio projects—no bridge software, no extra licensing, just a connection string and we’re ready to go.”
— Bernadette Theologidy, Manager, Student Analytics, UC Irvine
The Athena ODBC driver introduces two new authentication modes for SageMaker Unified Studio:
You connect Microsoft Power BI to Amazon SageMaker Unified Studio through Athena. The Athena ODBC driver supports using two connection methods that use these authentication modes:
Method 1: DSN-based (Athena Power BI connector): You configure an ODBC Data Source Name (DSN) and use the Athena connector in Power BI. This method supports DirectQuery and Import mode with both SageMakerBrowserIdc and SageMakerIam authentication.
Method 2: DSN-less (Power BI ODBC connector): You use the Power BI ODBC connector with a connection string, requiring no DSN configuration. This method supports Import mode only with SageMakerIam authentication. DirectQuery isn’t available because the Power BI ODBC connector doesn’t support it. The connection string in Power BI Desktop must match exactly the one on Power BI Service. Because the gateway runs as a Windows service without interactive browser access, both ends must use SageMakerIam.
| Feature | Method 1: DSN-based | Method 2: DSN-less |
| Power BI Connector | Amazon Athena connector | ODBC connector |
| Data connectivity mode | DirectQuery and Import | Import only |
| Requires DSN configuration | Yes | No |
| Data freshness | Real-time (DirectQuery) or scheduled (Import) | Scheduled refresh only |
| Authentication types | SageMakerIam and SageMakerBrowserIdc | SageMakerIam only |
| Domain types supported | IAM-based and IDC-based | IAM-based and IDC-based |
| Best for | Dashboards requiring live data | Scenarios where DSN management is not possible or scheduled refresh is acceptable |
This is Part 1 of a two-part series. This post covers IDC-based domains using both connection methods. Part 2 covers IAM-based domains.
In this walkthrough, you take the role of a data analyst at an energy company. You need to understand the current state and future direction of the U.S. power generation fleet using the Public Utility Data Liberation Project, available on the Registry of Open Data on AWS. Our goal is to analyze generation capacity and identify where new investment is flowing. We connect Power BI to Athena through Amazon SageMaker Unified Studio and query the EIA-860 generators dataset directly from our data catalog. The result is a single visualization that reveals the energy transition.
The following diagram illustrates the solution architecture for connecting Power BI to Amazon SageMaker Unified Studio through Amazon Athena.
Figure 1: Architecture diagram
The following architecture demonstrates a six-step workflow.
On the AWS side, Athena queries the data catalog managed by AWS Glue Data Catalog. The catalog references data stored in Amazon Simple Storage Service (Amazon S3). An Amazon SageMaker Unified Studio project governs all access.
In an IDC-based domain (covered in this post), Power BI Desktop uses SageMakerBrowserIdc for Method 1 and SageMakerIam for Method 2. Power BI Desktop can run on-premises or on an EC2 instance. The gateway always uses SageMakerIam (it runs as a Windows service without browser access) and authenticates using instance profile credentials, which rotate automatically. The gateway can only query data within projects where its IAM role has been added as a member. For IAM-based domains, see Part 2.
Before connecting Power BI to Amazon SageMaker Unified Studio, verify that your environment meets these requirements:
You need an Amazon SageMaker Unified Studio project with data assets. For detailed instructions, refer to the Amazon SageMaker Unified Studio User Guide.
The following screenshot shows the Amazon SageMaker Unified Studio project Query Editor interface, which runs a preview query against the EIA-860 generators dataset.
Figure 2: SageMaker Unified Studio project with the EIA-860 generators dataset available in the data catalog
This method uses the Amazon Athena Power BI connector with an ODBC Data Source Name (DSN), supporting DirectQuery and Import mode.
You configure Power BI Desktop to connect to your data assets in Amazon SageMaker Unified Studio using the SageMakerBrowserIdc authentication mode. The driver opens a browser window and authenticates through IAM Identity Center (and your external identity provider, if configured).
Your single sign-on (SSO) user needs project-level access to query data with Athena. Verify your user is listed as a project member or add it by following Add project members in the Amazon SageMaker Unified Studio User Guide.
The following screenshot shows the SageMaker Unified Studio project user management page, where project owners can add or remove project users and roles.
Figure 3: Members of a SageMaker Unified Studio project
Gather the following values from your Amazon SageMaker Unified Studio project:
The following screenshot shows the Amazon SageMaker Unified Studio project overview page, where you can copy these details.
Figure 4: ODBC connection details
Create a System DSN using the Amazon Athena ODBC driver. For the general DSN creation steps, see Configuring a data source name on Windows in the Amazon Athena User Guide.
Enter the following values:
| Field | Value |
| Data Source Name | Name your datasource (for example, pbi-idcdomain) |
| Region | The AWS Region where your Amazon SageMaker domain is provisioned (for example, us-east-1) |
| Catalog | AwsDataCatalog |
| Database | default |
| Workgroup | Your Athena workgroup name (for example, workgroup-abcdefghij-klmexample) |
In the Authentication Options, configure the following values:
| Field | Value |
| Authentication Type | SageMakerBrowserIdc |
| SSO Start URL | IAM Identity Center entry point (for example, https://identitycenter.amazonaws.com/ssoins-0example) |
| SSO Region | Region of IAM Identity Center (for example, us-east-1) |
| SageMaker Domain ID | dzd-123456example |
| SageMaker Project ID | abcd12example |
| SageMaker Domain Region | Region of your Amazon SageMaker Unified Studio project (for example, us-east-1) |
Choose OK, then Test to verify the connection. Choose Allow Access when prompted by the browser.
The following screenshot shows the consent prompt.
Figure 5: Browser consent prompt
The following screenshot shows the successful connection test.
Figure 6: Successful connection test in the ODBC DSN configuration with SageMakerBrowserIdc authentication
With the DSN configured, you can connect Power BI Desktop to your data catalog and load the generators dataset.
pbi-idcdomain.The following screenshot shows Power BI Desktop successfully connected to the AWS data catalog.
Figure 7: Power BI Desktop connected to the data catalog with the generators table loaded using SageMakerBrowserIdc authentication
You can create a dashboard to visualize U.S. power generation data. To create a visualization, complete the following steps:
technology_description to the Y-Axis.capacity_mw to the X-Axis (automatically summed).operational_status to the Legend field.generation-idcdomain) and choose Save.
Figure 8: Power BI Desktop report using the EIA-860 generators dataset
After publishing, the report structure is available on Power BI Service.
In this method, you use the Power BI ODBC connector with a connection string (no DSN required). This method supports Import mode only and SageMakerIam authentication. Because the gateway cannot perform browser authentication, both Desktop and gateway must use SageMakerIam. If your workflow requires SageMakerBrowserIdc, use Method 1.
If your machine already has AWS credentials through another method in the default credential provider chain, skip the following setup.
Create a custom permission set named SageMakerDataAnalyst in IAM Identity Center with the following inline policy. For detailed steps, see Create a permission set in the AWS IAM Identity Center User Guide.
Assign your user to this permission set for the AWS account containing your SageMaker Unified Studio domain. Then configure your AWS Command Line Interface (AWS CLI) SSO profile by running aws configure sso. For the full CLI configuration walkthrough with detailed steps, see Part 2. After your profile is configured, run aws sso login to authenticate.
The IAM identity providing credentials needs both domain-level and project-level access to query data through Athena.
Figure 9: List of users of your SageMaker Unified Studio domain including the IAM identity
Figure 10: Members of a SageMaker Unified Studio project including the IAM identity
Gather the following connection values from your Amazon SageMaker Unified Studio project:
Figure 11: ODBC connection string on the SageMaker Unified Studio project overview
With the configuration parameters of your project, you can connect Power BI Desktop to your data catalog and load the generators dataset.
Driver={Amazon Athena ODBC (x64)};AwsRegion=us-east-1;Catalog=AwsDataCatalog;Schema=default;Workgroup=workgroup-abcdefghij-klmexample;SageMakerDomainId= dzd-123456example;SageMakerProjectId= abcd12example;SageMakerDomainRegion=us-east-1;AuthenticationType=SageMakerIam;When publishing, name your report generation-idcdomain-dsnless.
After creating your reports in Power BI Desktop, configure the on-premises data gateway to view your report on Power BI Service.
You can configure the gateway using either a DSN or a DSN-less connection string, matching the method you used in Power BI Desktop.
Create an IAM role for the EC2 instance that will host your Power BI gateway. Name the role pbi-gateway-role (or a name of your choice). The role must use EC2 as the trusted entity and include the following inline policy:
Attach this role to your Power BI Gateway EC2 instance. For detailed steps on creating and attaching an IAM role to an EC2 instance, refer to IAM roles for Amazon EC2 in the Amazon EC2 User Guide.
The gateway IAM role needs project-level access to query data through Athena.
The following screenshot, from the Amazon SageMaker page of the AWS Management Console, shows the list of users of your Amazon SageMaker Unified Studio domain, including the IAM gateway role.
Figure 12: List of users of your SageMaker Unified Studio domain including the IAM gateway role
Add the IAM pbi-gateway-role role as a project member: see Add project members in the Amazon SageMaker Unified Studio User Guide.
The following screenshot shows the Amazon SageMaker Unified Studio project user management page listing the project members.
Figure 13: Members of a SageMaker Unified Studio project including the IAM gateway role
How you configure the data source depends on the method you used in Power BI Desktop.
Method 1 (DSN-based)
Configure a System DSN on the gateway EC2 instance following the same ODBC DSN steps described in Method 1. When configuring, make sure that:
pbi-idcdomain)Method 2 (DSN-less)
No configuration is needed on the gateway machine itself. You configure the data source directly in Power BI Service.
To view your report, complete the following steps:
generation-idcdomain) and choose the More options icon (three dots).pbi-idcdomain).The next step depends on the method that you chose:
pbi-idcdomain) that matches exactly the one configured on Power BI Desktop.Next, continue with the configuration:
pbi-idcdomain).generation-idcdomain).The following screenshot shows a Power BI report on Power BI Service.
Figure 14: Power BI report on Power BI Service
You can now see your report online with the data from your Amazon SageMaker Unified Studio project.
To avoid additional charges after testing, delete the Amazon SageMaker Unified Studio domain and EC2 instances. Refer to Delete domains and Terminate Instances for instructions.
In this post, you connected Microsoft Power BI to Amazon SageMaker Unified Studio using an IDC-based domain with both DSN-based and DSN-less methods. This provides a direct connection, with no third-party licensing, that maintains data governance. In Part 2, we cover IAM-based domains.
You can automate many steps of this process. For information about automating DSN creation on the Power BI Gateway or Service, refer to How ENGIE automates the deployment of Amazon Athena data sources on Microsoft Power BI. If you don’t want users adding the gateway IAM role directly, you can create a custom blueprint as a self-service tool for gateway role addition. The blueprint uses a ProjectMembership resource with a configurable parameter that project owners can activate at project creation, automatically adding the gateway role as a project contributor.
For additional best practices, refer to the Using Microsoft Power BI with the AWS Cloud Whitepaper. To learn more, visit Amazon SageMaker Unified Studio and Amazon Athena.
Post Syndicated from Ramesh H Singh original https://aws.amazon.com/blogs/big-data/connect-amazon-sagemaker-unified-studio-to-microsoft-power-bi-part-2-iam-based-domains/
In Part 1 of this series, we connected Microsoft Power BI to Amazon SageMaker Unified Studio using an IAM Identity Center (IDC)-based domain. The Amazon Athena ODBC driver (version 2.2.0 and later) supports Amazon SageMaker Unified Studio authentication natively, removing the third-party ODBC-JDBC bridge previously required. We walked through both the DSN-based connection and the DSN-less connection, from Power BI Desktop through the on-premises data gateway to Power BI Service, where report viewers access published dashboards.
In this post, you create the same direct connection using an AWS Identity and Access Management (IAM)-based domain. The walkthrough covers the same two connection methods. The differences are the Amazon SageMaker Unified Studio console navigation paths, the configuration values, and an additional administrator setup that provides AWS credentials through AWS IAM Identity Center. This is Part 2 of a two-part series. For a detailed comparison of the two connection methods, see Part 1.
The architecture is the same as the previous post (see the architecture diagram and walkthrough scenario in Part 1). Power BI Desktop connects to Amazon Athena through the ODBC driver and the Amazon SageMaker Unified Studio project governs all data access. At the same time, the on-premises data gateway on an Amazon Elastic Compute Cloud (Amazon EC2) instance bridges the connection to Power BI Service so report viewers can access published dashboards.
The difference is in authentication: An IAM-based domain uses SageMakerIam authentication for both connection methods. The driver retrieves credentials from the AWS default credential provider chain. For this walkthrough, AWS IAM Identity Center provides those credentials through a custom permission set. Power BI Desktop can run on-premises or on an EC2 instance in the AWS Cloud. The gateway EC2 instance authenticates using its attached IAM role.
Complete the prerequisites from Part 1. Additionally, you need:
The following screenshot shows the Amazon SageMaker Unified Studio (IAM-based domain) project Query Editor interface. It runs a preview query on the EIA-860 generators dataset.
Figure 1: SageMaker Unified Studio (IAM-based domain) project with the EIA-860 generators dataset available in the data catalog
This section configures AWS IAM Identity Center to provide credentials for the SageMakerIam authentication mode. It applies to Method 1 (IAM-based domain) and Method 2 (both domain types). If your machine already has AWS credentials available through another method in the default credential provider chain, you can skip this section and proceed directly to the method of your choice. For the full list of credential sources, refer to Credential providers in the AWS SDKs and Tools Reference Guide.
Create a custom permission set named SageMakerDataAnalyst in IAM Identity Center with the following inline policy. For detailed steps, see Create a permission set in the AWS IAM Identity Center User Guide.
The "Resource": "*" is required because these API actions do not support resource-level permissions. For more information, see Actions, resources, and condition keys for Amazon DataZone.
This doesn’t grant broad access to your data. These are read-only metadata actions that allow the ODBC driver to discover connection details and retrieve temporary Athena credentials. The actual data access is governed by Amazon SageMaker Unified Studio project membership: Users can only query data within projects where they have been explicitly added as members. The Amazon SageMaker Unified Studio project IAM role provides Athena and Amazon S3 permissions separately.
To assign users or groups to the target AWS account, complete the following steps:
To configure the AWS IAM Identity Center profile, run the following command in your terminal on Windows:
When prompted, enter the following values:
| Prompt | Value |
| SSO session name | For example, smus |
| SSO start URL | The IDC issuer URL. For example, https://identitycenter.amazonaws.com/ssoins-0example |
| SSO region | The SSO Region. For example, us-east-1 |
| SSO registration scopes | sso:account:access |
A browser window opens for authentication. After authentication, select your account and the SageMakerDataAnalyst role.
The following screenshots show the consent window and the successful authentication message.
Figure 2: Browser consent prompt
Figure 3: Browser authentication successful message
When prompted, enter the following values:
| Prompt | Value |
| Default client Region | None |
| CLI default output format | None |
| Profile Name | Change value by default |
The resulting ~/.aws/config file should look like the following:
To verify that your SSO profile is working correctly, run the following command:
You should receive a response like the following:
For daily use, no passwords or EC2 instance roles are required. When your SSO session expires, run the following command to quickly refresh it:
The IAM identity providing credentials to the ODBC driver needs project-level access to query data through Athena. If you completed the administrator setup, this is the SSO role associated with your permission set (for example, AWSReservedSSO_SageMakerDataAnalyst_1234example). If you’re using another credential source, add the IAM role or user that provides those credentials. For detailed steps, see Managing users for IAM-based domains in the Amazon SageMaker Unified Studio Administrator Guide.
The following screenshot shows the Amazon SageMaker Unified Studio domain management page, which lists the members in a project.
Figure 4: List of members of your SageMaker Unified Studio project
To get the parameters that you need to authenticate, complete these steps:
https://identitycenter.amazonaws.com/ssoins-0exampleThe following screenshot shows the Amazon SageMaker Unified Studio domain management page with SSO connection details.
Figure 5: AWS IAM Identity Center information
Figure 6: User icon with the Region information
In this method, you configure an ODBC Data Source Name (DSN) and use the Amazon Athena connector in Power BI. This method uses SageMakerIam authentication mode and supports both DirectQuery and Import mode.
This section covers IAM-based domains. For IDC-based domains, see Part 1.
Before configuring the ODBC DSN, gather the following connection values from your Amazon SageMaker Unified Studio project:
The following screenshot shows the Amazon SageMaker Unified Studio project overview page, which provides the project details to copy.
Figure 7: Project details with SageMaker domain ID, SageMaker project ID, Region, and Athena workgroup
Create a System DSN using the Amazon Athena ODBC driver. For the general DSN creation steps, see Configuring a data source name on Windows in the Amazon Athena User Guide. Enter the following values:
| Field | Value |
| Data Source Name | Name your datasource (for example, pbi-iamdomain) |
| Region | The AWS Region where your Amazon SageMaker domain is provisioned (for example, us-east-1) |
| Catalog | AwsDataCatalog |
| Database | default |
| Workgroup | Your Athena workgroup name (for example, workgroup-abcdefghij-klmexample) |
In the Authentication Options, configure the following values:
| Field | Value |
| Authentication Type | SageMakerIam |
| SageMaker Domain ID | dzd-123456example |
| SageMaker Project ID | abcd12example |
| SageMaker Region | Region of your SageMaker Unified Studio project (for example, us-east-1) |
Choose OK, then Test to verify the connection. Choose Allow Access when prompted by the browser.
The following screenshot shows the successful connection test.
Figure 8: Successful connection test in the ODBC DSN configuration with SageMakerIam authentication
With the DSN configured, you can connect Power BI Desktop to your data catalog and load the generators dataset.
pbi-iamdomain.The following screenshot shows Power BI Desktop successfully connected to the data catalog.
Figure 9: Power BI Desktop connected to the data catalog with the generators table loaded using SageMakerIam authentication
You can create a dashboard to visualize U.S. power generation data. To create a visualization, complete the following steps:
technology_description to the Y-Axis.capacity_mw to the X-Axis (automatically summed).operational_status to the Legend field.generation-iamdomain) and choose Save.The following screenshot shows the Power BI dashboard with U.S. power generation data.
Figure 10: Power BI dashboard with U.S. power generation data
After you publish, the report structure becomes available on Microsoft Power BI Service.
In this method, you use the Power BI ODBC connector with a connection string (no DSN required). This method supports Import mode only and SageMakerIam authentication. Because the gateway can’t perform browser authentication and connection strings need to match, both Desktop and gateway must use SageMakerIam.
This section covers IAM-based domains. For IDC-based domains, see Part 1.
Gather the following connection values from your Amazon SageMaker Unified Studio project:
The following screenshot shows the Amazon SageMaker Unified Studio project overview page with the ODBC connection string to copy.
Figure 11: Project details with ODBC connection string
With the configuration parameters of your project, you can connect Power BI Desktop to your data catalog and load the generators dataset.
Driver={Amazon Athena ODBC (x64)};AwsRegion=us-east-1;Catalog=AwsDataCatalog;Schema=default;Workgroup=workgroup-abcdefghij-klmexample;SageMakerDomainId= dzd-123456example;SageMakerProjectId= abcd12example;SageMakerDomainRegion=us-east-1;AuthenticationType=SageMakerIam;When publishing, name your report generation-iamdomain-dsnless.
After creating your reports in Power BI Desktop, configure the on-premises data gateway to view your report on Power BI Service.
You can configure the gateway using either a DSN or a DSN-less connection string, matching the method you used in Power BI Desktop.
Create an IAM role for the EC2 instance that will host your Power BI gateway. Name the role pbi-gateway-role (or a name of your choice). The role must use EC2 as the trusted entity and include the following inline policy:
Attach this role to your Power BI Gateway EC2 instance. For detailed steps on creating and attaching an IAM role to an EC2 instance, refer to IAM roles for Amazon EC2 in the Amazon EC2 User Guide.
The gateway IAM role needs project-level access to query data through Athena. The steps to add the role differ depending on your domain type.
pbi-gateway-role).The following screenshot shows the Amazon SageMaker Unified Studio project domain management page with options to add members to a project.
Figure 12: List of members of a SageMaker Unified Studio project with the IAM gateway role
How you configure the data source depends on the method you used in Power BI Desktop.
Method 1 (DSN-based)
Configure a System DSN on the gateway EC2 instance following the same ODBC DSN steps described in Method 1. When configuring, make sure that:
pbi-iamdomain).Method 2 (DSN-less)
No configuration is needed on the gateway machine itself. You configure the data source directly in Power BI Service.
To view your report, complete the following steps:
generation-iamdomain) and choose the More options icon (three dots).pbi-iamdomain).The next step depends on the method that you chose:
pbi-iamdomain) that matches exactly the one configured on Power BI Desktop.Next, continue with the configuration:
pbi-iamdomain).generation-iamdomain).The following screenshot shows a report on Power BI Service.
Figure 13: Power BI report on Power BI Service
You can now see your report online with the data from your Amazon SageMaker Unified Studio project.
To avoid additional charges after testing, delete the Amazon SageMaker Unified Studio domain and EC2 instances. Refer to Delete domains and Terminate Instances for instructions.
In this two-part series, you connected Power BI to Amazon SageMaker Unified Studio through Amazon Athena. Part 1 covered IDC-based domains. This post covered IAM-based domains using SageMakerIam authentication. This provides a direct connection path, with no third-party licensing, while maintaining data governance and security.
You can automate many steps of this process. For information about automating DSN creation on the Power BI Gateway or Service, refer to How ENGIE automates the deployment of Amazon Athena data sources on Microsoft Power BI. If you don’t want users adding the gateway IAM role directly, you can create a custom blueprint as a self-service tool for gateway role addition. The blueprint uses a ProjectMembership resource with a configurable parameter that project owners can activate at project creation, automatically adding the gateway role as a project contributor.
For additional best practices, refer to the Using Microsoft Power BI with the AWS Cloud Whitepaper. To learn more, visit Amazon SageMaker Unified Studio and Amazon Athena.
Post Syndicated from The Atlantic original https://www.youtube.com/shorts/SNX5BrDS2CI
Post Syndicated from Abrom Douglas original https://aws.amazon.com/blogs/security/architecting-resilient-authentication-with-amazon-cognito-multi-region-replication/
Your consumer identity and access management (CIAM) system is the foundation of your customer experience. It’s how users sign in, access services, and engage with your applications. As your business scales across geographies, ensuring authentication is always available becomes a core architectural requirement. However, building multi-Region authentication has traditionally required complex custom replication solutions that synchronize user data, manage consistency, and handle failover, all adding significant operational overhead. Amazon Cognito simplifies this with multi-Region replication (MRR), which automatically replicates user pools across AWS Regions with near-real-time synchronization, built-in failover, and seamless sign-in, while keeping operational complexity and costs optimized.
In this post, we show you how to prepare your user pool for MRR, provide architectural decisions and reference architectures for business to consumer (B2C), business to business (B2B), and machine to machine (M2M) use cases, and practical guidance on implementing failover strategies.
Amazon Cognito MRR creates a replica user pool in another AWS Region (a replica Region) that shares the same user pool ID as your primary user pool. The primary user pool (the user pool in your primary Region) remains authoritative, and its configurations (app client IDs, client secrets), user data (attributes, hashed credentials, group memberships), and external identity provider (IdP) settings are replicated to the replica with eventual consistency.
The user pool in the replica Region (replica user pool) supports user authentication operations (such as sign-in, token generation and revocation) and read-only operations towards user pool configurations and user attributes (such as list users and groups and describe user pool configurations). Write operations against user pool configurations and updating user attributes aren’t enabled in the replica user pool and can only be made in the primary user pool. Amazon Cognito returns an Action temporarily unavailable error when using managed login, or an OperationNotEnabledException when using an AWS SDK for those operations. See Supported API operations in secondary Regions for a list of API operations supported in replica Regions.
JSON web tokens (JWTs) and active sessions are interoperable between Regions; for example, a refresh token issued by the primary Region is accepted in the replica Region to retrieve new ID and access tokens.
While this post primarily focuses on MRR architecture patterns and considerations, you can visit the following posts to learn more about MRR basics and the next-generation infrastructure behind it:
In this section, we show you architectural decisions and preparation work for a successful MRR deployment.
Without MRR enabled, data is encrypted at rest with an AWS owned AWS Key Management Service (AWS KMS) key and encrypted in transit with TLS 1.2 and TLS 1.3 with hybrid post-quantum key exchange. Before enabling MRR, you must configure your user pool to use a customer managed key. This must be a symmetric multi-Region AWS KMS customer managed key.
Architectural considerations for your KMS key:
kms:EncryptionContext:aws:cognito-idp:<userpool-arn>. See the data encryption section in the Amazon Cognito developer guide for a full example key policy.In each ID and access tokens, Amazon Cognito includes a default Issuer claim in the JWT payload, referred as iss, to represent the identity provider that issued the token. The OpenID Connect (OIDC) specification dictates that the iss format must be a URL that uses https scheme and publishes a JSON metadata document about the identity provider available at the <iss>/.well-known/openid-configuration path. The metadata document must also include the JSON Web Key (JWK) document in the <iss>/.well-known/jwks.json path, which contains the signing keys to validate the token signatures for its integrity.
The original issuer type follows the format as https://cognito-idp.<region>.amazonaws.com/<userpool_id>. However, this issuer URL format and the OIDC well-known metadata are regional resources. As part of the MRR capability, Cognito introduces a new multi-Region OIDC issuer type, the updated issuer, and follows the format as https://issuer-cognito-idp.<region>.amazonaws.com/<userpool_id>. This new updated issuer type replaces the original single Region type and maintains availability of the issuer endpoint regardless of the state of primary or replica Region.
Based on the issuer URL format you select, your OpenID Connect discovery endpoint is hosted at <iss>/.well-known/openid-configuration and your JSON Web Key Set (JWKS) endpoint at <iss>/.well-known/jwks.json. Both original type and updated type are supported with the Amazon Cognito MRR capability. You can change the issuer type at any stage in your MRR journey, and the newly issued tokens, including those generated by refresh tokens, will reflect the most current issuer type configurations.
We recommend adopting the updated issuer type. With the updated issuer type, the OpenID Connect discovery document and JWKS endpoint remain consistent and available regardless of which Region is servicing requests. This means your applications can always fetch signing keys for token verification, even during a regional impairment.
To adopt the updated issuer type, update your applications and downstream dependencies to validate against the new updated iss value. If you use the aws-jwt-verify library, update to v5.2.1 or later that supports updated issuer type. Plan this as a coordinated deployment; existing ID and access tokens with the original issuer type remain valid and accepted by Amazon Cognito endpoints until they expire. When using an existing refresh token to exchange for a new set of ID and access tokens, new tokens always carry the current issuer format configuration at the time of token refresh operation, providing interoperability across two issuer formats.
If you can’t immediately adopt the multi-Region issuer—for example, because downstream services or third-party integrations validate the iss claim against a hard-coded original format pattern—you can enable MRR while continuing to use the original issuer type. However, in this configuration the OIDC discovery endpoint and JWKS endpoint are tied to a single Region and might be unavailable during a regional impairment. Your multi-Region application might not be able to fetch public keys dynamically and validate token signatures. To mitigate this, it’s a good practice to implement a JWKS caching strategy in your token verification layer. Cache the signing keys locally (respecting the Cache-Control headers) so your applications can continue to validate tokens using cached keys when the JWKS endpoint is unreachable. This approach lets you benefit from MRR for user authentication while maintaining token verification continuity until you’re ready to complete the issuer migration. To learn more about the original and updated issuer types, see the Amazon Cognito user pools as an OIDC issuer section of the developer guide.
Amazon Cognito user pools support several integrations with AWS services for extended customization functionalities. Those AWS services are regional services and must be configured independently in the replica Region, including:
Use infrastructure-as-code tools like AWS CloudFormation or AWS Cloud Development Kit (AWS CDK) to maintain consistent configurations and deployments across Regions and environments. You should also monitor for any configuration drifts between assets.
For authentication use cases that rely on managed login and OAuth 2.0 endpoints—including federated authentication and M2M authorization—Amazon Cognito supports automatic failover to the replica Region with an Amazon Route 53 health check. Cognito uses the health status of Route 53 health check to control whether traffic routes to the primary or replica user pool. The health check can be set up to monitor the health of an endpoint, a CloudWatch alarm, or a calculated number of other health checks, so you determine what triggers a healthy or unhealthy state and can adjust traffic routing as needed.
Both the Amazon Cognito prefix domain (for example, auth.us-east-1.amazoncognito.com) and custom domain (for example, auth.example.com) support automatic domain failover. Your domain serves as the single entry point for the user pool OAuth 2.0 endpoints and directs traffic to the managed login pages. Cognito automatically fails over domain traffic to the replica Region when a Route 53 health check becomes unhealthy and fails back to the primary Region when the check is healthy. You don’t need to create another prefix domain in the replica user pool for failover use cases.
With the automatic failover capability, you can use a single domain to serve external IdP configurations, including redirect URIs and SAML assertion consumer URLs. For example, use https://auth.example.com/saml2/logout to send SAML 2.0 sign-out responses. Because the domain can serve traffic to both the primary and replica Regions and remains unchanged across Regions, your external IdP configurations stay consistent across Regions, and existing federated users continue to authenticate without disruption. This means that you can enable MRR without having to contact external IdP admins to update configurations; all existing configurations will continue to work.
For SDK-based authentication use cases without managed login, a custom domain isn’t strictly required. We recommend configuring a custom endpoint for SDK requests to simplify failover orchestration, so you don’t have to modify the Region parameter in the SDK configuration. Behind your custom endpoint, you can use the same Route 53 health check or a custom load balancing strategy to proxy API requests to primary or replica Region endpoints. You might also consider load balancing user authentication traffic, by referring to an X-Amz-Target HTTP header (for example, X-Amz-Target: AWSCognitoIdentityProviderService.InitiateAuth), to both the primary and replica Regions, while keeping user sign-up operations in the primary Region. If you use both managed login and SDK authentication in the same user pool, you can consider using the custom domain as the custom endpoint of the SDK for a streamlined operation, where Route 53 health check initiates failover and failback between the primary and replica Regions.
Time-Based One-Time Password (TOTP) MFA isn’t supported in replica user pools. Users configured to use TOTP MFA must authenticate through the primary Region. If your application relies on TOTP as a second factor, this limitation requires careful planning because you want to enable an alternative MFA for your users, such as SMS OTP, email OTP, or passkey.
When you activate a replica user pool, you gain a separate set of default quotas in the replica Region. Previously reserved higher quotas for your user pool in the primary Region aren’t carried over to the replica Region.
When selecting a replica Region for your user pool, consider your organization’s data sovereignty and residency requirements, as user identity data will be replicated to and stored in that Region. For guidance on navigating compliance, continuity, and control obligations that may influence your Region selection, see Practical digital sovereignty: Navigating the pillars of compliance, continuity, and control.
In this section, we show you reference architectures for common authentication patterns using the Amazon Cognito MRR capability. Each architecture demonstrates how Cognito MRR works with different authentication use cases.
Amazon Cognito managed login provides a fully managed authentication UI that handles sign-in, sign-up, and federation flows. With MRR, managed login endpoints are served from the healthy user pool based on your Route 53 health check configuration. Managed login also includes OAuth 2.0 endpoints and can be used with local Cognito accounts and federated users. Figure 1 depicts a reference architecture for using managed login to authenticate Cognito users.
Figure 1: Cognito MRR reference architecture for managed login and federation use cases
When using Amazon Cognito with managed login, the process flow is:
In an M2M architecture, services authenticate using the OAuth 2.0 client credentials grant. This flow doesn’t involve users; instead, backend services exchange client credentials for access tokens.
Figure 2: Cognito MRR reference architecture for machine-to-machine use case
The authentication flow is:
/token endpoint with client credentials./token endpoint will flow to the primary Region user pool./token endpoint will flow to the replica Region user pool. After the health check returns to a healthy status, traffic will return to routing to the primary user pool.For applications that use AWS SDK or Amazon Cognito APIs directly (rather than through managed login), the authentication flow is embedded in your application code. This gives you more control over the user experience but requires additional considerations for failover.
Figure 3: Cognito MRR reference architecture for SDK use cases
The process shown in Figure 3 is:
In an SDK-based architecture, Amazon Cognito regional endpoints can also be called directly. You can also set up custom routing to use replica Region endpoints to load balance user authentication requests by routing read-only requests to both the primary and replica Region endpoints while keeping write requests in the primary Region.
Now that you’ve set up multi-Region replication with Amazon Cognito, the next step is to test and monitor your multi-Region configuration. In this section, we walk through strategies for monitoring your endpoints, determining when to trigger failover, and testing your failover readiness.
Failover for Managed Login and all OAuth 2.0 flows is driven by Amazon Route 53 health checks associated with your Amazon Cognito prefix or custom domain. You’re responsible for what determines the state of this health check. The health check isn’t tied to your DNS CNAME record but is the signal that tells Amazon Cognito whether to route traffic to the primary or replica Region for all managed login endpoints. When the health check fails, Amazon Cognito routes traffic to the replica user pool. When the health check recovers, traffic is restored to the primary user pool.
A practical approach to get started to build a health check:
/oauth2/token endpoint or execute a full AdminInitiateAuth API call with test credentials. This validates that the end-to-end authentication path is functional, not just that an endpoint is responding.SuccessPercent CloudWatch metric. Set a threshold that accounts for transient errors (for example, alarm when success drops below 90% for three consecutive evaluation periods).After you have your health check, associate it with your Amazon Cognito domain using the UpdateUserPoolDomain API or the Amazon Cognito console.
Before implementing failover, consider how your authentication layer relates to the rest of your application stack. There are two common patterns:
Use Amazon Application Recovery Controller (ARC) to coordinate failover across all components with a single action. ARC provides three capabilities that are particularly relevant for multi-Region authentication architectures:
ARC is particularly valuable when your Amazon Cognito Lambda triggers, WAF rules, SNS and SES configurations, and downstream services all need to switch Regions in lockstep. Rather than managing failover for each component independently, you can use ARC to define a single recovery group that treats your authentication stack and application stack as one unit. To learn more about the capabilities and use cases of ARC, see Introducing Amazon Route 53 Application Recovery Controller.
The right choice depends on your recovery scope. Map the dependencies in your authentication flow: if your Lambda triggers call regional DynamoDB tables or your post-authentication logic writes to a regional event bus, those tight couplings point to full-stack failover. If your application validates tokens independently and doesn’t make real-time calls back to Amazon Cognito after token issuance, authentication-only failover keeps both your blast radius and operational overhead smaller.
Triggering failover too aggressively risks unnecessary disruptions; too conservatively risks a drop in desired availability. Here are the factors to balance:
Understanding and determining the recovery time objective (RTO) and recovery point objective (RPO) should also be the key factor in determining when and why to failover. See the Establishing RPO and RTO Targets for Cloud Applications blog post to learn more.
If using Route 53 health check, start by manually inverting your Route 53 health check during a maintenance window. In the Route 53 console, enable Invert health check status to force the health check into a failed state; this triggers failover to the replica Region without requiring any infrastructure changes. While traffic is routing to the replica Region, validate that your critical authentication flows (sign-in, token refresh, federation) work correctly, then disable the inversion to restore traffic to the primary. This test confirms your end-to-end failover path is functional.
When you’re confident in the basic failover path, graduate to more realistic failure simulations with AWS Fault Injection Service (FIS). Create FIS experiment templates that disrupt your primary Region’s Amazon Cognito dependencies; for example, block network access to a dependent resource or inject latency into downstream API calls. Use FIS stop conditions (guardrails) to automatically halt experiments if unexpected impacts are detected. These experiments validate not just that failover triggers correctly, but that your replica Region handles real authentication load under degraded conditions.
We recommend conducting failover tests on a predefined and regular cadence and after any significant changes to your authentication architecture. Document your runbooks and make sure your operations team is familiar with both the failover and recovery procedures.
In this post, we built on the foundational knowledge of the Amazon Cognito MRR capability and showed you how to architect resilient authentication for real-world use cases:
To get started, make sure your user pool is on the Essentials or Plus feature plan, configure your multi-Region KMS key and OIDC issuer, and create your first replica. For step-by-step setup instructions, see Multi-Region replication for user pools
If you have feedback or thoughts about this post, submit comments below. If you have questions, start a new thread on Amazon Cognito re:Post or contact AWS Support.
Post Syndicated from Ryan Smith original https://www.servethehome.com/astera-labs-releases-leo-2-cxl-memory-controllers-and-leo-x-controller-for-rackscale-fabric-attached-memory/
Astera Labs is launching a new generation of Leo smart memory controllers. The Leo 2 series adds support for CXL 3.2 and PCIe Gen6, while the ambitious Leo X brings the ability to attach memory expanders directly to the fabric networks of AI accelerators
The post Astera Labs Releases Leo 2 CXL Memory Controllers and Leo X Controller for Rackscale Fabric-Attached Memory appeared first on ServeTheHome.
Post Syndicated from Vaibhav Agrawal original https://aws.amazon.com/blogs/big-data/how-united-airlines-uses-amazon-redshift-and-aws-glue-data-catalog-federation-to-query-databricks-managed-data/
This post was co-written with Ankit Aggarwal and Raja Kalluri from United Airlines.
United Airlines processes billions of events daily across its data platform, which spans Amazon Redshift and Databricks with Unity Catalog. To bridge these platforms without duplicating data, the team turned to AWS Glue Data Catalog federation.
In this post, we walk through how to configure AWS Glue Data Catalog federation to connect with Databricks Unity Catalog, so you can run live SQL queries from Amazon Redshift without moving or duplicating data.
United Airlines curates petabytes of data through a medallion architecture (bronze to silver to gold) on Amazon Simple Storage Service (Amazon S3). The airline user interaction data layer alone is several double-digit terabytes of near real-time streamed data. Teams use it to measure customer engagement patterns, feature adoption, and conversion behavior across web and mobile touchpoints. Analysts need to query this curated data through Amazon Redshift Serverless. As part of the existing data platform architecture these data tables are cataloged in Databricks Unity Catalog, not in the AWS Glue Data Catalog. As a result, Amazon Redshift has no native visibility into them. Without catalog federation, the only way to make this data queryable from Amazon Redshift would have been to duplicate it into Amazon Redshift Managed Storage (RMS) and build pipelines to keep it in sync.
AWS Glue Data Catalog federation removed this need. Amazon Redshift users now query the gold layer stored in Amazon S3 directly, with Iceberg metadata resolved from Unity Catalog at query time and no data movement. AWS Glue Data Catalog federation connects Amazon Redshift to external catalogs like Unity Catalog, so analysts query cross-platform data without building sync pipelines or duplicating storage.
Amazon Redshift Serverless is powered by the same Graviton-based query engine used in the new RG instance family, which delivers up to 2x faster data lake query performance compared to prior generations. This engine is purpose-built for reading Apache Iceberg tables directly from Amazon S3, making it well-suited for such federated query workloads.
United Airlines is taking a phased approach to adopting AWS Glue Data Catalog federation across its data platform. The initial focus is the most heavily used user interaction data tables, with 30 tables currently federated in production and 70 more in active rollout. Several hundred additional tables across different business domains are planned for production in the coming months.
AWS Glue Data Catalog federation bridges these platforms at the metadata layer. Here’s how the architecture works.
The architecture follows a four-layer federation chain:
Key services or service features used in this solution:
Figure 1: Federation chain from Databricks Unity Catalog to Amazon Redshift Serverless through AWS Glue and Lake Formation
The architecture follows a six-step flow:
Before you begin, make sure the following are in place:
Note: For setting up the Databricks Unity Catalog side (Phase 1), follow the steps in the AWS blog post Access Databricks Unity Catalog data using catalog federation in the AWS Glue Data Catalog. This walkthrough picks up after the federated catalog has been created in AWS Glue.
The walkthrough is organized into six steps covering Lake Formation configuration, the resource link pattern, IAM role setup, and querying Databricks tables from Amazon Redshift.
1a. Add a data lake administrator
1b. Confirm the federated catalog is registered
This step is the key architectural detail in the walkthrough. Amazon Redshift resolves CREATE EXTERNAL SCHEMA only against the default AWS Glue Data Catalog. The federated catalog (databricks-federated-catalog) is a separate, non-default catalog object. To give Amazon Redshift a path to the federated data, you create a resource link database in the default catalog that points to the federated catalog’s database.
A resource link does not copy data or metadata. It’s a pointer that Lake Formation resolves at query time.
To create the resource link in the Lake Formation console:
Alternatively, use the AWS CLI:
When Amazon Redshift queries through the resource link, it uses the IAM role attached to the Amazon Redshift Serverless namespace to call the Lake Formation GetDataAccess API. Lake Formation permissions must be granted to this namespace role.
Choose one of these two approaches:
Attach the following IAM policy to the role:
Note: The Resource: “*” in this policy is shown for simplicity. In production, scope resources to specific AWS Glue catalog ARNs, database ARNs, and table ARNs based on your use case.*
After creating or updating the role, associate it with your Amazon Redshift Serverless namespace:
4a. Grant DESCRIBE on the resource link database (default catalog)
4b. Grant SELECT and DESCRIBE on the target tables (Grant on Target)
Resource links permit only DESCRIBE and DROP permissions on the link itself. To allow Amazon Redshift to actually read data, you must separately grant SELECT on the target tables in the federated catalog. This is the Lake Formation Grant on Target pattern.
Important: SELECT must be granted on the TARGET tables in the federated catalog, not on the resource link. Granting SELECT only on the resource link won’t work. This is a common configuration error.
With the resource link in place and permissions granted, you can now create an external schema in Amazon Redshift that points to the resource link database. The external schema is the query interface. When a user runs SQL against it, Amazon Redshift traverses the link to the federated catalog and retrieves metadata and data from Databricks Unity Catalog.
The DATABASE parameter must reference the resource link database name in the default AWS Glue catalog (databricks_federated_db_link), not the federated catalog name directly. The CATALOG_ARN parameter isn’t required here because the resource link lives in the default catalog and Amazon Redshift resolves it automatically.
Connect to your Amazon Redshift cluster as a superuser (for example, using Amazon Redshift Query Editor v2) and run:
A key design principle in this architecture is the clear separation between data physically stored in Amazon Redshift and data accessed externally through federation. External schemas provide a transparent abstraction layer, so Amazon Redshift users can query data stored in S3 without ingestion. For consistency and clarity, United Airlines follows a standard naming convention for all federated schemas in Amazon Redshift: {domain}_iceberg. This convention makes it immediately clear that the data isn’t natively stored within Amazon Redshift but is accessed by using federation through AWS Glue and Lake Formation. This distinction is critical for analysts and engineers, because it improves discoverability, avoids ambiguity between storage layers, and reinforces architectural discipline when working across hybrid data environments.
The User Interactions domain exposes curated datasets representing customer interaction activity, engagement behavior, and channel usage patterns. Operational datasets follow the same pattern, providing governed access to supporting business events and reference information through a common federation framework.
You create a view layer over each external schema using WITH NO SCHEMA BINDING, so that analysts always resolve the freshest schema on each query execution. For example:
After creating the external schema, verify that the Databricks tables are visible and run a test query.
Verify table visibility
Query a Databricks Unity Catalog table
When a query runs, Amazon Redshift calls Lake Formation GetDataAccess using the namespace IAM role to obtain temporary credentials. It then contacts the AWS Glue federated catalog, which in turn calls the Databricks Unity Catalog Iceberg REST API to retrieve metadata and read table data. The result is returned to the Amazon Redshift user transparently.
For SAML-authenticated users, connect using your IdP JDBC plugin:
The Amazon Redshift JDBC driver handles authentication automatically. It authenticates with your IdP, receives a SAML assertion, and calls sts:AssumeRoleWithSAML for temporary IAM credentials. It then calls redshift-serverless:GetCredentials to connect as the mapped database user.
AWS Glue Data Catalog federation delivered measurable architectural and operational improvements for United Airlines:
| Area | Before | After | Impact |
| Data access | Delta Lake and Amazon Redshift data were completely siloed, so Amazon Redshift users had no access to curated datasets on Databricks-managed S3 data | Amazon Redshift users get real-time access to Databricks-managed data through AWS Glue Data Catalog federation | ~100 analysts gained access to user interaction data tables in the first phase without adding new pipelines. |
| Disaster recovery | Cross-Region DR relied on Amazon Redshift snapshots every 3 hours (recovery point objective, or RPO, of 3 hours or more) | Amazon S3 cross-Region replication on the Delta Lake provides a near-continuous RPO. A new Amazon Redshift Serverless workgroup in the DR Region can federate to the same S3 data | More resilient architecture. Reduces cost for Amazon Redshift snapshot and copy maintenance across Regions |
| Architecture simplification | Data processing happened in both Databricks and Amazon Redshift, requiring manual catalog synchronization between the two platforms which was operationally expensive and prone to drift | With the federated architecture, data processing is consolidated in Databricks, and Amazon Redshift acts solely as a query engine powering user queries and dashboards through catalog federation | Single processing platform, zero sync pipelines, single source of truth |
| Infrastructure cost | Running dedicated Amazon Redshift ETL cluster with RMS storage, snapshots, and compute for data processing | For this use case with federation, Amazon Redshift is not needed for ETL but only as a query engine. No RMS storage duplication, no snapshot replication required | ~$30K/month in redundant ETL infrastructure cost reduced |
At United Airlines, identity governance is unified through Azure Active Directory groups. On the AWS consumption side, users authenticate to Amazon Redshift Serverless through SAML federation. AD group membership determines database-level access to federated schemas. On the Databricks side, the same AD groups govern access to Unity Catalog schemas. This single-identity model provides consistent access control across both platforms without requiring separate user provisioning. Lake Formation handles credential vending for S3 data access during federated queries, while schema-level access decisions are managed through the AD group mappings on each platform.
The architecture also provides multiple layers of security controls built into the federation chain:
Review the catalog federation service limitations before deploying. Key requirements:
To avoid ongoing charges for resources created in this walkthrough, remove them in the following order. This teardown doesn’t affect Databricks metadata or your underlying data stored in Amazon S3.
DROP SCHEMA databricks_schema;.In this post, we showed how United Airlines uses AWS Glue Data Catalog federation to give Amazon Redshift Serverless analysts real-time access to double-digit terabytes of curated user interaction data on Amazon S3, without duplicating a single byte or building sync pipelines.
The architecture uses the Iceberg REST API, resource link databases, and Lake Formation credential vending to create a governed query path between Amazon Redshift and Unity Catalog. For United Airlines, this eliminated redundant ETL infrastructure costs, removed the need for catalog synchronization, and turned Amazon Redshift Serverless into a dedicated high-performance query engine for analysts and dashboards.
For questions or feedback, leave a comment on this post.
Post Syndicated from Luis Pastor original https://aws.amazon.com/blogs/security/operationalizing-least-privilege-automate-iam-remediation-through-your-ci-cd-pipeline/
The principle of least privilege is straightforward to articulate but challenging to maintain at scale. When teams first deploy applications to AWS, they often grant broader permissions than strictly necessary; it’s faster to get things working, and the plan is always to tighten permissions later. But later rarely comes. Permissions accumulate, AWS Identity and Access Management (IAM) principals that once needed broad access for initial deployment retain those permissions long after they’re necessary, and some principals stop being used entirely. Even small teams face this challenge—permission reviews aren’t a one-time task but an ongoing operational burden that demands automation.
AWS IAM Access Analyzer addresses detection and recommendation. It identifies unused permissions across IAM roles and users: actions that haven’t been exercised, services that haven’t been accessed, and principals that aren’t being assumed at all. For each finding, it generates a recommended policy with the excess permissions removed. Security teams can see exactly what to fix, but manual remediation doesn’t persist. A security engineer can right-size a role today, but if that role is defined in an AWS CloudFormation template or AWS Cloud Development Kit (AWS CDK) stack, the next deployment restores the original permissions. The fix must live where the role is defined, and not every role starts in the same place. Some are managed through infrastructure-as-code (IaC), where remediation means updating source code and deploying through a pipeline. Others were created manually through the AWS Management Console and have no code representation. And some principals aren’t being used at all and need a controlled decommission path. Each scenario requires a different remediation strategy.
This post walks through an automated remediation workflow that bridges the gap between detection and action. Instead of findings accumulating in a dashboard waiting for someone to investigate, the automation classifies each role by how it was created and produces a ready-to-review remediation artifact: a pull request with production-ready CDK code and a plain-English explanation for IaC-managed roles, an issue with the recommended policy and step-by-step IaC migration guidance for manually created roles, or a soft-disable issue with a monitored decommission plan for unused principals. Each output flows through your existing code review and issue tracking processes—the same workflows your teams already follow. By the end of this post, you’ll have a pattern that converts IAM Access Analyzer findings into tested, deployable code changes rather than a growing backlog of security tickets.
Unused IAM permissions increase the attack surface. Removing unused permissions limits the actions available to any compromised credentials, reducing potential impact. Roles that aren’t being assumed represent unused resources; removing them simplifies your IAM inventory and reduces potential access paths that aren’t actively monitored.
The challenge isn’t knowing what to fix. As we said earlier, Access Analyzer provides both the findings and the recommended policies. The challenge is acting on that knowledge consistently across your environment. Each finding requires context:
Multiply this by hundreds of roles and security teams face a backlog that grows faster than they can address it.
Manual remediation compounds the problem. A security engineer can right-size a role directly in the console, but that fix is fragile. If the role is defined in an IaC template, the next deployment restores the original permissions. If it was created manually, there’s no record of what changed or why, and no easy way to revert if the change causes issues.
This is where IaC changes the equation. When roles are defined in code, remediation means updating that code. Changes flow through pull requests, are reviewed by the team that owns the role, and deploy consistently across environments. The fix becomes permanent, not a point-in-time correction that drifts back on the next deployment. And because every change is tracked in version control, teams can confidently remove permissions knowing they can revert if something breaks. That safety net matters; it’s often the difference between a team acting on a finding and leaving it in the backlog.
The solution automates remediation by connecting four capabilities: IAM Access Analyzer for detection and policy recommendations, CloudTrail for role attribution, Amazon Bedrock for CDK code generation and plain-English explanations, and your existing continuous integration and delivery (CI/CD) pipeline for remediation execution. The workflow operates on a core principle: every IAM role has an origin, and that origin determines the remediation path.
Figure 1 shows the solution architecture: Amazon EventBridge triggers an AWS Lambda orchestrator on a daily schedule. The Lambda orchestrator integrates with IAM Access Analyzer, CloudTrail, Amazon Bedrock, and Amazon CloudWatch. Each finding is routed to one of three remediation paths: a pull request for IaC-managed roles, an issue for manually created roles, and a soft-disable issue for unused roles.
Figure 1: The daily remediation workflow; from scheduled trigger to the three role-based remediation paths
On each scheduled run, the automation retrieves active findings from IAM Access Analyzer and queries CloudTrail to determine how each role was created. Roles created through CloudFormation or AWS CDK have a traceable origin: the service principal, stack name, and originating repository. Roles created manually through the console have a different origin: the IAM user who created them and the timestamp. This distinction drives the remediation strategy.
For IaC-managed roles, the automation retrieves the IAM Access Analyzer-recommended policy and uses Amazon Bedrock to wrap it in production-ready CDK code that includes the role definition and policy statements and imports what your CI/CD pipeline needs to deploy the update. It then creates a pull request in the originating repository. The pull request (PR) includes the updated CDK code, a policy diff showing exactly which permissions are being removed, and a plain-English explanation of the changes, for example, “This change removes write access to S3, keeping only read and list permissions.” Your existing code review process evaluates the change, and after being merged, the fix deploys consistently across environments.
For manually created roles, the automation creates an issue that includes the IAM Access Analyzer-recommended policy with unused permissions removed, a diff highlighting the changes, and an Amazon Bedrock-generated explanation of what the permission changes accomplish. The issue also provides guidance on importing the role into your IaC codebase. This gives teams an immediate remediation path while encouraging long-term governance through IaC adoption.
For roles that aren’t being assumed at all, the automation takes a more cautious approach. Instead of taking direct action, it creates an issue recommending a soft-disable workflow: attach a deny-all policy to the role, monitor for 30 days to confirm no workload depends on it, then delete. The issue provides the steps and context, the team executes the decommission through their preferred process, whether that’s a console change, an AWS Command Line Interface (AWS CLI) script, or a PR removing the role from the IaC. This controlled decommission path reduces the risk of removing a role that’s used infrequently or seasonally.
The solution supports both single-account and organization-wide deployment. In single-account mode, it uses an ACCOUNT_UNUSED_ACCESS analyzer to process findings for one account. In organization mode, it uses an ORGANIZATION_UNUSED_ACCESS analyzer deployed in a delegated administrator account, which generates findings across all member accounts from a single vantage point. The Lambda function automatically detects which analyzer type is available and extracts the account ID from each finding’s resource Amazon Resource Name (ARN), so role attribution and remediation routing work the same way regardless of scope.
This three-path strategy acknowledges operational reality. Not all roles start in IaC, not all unused roles are safe to delete immediately, and forcing immediate migration isn’t always practical. The solution provides a clear path forward for each scenario: remediate IaC roles through code, give teams actionable recommendations for manually created roles, and safely decommission what’s no longer needed. Over time, your infrastructure becomes increasingly code-driven, and remediation becomes a routine part of your CI/CD process rather than a manual security task.
Consider a company—call them AnyCompany—running 200 IAM roles across three AWS accounts. Some roles were created through AWS CDK stacks during initial deployment. Others were created manually through the console by engineers who needed quick access during incident response or prototyping. A handful haven’t been assumed in over 6 months. AnyCompany’s security team wants to act on their IAM Access Analyzer findings, but each role requires different handling. The solution’s architecture addresses this by routing each finding through a classification and remediation pipeline.
Figure 2 shows how each IAM Access Analyzer finding is processed:
UnusedPermission findings retrieve a recommended policy from IAM Access Analyzer and then query CloudTrail for role origin, while UnusedIAMRole findings follow the unused role path.
Figure 2: Detailed component interactions—the orchestrator’s five steps, its four service integrations, and the three remediation paths
The rest of this section walks through each component using AnyCompany’s roles as examples.
Before processing any finding, the Lambda function loads an exclusion configuration and checks whether the role should be skipped. This prevents the automation from creating remediation items for roles that legitimately need broad permissions.
AnyCompany excludes their break-glass role (used only during incidents), any service-linked roles, and roles tagged CriticalService. The min_unused_days threshold prevents false positives from seasonal workloads; a role that ran a quarterly batch job 25 days ago won’t generate a finding.
IAM Access Analyzer generates two types of findings relevant to this solution. UnusedPermission findings identify roles with permissions that haven’t been exercised within the analysis period. UnusedIAMRole findings identify roles that haven’t been assumed at all. The Lambda function queries both finding types separately because they follow different remediation paths.
The Lambda function auto-detects the analyzer type at startup. When ANALYZER_SCOPE is set to organization, it checks for an ORGANIZATION_UNUSED_ACCESS analyzer first and falls back to ACCOUNT_UNUSED_ACCESS if none exists. If multiple analyzers of the same type exist in the account, the Lambda function selects the first active analyzer returned by the API. To target a specific analyzer, set the ANALYZER_ARN environment variable explicitly. With an organization-level analyzer, findings include roles from all member accounts. The Lambda function extracts the account ID from each finding’s resource ARN (for example, account 111122223333 from arn:aws:iam::111122223333:role/MyRole) and carries that context through the entire pipeline: attribution, remediation, and issue or PR creation all include the originating account.
For UnusedPermission findings, the Lambda function calls GenerateFindingRecommendation to initiate policy generation, then retrieves the IAM Access Analyzer-recommended policy through the GetFindingRecommendation API. This is a key integration point: IAM Access Analyzer provides the right-sized policy with unused permissions removed, so the automation doesn’t need to generate policies itself.
Here’s what a typical finding looks like for one of AnyCompany’s application roles:
The OrderProcessorRole has write and delete permissions for Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB, but only uses read operations. The IAM Access Analyzer recommendation removes the four unused actions while preserving the three active ones.
For UnusedIAMRole findings, no recommendation is needed: the role isn’t being assumed at all, so the remediation is to disable or delete it. The Lambda function caps the number of unused role issues per run (configurable using MAX_UNUSED_ROLE_ISSUES, default 10) to avoid overwhelming teams with a flood of issues on the first execution.
For each finding, the Lambda function queries CloudTrail to determine how the role was created. The CreateRole event contains the information needed to classify the role’s origin.
An IaC-created role looks like this in CloudTrail:
The cloudformation.amazonaws.com service principal and user agent tell the automation this role was created through a CloudFormation or AWS CDK deployment. The Lambda function then looks up the role’s tags to find the originating repository (stored in a Repository tag set during deployment).
A manually-created role looks different:
Here, the IAMUser type and console.amazonaws.com user agent indicate someone created this role through the console. Roles created through the AWS CLI show a similar pattern: the IAMUser type with a user agent like aws-cli/2.x.x. The automation classifies both console and AWS CLI-created roles as manually created, because neither has an IaC origin that can be updated programmatically. The automation captures the username and timestamp for the remediation issue.
When the Lambda function processes findings from an organization-level analyzer, the role might live in a different account than the one running the function. The automation handles this by assuming a cross-account role (configurable using CROSS_ACCOUNT_ROLE_NAME, defaulting to OrganizationAccountAccessRole) in the member account, then querying that account’s CloudTrail and IAM APIs for the CreateRole event. If the cross-account assume fails—because the role doesn’t exist in that account or permissions aren’t configured—the automation falls back gracefully, classifying the role as unknown origin and creating an issue with the account ID and available context. This approach helps the automation produce an actionable output for findings even when attribution is incomplete.
For IaC-managed roles with UnusedPermission findings, the Lambda function retrieves the IAM Access Analyzer-recommended policy and sends it to Amazon Bedrock to generate production-ready AWS CDK code. This is an important distinction: IAM Access Analyzer decides what the policy should be, and Amazon Bedrock wraps that policy in the AWS CDK constructs, imports, and resource definitions that the CI/CD pipeline needs to deploy the update.
The prompt instructs Amazon Bedrock to convert the recommended policy to AWS CDK code exactly as provided, with no modifications:
IAM Access Analyzer generates recommendations for both inline policies and customer managed policies. When a managed policy has partially unused permissions, the recommendation contains the full right-sized policy. The automation wraps this in AWS CDK code as an iam.ManagedPolicy construct. Note that if a managed policy is shared across multiple roles, the recommendation applies to the specific role’s usage pattern. In this case, the automation generates an issue for manual review rather than a PR, because modifying a shared policy could affect other roles.
The generated code goes through a validation step before inclusion in any PR. The Lambda function compiles the Python code to check for syntax errors and verifies that required AWS CDK patterns (iam, PolicyStatement) are present. If validation fails, the finding is logged as an error rather than creating a broken PR.
The solution doesn’t currently invoke the IAM Access Analyzer ValidatePolicy API to check the generated policy for errors or overly permissive statements. However, this is a natural extension point. Teams can add a validation step that calls ValidatePolicy on the Amazon Bedrock-generated policy before including it in a PR, detecting issues like missing resource constraints or invalid action names.
Amazon Bedrock also generates a plain-English explanation of the policy changes. For AnyCompany’s OrderProcessorRole, the explanation might read:
“The role currently has full S3 write access and DynamoDB delete permissions, but only uses read operations. Removing s3:PutObject, s3:DeleteObject, s3:PutBucketPolicy, and dynamodb:DeleteItem reduces the scope of impact if credentials are compromised, while preserving the s3:GetObject, s3:ListBucket, and dynamodb:Query permissions the application needs.”
The solution uses the Anthropic Claude Sonnet model on Amazon Bedrock for CDK code generation (where accuracy matters) and Claude Haiku on Amazon Bedrock for explanations (where speed and cost efficiency matter more).
The Lambda function evaluates each finding’s origin and routes it to one of three remediation paths.
Path 1: IaC-managed roles (pull request) – For AnyCompany’s OrderProcessorRole, the automation creates a PR in the originating repository. The PR includes:
security, iam-remediation, automated) for filtering and trackingThe team that owns the role reviews the PR through their normal code review process. Once merged, the fix deploys consistently across environments through the existing CI/CD pipeline.
Path 2: Manually-created roles (issue) – For AnyCompany’s IncidentResponseRole, the automation creates an issue that includes the Access Analyzer-recommended policy with unused permissions removed, a diff highlighting the changes, an Amazon Bedrock-generated explanation, and step-by-step guidance on importing the role into IaC. This gives the team an immediate remediation path (apply the recommended policy) while encouraging long-term governance through IaC adoption.
Path 3: Unused roles (soft-disable issue) – For roles that haven’t been assumed at all, the automation creates an issue recommending a three-stage decommission workflow: attach a deny-all policy to the role, monitor for 30 days to confirm no workload depends on it, then delete. This controlled approach reduces the risk of removing a role that’s used infrequently or seasonally – if something breaks during the monitoring period, removing the deny-all policy restores access immediately.
Dry-run mode
Before creating real PRs and issues, you can run the automation in dry-run mode by setting “dry_run": true in the CI/CD configuration or setting the CI_CD_PLATFORM environment variable to dryrun. In this mode, the Lambda function processes findings, classifies roles, and generates remediation data, but logs what it would create instead of making actual API calls to your repository platform. You can use the log to validate the automation’s behavior, review the classification accuracy, and tune exclusions before going live.
Operational metrics
The Lambda function publishes CloudWatch metrics after each run:
| findings_processed | Total UnusedPermission findings evaluated |
| iac_roles_found | Roles classified as IaC-managed |
| manual_roles_found | Roles classified as manually created |
| unused_roles_found | Roles with no assume activity (UnusedIAMRole findings) |
| prs_created | Pull requests created for IaC roles |
| issues_created | Issues created (manual roles and unused roles) |
| errors | Processing errors (failed classifications, API failures) |
These metrics feed into dashboards and alarms. AnyCompany sets an alarm on errors > 5 to catch API throttling or configuration issues, and tracks prs_created + issues_created over time to measure remediation velocity.
The solution ships as two AWS CDK stacks and deploys in minutes. The accompanying GitHub repository contains the complete source code, AWS CDK stacks, configuration templates, and step-by-step deployment instructions.
At a high level, deployment involves:
ACCOUNT_UNUSED_ACCESS or ORGANIZATION_UNUSED_ACCESS analyzer enabled, Python 3.11 or later, AWS CDK v2, a CI/CD platform API token stored in AWS Secrets Manager, and Amazon Bedrock model access for the Anthropic Claude models you plan to use. The model IDs are configurable environment variables (BEDROCK_CODEGEN_MODEL and BEDROCK_EXPLANATION_MODEL); Amazon Bedrock retires older foundation models over time, so if the shipped defaults stop working, set these variables to current models you have enabled and redeploy. The repository README documents this.config/ directory control behavior. exclusions.json defines which roles and permissions to skip (break-glass roles, service-linked roles, tagged exceptions), and ci_cd_config.json configures your repository platform integration (GitLab or GitHub), labels, and throttling limits.cdk deploy --all to create the Lambda function, EventBridge schedule, IAM roles, and CloudWatch alarms.“dry_run": true to see how the automation classifies your roles without creating real PRs or issues. Review the CloudWatch logs to confirm attribution accuracy and tune exclusions.“dry_run": false and redeploy. The Lambda function runs on schedule (daily by default) and begins creating PRs and issues.The repository README covers each step in detail, including organization-wide deployment, cross-account configuration, and platform-specific setup for GitLab and GitHub.
Deploying the automation is only the starting point. Running it in production means making decisions about how roles are retired, how the volume of findings is managed at scale, which roles warrant human review before any change is proposed, and how you measure the automation’s impact over time. The following practices keep remediation sustainable as your IAM footprint grows, so the automation reduces operational burden rather than adding to it.
Unused roles follow a three-stage decommission workflow. When the automation identifies a role that hasn’t been assumed within the analysis period, it creates an issue with the recommended decommission steps; the automation doesn’t modify the role directly. The team then follows the soft-disable approach:
This approach is deliberately conservative. Deleting a role is irreversible; you lose the trust policy, attached policies, and any resource-based policies that reference it. The soft-disable step gives teams a safety net while still making progress on reducing their unused role inventory.
On AnyCompany’s first run, the automation found 47 unused permission findings and 4 unused roles. That’s manageable. But organizations with hundreds of accounts and thousands of roles might see significantly more findings on initial deployment.
This is especially true with an organization-level analyzer. A single-account deployment might surface dozens of findings; an organization-level analyzer across multiple accounts could surface hundreds or thousands on the first run. The throttling controls become critical at this scale.
Two throttling controls prevent the automation from overwhelming teams:
max_findings_per_run (default 50): Caps the total UnusedPermission findings processed per Lambda function execution. Remaining findings are picked up on the next scheduled run.MAX_UNUSED_ROLE_ISSUES (default 10): Caps unused role issues per run. This is especially important during initial deployment when you might have a large backlog of roles that haven’t been assumed in months.Start with conservative limits and increase them as your team builds confidence in the review process. A team that can review 10 PRs per week shouldn’t receive 50 on Monday morning.
Not every role should receive automated PRs. Roles with administrative permissions or access to sensitive data might warrant manual review before any remediation is created. The exclusion configuration supports this through the approval_required_for_tags field:
Roles matching these tags generate issues for manual review instead of automated PRs, regardless of whether they’re IaC-managed. This gives security teams a checkpoint for high-risk roles while still automating remediation for standard application roles.
The metrics published after each Lambda function run (covered in the Technical details section) feed into CloudWatch dashboards and alarms. A few patterns worth setting up:
errors > 5 per run to catch API throttling, expired CI/CD tokens, or Amazon Bedrock availability issues.prs_created + issues_created over time. A healthy trend shows this number decreasing as your environment converges toward least privilege.unused_roles_found as a leading indicator. A sudden increase might signal a team spinning up roles for a project and not cleaning up afterward.iac_roles_found to manual_roles_found over time. As teams adopt IaC, the ratio should shift toward IaC-managed roles, which means more automated remediation and less manual work.The solution uses Lambda (minimal cost at daily execution), CloudTrail (typically already enabled), IAM Access Analyzer (charges per IAM role or user analyzed per month for the unused access analyzer), and Amazon Bedrock (pay-per-token for AWS CDK code generation and explanations). For most organizations the ongoing cost is low, and Amazon Bedrock token usage is the largest variable, scaling with the number of findings processed per day and the complexity of each policy. Review the pricing pages for each service for current rates.
For organization-level deployments, the IAM Access Analyzer cost scales with the number of IAM roles analyzed across all member accounts. The ORGANIZATION_UNUSED_ACCESS analyzer charges per role per month across the organization, so an organization with 500 roles across 20 accounts will see higher analyzer costs than a single account with 50 roles. Review the IAM Access Analyzer pricing page for current rates.
To remove the solution, run cdk destroy --all from the infrastructure/ directory. This removes the Lambda function, EventBridge rule, CloudWatch alarms, and IAM roles created by the stacks.
If you stored a CI/CD platform API token in Secrets Manager as part of deployment, delete it with aws secretsmanager delete-secret --secret-id <your-secret-name> --recovery-window-in-days 7. The 7-day recovery window lets you restore the secret if the deletion was accidental. After 7 days, the secret is permanently deleted and can’t be recovered. To delete immediately without a recovery window, add --force-delete-without-recovery.
Lambda automatically creates a CloudWatch Logs log group at /aws/lambda/<function-name> that persists after cdk destroy --all and continues to incur log storage charges. To remove it, run aws logs delete-log-group --log-group-name /aws/lambda/<function-name>. WARNING: This permanently deletes all execution logs.
The IAM Access Analyzer isn’t created by the AWS CDK stacks. WARNING: Deleting the analyzer permanently removes all findings, analysis history, and unused permission data. Export any findings you need to retain before deletion. After exporting, run aws accessanalyzer delete-analyzer --analyzer-name <your-analyzer-name> to delete it. The ACCOUNT_UNUSED_ACCESS and ORGANIZATION_UNUSED_ACCESS analyzer types incur charges based on the number of IAM roles and users analyzed per month.
If you deployed in organization mode and created cross-account roles (default name: OrganizationAccountAccessRole) in member accounts solely for this solution, remove them from those accounts.
Any PRs or issues already created in your CI/CD platform remain after stack deletion; they’re artifacts in your repository, not AWS resources. See the repository README for detailed cleanup instructions.,
Automating IAM permission remediation turns least privilege from a periodic compliance exercise into an operational practice. By connecting IAM Access Analyzer findings and recommendations to your CI/CD pipeline, remediation shifts from manual security tasks to code review processes that your teams already follow.
The three-path strategy acknowledges how infrastructure evolves. IaC-managed roles receive pull requests with production-ready AWS CDK code and plain-English explanations. Manually created roles receive actionable issues with recommended policies and IaC migration guidance. Unused roles are put on a controlled decommission path that protects against accidental disruption. Over time, the manual role count decreases as teams adopt IaC, and remediation becomes a routine part of your deployment pipeline.
Start with a pilot. Choose 10–20 non-production roles, deploy in dry-run mode, and review the classification results. Tune your exclusions, confirm the CloudTrail attribution is accurate for your environment, and then enable live remediation. Expand to production roles after your team is comfortable with the review cadence.
When you’re ready to scale beyond a single account, switch to an organization-level analyzer and the same Lambda function will process findings across all member accounts with no architectural changes required, only a configuration toggle.
The complete source code, AWS CDK stacks, and configuration templates are available in the accompanying GitHub repository.
If you have feedback about this post, submit comments in the Comments section below.