Balance deployment speed and stability with DORA metrics

2024-07-31 Rostislav Markov

Post Syndicated from Rostislav Markov original https://aws.amazon.com/blogs/devops/balance-deployment-speed-and-stability-with-dora-metrics/

Development teams adopt DevOps practices to increase the speed and quality of their software delivery. The DevOps Research and Assessment (DORA) metrics provide a popular method to measure progress towards that outcome. Using four key metrics, senior leaders can assess the current state of team maturity and address areas of optimization.

This blog post shows you how to make use of DORA metrics for your Amazon Web Services (AWS) environments. We share a sample solution which allows you to bootstrap automatic metric collection in your AWS accounts.

Benefits of collecting DORA metrics

DORA metrics offer insights into your development teams’ performance and capacity by measuring qualitative aspects of deployment speed and stability. They also indicate the teams’ ability to adapt by measuring the average time to recover from failure. This helps product owners in defining work priorities, establishing transparency on team maturity, and developing a realistic workload schedule. The metrics are appropriate for communication with senior leadership. They help commit leadership support to resolve systemic issues inhibiting team satisfaction and user experience.

Use case

This solution is applicable to the following use case:

Development teams have a multi-account AWS setup including a tooling account where the CI/CD tools are hosted, and an operations account for log aggregation and visualization.
Developers use GitHub code repositories and AWS CodePipeline to promote code changes across application environment accounts.
Tooling, operations, and application environment accounts are member accounts in AWS Control Tower or workload accounts in the Landing Zone Accelerator on AWS solution.
Service impairment resulting from system change is logged as OpsItem in AWS Systems Manager OpsCenter.

Overview of solution

The four key DORA metrics

The ‘four keys’ measure team performance and ability to react to problems:

Deployment Frequency measures the frequency of successful change releases in your production environment.
Lead Time For Changes measures the average time for committed code to reach production.
Change Failure Rate measures how often changes in production lead to service incidents/failures, and is complementary to Mean Time Between Failure.
Mean Time To Recovery measures the average time from service interruption to full recovery.

The first two metrics focus on deployment speed, while the other two indicate deployment stability (Figure 1). We recommend organizations to set their own goals (that is, DORA metric targets) based on service criticality and customer needs. For a discussion of prior DORA benchmark data and what it reveals about the performance of development teams, consult How DORA Metrics Can Measure and Improve Performance.

Figure 1. Overview of DORA metrics

Consult the GitHub code repository Balance deployment speed and stability with DORA metrics for a detailed description of the metric calculation logic. Any modifications to this logic should be made carefully.

For example, the Change Failure Rate focuses on changes that impair the production system. Limiting the calculation to tags (such as hotfixes) on pull requests would exclude issues related to the build process. It’s important to match system change records that lead to actual impairments in production. Limiting the calculation to the number of failed deployments from the deployment pipeline only considers deployments that didn’t reach production. We use AWS Systems Manager OpsCenter as the system of records for change-related outages, rather than relying solely on data from CI/CD tools.

Similarly, Mean Time To Recovery measures the duration from a service impairment in production to a successful pipeline run. We encourage teams to track both pipeline status and recovery time, as frequent pipeline failure can indicate insufficient local testing and potential pipeline engineering issues.

Gathering DORA events

Our metric calculation process runs in four steps:

In the tooling account, we send events from CodePipeline to the default event bus of Amazon EventBridge.
Events are forwarded to custom event buses which process them according to the defined metrics and any filters we may have set up.
The custom event buses call AWS Lambda functions which forward metric data to Amazon CloudWatch. CloudWatch gives us an aggregated view of each of the metrics. From Amazon CloudWatch, you can send the metrics to another designated dashboard like Amazon Managed Grafana.
As part of the data collection, the Lambda function will also query GitHub for the relevant commit to calculate the lead time for changes metric. It will query AWS Systems Manager for OpsItem data for change failure rate and mean time to recovery metrics. You can create OpsItems manually as part of your change management process or configure CloudWatch alarms to create OpsItems automatically.

Figure 2 visualizes these steps. This setup can be replicated to a group of accounts of one or multiple teams.

This figure visualizes the aforementioned four steps of our metric calculation process. AWS Lambda functions process all events and publish custom metrics in Amazon CloudWatch.

Figure 2. DORA metric setup for AWS CodePipeline deployments

Walkthrough

Follow these steps to deploy the solution in your AWS accounts.

Prerequisites

For this walkthrough, you should have the following prerequisites:

AWS accounts for tooling, operations, and application environments
Install Python version 3.9 or later
AWS Cloud Development Kit (AWS CDK) v2 installed
Set up an AWS CDK pipeline
Access to AWS CodePipeline, AWS Systems Manager OpsCenter and GitHub

Deploying the solution

Clone the GitHub code repository Balance deployment speed and stability with DORA metrics.

Before you start deploying or working with this code base, there are a few configurations you need to complete in the constants.py file in the cdk/ directory. Open the file in your IDE and update the following constants:

TOOLING_ACCOUNT_ID & TOOLING_ACCOUNT_REGION: These represent the AWS account ID and AWS region for AWS CodePipeline (that is, your tooling account).
OPS_ACCOUNT_ID & OPS_ACCOUNT_REGION: These are for your operations account (used for centralized log aggregation and dashboard).
TOOLING_CROSS_ACCOUNT_LAMBDA_ROLE: The IAM Role for cross-account access that allows AWS Lambda to post metrics from your tooling account to your operations account/Amazon CloudWatch dashboard.
DEFAULT_MAIN_BRANCH: This is the default branch in your code repository that’s used to deploy to your production application environment. It is set to “main” by default, as we assumed feature-driven development (GitFlow) on the main branch; update if you use a different naming convention.
APP_PROD_STAGE_NAME: This is the name of your production stage and set to “DeployPROD” by default. It’s reserved for teams with trunk-based development.

Setting up the environment

To set up your environment on MacOS and Linux:

Create a virtual environment:
```
$ python3 -m venv .venv
```
Activate the virtual environment: On MacOS and Linux:
```
$ source .venv/bin/activate
```

Alternatively, to set up your environment on Windows:

Create a virtual environment:
```
% .venv\Scripts\activate.bat
```
Install the required Python packages:
```
$ pip install -r requirements.txt
```

To configure the AWS Command Line Interface (AWS CLI):

Follow the configuration steps in the AWS CLI User Guide.
```
$ aws configure sso
```
Configure your user profile (for example, Ops for operations account, Tooling for tooling account). You can check user profile names in the credentials file.

Deploying the CloudFormation stacks

Switch directory
```
$ cd cdk
```
Bootstrap CDK
```
$ cdk bootstrap –-profile Ops
```
Synthesize the AWS CloudFormation template for this project:
```
$ cdk synth
```
To deploy a specific stack (see Figure 3 for an overview), specify the stack name and AWS account number(s) in the following command:
```
$ cdk deploy <Stack-Name> --profile {Tooling, Ops}
```
To launch the DoraToolingEventBridgeStack stack in the Tooling account:
```
$ cdk deploy DoraToolingEventBridgeStack --profile Tooling
```
To launch the other stacks in the Operations account (including DoraOpsGitHubLogsStack, DoraOpsDeploymentFrequencyStack, DoraOpsLeadTimeForChangeStack, DoraOpsChangeFailureRateStack, DoraOpsMeanTimeToRestoreStack, DoraOpsMetricsDashboardStack):
```
$ cdk deploy DoraOps* --profile Ops
```

The following figure shows the resources you’ll launch with each CloudFormation stack. This includes six AWS CloudFormation stacks in operations account. The first stack sets up log integration for GitHub commit activity. Four stacks contain a Lambda function which creates one of the DORA metrics. The sixth stack creates the consolidated dashboard in Amazon CloudWatch.

Figure 3. Resources provisioned with this solution

Testing the deployment

To run the provided tests:

$ pytest

Understanding what you’ve built

Deployed resources in tooling account

The DoraToolingEventBridgeStack includes Amazon EventBridge rules with a target of the central event bus in the operations account, plus an AWS IAM role with cross-account access to put events in the operations account. The event pattern for invoking our EventBridge rules listens for deployment state changes in AWS CodePipeline:

{
  "detail-type": ["CodePipeline Pipeline Execution State Change"],
  "source": ["aws.codepipeline"]
}

Deployed resources in operations account

The Lambda function for Deployment Frequency tracks the number of successful deployments to production, and posts the metric data to Amazon CloudWatch. You can add a dimension with the repository name in Amazon CloudWatch to filter on particular repositories/teams.
The Lambda function for the Lead Time For Change metric calculates the duration from the first commit to successful deployment in production. This covers all factors contributing to lead time for changes, including code reviews, build, test, as well as the deployment itself.
The Lambda function for Change Failure Rate keeps track of the count of successful deployments and the count of system impairment records (OpsItems) in production. It publishes both as metrics to Amazon CloudWatch and the latter calculates the ratio, as shown in below example.
The Lambda function for Mean Time To Recovery keeps track of all deployments with status SUCCEEDED in production and whose repository branch name references an existing OpsItem ID. For every matching event, the function gets the creation time of the OpsItem record and posts the duration between OpsItem creation and successful re-deployment to the CloudWatch dashboard.

All Lambda functions publish metric data to Amazon CloudWatch using the PutMetricData API. The final calculation of the four keys is performed on the CloudWatch dashboard. The solution includes a simple CloudWatch dashboard so you can validate the end-to-end data flow and confirm that it has deployed successfully:

This simple CloudWatch dashboard displays the four DORA metrics for three reporting periods: per day, per week, and per month.

Cleaning up

Remember to delete example resources if you no longer need them to avoid incurring future costs.

You can do this via the CDK CLI:

$ cdk destroy <Stack-Name> --profile {Tooling, Ops}

Alternatively, go to the CloudFormation console in each AWS account, select the stacks related to DORA and click on Delete. Confirm that the status of all DORA stacks is DELETE_COMPLETE.

Conclusion

DORA metrics provide a popular method to measure the speed and stability of your deployments. The solution in this blog post helps you bootstrap automatic metric collection in your AWS accounts. The four keys help you gain consensus on team performance and provide data points to back improvement suggestions. We recommend using the solution to gain leadership support for systemic issues inhibiting team satisfaction and user experience. To learn more about developer productivity research, we encourage you to also review alternative frameworks including DevEx and SPACE.

Further resources

If you enjoyed this post, you may also like:

Author bio

Приятелство без граници. Как Китай очарова Русия?

2024-07-31 Искрен Иванов

Post Syndicated from Искрен Иванов original https://www.toest.bg/priyatelstvo-bez-granitsi-kak-kitaiu-ocharova-rusia/

Приятелство без граници. Как Китай очарова Русия?

Съществуват множество примери как държави с различна стратегическа култура се обединяват от идеята да наложат нов дневен ред в глобалната политика. В годините след края на Студената война Русия ревизира старите съветски доктрини и успява да изгради гъвкави механизми за влияние в системата на международните отношения, но тяхната ефективност едва ли щеше да е толкова очевидна без партньорството на Москва с Пекин.

Ето защо си струва да си зададем въпроса дали съвременните тенденции в отношенията между Русия и Китай могат да бъдат възприети единствено през призмата на националния интерес, или съществуват споделени ценности, които карат Москва и Пекин да развиват сътрудничеството си?

Дали стремежът към многополюсен свят, така често присъстващ в речите на Владимир Путин и Си Дзинпин, не е просто външното лице на една доста по-грандиозна стратегия, целяща глобална подмяна на ценностите и правилата, на които стъпва системата на международните отношения след края на Студената война? И най-сетне, ще съумеят ли двете държави да начертаят червени линии помежду си, или наистина вярват, че тяхното сътрудничество може да се превърне в гарант за баланса на силите в глобалната архитектура за сигурност?

Русия и Китай – не един, а два свята

Слагането на знак за равенство между стратегическите култури на Москва и Пекин е сериозна теоретична и практическа грешка. Двете държави може и да имат сходна визия кои са техните стратегически опоненти, но политически и културно се различават съществено. Стратегическата култура на Китай вече беше засегната подробно в предишния материал, затова си заслужава да кажем малко повече по какво тя се отличава от руската, която възниква и се формира на няколко етапа.

В началото следва да направим уточнението, че понятието „руска култура“ е неточно от научноисторическа гледна точка, защото авторите, които систематизират основните аспекти на руските културни образци, се идентифицират преди всичко като евразийци, а след това като руснаци.

Първите трудове, които систематизират последователно евразийството като идейно-философско учение, обясняващо Русия и нейната традиция, се появяват в България и са дело на руски белоемигранти, напуснали страната си след Октомврийската революция. Въпреки че много историци са склонни да търсят корените на руските идеи у автори като Константин Леонтиев, Николай Данилевски или Владимир Соловьов, класическата евразийска мисъл възниква с трудовете на Николай Трубецкой, Петър Сувчински и Пьотр Савицки.

Така в първия етап на формиране на евразийската стратегическа култура нейното съдържание се свежда до три основни стълба. Първият е идеята, че самата Русия е Евразия – жив политически организъм, който е самобитен, защото не принадлежи нито към Европа, нито към Азия. В този смисъл ранното евразийство е идея, която категорично се противопоставя на това Русия да бъде причислявана към семейството на азиатските държави или към семейството на европейските.

Вторият стълб е силно идеологизиран, тъй като той отрича Октомврийската революция, но хвърля вината за нея върху опитите на династията Романови да европеизира Русия. Ето защо евразийството в своята същност е отрицание както на азиатските, така и на европейските ценности.

Третият стълб дефинира Русия като културно-политическо цяло, което съчетава строгата централизирана система на татарската Златна орда с православното наследство на Източната Римска империя (Византия). За евразийците външната политика на това културно-политическо цяло и на неговия исторически приемник СССР следва да бъде насочена към „месторазвитие“ – обединението на всички славянски народи в един огромен славяно-православен пояс под опеката на Москва. Тези идеи са ясно изразени в евразийските манифести от 20-те години на миналия век, а по-късно са подробно развити и от автори като Шахматов, Вернадски и Степанов.

Вторият етап във формирането на руската стратегическа култура е белязан от творчеството на съветския писател и философ Лев Гумильов, който в своя основен труд „Етногенезисът и биосферата на Земята“ създава теорията си за пасионариите, съгласно която легитимира съществуването на „евразийски суперетнос“, населяващ територията на тогавашния СССР.

Гумильов твърди, че този етнос е източник на т.нар. пасионарии – лидери, които се намират под влиянието на самобитните евразийски ценности и руското православие. Те естествено се утвърждават като противообразци на политическите режими в западните демокрации, където политиците се избират от гражданите. По същество тази теория легитимира идеята за съветския вождизъм и това е причината, поради която книгите на Гумильов срещат толкова голяма подкрепа в ръководните среди на СССР.

Последният етап, който доизгражда стратегическата култура на Русия такава, каквато я познаваме днес, настъпва с написването на трудовете на небезизвестния руски философ и политолог Александър Дугин. Дугин създава неоевразийската концепция, съгласно която системата на международните отношения се намира в естествено и неизбежно (както той го дефинира) геополитическо противопоставяне между „сухопътните“ цивилизации начело с Русия и „морските“ начело със САЩ и Британия.

В контекста на този сблъсък неоевразийството целеполага като основен приоритет в руската външна политика създаването на многополюсен свят, в който Москва и нейните партньори да могат да преследват външната политика в своите сфери на влияние, а всяка държава да бъде свободна сама да избира какви ценности да преследва. По същество тези идеи представляват отрицание на глобалния либерален ред, установен от САЩ след края на Студената война, и предвиждат възстановяването на старите сфери на влияние на Москва от времето на СССР. Това следва да бъде крайната цел на руската външна политика, така както е залегнала в идейните основи на неоевразийската философия, която възприема тези процеси като приоритет само за Русия и затова не предвижда създаването на система от съюзи, подобно на западната.

От два различни свята към единна външна политика

След като Русия и Китай са толкова различни като стратегическа култура, то кое е онова, което ги обединява? Отговорът е очевиден – стремежът им да променят баланса на силите, установен след разпадането на СССР. Макар и единна по своя характер, тази външна политика се възприема много различно от Москва и Пекин и следователно едно толкова просто обяснение не би могло да ни даде отговор на въпроса какви действително са отношенията между двете държави. Отправна точка за по-ясното им разбиране е самата концепция за „приятелство без граници“, която се превърна в крайъгълен камък на руско-китайските отношения в последните години.

Тук е редно да направим уточнението, че формулата очевидно е лансирана и изцяло изградена от Пекин поради една много проста причина – в китайската външнополитическа философия не присъства геополитическата категория „съюзници“. Китай не дефинира държавите като съюзници или врагове и никога не е оцветявал политиката си в черно и бяло, освен ако не става въпрос за териториалната цялост и суверенитета му. Конфуцианското разбиране, че Пекин е центърът на вселената, продължава да доминира политиката на Китайската комунистическа партия и в този смисъл значението на всяка държава се определя от това до каква степен тя допринася за увеличаването или намаляването на глобалното икономическо влияние на Китай. В духа на тази концепция Русия заема особено важно място заради огромните си залежи на ресурси и ядрения си арсенал, който може да послужи като сдържащ фактор пред САЩ, тъй като на този етап Пекин все още не е достигнал ядрен паритет с Вашингтон.

В същото време между Русия и Китай не съществуват никакви договорености, които да надхвърлят рамките на икономическото сътрудничество или да обвързват двете страни със задължения да се защитават една друга. Първата двустранна инициатива за сигурност беше съобщена символично на 4 юли тази година и тя предвижда създаването на Евразийски клуб за сигурност, който да гарантира неприкосновеността на сътрудничеството между двете държави и да работи за постигането на многополюсен свят в системата на международните отношения. Прави впечатление обаче, че границите на този клуб се фокусират върху Евразия и по всяка вероятност той просто ще формализира някои механизми за обмен на технологии между двете държави.

Това ще бъде необходимо за следващите десетилетия, тъй като, както стана ясно от думите на държавния секретар на САЩ Антъни Блинкен, Америка възнамерява да създаде специална програма за подкрепа на Украйна, която да позволи на Вашингтон да ѝ отпуска редовна помощ в следващите десет години. Казано с други думи, Евразийският клуб за сигурност, който по всяка вероятност ще включва държавите от настоящата Организация на Договора за колективна сигурност, по-скоро ще превърне Русия в поредния източник на геополитическо влияние за Китай. По този начин преходът от рускоцентрична към синоцентрична архитектура за сигурност в Евразия окончателно ще приключи.

Най-съществената разлика във визията на Русия и Китай обаче си остава представата им върху какви механизми трябва да стъпва многополюсният свят. За Москва такива не следва да има, защото Русия неведнъж е демонстрирала, че за нея важи правилото на по-силния. Привилегията да си ядрена сила не е чужда на нито една държава, която разполага с оръжия за масово унищожение, но идеята, че по-силният винаги печели, е особено силно застъпена в руската външна политика и във визията на Русия, че Москва следва да има своя сфера на влияние.

Концепцията за многополюсен свят на Китай, от друга страна, е далеч по-изтънчена и в този смисъл става привлекателна за онези държави, които не желаят да бъдат част от семейството на либералните демокрации. Тук има и един много важен фактор, който често се подценява от Москва. И Русия, и Китай отстояват тезата, че всяка цивилизация трябва да има свободата сама да избира образците, по които да се развива, но само Пекин следва този принцип във външната си политика.

И това не е изненада, тъй като доктрините, с които работи Китай, все пак имат хилядолетна история, която дори Културната революция не успява да заличи, докато доктрините на Русия в повечето случаи или копират американските, или се опитват да подражават на френските. Или казано накратко, ако Китай иска свят, основан на правила, то Русия иска свят, основан на силата. Това може да създаде разцепление в приятелството без граници в дългосрочен план. Но като вземем предвид, че Китай има навика да планира стратегиите си едва ли не с хилядолетия напред, то в бъдеще Русия вероятно ще е доста слаба, за да задава правилата в Евразийския клуб за сигурност.

И нека не забравяме, че накрая опираме до ултимативния фактор, дефиниращ ролята на всяка държава в глобалната архитектура за сигурност – наличието на ядрен арсенал. Ако има нещо, което сдържа Китай да не бъде по-категоричен в позицията си Русия, това са руските ядрени оръжия. Москва разполага с най-големия ядрен арсенал в света и въпреки че доста експерти спорят каква част от него е в оперативна готовност, всички все пак са обединени от мнението, че в една ядрена война не може да има победител.

Само преди няколко месеца в изтекли руски секретни файлове се появи информация, че Москва се е подготвяла да отговори при евентуална военна интервенция от страна на Пекин. Наличието на толкова сериозна стратегия за отговор означава, че Русия няма доверие на Китай и ако възстанови сферите си на влияние от бившия СССР, едва ли ще гледа на Пекин като на безграничен приятел.

Видимо е, че Пекин има сериозни резерви към стратегията на Русия и поради сближаването ѝ със Северна Корея. Китайското влияние в Пхенян е въпрос на чест за Пекин от времето на Мао, а ако Путин и Ким Чен Ун наистина изработят трайни механизми за сътрудничество в региона, това ще постави под въпрос ролята на Китай като балансьор в отношенията между Северна и Южна Корея. Китай винаги е спазвал рационална дистанция от режима в Пхенян. Китайската комунистическа партия не би рискувала да развали икономическите си отношения със Сеул в името на недвусмислена подкрепа за Северна Корея. По същия начин и Сеул не би жертвал отношенията си с Пекин в името на безусловна подкрепа за остров Тайван.

И разбира се, следва да отчетем, че китайският президент Си Дзинпин на няколко пъти спира руския си колега от това да натисне червеното копче – доказателство, че Китай няма да рискува сферите си на влияние в Европа по линия на „Един пояс – един път“ само за да легитимира ядрения шантаж на Русия. По-скоро ще използва Москва като буфер, за да отслаби позициите на Вашингтон на Стария континент.

Ще стане ли приятелството предателство?

Това, разбира се, е съкровената мечта на повечето западни политици. В първите месеци след войната в Украйна Китай беше поставен под огромен международен натиск да осъди действията на Русия и да се разграничи от нея. Западът сякаш не разбира, че китайците нямат интерес Москва нито да спечели, нито да загуби тази война. Ако спечели, това ще обърне баланса на силите в Евразия. Ако загуби, Китай би останал сам срещу САЩ и техните съюзници в Пасифика. Удължаването на войната, от друга страна, би изтощило Америка и Европа, а неутралната позиция на Китай му гарантира образа на трета страна, която, както по-късно беше оповестено в китайското предложение за мир, призовава за мир и взаимно зачитане на суверенитета и на двете страни.

И все пак нито един западен политик не успя да „вкара“ Китай в руската схема или да „изкара“ Русия от сметките на Пекин поради един много просто факт – триъгълната дипломация на Хенри Кисинджър вече е мъртва. В годините след Студената война неговата парадигма беше последователно отричана от политици като бившия американски президент Барак Обама и бившата държавна секретарка Хилъри Клинтън, които пренебрегнаха съветите на Кисинджър, че поучаването на Китай на тема „човешки права“ само ще го дистанцира от САЩ.

В този смисъл представите как Москва и Пекин се обръщат един срещу друг или как Си Дзинпин се скарва с Путин, са също толкова илюзорни (поне докато и двамата са на власт), колкото и схващането, че Западът може да спечели ядрена война с Русия. Най-малкото защото за Китай войната в Украйна е възможност Америка да бъде сдържана, без Пекин директно да участва в конфликт с нея. А за Русия различията между САЩ и Китай са възможност тя да сбъдне неоевразийската мечта на Александър Дугин за многополюсен свят.

Краят на триъгълната дипломация всъщност полага и началото на преход от глобален либерален ред към двуполюсен модел, в който Китай остава сам срещу Запада и се насочва към приятелство без граници с Русия. За Русия това е добре дошло, тъй като в края на 2022 г. Украйна изглеждаше като победител, но в момента, в който Си Дзинпин кацна в Москва, нещата се промениха коренно. Путин придоби увереност, че не е сам, а диалогът със Си Дзинпин привлече още партньори, като Иран и Северна Корея.

Европа се раздели, тъй като част от нея не искаше да обръща гръб на Пекин, а и защото много европейски политици изразиха съмнение, че САЩ биха могли да спечелят стратегическа надпревара срещу втората икономика в света и нейния ядрен щит. Макар и този въпрос все още да остава отворен, е ясно, че без подкрепата на Китай Русия не може да спечели срещу Запада. И още: ако Вашингтон и неговите съюзници искат да надделеят в тази надпревара, те трябва да се върнат към стратегиите си от времето на Студената война или да извадят нов заек от шапката.

За Русия това има огромно значение, тъй като, ако САЩ възродят триъгълната дипломация в един момент, а управлението в Китай се смени, тогава бъдещето на Евразийския клуб за сигурност определено може да се промени в посока по-голямо влияние за Пекин. Факт е обаче, че на този етап в най-удобна позиция се намира Китай, защото той няма да загуби в нито един от двата сценария за изхода от войната в Украйна, освен ако не излезе в директен сблъсък със САЩ, което ще доведе до Трета световна война. Пред Китай стои перспективата или да наследи огромния ядрен арсенал на Русия и нейните природни ресурси, или да се върне към доброто старо време, когато изигра ролята на доброто ченге срещу СССР.

И все пак как ще завърши приятелството без граници? Не и като онова между Германия и СССР. Защото, както изяснихме, стратегическите култури на двете страни са различни, макар дългосрочната им външнополитическа цел да е единна. За Русия това приятелство по-скоро ще бъде една геополитическа епитафия, предшестваща постепенния упадък, към който ще тръгне страната след края на войната в Украйна. Това се обуславя от много фактори, като демографския срив в страната, жертвите, които тя даде във войната срещу Украйна, и изолацията ѝ в глобален план. От друга страна, възходът на Азия – не просто на Китай – е необратим, освен ако ядрените сили не загубят рационалното си мислене и не решат да впрегнат арсенала си в една последна, безсмислена война. Така че да, приятелството без граници ще продължи, но както обикновено става и в най-добрите приятелства, единият приятел надживява другия.

Operation Cowboy – Saving the Lipizzaner horses

2024-07-31 The History Guy: History Deserves to Be Remembered

Post Syndicated from The History Guy: History Deserves to Be Remembered original https://www.youtube.com/watch?v=llprFG0-7GE

Comic for 2024.07.31 – Anime Figure

2024-07-31 Explosm.net

Post Syndicated from Explosm.net original https://explosm.net/comics/anime-figure

New Cyanide and Happiness Comic

Exam Numbers

2024-07-31 xkcd.com

Post Syndicated from xkcd.com original https://xkcd.com/2966/

Calligraphy exam: Write down the number 37, spelled out, nicely.

F-14 The Last Analogue Fighter – The Deadly Tomcat

2024-07-31 Curious Droid

Post Syndicated from Curious Droid original https://www.youtube.com/watch?v=GTHY68S4Wxo

Soldering Things 102 – LILYGO POE ESP32-S3 Boards

2024-07-31 digiblur DIY

Post Syndicated from digiblur DIY original https://www.youtube.com/watch?v=YgmoxtwqbZk

[$] Famfs: a filesystem interface to shareable memory

2024-07-31 jake

Post Syndicated from jake original https://lwn.net/Articles/983105/

At the 2024 Linux
Storage, Filesystem, Memory Management, and BPF Summit, John Groves led
a session on famfs, which is a filesystem he has developed that uses the
kernel’s direct-access (DAX)
mechanism to access memory that is shareable between hosts. The discussion
was aimed at whether a different approach should be taken and, in
particular, whether FUSE should be used instead of implementing as an
in-kernel filesystem. As noted in the thread about his
proposal for an LSFMM+BPF session, and the mailing-list discussions on the first and second
version of his patch set, there is some skepticism that a new in-kernel
filesystem is warranted for the use case.

How the Paris 2024 Summer Olympics has impacted Internet traffic

2024-07-30 João Tomé

Post Syndicated from João Tomé original https://blog.cloudflare.com/paris-2024-summer-olympics-impacted-internet-traffic

The Paris 2024 Summer Olympics, themed “Games Wide Open” (“Ouvrons grand les Jeux”), kicked off on Friday, July 26, 2024, and will run until August 11. A total of 10,714 athletes from 204 nations, including individual and refugee teams, will compete in 329 events across 32 sports. This blog post focuses on the opening ceremony and the initial days of the event, examining associated impact on Internet traffic, especially in France, the popularity of Olympic websites by country, and the rise in Olympics-related spam and malicious emails.

Cloudflare has a global presence with data centers in over 320 cities, supporting millions of customers, which provides a global view of what’s happening on the Internet. This is helpful for improving security, privacy, efficiency, and speed, but also for observing Internet disruptions and traffic trends.

We are closely monitoring the event through our 2024 Olympics report on Cloudflare Radar and will provide updates on significant Internet trends as they develop.

An opening ceremony to remember

For the first time in modern Olympic history, the opening ceremony was held outside a stadium, lasting nearly four hours and clearly impacting Internet traffic in France. The nation’s engagement was evident during the TV broadcast, leading to noticeable traffic drops similar to those observed during Euro 2024 – we’ve seen that national TV broadcast events usually come with drops in Internet traffic.

The Olympics are more than just sporting events – they are filled with inspiring moments and stories that capture global attention in real time, and create stories that live on. Significant traffic dips during the ceremony coincided with performances by Celine Dion and Lady Gaga, the lighting of the Olympic cauldron, and John Lennon’s “Imagine” performed by Juliette Armanet. Here is a breakdown of the top five traffic drops compared to the previous week that occurred during the ceremony, detailing the events occurring at those times. Our data provides insights with 15-minute granularity.

Moments of the ceremony by traffic drop

	Time of drop (UTC)	Drop %	Events at the time
#1	~21:15	-20%	The Olympic cauldron is lit and floats into the Paris sky via air balloon; Celine Dion serenades Paris from the Eiffel Tower.
#2	~17:45	-17%	Lady Gaga sings the French classic “Mon truc en plumes” by Zizi Jeanmaire.
#3	~19:45	-16.9%	Team USA boat takes to the river, followed by Team France – the last boat en route to the Eiffel Tower.
#4	~20:15	-16.9%	Dionysus performs the song “Naked” (Philippe Katerine); John Lennon’s “Imagine” is sung from the middle of the Seine by Juliette Armanet; a metal horse rides down the river.
#5	~18:00	-16.7%	As the boats continue along the Seine, around 80 artists from the Moulin Rouge perform the famous French cabaret dance, the can-can.

During the opening ceremony on July 26, between 17:30 to 21:20 UTC, traffic in France was noticeably lower than the previous week, with losses between 15% and 20%. However, there were moments with smaller drops. For example, at 19:30 UTC, traffic only fell by 4% during the middle of the boat parade of athletes on the Seine River. Right after the event, at 21:45 UTC, traffic increased by as much as 8% compared to the previous week.

The opening ceremony also resulted in a higher mobile share of traffic than usual in France. At 20:45 UTC, close to the end of the ceremony, the mobile share of Internet traffic was 61%, up from 57% the previous week.

Parisians leaving town before the Olympics

With the Olympics in Paris, many locals left the city, either for vacations or quieter places, while tourists arrived for the games. Our data shows that two French regions, Île-de-France, where Paris is located, and Grand Est, east of Paris, experienced the most significant traffic drops. The chart below illustrates daily traffic to these regions, with a noticeable decline visible during the weekend before the Olympics in Île-de-France.

Analyzing the percentage change in request traffic from the previous week, Île-de-France saw its largest drops in the first week of July (July 1-7), with a 15% decrease, and the week before the Olympics started, with an 8% decrease. Interestingly, there was no percentage change in traffic during the week of the Olympics (July 22-28) – that was also the week when most visitors for the Olympics started to arrive.

The daily share of mobile device traffic from France also reveals shifts in typical patterns, with increases noted especially after the June 30 weekend, indicative of vacation periods and leisure Internet use. Mobile device traffic peaked during the first Olympic weekend, reaching 53% on July 26, the day of the opening ceremony – higher than any previous Friday since June. On Sunday, July 28, mobile device traffic peaked at 58%, the highest since June.

Impact to Internet traffic outside of France

Globally, Internet traffic variations were less pronounced than in France. However, on July 26, the day of the opening ceremony, a noticeable global drop occurred during the event. This was particularly evident during two key moments previously highlighted: during song performances at 20:15 UTC, traffic dropped 3% compared to the previous week, and around the end of the ceremony, at 21:15 UTC, it dropped 2%.

Expanding our view to other countries, moments of significant drops in traffic during the opening ceremony were clearly visible. Below is a summary list of 30 countries selected based on their tally of Summer Olympic medals.

Country	Drop in traffic (%)	Time of drop (UTC)
United States	-4%	20:15
Great Britain	-8%	20:15
France	-20%	21:15
Germany	-4%	20:15
China	-4%	21:00
Italy	-11%	18:15
Australia	-2%	20:00
Hungary	-5%	21:15
Sweden	-4%	21:15
Japan	-12%	21:15
Russia	-7%	19:45
Canada	-3%	20:15
Netherlands	-6%	21:15
Romania	-12%	20:00
Finland	-12%	17:30
Poland	-5%	21:15
South Korea	-4%	20:15
Cuba	-3%	19:00
Bulgaria	-6%	21:15
Switzerland	-10%	18:15
Denmark	-2%	21:15
Spain	-8%	18:15
Norway	-2%	21:15
Belgium	-5%	21:15
Brazil	-3%	18:15
Czech Republic	-10%	18:00
Slovakia	-11%	20:15
Ukraine	-2%	20:45
New Zealand	-9%	21:15
Greece	-11%	18:00

Additionally, the world map below highlights the countries that experienced notable Internet traffic impacts during the opening ceremony.

(Source: Cloudflare; created with Datawrapper)

Outside Europe, the countries with the most substantial drops were New Zealand (-9%), Uzbekistan (-12%), Argentina (-13%), and Mongolia -(20%), all experiencing greater declines than those in Europe.

Significant moments at the games: from Simone Biles to Olympic records

Below, we highlight specific Olympic events affecting Internet traffic, starting from the first full competition day on Saturday, July 27, 2024.

United States: The artistic gymnastics competition featuring four-time Olympic gold medalist Simone Biles notably impacted US Internet traffic more than the opening ceremony. On July 26-28, traffic dipped most significantly during Biles’ events. At 10:00 UTC, concurrent with her beam routine, traffic was already 4% lower than the previous week. It dropped by 6% at 10:45 UTC during her floor and vault routines.

France: French swimmer Léon Marchand’s gold medal and Olympic record-setting performance in the men’s 400-meter individual medley on July 28 had the most significant impact in the host nation. Traffic fell by 17% at 18:30 UTC during his event. However, as we noted above, the opening ceremony drove a bigger drop in traffic.

Australia: During Mollie O’Callaghan’s victory in the women’s 200m freestyle on July 29, at around 20:00 UTC, Australian traffic was 5% lower than the previous week This was larger than during the opening ceremony, which saw a 2% drop.

South Korea: The Korean women’s archery team’s gold medal win on July 28 at 15:30 UTC led to an 8% drop in traffic, the most significant decrease noted in the country from July 26 to July 29.

Brazil: Traffic in Brazil was15% lower than the previous week on July 27 at around 19:30 UTC, surpassing the opening ceremony’s impact. This occurred as Brazilian swimmers Guilherme Costa and Maria Fernanda Costa competed in the men’s and women’s 400 m freestyle events.

DNS trends to official Olympic websites by country

On July 22, before the Olympics started, we reported on the heightened interest in official Olympic websites based on request data from our 1.1.1.1 DNS resolver. We noted France’s dominance with 24% of DNS traffic to official Olympic websites, followed by the UK (20%) and the US (17%). However, the start of the Olympics marked a shift, with the US taking the lead.

On the first full day of competitions, July 27, the US led with 16% of all DNS request traffic to official Olympic sites. This change indicates a broader spread of interest across countries during the Olympics. A dynamic version of the map below is available in our Paris 2024 Olympics report.

Here are the top 10 countries with the highest shares of DNS request traffic for the first full day of competitions, July 27, to Olympic sites (percentages rounded):

United States: 16%
Germany: 12%
France: 9%
Vietnam: 9%
Brazil: 5%
Australia: 5%
United Kingdom: 4%
Netherlands: 4%
Canada: 3%
South Africa: 2%

Growth in interest as the Olympics drew closer

Global daily DNS request traffic to official Olympic websites began climbing to the highest levels seen year to date starting on July 23, showing a steady increase. It peaked on July 28, the second full day of events, with a fivefold (509%) increase from the previous week. On the opening ceremony day, traffic was already 110% higher than the previous week.

Country-specific peaks included the US, where traffic to Olympic sites surged 719% on July 28, coinciding with Simone Biles’ first competition day. In France, traffic peaked on the same day with a 391% increase, and in Germany, it skyrocketed by 2300% on July 27.

The evolving DNS ranking of Olympic site traffic by country reveals that from July 19, the US overtook France. Also, Germany ascended to the #2 spot on July 27, the first full day of competitions, while Australia climbed to #4 on July 28, and Canada’s peak day was also July 28.

Railway attacks on opening ceremony day cause surge in traffic

The opening ceremony day, July 26, was also disrupted by railway arson attacks in France, affecting the 800,000 passengers on the high-speed railway system. At 10:00 UTC, there was a significant surge in DNS traffic to public transportation websites, including high-speed railway services. Traffic spiked by 2000% compared to the previous week as users accessed websites to check updates.

DDoS attacks: always around

As we’ve observed with elections in 2024, including the French elections, political parties are not the only targets of DDoS (Distributed Denial of Service) attacks during significant events. While we haven’t seen any coordinated flow of major DDoS attacks targeting services potentially used during the Olympics in France, we have observed a few incidents.

A generally used French government website was targeted by a DDoS attack on July 29, 2024, lasting nine minutes and peaked at 207,000 requests per second at 20:34 UTC.

Before the Olympics began, a national transportation website was also targeted by a smaller DDoS attack, lasting only a couple of minutes and peaking at 10,000 requests per second on July 21 at 10:20 UTC.

As highlighted in our Q2 DDoS report, most DDoS attacks are short-lived, as exemplified by the two mentioned attacks. Also, 81% of HTTP DDoS attacks peak at under 50,000 requests per second (rps), and only 7% reach between 100,000 and 250,000 rps. While a 10,000 rps attack might seem minor to Cloudflare, it can be devastating for websites not equipped to handle such high levels of traffic.

“Olympics” and “Paris 2024” emails on the rise

From another cybersecurity perspective, major events often attract phishing and spam, and the Olympics are no exception. From January 2024 through late July, Cloudflare’s Cloud Email Security service processed over a million emails containing “Olympics” or “Paris 2024” in the subject. During the week of July 22-28, coinciding with the first few days of the Olympics, there was a 304% increase in such emails compared to the previous week and a staggering 3111% increase compared to the busiest week in January.

Regarding unwanted messages, spam accounted for 1.5% of all emails with “Olympics” or “Paris 2024” in the subject, while malicious emails made up 0.1% since January 2024. This means that in a sample of 1000 emails, roughly 15 would be spam and 1 would be malicious. The peak for malicious Olympic-related emails occurred the week of May 6, with 0.6% classified as malicious. Although there was a decline after this peak, rates increased slightly in July, reaching 0.4% on July 8. Despite the surge in volume during the week of July 22, only 0.05% of emails were malicious.

That same week, when the Olympics started, also saw an increase in spam emails to over 2%, the highest since the 7% peak the week of June 24.

Conclusion

The Paris 2024 Olympics started on July 26, with a clear impact on Internet traffic in different countries, most notably in France, the host nation. The significant traffic drops during key moments of the opening ceremony, and the reactive spikes following major events highlight the ever-present interplay between physical events and the way humans interact with the online world. Not many events take the focus away from the Internet, and in this case, into TV broadcast.

We’ve also observed how the interest in official Olympic websites surged, with clear increases in DNS traffic after the event started, in different countries, with the US ultimately taking the gold.

Regarding the July 29, 2024 sabotage of French fiber optic cables, we did not observe any notable disruptions of Internet traffic in France or its cities during the day.

As the games continue, we will maintain a Paris 2024 Olympics report on Cloudflare Radar, updating it as significant Internet trends related to the event emerge.

How the Paris 2024 Summer Olympics has impacted Internet traffic

2024-07-30 João Tomé

Post Syndicated from João Tomé original https://blog.cloudflare.com/paris-2024-summer-olympics-impacted-internet-traffic

We are closely monitoring the event through our 2024 Olympics report on Cloudflare Radar and will provide updates on significant Internet trends as they develop.

An opening ceremony to remember

Moments of the ceremony by traffic drop

	Time of drop (UTC)	Drop %	Events at the time
#1	~21:15	-20%	The Olympic cauldron is lit and floats into the Paris sky via air balloon; Celine Dion serenades Paris from the Eiffel Tower.
#2	~17:45	-17%	Lady Gaga sings the French classic “Mon truc en plumes” by Zizi Jeanmaire.
#3	~19:45	-16.9%	Team USA boat takes to the river, followed by Team France – the last boat en route to the Eiffel Tower.
#4	~20:15	-16.9%	Dionysus performs the song “Naked” (Philippe Katerine); John Lennon’s “Imagine” is sung from the middle of the Seine by Juliette Armanet; a metal horse rides down the river.
#5	~18:00	-16.7%	As the boats continue along the Seine, around 80 artists from the Moulin Rouge perform the famous French cabaret dance, the can-can.

Parisians leaving town before the Olympics

Impact to Internet traffic outside of France

Country	Drop in traffic (%)	Time of drop (UTC)
United States	-4%	20:15
Great Britain	-8%	20:15
France	-20%	21:15
Germany	-4%	20:15
China	-4%	21:00
Italy	-11%	18:15
Australia	-2%	20:00
Hungary	-5%	21:15
Sweden	-4%	21:15
Japan	-12%	21:15
Russia	-7%	19:45
Canada	-3%	20:15
Netherlands	-6%	21:15
Romania	-12%	20:00
Finland	-12%	17:30
Poland	-5%	21:15
South Korea	-4%	20:15
Cuba	-3%	19:00
Bulgaria	-6%	21:15
Switzerland	-10%	18:15
Denmark	-2%	21:15
Spain	-8%	18:15
Norway	-2%	21:15
Belgium	-5%	21:15
Brazil	-3%	18:15
Czech Republic	-10%	18:00
Slovakia	-11%	20:15
Ukraine	-2%	20:45
New Zealand	-9%	21:15
Greece	-11%	18:00

Additionally, the world map below highlights the countries that experienced notable Internet traffic impacts during the opening ceremony.

(Source: Cloudflare; created with Datawrapper)

Outside Europe, the countries with the most substantial drops were New Zealand (-9%), Uzbekistan (-12%), Argentina (-13%), and Mongolia -(20%), all experiencing greater declines than those in Europe.

Significant moments at the games: from Simone Biles to Olympic records

Below, we highlight specific Olympic events affecting Internet traffic, starting from the first full competition day on Saturday, July 27, 2024.

South Korea: The Korean women’s archery team’s gold medal win on July 28 at 15:30 UTC led to an 8% drop in traffic, the most significant decrease noted in the country from July 26 to July 29.

DNS trends to official Olympic websites by country

Here are the top 10 countries with the highest shares of DNS request traffic for the first full day of competitions, July 27, to Olympic sites (percentages rounded):

United States: 16%
Germany: 12%
France: 9%
Vietnam: 9%
Brazil: 5%
Australia: 5%
United Kingdom: 4%
Netherlands: 4%
Canada: 3%
South Africa: 2%

Growth in interest as the Olympics drew closer

Railway attacks on opening ceremony day cause surge in traffic

DDoS attacks: always around

A generally used French government website was targeted by a DDoS attack on July 29, 2024, lasting nine minutes and peaked at 207,000 requests per second at 20:34 UTC.

“Olympics” and “Paris 2024” emails on the rise

That same week, when the Olympics started, also saw an increase in spam emails to over 2%, the highest since the 7% peak the week of June 24.

Conclusion

Regarding the July 29, 2024 sabotage of French fiber optic cables, we did not observe any notable disruptions of Internet traffic in France or its cities during the day.

As the games continue, we will maintain a Paris 2024 Olympics report on Cloudflare Radar, updating it as significant Internet trends related to the event emerge.

MikroTik CRS520-4XS-16XQ-RM Review MikroTik Scales Up 100GbE

2024-07-30 Rohit Kumar

Post Syndicated from Rohit Kumar original https://www.servethehome.com/mikrotik-crs520-4xs-16xq-rm-review-mikrotik-scales-up-100gbe-marvell-annapurna-arm/

We review the MikroTik CRS520-4XS-16XQ-RM, the company’s first 16-port 100GbE switch with a big Arm CPU, a low price, and easy management

The post MikroTik CRS520-4XS-16XQ-RM Review MikroTik Scales Up 100GbE appeared first on ServeTheHome.

Integrate Amazon MWAA with Microsoft Entra ID using SAML authentication

2024-07-30 Satya Chikkala

Post Syndicated from Satya Chikkala original https://aws.amazon.com/blogs/big-data/integrate-amazon-mwaa-with-microsoft-entra-id-using-saml-authentication/

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) provides a fully managed solution for orchestrating and automating complex workflows in the cloud. Amazon MWAA offers two network access modes for accessing the Apache Airflow web UI in your environments: public and private. Customers often deploy Amazon MWAA in private mode and want to use existing login authentication mechanisms and single sign-on (SSO) features to have seamless integration with the corporate Active Directory (AD). Also, the end-users don’t need to log in to the AWS Management Console to access the Airflow UI.

In this post, we illustrate how to configure an Amazon MWAA environment deployed in private network access mode with customer managed VPC endpoints and authenticate users using SAML federated identity using Microsoft Entra ID and Application Load Balancer (ALB). Users can seamlessly log in to the Airflow UI with their corporate credentials and access the DAGs. This solution can be modified for Amazon MWAA public network access mode as well.

Solution overview

The architectural components involved in authenticating the Amazon MWAA environment using SAML SSO are depicted in the following diagram. The infrastructure components include two public subnets and three private subnets. The public subnets are required for the internet-facing ALB. Two private subnets are used to set up the Amazon MWAA environment, and the third private subnet is used to host the AWS Lambda authorizer function. This subnet will have a NAT gateway attached to it, because the function needs to verify the signer to confirm the JWT header has the expected LoadBalancer ARN.

The workflow consists of the following steps:

For SAML configuration, Microsoft Entra ID serves as the identity provider (IdP).
Amazon Cognito serves as the service provider (SP).
ALB has built-in support for Amazon Cognito and authenticates requests.
Post-authentication, ALB forwards the requests to the Lambda authorizer function. The Lambda function decodes the user’s JWT token and validates whether the user’s AD group is mapped to the relevant AWS Identity and Access Management (IAM) role.
If valid, the function creates a web login token and redirects to the Amazon MWAA environment for successful login.

The following are the high-level steps to deploy the solution:

Create an Amazon Simple Storage Service (Amazon S3) bucket for artifacts.
Create an SSL certificate and upload it to AWS Certificate Manager (ACM).
Deploy the Amazon MWAA infrastructure stack using AWS CloudFormation.
Configure Microsoft Entra ID services and integrate the Amazon Cognito user pool.
Deploy the ALB CloudFormation stack.
Log in to Amazon MWAA using Microsoft Entra ID user credentials.

Prerequisites

Before you get started, make sure you have the following prerequisites:

An AWS account
Appropriate IAM permissions to deploy AWS CloudFormation stack resources
A Microsoft Azure account is required for creating the Microsoft Entra ID app (IdP config) and Microsoft Entra ID P2.
A public certificate for the ALB in the AWS Region where the infrastructure is being deployed and a custom domain name relevant to the certificate.

Create an S3 bucket

In this step, we create an S3 bucket to store your Airflow DAGs, custom plugins in a plugins.zip file, and Python dependencies in a requirements.txt file. This bucket is used by the Amazon MWAA environment to fetch DAGs and dependency files.

On the Amazon S3 console, choose the Region where you want to create a bucket.
In the navigation pane, choose Buckets.
Choose Create bucket.
For Bucket type, select General purpose.
For Bucket name, enter a name for your bucket (for this post, mwaa-sso-blog-<your-aws-account-number>).
Choose Create bucket.
Navigate to the bucket and choose Create folder.
For Folder name, enter a name (for this post, we name the folder dags).
Choose Create folder.

Import certificates into ACM

ACM is integrated with Elastic Load Balancing (ALB). In this step, you can request a public certificate using ACM or import a certificate into ACM. To import organization certificates linked to a custom DNS into ACM, you must provide the certificate and its private key. To import a certificate signed by a non-AWS Certificate Authority (CA), you must also include the private and public keys of the certificate.

On the ACM console, choose Import certificate in the navigation pane.
For Certificate body, enter the contents of the cert.pem file.
For Certificate private key, enter the contents of the privatekey.pem file.
Choose Next.
Choose Review and import.
Review the metadata about your certificate and choose Import.

After the import is successful, the status of the imported certificate will show as Issued.

Create the Azure AD service, users, groups, and enterprise application

For the SSO integration with Azure, an enterprise application is required, which acts as the IdP for the SAML flow. We add relevant users and groups to the application and configure the SP (Amazon Cognito) details.

Airflow comes with five default roles: Public, Admin, Op, User, Viewer. In this post, we focus on three: Admin , User and Viewer. We create three roles and three corresponding users and assign memberships appropriately.

Log in to the Azure portal.
Navigate to Enterprise applications and choose New application.
Enter a name for your application (for example, mwaa-environment) and choose Create.

You can now view the details of your application.

Now you create two groups.
In the search bar, search for Microsoft Entra ID.
On the Add menu, choose Group.
For Group type, choose a type (for this post, Security).
Enter a group name (for example, airflow-admins) and description.
Choose Create.
Repeat these steps to create two more groups, named airflow-users and airflow-viewers.
Note the object IDs for each group (these are required in a later step).

Next, you create users.
On the Overview page, on the Add menu, choose User and Create new user.
Enter a name for your user (for example, mwaa-user), display name, and password.
Choose Review + create.
Repeat these steps to create a user called mwaa-admin.
In your airflow-users group details page, choose Members in the navigation pane.
Choose Add members.
Search for and select the users you created and choose Select.
Repeat these steps to add the users to each group.
Navigate to your application and choose Assign users and groups.
Choose Add user/group.
Search for and select the groups you created, then choose Select.

Deploy the Amazon MWAA environment stack

For this solution, we provide two CloudFormation templates that set up the services illustrated in the architecture. Deploying the CloudFormation stacks in your account incurs AWS usage charges.

The first CloudFormation stack creates the following resources:

A VPC with two public subnets and three private subnets and relevant route tables, NAT gateway, internet gateway, and security group
VPC endpoints required for the Amazon MWAA environment
An Amazon Cognito user pool and user pool domain
Application Load Balancer

Deploy the stack by completing the following steps:

Choose Launch Stack to launch the CloudFormation stack.
For Stack name, enter a name (for example, sso-blog-mwaa-infra-stack).
Enter the following parameters:
1. For MWAAEnvironmentName, enter the environment name.
2. For MwaaS3Bucket, enter the S3 artifacts bucket you created.
3. For VpcCIDR, enter the specify IP range (CIDR notation) for this VPC.
4. For PrivateSubnet1CIDR, enter the IP range (CIDR notation) for the private subnet in the first Availability Zone.
5. For PrivateSubnet2CIDR, enter the IP range (CIDR notation) for the private subnet in the second Availability Zone.
6. For PrivateSubnet3CIDR, enter the IP range (CIDR notation) for the private subnet in the third Availability Zone.
7. For PublicSubnet1CIDR, enter the IP range (CIDR notation) for the public subnet in the first Availability Zone.
8. For PublicSubnet2CIDR, enter the IP range (CIDR notation) for the public subnet in the second Availability Zone.
Choose Next
Review the template and choose Create stack.

After the stack is deployed successfully, you can view the resources on the stack’s Outputs tab on the AWS CloudFormation console. Note the ALB URL, Amazon Cognito user pool ID, and domain.

Integrate the Amazon MWAA application with the Azure enterprise application

Next, you configure the SAML configuration in the enterprise application by adding the SP details and redirect URLs (in this case, the Amazon Cognito details and ALB URL).

In the Azure portal, navigate to your environment.
Choose Set up single sign on.
For Identifier, enter urn:amazon:cognito:sp:<your cognito user_id>.
For Reply URL, enter https://<Your user pool domain>/saml2/idpresponse.
For Sign on URL, enter https://<Your application load balancer DNS>.
In the Attributes & Claims section, choose Add a group claim.
Select Security groups.
For Source attribute, choose Group ID.
Choose Save.
Note the values for App Federation Metadata Url and Login URL.

Deploy the ALB stack

When the SAML configuration is complete on the Azure end, the IdP details have to be configured in Amazon Cognito. When users access the ALB URL, they will be authenticated against the corporate identity using SAML through Amazon Cognito. After they’re authenticated, they’re redirected to the Lambda function for authorization against the group they belong to. The user’s group is then validated against matching IAM role. If it’s valid, the Lambda function adds the web login token to the URL, and the user will gain access to the Amazon MWAA environment.

This CloudFormation stack creates the following resources:

Two target groups: the Lambda target group and Amazon MWAA target group
Listener rules for the ALB to redirect URL requests to the relevant target groups
A user pool client and SAML provider (Azure) details to the Amazon Cognito user pool
IAM roles for Admin, User, and Viewer personas required for Airflow
The Lambda authorizer function to validate the JWT token and map Azure groups to IAM roles for appropriate Airflow UI access

Deploy the stack by completing the following steps:

Choose Launch Stack to launch the CloudFormation stack:
For Stack name, enter a name (for example, sso-blog-mwaa-alb-stack).
Enter the following parameters:
1. For MWAAEnvironmentName, enter your environment name.
2. For ALBCertificateArn, enter the certificate ARN required for ALB.
3. For AzureAdminGroupID, enter the group name for the Azure Admin persona.
4. For AzureUserGroupID, enter the group name for the Azure User persona.
5. For AzureViewerGroupID, enter the group name for the Azure Viewer persona.
6. For EntraIDLoginURL, enter the Azure IdP URI.
7. For AppFederationMetadataURL, enter the URL of the metadata file for the SAML provider.
Choose Next.
Review the template and choose Create stack.

Test the solution

Now that the SAML configuration and relevant AWS services are created, it’s time to access the Amazon MWAA environment.

Open your web browser and enter the ALB DNS name.
The SP initiates the sign-in request process and the browser redirects you to the Microsoft login page for credentials.
Enter the Admin user credentials.

The SAML request sign-in process completes and the SAML response is redirected to the Amazon Cognito user pool attached to the ALB.

The listener rules will validate the query URL and pass the requests to the Lambda authorizer to validate the JWT and assign the appropriate group (Azure) to role (AWS) mapping.
Repeat the steps to log in with User and Viewer credentials and observe the differences in access.

Clean up

When you’re done experimenting with this solution, it’s essential to clean up your resources to avoid incurring AWS charges.

On the AWS CloudFormation console, delete the stacks you created.

Remove the SSM parameters and private webserver and database VPC endpoints (created by the Lambda events function):

aws ssm delete-parameters --names "MyFirstParameter" "MySecondParameter"
aws ec2 delete-vpc-endpoints --vpc-endpoint-ids "Endpoint1" "Endpoint2"

Delete the users, groups, and enterprise application in the Azure environment.

Conclusion

In this post, we demonstrated how to integrate Amazon MWAA with organization Azure AD services. We walked through the solution that solves this problem using infrastructure as code. This solution allows different end-user personas in your organization to access the Amazon MWAA Airflow UI using SAML SSO.

For additional details and code examples for Amazon MWAA, visit the Amazon MWAA User Guide and the Amazon MWAA examples GitHub repo.

About the Authors

Satya Chikkala is a Solutions Architect at Amazon Web Services. Based in Melbourne, Australia, he works closely with enterprise customers to accelerate their cloud journey. Beyond work, he is very passionate about nature and photography.

Vijay Velpula is a Data Lake Architect with AWS Professional Services. He assists customers in building modern data platforms by implementing big data and analytics solutions. Outside of his professional responsibilities, Velpula enjoys spending quality time with his family, as well as indulging in travel, hiking, and biking activities.

Funtoo Linux is being discontinued

2024-07-30 jzb

Post Syndicated from jzb original https://lwn.net/Articles/983977/

Daniel Robbins, founder of the Gentoo Linux distribution and its
spinoff Funtoo Linux, has
announced
that he has decided to end the Funtoo project:

Funtoo started as a philosophy to create a fun
community of contributors building something great together. For me,
it’s no longer that so I need to move on to other things. There is not
a successor BDFL for Funtoo nor am I interested in trying to find one,
or hand the project off to someone else. You can expect the project to
wind down through August. If you have a Funtoo
container, it will continue to be online through the end of August
so you have time to find another hosting solution if you need one.

Get Excited Marvell Structera CXL Memory with Arm Neoverse V2

2024-07-30 Patrick Kennedy

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/everyone-reading-sth-get-excited-marvell-structera-cxl-memory-with-arm-neoverse-v2/

Marvell Structera CXL Memory Expansion modules accept DDR4 or DDR5. The line also has a 16-core Arm Neoverse V2 accelerated memory expander

The post Get Excited Marvell Structera CXL Memory with Arm Neoverse V2 appeared first on ServeTheHome.

Federating access to Amazon DataZone with AWS IAM Identity Center and Okta

2024-07-30 Carlos Gallegos

Post Syndicated from Carlos Gallegos original https://aws.amazon.com/blogs/big-data/federating-access-to-amazon-datazone-with-aws-iam-identity-center-and-okta/

Many customers rely today on Okta or other identity providers (IdPs) to federate access to their technology stack and tools. With federation, security teams can centralize user management in a single place, which helps simplify and brings agility to their day-to-day operations while keeping highest security standards.

To help develop a data-driven culture, everyone inside an organization can use Amazon DataZone. To realize the benefits of using Amazon DataZone for governing data and making it discoverable and available across different teams for collaboration, customers integrate it with their current technology stack. Handling access through their identity provider and preserving a familiar single sign-on (SSO) experience enables customers to extend the use of Amazon DataZone to users across teams in the organization without any friction while keeping centralized control.

Amazon DataZone is a fully managed data management service that makes it faster and simpler for customers to catalog, discover, share, and govern data stored across Amazon Web Services (AWS), on premises, and third-party sources. It also makes it simpler for data producers, analysts, and business users to access data throughout an organization so that they can discover, use, and collaborate to derive data-driven insights.

You can use AWS IAM Identity Center to securely create and manage identities for your organization’s workforce, or sync and use identities that are already set up and available in Okta or other identity provider, to keep centralized control of them. With IAM Identity Center you can also manage the SSO experience of your organization centrally, across your AWS accounts and applications.

This post guides you through the process of setting up Okta as an identity provider for signing in users to Amazon DataZone. The process uses IAM Identity Center and its native integration with Amazon DataZone to integrate with external identity providers. Note that, even though this post focuses on Okta, the presented pattern relies on the SAML 2.0 standard and so can be replicated with other identity providers.

Prerequisites

To build the solution presented in this post, you must have:

A developer or licensed Okta account along with administrative access to manage users and permissions.
IAM Identity Center administrator access for your single or multi-account environment. For more information, see Enabling AWS IAM Identity Center.
A provisioned Amazon DataZone domain. For more information, see Create domains.
Have IAM Identity Center for Amazon DataZone enabled. For more information, see Setting up AWS IAM Identity Center for Amazon DataZone.

Process overview

Throughout this post you’ll follow these high-level steps:

Establish a SAML connection between Okta and IAM Identity Center
Set up automatic provisioning of users and groups in IAM Identity Center so that users and groups in the Okta domain are created in Identity Center.
Assign users and groups to your AWS accounts in IAM Identity Center by assuming an AWS Identity and Access Management (IAM) role.
Access the AWS Management Console and Amazon DataZone portal through Okta SSO.
Manage Amazon DataZone specific permissions in the Amazon DataZone portal.

Setting up user federation with Okta and IAM Identity Center

This guide follows the steps in Configure SAML and SCIM with Okta and IAM Identity Center.

Before you get started, review the following items in your Okta setup:

Every Okta user must have a First name, Last name, Username and Display name value specified.
Each Okta user has only a single value per data attribute, such as email address or phone number. Users that have multiple values will fail to synchronize. If there are users that have multiple values in their attributes, remove the duplicate attributes before attempting to provision the user in IAM Identity Center. For example, only one phone number attribute can be synchronized. Because the default phone number attribute is work phone, use the work phone attribute to store the user’s phone number, even if the phone number for the user is a home phone or a mobile phone.
If you update a user’s address you must have streetAddress, city, state, zipCode and the countryCode value specified. If any of these values aren’t specified for the Okta user at the time of synchronization, the user (or changes to the user) won’t be provisioned.

Okta account

1) Establish a SAML connection between Okta and AWS IAM Identity Center

Now, let’s establish a SAML connection between Okta and AWS IAM Identity Center. First, you’ll create an application in Okta to establish the connection:

Sign in to the Okta admin dashboard, expand Applications, then select Applications.
On the Applications page, choose Browse App Catalog.
In the search box, enter AWS IAM Identity Center, then select the app to add the IAM Identity Center app.

IAM identity center app in Okta

Choose the Sign On tab.

IAM identity center app in Okta - sign on

Under SAML Signing Certificates, select Actions, and then select View IdP Metadata. A new browser tab opens showing the document tree of an XML file. Select all of the XML from <md:EntityDescriptor> to </md:EntityDescriptor> and copy it to a text file.
Save the text file as metadata.xml.

Identity provider metadata in Okta

Leave the Okta admin dashboard open, you will continue using it in the later steps.

Second, you’re going to set up Okta as an external identity provider in IAM Identity Center:

Open the IAM Identity Center console as a user with administrative privileges.
Choose Settings in the navigation pane.
On the Settings page, choose Actions, and then select Change identity source.

Identity provider source in IAM identity center

Under Choose identity source, select External identity provider, and then choose Next.

Identity provider source in IAM identity center

Under Configure external identity provider, do the following:
1. Under Service provider metadata, choose Download metadata file to download the IAM Identity Center metadata file and save it on your system. You will provide the Identity Center SAML metadata file to Okta later in this tutorial.
  1. Copy the following items to a text file for easy access (you’ll need these values later):
    - IAM Identity Center Assertion Consumer Service (ACS) URL
    - IAM Identity Center issuer URL
2. Under Identity provider metadata, under IdP SAML metadata, choose Choose file and then select the metadata.xml file you created in the previous step.
3. Choose Next.
After you read the disclaimer and are ready to proceed, enter accept.
Choose Change identity source.

Identity provider source in IAM identity center

Leave the AWS console open, because you will use it in the next procedure.

Return to the Okta admin dashboard and choose the Sign On tab of the IAM Identity Center app, then choose Edit.
Under Advanced Sign-on Settings enter the following:
1. For ACS URL, enter the value you copied for IAM Identity Center Assertion Consumer Service (ACS) URL.
2. For Issuer URL, enter the value you copied for IAM Identity Center issuer URL.
3. For Application username format, select one of the options from the drop-down menu.
  Make sure the value you select is unique for each user. For this tutorial, select Okta username.
Choose Save.

IAM identity center app in Okta - sign on

2) Set up automatic provisioning of users and groups in AWS IAM Identity Center

You are now able to set up automatic provisioning of users from Okta into IAM Identity Center. Leave the Okta admin dashboard open and return to the IAM Identity Center console for the next step.

In the IAM Identity Center console, on the Settings page, locate the Automatic provisioning information box, and then choose Enable. This enables automatic provisioning in IAM Identity Center and displays the necessary System for Cross-domain Identity Management (SCIM) endpoint and access token information.

Automatic provisioning in IAM identity center

In the Inbound automatic provisioning dialog box, copy each of the values for the following options:
- SCIM endpoint
- Access token

You will use these values to configure provisioning in Okta later.

Choose Close.

Automatic provisioning in IAM identity center

Return to the Okta admin dashboard and navigate to the IAM Identity Center app.
On the AWS IAM Identity Center app page, choose the Provisioning tab, and then in the navigation pane, under Settings, choose Integration.
Choose Edit, and then select the check box next to Enable API integration to enable provisioning.
Configure Okta with the SCIM provisioning values from IAM Identity Center that you copied earlier:
1. In the Base URL field, enter the SCIM endpoint Make sure that you remove the trailing forward slash at the end of the URL.
2. In the API Token field, enter the Access token value.
Choose Test API Credentials to verify the credentials entered are valid. The message AWS IAM Identity Center was verified successfully! displays.
Choose Save. You are taken to the Settings area, with Integration selected.

API Integration in Okta

Review the following setup before moving forward. In the Provisioning tab, in the navigation pane under Settings, choose To App. Check that all options are enabled. They should be enabled by default, but if not, enable them.

Application provision in Okta

3) Assign users and groups to your AWS accounts in AWS IAM Identity Center by assuming an AWS IAM role

By default, no groups nor users are assigned to your Okta IAM Identity Center app. Complete the following steps to synchronize users with IAM Identity Center.

In the Okta IAM Identity Center app page, choose the Assignments tab. You can assign both people and groups to the IAM Identity Center app.
1. To assign people:
  1. In the Assignments page, choose Assign, and then choose Assign to people.
  2. Select the Okta users that you want to have access to the IAM Identity Center app. Choose Assign, choose Save and Go Back, and then choose Done.
    This starts the process of provisioning the individual users into IAM Identity Center.
1. To assign groups:
  1. Choose the Push Groups tab. You can create rules to automatically provision Okta groups into IAM Identity Center.
  1. Choose the Push Groups drop-down list and select Find groups by rule.
  2. In the By rule section, set a rule name and a condition. For this post we’re using AWS SSO Rule as rule name and starts with awssso as a group name condition. This condition can be different depending on the name of the group you want to sync.
  3. Choose Create Rule
  1. (Optional) To create a new group choose Directory in the navigation pane, and then choose Groups.
  1. Choose Add group and enter a name, and then choose Save.
  1. After you have created the group, you can assign people to it. Select the group name to manage the group’s users.
  1. Choose Assign people and select the users that you want to assign to the group.
  1. You will see the users that are assigned to the group.
  1. Going back to Applications in the navigation pane, select the AWS IAM Identity Center app and choose the Push Groups tab. You should have the groups that match the rule synchronized between Okta and IAM Identity Center. The group status should be set to Active after the group and its members are updated in Identity Center.

Return to the IAM Identity Center console. In the navigation pane, choose Users. You should see the user list that was updated by Okta.

Active users in IAM identity center

In the left navigation, select Groups, you should see the group list that was updated by Okta.

Active groups in IAM identity center

Congratulations! You have successfully set up a SAML connection between Okta and AWS and have verified that automatic provisioning is working.

OPTIONAL: If you need to provide Amazon DataZone console access to the Okta users and groups, you can manage these permissions through the IAM Identity Center console.

In the IAM Identity Center navigation pane, under Multi-account permissions, choose AWS accounts.
On the AWS accounts page, the Organizational structure displays your organizational root with your accounts underneath it in the hierarchy. Select the checkbox for your management account, then choose Assign users or groups.

IAM Roles in IAM identity center

The Assign users and groups workflow displays. It consists of three steps:
1. For Step 1: Select users and groups choose the user that will be performing the administrator job function. Then choose Next.
2. For Step 2: Select permission sets choose Create permission set to open a new tab that steps you through the three sub-steps involved in creating a permission set.
  1. For Step 1: Select permission set type complete the following:
    - In Permission set type, choose Predefined permission set.
    - In Policy for predefined permission set, choose AdministratorAccess.
  2. Choose Next.
  3. For Step 2: Specify permission set details, keep the default settings, and choose Next.
    The default settings create a permission set named AdministratorAccess with session duration set to one hour. You can also specify reduced permissions with a custom policy just to allow Amazon DataZone console access.
  4. For Step 3: Review and create, verify that the Permission set type uses the AWS managed policy AdministratorAccess or your custom policy. Choose Create. On the Permission sets page, a notification appears informing you that the permission set was created. You can close this tab in your web browser now.
On the Assign users and groups browser tab, you are still on Step 2: Select permission sets from which you started the create permission set workflow.
In the Permissions sets area, Refresh. The AdministratorAccess permission or your custom policy set you created appears in the list. Select the checkbox for that permission set, and then choose Next.

IAM Roles in IAM identity center

1. For Step 3: Review and submit review the selected user and permission set, then choose Submit.
  The page updates with a message that your AWS account is being configured. Wait until the process completes.
2. You are returned to the AWS accounts page. A notification message informs you that your AWS account has been re-provisioned, and the updated permission set is applied. When a user signs in, they will have the option of choosing the AdministratorAccess role or a custom policy role.

4) Access the AWS console and Amazon DataZone portal through Okta SSO

Now, you can test your user access into the console and Amazon DataZone portal using the Okta external identity application.

Sign in to the Okta dashboard using a test user account.
Under My Apps, select the AWS IAM Identity Center icon.

IAM identity center access in Okta

Complete the authentication process using your Okta credentials.

IAM identity center access in Okta

4.1) For administrative users

You’re signed in to the portal and can see the AWS account icon. Expand that icon to see the list of AWS accounts that the user can access. In this tutorial, you worked with a single account, so expanding the icon only shows one account.
Select the account to display the permission sets available to the user. In this tutorial you created the AdministratorAccess permission set.
Next to the permission set are links for the type of access available for that permission set. When you created the permission set, you specified both management console and programmatic access be enabled, so those two options are present. Select Management console to open the console.

AWS Management console

The user is signed in to the console. Using the search bar, look for Amazon DataZone service and open it.
Open the Amazon DataZone console and make sure you have enabled SSO users through IAM Identity Center. In case you haven’t, you can follow the steps in Enable IAM Identity Center for Amazon DataZone.

Note: In this post, we followed the default IAM Identity Center for Amazon DataZone configuration, which has implicit user assignment mode enabled. With this option, any user added to your Identity Center directory can access your Amazon DataZone domain automatically. If you opt for using explicit user assignment instead, remember that you need to manually add users to your Amazon DataZone domain in the Amazon DataZone console for them to have access.
To learn more about how to manage user access to an Amazon DataZone domain, see Manage users in the Amazon DataZone console.

Choose the Open data portal to access the Amazon DataZone Portal.

DataZone console

4.2) For all other users

Choose the Applications tab in the AWS access portal window and choose the Amazon DataZone data portal application link.

DataZone application

In the Amazon DataZone data portal, choose SIGN IN WITH SSO to continue

DataZone portal

Congratulations! Now you’re signed in to the Amazon DataZone data portal using your user that’s managed by Okta.

DataZone portal

5) Manage Amazon DataZone specific permissions in the Amazon DataZone portal

After you have access to the Amazon DataZone portal, you can work with projects, the data assets within, environments, and other constructs that are specific to Amazon DataZone. A project is the overarching construct that brings together people, data, and analytics tools. A project has two roles: owner and contributor. Next, you’ll learn how a user can be made an owner or contributor of existing projects.

These steps must be completed by the existing project owner in the Amazon DataZone portal:

Open the Amazon DataZone portal, select the project in the drop-down list on the left top of the portal and choose the project you own

DataZone project

In the project window, choose the Members tab to see the current users in the project and add a new one.

DataZone project members

Choose Add Members to add a new user. Make sure the User type is SSO User to add an Okta user. Look for the Okta user in the name drop-down list, select it, and select a project role for it. Finally, choose Add Members to add the user.

DataZone project members

The Okta user has been granted the selected project role and can interact with the project, assets, and tools.

DataZone project members

You can also grant permissions to SSO Groups. Choose Add members, then select SSO group in the drop-down list, next select the Group name, set the assigned project role, and choose Add Members.

DataZone project members

The Okta group has been granted the project role and can interact with the project, assets, and tools.

DataZone project members

You can also manage SSO user and group access to the Amazon DataZone data portal from the console. See Manage users in the Amazon DataZone console for additional details.

Clean up

To ensure a seamless experience and avoid any future charges, we kindly request that you follow these steps:

Delete the Amazon DataZone project if you created it as part of this blog post.
Delete the Amazon DataZone domain if you created it as part of this blog post.
If you changed your identity source from IAM Identity Center to an external identity provider (IdP), please revert it back to IAM Identity Center.
If you added an AWS IAM Identity Center app integration to your Okta account as part of this post, please delete the app integration.
Delete the permission set that you created specifically for this blog post.

By following these steps, you can effectively clean up the resources utilized in this blog post and prevent any unnecessary charges from accruing.

Summary

In this post, you followed a step-by-step guide to set up and use Okta to federate access to Amazon DataZone with AWS IAM Identity Center. You also learned how to group users and manage their permission in Amazon DataZone. As a final thought, now that you’re familiar with the elements involved in the integration of an external identity provider such as Okta to federate access to Amazon DataZone, you’re ready to try it with other identity providers.

To learn more about, see Managing Amazon DataZone domains and user access.

About the Authors

Carlos Gallegos is a Senior Analytics Specialist Solutions Architect at AWS. Based in Austin, TX, US. He’s an experienced and motivated professional with a proven track record of delivering results worldwide. He specializes in architecture, design, migrations, and modernization strategies for complex data and analytics solutions, both on-premises and on the AWS Cloud. Carlos helps customers accelerate their data journey by providing expertise in these areas. Connect with him on LinkedIn.

Jose Romero is a Senior Solutions Architect for Startups at AWS. Based in Austin, TX, US. He’s passionate about helping customers architect modern platforms at scale for data, AI, and ML. As a former senior architect in AWS Professional Services, he enjoys building and sharing solutions for common complex problems so that customers can accelerate their cloud journey and adopt best practices. Connect with him on LinkedIn.

Arun Pradeep Selvaraj is a Senior Solutions Architect at AWS. Arun is passionate about working with his customers and stakeholders on digital transformations and innovation in the cloud while continuing to learn, build, and reinvent. He is creative, fast-paced, deeply customer-obsessed and uses the working backwards process to build modern architectures to help customers solve their unique challenges. Connect with him on LinkedIn.

How to deploy an Amazon OpenSearch cluster to ingest logs from Amazon Security Lake

2024-07-30 Kevin Low

Post Syndicated from Kevin Low original https://aws.amazon.com/blogs/security/how-to-deploy-an-amazon-opensearch-cluster-to-ingest-logs-from-amazon-security-lake/

January 30, 2025: This post was republished to make the instructions clearer and compatible with OCSF 1.1.

Customers often require multiple log sources across their AWS environment to empower their teams to respond and investigate security events. In part one of this two-part blog post, I show you how you can use Amazon OpenSearch Service to ingest logs collected by Amazon Security Lake to facilitate near real-time monitoring.

Many customers use Security Lake to automatically centralize security data from Amazon Web Services (AWS) environments, software as a service (SaaS) providers, on-premises workloads, and cloud sources into a purpose-built data lake in their AWS environment. OpenSearch Service is a managed service that customers can use to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. It natively integrates with Security Lake to enable customers to perform interactive log analytics and searches across large datasets, create enterprise visualization and dashboards, and perform analysis across disparate applications and logs. With Amazon OpenSearch Security Analytics, customers can also gain visibility into the security posture of their organization’s infrastructure, monitor for anomalous activity, detect potential security threats in near real time, and initiate alerts to pre-configured destinations.

Without using Amazon OpenSearch Service, customers would need to build, deploy and manage infrastructure for an analytics solution, such as an ELK stack.

Prerequisites

Security Lake should already be deployed. For details on how to deploy Security Lake, see Getting started with Amazon Security Lake. You will need AWS Identity and Access Management (IAM) permissions to manage Security Lake, OpenSearch Service, Amazon Cognito, AWS Secrets Manager, and Amazon Elastic Compute Cloud (Amazon EC2), and to create IAM roles to follow along with this post. The solution can be deployed in any AWS Region that has at least 3 Availability Zones, supports Security Lake, OpenSearch, and OpenSearch Ingestion.

Solution overview

The architecture diagram in Figure 1 shows the completed architecture of the solution.

The OpenSearch Service cluster is deployed within a virtual private cloud (VPC) across three Availability Zones for high availability.
The OpenSearch Service cluster ingests logs from Security Lake using an OpenSearch Ingestion pipeline.
The cluster is accessed by end users through a public-facing proxy hosted on an Amazon EC2 instance.
1. To reduce costs, the template doesn’t deploy a dead letter queue (DLQ) for the OpenSearch Ingestion pipeline. You can add one later if you want.
2. Instead of a public facing proxy, you can deploy a VPN to access your cluster.
Authentication to the cluster is managed with Amazon Cognito.

Figure 1: Solution architecture

Planning the deployment

This section will help you plan your OpenSearch service deployment, including what nodes you should choose, the amount of storage to allocate, and where to deploy the cluster.

Deciding instances for the OpenSearch Service master and data nodes

First, determine what instance type to use for the master and data nodes. If your workload generates less than 100 GB of Security Lake logs per day, we recommend using three m6g.large.search master nodes and three r6g.large.search data nodes. You can start small and scale up or scale out later. For more information about deciding the size and number of instances, see Get started with Amazon OpenSearch Service. Note the instance types that you have selected on a text editor because you will use this as an input for the AWS CloudFormation template that you will deploy later.

Configuring storage

To optimize your storage costs, you need to plan your data strategy. In this architecture, Security Lake is used for long-term log storage. Because Security Lake uses Amazon Simple Storage Service (Amazon S3), you can optimize long-term storage costs. You can configure OpenSearch Service to ingest priority logs based on the recent data that you can use for near-real time detection and alerting. Your team can query logs in Security Lake using its Zero-ETL integration with OpenSearch Service to analyze older logs.

Therefore, Security Lake should serve as your primary long-term log storage, with OpenSearch Service storing only the most recent logs.

The number of days of logs in OpenSearch Service will depend on how many days’ worth of data you need to investigate at a given time. I recommend storing 15 days of data in OpenSearch Service. This allows you to react to and investigate the most immediate security events while optimizing storage costs for older logs.

The next step is to determine the volume of logs generated by Security Lake.

Sign in to the Security Lake delegated administrator account.
Go to the AWS Management Console for Security Lake. Choose Usage in the navigation pane.
On the Usage screen, select Last 30 days as the range of usage.
Add the total Actual usage for the last 30 days for the data sources that you intend to send to OpenSearch. If you have used Security Lake for less than 30 days, you can use the Total predicted usage per month. Divide this figure by 30 to get the daily data volume.

Figure 2: Select range of usage

To determine the total storage needed, multiply the data generated by Security Lake per day by the retention period you chose, then by 1.1 to account for the indexes, then multiply that number by 1.15 for overhead storage. For more information about calculating storage, see Get started with Amazon OpenSearch Service.

To determine the amount of Amazon Elastic Block Store (Amazon EBS) storage that you need per node, take the total amount of storage and divide it by the number of nodes that you have. Round that number up to the nearest whole number. You can increase the amount of storage after deployment when you have a better understanding of your workload. Make a note of this number in a text editor because you’ll use it as an input in the CloudFormation template later.

Example 1: 10 GB of Security Lake logs generated per day, stored for 30 days in OpenSearch Service in three nodes

10 GB of Security Lake logs stored for 30 days = 10 GB * 30 = 300 GB
Account for additional space for indexes and overhead space = 300 GB * 1.1 * 1.15 = 379.5 GB
Divide the storage required across three nodes, rounded up = 379.5/3 ≈ 127 GB per node
You would need 127 GB per node in OpenSearch Service

Example 2: 200 GB of Security Lake logs generated per day, stored for 15 days in OpenSearch Service across six nodes

200 GB of Security Lake logs stored for 15 days = 200 GB * 15 = 3000 GB
Account for additional space for indexes and overhead space = 3000 GB * 1.1 * 1.15 = 3795 GB
Divide the storage required across three nodes, rounded up = 3795/6 ≈ 633 GB per node
You would need 633 GB per node in OpenSearch Service

Where to deploy the cluster?

If you have an AWS Control Tower deployment or have a deployment modelled after the AWS Security Reference Architecture (AWS SRA), Security Lake should be deployed in the Log Archive account. Because security best practices recommend that the Log Archive account should not be frequently accessed, the OpenSearch Service cluster should be deployed into your Audit account or Security Tooling account.

You need to deploy your Security Lake subscriber in the same Region as your Security Lake roll-up Region. If you have more than one roll-up Region, choose the Region that collects logs from the Regions you want to monitor.

Your cluster needs to be deployed in the same Region as your Security Lake subscriber be able to access data.

Setting up the Security Lake subscriber

Before deploying the solution, create a Security Lake subscriber in your Security Lake roll-up Region so that OpenSearch Service can access data from Amazon Security Lake.

Access the Security Lake console in your Log Archive account.
Choose Subscribers in the navigation pane.
Choose Create subscriber.
On the Create subscriber page, enter a name, such as OpenSearch-subscriber.
Under Data Access, select Under S3 notification type, select SQS queue.
Under Subscriber credentials, enter the AWS account ID for the account you plan to deploy the OpenSearch cluster to, which should be your Security Tooling
Enter OpenSearchIngestion-<AWS account ID> under External ID.

Figure 3: Configuring the Security Lake subscriber
Leave All log and event sources selected and choose Create.

After the subscriber has been created, you will need to collect information to facilitate the deployment.

To gather necessary information:

Select the subscriber that you just created.
Derive the S3 bucket name from the S3 bucket ARN and store it in a text editor. The Amazon Resource Name (ARN) is formatted as arn:aws:s3:::<bucket name>. The bucket name should look like aws-security-data-lake-<region>-xxxxx.

Figure 4: Derive the S3 bucket name from the Subscriber details page
Go to the Amazon Simple Queue Service (Amazon SQS) console and select the SQS queue created as part of the Security Lake subscriber. It should look like AmazonSecurityLake-xxxxxxxxx-Main-Queue. Note the queue’s ARN and URL in your text editor.

Figure 5: Relevant details from the SQS queue

Deploy the solution

To deploy the solution in your Security Tooling account, use a CloudFormation template. This template deploys the OpenSearch Service cluster, OpenSearch Ingestion pipeline, and an AWS Lambda function to initialize the cluster.

To deploy the OpenSearch cluster:

To deploy the CloudFormation template that builds the OpenSearch service cluster, select the Launch Stack button.
In the CloudFormation console, make sure that you are in the correct AWS account. You should be in your Security Tooling account. Also make sure that you have selected the same Region as your Security Lake subscriber.
Enter a name for your stack. A name like os-stack-<day>-<month> can help you keep track of deployments.
Enter the instance types and Amazon EBS volume size that you noted earlier.
Enter the IP address range that you want to allow to access the proxy’s security group. You should limit this to your corporate IP range. You can set it as 0.0.0/0 if you want to expose it to the public internet.
Fill in the details of the Security Lake bucket and the subscriber Amazon SQS queue ARN, URL, and Region.

Figure 6: Add stack parameters
Check the acknowledgements in the Capabilities section.
Choose Create stack to begin deploying the resources.
It will take 20–30 minutes to deploy the multiple nested templates. Wait for the main stack (not the nested ones) to achieve the CREATE_COMPLETE status before proceeding to the next step.

Note: If you encounter failures while deployment, you can download the CloudFormation file here and select Preserve successfully provisioned resources under Stack failure options while deploying. This will allow you to troubleshoot the stack deployment.
Go to the Outputs pane of the main CloudFormation stack. Save the DashboardsProxyURL, OpenSearchInitRoleARN, and PipelineRole values in a text editor to refer to later.

Figure 7: The stacks in the CREATE_COMPLETE state with the outputs panel shown
Open the DashboardsProxyURL value in a new tab.

Note: Because the proxy relies on a self-signed certificate, you will get an insecure certificate warning. You can safely ignore this warning and proceed. For a production workload, you should issue a trusted private certificate from your internal public key infrastructure or use AWS Private Certificate Authority.
You will be presented with the Amazon Cognito sign-in page. Use administrator as the username.
Access Secrets Manager to find the password. Select the secret that was created as part of the stack.

Figure 8: The Cognito password in Secrets Manager
Choose Retrieve secret value to get the password.

Figure 9: Retrieve the secret value
After signing in, you will be prompted to change your password and will be redirected to the OpenSearch dashboard.
If you see a pop-up that states Start by adding your own data, select Explore on my own. On the next page, Introducing new OpenSearch Dashboards look & feel, choose Dismiss.
If you see a pop-up that states Select your tenant, select Global, and then choose Confirm.

Figure 10: Select and confirm your tenant

To initialize the OpenSearch cluster:

Choose the menu icon (three stacked horizontal lines) on the top left and select Security under the Management section.

Figure 11: Navigating to the Security page in the OpenSearch console
Select Roles. On the Roles page, search for the all_access role and select it.
Select Mapped users, and then select Manage mapping.
On the Map user screen, choose Add another backend role. Paste the value for the OpenSearchInitRoleARN from the list of CloudFormation outputs. Choose Map.

Figure 12: Mapping the role on the Security page in the OpenSearch console
Leave this tab open and return to the AWS Management console. Go to the AWS Lambda console and select the function named xxxxxx-OS_INIT.
In the function screen, choose Test, and then Create new test event.

Figure 13: Creating the test event in the Lambda console
Choose Invoke. The function should run for about 30 seconds. The execution results should show the component templates that have been created. This Lambda function creates the component and index templates to ingest Open Cybersecurity Framework (OCSF) formatted data, a set of indices and aliases that correspond with the OCSF classes generated by Security Lake, and a rollover policy that will rollover the index daily or if it becomes larger than 40 GB.

Figure 14: Invoking the Lambda function in the Lambda console

To set up the pipeline

Return to the Map user page on the OpenSearch console.
Choose Add another backend role. Paste the value of the PipelineRole from the CloudFormation template output. Choose This will allow the OpenSearch Ingestion to write to the cluster.

Figure 15: Mapping the OpenSearch Ingestion role
Access the Amazon S3 console in the Log Archive account where Security Lake is hosted.
Select the Security Lake bucket in your roll-up Region. It should look like aws-security-data-lake-region-xxxxxxxxxx.
Choose Permissions, then Edit under Bucket policy.

Add this policy to the end of the existing bucket policy. Replace the Principal with the ARN of the PipelineRole and the name of your Security Lake bucket in the Resource section.

{
            "Sid": "Cross Account Permissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "<Pipeline role ARN>"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<Security Lake bucket name>/*",
                "arn:aws:s3:::<Security Lake bucket name>"
            ]
        }

Figure 16: The modified S3 bucket access policy

Choose Save changes.

To upload the index patterns and dashboards

Download the Security-lake-objects.ndjson file by right-clicking on this link and selecting Save link as.
Access the Dashboards Management page through the navigation menu.
Choose Saved objects in the navigation pane.
On the Saved Objects page, choose Import on the right side of the screen.

Figure 17: Import saved objects
Choose Import and select the Security-lake-objects.ndjson file that you downloaded previously.
Leave Create new objects with unique IDs selected and choose Import.
You can now view the ingested logs on the Discover page and visualizations on the Dashboards page, which you can find on the navigation bar.

Figure 18: The Discover page displaying ingested logs

Clean up

To avoid unwanted charges, delete the main CloudFormation template, named os-stack-<day>-<month> (not the nested stacks).

Figure 19: Select the main stack in the CloudFormation console

Modify the Security Lake bucket policy in the logging account to remove the section you added that trusted the PipelineRole. Be careful not to modify the rest of the policy because it could impact the functioning of Security Lake and other subscribers.

Figure 20: The S3 bucket policy with the relevant sections that needed to be deleted

Conclusion

In this post, you learned how to plan an OpenSearch deployment with Amazon OpenSearch Service to ingest logs from Amazon Security Lake. With this solution, you’re able to aggregate and manage logs with Security Lake and visualize and monitor those logs with OpenSearch Service. After deployment, monitor the OpenSearch Service metrics to determine if you need to scale this up or out for improved performance. In part 2, I will show you how to set up the Security Analytics detector to generate alerts to security findings in near-real time.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Nadina Galle | The Nature of our Cities | Talks at Google

2024-07-30 Talks at Google

Post Syndicated from Talks at Google original https://www.youtube.com/watch?v=gF4vKaOqVj4

How to Future-Proof Your Archives: File Formats That Stand the Test of Time

2024-07-30 Stephanie Doyle

Post Syndicated from Stephanie Doyle original https://www.backblaze.com/blog/how-to-future-proof-your-archives-file-formats-that-stand-the-test-of-time/

A decorative image showing a vault door with a series of 0s and 1s behind it.

You may have heard us talk about backup a time or two, and hopefully our love has been clear. So, when Wired gave us a shoutout in their recent, astute article about archiving, allow us to say we were flattered.

As both a tribute and a collaboration, we’re happy to build on their article’s premise about choosing the correct file type for archiving purposes, and we’ll highlight a few tools to help you protect your files in the long term.

Wired reports: Archived files are especially vulnerable to changing file types

Archives are distinct from backups and have their own demands. Backups are intended to give you the ability to restore files or your whole environment—they need to be both in step with your current environment and flexible enough to respond to both a point in time restore of your whole system or a single file, depending on what you need. Archiving, on the other hand, is about preservation when you can’t depend on the continuity of devices or digital tools—those tools might look quite different (or not exist at all!) down the line. That said, backups are also an essential building block of digital archives.

Making something last long-term takes more work than you’d think. And, interestingly, digital archiving suffers from the opposite problem of “traditional” archiving. Whereas with books, magazines, and other paper-based media, you want to touch them as little as possible, with digital archives, you actually need to do some active maintenance to make sure you’re converting files to accessible formats that you can open well into the future.

Here’s an expert from the Wired report telling us about just one part of the practical concerns of digital archiving:

“Twenty years, in the digital realm, is ancient,” says Lance Stuchell, director of digital preservation services at the University of Michigan. His team is frequently tasked with recovering digital files from old computers and storage mediums. “We have a lab that can deal with old media—floppy drives, CDs, older computers. We can get that off of those types of media and move it into our preservation system while ensuring we don’t mess it up while we’re doing it.”

Wired goes on to report that the problem isn’t just having the correct device, but actually having the correct file type. Their biggest takeaways for making sure your files hold up over time?

Use open source file types.
If you’re storing media, store files uncompressed.
Back up absolutely everything.

Check out the rest of the article for details—it’s worth a read. And, thanks for the shoutout as a good option for folks looking to back up, Wired.

Some tools for converting files

So, now that you’re all geared up to get your archive in order, here are some free, open source tools that will help you convert your files. One note when you’re using open source (and we’re big fans) is to make sure you’re using a tool you trust. And, some tools, especially web-based tools, may collect user data or can expose sensitive information.

With that in mind, here are a few to get you started:

LibreOffice: The successor to OpenOffice, LibreOffice is a well-respected open source alternative to Microsoft Office and supports several open source and older document file formats.
Pandoc: Pandoc calls itself the Swiss army knife of file converters for markup formats, which includes documents, HTML formats, spreadsheets, and more. It’s got a very helpful list of file formats and indicates whether they can convert from/to each of them.
ImageMagick: ImageMagick can certainly convert your files, and it’s also beloved because it can edit files, including support for scripting and automation. But, for our purposes, it converts image formats and has continuous support from the open source community.
FFmpeg: FFmpeg is a community supported audio and video tool.
VLC media player: Another audio and video tool that supports conversion.

Keep in mind that while we’re recommending many of these for conversion purposes, many are actually fully-fledged programs with some very cool features—and, some can even replace traditional paid tool options, if you’re the budget-conscious type.

Archiving hardware of the future

Standard tech right now for storing archived files is on hard disk drives (HDDs) or solid state drives (SSDs), and you’ll even find DVDs that can keep your data stored for 1,000 years or more. All storage media types are known to degrade over time—when you’re storing long-term, you can’t just leave your drive disconnected from power forever, for instance.

That said, developing storage media types, while not as common, offer some interesting (if not yet widely practical) options. If you’re willing to drop some cash, DNA (yes, the biological kind) or ceramic might be for you. And, if you want to get super sci-fi with it, PhysicsWorld has reported on the “Superman memory crystal” that could keep data intact for millions of years.

Build your archive for alien circumstances

When we ambitiously sent out messages to the (potential) sentient life in the universe almost 50 years ago, on golden records no less, we apparently thought it was enough to also include a phonograph needle and some symbolic instructions on how to play the record. In practice, we sent a message with no guarantees that someone could decode and play it.

That may be fine for our space-age time capsule, but for our everyday archives, we do want to do our best to make sure we’re able to open them in the future. While we can’t anticipate where technology will be in 20, 40, or 100 years, we can follow digital archiving best practices to give future generations the best chance of opening files. At least they’ll likely share a language with us, as opposed to our alien friends.

The post How to Future-Proof Your Archives: File Formats That Stand the Test of Time appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Gratkit Smart Filament Dryer or is it?

2024-07-30 BeardedTinker

Post Syndicated from BeardedTinker original https://www.youtube.com/watch?v=UN2mtcmSOe4

[$] Report from the annual general meeting at GUADEC

2024-07-30 jzb

Post Syndicated from jzb original https://lwn.net/Articles/983203/

At GUADEC
in Denver, Colorado on July 21, the GNOME Foundation held its annual
general meeting (AGM) to provide updates from the foundation’s board and committees.
Topics included work accomplished in the past year, challenges
facing the GNOME Foundation–including fundraising and finding a new
executive director–and some insight into plans for the next year. And
last, but not least, the awarding of the Pants of Thanks.