Пътеводител за летния политически хаос в САЩ

Post Syndicated from Йоанна Елми original https://www.toest.bg/putevoditel-za-letniya-politicheski-haos-v-usa/

Пътеводител за летния политически хаос в САЩ

Лош месец за американската демокрация, но добър месец в историята на меметата – така обобщиха този юли потребителите в социалните мрежи в САЩ. За по-малко от 30 дни бяхме свидетели на редица събития: един от най-странните кандидатпрезидентски дебати от доста време насам (което само по себе си е постижение); две противоречиви решения на Върховния съд, които имат потенциала да променят страната из основи; един неуспешен атентат срещу кандидат-президента Доналд Тръмп; оттеглянето на кандидата на демократите и настоящ президент Джо Байдън от надпреварата… 

Преподаватели по история от различни престижни университети се пошегуваха в X (Туитър), че ще трябва да печатат нови конспекти: история на САЩ, 2024 – 2024 г. И въпреки че последният текст в рубриката ни преди ваканцията на „Тоест“ трябваше да бъде продължение на темата за най-важните проблеми за американските избиратели, изминалите четири седмици бяха така наситени със събития, че няма как да не им обърнем внимание. И така, представяме на развълнувания читател пътеводителя на летния политически хаос в САЩ. 

Катастрофалният дебат 

Миналия месец споменахме, че дебатите не заслужават кой знае какво внимание, защото не променят мнението на твърдия електорат и към настоящия момент са по-скоро ритуал, отколкото сериозна политическа дискусия. 

Е, скъпи читатели, дори и най-добрите грешат. 

Според някои изследователи съвременната американска политика преживява период на втвърдяване. Това означава, че и двете партии имат твърд електорат, който трудно би могъл да промени мнението си и който гласува по-скоро по идеологически съображения, отколкото след рационална преценка на конкретни политики.

Тази теория може да бъде възможно обяснение защо присъдата срещу Доналд Тръмп не повлия на мнението на неговия електорат например. Затова мнозинството специалисти смятат, че в подобна среда дебатите нямат особено значение, защото повечето гласоподаватели вече са взели решение кого ще подкрепят. Е, това може да е истина в повечето случаи. В края на юни историята показа, че един дебат може да обърне посоката на цяла партия и в крайна сметка да завърши с нов кандидат-президент. 

След изключително слабото представяне на настоящия президент Джоузеф Байдън, опасенията за възрастта му и способността му да осигури победа на демократите през ноември се превърнаха в тема от първостепенно значение. Въпреки че Джо Байдън има с какво да се похвали за четиригодишния си мандат, все по-видимите затруднения поради възрастта му бяха повод за дискусия още преди да обяви кандидатурата си, и се превърнаха в основна мишена за атаки от страна на републиканците.

През 2020 г. Байдън се кандидатира с аргумента, че е „президент – мост“ между ерата на Тръмп и бъдещето. Четири години по-късно обаче се оказва, че американците са принудени да избират между двама кандидати от миналото, макар и само единият да иска да върне държавата обратно там (макар и неясно точно в кой период).

И ако всички са свикнали с арогантността, с която Тръмп демонстрира некомпетентност, и така или иначе никой не очаква напълно кохерентно поведение от него (бившият президент е известен с нелогичните си, непоследователни изказвания, често прескача от тема на тема и рядко отговаря конкретно на зададените му въпроси), то слабото представяне на Джо Байдън, който се оплиташе в собствените си думи, беше кулминацията на серия подобни публични издънки, включително и падания, в допълнение на общия вид, издаващ физическа слабост. 

„Политико“ основателно нарече дебата „най-слабия в историята на САЩ“. И въпреки че „хвърлянето на кал“ е традиция за страната (терминът влиза в речниците след изборите през 1828 г., когато и двамата кандидати инвестират доста усилия да се плюят взаимно), дебатът на 27 юни 2024 г. е сред малкото събития, за които няма достоен еквивалент в историята. Вероятно един от редките случаи, в които гръмките медийни заглавия са точно намясто. 

Срамна глава в историята на Върховния съд 

Хората с чувство за хумор успяха да намерят механизми за справяне с абсурда на дебата. В един момент кандидатите започнаха да спорят кой е по-добър играч на голф, и ситуацията неудобно напомняше на спор на двама възрастни мъже пред входа в късен летен следобед, отколкото на сериозен дебат между кандидати за държавен глава на една от най-важните държави в света. Последните решения на Върховния съд на САЩ обаче са трудни за преглъщане дори с добър запас от оптимизъм и шеги. 

Решението „Грантс Пас срещу Джонсън“ (Grants Pass v. Johnson) беше произнесено в рамките на едно от най-важните дела за правата на бездомните в историята на САЩ. Върховният съд постанови, че Осмата поправка, забраняваща произволни арести, жестоки глоби и гаранции, не може да се използва като аргумент в случаи, в които бездомниците са третирани от местните власти като престъпници.

Това означава, че бездомниците могат да бъдат арестувани и глобявани заради факта, че са бездомни и например спят на улицата – макар да няма къде другаде да отидат. Стратегиите за справяне с бездомните остават в ръцете на съответните населени места, но вече е много по-лесно за граждани и организации да съдят градската управа с аргумента, че бездомните замърсяват или вдигат шум. 

Ако „Грантс Пас“ все пак оставя някаква надежда, че прилагането на закона зависи от местните власти и вероятно много малко бездомни ще бъдат де факто арестувани само заради това, че са бездомни, то решението „Тръмп срещу САЩ“ (Trump v. United States) със сигурност ще остане в историята като едно от най-мрачните на председателствания от Джон Робъртс съд и ще бъде цитирано наравно с други срамни решения през годините, много от които отдавна са отхвърлени (например узаконяващите робството на практика). Според решението президентите имат почти пълен наказателен имунитет за официалните си действия. Казано на прост език, президентът е над закона в повечето случаи, а доказването на вина в останалите става изключително трудно, защото трябва да се докаже намерение. 

Последното срамно решение от гледна точка на историята е „Лопър Брайт Ентърпрайзес срещу Реймондо“ (Loper Bright Enterprises v. Raimondo). То не дава повече власт на президента, а на съдилищата – и то власт, която надхвърля тази на експертите, вземащи решения по редица важни въпроси – от това какви вещества могат да съдържат храните и какви да бъдат регулациите за стоки, произведени в САЩ, до мерките за справянето с климатичните промени. 

„Лопър Брайт Ентърпрайзес срещу Реймондо“ бетонира властта на съдилищата над федералните регулаторни органи. Свидетели сме на радикално пренареждане на разделението на властите в САЩ и прехвърляне на властта към хора, които не са избирани, включително на правомощията, които избраният от гражданите Конгрес е дал на изпълнителната власт“, пише Иън Милхайзър, старши кореспондент и експерт в областта на Върховния съд.

С други думи, ако се стигне до съдебен процес, решението дали дадено вещество е лекарство, или хранителна добавка, в крайна сметка ще се вземе не от федералната агенция, в която работят експерти със съответните познания, а от съдията, който разглежда делото. Дали съдиите имат подобни познания, стана ясно мигновено и нагледно: Върховният съд трябваше да коригира свое решение, в което съдия Нийл Горсъч системно бърка азотните оксиди, допринасящи за парниковия ефект, с диазотен оксид, или райски газ, използван за упояване на пациенти. 

Покушението срещу Тръмп 

Идиокрацията* не свършва дотук. На 13 юли в рамките на предизборен митинг е извършен опит за покушение срещу бившия президент Доналд Тръмп, който държи реч. Докато говори по една от основните си теми – нелегалната имиграция като екзистенциална заплаха за американското общество, – някъде около него проехтяват изстрели, след което Тръмп се хваща за дясното ухо и заляга. По-късно в изтекъл разговор описва болката като „ухапване от най-големия комар, който сте виждали“. Тръмп се изправя, заобиколен от агенти на тайните служби, и започва да крещи „Бийте се, бийте се, бийте се!“ с юмрук във въздуха, а фотографът на „Асошиейтед Прес“ Евън Вучи прави една от знаковите снимки в съвременната история. 

Списъкът с опити за покушения срещу президентите на САЩ е достатъчно богат, за да си има собствена страница в Уикипедия. На ужасените европейци, които веднага определиха шокиращото събитие като края на демокрацията в САЩ, можем само да кажем, че за съжаление, това е една съвсем стандартна част от историята на страната, в чийто парламент Капитолия все още стои знак, свидетелстващ, че в зората на историята на страната разгорещените политически спорове са се решавали с юмручни боеве и понякога дори с изстрели. 

Мотивите на стрелеца Томас Матю Крукс остават неясни. Макар преди години да е дарил 15 долара на демократите, Крукс е регистриран републиканец. Впоследствие разследването разкри, че е търсил възможности за атентат не само срещу Тръмп, но и срещу Байдън. Атентатът е извършен с едно от „най-популярните“ оръжия, използвани в масови престрелки. Самите републиканци и оръжейното лоби работят срещу ограничения за свободното закупуване на оръжия, позовавайки се на втората поправка.

Тези нюанси сериозно затрудниха републиканците, които още преди всякаква яснота около опита за покушение обвиниха „радикалното ляво“. Разбира се, веднага след покушението се развиха редица конспиративни теории, включително и че атентатът е нагласен от Тръмп, за да спечели симпатизанти.

Има една българска поговорка, която обобщава ситуацията сравнително добре: „Който сее ветрове, жъне бури“. Насилието на думи, независимо от политическите пристрастия, рано или късно ражда насилие в действие. И колкото и някои републиканци да се опитват да припишат вината за атентата на Джо Байдън, надали има обективен наблюдател, който да отрече насилието, заложено в марката „Тръмп“. И да не се страхува от последствията. 

Оттеглянето на Джо Байдън и новата кандидатура за президент на демократите

Доналд Тръмп не е първопричина, а симптом на много по-дълбока криза в американското общество – тази на провинцията срещу метрополиса, на необразованите срещу образованите, на бедните срещу богатите, на средната класа срещу безконтролния капитализъм, на различните визии за света и как той трябва да бъде подреден. Всички тези противоречия не са разпределени по партии, а се преплитат в сложни конфигурации и пораждат събития и тенденции, които определят политиката в страната през последните 50 години. 

Независимо какво прави Тръмп – дали ще бъде осъден, или ще изтече запис, в който говори унизително за жените, – култът към личността му е прекалено силен, за да бъде критикуван от съпартийците и симпатизантите си, без значение колко ги отвращава в действителност (настоящият му кандидат-вицепрезидент Джей Ди Ванс го сравнява с Хитлер). Всяко действие и всеки коментар на бившия президент, независимо колко е скандален и в противовес на ценностите, уж изповядвани от партията му, бива извъртан не само от неговите поддръжници, но и от консервативната медийна машина, която на този етап представлява съвсем отделна екосистема и алтернативна реалност. Това се дължи не толкова на качествата на Доналд Тръмп, колкото на онова, което символизира – недоверие в институциите и медиите, както и силна антипатия към политическия елит в страната.

Джо Байдън обаче няма тази привилегия. След катастрофалното му представяне по време на дебата, критиката сред демократите прерасна от притеснена дискусия за възрастта и издръжливостта му в публични призиви той да се оттегли от надпреварата и да даде път на по-млад кандидат с по-голям шанс да спечели изборите. Цялата кампания на републиканците така или иначе бе насочена именно в тази посока: обрисуване на Байдън като възрастен и физически негоден за изпълнението на длъжността си кандидат.

На 21 юли, след продължителни дискусии и противоречиви изказвания в медиите и с оглед на данните за предизборните нагласи, които показваха преимущество на Тръмп след дебата, Джоузеф Байдън обяви, че се оттегля от надпреварата, и публично застана зад номинацията на вицепрезидентката си Камала Харис. Така Доналд Тръмп стана най-възрастният кандидат-президент в историята на САЩ. Дали стратезите му ще успеят да се измъкнат от капана, който сами си поставиха, надали има значение – твърдият електорат на Тръмп е свикнал да съществува в очевидни противоречия. Но този ход може да повлияе на избирателите, които все още не са решили кой кандидат ще подкрепят. 

„Има място и време за нови гласове, свежи гласове, по-млади гласове. И това време е сега“, заяви Байдън в реч от Овалния кабинет няколко дни по-късно. Стратегията със сигурност отразява очакванията на повечето хора – мнозинството от американските избиратели подкрепят ограничения във възрастта и мандатите на държавните служители. Процентът е дори по-висок сред привържениците на републиканците, отколкото сред демократите, и то в не едно и две проучвания

Действително решението на Байдън незабавно наля енергия в редиците на демократическата партия: Vote.org, независима платформа за регистриране на гласоподаватели, отчете 700% ръст на новорегистрираните гласоподаватели – 38 500 новорегистрирани, повечето под 35-годишна възраст.

Камала Харис разби досегашните рекорди за фондонабиране, регистрирайки 81 млн. долара от дарения в рамките на 24 часа. Future Forward – най-големият политически комитет на демократическата партия, обяви, че е подсигурил 150 млн. долара за същия период от донори, които досега са се колебали или не са изразявали подкрепа за нито един кандидат. Кандидатурата на Харис активира гласоподаватели, които не са били активни участници в предишни избори или чиято демография е слабо представена, а голям ръст в даренията се наблюдава в решаващи щати, като Тексас и Пенсилвания. 

Какво предстои 

В рамките на броени часове Камала Харис се превърна и в интернет феномен, а маневрите в демократическата партия изместиха Тръмп и новия му кандидат-вицепрезидент от прожекторите на медиите. Ако кампанията дотук е индикатор за ритъм и ако се позоваваме на естественото развитие на американската политика, това няма да продължи дълго и тепърва предстоят изненади. Въпреки че многократно призоваваха за оттеглянето на Джо Байдън, републиканците веднага заведоха дело, че не може средствата, събирани от неговата кампания, да бъдат дарени за кампанията на Харис. Според експерти ходът не почива на солидни правни аргументи, но Тръмп всячески ще се опита да блокира или затрудни кампанията на Харис, така че това е част от цялостната стратегия. 

Прогнозите на повечето проучвания на общественото мнение показват близки проценти на Тръмп и Харис със също толкова близки граници на грешката, в които кандидатите се припокриват. Доколко обаче може да се вярва на проучванията? През 2015 г., когато кампанията беше точно толкова интензивна и непредвидима, никой не успя да предскаже победата на Тръмп. Три месеца са много време в предизборна година. Сред факторите, които задължително трябва да вземем предвид, са действията на Израел в Палестина, развитието на пълномащабната руска инвазия в Украйна, изненадващи вътрешнополитически промени (с надежда да няма повече опити за покушение), както и как двамата кандидати ще водят кампанията си оттук нататък. 

Едно е ясно: изборите през ноември ще са сблъсък не само между миналото на Америка, която трябва да е велика отново по библейски мотив, и бъдещето, което включва преосмисляне на модела на функциониране на страната. Те са и борба между расизма и сексизма и равенството – вицепрезидентът на републиканците смята, че гласът на хората с деца трябва да има повече тежест, и се обяви срещу „бездетните жени с котки“, които „съсипват държавата“; Тръмп каза в реч, че гласоподавателите предпочитат белия мъж пред черния мъж. Харис, която не само че би била първата жена президент в историята на САЩ, но е и дъщеря на индийски емигранти, съвършено представлява другия полюс на двамата републикански кандидати. Харис представя изборите и като сблъсък между свободата и насаждането на омраза: „Ние искаме да забраним оръжията, те искат да забраняват книги“, се казва в първата ѝ изборна реклама. 

Готова ли е обаче страната да стъпи в бъдещето? И колко минало трябва да изтече, с колко хаос, за да се върне САЩ в консенсуса на настоящето – климатът се затопля вследствие на човешката дейност, жените и мъжете са равни, религията няма място в политиката, корпорациите не са хора, държавата не може да абдикира от задълженията си спрямо хората, жените не могат да износват деца насила, обикновените граждани нямат нужда от оръжия… И колко още може да се нажежи дискусията – опонентите да бъдат наричани фашисти или комунисти, а онези, които избират да живеят различно от нас, да бъдат очерняни, – без да се стигне до насилие? 

След горещото лято идва бурна есен. Тогава ще продължи и нашият разказ за Америка. 

* „Идиокрация“ (Idiocracy) е комедия от 2006 г., която представя живота в САЩ през 2505 г. Филмът често се използва като културна препратка и се е превърнал в нарицателно особено напоследък, тъй като някои от сцените в него изглеждат притеснително пророчески от настояща гледна точка. Изпълнението на кечиста Хълк Хоган по време на републиканската конвенция например беше сравнено неколкократно със сцени от комедията. 

Deliver Amazon CloudWatch logs to Amazon OpenSearch Serverless

Post Syndicated from Balaji Mohan original https://aws.amazon.com/blogs/big-data/deliver-amazon-cloudwatch-logs-to-amazon-opensearch-serverless/

Amazon CloudWatch Logs collect, aggregate, and analyze logs from different systems in one place. CloudWatch provides subcriptions as a real-time feed of these logs to other services like Amazon Kinesis Data Streams, AWS Lambda, and Amazon OpenSearch Service. These subscriptions are a popular mechanism to enable custom processing and advanced analysis of log data to gain additional valuable insights. At the time of publishing this blog post, these subscription filters support delivering logs to Amazon OpenSearch Service provisioned clusters only. Customers are increasingly adopting Amazon OpenSearch Serverless as a cost-effective option for infrequent, intermittent and unpredictable workloads.

In this blog post, we will show how to use Amazon OpenSearch Ingestion to deliver CloudWatch logs to OpenSearch Serverless in near real-time. We outline a mechanism to connect a Lambda subscription filter with OpenSearch Ingestion and deliver logs to OpenSearch Serverless without explicitly needing a separate subscription filter for it.

Solution overview

The following diagram illustrates the solution architecture.

  1. CloudWatch Logs: Collects and stores logs from various AWS resources and applications. It serves as the source of log data in this solution.
  2. Subscription filter : A CloudWatch Logs subscription filter filters and routes specific log data from CloudWatch Logs to the next component in the pipeline.
  3. CloudWatch exporter Lambda function: This is a Lambda function that receives the filtered log data from the subscription filter. Its purpose is to transform and prepare the log data for ingestion into the OpenSearch Ingestion pipeline.
  4. OpenSearch Ingestion: This is a component of OpenSearch Service. The Ingestion pipeline is responsible for processing and enriching the log data received from the CloudWatch exporter Lambda function before storing it in the OpenSearch Serverless collection.
  5. OpenSearch Service: This is fully managed service that stores and indexes log data, making it searchable and available for analysis and visualization. OpenSearch Service offers two configurations: provisioned domains and serverless. In this setup, we use serverless, which is an auto-scaling configuration for OpenSearch Service.

Prerequisites

Deploy the solution

With the prerequisites in place, you can create and deploy the pieces of the solution.

Step 1: Create PipelineRole for ingestion

  • Open the AWS Management Console for AWS Identity and Access Management (IAM).
  • Choose Policies, and then choose Create policy.
  • Select JSON and paste the following policy into the editor:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "aoss:BatchGetCollection",
                "aoss:APIAccessAll"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:aoss:us-east-1:{accountId}:collection/{collectionId}"
        },
        {
            "Action": [
                "aoss:CreateSecurityPolicy",
                "aoss:GetSecurityPolicy",
                "aoss:UpdateSecurityPolicy"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aoss:collection": "{collection}"
                }
            }
        }
    ]
}

// Replace {accountId}, {collectionId}, and {collection} with your own values
  • Choose Next, choose Next, and name your policy collection-pipeline-policy.
  • Choose Create policy.
  • Next, create a role and attach the policy to it. Choose Roles, and then choose Create role.
  • Select Custom trust policy and paste the following policy into the editor:
{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Principal":{
            "Service":"osis-pipelines.amazonaws.com"
         },
         "Action":"sts:AssumeRole"
      }
   ]
}
  • Choose Next, and then search for and select the collection-pipeline-policy you just created.
  • Choose Next and name the role PipelineRole.
  • Choose Create role.

Step 2: Configure the network and data policy for OpenSearch collection

  • In the OpenSearch Service console, navigate to the Serverless menu.
  • Create a VPC endpoint by following the instruction in Create an interface endpoint for OpenSearch Serverless.
  • Go to Security and choose Network policies.
  • Choose Create network policy.
  • Configure the following policy
[
  {
    "Rules": [
      {
        "Resource": [
          "collection/{collection name}"
        ],
        "ResourceType": "collection"
      }
    ],
    "AllowFromPublic": false,
    "SourceVPCEs": [
      "{VPC Enddpoint Id}"
    ]
  },
  {
    "Rules": [
      {
        "Resource": [
          "collection/{collection name}"
        ],
        "ResourceType": "dashboard"
      }
    ],
    "AllowFromPublic": true
  }
]
  • Go to Security and choose Data access policies.
  • Choose Create access policy.
  • Configure the following policy:
[
  {
    "Rules": [
      {
        "Resource": [
          "index/{collection name}/*"
        ],
        "Permission": [
          "aoss:CreateIndex",
          "aoss:UpdateIndex",
          "aoss:DescribeIndex",
          "aoss:ReadDocument",
          "aoss:WriteDocument"
        ],
        "ResourceType": "index"
      }
    ],
    "Principal": [
      "arn:aws:iam::{accountId}:role/PipelineRole",
      "arn:aws:iam::{accountId}:role/Admin"
    ],
    "Description": "Rule 1"
  }
]

Step 3: Create an OpenSearch Ingestion pipeline

  • Navigate to the OpenSearch Service.
  • Go to the Ingestion pipelines section.
  • Choose Create pipeline.
  • Define the pipeline configuration.
version: "2"
 cwlogs-ingestion-pipeline:

  source:

    http:

      path: /logs/ingest

  sink:

    - opensearch:

        # Provide an AWS OpenSearch Service domain endpoint

        hosts: ["https://{collectionId}.{region}.aoss.amazonaws.com"]

        index: "cwl-%{yyyy-MM-dd}"

        aws:

          # Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com

          sts_role_arn: "arn:aws:iam::{accountId}:role/PipelineRole"

          # Provide the region of the domain.

          region: "{region}"

          serverless: true

          serverless_options:

            network_policy_name: "{Network policy name}"
 # To get the values for the placeholders: 
 # 1. {collectionId}: You can find the collection ID by navigating to the Amazon OpenSearch Serverless Collection in the AWS Management Console, and then clicking on the Collection. The collection ID is listed under the "Overview" section. 
 # 2. {region}: This is the AWS region where your Amazon OpenSearch Service domain is located. You can find this information in the AWS Management Console when you navigate to the domain. 
 # 3. {accountId}: This is your AWS account ID. You can find your account ID by clicking on your username in the top-right corner of the AWS Management Console and selecting "My Account" from the dropdown menu. 
 # 4. {Network policy name}: This is the name of the network policy you have configured for your Amazon OpenSearch Serverless Collection. If you haven't configured a network policy, you can leave this placeholder as is or remove it from the configuration.
 # After obtaining the necessary values, replace the placeholders in the configuration with the actual values.            

Step 4: Create a Lambda function

  • Create a Lambda layer for requests and sigv4 packages. Run the following commands in AWS Cloudshell.
mkdir lambda_layers
 cd lambda_layers
 mkdir python
 cd python
 pip install requests -t ./
 pip install requests_auth_aws_sigv4 -t ./
 cd ..
 zip -r python_modules.zip .


 aws lambda publish-layer-version --layer-name Data-requests --description "My Python layer" --zip-file fileb://python_modules.zip --compatible-runtimes python3.x
import base64
 import gzip
 import json
 import logging
 import json
 import jmespath
 import requests
 from datetime import datetime
 from requests_auth_aws_sigv4 import AWSSigV4
 import boto3


 LOGGER = logging.getLogger(__name__)
 LOGGER.setLevel(logging.INFO)


 def lambda_handler(event, context):

    """Extract the data from the event"""

    data = jmespath.search("awslogs.data", event)

    """Decompress the logs"""

    cwLogs = decompress_json_data(data)

    """Construct the payload to send to OpenSearch Ingestion"""

    payload = prepare_payload(cwLogs)

    print(payload)

    """Ingest the set of events to the pipeline"""    

    response = ingestData(payload)

    return {

        'statusCode': 200

    }
 def decompress_json_data(data):

    compressed_data = base64.b64decode(data)

    uncompressed_data = gzip.decompress(compressed_data)

    return json.loads(uncompressed_data)


 def prepare_payload(cwLogs):

    payload = []

    logEvents = cwLogs['logEvents']

    for logEvent in logEvents:

        request = {}

        request['id'] = logEvent['id']

        dt = datetime.fromtimestamp(logEvent['timestamp'] / 1000) 

        request['timestamp'] = dt.isoformat()

        request['message'] = logEvent['message'];

        request['owner'] = cwLogs['owner'];

        request['log_group'] = cwLogs['logGroup'];

        request['log_stream'] = cwLogs['logStream'];

        payload.append(request)

    return payload

 def ingestData(payload):

    ingestionEndpoint = '{OpenSearch Pipeline Endpoint}'

    endpoint = 'https://' + ingestionEndpoint

    headers = {'Content-Type': 'application/json', 'Accept':'application/json'}

    r = requests.request('POST', f'{endpoint}/logs/ingest', json=payload, auth=AWSSigV4('osis'), headers=headers)

    LOGGER.info('Response received: ' + r.text)

    return r
  • Replace {OpenSearch Pipeline Endpoint}’ with the endpoint of your OpenSearch Ingestion pipeline.
  • Attach the following inline policy in execution role.
{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Sid": "PermitsWriteAccessToPipeline",

            "Effect": "Allow",

            "Action": "osis:Ingest",

            "Resource": "arn:aws:osis:{region}:{accountId}:pipeline/{OpenSearch Pipeline Name}"

        }

    ]
 }
  • Deploy the function.

Step 5: Set up a CloudWatch Logs subscription

  • Grant permission to a specific AWS service or AWS account to invoke the specified Lambda function. The following command grants permission to the CloudWatch Logs service to invoke the cloud-logs Lambda function for the specified log group. This is necessary because CloudWatch Logs cannot directly invoke a Lambda function without being granted permission. Run the following command in CloudShell to add permission.
aws lambda add-permission
 --function-name "{function name}"
 --statement-id "{function name}"
 --principal "logs.amazonaws.com"
 --action "lambda:InvokeFunction"
 --source-arn "arn:aws:logs:{region}:{accountId}:log-group:{log_group}:*"
 --source-account "{accountId}"
  • Create a subscription filter for a log group. The following command creates a subscription filter on the log group, which forwards all log events (because the filter pattern is an empty string) to the Lambda function. Run the following command in Cloudshell to create the subscription filter.
aws logs put-subscription-filter
 --log-group-name {log_group}
 --filter-name {filter name}
 --filter-pattern ""
 --destination-arn arn:aws:lambda:{region}:{accountId}:function:{function name}

Step 6: Testing and verification

  • Generate some logs in your CloudWatch log group. Run the following command in Cloudshell to create sample logs in log group.
aws logs put-log-events --log-group-name {log_group} --log-stream-name {stream_name} --log-events "[{\"timestamp\":{timestamp in millis} , \"message\": \"Simple Lambda Test\"}]"
  • Check the OpenSearch collection to ensure logs are indexed correctly.

Clean up

Remove the infrastructure for this solution when not in use to avoid incurring unnecessary costs.

Conclusion

You saw how to set up a pipeline to send CloudWatch logs to an OpenSearch Serverless collection within a VPC. This integration uses CloudWatch for log aggregation, Lambda for log processing, and OpenSearch Serverless for querying and visualization. You can use this solution to take advantage of the pay-as-you-go pricing model for OpenSearch Serverless to optimize operational costs for log analysis.

To further explore, you can:


About the Authors

Balaji Mohan is a senior modernization architect specializing in application and data modernization to the cloud. His business-first approach ensures seamless transitions, aligning technology with organizational goals. Using cloud-native architectures, he delivers scalable, agile, and cost-effective solutions, driving innovation and growth.

Souvik Bose is a Software Development Engineer working on Amazon OpenSearch Service.

Muthu Pitchaimani is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search applications and solutions. Muthu is interested in the topics of networking and security, and is based out of Austin, Texas.

Synchronize data lakes with CDC-based UPSERT using open table format, AWS Glue, and Amazon MSK

Post Syndicated from Shubham Purwar original https://aws.amazon.com/blogs/big-data/synchronize-data-lakes-with-cdc-based-upsert-using-open-table-format-aws-glue-and-amazon-msk/

In the current industry landscape, data lakes have become a cornerstone of modern data architecture, serving as repositories for vast amounts of structured and unstructured data. Change data capture (CDC) refers to the process of identifying and capturing changes made to data in a database and then delivering those changes in a downstream system. Capturing every change from transactions in a source database and moving them to the target keeps the systems synchronized, and helps with analytics use cases and zero-downtime database migrations.

However, efficiently managing and synchronizing data within these lakes presents a significant challenge. Maintaining data consistency and integrity across distributed data lakes is crucial for decision-making and analytics. Inaccurate or outdated data can lead to flawed insights and business decisions. Businesses require synchronized data to gain actionable insights and respond swiftly to changing market conditions. Scalability is a critical concern for data lakes, because they need to accommodate growing volumes of data without compromising performance or incurring exorbitant costs.

To address these issues effectively, we propose using Amazon Managed Streaming for Apache Kafka (Amazon MSK), a fully managed Apache Kafka service that offers a seamless way to ingest and process streaming data. We use MSK connect—an AWS managed service to deploy and run Kafka Connect to build an end-to-end CDC application that uses Debezium MySQL connector to process, insert, update, and delete records from MySQL and a confluent Amazon Simple Storage Service (Amazon S3) sink connector to write to Amazon S3 as raw data that can be consumed by other downstream application for further use cases. To process batch data effectively, we use AWS Glue, a serverless data integration service that uses the Spark framework to process the data from S3 and copies the data to the open table format layer. Open table format manages large collections of files as tables and supports modern analytical data lake operations such as record-level insert, update, delete, and time travel queries. We chose Delta Lake as an example open table format, but you can achieve the same results using Apache Iceberg or Apache Hudi.

The post illustrates the construction of a comprehensive CDC system, enabling the processing of CDC data sourced from Amazon Relational Database Service (Amazon RDS) for MySQL. Initially, we’re creating a raw data lake of all modified records in the database in near real time using Amazon MSK and writing to Amazon S3 as raw data. This raw data can then be used to build a data warehouse or even a special type of data storage that’s optimized for analytics, such as a Delta Lake on S3. Later, we use an AWS Glue exchange, transform, and load (ETL) job for batch processing of CDC data from the S3 raw data lake. A key advantage of this setup is that you have complete control over the entire process, from capturing the changes in your database to transforming the data for your specific needs. This flexibility allows you to adapt the system to different use cases.

This is achieved through integration with MSK Connect using the Debezium MySQL connector, followed by writing data to Amazon S3 facilitated by the Confluent S3 Sink Connector. Subsequently, the data is processed from S3 using an AWS Glue ETL job, and then stored in the data lake layer. Finally, the Delta Lake table is queried using Amazon Athena.

Note: If you require real-time data processing of the CDC data, you can bypass the batch approach and use an AWS Glue streaming job instead. This job would directly connect to the Kafka topic in MSK, grabbing the data as soon as changes occur. It can then process and transform the data as needed, creating a Delta Lake on Amazon S3 that reflects the latest updates according to your business needs. This approach ensures you have the most up-to-date data available for real-time analytics.

Solution overview

The following diagram illustrates the architecture that you implement through this blog post. Each number represents a major component of the solution.

The workflow consists of the following:

  1. Near real-time data capture from MySQL and streaming to Amazon S3
    1. The process starts with data originating from Amazon RDS for
    2. A Debezium connector is used to capture changes to the data in the RDS instance in near real time. Debezium is a distributed platform that converts information from your existing databases into event streams, enabling applications to detect and immediately respond to row-level changes in the databases. Debezium is built on top of Apache Kafka and provides a set of Kafka Connect compatible connectors.
    3. The captured data changes are then streamed to an Amazon MSK topic. MSK is a managed service that simplifies running Apache Kafka on AWS.
    4. The processed data stream (topic) is streamed from MSK to Amazon S3 in JSON format. The Confluent S3 Sink Connector allows near real-time data transfer from an MSK cluster to an S3 bucket.
  2. Batch processing the CDC raw data and writing it into the data lake
    1. Set up an AWS Glue ETL job to process the raw CDC
    2. This job reads bookmarked data from an S3 raw bucket and writes into the data lake in open file format (Delta). The job also creates the Delta Lake table in AWS Glue Data Catalog.
    3. Delta Lake is an open-source storage layer built on top of existing data lakes. It adds functionalities like ACID transactions and versioning to improve data reliability and manageability.
  3. Analyze the data using serverless interactive query service
    1. Athena, a serverless interactive query service, can be used to query the Delta Lake table created in Glue Data Catalog. This allows for interactive data analysis without managing infrastructure.

For this post, we create the solution resources in the us-east-1 AWS Region using AWS CloudFormation templates. In the following sections, we show you how to configure your resources and implement the solution.

Configure resources with AWS CloudFormation

In this post, you use the following two CloudFormation templates. The advantage of using two different templates is that you can decouple the resource creation of the CDC pipeline and AWS Glue processing according to your use case, and if you have requirements to create specific process resources only.

  1. vpc-msk-mskconnect-rds-client.yaml – This template sets up the CDC pipeline resources such as a virtual private cloud (VPC), subnet, security group, AWS Identity and Access Management (IAM) roles, NAT, internet gateway, Amazon Elastic Compute Cloud (Amazon EC2) client, Amazon MSK, MSKConnect, RDS, and S3
  2. gluejob-setup.yaml – This template sets up the data processing resources such as the AWS Glue table, database and ETL

Configure MSK and MSK connect

To start, you’ll configure MKS and MSK connect using Debezium connector to capture incremental changes in table and write into Amazon S3 using an S3 sink connector. The vpc-msk-mskconnect-rds-client.yaml stack creates a VPC, private and public subnets, security groups, S3 buckets, Amazon MSK cluster, EC2 instance with Kafka client, RDS database, and MSK connectors, and its worker configurations.

  1. Launch the stack vpc-msk-mskconnect-rds-client using the CloudFormation template:
    BDB-4100-CFN-Launch-Stack
  2. Provide the parameter values as listed in the following
. A B C
1 Parameters Description Sample value
2 EnvironmentName An environment name that is prefixed to resource names. msk-delta-cdc-pipeline
3 DatabasePassword Database admin account password. S3cretPwd99
4 InstanceType MSK client EC2 instance type. t2.micro
5 LatestAmiId Latest AMI ID of Amazon Linux 2023 for EC2 instance. You can use the default value. /aws/service/ami-amazon-linux- latest/al2023-ami-kernel-6.1-x86_64
6 VpcCIDR IP range (CIDR notation) for this VPC. 10.192.0.0/16
7 PublicSubnet1CIDR IP range (CIDR notation) for the public subnet in the first Availability Zone. 10.192.10.0/24
8 PublicSubnet2CIDR IP range (CIDR notation) for the public subnet in the second Availability Zone. 10.192.11.0/24
9 PrivateSubnet1CIDR IP range (CIDR notation) for the private subnet in the first Availability Zone. 10.192.20.0/24
10 PrivateSubnet2CIDR IP range (CIDR notation) for the private subnet in the second Availability Zone. 10.192.21.0/24
11 PrivateSubnet3CIDR IP range (CIDR notation) for the private subnet in the third Availability Zone. 10.192.22.0/24
  1. The stack creation process can take approximately one hour to complete. Check the Outputs tab for the stack after the stack is created.

Next, you set up the AWS Glue data processing resources such as the AWS Glue database, table, and ETL job.

Implement UPSERT on an S3 data lake with Delta Lake using AWS Glue

The gluejob-setup.yaml CloudFormation template creates a database, IAM role, and AWS Glue ETL job. Retrieve the values for S3BucketNameForOutput, and S3BucketNameForScript from the vpc-msk-mskconnect-rds-client stack’s Outputs tab to use in this template. Complete the following steps:

  1. Launch the stack gluejob-setup.
    Launch Cloudformation Stack
  2. Provide parameter values as listed in the following
. A B C
1 Parameters Description Sample value
2 EnvironmentName Environment name that is prefixed to resource names. gluejob-setup
3 GlueDataBaseName Name of the Data Catalog database. glue_cdc_blog_db
4 GlueTableName Name of the Data Catalog table. blog_cdc_tbl
5 S3BucketForGlueScript Bucket name for the AWS Glue ETL script. Use the S3 bucket name from the previous stack. For example, aws- gluescript-${AWS::AccountId}-${AWS::Region}-${EnvironmentNam e
6 GlueWorkerType Worker type for AWS Glue job. For example, G.1X G.1X
7 NumberOfWorkers Number of workers in the AWS Glue job. 3
8 S3BucketForOutput Bucket name for writing data from the AWS Glue job. aws-glueoutput-${AWS::AccountId}-${AWS::Region}-${EnvironmentName}
9 S3ConnectorTargetBucketname Bucket name where the Amazon MSK S3 sink connector writes the data from the Kafka topic. msk-lab-${AWS::AccountId}- target-bucket
  1. The stack creation process can take approximately 2 minutes to complete. Check the Outputs tab for the stack after the stack is created.

In the gluejob-setup stack, we created an AWS Glue database and AWS Glue job. For further clarity, you can examine the AWS Glue database and job generated using the CloudFormation template.

After successfully creating the CloudFormation stack, you can proceed with processing data using the AWS Glue ETL job.

Run the AWS Glue ETL job

To process the data created in the S3 bucket from Amazon MSK using the AWS Glue ETL job that you set up in the previous section, complete the following steps:

  1. On the CloudFormation console, choose the stack gluejob-setup.
  2. On the Outputs tab, retrieve the name of the AWS Glue ETL job from the GlueJobName In the following screenshot, the name is GlueCDCJob-glue-delta-cdc.

  1. On the AWS Glue console, choose ETL jobs in the navigation pane.
  2. Search for the AWS Glue ETL job named GlueCDCJob-glue-delta-cdc.
  3. Choose the job name to open its details page.
  4. Choose Run to start the On the Runs tab, confirm if the job ran without failure.

  1. Retrieve the OutputBucketName from the gluejob-setup template output.
  2. On the Amazon S3 console, navigate to the S3 bucket to verify the data.

Note: We have enabled AWS Glue job bookmark, which will make sure job will process the new data in each job run.

Query the Delta Lake table using Athena

After the AWS Glue ETL job has successfully created the Delta Lake table for the processed data in the Data Catalog, follow these steps to validate the data using Athena:

  1. On the Athena console, navigate to the query editor.
  2. Choose the Data Catalog as the data source.
  3. Choose the database glue_cdc_blog_db created using gluejob-setup stack.
  4. To validate the data, run the following query to preview the data and find the total count.
SELECT * FROM "glue_cdc_blog_db"."blog_cdc_tbl" ORDER BY cust_id DESC LIMIT 40;
SELECT COUNT(*) FROM "glue_cdc_blog_db"."blog_cdc_tbl";

The following screenshot shows the output of our example query.

Upload incremental (CDC) data for further processing

After we process the initial full load, let’s perform insert, update, and delete records in MySQL, which will be processed by the Debezium mysql connector and written to Amazon S3 using a confluent S3 sink connector.

  1. On the Amazon EC2 console, go to the EC2 instance named KafkaClientInstance that you created using the CloudFormation template.

  1. Sign in to the EC2 instance using SSM. Select KafkaClientInstance and then choose Connect.

  1. Run the following commands to insert the data into the RDS table. Use the database password from the CloudFormation stack parameter tab.
sudo su - ec2-user
RDS_AURORA_ENDPOINT=`aws rds describe-db-instances --region us-east-1 | jq -r '.DBInstances[] | select(.DBName == "salesdb") | .Endpoint.Address'`
mysql -f -u master -h $RDS_AURORA_ENDPOINT  --password
  1. Now perform the insert into the CUSTOMER table.
use salesdb;
INSERT into CUSTOMER values(8887,'Customer Name 8887','Market segment 8887');
INSERT into CUSTOMER values(8888,'Customer Name 8888','Market segment 8888');
INSERT into CUSTOMER values(8889,'Customer Name 8889','Market segment 8889');

  1. Run the AWS Glue job again to update the Delta Lake table with new records.
  2. Use the Athena console to validate the data.
  3. Perform the insert, update, and delete in the CUSTOMER table.
    UPDATE CUSTOMER SET NAME='Customer Name update 8888',MKTSEGMENT='Market segment update 8888' where CUST_ID = 8888;
    UPDATE CUSTOMER SET NAME='Customer Name update 8889',MKTSEGMENT='Market segment update 8889' where CUST_ID = 8889;
    DELETE FROM CUSTOMER where CUST_ID = 8887;
    INSERT into CUSTOMER values(9000,'Customer Name 9000','Market segment 9000');
    

  4. Run the AWS Glue job again to update the Delta Lake table with the insert, update, and delete records.
  5. Use the Athena console to validate the data to verify the update and delete records in the Delta Lake table.

Clean up

To clean up your resources, complete the following steps:

  1. Delete the CloudFormation stack gluejob-setup.
  2. Delete the CloudFormation stack vpc-msk-mskconnect-rds-client.

Conclusion

Organizations continually seek high-performance, cost-effective, and scalable analytical solutions to extract value from their operational data sources in near real time. The analytical platform must be capable of receiving updates to operational data as they happen. Traditional data lake solutions often struggle with managing changes in source data, but the Delta Lake framework addresses this challenge. This post illustrates the process of constructing an end-to-end change data capture (CDC) application using Amazon MSK, MSK Connect, AWS Glue, and native Delta Lake tables, alongside guidance on querying Delta Lake tables from Amazon Athena. This architectural pattern can be adapted to other data sources employing various Kafka connectors, enabling the creation of data lakes supporting UPSERT operations using AWS Glue and native Delta Lake tables. For further insights, see the MSK Connect examples.


About the authors

Shubham Purwar is a Cloud Engineer (ETL) at AWS Bengaluru specializing in AWS Glue and Athena. He is passionate about helping customers solve issues related to their ETL workload and implement scalable data processing and analytics pipelines on AWS. In his free time, Shubham loves to spend time with his family and travel around the world.

Nitin Kumar is a Cloud Engineer (ETL) at AWS, specializing in AWS Glue. With a decade of experience, he excels in aiding customers with their big data workloads, focusing on data processing and analytics. He is committed to helping customers overcome ETL challenges and develop scalable data processing and analytics pipelines on AWS. In his free time, he likes to watch movies and spend time with his family.

[$] Showing up for Python in GNOME

Post Syndicated from jzb original https://lwn.net/Articles/983897/

A few years ago, PyGObject—the Python
package that provides bindings for GTK and GNOME applications—was not
faring particularly well. Several maintainers had left the project and its
development was not keeping pace with changes in GTK. At this year’s
GUADEC, Dan Yeaw presented a talk
about the project’s decline, improvements in the last year, and his
experience getting involved in an undermaintained project.

Plan your advertising campaigns with Amazon Marketing Cloud on AWS Clean Rooms, now generally available

Post Syndicated from Veliswa Boya original https://aws.amazon.com/blogs/aws/plan-your-advertising-campaigns-with-amazon-marketing-cloud-on-aws-clean-rooms-now-generally-available/

Today, we are announcing the general availability of Amazon Marketing Cloud (AMC) on AWS Clean Rooms to help advertisers use their first-party signals to collaborate with Amazon Ads unique signals. With this collaboration, advertisers can generate differentiated insights, discover new audiences, and enable advertising campaign planning, activation, and measurement use cases, all without having to move their underlying signals outside of their AWS account. With AMC on AWS Clean Rooms, customers can easily prepare their data, match and create audiences, use custom insights to activate more relevant advertising campaigns with Amazon Ads, and measure return on ad spend. All of this can be accomplished from the most secure cloud computing environment available today.

Advertisers continually strive to reach new audiences and deliver relevant, marketing campaigns to better engage their customers. Yet, the advertising and marketing landscape is undergoing a fundamental shift with signal loss and fragmentation. As such, advertisers and their partners need to collaborate together using signals that are stored across many applications to personalize their advertising campaigns. However, to work with one another to gather insights, companies typically need to share a copy of their signals with their partners, which is often not aligned with their data governance, security and privacy, IT, and legal teams’ policies. As a result, many businesses miss opportunities to fully maximize the value of their first-party signals and improve planning, activation, and measurement outcomes for their campaigns.

AMC on AWS Clean Rooms makes it easier and scalable for advertisers to use their first-party signals with Amazon Ads, including collaborating across event-level signals and modeling unique audiences to help improve media planning, activation, and outcomes without having to move underlying signals outside their cloud environment.

AMC on AWS Clean Rooms prerequisites (environment setup)
To get started with AMC on AWS Clean Rooms, the advertiser needs an AWS account and a dataset that contains user population and event-level data stored in open data formats (CSV, Parquet, or Iceberg) in an Amazon Simple Storage Service (Amazon S3) bucket. The next step is to send an email to the Amazon Ads team to request the creation of an AMC instance. Once an instance has been created, the Amazon Ads team will create an AWS Clean Rooms collaboration and invite the advertiser to join the collaboration.

How it works
1. Join an AWS Clean Rooms collaboration and create an ID namespace.
2. Configure and associate tables to an AMC collaboration.
3. Run an ID mapping workflow to create and populate the ID mapping table.
4. Run a query in AMC.

Walkthrough

1. Join an AWS Clean Rooms collaboration and create an ID namespace.
The advertiser will accept the collaboration invite by creating a membership in their AWS account. Once in the collaboration, the advertiser will access the AWS Clean Rooms console and then select the AWS Entity Resolution ID namespace generated when the collaboration was created to start the process of using their data for matching and collaboration in AWS Clean Rooms. Next, specify the AWS Glue table and the associated schema mapping and choose the S3 bucket in the same AWS Region as the collaboration for temporarily storing your data while it processes. Lastly, the advertiser will provide permissions to read your data input from AWS Glue and write to Amazon S3 on their behalf.

In the AirportLink collaboration shown in the following screenshot, the advertiser (member AirportLink2) accepts a collaboration invite sent by member AirportLink1.


2. Configure and associate tables to an AMC collaboration.
After joining the collaboration, the advertiser will create configured tables on their purchase data, add custom analysis rule, and associate the configured table to the collaboration.



Within the collaboration, the advertiser will set up a collaboration analysis rule to control which party can receive the result of a query run on the associated table.


3. Run an ID mapping workflow to create and populate the ID mapping table.
Now that the ID namespace is associated with the collaboration, the Amazon Ads team will create an ID mapping table in the AWS Clean Rooms console. This step requires both the advertiser (source) and the Amazon Ads team (target) to associate their ID namespace resources to the collaboration. Amazon Ads will provide the methods of mapping and configuration, add the details for querying to name the ID mapping table, and provide permission for AWS Clean Rooms to execute and track the ID mapping workflow job on their behalf. Finally, the Amazon Ads team will select Create and Populate to start the mapping workflow and generate an ID mapping table that captures a common user cohort, who were matched on the rules provided in Step 2.

4. Run a query in AMC.
Advertisers can either use templates or write a SQL query to run for analysis and get query results for further insights. They can run the SQL query in the following ways:

  • Run a SQL query with AMC data and the advertiser’s data that return the results to the advertiser’s S3 bucket using aggregate analysis. An example query is “How many of the customers who are registered for my email list saw the ads I’m running on Amazon?”
  • Run a SQL query to create an audience on the advertiser’s data or overlap with AMC signals that returns results to the S3 bucket of Amazon Ads. An example query is to generate an audience to target in an ad campaign.
  • Run an AWS Clean Rooms ML lookalike modeling job where Amazon Ads contributes the configured model and the advertiser contributes a seed audience. The resulting segment (list of user ad IDs) is sent to Amazon Ads.


After running the query, the advertiser can create an audience using a rule-based audience or a similar audience by navigating to the Audience tab in AMC. The output of the audience query will be sent directly to Amazon Demand Side Platform (DSP). The following table shows the options available to you when creating the audience:

If you want to
Then
Use pre-built audience templates Select Create with instructional query from the dropdown list
Create custom audience queries Select Create new query from the dropdown list

When creating a new query, the advertiser will configure various options such as name, description, and date adjustments. Additionally, the advertiser can choose from the two following audience types:

Rule-based audience – Create audience-based on the audience query.
Similar audience – Create machine learning (ML) based audiences based on the seed audience outputs from the audience query.

Now available
AMC on AWS Clean Rooms is available in in the US East (N. Virginia) Region. Be sure to check the full Region list for future updates. Learn more about AMC on AWS Clean Rooms in the AWS documentation.

Give it a try by emailing the Amazon Ads team to get started and send feedback to the AWS re:Post for AWS Clean Rooms or through your usual AWS Support contacts.

Veliswa

Let’s Architect! Designing Well-Architected systems

Post Syndicated from Vittorio Denti original https://aws.amazon.com/blogs/architecture/lets-architect-well-architected-systems/

The design of cloud workloads can be a complex task, where a perfect and universal solution doesn’t exist. We should balance all the different trade-offs and find an optimal solution based on our context. But how does it work in practice? Which guiding principles should we follow? Which are the most important areas we should focus on?

In this blog, we will try to answer some of these questions by sharing a set of resources related to the AWS Well-Architected Framework. The Framework shares a set of methods to help you understand the pros and cons of decisions you make while building cloud systems. By following this resource, you will learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems in the cloud. The framework is constantly updated; it evolves as the technology landscape changes. Check out the latest updates from June 2024.

Build secure applications on AWS the Well-Architected way

The AWS Well-Architected Framework is constantly updated across all six pillars. The security pillar added a new best practice area: application security (AppSec). In this session, you can learn about the best practices highlighted in this area. Review four key domains: organization and culture, security of the pipeline, security in the pipeline, and dependency management. Each area provides a set of principles that you can implement and provides a complete view of how you design, develop, build, deploy, and operate secure workloads in the cloud.

Security should be part of the end-to-end development process, and implementing best practices both in the application code as well as in the underlying infrastructure components.

Figure 1. Security should be part of the end-to-end development process, and implementing best practices both in the application code as well as in the underlying infrastructure components.

Take me to this video

Announcing the AWS Well-Architected Mergers and Acquisitions Lens

How can we integrate different systems as a consequence of an acquisition? Mergers and acquisitions operations bring different people with different backgrounds together, with a need of driving systems convergence. Both organization and technical challenges can arise in this scenario. The Mergers and Acquisitions (M&A) Lens is a collection of customer-proven design principles, best practices, and prescriptive guidance to help you integrate the IT systems of two or more organizations. This lens helps companies follow AWS prescribed best practices during technical integration, drive cost optimization, and expedite merger and acquisition value realization.

If the seller company runs on another cloud platform or on-premises, the acquirer should plan a cloud migration while guaranteeing continuity of service.

Figure 2. If the seller company runs on another cloud platform or on-premises, the acquirer should plan a cloud migration while guaranteeing continuity of service.

Take me to this blog

AWS Well-Architected Labs

One of the best ways to become familiar with new concepts and methodologies consist of doing hands-on work to absorb the techniques properly. For each Let’s Architect! blog, we tend to share at least one workshop associated with the topic. The AWS Well-Architected Framework covers six different pillars, so today we share the AWS Well-Architected Labs to cover each area of the framework. Feel free to jump across the different workshops and start building!

Sustainability is one of the pillars in the framework. Asynchronous and scheduled processing are key techniques for improving the sustainability and costs of cloud architectures.

Figure 3. Sustainability is one of the pillars in the framework. Asynchronous and scheduled processing are key techniques for improving the sustainability and costs of cloud architectures.

Take me to this workshop

Gain confidence in system correctness and resilience with formal methods

Distributed systems are difficult to design. It’s even more difficult to test them and prove they are working. Formal methods enable the early discovery of design bugs that can escape the guardrails of design reviews and automated testing only to get uncovered in production. This video shows how AWS uses P, an open source, state machine–based programming language for formal modelling and analysis of distributed systems.

You can learn from AWS engineers and architects how to use P for your own applications to find bugs early in the development process and increase developer velocity. This tool is used in AWS to reason out the correctness of cloud services (for example, Amazon Simple Storage Service and Amazon DynamoDB).

An example of a distributed system for processing transactions.

Figure 4. An example of a distributed system for processing transactions.

Take me to this video

See you next time!

Thanks for reading! Hopefully, you got interesting insights into the methodologies for designing Well-Architected systems. In the next blog, we will talk about multi-region architectures. We will understand when they are actually needed, and which design principles should be applied.

To revisit any of our previous posts or explore the entire series, visit the Let’s Architect! page.

Forgejo v8.0 released

Post Syndicated from jzb original https://lwn.net/Articles/984116/

Version 8.0 of the Forgejo
software-development platform has been released. Notable
changes include the removal
of non-free software found in the codebase, improved stability, and a
reduction
in “seemingly random User Interface changes“:

A gentle way of describing Forgejo User eXperience is that it is an
acquired taste: it grew over the years, driven by the inspiration of
the person with the keyboard in their hand. Once implemented it almost
never changed. A user who started with Forgejo in 2022 can only see
minor changes in 2024 and not all of them make intuitive sense. The
solution to this problem is simple and was identified early on: User
Research. But only in the making of Forgejo v8.0 did it get some
momentum.

See the release
notes
for a full list of changes.

Ampere AmpereOne Pricing and SKU List with Current OEM Partners

Post Syndicated from John Lee original https://www.servethehome.com/ampere-ampereone-pricing-and-sku-list-with-current-oem-partners-supermicro-asrock-rack-gigabyte-ieit-byd-arm/

Ampere just published the AmpereOne SKU list and pricing information as well as listing several OEM partners with systems

The post Ampere AmpereOne Pricing and SKU List with Current OEM Partners appeared first on ServeTheHome.

[$] Pulling Linux up by its bootstraps

Post Syndicated from daroc original https://lwn.net/Articles/983340/

A
bootstrappable build
is one that builds existing
software from scratch — for example, building GCC without relying on an existing
copy of GCC. In 2023, the Guix project
announced that the project had reduced the size
of the binary bootstrap seed needed to build its operating system to just 357-bytes —
not counting the Linux kernel required to run the build process. Now, the
live-bootstrap project
has gone a step further and removed the need for an existing kernel at all.

Security updates for Wednesday

Post Syndicated from jzb original https://lwn.net/Articles/984080/

Security updates have been issued by Fedora (xdg-desktop-portal-hyprland), Red Hat (freeradius, freeradius:3.0, git-lfs, httpd, kernel, openssh, and varnish:6), SUSE (cdi-apiserver-container, cdi-cloner-container, cdi- controller-container, cdi-importer-container, cdi-operator-container, cdi- uploadproxy-container, cdi-uploadserver-container, cont, git, gtk2, gtk3, kubevirt, virt-api-container, virt-controller-container, virt-exportproxy-container, virt-exportserver-container, virt-handler-container, virt-launcher-container, virt-libguestfs-t, orc, postgresql14, python-dnspython, python-urllib3, shadow, and xen), and Ubuntu (openjdk-17, openjdk-21, openjdk-8, openjdk-lts, and python3.10, python3.8).

Celebrating Excellence: Rapid7 Recognized in Newsweek’s Greatest Workplaces in America 2024

Post Syndicated from Rapid7 original https://blog.rapid7.com/2024/07/31/celebrating-excellence-rapid7-recognized-in-newsweeks-greatest-workplaces-in-america-2024/

Celebrating Excellence: Rapid7 Recognized in Newsweek's Greatest Workplaces in America 2024

In a testament to its commitment to fostering an exceptional workplace environment, Rapid7 is proud to be included in Newsweek’s Greatest Workplaces in America for 2024. This recognition not only underscores Rapid7’s dedication to its people, but also cements its standing among companies that invest in employee satisfaction and well-being as a critical component of business success.

The Importance of Employee Engagement in the Workplace

Employee engagement in the workplace is linked to higher levels of productivity and positive business outcomes. According to Gallup’s State of the Global Workplace: 2023 Report, employees who are not engaged, or those who are actively disengaged, could cost the world $8.8 trillion in lost productivity.

Rapid7 understands this connection, and is intentional in providing a workplace where employees can do their best work while feeling valued and supported. Christina Luconi, Chief People Officer at the firm, states “As a business, we are relentlessly in pursuit of delivering the best possible outcomes for our customers and the cybersecurity community we are a part of. In order to do that, we need people who are prepared to challenge convention, offer constructive feedback, and work collaboratively to drive impact. We make sure this is possible by providing a work environment that offers flexibility, great teaming experiences, opportunities to grow and learn new skills, and a shared commitment to Rapid7’s mission and vision.”

America’s Greatest Workplaces: Award Criteria

More than 250,000 U.S. employees were interviewed for the ranking, resulting in over 1.5 million company reviews spanning 78 individual sectors. The survey covered topics such as compensation and benefits, training and career progression, work-life balance and company culture. Also, post-survey research considered each ranked company’s online mentions, diversity and inclusion ratings, and reviews of senior management.

How Rapid7 Supports Its Employees

Flexibility: Our default model is hybrid, with employees spending three days per week in the office. This flexibility aligns with our culture of collaboration and teamwork, and is designed to foster meaningful connections and trusting relationships.

Career Development: At Rapid7, we provide a platform for career development and growth. Through a combination of programs and hands-on experiences, employees have the ability to drive their career forward and own their development journey.

Collaboration: We encourage our people to seek out opportunities to learn about other teams and areas of the business. Collaboration is something that happens every day, whether we are gathering feedback and perspectives to get the best solution, or setting up an insight coffee to learn more about a person’s role. We encourage employees at all levels of the business to proactively find ways to partner, collaborate, and learn from one another.

Wellbeing and Benefits: Benefits vary by country, and in the United States, employees enjoy unlimited PTO, competitive paid leave options for new parents, mental health resources, access to financial advisers, and more. We also undergo an annual compensation analysis to ensure our pay practices are both competitive and equitable.

Looking Ahead

Being named one of Newsweek’s Greatest Workplaces in America for 2024 is a significant achievement for Rapid7. Our commitment to fostering a positive and inclusive work environment remains unwavering, and we look forward to continuing to evolve our programs as  we expand our teams and tackle new challenges in cybersecurity.

The collective thoughts of the interwebz