Post Syndicated from Николай Марченко original https://bivol.bg/%D0%B1%D1%81%D0%BF-%D1%83%D0%B4%D0%B0%D1%80%D0%BD%D0%BE-%D0%BD%D0%B0%D0%B7%D0%BD%D0%B0%D1%87%D0%B0%D0%B2%D0%B0-%D0%B2-%D0%B3%D0%BE%D1%80%D0%B8%D1%82%D0%B5-%D1%81%D1%8A%D0%BF%D0%B0%D1%80%D1%82%D0%B8.html
Българска социалистическа партия (БСП), от чиято квота е и министърът на земеделието, храните и горите Иван Иванов, ударно си назначава областните, общинските и др. кадри на местно ниво в държавните…
Post Syndicated from Йоанна Елми original https://toest.bg/kak-prepodavame-i-govorim-za-komunizma/
Луиза Славкова е основател и изпълнителен директор на Фондация „Софийска платформа“, която за поредна година организира лятно училище в гр. Белене по темите за паметта и демокрацията. Славкова е член на Консултативния съвет на Европейската мрежа за гражданско образование (NECE), през годините е била програмен мениджър в Европейския съвет за външна политика, гост-изследовател в Института за изследване на правата на човека към Колумбийския университет и съветник на министъра на външните работи Николай Младенов. Автор и редактор е на няколко книги и публикации, свързани с международни политики, демократично развитие и гражданско образование, и е съавтор на учебник по гражданско образование по учебната програма на новия предмет.
Йоанна Елми разговаря с Луиза Славкова за лятното училище, за паметта за комунистическата диктатура и за важността на гражданското образование при малки и големи.
Лятното училище се провежда под надслов „Защо ни е да помним?“. Безспорно отговорът е важен и питам и за него, но първо ми е интересно нещо друго – не е ли важно и как помним? Защото българската образователна система тежко залага на помненето на определени исторически моменти, дори на механично наизустяване, но има ли смисъл от такова помнене?
Разговорът за паметта е многопластов. И в теорията, и в практиката има различни подходи, включително радикални, според които е по-добре да не се помни нищо. Защото паметта само репродуцирала определени нагласи и никога не можело да има трайно разбирателство, особено където е имало конфликт между две групи. Ако една група от обществото е представена като палач, а друга – като жертва, помненето затвърждава тази представа. Известен представител на този подход е синът на Сюзън Зонтаг – Дейвид Рийф, който години наред отразява войните в бивша Югославия и е убеден, че е по-добре на Балканите нищо да не се помни. Аз не съм съгласна с него, но това не означава, че не обсъждаме това гледище в нашето лятно училище. Ние питаме всеки участник дали трябва да помним комунизма с неговите репресии като тоталитарен режим, или не. И оттам: ако го помним – защо да го помним?
В лятното училище не бягаме от различните пластове на разказа за комунизма, но правим разлика между наративите – които се менят – и фактите. Факт е, че комунизмът е осъден като престъпен режим редом с фашизма и националсоциализма, независимо че има българи, които никога не са чували за лагерите за политзатворници в България, тогава носещи срамното име „трудово-възпитателни общежития“ (ТВО). Носталгията по живота по време на комунизма, ако и да е измамно усещане за предвидимост, е легитимно усещане и никой от нас няма право да се присмива на тези емоции у някои хора. Но това не променя факта, че други стотици хиляди са белязани – често произволно и безвъзвратно – от режима. Така че и в лятното училище ние отваряме темата като един голям сноп, в който побираме всякакви детайли от историята на комунизма, в рамките на една седмица отсяваме и подреждаме кое е факт, кое е спомен и кое е част от държавната политика за паметта (или отсъствието на такава).
Разбира се, много по-лесно се помнят, разказват и преподават славните откъси от историята; онези, в които ние сме победители или пък жертви. Обаче как говорим за един период, в който системата на комунизма е палач, ние като общество сме и жертва, и палач, но сме и онова сиво пространство между двете крайности, където има всякакви хора – тихи свидетели на издевателства; хора, които под натиска на службите стават доносници; хора, наивно вярващи във фасадата на комунизма, и т.н. Разговорът за тази сива зона ни е особено важен. Защото нищо в живота не е само черно или бяло. И колкото по-добре разбира един млад човек, че историята на близкото минало е нещо комплексно, толкова по-вероятно е, занимавайки се с нея, да се учи на критично мислене, да размишлява за противоречията и за собствените си пристрастия, за пропагандата или за ролята на гражданското участие.
По Ваши наблюдения какви са промените, ако има такива, в преподаването на историята на тоталитарния режим на деца и студенти в последните години?
В последните години се случиха много неща. Имаше промяна в учебните програми в класовете, в които се изучава съвременна българска история, и комунизмът е застъпен много по-детайлно и най-вече правдиво. В старите програми бяха изпускани термини като „Народен съд“, „Държавна сигурност“ и още много, а в така или иначе малкото отделени часове по тази тема учениците остават с усещането, че комунизмът не е някаква история, заслужаваща внимание. По време на пандемията не успяхме да проследим какво е въздействието на промяната на учебните програми, но се надявам, че с повечето часове, посветени на този период, и познанията ще набъбват.
Разбира се, преподаването е другият препъникамък. Особено учителите, живели преди 89-та, трудно могат да се дистанцират от личния си опит и да си признаят, че историята, която те са учили като педагози, е продукт на същия пропаганден апарат, който би трябвало да могат да осъдят в клас. При по-младите учители невинаги е по-различно, защото, за съжаление, в университетите ни сред историците продължава да има ядра, които не гледат на комунизма като на престъпен режим. Самите методи на преподаване пък са другата трудност – у нас не е много популярно да се преподава с интерактивни методи, а дидактическият подход все още е най-следван. Това често води и до асоцииране на историята със скучна фактология – освен когато става въпрос за история, свързана с националните ни герои Ботев и Левски, за които впрочем също рядко се знае нещо отвъд клишетата.
Засегнахте какво липсва в преподаването и какво може да се подобри. Вие правите ли усилия в тази посока?
Усилията, които ние полагаме, са за демократизиране на самото преподаване и за постоянно създаване на актуални материали, помагала, възможности за участие в проекти в помощ на учители и местни активни граждани, които работят по темите за историята на комунизма и гражданското образование в цялата страна. В обученията си за учители наблягаме на интерактивните методи, така че в час по история или гражданско образование преподавателите да насочват учениците, да модерират процеса на работа в час, а не да държат лекции от по 45 минути. Обученията, които организираме, са базирани на тези методи, така че преподавателите изпитват сами добавената стойност на ученето през преживяване, симулация – или направо през топография, на терен. Именно това е и основен метод в работата ни в лятното училище – работа с история през преживяване на мястото, неговите обитатели сега, артефактите, запечатаните истории, спомените на очевидци.
За нас разговорите за историята на комунизма не са самоцел, те са част от опита ни да усилим значението на демократичната култура в нашето общество. Войната в Украйна е най-радикалното доказателство защо познаването на миналото е важно. То показва не само защо е от съществено значение едно общество да е демократично, в общност с други демокрации, които имат колективен ангажимент да си помагат, но и защо паметта за тоталитаризмите наистина предпазва от това една държава и най-вече гражданите ѝ да се плъзнат надолу по авторитарната крива.
Тази пролет, веднага след началото на войната в Украйна, стартирахме мащабен проект в Белене, посветен на паметта за лагера. Освен че ще разработим виртуален тур, започваме записването на разказите на малкото останали сред нас оцелели от лагера посредством нови технологии, така че техните свидетелства да останат запечатани за следващите поколения. След година вие ще можете от екраните си да водите разговор с тях и те да отговарят на въпросите ви, свързани с живота в лагера, без да сте физически на едно и също място. Тези подходи не само съхраняват паметта и запечатват гласовете на малкото оцелели сред нас, но вървят и в крак с времето и с начина, по който младите хора консумират информация.
Умишлено казвам „тоталитарен режим“ във въпросите си и забелязвам, че и Вие използвате тази формулировка в анонсите на събитието. Засегнахме темата за Украйна, затова питам: в България като че ли асоциираме тоталитаризма единствено с „комунизъм“, но днес виждаме например един руски тоталитарен режим, който няма общо с конкретна идеология и по-скоро заема от всичко по малко. Трудно ли се обясняват тези динамики на ученици и студенти?
Разбира се, ние сме запознати с дискусиите около термина „тоталитарен режим“ и водим такива помежду си като екип, но и с хората, които срещаме в различните си формати. Спорът за термина е дали един режим може да е толкова репресивен, че да е тотален в подтисничеството си и да не оставя нито една брънка от личния и обществен живот незасегната. И честно казано, като чета за броя на доносниците в България по време на комунизма, като гледам снимки за абсурдните граждански ритуали, с които партията се опитва да изземе ролята на Църквата (като гражданското кръщене например), и т.н., съм склонна да кажа, че комунизмът е тоталитарен, наистина се опитва да присъства навсякъде. Затова и щетите, които нанася на гражданската ни култура, са толкова големи, че продължаваме да не можем да се отърсим от тях. И до днес имаме очаквания държавата да се погрижи за нас, и то не в онзи социалдемократичен смисъл, а с готовност жертваме някоя от свободите си за тази грижа. Това граничи с поданическите нагласи, не с гражданските, и принадлежи към друг, отдавна отминал век.
По сходен начин са белязани и руснаците, струва ми се, въпреки че не познавам Русия толкова добре. Това ги прави и лесна жертва на пропагандата, с която ги облъчва режимът на Путин. След края на Студената война антидемократичните режими трудно влизат в калъпа на идеологиите, както ги познаваме от ХХ век, и дори на нас, възрастните, ни е трудно да се ориентираме в тази нова координатна система. Помислете само колко ни е трудно да кажем какво е популизъм, защо някои държави са полудемокрации, а други – полуавторитарни. Освен че е сложно да се обясни, защото е сложно по принцип, допълнително се затрудняваме, когато средата, в която растат младите хора, е пропутински настроена. Тези нагласи не са дълбоки, те са базирани на клишета, зад които дори няма сериозни размисли. Но те са базирани и на тази наследена култура на пасивност, а Путин има образ на мъж с твърда ръка, който брани руснаците от всички реални и (най-вече) конструирани врагове.
В този дух ми направи впечатление и преподаването на „демократични ценности“. Пандемията показа според мен, че трудно разбираме границата между лична свобода и колективно добруване, между необходими за съвместния живот правила и репресия. Същевременно виждаме, че през последните години някои европейски лидери бяха по-склонни да загърбят базови ценности в името на определена realpolitik, която всъщност често пъти води до повече беди в дългосрочен план. Как обясняваме и изобщо дефинираме демократичните ценности в такива времена? Успяваме ли да излезем от клишето, че Западът е задължително демократичен, а Изтокът – задължително недемократичен?
Аз мисля, че човек може да губи за моменти вярата си в общото благо, демокрацията, солидарността, равноправието, но това не значи, че демократичните ценности губят валидност по принцип. И да, дори сегашната ситуация с руския газ и с това кой какво е договорил с „Газпром“ може да накара човек да погледне обединена Европа с повдигната вежда. Но да капитулира тотално и цинично да каже, че всички са маскари и големите винаги печелят, е твърде елементарно. Демократичната култура, или по-скоро демократичните компетенции, които ние се опитваме да насърчаваме, не повеляват това. Те възпитават точно обратното – че демокрацията не е перфектна, но е най-свободната и най-справедлива система, която сме могли да измислим от атиняните насам. Тъй като тя е форма на управление на съжителстването ни в големи групи, няма как да бъде лесна, без конфликти, защото всички имаме различни интереси и предпочитания. Менажирането на динамиките в едно семейство е сложно, какво остава в една нация. И както нямаме очаквания отношенията в най-малките ядра да са елементарни, така не би трябвало да ги имаме и на ниво общество.
В този толкова комплексен свят е нелогично да си обясняваме нещата едновалентно или с дихотомии, въпреки че разбирам защо е примамливо, оттам и защо хората стават жертви на конспирации. Те, конспирациите, правят точно това – яхват разочарованието от сложната реалност, особено на онези, които са ощетени от нея по някакъв начин, и намират лесно, елементарно обяснение за несгодата. За съжаление обаче, рядко тези обяснения са правдиви. Те не само че създават илюзии у хората, но и елиминират изцяло собствената роля в битието, отнемат свободата на действие на човека. За мен ценностите трябва винаги с еднаква сила да се насърчават и в особено трудно моменти усещаме последствията от тяхната липса.
В работата си ние говорим повече за компетенции, отколкото за ценности. Без да навлизам в подробности, на практика компетенцията е онова, което те прави демократичен гражданин, и е комбинация от ценности, нагласи, умения – имаме безплатен видеоурок на тази тема с платформата „Уча.се“, както, между другото, и на всички други теми от учебната програма по гражданско образование за ХI и ХII клас.
Да поговорим и за самите ученици. Как приемат те програмата? Какви въпроси задават най-често? С какви впечатления идват в лятното училище и как те се променят към края на програмата?
В лятното ни училище в Белене идват всякакви младежи. Такива с познания и такива без; участници, чиито семейства имат история на репресии или пък са били облагодетелствани по време на комунизма; деца от малки и големи населени места, от цялата страна. Не изискваме знания или успех, търсим грамотно написана мотивация и обоснован интерес, та дори и той да е артикулиран като „Не зная нищо, но ми е любопитно“. Срещите с оцелели от лагера, обходът на разрушените сгради от Обект 2 в лагера „Белене“, разговорите с лектори, креативните саморефлексии със собствените снимки и написани мисли, работата с архиви от особено голямо значение в Държавния архив в Плевен, срещите с беленската общност – всичко това изисква време, за да се осмисли, преживее и подреди. Преживяването със сигурност е едно от онези неща, които остават за цял живот, защото е емоционално, свързано с мястото и с живите следи.
Въпросите, които чуваме, са от всякакво естество, но последната година ни направи впечатление, че участниците имаха нужда да говорим и за Прехода като исторически период, който им помага да осмислят комунизма. Очевидно са достатъчно далеч от Прехода, за да е той в тяхната перцепция едно безвъзвратно приключено минало. Със сигурност си тръгват с много повече познания за периода, с осъзнаване за дупките от училище, за тишината или носталгията по темата у дома, но най-вече за влиянието на периода и до днес върху нас като общество. Началото на лятното ни училище е посветено на демокрацията, защото тя е реалността на участниците. Оттам вървим хронологично назад към комунизма и завършваме с паметта в демократичното ни общество, вкл. и политизирането ѝ.
Като че ли винаги сме склонни да залагаме надеждите си на по-младите поколения. Според Вас нужни ли са обаче усилия за образоване на предишни поколения относно историческото наследство на близкото минало? Как биха изглеждали такива усилия? Планирате ли нещо подобно?
Като фондация работим точно толкова с възрастни, колкото и с млади хора. В България т.нар. гражданско образование за възрастни не е толкова популярно. Най-вероятно защото гражданското образование не е много популярно като цяло, но и защото с млади хора се работи по-лесно. Нагласите им не са толкова циментирани като тези на възрастните, затова и с тях се постига по-голямо въздействие. С възрастните са нужни по-различни подходи.
Ще дам пример – няколко пъти организираме представления със стендъп комедия, посветена на ЕС и на Прехода. Така през хумора и най-вече през интересите и естествената среда на възрастните инициираме размисли на теми като членството ни в ЕС, финансовите измерения на отворените пазари за стоки и услуги, но и обратната страна на медала – обезлюдяването и липсата на медицински кадри заради емиграцията на Запад. Работим много с преподаватели, както споменах, но нагласите не се променят бързо, възрастните не стават по-демократични от една вечер със стендъп комедия или един семинар. Изисква се работа в продължение на месеци, дори години и разбира се, е много трудно да се изолира влиянието на гражданското образование от всякакви други случки от живота на възрастния.
В този смисъл е трудно и да се каже, че именно гражданското образование за възрастни ги прави по-демократични. Но има достатъчно изследвания, които показват, че то води до по-голяма ангажираност. Че наличието на граждански организации и пространства е предпоставка за повече граждански инициативи и съответно липсата им се свързва с липсата на организиран граждански живот.
Post Syndicated from Светла Енчева original https://toest.bg/ne-strelyayte-po-kalina-kostadinova/
През последната седмица либерално-демократичната общност у нас (към която се причислява и авторката на този текст) намери нова персонализация на гражданското си недоволство. Става въпрос за вицепремиерката Калина Константинова. Обвинителният ѝ тон по адрес на украинските бежанци трудно може да остави безразлични хората с чувствителност към човешките права. Още повече че бягащите от агресията на Русия срещат у обществото ни многократно повече съпричастност, отколкото тези от Сирия или Афганистан. Макар в последно време това да се променя – не на последно място, в резултат на масираната проруска пропаганда у нас.
Правителството интерпретира като изключителна щедрост от страна на държавата настаняването на украинците в луксозни хотели по морето вместо в бежански центрове и палаткови лагери. Мярката беше ограничена до 31 май, а краят ѝ съвпада с началото на туристическия сезон у нас. Говорейки за тази необичайна управленска мярка обаче, като че вече сме позабравили как точно се породи тя.
Всъщност благородната идея да се настаняват украинските бежанци в хотели съвсем не е на правителството. Хотелиерите сами решиха да проявят инициатива и гостоприемство. А после поискаха държавата да им плати за човещината, та да поспечелят нещо извън активния сезон. За разлика от частните лица, които прибраха украинци в домовете си безвъзмездно. Така правителството се оказа пред свършен факт. И тъй като държавата не беше измислила нищо по-добро, реши поне да финансира създадената система за подкрепа. При известни условия – до 40 лв. на човек за нощувка и храна при настаняване в хотели и къщи за гости, вписани в Националния туристически регистър.
Според новата мярка за настаняване на бежанците, анонсирана от заместник-премиерката по ефективно управление, те следваше да бъдат настанени в държавни бази, хотели с по-ниски категории и къщи за гости срещу 15 лв. на ден с храна или 10 – без храна. Ако се чудите колко качествени три хранения може да си осигурите за общо 5 лв., не сте единствените, които си задават този въпрос.
Все пак са се намерили, ако се вярва на правителството, достатъчно собственици на места за настаняване, които да се включат в програмата. Голяма част от тях са в планински курорти и в населени места, които не са известни туристически дестинации. Собствениците им вероятно са преценили, че и малкото държавни пари ще са повече, отколкото те обичайно биха получили. А украинците… все ще се намери с какво да ги хранят.
Бежанците трябваше да напуснат хотелите, в които са настанени, до 31 май, а държавата – да осигури автобуси и влакове за извозването им. На тях обаче не се качи почти никой, въпреки че много от украинците вече бяха попълнили онлайн въпросник за нуждите и намеренията си. Именно това предизвика гнева на Константинова – от нейна гледна точка „неблагодарниците“ са пратили усилията на правителството на вятъра и имат претенции да продължават да си живеят в луксозни хотели по Черноморието.
Само че украинците бяха третирани като чували с картофи. От тяхна гледна точка ситуацията изглежда съвсем иначе – те просто не са получили никаква информация въпреки обещанието за отговор до 48 часа след попълването на въпросника. Ако не са научили по някакъв неформален начин, е нямало как да разберат къде и кога ще ги чакат автобусите. А къде точно ще ги закарат – това съвсем не са знаели. Проблемът е особено сериозен за имащите личен автомобил. За много от тях това е най-скъпото им имущество, оцеляло през войната, и не биха го зарязали просто ей така, за да се отправят в неизвестна посока.
Обидена на „претенциозните“ украинци, Калина Константинова заяви, че вече няма да се допускат празни автобуси и влакове. Вместо това най-нуждаещите се бежанци ще бъдат транспортирани до буферни центрове, а оттам – до държавни бази във вътрешността на страната. Ако бягащите от руската агресия просто бяха получили информация, тази по същество наказателна мярка щеше да е излишна, но уви.
И така, стотици украинци бяха настанени в буферни центрове – фургони без климатици, хладилници и топла вода. Там вечерта не получиха храна, на сутринта закуска имаше, но не и подходяща за малки деца. Държавата оправдава тези условия с аргумента, че те все пак са само за 72 часа. А поуката за бягащите от Украйна е да видят какво им е на „истинските бежанци“, които „не ги глезят“. На сирийските и афганистанските майки с деца в бежански лагери да не им е леко? Щом са нуждаещи се, трябва да са благодарни и да търпят.
Отделен въпрос е, че някои от украинците вече имат личен лекар и не знаят дали на новото място ще могат да се сдобият с такъв. Както и че в изолирани курорти и в обезлюдени градчета не е лесно и за българи да си намерят работа, а какво остава за бежанци, които искат да се устроят и да се интегрират. В същото време има немалко фирми, които биха наели украински бежанци, две трети от които са с висше образование, но освен административните спънки, като например признаване на дипломи, трудно може да се наеме висококвалифицирана специалистка, разпределена в Паничище.
Държавата обаче не мисли особено как да облекчи пълноценното включване на бежанците от Украйна в българското общество. Едно – заради аргумента „ами те искат да си ходят“, друго – заради устойчивата нагласа у нас, че интеграцията е проблем единствено на интегриращите се. Именно тази нагласа е в основата на известната фраза „те не искат да се интегрират“. Независимо кои са „те“.
Интеграцията на чужденци (и в частност на бежанци) е нещо, с което българската държава традиционно не желае да се занимава. След влизането на страната ни в Европейския съюз беше приета Национална стратегия на Република България по миграция и интеграция 2008–2015 г. Тази стратегия вървеше заедно с европейски пари за интеграция, разпределяни от българските институции за проекти. В екипа на един от първите спечелили проекти бях и аз, така че ще споделя малко личен опит.
Задачата ни беше да изработим индикатори за интеграция на имигранти. За тази цел проучихме европейския опит. Силно впечатление ми направи германската система, в която фигурираха включително такива индикатори за качеството на живот – дали чужденецът се задъхва, когато се качва по стълби. За контраст, наличните български индикатори бяха „брой подадени проекти“, „брой спечелени проекти“, „брой проведени кръгли маси“ – все в този дух. Представихме предложенията си за индикатори и… на следващата година беше обявен проект по същата тема. Така държавата плащаше за разработване на индикатори след индикатори, без да въведе никой от тях.
Стратегията за периода 2015–2020 г. вече включваше в заглавието си и думата „убежище“. И този документ си остана само на думи. Отношението към сирийските бежанци в този период показва на какво е способна държавата, ако от нея, а не от предприемчиви хотелиери зависи настаняването на бягащите от война. Единайсет хиляди бежанци от Сирия (колкото беше максималният им брой у нас в един и същи момент) се оказаха непосилно бреме. Те бяха поставени в условия, които дори при огромните усилия на доброволците си оставаха нечовешки.
В заглавието на проекта за стратегия за периода 2021–2025 г., предложен в края на управлението на ГЕРБ и „Обединени патриоти“, интеграцията изобщо отсъства. Както и убежището. До ден днешен документът не е приет, така че понастоящем България е без официална стратегия какво да прави с чуждите граждани на територията си. Като се има предвид въздействието на предишните стратегии, загубата не е голяма. Отсъствието ѝ обаче е показателно за липсата на ангажимент на държавата към чужденците (и бежанците в частност) у нас.
На фона на приоритетите за правосъдна реформа и борба с корупцията чужденците не бяха на дневен ред. Докато един ден украинските майки с деца не започнаха да идват у нас. За разлика от комай всички премиери в най-новата ни история, отношението на Кирил Петков към бежанците беше позитивно и приемащо, а не ксенофобско. Правителството предприе редица мерки за оптимизиране на работата на институциите, включително смени ръководството на Държавната агенция за бежанците.
Ала няма как цялата система за предоставяне на публични услуги, имащи отношение към бежанците, да се подмени като с магическа пръчица. В нея работят администратори, експерти и всевъзможни служители, много от които не могат да си представят, че трябва да служат на хората, а не да упражняват власт над тях, да ги дисциплинират, наказват и унижават. Това важи за публичните услуги в България като цяло, не само по отношение на бежанците, но спрямо най-уязвимите дехуманизиращото отношение достига невъобразими висини.
Вместо управляващите да успеят да реформират държавната администрация, свикнала, че над бежанците се издевателства по дефиниция, се получи обратното. Не успявайки да овладее бюрократичната и ксенофобска система, правителството неусетно възприе нейната реторика. Която неприятно напомня псевдопатриотичния патос на предишното управление, от чиито прийоми настоящата власт се опитва да се еманципира.
Тази неприятна еволюция обаче си има и обратна страна – управляващите поне се опитват да поемат лична отговорност за действията на администрацията, макар и по не най-подходящия начин. Провалът на първия опит за разселване на украинците и въвеждането на новите „дисциплиниращи“ мерки се представят лично от Калина Константинова.
която всъщност е плод на институционална некадърност. А тя – резултат на десетилетно възпроизвеждани вредни модели. Поведението на вицепремиерката не е похвално. Но то е само върхът на айсберга, чиято основа стига надълбоко, далеч отвъд настоящото правителство. По-лесно е да хвърляме вината върху един човек, отколкото да си даваме сметка за принципите на функциониране на социалната система, част от която впрочем сме и самите ние.
За управлението на Кирил Петков настоящата ситуация е тест – дали ще продължат усилията за „продължаване на промяната“, или системата ще претопи опитващите се да я реформират. Случващото се с украинските бежанци е само един пример за ситуацията в почти всяка сфера на държавата.
Post Syndicated from Grab Tech original https://engineering.grab.com/graph-concepts
In an introductory article, we talked about the importance of Graph Networks in fraud detection. In this article, we will be adding some further context on graphs, graph technology and some common use cases.
Connectivity is the most prominent feature of today’s networks and systems. From molecular interactions, social networks and communication systems to power grids, shopping experiences or even supply chains, networks relating to real-world systems are not random. This means that these connections are not static and can be displayed differently at different times. Simple statistical analysis is insufficient to effectively characterise, let alone forecast, networked system behaviour.
As the world becomes more interconnected and systems become more complex, it is more important to employ technologies that are built to take advantage of relationships and their dynamic properties. There is no doubt that graphs have sparked a lot of attention because they are seen as a means to get insights from related data. Graph theory-based approaches show the concepts underlying the behaviour of massively complex systems and networks.
Graphs are mathematical models frequently used in network science, which is a set of technological tools that may be applied to almost any subject. To put it simply, graphs are mathematical representations of complex systems.
The first graph was produced in 1736 in the city of Königsberg, now known as Kaliningrad, Russia. In this city, there were two islands with two mainland sections that were connected by seven different bridges.
Famed mathematician Euler wanted to plot a journey through the entire city by crossing each bridge only once. Euler proceeded to abstract the four regions of the city and the seven bridges into edges but he demonstrated that the problem was unsolvable. A simplified abstract graph is shown in Fig 1.
The graph’s four dots represent Königsberg’s four zones, while the lines represent the seven bridges that connect them. Zones connected by an even number of bridges is clearly navigable because several paths to enter and exit are available. Zones connected by an odd number of bridges can only be used as starting or terminating locations because the same route can only be taken once.
The number of edges associated with a node is known as the node degree. If two nodes have odd degrees and the rest have even degrees, the Königsberg problem could be solved. For example, exactly two regions must have an even number of bridges while the rest have an odd number of bridges. However, as illustrated in Fig 1, no Königsberg location has an even number of bridges, rendering this problem unsolvable.
A graph is a structure that consists of vertices and edges. Vertices, or nodes, are the objects in a problem, while edges are the links that connect vertices in a graph.
Vertices are the fundamental elements that a graph requires to function; there should be at least one in a graph. Vertices are mathematical abstractions that refer to objects that are linked by a condition.
On the other hand, edges are optional as graphs can still be defined without any edges. An edge is a link or connection between any two vertices in a graph, including a connection between a vertex and itself. The idea is that if two vertices are present, there is a relationship between them.
We usually indicate V={v1, v2, …, vn} as the set of vertices, and E = {e1, e2, …, em} as the set of edges. From there, we can define a graph G as a structure G(V, E) which models the relationship between the two sets:
It is worth noting that the order of the two sets within parentheses matters, because we usually express the vertices first, followed by the edges. A graph H(X, Y) is therefore a structure that models the relationship between the set of vertices X and the set of edges Y, not the other way around.
Now that we have covered graphs and their typical components, let us move on to graph data models, which help to translate a conceptual view of your data to a logical model. Two common graph data formats are Resource Description Framework (RDF) and Labelled Property Graph (LPG).
RDF is typically used for metadata and facilitates standardised exchange of data based on their relationships. RDFs typically consist of a triple: a subject, a predicate, and an object. A collection of such triples is an RDF graph. This can be depicted as a node and a directed edge diagram, with each triple representing a node-edge-node graph, as shown in Fig 3.
The three types of nodes that can exist are:
Let us use an example to illustrate this. We have a person with the name Art and we want to plot all his relationships. In this case, the IRI is http://example.org/art and this can be shortened by defining a prefix like ex.
In this example, the IRI http://xmlns.com/foaf/0.1/knows defines the relationship knows. We define foaf as the prefix for http://xmlns.com/foaf/0.1/. The following code snippet shows how a graph like this will look.
@prefix foaf: <http://xmlns.com/foaf/0.1/>
@prefix ex: <http://example.org/>
ex:art foaf:knows ex:bob
ex:art foaf:knows ex:bea
ex:bob foaf:knows ex:cal
ex:bob foaf:knows ex:cam
ex:bea foaf:knows ex:coe
ex:bea foaf:knows ex:cory
ex:bea foaf:age 23
ex:bea foaf:based_near_:o1
In the last two lines, you can see how a literal and blank node would be depicted in an RDF graph. The variable foaf:age is a literal node with the integer value of 23, while foaf:based_near is an anonymous spatial entity with a node identifier of underscore. Outside the context of this graph, o1 is a data identifier with no meaning.
Multiple IRIs, intended for use in RDF graphs, are typically stored in an RDF vocabulary. These IRIs often begin with a common substring known as a namespace IRI. In some cases, namespace IRIs are also associated with a short name known as a namespace prefix. In the example above, http://xmlns.com/foaf/0.1/ is the namespace IRI and foaf and ex are namespace prefixes.
Note: RDF graphs are considered atemporal as they provide a static snapshot of data. They can use appropriate language extensions to communicate information about events or other dynamic properties of entities.
An RDF dataset is a set of RDF graphs that includes one or more named graphs as well as exactly one default graph. A default graph is one that can be empty, and has no associated IRI or name, while each named graph has an IRI or a blank node corresponding to the RDF graph and its name. If there is no named graph specified in a query, the default graph is queried (hence its name).
A labelled property graph is made up of nodes, links, and properties. Each node is given a label and a set of characteristics in the form of arbitrary key-value pairs. The keys are strings, and the values can be any data type. A relationship is then defined by adding a directed edge that is labelled and connects two nodes with a set of properties.
In Fig 4, we have an LPG that shows two nodes: art and bea. The bea node has two characteristics, age and proximity, that are connected by a known edge. This edge has the attribute since because it commemorates the year that art and bea first met.
Nodes, edges and properties must be defined when designing an LPG data model. In this scenario, based_near might not be applicable to all vertices, but they should be defined. You might be wondering, why not represent the city Seattle as a node and add an edge marked as based_near that connects a person and the city?
In general, if there is a value linked to a large number of other nodes in the network and it requires additional properties to correlate with other nodes, it should be represented as a node. In this scenario, the architecture defined in Fig 5 is more appropriate for traversing based_near connections. It also gives us the ability to link any new attributes to the based_near relationship.
Now that we have the context of graphs, let us talk about graph databases, how they help with large data queries and the part they play in Graph Technology.
A graph database is a type of NoSQL database that stores data using network topology. The idea is derived from LPG, which represents data sets with vertices, edges, and attributes.
Graph databases provide a more conceptual view of data that is closer to reality. Modelling complex linkages becomes simpler because interconnections between data points are given the same weight as the data itself.
Relational databases are currently the industry norm and take a structured approach to data, usually in the form of tables. On the other hand, graph databases are agile and focus on immediate relationship understanding. Neither type is designed to replace the other, so it is important to know what each database type has to offer.
There is a domain for both graph and relational databases. Graph databases outperform typical relational databases, especially in use cases involving complicated relationships, as they take a more naturalistic and flowing approach to data.
The key distinctions between graph and relational databases are summarised in the following table:
| Type | Graph | Relational |
|---|---|---|
| Format | Nodes and edges with properties | Tables with rows and columns |
| Relationships | Represented with edges between nodes | Created using foreign keys between tables |
| Flexibility | Flexible | Rigid |
| Complex queries | Quick and responsive | Requires complex joins |
| Use case | Systems with highly connected relationships | Transaction focused systems with more straightforward relationships |
Every database type has its advantages and disadvantages; knowing the distinctions as well as potential options for specific challenges is crucial. Graph databases are a rapidly evolving technology with improved functions compared with other database types.
Some advantages of graph databases include:
The general disadvantages of graph databases are:
Graph technology is the next step in improving analytics delivery. Traditional analytics is insufficient to meet complicated business operations, distribution, and analytical concerns as data quantities expand.
Graph technology aids in the discovery of unknown correlations in data that would otherwise go undetected or unanalysed. When the term graph is used to describe a topic, three distinct concepts come to mind: graph theory, graph analytics, and graph data management.
While the architecture and terminology are sometimes misunderstood, graph analytics’ output can be viewed through visualisation tools, knowledge graphs, particular applications, and even some advanced dashboard capabilities of business intelligence tools. All three concepts above are frequently used to improve system efficiency and even to assist in dynamic data management. In this approach, graph theory and analysis are inextricably linked, and analysis may always rely on graph databases.
Traditional fraud prevention methods concentrate on discrete data points such as individual accounts, devices, or IP addresses. However, today’s sophisticated fraudsters avoid detection by building fraud rings using stolen and fake identities. To detect such fraud rings, we need to look beyond individual data points to the linkages that connect them.
Graph technology greatly transcends the capabilities of a relational database, by revealing hard-to-find patterns. Enterprise businesses also employ Graph technology to supplement their existing fraud detection skills to tackle a wide range of financial crimes, including first-party bank fraud, fraud, and money laundering.
An online business’s success depends on systems that can generate meaningful recommendations in real time. To do so, we need the capacity to correlate product, customer, inventory, supplier, logistical, and even social sentiment data in real time. Furthermore, a real-time recommendation engine must be able to record any new interests displayed during the consumer’s current visit in real time, which batch processing cannot do.
Graph databases outperform relational and other NoSQL data stores in terms of delivering real-time suggestions. Graph databases can easily integrate different types of data to get insights into consumer requirements and product trends, making them an increasingly popular alternative to traditional relational databases.
With complicated scenarios like supply chains, there are many different parties involved and companies need to stay vigilant in detecting issues like fraud, contamination, high-risk areas or unknown product sources. This means that there is a need to efficiently process large amounts of data and ensure transparency throughout the supply chain.
To have a transparent supply chain, relationships between each product and party need to be mapped out, which means there will be deep linkages. Graph databases are great for these as they are designed to search and analyse data with deep links. This means they can process enormous amounts of data without performance issues.
Managing multiple changing roles, groups, products and authorisations can be difficult, especially in large organisations. Graph technology integrates your data and allows quick and effective identity and access control. It also allows you to track all identity and access authorisations and inheritances with significant depth and real-time insights.
Because of the scale and complexity of network and IT infrastructure, you need a configuration management database (CMDB) that is far more capable than relational databases. Neptune is an example of a CMDB and graph database that allows you to correlate your network, data centre, and IT assets to aid troubleshooting, impact analysis, and capacity or outage planning.
A graph database allows you to integrate various monitoring tools and acquire important insights into the complicated relationships that exist between various network or data centre processes. Possible applications of graphs in network and IT operations range from dependency management to automated microservice monitoring.
Risk assessment is crucial in the fintech business. With multiple sources of credit data such as ecommerce sites, mobile wallets and loan repayment records, it can be difficult to accurately assess an individual’s credit risk. Graph Technology makes it possible to combine these data sources, quantify an individual’s fraud risk and even generate full credit reviews.
One clear example of this is IceKredit, which employs artificial intelligence (AI) and machine learning (ML) techniques to make better risk-based decisions. With Graph technology, IceKredit has also successfully detected unreported links and increased efficiency of financial crime investigations.
Whether you’re using stated social connections or inferring links based on behaviour, social graph databases like Neptune introduce possibilities for building new social networks or integrating existing social graphs into commercial applications.
Having a data model that is identical to your domain model allows you to better understand your data, communicate more effectively, and save time. By decreasing the time spent data modelling, graph databases increase the quality and speed of development for your social network application.
AI and ML use statistical and analytical approaches to find patterns in data and provide insights. However, there are two prevalent concerns that arise – the quality of data and effectiveness of the analytics. Some AI and ML solutions have poor accuracy because there is not enough training data or variants that have a high correlation to the outcome.
These ML data issues can be solved with graph databases as it’s possible to connect and traverse links, as well as supplement raw data. With Graph technology, ML systems can recognise each column as a “feature” and each connection as a distinct characteristic, and then be able to identify data patterns and train themselves to recognise these relationships.
Graphs are a great way to visually represent complex systems and can be used to easily detect patterns or relationships between entities. To help improve graphs’ ability to detect patterns early, businesses should consider using Graph technology, which is the next step in improving analytics delivery.
Graph technology typically consists of:
Although predominantly used in fraud detection, Graph technology has many other use cases such as making real-time recommendations based on consumer behaviour, identity and access control, risk assessment and monitoring, AI and ML, and many more.
Check out our next blog article, where we will be talking about how our Graph Visualisation Platform enhances Grab’s fraud detection methods.
Grab is the leading superapp platform in Southeast Asia, providing everyday services that matter to consumers. More than just a ride-hailing and food delivery app, Grab offers a wide range of on-demand services in the region, including mobility, food, package and grocery delivery services, mobile payments, and financial services across 428 cities in eight countries.
Powered by technology and driven by heart, our mission is to drive Southeast Asia forward by creating economic empowerment for everyone. If this mission speaks to you, join our team today!
Post Syndicated from original https://lwn.net/Articles/896349/
The LWN.net Weekly Edition for June 2, 2022 is available.
Post Syndicated from Oglaf! -- Comics. Often dirty. original https://www.oglaf.com/be-water/
Post Syndicated from original https://lwn.net/Articles/896746/
Adding support for an in-kernel TLS
handshake was the topic of a combined storage and filesystem session at the
2022 Linux Storage,
Filesystem, Memory-management and BPF Summit (LSFMM). Chuck Lever and
Hannes Reinecke led the discussion on ways to add that support; they are
interested in order to provide TLS for network storage and filesystems.
But there are likely other features, such as QUIC support, that could use an in-kernel
TLS implementation.
Post Syndicated from CJ Moses original https://aws.amazon.com/blogs/security/join-me-in-boston-this-july-for-aws-reinforce-2022/
I’d like to personally invite you to attend the Amazon Web Services (AWS) security conference, AWS re:Inforce 2022, in Boston, MA on July 26–27. This event offers interactive educational content to address your security, compliance, privacy, and identity management needs. Join security experts, customers, leaders, and partners from around the world who are committed to the highest security standards, and learn how to improve your security posture.
As the new Chief Information Security Officer of AWS, my primary job is to help our customers navigate their security journey while keeping the AWS environment safe. AWS re:Inforce offers an opportunity for you to understand how to keep pace with innovation in your business while you stay secure. With recent headlines around security and data privacy, this is your chance to learn the tactical and strategic lessons that will help keep your systems and tools secure, while you build a culture of security in your organization.
AWS re:Inforce 2022 will kick off with my keynote on Tuesday, July 26. I’ll be joined by Steve Schmidt, now the Chief Security Officer (CSO) of Amazon, and Kurt Kufeld, VP of AWS Platform. You’ll hear us talk about the latest innovations in cloud security from AWS and learn what you can do to foster a culture of security in your business. Take a look at the most recent re:Invent presentation, Continuous security improvement: Strategies and tactics, and the latest re:Inforce keynote for examples of the type of content to expect.
For those who are just getting started on AWS, as well as our more tenured customers, AWS re:Inforce offers an opportunity to learn how to prioritize your security investments. By using the Security pillar of the AWS Well-Architected Framework, sessions address how you can build practical and prescriptive measures to protect your data, systems, and assets.
Sessions are offered at all levels and for all backgrounds, from business to technical, and there are learning opportunities in over 300 sessions across five tracks: Data Protection & Privacy; Governance, Risk & Compliance; Identity & Access Management; Network & Infrastructure Security; and Threat Detection & Incident Response. In these sessions, connect with and learn from AWS experts, customers, and partners who will share actionable insights that you can apply in your everyday work. At AWS re:Inforce, the majority of our sessions are interactive, such as workshops, chalk talks, boot camps, and gamified learning, which provides opportunities to hear about and act upon best practices. Sessions will be available from the intermediate (200) through expert (400) levels, so you can grow your skills no matter where you are in your career. Finally, there will be a leadership session for each track, where AWS leaders will share best practices and trends in each of these areas.
At re:Inforce, hear directly from AWS developers and experts, who will cover the latest advancements in AWS security, compliance, privacy, and identity solutions—including actionable insights your business can use right now. Plus, you’ll learn from AWS customers and partners who are using AWS services in innovative ways to protect their data, achieve security at scale, and stay ahead of bad actors in this rapidly evolving security landscape.
A full conference pass is $1,099. However, if you register today with the code ALUMkpxagvkV you’ll receive a $300 discount (while supplies last).
We’re excited to get back to re:Inforce in person; it is emblematic of our commitment to giving customers direct access to the latest security research and trends. We’ll continue to release additional details about the event on our website, and you can get real-time updates by following @AWSSecurityInfo. I look forward to seeing you in Boston, sharing a bit more about my new role as CISO and providing insight into how we prioritize security at AWS.
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
Want more AWS Security news? Follow us on Twitter.
Post Syndicated from Luca Mezzalira original https://aws.amazon.com/blogs/architecture/lets-architect-architecting-for-governance-and-management/
As you develop next-generation cloud-native applications and modernize existing workloads by migrating to cloud, you need cloud teams that can govern centrally with policies for security, compliance, operations and spend management.
In this edition of Let’s Architect!, we gather content to help software architects and tech leaders explore new ideas, case studies, and technical approaches to help you support production implementations for large-scale migrations.
A multi-account AWS environment helps businesses migrate, modernize, and innovate faster. With the large number of design choices, setting up a multi-account strategy can take a significant amount of time, because it involves configuring multiple accounts and services and requires a deep understanding of AWS.
This blog post shows you how AWS Control Tower helps customers achieve their desired business outcomes by setting up a scalable, secure, and governed multi-account environment. This post describes a strategic migration of 140 AWS accounts from customer Landing Zone to an AWS Control Tower-based solution.
Multi-account landing zone architecture that uses AWS Control Tower
How do you use your existing Microsoft Active Directory (AD) to reliably authenticate access for AWS accounts, infrastructure running on AWS, and third-party applications?
The architecture shown in this blog post is designed to be highly available and extends access to your existing AD to AWS, which enables your users to use their existing credentials to access authorized AWS resources and applications. This post highlights the importance of implementing a cloud authentication and authorization architecture that addresses the variety of requirements for an organization’s AWS Cloud environment.
Multi-account Complete AD architecture with trusts and AWS SSO using AD as the identity source
AWS customers often start their cloud journey with one AWS account, and over time they deploy many resources within that account. Eventually though, they’ll need to use more accounts and migrate resources across AWS Regions and accounts to reduce latency or increase resiliency.
This blog post shows four approaches to migrate resources based on type, configuration, and workload needs across AWS accounts.
Migration infrastructure approach
As enterprises seek digital transformation, their efforts to use cloud technology within their organizations can be a bit disjointed. This video introduces you to the Cloud Center of Excellence (CCoE) and shows you how it can help transform your business via cloud adoption, migration, and operations. By using the CCoE, you’ll establish and us a cross-functional team of people for developing and managing your cloud strategy, governance, and best practices that your organization can use to transform the business using the cloud.
Benefits of CCoE
Thanks for reading! If you want to dive into this topic even more, don’t miss the Management and Governance on AWS product page.
See you in a couple of weeks with novel ways to architect for front-end web and mobile!
AWS Architecture Center provides reference architecture diagrams, vetted architecture solutions, Well-Architected best practices, patterns, icons, and more!
Post Syndicated from Talks at Google original https://www.youtube.com/watch?v=3SPywoJWlxg
Post Syndicated from Jakub Oleksy original https://github.blog/2022-06-01-github-availability-report-may-2022/
In May, we experienced three distinct incidents that resulted in significant impact and degraded state of availability to multiple services across GitHub.com. This report also sheds light into the billing incident that impacted GitHub Actions and Codespaces users in April.
During this incident, our alerting systems detected increased CPU utilization on one of the GitHub Container registry databases. When we received the alert we immediately began investigating. Due to this preemptive monitoring added from the last incident in April at 8:59 UTC, the on-call was readily monitoring and prepared to run mitigation for this incident.
As CPU utilization on the database continued to rise, the Container registry began responding to requests with increased latency, followed by an internal server error for a percentage of requests. At this point we knew there was customer impact and changed the public status of the service. This increased CPU activity was due to a high volume of the “Put Manifest” command. Other package registries were not impacted.
The reason for the CPU utilization was that the throttling criteria configured at the API side for this command was too permissive, and a database query was found to be non-performant under that degree of scale. This caused an outage for anyone using the GitHub Container registry. Users were experiencing latency issues when pushing or pulling packages, as well as getting slow access to the packages UI.
In order to limit impact we throttled the requests from all organizations/users and to restore normal operation, we had to reset our database state by restarting our front-end servers and then the database.
To avoid this in the future, we have added separate rate limiting for operation types from specific organizations/users and will continue working on performance improvements for SQL queries.
Our alerting systems detected degraded availability for API requests during this time. Due to the recency of these incidents, we are still investigating the contributing factors and will provide a more detailed update on the causes and remediations in the June Availability Report, which will be published the first Wednesday of July.
During this incident, services including GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests, and Webhooks were impacted. As we continue to investigate the contributing factors, we will provide a more detailed update in the June Availability Report. We will also share more about our efforts to minimize the impact of similar incidents in the future.
As we mentioned in the April Availability Report, we are now providing a more detailed update on this incident following further investigation.
On April 14, GitHub Actions and Codespaces customers started reporting incorrect charges for metered services shown in their GitHub billing settings. As a result, customers were hitting their GitHub spending limits and unable to run new Actions or create new Codespaces. We immediately started an incident bridge. Our first step was to unblock all customers by giving unlimited Actions and Codespaces usage for no additional charge during the time of this incident.
From looking at the timing and list of recently pushed changes, we determined that the issue was caused by a code change in the metered billing pipeline. When attempting to improve performance of our metered usage processor, Actions and Codespaces minutes were mistakenly multiplied by 1,000,000,000 to convert gigabytes into bytes when this was not necessary for these products. This was due to a change to shared metered billing code that was not thought to impact these products.
To fix the issue, we reverted the code change and started repairing the corrupted data recorded during the incident. We did not re-enable metered billing for GitHub products until we had repaired the incorrect billing data, which happened 24 hours after this incident.
To prevent this incident in the future, we added a Rubocop (Ruby static code analyzer) rule to block pull requests containing non-safe billing code updates. In addition, we added anomaly monitoring for the billed quantity, so next time we are alerted before impacted customers. We also tightened the release process to require a feature flag and end-to-end test when shipping such changes.
We will continue to keep you updated on the progress and investments we’re making to ensure the reliability of our services. To receive real-time updates on status changes, please follow our status page. You can also learn more about what we’re working on on the GitHub Engineering Blog.
Post Syndicated from James Beswick original https://aws.amazon.com/blogs/compute/testing-amazon-eventbridge-events-using-aws-step-functions/
This post is written by Siarhei Kazhura, Solutions Architect and Riaz Panjwani, Solutions Architect.
Amazon EventBridge is a serverless event bus that can be used to ingest and process events from a variety of sources, such as AWS services and SaaS applications. With EventBridge, developers can build loosely coupled and independently scalable event-driven applications.
It can be useful to know with EventBridge when events are not able to reach the desired destination. This can be caused by multiple factors, such as:
EventBridge sends metrics to Amazon CloudWatch, which allows for the detection of failed invocations on a given event rule. You can also use EventBridge rules with a dead-letter queue (DLQ) to identify any failed event deliveries. The messages delivered to the queue contain additional metadata such as error codes, error messages, and the target ARN for debugging.
However, understanding why events fail to deliver is still a manual process. Checking CloudWatch metrics for failures, and then the DLQ takes time. This is evident when developing new functionality, when you must constantly update the event matching patterns and event transformers, and run tests to see if they provide the desired effect. EventBridge sandbox functionality can help with manual testing but this approach does not scale or help with automated event testing.
This post demonstrates how to automate testing for EventBridge events. It uses AWS Step Functions for orchestration, along with Amazon DynamoDB and Amazon S3 to capture the results of your events, Amazon SQS for the DLQ, and AWS Lambda to invoke the workflows and processing.
Using the solution provided in this post, users can track events from its inception to delivery and identify where any issues or errors are occurring. This solution is also customizable, and can incorporate integration tests against events to test pattern matching and transformations.
At a high level:
After an API call, this event is sent to the EventTester express workflow. This orchestrates the automated testing, and returns the results of the test.
In this workflow:
1. The test event is sent to EventBridge to see if the event matches the rule and can be transformed. The result is stored in a DynamoDB table.
2. The PollEventStatus synchronous Express Workflow is invoked. It polls the DynamoDB table until a record with the event ID is found or it reaches the timeout. The configurable timeout is 15 seconds by default.
3. If a record is found, it checks the event status.
From here, there are three possible states. In the first state, if the event status has succeeded:
4. The response from the PollEventStatus workflow is parsed and the payload is formatted.
5. The payload is stored in an S3 bucket.
6. The final response is created, which includes the payload, the event ID, and the event status.
7. The execution is successful, and the final response is returned to the user.
In the second state, if no record is found in the table and the PollEventStatus workflow reaches the timeout:
8. The most likely explanation for reaching the timeout is that the event pattern does not match the rule, so the event is not processed. You can build a test to verify if this is the issue.
9. From the EventBridge SDK, the TestEventPattern call is made to see if the event pattern matches the rule.
10. The results of the TestEventPattern call are checked.
11. If the event pattern does not match the rule, then the issue has been successfully identified and the response is created to be sent back to the user. If the event pattern matches the rule, then the issue has not been identified.
12. The response shows that this is an unexpected error.
In the third state, this acts as a catch-all to any other errors that may occur:
13. The response is created with the details of the unexpected error.
14. The execution has failed, and the final response is sent back to the user.
The following diagram shows how events are sent to EventBridge and their results are captured in S3 and DynamoDB. This is the first step of the EventTester workflow:
When the event is tested:
When the poller runs:
Visit Composing AWS Step Functions to abstract polling of asynchronous services for additional details.
The EventTester workflow uses Express Workflows, which can handle testing high volume event workloads. For example, you can run the solution against large volumes of historical events stored in S3 or CloudWatch.
This can be achieved by using services such as Lambda or AWS Fargate to read the events in batches and run tests simultaneously. To achieve optimal performance, some performance tuning may be required depending on the scale and events that are being tested.
To minimize the cost of the demo, the DynamoDB table is provisioned with 5 read capacity units and 5 write capacity units. For a production system, consider using on-demand capacity, or update the provisioned table capacity.
In this implementation, the EventBridge EventTester can be used to periodically sample events from your system for testing:
This sampling can complement existing metrics exposed for EventBridge via CloudWatch.
To follow the solution walkthrough, visit the solution repository. The walkthrough explains:
This blog post outlines how to use Step Functions, Lambda, SQS, DynamoDB, and S3 to create a workflow that automates the testing of EventBridge events. With this example, you can send events to the EventBridge Event Tester endpoint to verify that event delivery is successful or identify the root cause for event delivery failures.
For more serverless learning resources, visit Serverless Land.
Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/amazon-emr-serverless-now-generally-available-run-big-data-applications-without-managing-servers/
At AWS re:Invent 2021, we introduced three new serverless options for our data analytics services – Amazon EMR Serverless, Amazon Redshift Serverless, and Amazon MSK Serverless – that make it easier to analyze data at any scale without having to configure, scale, or manage the underlying infrastructure.
Today we announce the general availability of Amazon EMR Serverless, a serverless deployment option for customers to run big data analytics applications using open-source frameworks like Apache Spark and Hive without configuring, managing, and scaling clusters or servers.
With EMR Serverless, you can run analytics workloads at any scale with automatic scaling that resizes resources in seconds to meet changing data volumes and processing requirements. EMR Serverless automatically scales resources up and down to provide just the right amount of capacity for your application, and you only pay for what you use.
During the preview, we heard from customers that EMR Serverless is cost-effective because they do not incur cost from having to overprovision resources to deal with demand spikes. They do not have to worry about right-sizing instances or applying OS updates, and can focus on getting products to market faster.
Amazon EMR provides various deployment options to run applications to fit varied needs such as EMR clusters on Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Kubernetes Service (Amazon EKS) clusters, AWS Outposts, or EMR Serverless.
Also, when you build an application using an EMR release (for example, a Spark job using EMR release 6.4), you can choose to run it on an EMR cluster, EMR on EKS, or EMR Serverless without having to rewrite the application. This allows you to build applications for a given framework version and retain the flexibility to change the deployment model based on future operational needs.
Getting Started with Amazon EMR Serverless
To get started with EMR Serverless, you can use Amazon EMR Studio, a free EMR feature which provides an end to end development and debugging experience. With EMR Studio, you can create EMR Serverless applications (Spark or Hive), choose the version of open-source software for your application, submit jobs, check the status of running jobs, and invoke Spark UI or Tez UI for job diagnostics.
When you select the Get started button in the EMR Serverless Console, you can create and set up EMR Studio with preconfigured EMR Serverless applications.
In EMR Studio, when you choose Applications in the Serverless menu, you can create one or more EMR Serverless applications and choose the open source framework and version for your use case. If you want separate logical environments for test and production or for different line-of-business use cases, you can create separate applications for each logical environment.
An EMR Serverless application is a combination of (a) the EMR release version for the open-source framework version you want to use and (b) the specific runtime that you want your application to use, such as Apache Spark or Apache Hive.
When you choose Create application, you can set your application Name, Type of either Spark or Hive, and supported Release version. You can also select the option of default or custom settings for pre-initialized capacity, application limits, and Amazon Virtual Private Cloud (Amazon VPC) connectivity options. Each EMR Serverless application is isolated from other applications and runs within a secure VPC.
Use the default option if you want jobs to start immediately. But charges apply for each worker when the application is started. To learn more about pre-initialized capacity, see Configuring and managing pre-initialized capacity.
When you select Start application, your application is setup to start with pre-initialized capacity of 1 Spark driver and 1 Spark executor. Your application is by default configured to start when jobs are submitted and stop when the application is idle for more than 15 minutes.
You can customize these settings and setup different application limits by selecting Choose custom settings.
In the Job runs menu, you can see a list of run jobs for your application.
Choose Submit job and set up job details such as the name, AWS Identity and Access Management (IAM) role used by the job, script location, and arguments of the JAR or Python script in the Amazon Simple Storage Service (Amazon S3) bucket that you want to run.
If you want logs for your Spark or Hive jobs to be submitted to your S3 bucket, you will need to setup the S3 bucket in the same Region where you are running EMR Serverless jobs.
Optionally, you can set additional configuration properties that you can specify for each job, such as Spark properties, job configurations to override the default configurations for applications (such as using the AWS Glue Data Catalog as its metastore), storing logs to Amazon S3, and retaining logs for 30 days.
The following is an example of running a Python script using the StartJobRun API.
$ aws emr-serverless start-job-run \
--application-id <application_id> \
--execution-role-arn <iam_role_arn> \
--job-driver '{
"sparkSubmit": {
"entryPoint": "s3://spark-scripts/scripts/spark-etl.py",
"entryPointArguments": "s3://spark-scripts/output",
"sparkSubmitParameters": "--conf spark.executor.cores=1 --conf spark.executor.memory=4g --conf spark.driver.cores=1 --conf spark.driver.memory=4g --conf spark.executor.instances=1"
}
}' \
--configuration-overrides '{
"monitoringConfiguration": {
"s3MonitoringConfiguration": {
"logUri": "s3://spark-scripts/logs/"
}
}
}'You can check on job results in your S3 bucket. For details, you can use Spark UI for Spark Application, and Hive/Tez UI in the Job runs menu to understand how the job ran or to debug it if it failed.
For more debugging, EMR Serverless will push event logs to the sparklogs folder in your S3 log destination for Spark applications. In the case of Hive applications, EMR Serverless will continuously upload the Hive driver and Tez tasks logs to the HIVE_DRIVER or TEZ_TASK folders of your S3 log destination. To learn more, see Logging in the AWS documentation.
Things to Know
With EMR Serverless, you can get all the benefits of running Amazon EMR. I want to quote some things to know about EMR Serverless from an AWS Big Data Blog post of preview announcements:
Now Available
Amazon EMR Serverless is available in US East (N. Virginia), US West (Oregon), Europe (Ireland), and Asia Pacific (Tokyo) Regions. With EMR Serverless, there are no upfront costs, and you pay only for the resources you use. You pay for the amount of vCPU, memory, and storage resources consumed by your applications. For pricing details, see the EMR Serverless pricing page.
To learn more, visit the Amazon EMR Serverless User Guide. Please send feedback to AWS re:Post for Amazon EMR Serverless or through your usual AWS support contacts.
Learn all the details about Amazon EMR Serverless and get started today.
– Channy
Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/06/clever-and-exploitable-windows-zero-day.html
Researchers have reported a still-unpatched Windows zero-day that is currently being exploited in the wild.
Here’s the advisory, which includes a work-around until a patch is available.
Post Syndicated from Umair Rehmat original https://aws.amazon.com/blogs/security/when-and-where-to-use-iam-permissions-boundaries/
Customers often ask for guidance on permissions boundaries in AWS Identity and Access Management (IAM) and when, where, and how to use them. A permissions boundary is an IAM feature that helps your centralized cloud IAM teams to safely empower your application developers to create new IAM roles and policies in Amazon Web Services (AWS). In this blog post, we cover this common use case for permissions boundaries, some best practices to consider, and a few things to avoid.
Developers often need to create new IAM roles and policies for their applications because these applications need permissions to interact with AWS resources. For example, a developer will likely need to create an IAM role with the correct permissions for an Amazon Elastic Compute Cloud (Amazon EC2) instance to report logs and metrics to Amazon CloudWatch. Similarly, a role with accompanying permissions is required for an AWS Glue job to extract, transform, and load data to an Amazon Simple Storage Service (Amazon S3) bucket, or for an AWS Lambda function to perform actions on the data loaded to Amazon S3.
Before the launch of IAM permissions boundaries, central admin teams, such as identity and access management or cloud security teams, were often responsible for creating new roles and policies. But using a centralized team to create and manage all IAM roles and policies creates a bottleneck that doesn’t scale, especially as your organization grows and your centralized team receives an increasing number of requests to create and manage new downstream roles and policies. Imagine having teams of developers deploying or migrating hundreds of applications to the cloud—a centralized team won’t have the necessary context to manually create the permissions for each application themselves.
Because the use case and required permissions can vary significantly between applications and workloads, customers asked for a way to empower their developers to safely create and manage IAM roles and policies, while having security guardrails in place to set maximum permissions. IAM permissions boundaries are designed to provide these guardrails so that even if your developers created the most permissive policy that you can imagine, such broad permissions wouldn’t be functional.
By setting up permissions boundaries, you allow your developers to focus on tasks that add value to your business, while simultaneously freeing your centralized security and IAM teams to work on other critical tasks, such as governance and support. In the following sections, you will learn more about permissions boundaries and how to use them.
A permissions boundary is designed to restrict permissions on IAM principals, such as roles, such that permissions don’t exceed what was originally intended. The permissions boundary uses an AWS or customer managed policy to restrict access, and it’s similar to other IAM policies you’re familiar with because it has resource, action, and effect statements. A permissions boundary alone doesn’t grant access to anything. Rather, it enforces a boundary that can’t be exceeded, even if broader permissions are granted by some other policy attached to the role. Permissions boundaries are a preventative guardrail, rather than something that detects and corrects an issue. To grant permissions, you use resource-based policies (such as S3 bucket policies) or identity-based policies (such as managed or in-line permissions policies).
The predominant use case for permissions boundaries is to limit privileges available to IAM roles created by developers (referred to as delegated administrators in the IAM documentation) who have permissions to create and manage these roles. Consider the example of a developer who creates an IAM role that can access all Amazon S3 buckets and Amazon DynamoDB tables in their accounts. If there are sensitive S3 buckets in these accounts, then these overly broad permissions might present a risk.
To limit access, the central administrator can attach a condition to the developer’s identity policy that helps ensure that the developer can only create a role if the role has a permissions boundary policy attached to it. The permissions boundary, which AWS enforces during authorization, defines the maximum permissions that the IAM role is allowed. The developer can still create IAM roles with permissions that are limited to specific use cases (for example, allowing specific actions on non-sensitive Amazon S3 buckets and DynamoDB tables), but the attached permissions boundary prevents access to sensitive AWS resources even if the developer includes these elevated permissions in the role’s IAM policy. Figure 1 illustrates this use of permissions boundaries.
Figure 1: Implementing permissions boundaries
You can use the following policy sample for your developers to allow the creation of roles only if a permissions boundary is attached to them. Make sure to replace <YourAccount_ID> with an appropriate AWS account ID; and the <DevelopersPermissionsBoundary>, with your permissions boundary policy.
You can also deny deletion of a permissions boundary, as shown in the following policy sample.
You can further prevent detaching, modifying, or deleting the policy that is your permissions boundary, as shown in the following policy sample.
Put together, you can use the following permissions policy for your developers to get started with permissions boundaries. This policy allows your developers to create downstream roles with an attached permissions boundary. The policy further denies permissions to detach, delete, or modify the attached permissions boundary policy. Remember, nothing is implicitly allowed in IAM, so you need to allow access permissions for any other actions that your developers require. To learn about allowing access permissions for various scenarios, see Example IAM identity-based policies in the documentation.
You can build on these concepts and apply permissions boundaries to different organizational structures and functional units. In the example shown in Figure 2, the developer can only create IAM roles if a permissions boundary associated to the business function is attached to the IAM roles. In the example, IAM roles in function A can only perform Amazon EC2 actions and Amazon DynamoDB actions, and they don’t have access to the Amazon S3 or Amazon Relational Database Service (Amazon RDS) resources of function B, which serve a different use case. In this way, you can make sure that roles created by your developers don’t exceed permissions outside of their business function requirements.
Figure 2: Implementing permissions boundaries in multiple organizational functions
You might consider restricting your developers by directly applying permissions boundaries to them, but this presents the risk of you running out of policy space. Permissions boundaries use a managed IAM policy to restrict access, so permissions boundaries can only be up to 6,144 characters long. You can have up to 10 managed policies and 1 permissions boundary attached to an IAM role. Developers often need larger policy spaces because they perform so many functions. However, the individual roles that developers create—such as a role for an AWS service to access other AWS services, or a role for an application to interact with AWS resources—don’t need those same broad permissions. Therefore, it is generally a best practice to apply permissions boundaries to the IAM roles created by developers, rather than to the developers themselves.
There are better mechanisms to restrict developers, and we recommend that you use IAM identity policies and AWS Organizations service control policies (SCPs) to restrict access. In particular, the Organizations SCPs are a better solution here because they can restrict every principal in the account through one policy, rather than separately restricting individual principals, as permissions boundaries and IAM identity policies are confined to do.
You should also avoid replicating the developer policy space to a permissions boundary for a downstream IAM role. This, too, can cause you to run out of policy space. IAM roles that developers create have specific functions, and the permissions boundary can be tailored to common business functions to preserve policy space. Therefore, you can begin to group your permissions boundaries into categories that fit the scope of similar application functions or use cases (such as system automation and analytics), and allow your developers to choose from multiple options for permissions boundaries, as shown in the following policy sample.
Finally, it is important to understand the differences between the various IAM resources available. The following table lists these IAM resources, their primary use cases and managing entities, and when they apply. Even if your organization uses different titles to refer to the personas in the table, you should have separation of duties defined as part of your security strategy.
| IAM resource | Purpose | Owner/maintainer | Applies to |
| Federated roles and policies | Grant permissions to federated users for experimentation in lower environments | Central team | People represented by users in the enterprise identity provider |
| IAM workload roles and policies | Grant permissions to resources used by applications, services | Developer | IAM roles representing specific tasks performed by applications |
| Permissions boundaries | Limit permissions available to workload roles and policies | Central team | Workload roles and policies created by developers |
| IAM users and policies | Allowed only by exception when there is no alternative that satisfies the use case | Central team plus senior leadership approval | Break-glass access; legacy workloads unable to use IAM roles |
This blog post covered how you can use IAM permissions boundaries to allow your developers to create the roles that they need and to define the maximum permissions that can be given to the roles that they create. Remember, you can use AWS Organizations SCPs or deny statements in identity policies for scenarios where permissions boundaries are not appropriate. As your organization grows and you need to create and manage more roles, you can use permissions boundaries and follow AWS best practices to set security guard rails and decentralize role creation and management. Get started using permissions boundaries in IAM.
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
Want more AWS Security news? Follow us on Twitter.
Post Syndicated from Barry Kaufman original https://www.backblaze.com/blog/how-to-back-up-and-restore-whatsapp-messages-and-files/
Some of your most valuable digital assets are memories, scattered across the digital ether. For a lot of us, these mementoes of our meta life are tied up in one of Meta’s most popular apps, WhatsApp. The last group chat you had with old college friends. An inspiring note from a loved one. A funny meme that proves your weird uncle does, in fact, have a sense of humor. They’re all out there, drifting through the cloud.
These are treasured memories, and worth preserving, especially in the always uncertain world online. You might find yourself gravitating towards a new messaging app as the technology changes. You might get locked out of your account. It might even be that your country winds up banning the app. (So far, it’s only happened in places like Cuba and Uganda, and while those were temporary, the app was “temporarily” blocked in China in 2017 and is still blocked. The point being, you never know which way the wind is going to blow).
So obviously, it’s worth it to make sure you have some kind of backup for these treasured memories. Now it’s just a matter of creating those backups, and finding somewhere (or, more accurately, several somewheres) to securely store them.
By default, WhatsApp automatically archives your chats every day onto your device, with the option to back them up to Google Drive. As you’ll see, this is not quite sufficient if you really want to preserve those memories. To create a backup that you can preserve elsewhere, use the following steps:
We suggest downloading your backup to your device at this point. This step creates a local backup on your phone. From here, you can either download the attachment onto your computer from that same email, or connect your phone and make a copy from local storage onto your desktop. This will create two copies, but we’re just getting started.
As with anything you back up, the best strategy to employ is the classic 3-2-1 backup strategy. In essence, this is creating a trio of redundant copies, giving your backups their own backups in case anything should go wrong. Typically, you’ll want two copies stored on two different local devices and a third in the cloud.
Is this a little bit of overkill? Absolutely, and we mean that in the best possible way. You’ll thank us when a spilled cup of coffee doesn’t wipe out some of your favorite WhatsApp chats.
If you’ve followed the directions so far, you’ll now have a copy of your chats on your phone as well as on your desktop. This constitutes your first local copy.
For your second local copy, you have a few options:
We’ll admit to being a bit biased here, but for true peace of mind that your backups will survive any number of disasters and mishaps, you have to incorporate cloud storage into your 3-2-1 solution. (If for no other reason than without it, you have a 2-1 situation which doesn’t quite have the same ring to it. But believe us, there are plenty of other reasons, mainly so you don’t lose your data when your computer suddenly dies or you drop your flash drive down the sewer.)
If you are one of the millions of extraordinarily clever people who use Backblaze Personal Backup, this might just be the easiest step in the process because it’s all done automatically. Simply by having your WhatsApp backups on your computer, you’ll rest easy knowing that the software is one step ahead of you, storing your chats and online memories safely in the cloud.
If space is limited locally, and you don’t necessarily need the WhatsApp files on your own computer, Backblaze B2 Cloud Storage gives you plenty of space in the cloud to stash them until they’re needed. Paired with local copies elsewhere, you could also use this method to achieve a 3-2-1 strategy without taking up a huge amount of space locally on your machine.
While it’s all well and good to have your data securely tucked away using the 3-2-1 strategy, what happens when you get a new phone and want to have easy access through the app to all of your old chats? When that happens, you’re going to be glad you squirreled it away, and thrilled that we’ve outlined step-by-step how to get your new phone up and running with all of your old memories intact.
Restoring your WhatsApp backup to an Android phone takes a few steps. In these steps, we will assume you have reinstalled WhatsApp on your new phone.
The restore process for WhatsApp on an iPhone is similar. We will assume you do not have WhatsApp installed with the Android steps.
When it comes to your digital assets, backups matter. Even the most mundane conversations you engage in on WhatsApp now might end up being a treasured memory of old friends later on. Preserving them is important, and if you’ve followed these steps and backed up your memories to your computer and to the cloud, you’ll have those memories available at a moment’s notice wherever you are.
The post How to Back Up and Restore WhatsApp Messages and Files appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.
Post Syndicated from Sam Mokhtari original https://aws.amazon.com/blogs/big-data/integrate-aws-glue-schema-registry-with-the-aws-glue-data-catalog-to-enable-effective-schema-enforcement-in-streaming-analytics-use-cases/
Metadata is an integral part of data management and governance. The AWS Glue Data Catalog can provide a uniform repository to store and share metadata. The main purpose of the Data Catalog is to provide a central metadata store where disparate systems can store, discover, and use that metadata to query and process the data.
Another important aspect of data governance is serving and managing the relationship between data stores and external clients, which are the producers and consumers of data. As the data evolves, especially in streaming use cases, we need a central framework that provides a contract between producers and consumers to enable schema evolution and improved governance. The AWS Glue Schema Registry provides a centralized framework to help manage and enforce schemas on data streaming applications using convenient integrations with Apache Kafka and Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon Kinesis Data Streams, Apache Flink and Amazon Kinesis Data Analytics for Apache Flink, and AWS Lambda.
In this post, we demonstrate how to integrate Schema Registry with the Data Catalog to enable efficient schema enforcement in streaming analytics use cases.
There are many different scenarios where customers want to run stream analytics on AWS while managing the schema evolution effectively. To manage the end-to-end stream analytics life cycle, there are many different applications involved for data production, processing, analytics, routing, and consumption. It can be quite hard to manage changes across different applications for stream analytics use cases. Adding/removing a data field across different stream analytics applications can lead to data quality issues or downstream application failures if it is not managed appropriately.
For example, a large grocery store may want to send orders information using Amazon KDS to it’s backend systems. While sending the order information, customer may want to make some data transformations or run analytics on it. The orders may be routed to different targets depending upon the type of orders and it may be integrated with many backend applications which expect order stream data in specific format. But the order details schema can change due to many different reasons such as new business requirements, technical changes, source system upgrades or something else.
The changes are inevitable but customers want a mechanism to manage these changes effectively while running their stream analytics workloads. To support stream analytics use cases on AWS and enforce schema and governance, customers can make use of AWS Glue Schema Registry along with AWS Stream analytics services.
You can use Amazon Kinesis Data Firehose data transformation to ingest data from Kinesis Data Streams, run a simple data transformation on a batch of records via a Lambda function, and deliver the transformed records to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon OpenSearch Service, Splunk, Datadog, NewRelic, Dynatrace, Sumologic, LogicMonitor, MongoDB, and an HTTP endpoint. The Lambda function transforms the current batch of records with no information or state from previous batches.
Lambda function also has the stream analytics capability for Amazon Kinesis Data Analytics and Amazon DynamoDB. This feature enables data aggregation and state management across multiple function invocations. This capability uses a tumbling window, which is a fixed-size, non-overlapping time interval of up to 15 minutes. When you apply a tumbling window to a stream, records in the stream are grouped by window and sent to the processing Lambda function. The function returns a state value that is passed to the next tumbling window.
Kinesis Data Analytics provides SQL-based stream analytics against streaming data. This service also enables you to use an Apache Flink application to process stream data. Data can be ingested from Kinesis Data Streams and Kinesis Data Firehose while supporting Kinesis Data Firehose (Amazon S3, Amazon Redshift, Amazon OpenSearch Service, and Splunk), Lambda, and Kinesis Data Streams as destinations.
Finally, you can use the AWS Glue streaming extract, transform, and load (ETL) capability as a serverless method to consume data from Kinesis and Apache Kafka or Amazon MSK. The job aggregates, transforms, and enriches the data using Spark streaming, then continuously loads the results into Amazon S3-based data lakes, data warehouses, DynamoDB, JDBC, and more.
Managing stream metadata and schema evolution is becoming more important for stream analytics use cases. To enable these on AWS, the Data Catalog and Schema Registry allow you to centrally control and discover schemas. Before the release of schema referencing in the Data Catalog, you relied on managing schema evolution separately in the Data Catalog and Schema Registry, which usually leads to inconsistencies between these two. With the new release of the Data Catalog and Schema Registry integration, you can now reference schemas stored in the schema registry when creating or updating AWS Glue tables in the Data Catalog. This helps avoid inconsistency between the schema registry and Data Catalog, which results in end-to-end data quality enforcement.
In this post, we walk you through a streaming ETL example in AWS Glue to better showcase how this integration can help. This example includes reading streaming data from Kinesis Data Streams, schema discovery with Schema Registry, using the Data Catalog to store the metadata, and writing out the results to an Amazon S3 as a sink.
The following high-level architecture diagram shows the components to integrate Schema Registry and the Data Catalog to run streaming ETL jobs. In this architecture, Schema Registry helps centrally track and evolve Kinesis Data Streams schemas.
At a high level, we use the Amazon Kinesis Data Generator (KDG) to stream data to a Kinesis data stream, use AWS Glue to run streaming ETL, and use Amazon Athena to query the data.
In the following sections, we walk you through the steps to build this architecture.
To set up a Kinesis data stream, complete the following steps:
ventilator_gsr_stream.You can use the KDG with the ventilator template available on the GitHub repo to generate sample data. The following diagram shows the template on the KDG console.
To add a new schema registry, complete the following steps:
MyDemoSchemaReg).To add a new schema, complete the following steps:
ventilatorstream_schema_gsr) and attach the schema to the schema registry defined in the previous step.Compatibility modes allow you to control how schemas can or cannot evolve over time. These modes form the contract between applications producing and consuming data. When a new version of a schema is submitted to the registry, the compatibility rule applied to the schema name is used to determine if the new version can be accepted. For more information on different compatibility modes, refer to Schema Versioning and Compatibility.
To add a new table in the Data Catalog, complete the following steps:
ventilator_gsr_stream.MyDemoSchemaReg registry created earlier and the schema (ventilatorstream_schema_gsr) with its respective version.You should be able to preview the schema.
To create your AWS Glue job, complete the following steps:
For the IAM role, specify a role that is used for authorization to resources used to run the job and access data stores. Because streaming jobs require connecting to sources and sinks, you need to make sure that the IAM role has permissions to read from Kinesis Data Streams and write to Amazon S3.
database, table_name, and checkpointLocation):Our AWS Glue job is ready to read the data from the Kinesis data stream and send it to Amazon S3 in Parquet format.
The processed streaming data is written in Parquet format to Amazon S3. Run an AWS Glue crawler on the Amazon S3 location where the streaming data is written; the crawler updates the Data Catalog. You can then run queries using Athena to start driving relevant insights from the data.
It’s always a good practice to clean up all the resources created as part of this post to avoid any undue cost. To clean up your resources, delete the AWS Glue database, tables, crawlers, jobs, service role, and S3 buckets.
Additionally, be sure to clean up all other AWS resources that you created using AWS CloudFormation. You can delete these resources on the AWS CloudFormation console by deleting the stack used for the Kinesis Data Generator.
This post demonstrated the importance of centrally managing metadata and schema evolution in stream analytics use cases. It also described how the integration of the Data Catalog and Schema Registry can help you achieve this on AWS. We used a streaming ETL example in AWS Glue to better showcase how this integration can help to enforce end-to-end data quality.
To learn more and get started, you can check out AWS Glue Data Catalog and AWS Glue Schema Registry.
Dr. Sam Mokhtari is a Senior Solutions Architect at AWS. His main area of depth is data and analytics, and he has published more than 30 influential articles in this field. He is also a respected data and analytics advisor, and has led several large-scale implementation projects across different industries, including energy, health, telecom, and transport.
Amar Surjit is a Sr. Solutions Architect based in the UK who has been working in IT for over 20 years designing and implementing global solutions for enterprise customers. He is passionate about streaming technologies and enjoys working with customers globally to design and build streaming architectures and drive value by analyzing their streaming data.
Post Syndicated from Dhanraj Gaikwad original https://aws.amazon.com/blogs/big-data/supercharging-dream11s-data-highway-with-amazon-redshift-ra3-clusters/
This is a guest post by Dhanraj Gaikwad, Principal Engineer on Dream11 Data Engineering team.
Dream11 is the world’s largest fantasy sports platform, with over 120 million users playing fantasy cricket, football, kabaddi, basketball, hockey, volleyball, handball, rugby, futsal, American football, and baseball. Dream11 is the flagship brand of Dream Sports, India’s leading Sports Technology company, and has partnerships with several national and international sports bodies and cricketers.
In this post, we look at how we supercharged our data highway, the backbone of our major analytics pipeline, by migrating our Amazon Redshift clusters to RA3 nodes. We also look at why we were excited about this migration, the challenges we faced during the migration and how we overcame them, as well as the benefits accrued from the migration.
The Dream11 Data Engineering team runs the analytics pipelines (what we call our Data Highway) across Dream Sports. In near-real time, we analyze various aspects that directly impact the end-user experience, which can have a profound business impact for Dream11.
Initially, we were analyzing upwards of terabytes of data per day with Amazon Redshift clusters that ran mainly on dc2.8xlarge nodes. However, due to a rapid increase in our user participation over the last few years, we observed that our data volumes increased multi-fold. Because we were using dc2.8xlarge clusters, this meant adding more nodes of dc2.8xlarge instance types to the Amazon Redshift clusters. Not only was this increasing our costs, it also meant that we were adding additional compute power when what we really needed was more storage. Because we anticipated significant growth during the Indian Premier League (IPL) 2021, we actively explored various options using our AWS Enterprise Support team. Additionally, we were expecting more data volume over the next few years.
After discussions with AWS experts and the Amazon Redshift product team, we at Dream11 were recommended the most viable option of migrating our Amazon Redshift clusters from dc2.8xlarge to the newer RA3 nodes. The most obvious reason for this was the decoupled storage from compute. As a result, we could use lesser nodes and move our storage to Amazon Redshift managed storage. This allowed us to respond to data volume growth in the coming years as well as reduce our costs.
To start off, we conducted a few elementary tests using an Amazon Redshift RA3 test cluster. After we were convinced that this wouldn’t require many changes in our Amazon Redshift queries, we decided to carry out a complete head-to-head performance test between the two clusters.
Because the user traffic on the Dream11 app tends to spike during big ticket tournaments like the IPL, we wanted to ensure that the RA3 clusters could handle the same traffic that we usually experience during our peak. The AWS Enterprise Support team suggested using the Simple Replay tool, an open-sourced tool released by AWS that you can use to record and replay the queries from one Amazon Redshift cluster to another. This tool allows you to capture queries on a source Amazon Redshift cluster, and then replay the same queries on a destination Amazon Redshift cluster (or clusters). We decided to use this tool to capture our performance test queries on the existing dc2.8xlarge clusters and replay them on a test Amazon Redshift cluster composed of RA3 nodes. During this time of our experimentation, the newer version of the automated AWS CloudFormation-based toolset (now on GitHub), was not available.
The first challenge came up when using the Simple Replay tool because there was no easy way to compare the performance of like-to-like queries on the two types of clusters. Although Amazon Redshift provides various statistics using meta-tables about individual queries and their performance, the Simple Replay tool adds additional comments in each Amazon Redshift query on the target cluster to make it easier to know if these queries were run by the Simple Replay tool. In addition, the Simple Replay tool drops comments from the queries on the source cluster.
Comparing each query performance with the Amazon Redshift performance test suite would mean writing additional scripts for easy performance comparison. An alternative would have been to modify the Simple Replay tool code, because it’s open source on GitHub. However, with the IPL 2022 beginning in just a few days, we had to explore another option urgently.
After further discussions with the AWS Enterprise Support team, we decided to use two test clusters: one with the old dc2.8xlarge nodes, and another with the newer RA3 nodes. The idea was to use the Simple Replay tool to run the captured queries from our original cluster on both test clusters. This meant that the queries would be identical on both test clusters, making it easier to compare. Although this meant running an additional test cluster for a few days, we went ahead with this option. As a side note, the newer automated AWS CloudFormation-based toolset does exactly the same in an automated way.
After we were convinced that most of our Amazon Redshift queries performed satisfactorily, we noticed that certain queries were performing slower on the RA3-based cluster than the dc2.8xlarge cluster. We narrowed down the problem to SQL queries with full table scans. We rectified it by following proper data modelling practices in the ETL workflow. Then we were ready to migrate to the newer RA3 nodes.
The migration from the old cluster to the new cluster was smoother than we thought. We used the elastic resize approach, which meant we only had a few minutes of Amazon Redshift downtime. We completed the migration successfully with a sufficient buffer timeline for more tests. Additional tests indicated that the new cluster performed how we wanted it to.
The new cluster performed satisfactorily during our peak performance loads in the IPL as well as the following ICC T20 Cricket World Cup. We’re excited that the new RA3 node-based Amazon Redshift cluster can support our data volume growth needs without needing to increase the number of instance nodes.
We migrated from dc2 to RA3 in April 2021. The data volume has grown by 50% since then. If we had continued with dc2 instances, the cluster cost would have increased by 50%. However, because of the migration to RA3 instances, even with an increase in data volume by 50% since April 2021, the cluster cost has increased by 0.7%, which is attributed to an increase in storage cost.
Migrating to the newer RA3-based Amazon Redshift cluster helped us decouple our computing needs from our storage needs, and now we’re prepared for our expected data volume growth for the next few years. Moreover, we don’t need to add compute nodes if we only need storage, which is expected to bring down our costs in the long run. We did need to fine-tune some of our queries on the newer cluster. With the Simple Replay tool, we could do a direct comparison between the older and the newer cluster. You can also use the newer automated AWS CloudFormation-based toolset if you want to follow a similar approach.
We highly recommend RA3 instances. They give you the flexibility to size your RA3 cluster based on the
amount of data stored without increasing your compute costs.
Dhanraj Gaikwad is a Principal Data Engineer at Dream11. Dhanraj has more than 15 years of experience in the field of data and analytics. In his current role, Dhanraj is responsible for building the data platform for Dream Sports and is specialized in data warehousing, including data modeling, building data pipelines, and query optimizations. He is passionate about solving large-scale data problems and taking unique approaches to deal with them.
Sanket Raut is a Principal Technical Account Manager at AWS based in Vasai ,India. Sanket has more than 16 years of industry experience, including roles in cloud architecture, systems engineering, and software design. He currently focuses on enabling large startups to streamline their cloud operations and optimize their cloud spend. His area of interest is in serverless technologies.
Post Syndicated from Geographics original https://www.youtube.com/watch?v=lUrfdk2m0u8
Post Syndicated from BeardedTinker original https://www.youtube.com/watch?v=OIUglC1lGLI
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.