New Bluetooth Attack

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/12/new-bluetooth-attack.html

New attack breaks forward secrecy in Bluetooth.

Three news articles:

BLUFFS is a series of exploits targeting Bluetooth, aiming to break Bluetooth sessions’ forward and future secrecy, compromising the confidentiality of past and future communications between devices.

This is achieved by exploiting four flaws in the session key derivation process, two of which are new, to force the derivation of a short, thus weak and predictable session key (SKC).

Next, the attacker brute-forces the key, enabling them to decrypt past communication and decrypt or manipulate future communications.

The vulnerability has been around for at least a decade.

Златните ябълки, партиите стомаси и триъгълникът на властта

Post Syndicated from Емилия Милчева original https://www.toest.bg/zlatnite-yabulki-partiite-stomasi-i-triugulnikut-na-vlastta/

Златните ябълки, партиите стомаси и триъгълникът на властта

Обидно за демокрацията, срамно за държавата, печелившо за управляващата коалиция, която си е съвсем истинска сглобка. Така изглежда политическата ситуация в България, вместена в малко думи. В дългия вариант за всяко от тези съждения има куп аргументи.

Пари и власт и тяхното разпределение между трите политически сили ПП–ДБ, ГЕРБ–СДС и ДПС – това са ябълките на раздора в последните седмици на 2023 г. За първи път, откакто ГЕРБ падна от власт през 2021 г., партиите на Бойко Борисов и на Ахмед Доган (пак) се включват в разпределението на разходите в бюджета – като част от конституционноТО, евроатлантическоТО и изобщо управляващоТО мнозинство. А тези партии стомаси са известни с апетитите си – „хранят“ доста бизнеси. Значително по-обиграни са в политически план от ПП–ДБ, а разполагат и с друго тактическо предимство – „играли“ са заедно и преди, сглобката им е смазана от години. 

За ПП–ДБ правителството е безалтернативно и затова се налага да го подкрепят съвместно с ГЕРБ. Присъствието на ДПС също обясняват с крайна необходимост, но не и с партньорство – нужни са 160 гласа за конституционните промени. Какво тогава търсят ДПС и председателят на парламентарната му група Делян Пеевски при вземането на всички решения от управляващото мнозинство?!

Квоти за регулаторите

Освен за бюджетните разходи, неминуеми са кавгите за назначенията в регулаторите и кадровиците на съдебната система, които ще се състоят след конституционните промени, насрочени за гласуване през декември. Дотогава се очаква да е готов и регламент за назначенията, за какъвто се договориха ПП–ДБ, ГЕРБ и ДПС, но не пречи да има назначени и преди приемането му. Как ще се поделят позициите в регулаторите, изглежда най-парливият въпрос, след като всички вече приехме, че най-нормалното нещо на света е това да стане в рамките на новия триъгълник на властта. 

Допреди няколко месеца върли политически противници, днес „добрите“ и „лошите“ управляват заедно, заедно ще реформират съдебната система и най-големият проблем, изглежда, са квотите в регулаторите. Назначените в тях ще укрепят всяка политическа сила, тъй като са с мандати от по няколко години, които надхвърлят средната продължителност на живот на правителствата, и контролират пазари за милиарди, както и информационни масиви. 

Оказа се, че в тази категория са и шефовете на спецслужбите. Но без реформата им, заложена в приоритетите, върху които стъпи правителството на Денков–Габриел. За отсъствието ѝ мълчаливо се съгласиха и трите политически сили. 

Пред bTV тази седмица председателят на ПГ на ГЕРБ Десислава Атанасова обяви, че членовете на регулаторите ще се назначават с мнозинство от 160 гласа – и ДПС ще участва в процеса, получавайки 1/3 тежест. От ПП–ДБ са уклончиви дали това е така, след като процесът още не е започнал. Но всъщност е – с избора на ръководство на БНБ, където ГЕРБ си запази гуверньора, избран още при управлението на Бойко Борисов, а ПП–ДБ и ДПС получиха по един подуправител. Публично потвърждение на думите на Атанасова обаче няма да донесе позитиви на ПП–ДБ, които бездруго понасят негативите от управлението.

Отговорът на най-съществения за общественото благо въпрос – как това кадруване ще повлияе на зле изградените и поради това неработещи демократични институции, ще дойде много по-късно. 

Блокадата за бюджета падна

Бюджетът за 2023 г. се прие в средата на годината и нямаше как да предизвика особени крамоли поради крайно належащата необходимост да го има. Предишният за 2022 г. също закъсня, макар и с по-малко. Гласуваха го в края на февруари и също не вдигна по-голям от обичайния шум, въпреки че правителството беше излъчено от широка коалиция между „Продължаваме промяната“, БСП, „Има такъв народ“ и „Демократична България“. 

Освобождаването на директора на „Пирогов“ от здравния министър д-р Христо Хинков блокира за кратко обсъждането на проектобюджета за 2024 г. От ГЕРБ обявиха, че са имали уговорка да не се пипа д-р Валентин Димитров – шефът на най-голямото лечебно заведение за спешна помощ в България, „ако няма доклади, които да показват обратното“. 

Преди това обаче с гласовете не само на ГЕРБ и ДПС, но и на други политически сили (без ПП–ДБ), бяха окастрени всички предложения на ръководеното от Асен Василев Министерство на финансите. Отпадна идеята за бонус, ако бъде подаден сигнал до данъчните за укрити данъци и осигуровки, стига да бъдат събрани поне 100 000 лева. Отпадна и предложението ресторантьорите да плащат заплати по банков път, в това число и за работодатели с под 100 работници. Отпаднаха и данък „бакшиш“ и превръщането на гражданските договори в трудови по преценка на НАП. Депутатите обаче одобриха предложение на Десислава Танева от ГЕРБ, макар да не беше подкрепено от бюджетната комисия. Става въпрос за данъчно облекчение за земеделските стопани – субсидии до 100 000 лв. няма да бъдат облагаеми, с тази сума ще се намалява данъчната основа преди облагане.  

Но странното е, че Василев не се ядоса на тeзи промени, а на бавенето на бюджета, и заяви, че: 

Ако един министър не може да си осигури подкрепа за политиката, която провежда, най-правилно е да си подаде оставката, вместо да прави циркове и да блокира държавата.

Де факто искането му за оставка на проф. Хинков беше подкрепено от лидера на ГЕРБ и разгневи ДБ – здравният министър е от тяхната квота (номинация на ДСБ). 

Но бюджетът все пак ще влезе в пленарна зала, след като парламентарната здравна комисия, чийто председател е Костадин Ангелов от ГЕРБ, предложи на министъра да оттегли протокола за отстраняването на шефа на „Пирогов“ до приключване на проверки от външни одити и останалите органи, които го проверяват. Остава въпросът дали Хинков ще се съобрази с тази… препоръка.

Напрежението около отстраняването на д-р Димитров извади наяве и вътрешни конфликти в коалицията ПП–ДБ, тъй като депутати от „Продължаваме промяната“ застанаха (не за първи път) срещу Хинков – било поради лични амбиции за постове или с други мотиви.

Конституционните промени – в галоп

Бързината, с която е програмирано да се приемат конституционните промени – на три четения през декември, изненада мнозина юристи. „Нямам обяснение за бързането с гласуването на промените в Конституцията“, каза тази седмица по БНР проф. Екатерина Михайлова, преподавател по конституционно право в НБУ и зам.-председател на 40-тото и 41-вото Народно събрание. Тя припомни и призива на правосъдния министър Атанас Славов: 

Депутатите да не пришпорват процеса, а по-скоро да се търси качество и сигурност в предлагането на окончателните текстове. 

Но вече стана ясно, че проектът ще претърпи доста промени, част от които свързани с правомощията на президента при назначаване на служебно правителство. Изглежда, че те няма да са така радикални, както беше обявено, а ще се търси възможност за колаборация между двете институции – президент и парламент, при подбора на фигурите в кабинета.

Други основни промени са свързани със съдебната реформа – най-съществената част от проекта, от която зависи дали Висшият съдебен съвет ще бъде разделен на прокурорска и съдийска колегия, в която съдиите да имат мнозинство, и дали прокуратурата ще има по-голяма отчетност и по-малко правомощия. По третата промяна – отпадане на изискването министрите и депутатите да са само български граждани, просто се мълчи.

Тази седмица по bTV депутатът от ПП–ДБ Ивайло Мирчев каза:

Ако Конституцията не бъде приета под тази форма, ключовите министри подават оставка. До момента такива индикации по отношение на Конституцията няма. 

Значи няма да има оставки, но най-вероятно ще се чака произнасяне на Конституционния съд.

„Лукойл“ се изтегля от България

Игрите с премахване на дерогацията за „Лукойл“, което ГЕРБ и ДПС уж все се опитват да изтеглят за 1 януари 2024 г. вместо за март, както се договориха трите политически сили, показват, че развръзката е близо. Предстоящата продажба на бургаската рафинерия очевидно е навлязла в крайна фаза и изостря апетитите на местните (политически) играчи. След като ГЕРБ и ДПС сменяха позиции на 180 градуса буквално за часове, в крайна сметка се разбраха с ПП–ДБ, че дерогацията се прекратява на 1 март 2024 г. – тоест от тази дата рафинерията трябва да преработва само неруски петрол. Това е 7 месеца по-рано от крайния срок, отпуснат от Европейската комисия миналото лято. 

Няма потвърждение с кой инвеститор преговаря частната руска компания – с азерската държавна петролна и газова компания SOCAR, която вече осигурява 30% от газа за България, с инвеститор от Дубай или с американска компания. Появиха се слухове за полския енергиен концерн Orlen и дори за купувачи от Украйна. Едно е сигурно – български политици играят за интересите на някои от потенциалните купувачи. Само така може да бъде обяснена настойчивостта на ГЕРБ и ДПС да бъде отменено изключението за внос на руски суров петрол по море колкото се може по-бързо. Но не и с евроатлантизма на „Д2“, както бяха наречени Делян Добрев от ГЕРБ и Делян Пеевски, които в продължение на дни размахваха пръст, че продажбите на „Лукойл“ финансират войната на Путин в Украйна.

Според енергийния експерт Мартин Владимиров към „Лукойл“ има интерес и от Азербайджан, и от Турция. Пред „Фрогнюз“ експертът от Центъра за изследване на демокрацията отбелязва, че в съседна Турция SOCAR оперира най-голямата рафинерия, като внася основно руски, а не азерски петрол. „В замяна е получила огромен заем от „Литаско“, която е собственик на бургаската рафинерия“, казва Владимиров. Ако тази сделката с азерския купувач се състои, България ще намали зависимостта си от руския петрол само на хартия, прогнозира той.

След като премине през всички тези плитчини, сглобката ще се закали още повече и догодина може и да не е така паянтова, както я нарича Бойко Борисов. Но на политически трусове тези постройки са по-издръжливи от монолитните – защото са си безалтернативни.

Светът на финтех. Разговор с Георги Пенев – директор на Българската финтех асоциация

Post Syndicated from Александър Нуцов original https://www.toest.bg/svetut-na-fintech-razgovor-s-georgi-penev/

Светът на финтех. Разговор с Георги Пенев – директор на Българската финтех асоциация

В четвъртата част от поредицата ни за бизнеса ще разгледаме една не съвсем позната, но все по-важна част от българската икономика – финтех. Какво значи финтех, какви проблеми решават финтех компаниите и с какви предизвикателства се сблъскват? Отговорите ще потърсим с директора на Българската финтех асоциация – Георги Пенев. 

Светът на финтех. Разговор с Георги Пенев – директор на Българската финтех асоциация
Георги Пенев започва работа в ИТ сектора преди 8 години, след което бързо преминава към водеща роля в бизнес развитието на печелеща награди българска финтех компания. Има бакалавърска степен по политически науки от УНСС и магистърска степен по публична политика и мениджмънт от Университета в Глазгоу.

Георги, може ли да обясните накратко какво стои зад понятието финтех? 

Терминът е съчетание от думите финанси и технологии. Финтех се отнася за компаниите, които прилагат технологии за улесняване, дигитализиране и автоматизиране на финансовите услуги. Може би най-известният пример за технологии във финансите е банкоматът. В най-общ смисъл обаче финтех обхваща иновациите във финансовия сектор, които конкурират традиционните финансови процеси и институции, като банките например.

Как се развива секторът у нас през годините?

Историята всъщност е много интересна. Ровейки се из различни източници, открих, че през 1968 г. Българската народна банка (БНБ) започва да използва компютър – колкото стая, за изчисляване на сметки. Затова може да приемем тази година за отправна точка. Финтех всъщност започва от банките. Първата финтех компания след падането на тоталитарния режим у нас е добре познатата на всички „Борика“, в която участие има и държавата. 

С технологичния бум обаче тежестта постепенно се измества към услуги, сходни на банковите и предлагани от малки и средни предприятия, но с по-високо качество от гледна точка на бързина, достъпност и цена. Днес финансите на всеки от нас са на само няколко клика разстояние през телефона. А разплащанията, кредитите и застраховките например са само част от широката гама от услуги и продукти, които обхваща финтех индустрията. 

Имате ли данни колко голяма всъщност е финтех индустрията в България и какъв дял представлява от икономиката ни?

В България оперират 156 финтех компании, без да броим банките, а заетите наброяват приблизително 12 000 души. По данни на годишните ни отчети приблизително 44% от служителите в сектора са жени. За сравнение, заетите във Великобритания жени в нашата сфера са едва 30%. Средната брутна заплата на работещите пък надхвърля 3000 лв. Редно е да отбележим, че извадката за данните е от 22 компании. Над 80% от финтех компаниите в България са малки и средни предприятия. И не са само български – на нашия пазар оперират около 60 чуждестранни компании с българските си дружества или с офиси у нас.

Бихте ли ни разказал как технологиите променят финансовия свят? 

С развитието на компютърните и информационните технологии финансовият свят става все по-достъпен за все повече хора. Процесът на financial inclusion, или включването на повече хора във финансовия свят, се ускори драстично през последните години. Необходимостта да посещаваме физически банкови офиси например става все по-малка. С едно обаждане можем да решаваме финансови въпроси, а с електронна идентификация през телефона можем да изпълним административни задачи бързо и лесно.

Всяка страна върви по този път, адаптирайки се към собствените си особености и развитие. Страните от Африка също търсят креативни решения и модернизация на финансовия сектор въпреки лошата си инфраструктура и условия. Хората в Кения например имат възможност за банкиране чрез SMS система, известна като M-PESA, която улеснява трансфера на средства, предоставяйки значителна сигурност на потребителите. 

С какво финтех е ценен за българската икономика? 

Компаниите в сектора са предимно малки и средни предприятия, които играят ключова роля във финансовия сектор, без да имат банков лиценз. Тоест те не оперират като банки, но предлагат сходни услуги и продукти по много по-лесен и удобен начин. В същото време финтех компаниите не подлежат на всички тежки финансови регулации като банките, което ги прави много по-гъвкави. 

Финтех практически съществува, защото финансовите иновации, технологии и дигитализацията изпреварват съществуващото законодателство, което трябва да ги регулира. А това води до няколко важни предимства за пазара.

Четири от пет компании в сектора са с реализиран висок експортен потенциал и са около четири пъти по-добре позиционирани на чуждестранните пазари от средното за българските фирми. 

Българската финтех асоциация работи за най-добро позициониране на страната ни на европейско и световно ниво. Добрият имидж и подготвените ИТ кадри у нас носят изключителна добавена стойност на страната ни, защото привличат ключови чуждестранни инвестиции и предприемачи. Немската Commerzbank например разкри отдела си за дигитална трансформация от около 300 души именно в България. 

Сама по себе си финтех индустрията произведе малко над 1,5 млрд. лв. приходи за 2021 г. И макар да е все още малка като дял от брутния вътрешен продукт, финтех нишата придобива все по-ключово значение за българската икономика заради високата конкурентоспособност на компаниите на европейските и глобалните пазари.

А с какви предизвикателства се сблъсква развитието на финтех средата?

Освен финтех компаниите, екосистемата обхваща множество други компании и хора с подпомагащи функции, например специалисти по киберсигурност, адвокати и кантори, специализирани в сферата, и др. За да продължи да се развива финтех екосистемата, е нужно засилено взаимодействие както с държавата, така и с академичните среди. 

Предизвикателствата пред сектора са свързани най-вече със създаването на устойчиви връзки между бизнеса и институциите. Преди всичко е нужно да се обясни важността на сектора както от икономическа, така и от чисто практическа перспектива за потребителите. БНБ като висш надзорен орган и Комисията за финансов надзор (КФН) като регулатор все още не третират финтех компаниите с нужното внимание. Липсата на надзор обаче крие рискове и създава условия за злоупотреби. 

Същевременно БНБ няма дългосрочна стратегия за развитието на финансовия сектор у нас, което води до липса на сигурност и предвидимост за компаниите. Затова е нужно спешно осъвременяване на работата на надзорните органи и регулаторите, за да посрещат изискванията на динамичната финансова среда.

Какви са конкретните стъпки за осъвременяването им?

Преди всичко трябва да се подобри комуникационната инфраструктура между институциите и бизнеса. Това може да стане чрез въвеждане на т.нар. иновационни фасилитатори. Те представляват единна централизирана точка за комуникация между БНБ, регулаторите и бизнеса. Може да звучи сложно, но всъщност става въпрос за създаване на обикновени канали за комуникация, като телефонна връзка и имейл, чрез които компаниите да имат постоянен достъп до експертност от БНБ, КФН и Комисията за защита на личните данни. 

Идеята е финтех компаниите и предприемачите в сектора да могат да предоставят своите идеи и план за развитие (pitch deck), да разберат дали са приложими спрямо местното законодателство, да научат на какви регулации трябва да отговарят, за да установят бизнеса си, трябва ли им определен лиценз и т.н. Въпреки че информацията, предоставена от органите, няма как да е законово обвързваща – тоест не можеш да се позоваваш на нея в съда например, – тя би била изключително ценна за компаниите, защото идва от най-висша инстанция. Такива иновационни фасилитатори действат успешно в прибалтийските държави. Само за миналата година в Естония са постъпили близо 300 запитвания към финансовия регулатор относно правенето на финтех бизнес в страната, 70% от които на английски език. Това свидетелства за огромния интерес към разгръщане на чужди бизнеси на територията на страната.

С какво могат да допринесат политиците? 

Липсата на адекватно законодателство към момента крие както рискове, така и възможности. Политиците могат да отключат солиден потенциал за развитие с правилните промени, от което да спечелим всички ние като граждани. Ето и един важен пример. Всеки от нас плаща данъци. Това става дигитално с карта или на установените физически локации. Законът досега не позволява транзакциите да се извършват през финтех платформи.

Проблемът тук е, че при всяко картово разплащане държавата плаща на съответната банка такса под формата на процент от сумата. Финтех компаниите могат да предоставят същата услуга, но на много по-ниска цена – таксите могат да бъдат намалени с поне 50%, което по наши изчисления би спестило на държавата и данъкоплатеца десетки милиони левове годишно. 

В по-глобален мащаб политиците ни трябва да издигнат развитието на финансовия сектор и иновациите като приоритет. Друга постсоциалистическа страна, като Литва, вече има над 200 финтех компании. Това в голяма степен е резултат от обявената през 2016 г. национална цел страната да се превърне в стартъп нация. Точно заради държавната подкрепа една от най-популярните финтех компании – Revolut, получи банков лиценз точно от Литва. Лиценз, който използва из целия Европейски съюз.

Ако погледнем към образованието, Финтех асоциацията е известна и с тясното си сътрудничество с академичните среди. Как си взаимодействате с учебните заведения? 

В колаборация със Стопанския факултет на Софийския университет „Св. Климент Охридски“ подкрепяме магистърската програма по финанси, инвестиции и финтех с преподаватели и практици от частния сектор. Заедно провеждаме ежегодно и финтех хакатон – From Zero to Fintech, с идеята да дадем възможност на студентите да създадат собствена финтех компания. В рамките на семинарите и уъркшопите участниците имат възможност да развият бизнес модела си, да тестват продукта си и да предоставят решение на реален финтех казус.

Освен на ниво висше образование подкрепяме въвеждането на образователни програми за финансова грамотност сред учениците. Целта е да се запознаят с функционирането на финансовия свят и да придобият чисто практически знания за традиционните и алтернативните финансови услуги и продукти, за положителните им страни и рисковете, които крият. 

Upgrading GitHub.com to MySQL 8.0

Post Syndicated from Jiaqi Liu original https://github.blog/engineering/infrastructure/upgrading-github-com-to-mysql-8-0/


Over 15 years ago, GitHub started as a Ruby on Rails application with a single MySQL database. Since then, GitHub has evolved its MySQL architecture to meet the scaling and resiliency needs of the platform—including building for high availability, implementing testing automation, and partitioning the data. Today, MySQL remains a core part of GitHub’s infrastructure and our relational database of choice.

This is the story of how we upgraded our fleet of 1200+ MySQL hosts to 8.0. Upgrading the fleet with no impact to our Service Level Objectives (SLO) was no small feat–planning, testing and the upgrade itself took over a year and collaboration across multiple teams within GitHub.

Motivation for upgrading

Why upgrade to MySQL 8.0? With MySQL 5.7 nearing end of life, we upgraded our fleet to the next major version, MySQL 8.0. We also wanted to be on a version of MySQL that gets the latest security patches, bug fixes, and performance enhancements. There are also new features in 8.0 that we want to test and benefit from, including Instant DDLs, invisible indexes, and compressed bin logs, among others.

GitHub’s MySQL infrastructure

Before we dive into how we did the upgrade, let’s take a 10,000-foot view of our MySQL infrastructure:

  • Our fleet consists of 1200+ hosts. It’s a combination of Azure Virtual Machines and bare metal hosts in our data center.
  • We store 300+ TB of data and serve 5.5 million queries per second across 50+ database clusters.
  • Each cluster is configured for high availability with a primary plus replicas cluster setup.
  • Our data is partitioned. We leverage both horizontal and vertical sharding to scale our MySQL clusters. We have MySQL clusters that store data for specific product-domain areas. We also have horizontally sharded Vitess clusters for large-domain areas that outgrew the single-primary MySQL cluster.
  • We have a large ecosystem of tools consisting of Percona Toolkit, gh-ost, orchestrator, freno, and in-house automation used to operate the fleet.

All this sums up to a diverse and complex deployment that needs to be upgraded while maintaining our SLOs.

Preparing the journey

As the primary data store for GitHub, we hold ourselves to a high standard for availability. Due to the size of our fleet and the criticality of MySQL infrastructure, we had a few requirements for the upgrade process:

  • We must be able to upgrade each MySQL database while adhering to our Service Level Objectives (SLOs) and Service Level Agreements (SLAs).
  • We are unable to account for all failure modes in our testing and validation stages. So, in order to remain within SLO, we needed to be able to roll back to the prior version of MySQL 5.7 without a disruption of service.
  • We have a very diverse workload across our MySQL fleet. To reduce risk, we needed to upgrade each database cluster atomically and schedule around other major changes. This meant the upgrade process would be a long one. Therefore, we knew from the start we needed to be able to sustain operating a mixed-version environment.

Preparation for the upgrade started in July 2022 and we had several milestones to reach even before upgrading a single production database.

Prepare infrastructure for upgrade

We needed to determine appropriate default values for MySQL 8.0 and perform some baseline performance benchmarking. Since we needed to operate two versions of MySQL, our tooling and automation needed to be able to handle mixed versions and be aware of new, different, or deprecated syntax between 5.7 and 8.0.

Ensure application compatibility

We added MySQL 8.0 to Continuous Integration (CI) for all applications using MySQL. We ran MySQL 5.7 and 8.0 side-by-side in CI to ensure that there wouldn’t be regressions during the prolonged upgrade process. We detected a variety of bugs and incompatibilities in CI, helping us remove any unsupported configurations or features and escape any new reserved keywords.

To help application developers transition towards MySQL 8.0, we also enabled an option to select a MySQL 8.0 prebuilt container in GitHub Codespaces for debugging and provided MySQL 8.0 development clusters for additional pre-prod testing.

Communication and transparency

We used GitHub Projects to create a rolling calendar to communicate and track our upgrade schedule internally. We created issue templates that tracked the checklist for both application teams and the database team to coordinate an upgrade.

Project Board for tracking the MySQL 8.0 upgrade schedule
Project Board for tracking the MySQL 8.0 upgrade schedule

Upgrade plan

To meet our availability standards, we had a gradual upgrade strategy that allowed for checkpoints and rollbacks throughout the process.

Step 1: Rolling replica upgrades

We started with upgrading a single replica and monitoring while it was still offline to ensure basic functionality was stable. Then, we enabled production traffic and continued to monitor for query latency, system metrics, and application metrics. We gradually brought 8.0 replicas online until we upgraded an entire data center and then iterated through other data centers. We left enough 5.7 replicas online in order to rollback, but we disabled production traffic to start serving all read traffic through 8.0 servers.

The replica upgrade strategy involved gradual rollouts in each data center (DC).
The replica upgrade strategy involved gradual rollouts in each data center (DC).

Step 2: Update replication topology

Once all the read-only traffic was being served via 8.0 replicas, we adjusted the replication topology as follows:

  • An 8.0 primary candidate was configured to replicate directly under the current 5.7 primary.
  • Two replication chains were created downstream of that 8.0 replica:
  • A set of only 5.7 replicas (not serving traffic, but ready in case of rollback).
  • A set of only 8.0 replicas (serving traffic).
  • The topology was only in this state for a short period of time (hours at most) until we moved to the next step.
To facilitate the upgrade, the topology was updated to have two replication chains.
To facilitate the upgrade, the topology was updated to have two replication chains.

Step 3: Promote MySQL 8.0 host to primary

We opted not to do direct upgrades on the primary database host. Instead, we would promote a MySQL 8.0 replica to primary through a graceful failover performed with Orchestrator. At that point, the replication topology consisted of an 8.0 primary with two replication chains attached to it: an offline set of 5.7 replicas in case of rollback and a serving set of 8.0 replicas.

Orchestrator was also configured to blacklist 5.7 hosts as potential failover candidates to prevent an accidental rollback in case of an unplanned failover.

Primary failover and additional steps to finalize MySQL 8.0 upgrade for a database
Primary failover and additional steps to finalize MySQL 8.0 upgrade for a database

Step 4: Internal facing instance types upgraded

We also have ancillary servers for backups or non-production workloads. Those were subsequently upgraded for consistency.

Step 5: Cleanup

Once we confirmed that the cluster didn’t need to rollback and was successfully upgraded to 8.0, we removed the 5.7 servers. Validation consisted of at least one complete 24 hour traffic cycle to ensure there were no issues during peak traffic.

Ability to Rollback

A core part of keeping our upgrade strategy safe was maintaining the ability to rollback to the prior version of MySQL 5.7. For read-replicas, we ensured enough 5.7 replicas remained online to serve production traffic load, and rollback was initiated by disabling the 8.0 replicas if they weren’t performing well. For the primary, in order to roll back without data loss or service disruption, we needed to be able to maintain backwards data replication between 8.0 and 5.7.

MySQL supports replication from one release to the next higher release but does not explicitly support the reverse (MySQL Replication compatibility). When we tested promoting an 8.0 host to primary on our staging cluster, we saw replication break on all 5.7 replicas. There were a couple of problems we needed to overcome:

  1. In MySQL 8.0, utf8mb4 is the default character set and uses a more modern utf8mb4_0900_ai_ci collation as the default. The prior version of MySQL 5.7 supported the utf8mb4_unicode_520_ci collation but not the latest version of Unicode utf8mb4_0900_ai_ci.
  2. MySQL 8.0 introduces roles for managing privileges but this feature did not exist in MySQL 5.7. When an 8.0 instance was promoted to be a primary in a cluster, we encountered problems. Our configuration management was expanding certain permission sets to include role statements and executing them, which broke downstream replication in 5.7 replicas. We solved this problem by temporarily adjusting defined permissions for affected users during the upgrade window.

To address the character collation incompatibility, we had to set the default character encoding to utf8 and collation to utf8_unicode_ci.

For the GitHub.com monolith, our Rails configuration ensured that character collation was consistent and made it easier to standardize client configurations to the database. As a result, we had high confidence that we could maintain backward replication for our most critical applications.

Challenges

Throughout our testing, preparation and upgrades, we encountered some technical challenges.

What about Vitess?

We use Vitess for horizontally sharding relational data. For the most part, upgrading our Vitess clusters was not too different from upgrading the MySQL clusters. We were already running Vitess in CI, so we were able to validate query compatibility. In our upgrade strategy for sharded clusters, we upgraded one shard at a time. VTgate, the Vitess proxy layer, advertises the version of MySQL and some client behavior depends on this version information. For example, one application used a Java client that disabled the query cache for 5.7 servers—since the query cache was removed in 8.0, it generated blocking errors for them. So, once a single MySQL host was upgraded for a given keyspace, we had to make sure we also updated the VTgate setting to advertise 8.0.

Replication delay

We use read-replicas to scale our read availability. GitHub.com requires low replication delay in order to serve up-to-date data.

Earlier on in our testing, we encountered a replication bug in MySQL that was patched on 8.0.28:

Replication: If a replica server with the system variable replica_preserve_commit_order = 1 set was used under intensive load for a long period, the instance could run out of commit order sequence tickets. Incorrect behavior after the maximum value was exceeded caused the applier to hang and the applier worker threads to wait indefinitely on the commit order queue. The commit order sequence ticket generator now wraps around correctly. Thanks to Zhai Weixiang for the contribution. (Bug #32891221, Bug #103636)

We happen to meet all the criteria for hitting this bug.

  • We use replica_preserve_commit_order because we use GTID based replication.
  • We have intensive load for long periods of time on many of our clusters and certainly for all of our most critical ones. Most of our clusters are very write-heavy.

Since this bug was already patched upstream, we just needed to ensure we are deploying a version of MySQL higher than 8.0.28.

We also observed that the heavy writes that drove replication delay were exacerbated in MySQL 8.0. This made it even more important that we avoid heavy bursts in writes. At GitHub, we use freno to throttle write workloads based on replication lag.

Queries would pass CI but fail on production

We knew we would inevitably see problems for the first time in production environments—hence our gradual rollout strategy with upgrading replicas. We encountered queries that passed CI but would fail on production when encountering real-world workloads. Most notably, we encountered a problem where queries with large WHERE IN clauses would crash MySQL. We had large WHERE IN queries containing over tens of thousands of values. In those cases, we needed to rewrite the queries prior to continuing the upgrade process. Query sampling helped to track and detect these problems. At GitHub, we use Solarwinds DPM (VividCortex), a SaaS database performance monitor, for query observability.

Learnings and takeaways

Between testing, performance tuning, and resolving identified issues, the overall upgrade process took over a year and involved engineers from multiple teams at GitHub. We upgraded our entire fleet to MySQL 8.0 – including staging clusters, production clusters in support of GitHub.com, and instances in support of internal tools. This upgrade highlighted the importance of our observability platform, testing plan, and rollback capabilities. The testing and gradual rollout strategy allowed us to identify problems early and reduce the likelihood for encountering new failure modes for the primary upgrade.

While there was a gradual rollout strategy, we still needed the ability to rollback at every step and we needed the observability to identify signals to indicate when a rollback was needed. The most challenging aspect of enabling rollbacks was holding onto the backward replication from the new 8.0 primary to 5.7 replicas. We learned that consistency in the Trilogy client library gave us more predictability in connection behavior and allowed us to have confidence that connections from the main Rails monolith would not break backward replication.

However, for some of our MySQL clusters with connections from multiple different clients in different frameworks/languages, we saw backwards replication break in a matter of hours which shortened the window of opportunity for rollback. Luckily, those cases were few and we didn’t have an instance where the replication broke before we needed to rollback. But for us this was a lesson that there are benefits to having known and well-understood client-side connection configurations. It emphasized the value of developing guidelines and frameworks to ensure consistency in such configurations.

Prior efforts to partition our data paid off—it allowed us to have more targeted upgrades for the different data domains. This was important as one failing query would block the upgrade for an entire cluster and having different workloads partitioned allowed us to upgrade piecemeal and reduce the blast radius of unknown risks encountered during the process. The tradeoff here is that this also means that our MySQL fleet has grown.

The last time GitHub upgraded MySQL versions, we had five database clusters and now we have 50+ clusters. In order to successfully upgrade, we had to invest in observability, tooling, and processes for managing the fleet.

Conclusion

A MySQL upgrade is just one type of routine maintenance that we have to perform – it’s critical for us to have an upgrade path for any software we run on our fleet. As part of the upgrade project, we developed new processes and operational capabilities to successfully complete the MySQL version upgrade. Yet, we still had too many steps in the upgrade process that required manual intervention and we want to reduce the effort and time it takes to complete future MySQL upgrades.

We anticipate that our fleet will continue to grow as GitHub.com grows and we have goals to partition our data further which will increase our number of MySQL clusters over time. Building in automation for operational tasks and self-healing capabilities can help us scale MySQL operations in the future. We believe that investing in reliable fleet management and automation will allow us to scale github and keep up with required maintenance, providing a more predictable and resilient system.

The lessons from this project provided the foundations for our MySQL automation and will pave the way for future upgrades to be done more efficiently, but still with the same level of care and safety.

The post Upgrading GitHub.com to MySQL 8.0 appeared first on The GitHub Blog.

Storage Tech of the Future: Ceramics, DNA, and More

Post Syndicated from Molly Clancy original https://www.backblaze.com/blog/storage-tech-of-the-future-ceramics-dna-and-more/

A decorative image showing a data drive with a health monitor indicator running through and behind it.

Two announcements had the Backblaze #social Slack channel blowing up this week, both related to “Storage Technologies of the Future.” The first reported “Video of Ceramic Storage System Surfaces Online” like some kind of UFO sighting. The second, somewhat more restrained announcement heralded the release of DNA storage cards available to the general public. Yep, you heard that right—coming to a Best Buy near you. (Not really. You absolutely have to special order these babies, but they ARE going to be for sale.)

We talked about DNA storage way back in 2015. It’s been nine years, so we thought it was high time to revisit the tech and dig into ceramics as well. (Pun intended.) 

What Is DNA Storage?

The idea is elegant, really. What is DNA if not an organic, naturally occuring form of code? 

DNA consists of four nucleotide bases: adenine (A), thymine (T), cytosine (C), and guanine (G). 

In DNA storage, information is encoded into sequences of these nucleotide bases. For example, A and C might represent 0, while T and G represent 1. This encoding allows digital data, such as text, images, or other types of information, to be translated into DNA sequences. Cool!

The appeal of DNA as a storage medium lies in its density and stability, as well as its ability to store vast amounts of information in a very compact space. It also boasts remarkable durability, with the ability to preserve information for thousands of years under suitable conditions. I mean, leave it to Mother Nature to put our silly little hard drives to shame.

Back in 2015, we shared that the storage density of DNA was about 2.2 petabytes per gram. In 2017, a study out of Columbia University and the New York Genome Center put it at an incredible 215 petabytes per gram. For comparison’s sake, a WDC 22TB drive (WDC WUH722222ALE6L4) that we currently use in our data centers is 1.5 pounds or 680 grams, which nets out at 0.032TB/gram or 0.000032PB/gram.

Another major advantage is its sustainability. Estimated global data center electricity consumption in 2022 was 240–340 TWh1, or around 1–1.3% of global final electricity demand. Current data storage technology uses rare earth metals which are environmentally damaging to mine. Drives take up space, and they also create e-waste at the end of their lifecycle. It’s a challenge anyone who works in the data storage industry thinks about a lot. 

DNA storage, on the other hand, requires less energy. A 2023 study found that data writing can be achieved in the DNA movable-type storage system under normal operating temperatures ranging from about 60–113°F and can be stored at room temperature. DNA molecules are also biodegradable and can be broken down naturally. 

The DNA data-writing process is chemical-based, and actually not the most environmentally friendly, but the DNA storage cards developed by Biomemory use a proprietary biosourced writing process, which they call “a significant advancement over existing chemical or enzymatic synthesis technologies.” So, there might be some trade-offs, but we’ll know more as the technology evolves. 

What’s the Catch?

Density? Check. Durability? Wow, yeah. Sustainability? You got it. But DNA storage is still a long way from sitting on your desk, storing your duplicate selfies. First, and we said this back in 2015 too, DNA takes a long time to read and write—DNA synthesis writes at a few hundred bytes per second. An average iPhone photo would take several hours to write to DNA. And to read it, you have to sequence the DNA—a time-intensive process. Both of those processes require specialized scientific equipment.

It’s also still too expensive. In 2015, we found a study that put 83 kilobytes of DNA storage at £1000 (about $1,500 U.S. dollars). In 2021, MIT estimated it would cost about $1 trillion to store one petabyte of data on DNA. For comparison, it costs $6,000 per month to store one petabyte in Backblaze B2 Cloud Storage ($6/TB/month). You could store that petabyte for a little over 13 million years before you’d hit $1 trillion.

Today, Biomemory’s DNA storage cards ring in at a cool €1000 (about $1,080 U.S. dollars). And they can hold a whopping one kilobyte of data or the equivalent of a short email. So, yeah …it’s ahh, gotten even more expensive for the commercial product. 

The discrepancy between the MIT theoretical estimate and the cost of the Biomemory cards really speaks to the expense of bringing a technology like this to market. The theoretical cost per byte is a lot different than the operational cost, and the Biomemory cards are really meant to serve as proof of concept.  All that said, as the technology improves, one can only hope that it becomes more cost-effective in the future. Folks are experimenting with different encoding schemes to make writing and reading more efficient, as one example of an advance that could start to tip the balance.  

Finally, there’s just something a bit spooky about using synthetic DNA to store data. There’s a Black Mirror episode in there somewhere. Maybe one day we can upload kung fu skills directly into our brain domes and that would be cool, but for now, it’s still somewhat unsettling.

What Is Ceramic Storage?

Ceramic storage makes an old school approach new again, if you consider that the first stone tablets were kind of the precursor to today’s hard drives. Who’s up for storing some cuneiform?

Cerabyte, the company behind the “video that surfaced online,” is working on storage technology that uses ceramic and glass substrates in devices the size of a typical HDD that can store 10 petabytes of data. They use a glass base similar to Gorilla Glass by Corning topped with a layer of ceramic 300 micrometers thick that’s essentially etched with lasers. (Glass is used in many larger hard drives today, for what it’s worth. Hoya makes them, for example.) The startup debuted a fully operational prototype system using only commercial off-the-shelf equipment—pretty impressive. 

The prototype consists of a single read-write rack and several library racks. When you want to write data, it moves one of the cartridges from the library to the read-write rack where it is opened to expose and stage the ceramic substrate. Two million laser beamlets then punch nanoscale ones and zeros into the surface. Once the data is written, the read-write arm verifies it on the return motion to its original position. 

Cerabyte isn’t the only player in the game. Others like MDisc use similar technology. Currently, MDisc stores data on DVD-sized disks using a “rock-like” substrate. Several DVD player manufacturers have included the technology in players. 

Similar to DNA storage, ceramic storage boasts much higher density than current data storage tech—terabytes per square centimeter versus an HDD’s 0.02TB per square centimeter. Also like DNA storage, it’s more environmentally friendly. Ceramic and glass can be stored within a wide temperature range between -460°F–570°F, and it’s a natural material that will last millennia and eventually decompose. It’s also incredibly durable: Cerabyte claims it will last 5000+ years, and with tons of clay pots still laying around from ancient times, that makes sense. 

One advantage it has on DNA storage though is speed. One laser pulse writes up to 2,000,000 bits, so data can be written at GBps speeds. 

What’s the Catch?

Ceramic also has density, sustainability, and speed to boot, but our biggest question is: who’s going to need that speed? There are only a handful of applications, like AI, that require that speed now. AI is certainly having a big moment, and it can only get bigger. So, presumably there’s a market, but only a small one that can justify the cost. 

One other biggie, at least for a cloud storage provider like us, though not necessarily for consumers or other enterprise users: it’s a write-once model. Once it’s on there, it’s on there. 

Finally, much like DNA tech, it’s probably (?) still too expensive to make it feasible for most data center applications. Cerabyte hasn’t released pricing yet. According to Blocks & Files, “The cost roadmap is expected to offer cost structures below projections of current commercial storage technologies.” But it’s still a big question mark.

Our Hot Take

Both of these technologies are really cool. They definitely got our storage geek brains fired up. But until they become scalable, operationally feasible, and cost-effective, you won’t see them in production—they’re still far enough out that they’re on the fiction end of the science fiction to science fact spectrum. And there are a couple roadblocks we see before they reach the ubiquity of your trusty hard drive. 

The first is making both technologies operational, not just theoretical in a lab. We’ll know more about both Biomemory’s and Cerabyte’s technologies as they roll out these initial proof of concept cards and prototype machines. And both have plans, naturally, for scaling the technologies to the data center. Whether they can or not remains to be seen. Lots of technologies have come and gone, falling victim to the challenges of production, scaling, and cost. 

The second is the attendant infrastructure needs. Getting 100x speed is great, if the device is right next to you. But we’ll need similar leaps in physical networking infrastructure to transfer the data anywhere else. Until that catches up, the tech remains lab-bound. 

All that said, I still remember using floppy disks that held mere megabytes of data, and now you can put 20TB on a hard disk. So, I guess the question is, how long will it be before I can plug into the Matrix?

The post Storage Tech of the Future: Ceramics, DNA, and More appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

AWS Clean Rooms proof of concept scoping part 1: media measurement

Post Syndicated from Shaila Mathias original https://aws.amazon.com/blogs/big-data/aws-clean-rooms-proof-of-concept-scoping-part-1-media-measurement/

Companies are increasingly seeking ways to complement their data with external business partners’ data to build, maintain, and enrich their holistic view of their business at the consumer level. AWS Clean Rooms helps companies more easily and securely analyze and collaborate on their collective datasets—without sharing or copying each other’s underlying data. With AWS Clean Rooms, you can create a secure data clean room in minutes and collaborate with any other company on Amazon Web Services (AWS) to generate unique insights.

One way to quickly get started with AWS Clean Rooms is with a proof of concept (POC) between you and a priority partner. AWS Clean Rooms supports multiple industries and use cases, and this blog is the first of a series on types of proof of concepts that can be conducted with AWS Clean Rooms.

In this post, we outline planning a POC to measure media effectiveness in a paid advertising campaign. The collaborators are a media owner (“CTV.Co,” a connected TV provider) and brand advertiser (“Coffee.Co,” a quick service restaurant company), that are analyzing their collective data to understand the impact on sales as a result of an advertising campaign. We chose to start this series with media measurement because “Results & Measurement” was the top ranked use case for data collaboration by customers in a recent survey the AWS Clean Rooms team conducted.

Important to keep in mind

  • AWS Clean Rooms is generally available so any AWS customer can sign in to the AWS Management Console and start using the service today without additional paperwork.
  • With AWS Clean Rooms, you can perform two types of analyses: SQL queries and machine learning. For the purpose of this blog, we will be focusing only on SQL queries. You can learn more about both types of analyses and their cost structures on the AWS Clean Rooms Features and Pricing webpages. The AWS Clean Rooms team can help you estimate the cost of a POC and can be reached at [email protected].
  • While AWS Clean Rooms supports multiparty collaboration, we assume two members in the AWS Clean Rooms POC collaboration in this blog post.

Overview

Setting up a POC helps define an existing problem of a specific use case for using AWS Clean Rooms with your partners. After you’ve determined who you want to collaborate with, we recommend three steps to set up your POC:

  • Defining the business context and success criteria – Determine which partner, which use case should be tested, and what the success criteria are for the AWS Clean Rooms collaboration.
  • Aligning on the technical choices for this test – Make the technical decisions of who sets up the clean room, who is analyzing the data, which data sets are being used, join keys and what analysis is being run.
  • Outlining the workflow and timing – Create a workback plan, decide on synthetic data testing, and align on production data testing.

In this post, we walk through an example of how a quick service restaurant (QSR) coffee company (Coffee.Co) would set up a POC with a connected TV provider (CTV.Co) to determine the success of an advertising campaign.

Business context and success criteria for the POC

Define the use case to be tested

The first step in setting up the POC is defining the use case being tested with your partner in AWS Clean Rooms. For example, Coffee.Co wants to run a measurement analysis to determine the media exposure on CTV.Co that led to sign up for Coffee.Co’s loyalty program. AWS Clean Rooms allows for Coffee.Co and CTV.Co to collaborate and analyze their collective datasets without copying each other’s underlying data.

Success criteria

It’s important to determine metrics of success and acceptance criteria to move the POC to production upfront. For example, Coffee.Co’s goal is to achieve a sufficient match rate between their data set and CTV.Co’s data set to ensure the efficacy of the measurement analysis. Additionally, Coffee.Co wants ease-of-use for existing Coffee.Co team members to set up the collaboration and action on the insights driven from the collaboration to optimize future media spend to tactics on CTV.Co that will drive more loyalty members.

Technical choices for the POC

Determine the collaboration creator, AWS account IDs, query runner, payor and results receiver

Each AWS Clean Rooms collaboration is created by a single AWS account inviting other AWS accounts. The collaboration creator specifies which accounts are invited to the collaboration, who can run queries, who pays for the compute, who can receive the results, and the optional query logging and cryptographic computing settings. The creator is also able to remove members from a collaboration. In this POC, Coffee.Co initiates the collaboration by inviting CTV.Co. Additionally, Coffee.Co runs the queries and receives the results, but CTV.Co pays for the compute.

Query logging setting

If logging is enabled in the collaboration, AWS Clean Rooms allows each collaboration member to receive query logs. The collaborator running the queries, Coffee.Co, gets logs for all data tables while the other collaborator, CTV.Co, only sees the logs if their data tables are referenced in the query.

Decide the AWS region

The underlying Amazon Simple Storage Service (Amazon S3) and AWS Glue resources for the data tables used in the collaboration must be in the same AWS Region as the AWS Clean Rooms collaboration. For example, Coffee.Co and CTV.Co agree on the US East (Ohio) Region for their collaboration.

Join keys

To join data sets in an AWS Clean Rooms query, each side of the join must share a common key. Key join comparison with the equal to operator (=) must evaluate to True. AND or OR logical operators can be used in the inner join for matching on multiple join columns. Keys such as email address, phone number, or UID2 are often considered. Third party identifiers from LiveRamp, Experian, or Neustar can be used in the join through AWS Clean Rooms specific work flows with each partner.

If sensitive data is being used as join keys, it’s recommended to use an obfuscation technique to mitigate the risk of exposing sensitive data if the data is mishandled. Both parties must use a technique that produces the same obfuscated join key values such as hashing. Cryptographic Computing for Clean Rooms can be used for this propose.

In this POC, Coffee.Co and CTV.Co are joining on hashed email or hashed mobile. Both collaborators are using the SHA256 hash on their plaintext email and phone number when preparing their data sets for the collaboration.

Data schema

The exact data schema must be determined by collaborators to support the agreed upon analysis. In this POC, Coffee.Co is running a conversion analysis to measure media exposures on CTV.Co that led to sign-up for Coffee.Co’s loyalty program. Coffee.Co’s schema includes hashed email, hashed mobile, loyalty sign up date, loyalty membership type, and birthday of member. CTV.Co’s schema includes hashed email, hashed mobile, impressions, clicks, timestamp, ad placement, and ad placement type.

Analysis rule applied to each configured table associated to the collaboration

An AWS Clean Rooms configured table is a reference to an existing table in the AWS Glue Data Catalog that’s used in the collaboration. It contains an analysis rule that determines how the data can be queried in AWS Clean Rooms. Configured tables can be associated to one or more collaborations.

AWS Clean Rooms offers three types of analysis rules: aggregation, list, and custom.

  • Aggregation allows you to run queries that generate an aggregate statistic within the privacy guardrails set by each data owner. For example, how large the intersection of two datasets is.
  • List allows you to run queries that extract the row level list of the intersection of multiple data sets. For example, the overlapped records on two datasets.
  • Custom allows you to create custom queries and reusable templates using most industry standard SQL, as well as review and approve queries prior to your collaborator running them. For example, authoring an incremental lift query that’s the only query permitted to run on your data tables. You can also use AWS Clean Rooms Differential Privacy by selecting a custom analysis rule and then configuring your differential privacy parameters.

In this POC, CTV.Co uses the custom analysis rule and authors the conversion query. Coffee.Co adds this custom analysis rule to their data table, configuring the table for association to the collaboration. Coffee.Co is running the query, and can only run queries that CTV.Co authors on the collective datasets in this collaboration.

Planned query

Collaborators should define the query that will be run by the collaborator determined to run the queries. In this POC, Coffe.Co runs the custom analysis rule query CTV.Co authored to understand who signed up for their loyalty program after being exposed to an ad on CTV.Co. Coffee.Co can specify their desired time window parameter to analyze when the membership sign-up took place within a specific date range, because that parameter has been enabled in the custom analysis rule query.

Workflow and timeline

To determine the workflow and timeline for setting up the POC, the collaborators should set dates for the following activities.

  1. Coffee.Co and CTV.Co align on business context, success criteria, technical details, and prepare their data tables.
    • Example deadline: January 10.
  2. [Optional] Collaborators work to generate representative synthetic datasets for non-production testing prior to production data testing.
    • Example deadline: January 15
  3. [Optional] Each collaborator uses synthetic datasets to create an AWS Clean Rooms collaboration between two of their owned AWS non-production accounts and finalizes analysis rules and queries they want to run in production.
    • Example deadline: January 30
  4. [Optional] Coffee.Co and CTV.Co create an AWS Clean Rooms collaboration between non-production accounts and tests the analysis rules and queries with the synthetic datasets.
    • Example deadline: February 15
  5. Coffee.Co and CTV.Co create a production AWS Clean Rooms collaboration and run the POC queries on production data.
    • Example deadline: Feb 28
  6. Evaluate POC results against success criteria to determine when to move to production.
    • Example deadline March 15

Conclusion

After you’ve defined the business context and success criteria for the POC, aligned on the technical details, and outlined the workflow and timing, the goal of the POC is to run a successful collaboration using AWS Clean Rooms to validate moving to production. After you’ve validated that the collaboration is ready to move to production, AWS can help you identify and implement automation mechanisms to programmatically run AWS Clean Rooms for your production use cases. Watch this video to learn more about privacy-enhanced collaboration and contact an AWS Representative to learn more about AWS Clean Rooms.

About AWS Clean Rooms

AWS Clean Rooms helps companies and their partners more easily and securely analyze and collaborate on their collective datasets—without sharing or copying one another’s underlying data. With AWS Clean Rooms, customers can create a secure data clean room in minutes, and collaborate with any other company on AWS to generate unique insights about advertising campaigns, investment decisions, and research and development.

Additional resources


About the authors

Shaila Mathias  is a Business Development lead for AWS Clean Rooms at Amazon Web Services.

Allison Milone is a Product Marketer for the Advertising & Marketing Industry at Amazon Web Services.

Ryan Malecky is a Senior Solutions Architect at Amazon Web Services. He is focused on helping customers build gain insights from their data, especially with AWS Clean Rooms.

[$] Controlling shadow-stack allocation in clone3()

Post Syndicated from corbet original https://lwn.net/Articles/953794/

User-space shadow stacks are a relatively new feature in Linux; support was
only added for 6.6, and is limited to the x86
architecture
. As support for other architectures (including arm64 and RISC-V) approaches readiness,
though, more thought is going into the API for this feature. As a recent
discussion on the integration of shadow stacks with the clone3() system call shows, there are
still some details to be worked out.

Simplify workforce identity management using IAM Identity Center and trusted token issuers

Post Syndicated from Roberto Migli original https://aws.amazon.com/blogs/security/simplify-workforce-identity-management-using-iam-identity-center-and-trusted-token-issuers/

AWS Identity and Access Management (IAM) roles are a powerful way to manage permissions to resources in the Amazon Web Services (AWS) Cloud. IAM roles are useful when granting permissions to users whose workloads are static. However, for users whose access patterns are more dynamic, relying on roles can add complexity for administrators who are faced with provisioning roles and making sure the right people have the right access to the right roles.

The typical solution to handle dynamic workforce access is the OAuth 2.0 framework, which you can use to propagate an authenticated user’s identity to resource services. Resource services can then manage permissions based on the user—their attributes or permissions—rather than building a complex role management system. AWS IAM Identity Center recently introduced trusted identity propagation based on OAuth 2.0 to support dynamic access patterns.

With trusted identity propagation, your requesting application obtains OAuth tokens from IAM Identity Center and passes them to an AWS resource service. The AWS resource service trusts tokens that Identity Center generates and grants permissions based on the Identity Center tokens.

What happens if the application you want to deploy uses an external OAuth authorization server, such as Okta Universal Directory or Microsoft Entra ID, but the AWS service uses IAM Identity Center? How can you use the tokens from those applications with your applications that AWS hosts?

In this blog post, we show you how you can use IAM Identity Center trusted token issuers to help address these challenges. You also review the basics of Identity Center and OAuth and how Identity Center enables the use of external OAuth authorization servers.

IAM Identity Center and OAuth

IAM Identity Center acts as a central identity service for your AWS Cloud environment. You can bring your workforce users to AWS and authenticate them from an identity provider (IdP) that’s external to AWS (such as Okta or Microsoft Entra), or you can create and authenticate the users on AWS.

Trusted identity propagation in IAM Identity Center lets AWS workforce identities use OAuth 2.0, helping applications that need to share who’s using them with AWS services. In OAuth, a client application and a resource service both trust the same authorization server. The client application gets an OAuth token for the user and sends it to the resource service. Because both services trust the OAuth server, the resource service can identify the user from the token and set permissions based on their identity.

AWS supports two OAuth patterns:

  • AWS applications authenticate directly with IAM Identity Center: Identity Center redirects authentication to your identity source, which generates OAuth tokens that the AWS managed application uses to access AWS services. This is the default pattern because the AWS services that support trusted identity propagation use Identity Center as their OAuth authorization server.
  • Third-party, non-AWS applications authenticate outside of AWS (typically to your IdP) and access AWS resources: During authentication, these third-party applications obtain an OAuth token from an OAuth authorization server outside of AWS. In this pattern, the AWS services aren’t connected to the same OAuth authorization server as the client application. To enable this pattern, AWS introduced a model called the trusted token issuer.

Trusted token issuer

When AWS services use IAM Identity Center as their authentication service, directory, and OAuth authorization server, the AWS services that use OAuth tokens require that Identity Center issues the tokens. However, most third-party applications federate with an external IdP and obtain OAuth tokens from an external authorization server. Although the identities in Identity Center and the external authorization server might be for the same person, the identities exist in separate domains, one in Identity Center, the other in the external authorization server. This is required to manage authorization of workforce identities with AWS services.

The trusted token issuer (TTI) feature provides a way to securely associate one identity from the external IdP with the other identity in IAM Identity Center.

When using third-party applications to access AWS services, there’s an external OAuth authorization server for the third-party application, and IAM Identity Center is the OAuth authorization server for AWS services; each has its own domain of users. The Identity Center TTI feature connects these two systems so that tokens from the external OAuth authorization server can be exchanged for tokens from Identity Center that AWS services can use to identify the user in the AWS domain of users. A TTI is the external OAuth authorization server that Identity Center trusts to provide tokens that third-party applications use to call AWS services, as shown in Figure 1.

Figure 1: Conceptual model using a trusted token issuer and token exchange

Figure 1: Conceptual model using a trusted token issuer and token exchange

How the trust model and token exchange work

There are two levels of trust involved with TTIs. First, the IAM Identity Center administrator must add the TTI, which makes it possible to exchange tokens. This involves connecting Identity Center to the Open ID Connect (OIDC) discovery URL of the external OAuth authorization server and defining an attribute-based mapping between the user from the external OAuth authorization server and a corresponding user in Identity Center. Second, the applications that exchange externally generated tokens must be configured to use the TTI. There are two models for how tokens are exchanged:

  • Managed AWS service-driven token exchange: A third-party application uses an AWS driver or API to access a managed AWS service, such as accessing Amazon Redshift by using Amazon Redshift drivers. This works only if the managed AWS service has been designed to accept and exchange tokens. The application passes the external token to the AWS service through an API call. The AWS service then makes a call to IAM Identity Center to exchange the external token for an Identity Center token. The service uses the Identity Center token to determine who the corresponding Identity Center user is and authorizes resource access based on that identity.
  • Third-party application-driven token exchange: A third-party application not managed by AWS exchanges the external token for an IAM Identity Center token before calling AWS services. This is different from the first model, where the application that exchanges the token is the managed AWS service. An example is a third-party application that uses Amazon Simple Storage Service (Amazon S3) Access Grants to access S3. In this model, the third-party application obtains a token from the external OAuth authorization server and then calls Identity Center to exchange the external token for an Identity Center token. The application can then use the Identity Center token to call AWS services that use Identity Center as their OAuth authorization server. In this case, the Identity Center administrator must register the third-party application and authorize it to exchange tokens from the TTI.

TTI trust details

When using a TTI, IAM Identity Center trusts that the TTI authenticated the user and authorized them to use the AWS service. This is expressed in an identity token or access token from the external OAuth authorization server (the TTI).

These are the requirements for the external OAuth authorization server (the TTI) and the token it creates:

  • The token must be a signed JSON Web Token (JWT). The JWT must contain a subject (sub) claim, an audience (aud) claim, an issuer (iss), a user attribute claim, and a JWT ID (JTI) claim.
    • The subject in the JWT is the authenticated user and the audience is a value that represents the AWS service that the application will use.
    • The audience claim value must match the value that is configured in the application that exchanges the token.
    • The issuer claim value must match the value configured in the issuer URL in the TTI.
    • There must be a claim in the token that specifies a user attribute that IAM Identity Center can use to find the corresponding user in the Identity Center directory.
    • The JWT token must contain the JWT ID claim. This claim is used to help prevent replay attacks. If a new token exchange is attempted after the initial exchange is complete, IAM Identity Center rejects the new exchange request.
  • The TTI must have an OIDC discovery URL that IAM Identity Center can use to obtain keys that it can use to verify the signature on JWTs created by your TTI. Identity Center appends the suffix /.well-known/openid-configuration to the provider URL that you configure to identify where to fetch the signature keys.

Note: Typically, the IdP that you use as your identity source for IAM Identity Center is your TTI. However, your TTI doesn’t have to be the IdP that Identity Center uses as an identity source. If the users from a TTI can be mapped to users in Identity Center, the tokens can be exchanged. You can have as many as 10 TTIs configured for a single Identity Center instance.

Details for applications that exchange tokens

Your OAuth authorization server service (the TTI) provides a way to authorize a user to access an AWS service. When a user signs in to the client application, the OAuth authorization server generates an ID token or an access token that contains the subject (the user) and an audience (the AWS services the user can access). When a third-party application accesses an AWS service, the audience must include an identifier of the AWS service. The third-party client application then passes this token to an AWS driver or an AWS service.

To use IAM Identity Center and exchange an external token from the TTI for an Identity Center token, you must configure the application that will exchange the token with Identity Center to use one or more of the TTIs. Additionally, as part of the configuration process, you specify the audience values that are expected to be used with the external OAuth token.

  • If the applications are managed AWS services, AWS performs most of the configuration process. For example, the Amazon Redshift administrator connects Amazon Redshift to IAM Identity Center, and then connects a specific Amazon Redshift cluster to Identity Center. The Amazon Redshift cluster exchanges the token and must be configured to do so, which is done through the Amazon Redshift administrative console or APIs and doesn’t require additional configuration.
  • If the applications are third-party and not managed by AWS, your IAM Identity Center administrator must register the application and configure it for token exchange. For example, suppose you create an application that obtains an OAuth token from Okta Universal Directory and calls S3 Access Grants. The Identity Center administrator must add this application as a customer managed application and must grant the application permissions to exchange tokens.

How to set up TTIs

To create new TTIs, open the IAM Identity Center console, choose Settings, and then choose Create trusted token issuer, as shown in Figure 2. In this section, I show an example of how to use the console to create a new TTI to exchange tokens with my Okta IdP, where I already created my OIDC application to use with my new IAM Identity Center application.

Figure 2: Configure the TTI in the IAM Identity Center console

Figure 2: Configure the TTI in the IAM Identity Center console

TTI uses the issuer URL to discover the OpenID configuration. Because I use Okta, I can verify that my IdP discovery URL is accessible at https://{my-okta-domain}.okta.com/.well-known/openid-configuration. I can also verify that the OpenID configuration URL responds with a JSON that contains the jwks_uri attribute, which contains a URL that lists the keys that are used by my IdP to sign the JWT tokens. Trusted token issuer requires that both URLs are publicly accessible.

I then configure the attributes I want to use to map the identity of the Okta user with the user in IAM Identity Center in the Map attributes section. I can get the attributes from an OIDC identity token issued by Okta:

{
    "sub": "00u22603n2TgCxTgs5d7",
    "email": "<masked>",
    "ver": 1,
    "iss": "https://<masked>.okta.com",
    "aud": "123456nqqVBTdtk7890",
    "iat": 1699550469,
    "exp": 1699554069,
    "jti": "ID.MojsBne1SlND7tCMtZPbpiei9p-goJsOmCiHkyEhUj8",
    "amr": [
        "pwd"
    ],
    "idp": "<masked>",
    "auth_time": 1699527801,
    "at_hash": "ZFteB9l4MXc9virpYaul9A"
}

I’m requesting a token with an additional email scope, because I want to use this attribute to match against the email of my IAM Identity Center users. In most cases, your Identity Center users are synchronized with your central identity provider by using automatic provisioning with the SCIM protocol. In this case, you can use the Identity Center external ID attribute to match with oid or sub attributes. The only requirement for TTI is that those attributes create a one-to-one mapping between the two IdPs.

Now that I have created my TTI, I can associate it with my IAM Identity Center applications. As explained previously, there are two use cases. For the managed AWS service-driven token exchange use case, use the service-specific interface to do so. For example, I can use my TTI with Amazon Redshift, as shown in Figure 3:

Figure 3: Configure the TTI with Amazon Redshift

Figure 3: Configure the TTI with Amazon Redshift

I selected Okta as the TTI to use for this integration, and I now need to configure the audclaim value that the application will use to accept the token. I can find it when creating the application from the IdP side–in this example, the value is 123456nqqVBTdtk7890, and I can obtain it by using the preceding example OIDC identity token.

I can also use the AWS Command Line Interface (AWS CLI) to configure the IAM Identity Center application with the appropriate application grants:

aws sso put-application-grant \
    --application-arn "<my-application-arn>" \
    --grant-type "urn:ietf:params:oauth:grant-type:jwt-bearer" \
    --grant '
    {
        "JwtBearer": { 
            "AuthorizedTokenIssuers": [
                {
                    "TrustedTokenIssuerArn": "<my-tti-arn>", 
                    "AuthorizedAudiences": [
                        "123456nqqVBTdtk7890"
                    ]
                 }
            ]
       }
    }'

Perform a token exchange

For AWS service-driven use cases, the token exchange between your IdP and IAM Identity Center is performed automatically by the service itself. For third-party application-driven token exchange, such as when building your own Identity Center application with S3 Access Grants, your application performs the token exchange by using the Identity Center OIDC API action CreateTokenWithIAM:

aws sso-oidc create-token-with-iam \  
    --client-id "<my-application-arn>" \ 
    --grant-type "urn:ietf:params:oauth:grant-type:jwt-bearer" \
    --assertion "<jwt-from-idp>"

This action is performed by an IAM principal, which then uses the result to interact with AWS services.

If successful, the result looks like the following:

{
    "accessToken": "<idc-access-token>",
    "tokenType": "Bearer",
    "expiresIn": 3600,
    "idToken": "<jwt-idc-identity-token>",
    "issuedTokenType": "urn:ietf:params:oauth:token-type:access_token",
    "scope": [
        "sts:identity_context",
        "openid",
        "aws"
    ]
}

The value of the scope attribute varies depending on the IAM Identity Center application that you’re interacting with, because it defines the permissions associated with the application.

You can also inspect the idToken attribute because it’s JWT-encoded:

{
    "aws:identity_store_id": "d-123456789",
    "sub": "93445892-f001-7078-8c38-7f2b978f686f",
    "aws:instance_account": "12345678912",
    "iss": "https://identitycenter.amazonaws.com/ssoins-69870e74abba8440",
    "sts:audit_context": "<sts-token>",
    "aws:identity_store_arn": "arn:aws:identitystore::12345678912:identitystore/d-996701d649",
    "aud": "20bSatbAF2kiR7lxX5Vdp2V1LWNlbnRyYWwtMQ",
    "aws:instance_arn": "arn:aws:sso:::instance/ssoins-69870e74abba8440",
    "aws:credential_id": "<masked>",
    "act": {
      "sub": "arn:aws:sso::12345678912:trustedTokenIssuer/ssoins-69870e74abba8440/c38448c2-e030-7092-0f0a-b594f83fcf82"
    },
    "aws:application_arn": "arn:aws:sso::12345678912:application/ssoins-69870e74abba8440/apl-0ed2bf0be396a325",
    "auth_time": "2023-11-10T08:00:08Z",
    "exp": 1699606808,
    "iat": 1699603208
  }

The token contains:

  • The AWS account and the IAM Identity Center instance and application that accepted the token exchange
  • The unique user ID of the user that was matched in IAM Identity Center (attribute sub)

AWS services can now use the STS Audit Context token (found in the attribute sts:audit_context) to create identity-aware sessions with the STS API. You can audit the API calls performed by the identity-aware sessions in AWS CloudTrail, by inspecting the attribute onBehalfOf within the field userIdentity. In this example, you can see an API call that was performed with an identity-aware session:

"userIdentity": {
    ...
    "onBehalfOf": {
        "userId": "93445892-f001-7078-8c38-7f2b978f686f",
        "identityStoreArn": "arn:aws:identitystore::425341151473:identitystore/d-996701d649"
    }
}

You can thus quickly filter actions that an AWS principal performs on behalf of your IAM Identity Center user.

Troubleshooting TTI

You can troubleshoot token exchange errors by verifying that:

  • The OpenID discovery URL is publicly accessible.
  • The OpenID discovery URL response conforms with the OpenID standard.
  • The OpenID keys URL referenced in the discovery response is publicly accessible.
  • The issuer URL that you configure in the TTI exactly matches the value of the iss scope that your IdP returns.
  • The user attribute that you configure in the TTI exists in the JWT that your IdP returns.
  • The user attribute value that you configure in the TTI matches exactly one existing IAM Identity Center user on the target attribute.
  • The aud scope exists in the token returned from your IdP and exactly matches what is configured in the requested IAM Identity Center application.
  • The jti claim exists in the token returned from your IdP.
  • If you use an IAM Identity Center application that requires user or group assignments, the matched Identity Center user is already assigned to the application or belongs to a group assigned to the application.

Note: When an IAM Identity Center application doesn’t require user or group assignments, the token exchange will succeed if the preceding conditions are met. This configuration implies that the connected AWS service requires additional security assignments. For example, Amazon Redshift administrators need to configure access to the data within Amazon Redshift. The token exchange doesn’t grant implicit access to the AWS services.

Conclusion

In this blog post, we introduced the trust token issuer feature of IAM Identity Center and what it offers, how it’s configured, and how you can use it to integrate your IdP with AWS services. You learned how to use TTI with AWS-managed applications and third-party applications by configuring the appropriate parameters. You also learned how to troubleshoot token-exchange issues and audit access through CloudTrail.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS IAM Identity Center re:Post or contact AWS Support.

Roberto Migli

Roberto Migli

Roberto is a Principal Solutions Architect at AWS. Roberto supports global financial services customers, focusing on security and identity and access management. In his free time, he enjoys building electronic gadgets, learning about space, and spending time with his family.

Ron Cully

Ron Cully

Ron is a Principal Product Manager at AWS where he leads feature and roadmap planning for workforce identity products at AWS. Ron has over 20 years of industry experience in product and program management in networking and directory related products. He is passionate about delivering secure, reliable solutions that help make it simple for customers to migrate directory-aware applications and workloads to the cloud.

Rafael Koike

Rafael Koike

Rafael is a Principal Solutions Architect supporting enterprise customers in the Southeast and is a Storage SME. Rafael has a passion to build, and his expertise in security, storage, networking, and application development has been instrumental in helping customers move to the cloud quickly and securely.

Just about every Windows and Linux device vulnerable to new LogoFAIL firmware attack (ars technica)

Post Syndicated from corbet original https://lwn.net/Articles/953985/

This
ars technica article
describes how secure-boot firmware on a huge range
of systems can be subverted with a malicious image file:

As its name suggests, LogoFAIL involves logos, specifically those
of the hardware seller that are displayed on the device screen
early in the boot process, while the UEFI is still running. Image
parsers in UEFIs from all three major IBVs [independent BIOS
vendors] are riddled with roughly a dozen critical vulnerabilities
that have gone unnoticed until now. By replacing the legitimate
logo images with identical-looking ones that have been specially
crafted to exploit these bugs, LogoFAIL makes it possible to
execute malicious code at the most sensitive stage of the boot
process.

Security updates for Thursday

Post Syndicated from jake original https://lwn.net/Articles/953977/

Security updates have been issued by Debian (tzdata), Fedora (gmailctl), Oracle (kernel), Red Hat (linux-firmware, postgresql:12, postgresql:13, and squid:4), SUSE (cdi-apiserver-container, cdi-cloner-container, cdi- controller-container, cdi-importer-container, cdi-operator-container, cdi- uploadproxy-container, cdi-uploadserver-container, cont, frr, libtorrent-rasterbar, qbittorrent, openssl-3, openvswitch, openvswitch3, and suse-build-key), and Ubuntu (bluez, curl, linux, linux-aws, linux-azure, linux-laptop, linux-lowlatency,
linux-oem-6.5, linux-oracle, linux-raspi, linux-starfive, linux-gcp, open-vm-tools, postgresql-12, postgresql-14, postgresql-15, and python-cryptography).

ML Ops Platform at Cloudflare

Post Syndicated from Keith Adler original http://blog.cloudflare.com/mlops/


ML Ops Platform at Cloudflare

We’ve been relying on ML and AI for our core services like Web Application Firewall (WAF) since the early days of Cloudflare. Through this journey, we’ve learned many lessons about running AI deployments at scale, and all the tooling and processes necessary. We recently launched Workers AI to help abstract a lot of that away for inference, giving developers an easy way to leverage powerful models with just a few lines of code. In this post, we’re going to explore some of the lessons we’ve learned on the other side of the ML equation: training.

Cloudflare has extensive experience training models and using them to improve our products. A constantly-evolving ML model drives the WAF attack score that helps protect our customers from malicious payloads. Another evolving model power bot management product to catch and prevent bot attacks on our customers. Our customer support is augmented by data science. We build machine learning to identify threats with our global network. To top it all off, Cloudflare is delivering machine learning at unprecedented scale across our network.

Each of these products, along with many others, has elevated ML models — including experimentation, training, and deployment — to a crucial position within Cloudflare. To help our team continue to innovate efficiently, our MLOps effort has collaborated with Cloudflare’s data scientists to implement the following best practices.

Notebooks

Given a use case and data, the first step for many Data Scientist/AI Scientists is to set up an environment for exploring data, feature engineering, and model experiments. Jupyter Notebooks are a common tool to satisfy these requirements. These environments provide an easy remote Python environment that can be run in the browser or connected to a local code editor. To make notebooks scalable and open to collaboration, we deploy JupyterHub on Kubernetes. With JupyterHub, we can manage resources for teams of Data Scientists and ensure they get a suitable development environment. Each team can tailor their environment by pre-installing libraries and configuring user environments to meet the specific needs, or even individual projects.

This notebook space is always evolving as well. Open source projects include further features, such as:

  • nbdev – a Python package to improve the notebook experience
  • Kubeflow – the kubernetes native CNCF project for machine learning
  • depoyKF – ML Platforms on any Kubernetes cluster, with centralized configs, in-place upgrades, and support for leading ML & Data tools like Kubeflow, Airflow, and MLflow

GitOps

Our goal is to provide an easy-to-use platform for Data Scientists and AI Scientists to develop and test machine learning models quickly. Hence, we are adopting GitOps as our continuous delivery strategy and infrastructure management on MLOps Platform in Cloudflare. GitOps is a software development methodology that leverages Git, a distributed version control system, as the single source of truth for defining and managing infrastructure, application configurations, and deployment processes. It aims to automate and streamline the deployment and management of applications and infrastructure in a declarative and auditable manner. GitOps aligns well with the principles of automation and collaboration, which are crucial for machine learning (ML) workflows. GitOps leverages Git repositories as the source of truth for declarative infrastructure and application code.

A data scientist needs to define the desired state of infrastructure and applications. This usually takes a lot of custom work, but with ArgoCD and model templates, all it takes is a simple pull request to add new applications. Helm charts and Kustomize are both supported to allow for configuration for different environments and jobs. With ArgoCD, declarative GitOps will then start the Continuous Delivery process. ArgoCD will continuously check the desired state of the infrastructure and applications to ensure that they are synched with the Git repository.

In the future, we plan to migrate our platform (including Jupyterhub) to Kubeflow, a machine learning workflow platform on Kubernetes that simplifies the development, deployment, and management of notebooks and end-to-end pipelines. This is best deployed using a new project, deployKF, which allows for distributed configuration management across multiple components available with Kubeflow, and others that extend beyond what is offered within Kubeflow.

Templates

Starting a project with the right tools and structure can be the difference between success and stagnation. Within Cloudflare, we’ve curated an array of model templates, which are production ready data science repositories with an example model. These model templates are deployed through production to continually ensure they are a stable foundation for future projects. To start a new project, all it takes is one Makefile command to build a new CICD project in the git project of the users’ choosing. These template utility packages are identical to those used in our Jupyter Notebooks and connections to R2 / S3 / GCS buckets, D1 / Postgres / Bigquery / Clickhouse databases. Data scientists can use these templates to instantly kickstart new projects with confidence. These templates are not yet publicly available, but our team plans to open source them in the future.

1. Training Template
Our model training template provides a solid foundation to build any model. This is configured to help extract, transform, and load data (ETL) from any data source. The template includes helper functions for feature engineering, tracking experiments with model metadata, and choose orchestration through a Directed Acyclic Graph (DAG) to productionalize the model pipeline. Each orchestration can be configured to use Airflow or Argo Workflows.

2. Batch Inference Template
Batch and micro-batch inference can make a significant impact on processing efficiency. Our batch inference model template to schedule models for consistent results, and can be configured to use Airflow or Argo Workflows.

3. Stream Inference Template
This template makes it easy for our team to deploy real-time inference. Tailored for Kubernetes as a microservice using FastAPI, this template allows our team to run inference using familiar Python in a container. This microservice already has built-in REST interactive documentation with Swagger and integration with Cloudflare Access authentication tokens in terraform.

4. Explainability Template
Our model template for explainability spins up dashboards to illuminate the model type and experiments. It is important to be able to understand key values such as a time window F1 score, the drift of features and data over time. Tools like Streamlit and Bokeh help to make this possible.

Orchestration

Organizing data science into a consistent pipeline involves a lot of data and several model versions. Enter Directed Acyclic Graphs (DAGs), a robust flow chart orchestration paradigm that weaves together the steps from data to model, and model to inference. There are many unique approaches to running DAG pipelines, but we have found that data science teams’ preference comes first. Each team has different approaches based on their use cases and experience.

Apache Airflow – The Standard DAG Composer
Apache Airflow is the standard as a DAG (Directed Acyclic Graphs)-based orchestration approach. With a vast community and extensive plugin support, Airflow excels in handling diverse workflows. The flexibility to integrate with a multitude of systems and a web-based UI for task monitoring make it a popular choice for orchestrating complex sequences of tasks. Airflow can be used to run any data or machine learning workflow.

Argo Workflows – Kubernetes-native Brilliance
Built for Kubernetes, Argo Workflows embraces the container ecosystem for orchestrating workflows. It boasts an intuitive YAML-based workflow definition and excels in running microservices-based workflows. The integration with Kubernetes enables scalability, reliability, and native container support, making it an excellent fit for organizations deeply rooted in the Kubernetes ecosystem. Argo Workflows can also be used to run any data or machine learning workflow.

Kubeflow Pipelines – A Platform for Workflows
Kubeflow Pipelines is a more specific approach tailored for orchestrating machine learning workflows. “KFP” aims to address the unique demands of data preparation, model training, and deployment in the ML landscape. As an integrated component of the Kubeflow ecosystem, it streamlines ML workflows with a focus on collaboration, reusability, and versioning. Its compatibility with Kubernetes ensures seamless integration and efficient orchestration.

Temporal – The Stateful Workflow Enabler
Temporal takes a stance by emphasizing the orchestration of long-running, stateful workflows. This relative newcomer shines in managing resilient, event-driven applications, preserving workflow state and enabling efficient recovery from failures. The unique selling point lies in its ability to manage complex, stateful workflows, providing a durable and fault-tolerant orchestration solution.

In the orchestration landscape, the choice ultimately boils down to the team and use case. These are all open source projects, so the only limitation is support for different styles of work, which we find is worth the investment.

Hardware

Achieving optimal performance necessitates an understanding of workloads and the underlying use cases in order to provide teams with effective hardware. The goal is to enable data scientists and strike a balance between enablement and utilization. Each workload is different, and it is important to fine tune each use case for the capabilities of GPUs and CPUs to find the perfect tool for the job.  For core datacenter workloads and edge inference, GPUs have leveled up the speed and efficiency that is core to our business. With observability and metrics consumed by Prometheus, metrics enable us to track orchestration to be optimized for performance, maximize hardware utilization, and operate within a Kubernetes-native experience.

Adoption

Adoption is often one of the most challenging steps in the MLops journey. Before jumping into building, it is important to understand the different teams and their approach to data science. At Cloudflare, this process began years ago, when each of the teams started their own machine learning solutions separately. As these solutions evolved, we ran into the common challenge of working across the company to prevent work from becoming isolated from other teams. In addition, there were other teams that had potential for machine learning but did not have data science expertise within their team. This presented an opportunity for MLops to step in — both to help streamline and standardize the ML processes being employed by each team, and to introduce potential new projects to data science teams to start the ideation and discovery process.

When able, we have found the most success when we can help get projects started and shape the pipelines for success. Providing components for shared use such as notebooks, orchestration, data versioning (DVC), feature engineering (Feast), and model versioning (MLflow) allow for teams to collaborate directly.

Looking forward

There is no doubt that data science is evolving our business and the businesses of our customers. We improve our own products with models, and have built AI infrastructure that can help us secure applications and applications built with AI. We can leverage the power of our network to deliver AI for us and our customers. We have partnered with machine learning giants to make it easier for the data science community to deliver real value from data.

The call to action is this: join the Cloudflare community in bringing modern software practices and tools to data science. Be on the lookout for more data science from Cloudflare. Help us securely leverage data to help build a better Internet.

Impressions from Coolest Projects South Africa 2023

Post Syndicated from Rujeko Moyo original https://www.raspberrypi.org/blog/coolest-projects-south-africa-2023/

The day after the successful meetup with our Global Clubs Partner organisations based in Africa, our team and some of our partners enjoyed participating in the Coolest Projects South Africa 2023 event to meet young tech creators and help out as project judges. Here are some of our impressions.

Our team and partners at Coolest Projects South Africa 2023.

A day of Coolest Projects

This is the fourth year of a partner-run, regional version of Coolest Projects — our world-leading showcase for young tech creators — taking place in South Africa, led by David Campey. David is Director of Coder LevelUp, one of our Global Clubs Partners growing and supporting a network of CoderDojos and Code Clubs in the country, and involved in the CoderDojo movement for a whole decade.

A waterbottle with a Coolest Projects South Africa sticker.

There was a buzz of anticipation and excitement at the Cape Town Science Centre as young coders from age 5 to 18 and various backgrounds gathered on this sunny Saturday morning to showcase their coding creations and inventions at Coolest Projects South Africa. From fun games and animations on Scratch, to cool websites created with HTML and CSS, to fantastic Python-based hardware solutions to real-world challenges — every young creator brought along a project they’d created to proudly showcase and celebrate.

Luhle’s language-inspired coding project

While chatting with the creators and discovering what had motivated their projects, we met up with 11-year-old Luhle, who was delighted to take us through the ‘Moon conversation’ animation she had coded in Scratch.

A young tech creator with her Scratch project at Coolest Projects South Africa 2023.
11-year-old Luhle proudly showcases her ‘Moon conversation’ Scratch animation at the Coolest Projects South Africa 2023 event.

The animation involved a Spanish conversation between two people who journeyed to the moon and back. Luhle had created her animation because of her love for languages and in response to a challenge posed to her class by her teacher: to learn 5 languages. While her mother tongue is isiXhosa, she is confident in English, is learning Afrikaans, has started teaching herself Spanish, and would love to learn Korean.

Kayden’s innovative hardware creation

We also met with 16-year-old Kayden, who showcased a project he’d made to address a real-world challenge. He told us he had always struggled to concentrate in class — a challenge that many young people face — and he wanted to build an alternative solution to the established medications. Using vibration sensors and two microcontrollers, he created a digital device to prompt users when they are no longer paying attention in class. With his friend Carl, he successfully tested the device on a meaningful sample of Grade 1–3 learners (ages 7–9).

A young tech creator with his hardware project at Coolest Projects South Africa 2023.
16-year-old Kayden listens intently as one of the Coolest Projects judges, Akwabi Paul from Kenya, commends his invention and advises him on next steps. Listening in are two other judges, Solomon from The Gambia and Sylvester from Malawi.

Kayden is now developing this low-cost innovative solution to include a heart rate monitor to help to detect when a user loses focus, and he wants this to be a solution that’s widely accessible and affordable for all South African children. One of the judges, our partner Akwabi Paul from Tech Kidz Africa in Kenya, was greatly impressed and motivated by Kayden’s work, and took time to advise Kayden on the next steps to turn his invention into a commercial product.

The coding club at CBC St Johns Parklands

During the event we also met members of Mrs Hill’s coding club and learnt about Mrs Hill’s experience of nurturing a love and interest for coding and robotics at CBC St Johns Parklands in Cape Town.

Since 2020, Mrs Hills has been providing coding lessons to all school classes — learners aged 6 to 12 years — as well as an after-school coding and robotics club. She approaches her lessons by introducing and demonstrating coding skills and then presenting her learners with a problem to solve collaboratively. In her words, ‘Learners find more interest in learning practically.’

That’s why Coolest Projects is the perfect fit for her and her young people. 4 of her club members took part in Coolest Projects South Africa 2022. This year, she was proud to enter 11 participants, 3 of whom were chosen as judges’ favourites.

Here’s to the young creators and more Coolest Projects events

After the showcasing and judging, the Coolest Projects South Africa event culminated in a hearty celebration of all that the young tech creators had presented. David Campey’s passion for nurturing coding literacy, digital making skills, and innovative thinking among learners from different walks of life made the whole day a truly enjoyable, inclusive event for the young creators.

Coolest Projects logo.

It was inspiring, no doubt, for our other African partners who participated as judges and are now keen to host Coolest Projects events back in their home countries.  

Get involved in Coolest Projects

If you and your young people based anywhere on the globe feel inspired to showcase digital tech creations, you can get involved in our Coolest Projects 2024 online showcase! It’s free and open to any young tech creator up to age 18.

Sign up to the Coolest Projects newsletter to be the first to hear all updates, for example when showcase registration opens on 14 February.

The post Impressions from Coolest Projects South Africa 2023 appeared first on Raspberry Pi Foundation.

Нагледно какво ще се строи в една част на София

Post Syndicated from Боян Юруков original https://yurukov.net/blog/2023/build-map-sofia/

Застрояването на един район се решава – често в разрез на закона – много преди да видим багерите да изриват детски площадки и градинки или да се вдигат кранове на метри от прозорците. С проекта GovAlert.eu се опитах да помогна с информираността, но тези решения се взимат много преди визите за проектиране и разрешенията за строеж. Истината е, че са скрити в градоустройствени заповеди, протоколи на общинските съвети, продажби на общинска земя, а нерядко – незаконно вписване на такава като частна и фалшифициране на становища и съгласуване. Контрол няма – къде умишлено, къде от некадърност.

Ситуацията в София класически пример за този начин на работа. Едно положително нещо обаче е, че в сайтът на НАГ може да се видят скиците от въпросните градоустройствени планове. Днес между другото седнах да си играя да въведа сградите в и около квартал Дианабад, които не са започнати или довършени заедно с височината им. За някои от тях са издадени визи, за други – разрешителни за строеж. Повечето обаче са одобрени от НАГ за строеж и е само формалност да получат разрешително. Някои одобрените сгради – като тези на 5-тата снимка – се намират на все още общински имоти, което показва, че вече се подготвя продажбата им.

Направих това като експеримент. Реших, че този начин на визуализация би бил полезен на живущите в квартала да си представят какво ги очаква и как ще се промени градската среда в близките години. В червено са въпросните планирани сгради. Останалите се виждат триизмерно и макар очертанията им да не са точни, достатъчно близо са, за да дадат представа. Списъкът не е изчерпателен като може да има няколко по-малки сгради.

Може да видите интерактивна версия тук:

На снимките виждате няколко ключови точки в квартала.

Ето тук ще намерите (почти) всички градоустройствени заповеди от района за последните 10 г. Някои от решенията за застрояване долу са от преди това. Други са укрити от сайта на НАГ.

The post Нагледно какво ще се строи в една част на София first appeared on Блогът на Юруков.

Уча се да уча

Post Syndicated from original https://www.toest.bg/ucha-se-da-ucha/

Уча се да уча

През XXI век най-нормалният въпрос, който може да си зададем, е „Защо трябва да уча каквото и да било, като мога да го намеря за секунди в Google?“ Дали не трябва да обърнем цялата си концепция за ученето и оттук нататък да учим само практически неща, които са ни нужни за живота, работата и хобитата ни, защото цялата останала информация и без това я има на един клик разстояние?

Разбира се, че няма да е адекватно да го направим, но тръгвайки в тази посока, отваряме пътя за въпроса не какво да учим, а как да учим предвид факта, че разполагаме с целия свят в телефона си.

Компютърът в главата ни

Ако извървим пътя на всяка информация, която достига до нас, не само че ще разберем как функционира ученето, но и съвсем логично след това сами ще се сетим как може да бъде подобрен процесът при нас. Всяка секунда до мозъка ни достигат тонове информация. И огромна част изчезва от него след съвсем кратко време. В краткосрочната ни памет остава само това, което може да се „закачи“ за вече съществуващо знание. Но дори и закачилите се късчета данни не оцеляват дълго – след около 20 минути повечето от тях също за забравени. От новопостъпилите факти след час ще помним онова, което е отишло в дългосрочната памет. Този процес можем да направляваме сами при активно учене, стига да знаем как.

Когато ми се налага да обясня на дете какво е ученето, винаги му казвам да си представи невроните в мозъка си като безброй приятели (всъщност са милиони), които живеят там. Те се свързват различно – подават си ръце, пращат си писма, опъват простор един между друг, строят си обща улица и какво ли още не. По всички тези връзки протича нашето знание, защото някакво наше преживяване или нова информация са ги накарали да ги изградят. 

Или иначе казано, сетивните стимули, които получаваме чрез гледане, слушане или четене, стартират дейността на синапсите в мозъка. Информацията се предава от една на друга нервна клетка чрез точките им на свързване. Броят на активните синапси и нервни клетки е определящ за успеха на ученето – колкото повече са те, толкова по-дълбоко в мозъка ни се намира търсената информация. Ако повтаряме чрез преразказване или си припомняме чрез препрочитане дадена информация, ние ѝ помагаме да остане по-дълго и ефективно в паметта ни, защото реактивираме създадените връзки между нервните клетки.

Класическото учене не е складиране на знания в отделни папки и на точно определени места в мозъка ни. По-скоро мозъкът ни разполага с временен склад, в който отива цялата нова информация и който играе ролята на разпределителна гара за пренасочване на информацията. Този склад се нарича хипокампус. Когато си спомняме нещо, не отваряме четвъртата папка от трети рафт на шестата етажерка, а активираме всички нервни клетки и техните взаимовръзки, участвали в запомнянето на точно тази информация. Запомнената от нас информация е плетеницата, която хипокампусът предава на мозъчната кора.

Само че хипокампусът не пуска всичко наведнъж, а предава малко по малко най-важното от деня към главния мозък, предимно през нощта. Докато спим, хипокампусът активира важните според него пътища между нервните клетки, които скрепяват новите взаимовръзки в главния мозък. Това от своя страна е основата на дългосрочната памет. Ето ви научно обосновано оправдание да си полегнете след час-два учене, защото иначе току-що наученото няма да намери пътя до главния мозък.

Дотук е ясно как да запомним нещо наизуст – повтаряме го, докато не стане автоматично. 

Но това означава ли задължително, че го разбираме? 

В училище и в университета пътят на ученето е начертан – имаме учебници, конспекти, планове, класически учебни стаи с учител отпред. Нищо чудно, че все още много учители смятат повторението за единствения начин за запаметяване на информация – „Прочетете урока пет пъти“ или „Напишете всяка непозната дума по три реда“. През стандартното устно или писмено изпитване се преминава изключително лесно, ако просто се наизустят уроците. 

Но не трябва ли знанието да добие още едно изражение – да се превърне в умение? Към това всеки сам намира път. Умението да учиш може да бъде придобито. Дори с не толкова големи усилия или инвестиция на време. И е крайно нужно, защото учениците в голяма степен са изоставени на собствения си усет как да боравят с количеството информация, което ги залива ежедневно в учебния процес. 

Затова и ученето не трябва да е процес на запаметяване на факти, които по-късно мозъкът ни ще „изплюе“. Ученето трябва да е разбиране, изграждане на мисловни модели, с помощта на които да сме в състояние да намерим сами решение. Наученото трябва да ни направи способни да приложим знанието извън рамката му и да го надградим, да ни направи критично мислещи, самостоятелни, комуникативни и търсещи предизвикателството. Пасивното консумиране на предварително зададена информация не може да постигне това. 

Пример как може да се разчупи сухото цитиране на факти, е да се обърне поредността и новият материал да се представи под формата на проблем, който трябва да бъде разрешен. Доказано е, че придобиваме знания най-добре, когато сме изправени пред препятствие, което не можем да преодолеем с наученото досега. Правилният отговор трябва да бъде оставен за накрая. Тогава ефективността (например колко нов материал е взет) ще отстъпи място на любопитството. Крайният резултат ще е всеобхватен: ще сме установили къде сме, ще сме добавили нови знания, но и ще имаме похват, който ще можем да приложим в друга сфера при различни обстоятелства.

Да раздвижим ученето

Учим навсякъде и по всяко време. Учебният процес в учебните заведения е структуриран така, че не предполага свободно да изберем работното си място. В най-добрия случай можем да изберем редицата, в която да седнем. 

На нас всъщност не са ни необходими задължително стол и бюро, за да разбере мозъкът ни, че е време да работи. Напротив, особено при децата е нужно движение и разнообразие вместо специално работно място. Затова когато учат вкъщи, ги оставете да седнат под масата, защо не и във ваната, да стоят прави и да пишат със специален маркер по прозореца, да скачат на батута, да лежат на земята. 

Друга особеност при децата е, че те не обичат да са сами, докато учат. Затова и толкова ги привлича да са около вас във всекидневната или в кухнята с всичките си тетрадки и учебници. Отделете си време, в което сядате заедно да работите: те – по домашните си, а вие – по нещо, което позволява детето да ви прекъсва. 

Работно време

Не е задължително количеството време, което изразходваме за научаването на даден обем информация – например когато учим за изпит, да отговаря на качеството на наученото. Може да прекараме седмици наред в усърдно учене, а накрая оценката да не отразява по никакъв начин усилията ни. В този случай е хубаво да обърнем внимание на целите си – дългосрочни и краткосрочни. „Да науча немски език“ звучи чудесно като желание, но не и като план. Разбиването на целта на по-малки части с междинни успехи, както и предварителна оценка на настоящото ниво и реалистична представа какво и за колко време може да бъде постигнато, биха улеснили неимоверно процеса.

Към работното време спадат и почивките. Не само че не трябва да ги делим от ученето, а следва да обърнем внимание как точно се почива. Не бива нито да се прекалява, нито да ги пестим, защото и в двата случая понижаваме качеството на наученото.

Още дълго може да се изброяват факторите, които влияят върху качеството на ученето. Има и безброй учебни техники, типове учене и хакове за тях, на които е важно да се обърне внимание и които са обект на работа от страна на психолози, учебни терапевти и учители, прозрели нуждата от промяна в учебния процес.

Може би бих поставила в основата им теорията за фиксирания и развиващ се майндсет, или на български език – умствена нагласа.

Ако сте родители на дете в ученическа възраст, сигурно ви е втръснало да чувате от него: „НЕ МОГААА!“

Просто добавете: „Все още.“

Типичен пример за фиксиран майндсет е някой, който не желае дори да се пробва с дадена задача, защото „аз и без това не мога“. Или пък някой, който не може да борави с поражения. Или най-общо – човек, който смята, че потенциалът му е предопределен от способностите му и интелигентността му, които са толкова, колкото са. 

Умствената ни нагласа, за наше щастие, подлежи на промяна. Дали с помощта на учител или родител, или пък със собствени сили, ние можем да развиваме нагласата си постоянно и така да преминем към развиващ се майндсет. Да станем хора, които се учат от грешките си. Които обичат предизвикателствата не само заради състезателния им характер. Които знаят какво да правят с обратната връзка дори когато тя не е много позитивна. И които са наясно, че успехът им зависи от собствените им усилия и нагласи.

Развиващата се умствена нагласа е много повече от която и да било мотивация, защото тя става стил на живот и се прехвърля върху всички области на живота ни.

Мозъкът ни е като мускул. Можем да го тренираме така, че дори малко да боли – като при мускулната треска. Редовното правилно трениране води до натрупване на мускулна маса, или в случая с мозъка ни – до повече взаимовръзки. Не може и не трябва да отписваме традиционното общо образование, колкото и остаряло и неадекватно да се струва на някои от нас. Но задължително трябва да го преустроим – със силна връзка с настоящата реалност и напредък. Познаването на фактите трябва да даде по-голямо предимство на познаването на процесите и взаимовръзките между тях. Защото за суровите факти винаги ще имаме интернет, но за навременното разпознаване и адекватното планиране на процесите, произтичащи от тях, не можем да разчитаме на която и да било търсачка. А ключът към ученето остава вниманието – но вече не само към детайла, а и един към друг.


Светът се променя с бясна скорост. Професиите, в които ще се развиват поколенията, започващи днес образователния си път, все още не са измислени. Подготвена ли е нашата образователна система, за да отговори на тези предизвикателства? Какво може и трябва да се промени? А как?

Веднъж месечно в рубриката „Възможното образование“ ще говорим за промяната – такава, каквато искаме да я видим, за добрите примери и за посоките, в които може би е добре да обърне поглед българската образователна система.

The collective thoughts of the interwebz