Научни новини: COVID-19, CRISPR, AI и Космос

Post Syndicated from Михаил Ангелов original https://toest.bg/science-news-covid19-crispr-ai-cosmos/

Веднъж-дваж месечно Михаил Ангелов – биолог, агроном и любим нърд от нашия екип, ще ни представя най-интересните скорошни новини от различни сфери на науката и ще обяснява защо тези постижения са толкова значими за света и човечеството. Или най-малкото – любопитни и забавни.

COVID-19

Въпреки че най-разпространените в момента варианти на SARS-CoV-2 предизвикват сравнително леки симптоми (особено при ваксинирани), продължават да се натрупват данни за дълготрайни и непредвидени до момента последствия.

В скорошно изследване на екип от Каталуния е констатирано, че една година след преболедуване в над една трета от неваксинираните не може да се засекат антитела, особено в групата на над 60-годишните и на пушачите. Резултатите от проучването потвърждават и че хибридният имунитет е по-силен и по-дълготраен от постинфекциозния, поради което учените препоръчват поставянето на бустерни дози и на вече преболедували. В комбинация с наблюдението, че след поставени две ваксини поне две седмици преди заразяване с вируса вероятността от дълъг ковид намалява почти наполовина, тези данни показват ясната полза от ваксинацията.

В друго изследване е установено, че дори леките случаи на заболяването по време на бременност увреждат способността за имунен отговор на плацентата при последващи инфекции. Според учените това може да се окаже само върхът на айсберга и те смятат, че тези деца ще трябва да се наблюдават в по-продължителен период, за да се проследи здравето им. Препоръката на изследователите към бременните е да се възползват от достъпните за тях ваксини, да носят маска и по възможност да ограничат контактите си, особено с неваксинирани.

Добрата новина е, че при анкетиране на бременни и кърмещи жени съобщените нежелани реакции към трета доза ваксина (бустер) 24 часа след поставянето ѝ са били рядкост. Въпреки че локалните симптоми (болка и зачервяване) са били завишени в сравнение с контролната група, едва 2,4% от бременните са съобщили за реакции, свързани с бременността (основно контракции при жени в трети триместър), а само 3,5% от кърмещите са отбелязали намаляване в количеството кърма.

Нови приложения на CRISPR

През десетте години след публикуването на статията, описваща възможностите на системата CRISPR/Cas9, учените постоянно откриват нови начини за нейното приложение.

Сливането на хромозоми е процес, който се среща в природата – понякога като следствие на еволюцията, друг път в резултат на злокачествени процеси, които водят до редица здравословни проблеми. В статия, публикувана в списание Science, е описано първото успешно подобно сливане в лабораторни условия при бозайници. С помощта на CRISPR/Cas9 учените са слели хромозоми на мишки, които впоследствие могат успешно да се възпроизвеждат с нередактирани партньори. Въпреки някои отчетени проблеми – по-малобройно потомство, симптоми на тревожност и общи поведенчески промени, тази публикация отваря вратите за много бъдещи изследвания. С помощта на този метод могат да бъдат проучени по-основни теми – като еволюцията на хромозомите и разпределението им по време на мейоза (деленето на половите клетки), но и по-практично ориентирани – например развитието на някои видове тумори.

Наборът с инструменти на генетичните инженери се увеличава и с други нови открития. За разлика от Cas9, който модифицира ДНК, Cas13 има способността да реже едноверижна РНК. Както при Cas9, и при него са възможни нежелани редакции на непредвидени места. Поради това учените са създали система за белязване на такива редакции и над 200 модифицирани варианта на протеина и така са успели да открият варианти, които запазват ефективността си, без да променят РНК на непредвидени места. Тъй като РНК молекулите се задържат за сравнително кратко време в клетките и не се интегрират в генома, тази технология дава възможност за временни промени в клетките – подход, който се счита за по-безопасен.

Освен да срязва РНК молекулите на точно специфично място, Cas13 има способността да прави това и с близки РНК молекули. Като се използва тази негова възможност, е създаден бърз и прецизен тест за SARS-CoV-2, който може да се извършва и извън лабораторни условия само със секрет от носа и гърлото на пациента. Тъй като във формата, в която се среща в природата, протеинът не е с достатъчна чувствителност за приложение в диагностиката, екипът го е модифицирал, като е подобрил способността му да се свързва с РНК. Така е постигната възможността за детекция на 12 вирусни копия в един микролитър. Това го прави много по-чувствителен и по-прецизен от познатите бързи антигенни тестове, тъй като взаимодейства директно с генетичния материал на вируса, а не с белтъците от обвивката му. Поради гъвкавостта на системата тестът може да се приложи и за други вирусни заболявания – като зика, денге, ебола.

Изкуственият интелект в полза на медицината

Изкуственият интелект става широкодостъпен и интерактивен – най-вероятно немалка част от интернет потребителите са се забавлявали с изображенията, генерирани от Craiyon, а преди няколко години AlphaGo стана първата компютърна програма, победила човек на играта Go. Поради това не е изненада, че сходни похвати навлизат все повече и в други полета на науката.

Употребата на втора версия на AlphaFold (изкуствен интелект за предсказване на пространствената структура на протеини) става изключително популярна след публикуването и отварянето на кода му през юли 2021 г. Според статистика на списание Nature само два месеца след публикацията вече се подават по над 75 статии на месец. Програмата е създадена от екип на DeepMind, подразделение на Alphabet (компанията майка на Google).

Сега учените правят следващата стъпка с използването на изкуствен интелект за дизайн на нови протеини, като резултатите им са публикувани в три статии в списание Science. Първата е за създаване на нова структура с помощта на един от двата подхода – „халюцинация“, който работи на сходен принцип като при създаването на изображения от зададен текст в Crayion, или „запълване“, при който се използват похватите на автоматичното допълване (autofill) при текстово търсене по ключова дума. Така може да се зададат желани региони в протеина, които да имат конкретна активност или структура. Следва генериране на аминокиселинната последователност с помощта на новосъздаден инструмент (ProteinMPNN), който значително ускорява тази стъпка. Финално последователностите се анализират с помощта на AlphaFold, за да се потвърди, че ще бъде получена желаната пространствена структура. Някои от тези протеини са изпитани и в лабораторни условия, където е потвърдено, че кристалната им структура отговаря на структурата, създадена от алгоритмите.

Защо това е революционна технология ли? Защото с нея може да се създадат напълно нови протеини без аналози в природата. Едно от приложенията, посочени от авторите на изследването, е проектирането и изработването на биоматериали и все по-сложни компоненти за наномашини – малки „роботи“, които могат например да навлизат в човешките артерии и да разбиват плака или пък да внасят лекарства само в определени тъкани. Но това е само малка част от хоризонтите, които методът разкрива – приложението му може да намери място в най-различни биотехнологични процеси, като създаване на ваксини и различни терапевтични средства, устойчиво производство на биомаса и биогорива и др.

Изкуственият интелект се използва успешно и при създаването на нови медикаменти. Ключова стъпка в този процес е откриването на подходящо взаимодействие между целевата молекула в тялото на пациента и тази на лекарството, но поради големия брой потенциални взаимодействия задачата не е лесна. Прилагането на машинно самообучение помага значително.

Нов модел (AttentionSiteDTI), базиран на т.нар. обработка на естествен език (NLP), е описан от авторите му като голяма стъпка напред. Основните предимства на този подход са улесненото въвеждане на данни, който позволява използването на метода за по-широк набор от взаимодействия, и по-високата му скорост, за разлика от методите, базирани на други алгоритми. Ефективността му е потвърдена успешно в лабораторно изпитване на способността за свързване на седем молекули със спайкпротеина на SARS-CoV-2 (или ACE-2 рецепторите на човешки клетки), при което пет от тях са показали очаквания от модела резултат. Авторите смятат, че благодарение на гъвкавостта и високата си възпроизводимост, този метод може да бъде ключов в процеса на въвеждане на нови лекарства на пазара.

Алгоритми се използват и за проверка на това дали новосъздадените медикаменти могат да проявят токсичност към организма. За съжаление, тези алгоритми имат и своята тъмна страна. Само с промяна на условието – да се търсят токсични вместо нетоксични молекули – за по-малко от 6 часа са генерирани 40 000 химични структури на отровни вещества. Част от тях са вече познати химически оръжия, включително изключително опасният нервнопаралитичен агент VX, но има и нови, някои от които са със значително по-висока средна летална доза спрямо VX. Както авторите отбелязват, „духът вече е излязъл от бутилката“ и предстои много сериозна дискусия относно бъдещото развитие на тези технологии и създаването на механизми за ограничаване на възможността за използването им със зъл умисъл.

Повече по темата можете да чуете в епизода 40,000 Recipes for Murder на подкаста Radiolab.

Необятният Космос

Наблюдаването на астрономически събития, като слънчевите затъмнения, може да окаже сплотяващ и помиряващ ефект върху обществата, възприемано като изживяване на колективно чудо и източник на вдъхновение и възхищение. До този извод са достигнали авторите на проучване, в което са анализирани реакциите на почти 3 млн. потребители на социалната мрежа Twitter, за да се изследва социалното въздействие на пълното слънчево затъмнение през 2017 г. Според резултатите от изследването вероятността потребителите, попаднали в зоната на пълното затъмнение, да използват думи на благоговение е била над два пъти по-висока от тези извън него. Над 1,5 пъти по-вероятно е било да използват думи, свързани с общност и принадлежност (например „ние“). Увеличили са се и позитивните емоции.

Изводите от това изследване ще можете да изпитате и сами – макар че от територията на България няма да може да бъде наблюдавано като пълно, на 25 октомври около 12:30 ч. ще започне слънчево затъмнение. Максимумът му ще настъпи около час след това, като колкото по̀ на изток се намирате, толкова по-голяма част от слънчевия диск ще бъде закрита. Ако възнамерявате да наблюдавате явлението, използвайте подходящи защитни очила или филтри.

Системата от пръстени на Сатурн е най-разпознаваемата в нашата Слънчева система поради възможността за наблюдението ѝ със сравнително малко увеличение. Нов модел предлага обяснение за тяхното образуване, както и за наклона на оста на планетата спрямо орбиталната равнина. Според хипотезата преди около 160 млн. години орбитата на една от луните му (наречена от екипа Chrysalis, „какавида“) се е дестабилизирала и вследствие на приливните сили Chrysalis е била разкъсана на парчета, които с времето са оформили системата от пръстени на планетата (преди около 100 млн. години). Това е нарушило и баланса между Сатурн и Нептун, променяйки наклона на оста му. Предполага се, че най-вероятната причина е „побутване“ от Титан (най-голямата от 83-те луни на Сатурн), като това би обяснило и по-ексцентричната му орбита.

Но пръстени имат и четирите гиганта в нашата Слънчева система. С помощта на детектора за близка инфрачервена светлина на телескопа „Джеймс Уеб“ са направени най-ясните снимки на пръстените на Нептун след получените от космическата сонда „Вояджър 2“ при прелитането ѝ покрай планетата през 1989 г. Заради използването на този детектор планетата няма характерния си син цвят и изглежда сравнително бледа – и двата ефекта са вследствие на наличието на метан в атмосферата ѝ. Видими са и облаци от метан, които са високо в атмосферата и успяват да отразят повече от слънчевата светлина. На снимката се виждат и 7 от 14-те известни луни на Нептун, като най-впечатляващ е Тритон, който с покривката си от замръзнал азот отразява около 70% от попадналата върху него слънчева светлина. Това затвърждава успешното въвеждане в експлоатация на телескопа „Джеймс Уеб“ като едно от най-значимите събития в астрономията през последните години.

Заглавно изображение: Микроскопска снимка на стъбло на петгодишен бор, напречен разрез, 40х увеличение. Източник: BCC Bioscience Image Library, CC0 / Flickr

Източник

По буквите: Списание „Съвременник“ и „Вдън прогледния мрак“

Post Syndicated from Зорница Христова original https://toest.bg/po-bukvite-suvremennik-vdun-progledniya-mrak/

В емблематичната си колонка, започната още през 2008 г. във в-к „Култура“, Марин Бодаков ни представяше нови литературни заглавия и питаше с какво точно тези книги ни променят. Вярваме, че е важно тази рубрика да продължи. От човек до човек, с нова книга в ръка.

Списание „Съвременник“, брой 3/2022

гл. ред. Владимир Зарев, София: Книгоиздателска къща „Труд“, 2022

Новият брой на сп. „Съвременник“ започва с един от по-старите романи на Дж. М. Кутси – „В сърцето на страната“, в превод на Пейчо Кънев. Тематично кълбо от клаустрофобия и копнеж, разлюлени равновесия между подчинение и власт, полупустиня, хтонична каша между въображаемо и реално, кръв, пръст, страст, екскременти, танцовите роли на расата и пола се завъртат без обяснения.

Тежък сюжет, на хартия, а всъщност го четох с носталгична наслада – от третия параграф, когато се хващаш на играта и започваш да дебнеш разказвачката: сега лъже ли, измисля ли си, или не си спомня добре? Първия път явно само си е изфантазирала убийството на баща си, защото той продължава да е жив в следващите страници и няма никаква нова невеста. Добре, а втория път тогава наистина ли е? Колко да е истина, след като самата тя, Магда, е въображаема, леко зловеща измислица на автора, близка до героиня на Стивън Кинг, но преди всичко – до сянката на човешкото съзнание, което не престава да фантазира докъде би могло да стигне, ако откаже да следва заповедите на морала, ако обърне, макар и наум, властовите си отношения с него. „Пътят, по който не поех“, в цялата му мрачна потентност. Потенциал, де.

Разказите, преведени от Божидар Стойков, са чудесно подбрани и издават общ вкус към нещо подобно. Ето примерно героинята на Хелън Симпсън – грижовна майка, която помага на тринайсетгодишния си син да си напише домашното по английски за събитие, променило живота му. И докато се чуди какво аджеба може да бъде то, успява да изфантазира развода си и всички произтичащи от това събития. Или разказа на Робърт Стоун, в който професиите на героите предполагат те да помагат на клиентите си, но вместо това ги зареждат с гняв и фрустрация – докато жената някак „не се оказва“ от погрешната страна на оръжието, с което мъжът е тръгнал да я защитава.

И отново: в „Касерола“ на Томас Макгуейн една млада двойка привидно има избор – да поеме или не наследственото ранчо, но този избор се оказва предрешен с пистолет в ръката и яхния в пластмасова кутия. Въобще из броя преминава темата за ужаса от „а можеше“, ужаса от нереализираните житейски сюжети. Усещането, че губиш контрол върху съдбата си, някой решава вместо теб (дори да е миналото ти Аз). В „Довиждане за довиждане“ на Стивън Диксън героят проиграва четири различни сценария, в които разбира за любовника на жена си – и собствената му роля се спихва от мъжкаря, който буквално изгонва навлека заради простото нахалство да седи на дивана му, до човек, принуден да приеме, че жена му просто отдавна не го обича и си е намерила нещо временно, колкото да има повод да се махне.

Две вариации по темата „другият възможен живот“ намираме и в разказите на Луиджи Пирандело, преведени от Огнян Стамболиев. Възможното бъдеще на една двойка със или без слепотата на мъжа. Възможното бъдеще на една диригентска вдовица със или без лъжата на нейната принадлежност към музиката, към семейните приятелски кръгове, към собствената ѝ досегашна орбита. В този случай отрязаният избор присъства като глупаво пропуснати шансове, като отказ на героините да позволят на сюжета (житейската фабула, която разказват в главите си) да се отклони от своята наративна логика.

И ето отгласа в разказите на Палми Ранчев – бездомното дете не си издава името и целият потенциален сюжет на неговото спасение отива на кино; трима български емигранти от различни разкази се движат според случая, като градацията от неща, които се случват без твоя воля и участие, набъбва все по-застрашително: по стечение на обстоятелствата заминаваш в чужбина, по стечение на обстоятелствата попадаш на някаква работа, разбираш – не разбираш, по стечение на обстоятелствата можеш да се ожениш, да спиш с жената на съседа, без да ѝ знаеш името, гаджето ти да забременее или ти да откриеш, че е спряло да ти пука дали си откраднал. Пестеливи разкази, написани в първо лице от името на своите немногословни герои, тук валенциите на „другото възможно“ не са разгърнати, дори споменати; ефектът идва тъкмо от неговата липса. Персонажите не се и сещат, че могат и да не следват жребия.

Точно обратното на персонажите в разказите на Йордан Славейков, за които вече писах в „Тоест“. Те, напук на правдоподобността, живеят в автентичността, в отказа от скепсис, в упоритостта на избора. И разбира се, неизбежно стигат до магическото мислене – все едно дали е предадено като „реална“ магическа рамка на разказа („Емилия“), или като въображаем свят на наивната героиня, която навсякъде вижда знаци от съдбата („Писател“). Всъщност двете героини съвсем не са толкова различни, едната би могла да е въображаем образ на другата.

Много интересен контрапункт на темата за възможния избор са и двата своеобразни „мемоарни“ сегмента в списанието. Единият – спомените на Джеки Стоев „Джеки, Джони и Чарли не са имена на кучета“ – е за свободата и смеха по черните пътища на официозната култура преди 1989-та. Напук на литературните герои от съседните страници, този глас съвсем не жали за пропуснати „други животи“, нито пък се усеща като каквато и да било пионка на съдбата. Напротив, той е този, който ще остави Съдбата да се тюхка пред военното поделение в Мичурин, защото някой ѝ е откраднал мотора на джипката.

Интересно, че същото жизнелюбие се усеща и в интервюто на Пламен Дойнов с проф. Михаил Неделчев – въпреки че темата е Съюзът на българските писатели. Като литературен историк проф. Неделчев е много повече разказвач, отколкото съдия – и неслучилото насъскване на „младите лъвчета“ срещу старите критици, и „белият мерцедес“ на Джагаров, демонстративно паркиран пред СБП, са свидетелство за времето, но и ефектна литература. Впрочем в броя е отбелязана неговата 80-годишнина и с апология от Владимир Зарев, и с текст от проф. Елка Трайкова за приноса на Михаил Неделчев в литературната ни история. Еднозначно си набелязах за четене „Цензурираните класици“ – антология на останалите извън канона неудобни произведения на Вазов, Пенчо Славейков, Страшимиров, Елин Пелин, Яворов, Симеон Радев, Йовков, Талев, Каралийчев и др. Пряко доказателство, че историята също е въпрос на избор – избор кое да разкажеш.

В броя е включена поезия от Камелия Спасова (цикъл стихотворения, в които основен образ е последният читател) и Никола Петров. Стихотворенията са нови, невключени в последната стихосбирка „Не са чудовища“ на Петров – също толкова силни, седем на брой. Най-ефектното е „Имаш болка срещу всяко лекарство“, най-много ми хареса това за очакването, обаче тъй като говорих за избора, избрах да ви цитирам ей този откъс:

Не можехме, защото кой избира да преследва,
кой избира да почива.
Кой избира да преживя или да къса плът
с муцуната, направена за късане на плът.
Кой избира, ако изборът е да кимаш на жребия.
Да избираш е да имаш тикове.

Много ми се щеше да завърша така, но нямам избор. Просто трябва да кажа: другари и другарки, вземете си коректор, де. Иначе броят е чуден!

„Вдън прогледния мрак“ от Виктор Самуилов

подзаглавие „Сказание за нощни птици“, илюстрации и оформление Невена Ангелова, Пловдив: ИК „Жанет 45“, 2022

Смятам брилянтните стихотворения за деца на Виктор Самуилов за едно от нещата, които безспорно ще останат във времето от целия наплив на съвременна детска литература. Защо? Заради съчетанието между мощно езиково майсторство и честно усещане за болезнените страни на днешното детство, примесени с топлота и смях. Това последното често бива пробутвано на децата „пет за четири“, по най-простия начин. Не и при Виктор Самуилов. Може би си струва някой да направи разбор на видовете „смешно“ в неговите стихотворения.

Но новата му детска книга е в проза. И не говори от името на дете. Напротив, говори от името на един бухал на неопределена възраст, който обръща всичко наопаки – денят му е противен, светулките ги гони с хавлиена кърпа, дразни се от досадните славеи… И разбира се, си води не дневник, а „нощник“. Така описаният герой се хвърля в класическо пътешествие с цел да се пресели в пределите на полярната нощ, където ще може на спокойствие да завърши астрономическия си труд. И както си му е редът в разказите за пътешествия, среща трима други герои, преди финалната среща да обърне нещата и да ги поведе обратно към епилога.

От диалозите с тях най-забавен ми се струва този със заблудената овца (на отиване) – когато разбираме колко тъжно е да се заблуждаваш, че си се заблудил нарочно. Разговорът с овцата ни дава и ценното наблюдение, че „чуждата мъка понякога ни изпълва с неподозирани сили“. На връщане пък ще срещнем нейните посестрими, Бухала ще ги разпита за нея… но в отговор ще научи само: „Всички сме заблуденииии! Цялото стадоооо!“

Другият основен източник на смях е обръщането на нашите „светлоцентристки“ понятия – за да снимаш нещо насред досадната светлина, ти трябва фотоапарат с „тъмнявица“, над работното ти бюро черните лъчи на лампата едва смогват да отъмнят записките ти… Изобщо, ако имате дребосъци наоколо, не пропускайте.

Активните дарители на „Тоест“ получават постоянна отстъпка в размер на 20% от коричната цена на всички заглавия от каталога на „Жанет 45“, както и на няколко други български издателства в рамките на партньорската програма Читателски клуб „Тоест“. За повече информация прочетете на toest.bg/club.
Заглавно изображение: Колаж от кориците на списанието и книгата и снимка на Annie Spratt / Unsplash

Източник

Metasploit Weekly Wrap-Up

Post Syndicated from Dean Welch original https://blog.rapid7.com/2022/09/30/metasploit-weekly-wrap-up-178/

Veritas Backup Exec Agent RCE

Metasploit Weekly Wrap-Up

This module kindly provided by c0rs targets the Veritas Backup Exec Agent in order to gain RCE as the system/root user.
The exploit itself is actually a chain of 3 separate CVEs (CVE-2021-27876, CVE-2021-27877 and CVE-2021-27878) which only makes it more impressive.
While you’re patching, why not take the time to test your backups too.

Hikvision IP Camera user impersonation

This vulnerability has been present in Hikvision products since 2014 and comes to us courtesy of h00die-gr3y.
The main culprit here is in Hikvisions authentication mechanism which allows you to login as any valid user using only their username and from that point this module allows you to set a new password for your chosen username so now you can log in "legitimately".

New module content (6)

  • Hikvision IP Camera Unauthenticated Password Change Via Improper Authentication Logic by Monte Crypto and h00die-gr3y, which exploits CVE-2017-7921 – A new module has been added in for CVE-2017-7921, an improper authentication logic bug in HikVision cameras. Successfully exploiting this vulnerability allows unauthenticated attackers to impersonate any valid user on the affected camera, which can be used to gain full control over the camera.
  • Netfilter nft_set_elem_init Heap Overflow Privilege Escalation by Arthur Mongodin and Redouane NIBOUCHA, which exploits CVE-2022-34918 – This is a local-privilege escalation exploit targeting CVE-2022-34918, a vulnerability in the Netfilter component of the Linux kernel.
  • qdPM 9.1 Authenticated Arbitrary PHP File Upload (RCE) by Giacomo Casoni, Leon Trappett (thepcn3rd), and Rishal Dwivedi (Loginsoft), which exploits CVE-2020-7246 – Adds an exploit that targets an authenticated arbitrary file upload vulnerability to gain code execution on qdPM 9.1 and lower.
  • Veritas Backup Exec Agent Remote Code Execution by Alexander Korotin, which exploits CVE-2021-27878 – This module exploits a chain of the vulnerabilities CVE-2021-27876, CVE-2021-27877, and CVE-2021-27878 in Veritas Backup Exec Agent which leads to remote code execution with privileges of system or root user.
  • Mobile Mouse RCE by CHOKRI HAMMEDI and h00die – This PR includes a module that uses default configuration in Unified Remote to spawn a run prompt and return a shell.
  • Wifi Mouse RCE by H4RK3NZ0, REDHATAUGUST, and h00die, which exploits CVE-2022-3218 – A new module has been added for CVE-2022-3218, an unpatched (at the time of publication) authentication bypass in WiFi Mouse (Mouse Server) from Necta LLC which can be used to gain RCE as the user running Wifi Mouse (Mouse Server).

Enhancements and features (2)

  • #16981 from bcoles – This PR fixes several bugs as well as style and documentation inconsistencies as well as implementing new library methods.
  • #17048 from bcoles – :
    This PR Updates the enum_token module by adding documentation, clarifying the description, improving efficiency, and leveraging library code.

Bugs fixed (3)

  • #16994 from zeroSteiner – Fixes multiple issues with registry manipulation on opened sessions.
  • #17054 from zeroSteiner – Fixes a crash when using the info and generate commands for adapted single (unstaged) payloads – such as cmd/windows/powershell/meterpreter/reverse_tcp.
  • #17073 from cgranleese-r7 – Fixes a bug where sessions opened by running one of the rexec_login / rlogin_login / rsh_login modules would die after module completion.

Get it

As always, you can update to the latest Metasploit Framework with msfupdate
and you can get more details on the changes since the last blog post from
GitHub:

If you are a git user, you can clone the Metasploit Framework repo (master branch) for the latest.
To install fresh without using git, you can use the open-source-only Nightly Installers or the
binary installers (which also include the commercial edition).

Amazon File Cache – A High Performance Cache On AWS For Your On-Premises File Systems

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/amazon-file-cache-a-high-performance-cache-on-aws-for-your-on-premises-file-systems/

I am pleased to announce today the availability of Amazon File Cache, a new high-speed cache service on AWS designed for processing file data stored in disparate locations—including on premises. File Cache accelerates and simplifies your most demanding cloud bursting and hybrid workflows by giving your applications access to files using a fast and familiar POSIX interface, no matter if the original files live on premises on any file system that can be accessed through NFS v3 or on Amazon Simple Storage Service (Amazon S3).

Imagine you have a large data set on on-premises storage infrastructure, and your end-of-month reporting typically takes two to three days to run. You want to move that occasional workload to the cloud to run it on larger machines with more CPU and memory to reduce the processing time. But you’re not ready to move the data set to the cloud yet.

Imagine another scenario where you have access to a large data set on Amazon Simple Storage Service (Amazon S3), spread across multiple Regions. Your application that wants to exploit this data set is coded for traditional (POSIX) file system access and uses command line tools like awk, sed, pipes, and so on. Your application requires file access with sub-millisecond latencies. You cannot update the source code to use the S3 API.

File Cache helps to address these use cases and many others, think about management and transformation of video files, AI/ML data sets, and so on. File Cache creates a file system–based cache in front of either NFS v3 file systems or S3 buckets in one or more Regions. It transparently loads file content and metadata (such as the file name, size, and permissions) from the origin and presents it to your applications as a traditional file system. File Cache automatically releases the less recently used cached files to ensure the most active files are available in the cache for your applications.

You can link up to eight NFS file systems or eight S3 buckets to a cache, and they will be exposed as a unified set of files and directories. You can access the cache from a variety of AWS compute services, such as virtual machines or containers. The connection between File Cache and your on-premises infrastructure uses your existing network connection, based on AWS Direct Connect and/or Site-to-Site VPN.

When using File Cache, your applications benefit from consistent, sub-millisecond latencies, up to hundreds of GB/s of throughput, and up to millions of operations per second. Just like with other storage services, such as Amazon Elastic Block Store (Amazon EBS), the performance depends on the size of the cache. The cache size can be expanded to petabyte scale, with a minimum size of 1.2 TiB.

Let’s See How It Works
To show you how it works, I create a file cache on top of two existing Amazon FSx for OpenZFS file systems. In a real-world scenario, it is likely you will create caches on top of on-premises file systems. I choose FSx for OpenZFS for the demo because I don’t have an on-premises data center at hand (I should maybe invest in seb-west-1). Both demo OpenZFS file systems are accessible from a private subnet in my AWS account. Finally, I access the cache from an EC2 Linux instance.

I open my browser and navigate to the AWS Management Console. I search for “Amazon FSx” in the console search bar and click on Caches in the left navigation menu. Alternatively, I go directly to the File Cache section of the console. To get started, I select Create cache.

Amazon File Cache consoleI enter a Cache name for my cache (AWSNewsBlog for this demo) and a Cache storage capacity. The storage capacity is expressed in tebibytes. The minimum value is 1.2 TiB or increments of 2.4 TiB. Notice that the Throughput capacity increases as you choose large cache sizes.

Amazon File Cache create 1I check and accept the default values provided for Networking and Encryption. For networking, I might select a VPC, subnet, and security group to associate with my cache network interface. It is recommended to deploy the cache in the same subnet as your compute service to minimize the latency when accessing files. For encryption, I might use an AWS KMS-managed key (the default) or select my own.

Then, I create Data Repository Association. This is the link between the cache and a data source. A data source might be an NFS file system or an S3 bucket or prefix. I might create up to eight data repository associations for one cache. All Data Repository Associations for a cache have the same type: they are all NFS v3 or all S3. If you need both, you can create two caches.

In this demo, I choose to link two OpenZFS file systems on my AWS account. You can link to any NFS v3 servers, including the ones you already have on premises. Cache path allows you to choose where the source file system will be mounted in the cache. The Data repository path is the URL to your NFS v3 or S3 data repository. The format is nfs://hostname/path or s3://bucketname/path.

The DNS server IP addresses allows File Cache to resolve the DNS name of your NFS server. This is useful when DNS resolution is private, like in my example. When you are associating NFS v3 servers deployed in a VPC, and when using the AWS-provided DNS server, the DNS server IP address of your VPC is the VPC Range + two. In my example, my VPC CIDR range is 172.31.0.0, hence the DNS server IP address is 172.31.0.2.

Do not forget to click on the Add button! Otherwise, your input is ignored. You can repeat the operation to add more data repositories.

File Cache - Create new Data Repository Association- dataset one File Cache - Create new Data Repository Association- dataset two

Once I have entered my two data repositories, I select Next, and I review my choices. When I am ready, I select Create cache.

File Cache - review choices

After a few minutes, the cache status becomes ✅ Available.

Amazon File cache status is available

The last part is to mount the cache on the machine where my workload is deployed. File Cache uses Lustre behind the scene. I have to install the Lustre client for Linux first, as explained in our documentation. Once done, I select the Attach button on the console to receive the instructions to download and install the Lustre client and to mount the cache file system.File Cache Attach To do so, I connect to an EC2 instance running in the same VPC. Then I type:

sudo mount -t lustre -o relatime,flock file_cache_dns_name@tcp:/mountname /mnt

This command mounts my cache with two options:

  • relatime – Maintains atime (inode access times) data, but not for each time that a file is accessed. With this option enabled, atime data is written to disk only if the file has been modified since the atime data was last updated (mtime) or if the file was last accessed more than a certain amount of time ago (one day by default). relatime is required for automatic cache eviction to work properly.
  • flock – Enables file locking for your cache. If you don’t want file locking enabled, use the mount command without flock.

Once mounted, processes running on my EC2 instance can access files in the cache as usual. As I defined at cache creation time, the first ZFS file system is available inside the cache at /dataset1, and the second ZFS file system is available as /dataset2.

$ echo "Hello File Cache World" > /mnt/zsf1/greetings

$ sudo mount -t lustre -o relatime,flock fc-0280000000001.fsx.us-east-2.aws.internal@tcp:/r3xxxxxx /mnt/cache

$ ls -al /mnt/cache
total 98
drwxr-xr-x 5 root root 33280 Sep 21 14:37 .
drwxr-xr-x 2 root root 33280 Sep 21 14:33 dataset1
drwxr-xr-x 2 root root 33280 Sep 21 14:37 dataset2

$ cat /mnt/cache/dataset1/greetings
Hello File Cache World

I can observe and measure the activity and the health of my caches using Amazon CloudWatch metrics and AWS CloudTrail log monitoring.

CloudWatch metrics for a File Cache resource are organized into three categories:

  • Front-end I/O metrics
  • Backend I/O metrics
  • Cache front-end utilization metrics

As usual, I can create dashboards or define alarms to be informed when metrics reach thresholds that I defined.

Things To Keep In Mind
There are a couple of key points to keep in mind when using or planning to use File Cache.

First, File Cache encrypts data at rest and supports encryption of data in transit. Your data is always encrypted at rest using keys managed in AWS Key Management Service (AWS KMS). You can use either service-owned keys or your own keys (customer-managed CMKs).

Second, File Cache provides two options for importing data from your data repositories to the cache: lazy load and preload. Lazy load imports data on demand if it’s not already cached, and preload imports data at user request before you start your workload. Lazy loading is the default. It makes sense for most workloads since it allows your workload to start without waiting for metadata and data to be imported to the cache. Pre loading is helpful when your access pattern is sensitive to first-byte latencies.

Pricing and Availability
There are no upfront or fixed-price costs when using File Cache. You are billed for the provisioned cache storage capacity and metadata storage capacity. The pricing page has the details. In addition to File Cache itself, you pay for S3 request costs, AWS Direct Connect charges, and the usual data transfer charges for inter-AZ, inter-Region, and internet egress traffic between File Cache and the data sources.

File Cache is available in US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), Europe (Frankfurt), Europe (Ireland), and Europe (London).

Now go build and create your first file cache today!

— seb

[$] Hybrid scheduling gets more complicated

Post Syndicated from original https://lwn.net/Articles/909611/

Just over ten years ago, the Arm big.LITTLE architecture posed a challenge for the kernel’s CPU
scheduler: how should processes be assigned to CPUs when not all CPUs have
the same capacity? The situation has not gotten simpler since then; new
systems bring new quirks that must be kept in mind for optimal scheduling.
At the 2022 Linux Plumbers Conference, Len
Brown and Ricardo Neri talked about Intel’s hybrid systems and the work
that is being done to schedule properly on those systems.

Security Vulnerabilities in Covert CIA Websites

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/09/security-vulnerabilities-in-covert-cia-websites.html

Back in 2018, we learned that covert system of websites that the CIA used for communications was compromised by—at least—China and Iran, and that the blunder caused a bunch of arrests, imprisonments, and executions. We’re now learning that the CIA is still “using an irresponsibly secured system for asset communication.”

Citizen Lab did the research:

Using only a single website, as well as publicly available material such as historical internet scanning results and the Internet Archive’s Wayback Machine, we identified a network of 885 websites and have high confidence that the United States (US) Central Intelligence Agency (CIA) used these sites for covert communication.

The websites included similar Java, JavaScript, Adobe Flash, and CGI artifacts that implemented or apparently loaded covert communications apps. In addition, blocks of sequential IP addresses registered to apparently fictitious US companies were used to host some of the websites. All of these flaws would have facilitated discovery by hostile parties.

[…]

The bulk of the websites that we discovered were active at various periods between 2004 and 2013. We do not believe that the CIA has recently used this communications infrastructure. Nevertheless, a subset of the websites are linked to individuals who may be former and possibly still active intelligence community employees or assets:

  • Several are currently abroad
  • Another left mainland China in the timeframe of the Chinese crackdown
  • Another was subsequently employed by the US State Department
  • Another now works at a foreign intelligence contractor

Citizen Lab is not publishing details, of course.

When I was a kid, I thought a lot about being a spy. And this, right here, was the one thing I worried about. It didn’t matter how clever and resourceful I was. If my handlers were incompetent, I was dead.

Another news article.

EDITED TO ADD (10/2): Slashdot thread.

How Launchmetrics improves fashion brands performance using Amazon EC2 Spot Instances

Post Syndicated from Ivo Pinto original https://aws.amazon.com/blogs/architecture/how-launchmetrics-improves-fashion-brands-performance-using-amazon-ec2-spot-instances/

Launchmetrics offers its Brand Performance Cloud tools and intelligence to help fashion, luxury, and beauty retail executives optimize their global strategy. Launchmetrics initially operated their whole infrastructure on-premises; however, they wanted to scale their data ingestion while simultaneously providing improved and faster insights for their clients. These business needs led them to build their architecture in AWS cloud.

In this blog post, we explain how Launchmetrics’ uses Amazon Web Services (AWS) to crawl the web for online social and print media. Using the data gathered, Launchmetrics is able to provide prescriptive analytics and insights to their clients. As a result, clients can understand their brand’s momentum and interact with their audience, successfully launching their products.

Architecture overview

Launchmetrics’ platform architecture is represented in Figure 1 and composed of three tiers:

  1. Crawl
  2. Data Persistence
  3. Processing
Launchmetrics backend architecture

Figure 1. Launchmetrics backend architecture

The Crawl tier is composed of several Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances launched via Auto Scaling groups. Spot Instances take advantage of unused Amazon EC2 capacity at a discounted rate compared with On-Demand Instances, which are compute instances that are billed per-hour or -second with no long-term commitments. Launchmetrics heavily leverages Spot Instances. The Crawl tier is responsible for retrieving, processing, and storing data from several media sources (represented in Figure 1 with the number 1).

The Data Persistence tier consists of two components: Amazon Kinesis Data Streams and Amazon Simple Queue Service (Amazon SQS). Kinesis Data Streams stores data that the Crawl tier collects, while Amazon SQS stores the metadata of the whole process. In this context, metadata helps Launchmetrics gain insight into when the data is collected and if it has started processing. This is key information if a Spot Instance is interrupted, which we will dive deeper into later.

The third tier, Processing, also makes use of Spot Instances and is responsible for pulling data from the Data Persistence tier (represented in Figure 1 with the number 2). It then applies proprietary algorithms, both analytics and machine learning models, to create consumer insights. These insights are stored in a data layer (not depicted) that consists of an Amazon Aurora cluster and an Amazon OpenSearch Service cluster.

By having this separation of tiers, Launchmetrics is able to use a decoupled architecture, where each component can scale independently and is more reliable. Both the Crawl and the Data Processing tiers use Spot Instances for up to 90% of their capacity.

Data processing using EC2 Spot Instances

When Launchmetrics decided to migrate their workloads to the AWS cloud, Spot Instances were one of the main drivers. As Spot Instances offer large discounts without commitment, Launchmetrics was able to track more than 1200 brands, translating to 1+ billion end users. Daily, this represents tracking upwards of 500k influencer profiles, 8 million documents, and around 70 million social media comments.

Aside from the cost-savings with Spot Instances, Launchmetrics incurred collateral benefits in terms of architecture design: building stateless, decoupled, elastic, and fault-tolerant applications. In turn, their stack architecture became more loosely coupled, as well.

All Launchmetrics Auto Scaling groups have the following configuration:

  • Spot allocation strategy: cost-optimized
  • Capacity rebalance: true
  • Three availability zones
  • A diversified list of instance types

By using Auto Scaling groups, Launchmetrics is able to scale worker instances depending on how many items they have in the SQS queue, increasing the instance efficiency. Data processing workloads like the ones Launchmetrics’ platform have, are an exemplary use of multiple instance types, such as M5, M5a, C5, and C5a. When adopting Spot Instances, Launchmetrics considered other instance types to have access to spare capacity. As a result, Launchmetrics found out that workload’s performance improved, as they use instances with more resources at a lower cost.

By decoupling their data processing workload using SQS queues, processes are stopped when an interruption arrives. As the Auto Scaling group launches a replacement Spot Instance, clients are not impacted and data is not lost. All processes go through a data checkpoint, where a new Spot Instance resumes processing any pending data. Spot Instances have resulted in a reduction of up to 75% of related operational costs.

To increase confidence in their ability to deal with Spot interruptions and service disruptions, Launchmetrics is exploring using AWS Fault Injection Simulator to simulate faults on their architecture, like a Spot interruption. Learn more about how this service works on the AWS Fault Injection Simulator now supports Spot Interruptions launch page.

Reporting data insights

After processing data from different media sources, AWS aided Launchmetrics in producing higher quality data insights, faster: the previous on-premises architecture had a time range of 5-6 minutes to run, whereas the AWS-driven architecture takes less than 1 minute.

This is made possible by elasticity and availability compute capacity that Amazon EC2 provides compared with an on-premises static fleet. Furthermore, offloading some management and operational tasks to AWS by using AWS managed services, such as Amazon Aurora or Amazon OpenSearch Service, Launchmetrics can focus on their core business and improve proprietary solutions rather than use that time in undifferentiated activities.

Building continuous delivery pipelines

Let’s discuss how Launchmetrics makes changes to their software with so many components.

Both of their computing tiers, Crawl and Processing, consist of standalone EC2 instances launched via Auto Scaling groups and EC2 instances that are part of an Amazon Elastic Container Service (Amazon ECS) cluster. Currently, 70% of Launchmetrics workloads are still running with Auto Scaling groups, while 30% are containerized and run on Amazon ECS. This is important because for each of these workload groups, the deployment process is different.

For workloads that run on Auto Scaling groups, they use an AWS CodePipeline to orchestrate the whole process, which includes:

I.  Creating a new Amazon Machine Image (AMI) using AWS CodeBuild
II. Deploying the newly built AMI using Terraform in CodeBuild

For containerized workloads that run on Amazon ECS, Launchmetrics also uses a CodePipeline to orchestrate the process by:

III. Creating a new container image, and storing it in Amazon Elastic Container Registry
IV. Changing the container image in the task definition, and updating the Amazon ECS service using CodeBuild

Conclusion

In this blog post, we explored how Launchmetrics is using EC2 Spot Instances to reduce costs while producing high-quality data insights for their clients. We also demonstrated how decoupling an architecture is important for handling interruptions and why following Spot Instance best practices can grant access to more spare capacity.

Using this architecture, Launchmetrics produced faster, data-driven insights for their clients and increased their capacity to innovate. They are continuing to containerize their applications and are projected to have 100% of their workloads running on Amazon ECS with Spot Instances by the end of 2023.

To learn more about handling EC2 Spot Instance interruptions, visit the AWS Best practices for handling EC2 Spot Instance interruptions blog post. Likewise, if you are interested in learning more about AWS Fault Injection Simulator and how it can benefit your architecture, read Increase your e-commerce website reliability using chaos engineering and AWS Fault Injection Simulator.

Don’t roll your own high cardinality analytics, use Workers Analytics Engine

Post Syndicated from Cloudflare original https://blog.cloudflare.com/analytics-engine-open-beta/

Don't roll your own high cardinality analytics, use Workers Analytics Engine

Don't roll your own high cardinality analytics, use Workers Analytics Engine

Workers Analytics Engine (or for short, Analytics Engine) is a new way for developers to store and analyze time series analytics about anything using Cloudflare Workers, and it’s now in open beta! Analytics Engine is really good at gathering time-series data for really high cardinality and high-volume data sets from Cloudflare Workers. At Cloudflare, we use Analytics Engine to provide insight into how our customers use Cloudflare products.

Log, log, logging!

As an example, Analytics Engine is used to observe the backend that powers Instant Logs. Instant Logs allows Cloudflare customers to stream a live session of the HTTP logs for their domain to the Cloudflare dashboard. The backend for Instant Logs is built on Cloudflare Workers.

Briefly, the Instant Logs backend works by receiving requests from each Cloudflare server that processes a customer’s HTTP traffic. These requests contain the HTTP logs for the customer’s HTTP traffic. The Instant Logs backend then forwards these HTTP logs to the customer’s browser via a WebSocket.

In order to ensure that the HTTP logs are being delivered smoothly to a customer’s browser, we need to track the request rates across all active Instant Logs sessions. We also need to track the request rates across all Cloudflare data centers, since Instant Logs is built on Cloudflare Workers, and Cloudflare Workers is built on Cloudflare’s massive network. As a result, the data set for the Instant Logs backend has really massive cardinality!

“Traditional” metrics systems like Prometheus are poorly suited to serving high cardinality data. Fortunately, this is exactly the problem that Analytics Engine is designed to solve. So, we sent all the Instant Logs backend request logs to Analytics Engine. Log, log, logging!

Using the Analytics Engine API (which has a SQL interface), we can visualize the Instant Logs backend request rates for the top sessions and top data centers over the previous month. “Zooming in” to an interesting period is also really fast. We’ve designed Analytics Engine so that queries always respond within the window of interactivity (more on this later). This makes it well-suited for interactive debugging with a dashboard tool (in this case we’re using Grafana).

What we learned in closed beta

We received a lot of great feedback during the closed beta. Developers were excited about the SQL API, ease of integration with Workers, the ability to query data in Grafana (with more integrations in future), and our simple pricing model (free!). However, there were a number of things that we needed to fix before moving on to the open beta phase.

Developers were supportive of our choice to use SQL (the world’s language for data) as the interface for the Analytics Engine API. However, when developers used the Analytics Engine API, they found that the error messages were opaque and difficult to debug. For the open beta, we have rewritten the API from the ground-up to provide much improved error messaging.

Before:
> SELECT column_that_does_not_exist FROM your_dataset FORMAT JSON
Sorry, we were unable to evaluate your query

After:
> SELECT column_that_does_not_exist FROM your_dataset FORMAT JSON
cannot select unknown column: "column_that_does_not_exist"

In addition to understanding what went wrong, developers also wanted to understand what the API is capable of doing. For the open beta, we’ve written a comprehensive SQL reference for Analytics Engine. We also have a few “How To” guides, including information on how to hook up the API to Grafana.

ABR and Analytics Engine

Analytics Engine uses Cloudflare’s ABR technology to make queries fast. This means that every query is satisfied by a resolution of the data that matches the query. For example, if we are looking at data for the last month, we might use a lower resolution version of the Analytics Engine data than if we are looking at the last hour. The lower resolution data will provide the correct answer, but will respond within the window of interactivity. By using multiple, different resolutions of the same data, ABR provides consistent response times.

To account for the different resolutions of data, each event carries with it information about the resolution of data that the event comes from. This information is encoded in the _sample_interval column. For example, if an event comes from a resolution of the data which is 1% of the original data, its _sample_interval will be set to 100. To reconstruct the number of events in the original data, we can use the query:

SELECT sum(_sample_interval) AS count FROM dataset

For the open beta, we are exposing _sample_interval directly to developers. In the future, we’ll make it easier to work with this field by providing convenience functions which automatically take into account varying resolutions of the data. We also want to provide the ability to understand the confidence level of the estimates that these functions return.

Coming soon

This is just the beginning for Workers Analytics Engine. Internally, there has been high demand for the ability to define alerts based on the data captured by Analytics Engine. This is also something that we want developers to be able to do.

As in the closed beta, fields are accessed via names that have 1-based indexing (blob1, blob2, double1, double2, etc.). In the future, we will allow developers to attach names to fields, and these names will be available to use to retrieve data via the SQL API.

Something we want to provide is a rich UX in the Cloudflare dashboard (imagine something like Grafana in the Cloudflare dashboard). Ultimately, we don’t want developers to have to set up their own infrastructure for exploring data captured with Analytics Engine.

Conclusion

Try Workers Analytics Engine today! Please let us know if you have any ideas or more advanced use cases that aren’t supported. We’re discussing everything about the Analytics Engine in our discord channel too – join the conversation!

Project A11Y: how we upgraded Cloudflare’s dashboard to adhere to industry accessibility standards

Post Syndicated from Emily Flannery original https://blog.cloudflare.com/project-a11y/

Project A11Y: how we upgraded Cloudflare’s dashboard to adhere to industry accessibility standards

Project A11Y: how we upgraded Cloudflare’s dashboard to adhere to industry accessibility standards

At Cloudflare, we believe the Internet should be accessible to everyone. And today, we’re happy to announce a more inclusive Cloudflare dashboard experience for our users with disabilities. Recent improvements mean our dashboard now adheres to industry accessibility standards, including Web Content Accessibility Guidelines (WCAG) 2.1 AA and Section 508 of the Rehabilitation Act.

Over the past several months, the Cloudflare team and our partners have been hard at work to make the Cloudflare dashboard1 as accessible as possible for every single one of our current and potential customers. This means incorporating accessibility features that comply with the latest Web Content Accessibility Guidelines (WCAG) and Section 508 of the US’s federal Rehabilitation Act. We are invested in working to meet or exceed these standards; to demonstrate that commitment and share openly about the state of accessibility on the Cloudflare dashboard, we have completed the Voluntary Product Accessibility Template (VPAT), a document used to evaluate our level of conformance today.

Conformance with a technical and legal spec is a bit abstract–but for us, accessibility simply means that as many people as possible can be successful users of the Cloudflare dashboard. This is important because each day, more and more individuals and businesses rely upon Cloudflare to administer and protect their websites.

For individuals with disabilities who work on technology, we believe that an accessible Cloudflare dashboard could mean improved economic and technical opportunities, safer websites, and equal access to tools that are shaping how we work and build on the Internet.

For designers and developers at Cloudflare, our accessibility remediation project has resulted in an overhaul of our component library. Our newly WCAG-compliant components expedite and simplify our work building accessible products. They make it possible for us to deliver on our commitment to an accessible dashboard going forward.

Our Journey to an Accessible Cloudflare Dashboard

In 2021, we initiated an audit with third party experts to identify accessibility challenges in the Cloudflare dashboard. This audit came back with a daunting 213-page document—a very, very long list of compliance gaps.

We learned from the audit that there were many users we had unintentionally failed to design and build for in Cloudflare dashboard user interfaces. Most especially, we had not done well accommodating keyboard users and screen reader users, who often rely upon these technologies because of a physical impairment. Those impairments include low vision or blindness, motor disabilities (examples include tremors and repetitive strain injury), or cognitive disabilities (examples include dyslexia and dyscalculia).

As a product and engineering organization, we had spent more than a decade in cycles of rapid growth and product development. While we’re proud of what we have built, the audit made clear to us that there was a great need to address the design and technical debt we had accrued along the way.

One year, four hundred Jira tickets, and over 25 new, accessible web components later, we’re ready to celebrate our progress with you. Major categories of work included:

  1. Forms: We re-wrote our internal form components with accessibility and developer experience top of mind. We improved form validation and error handling, labels, required field annotations, and made use of persistent input descriptions instead of placeholders. Then, we deployed those component upgrades across the dashboard.
  2. Data visualizations: After conducting a rigorous re-evaluation of their design, we re-engineered charts and graphs to be accessible to keyboard and screen reader users. See below for a brief case study.
  3. Heading tags: We corrected page structure throughout the dashboard by replacing all our heading tags (<h1>, <h2>, etc.) with a technique we borrowed from Heydon Pickering. This technique is an approach to heading level management that uses React Context and basic arithmetic.
  4. SVGs: We reworked how we create SVGs (Scalable Vector Graphics), so that they are labeled properly and only exposed to assistive technology when useful.
  5. Node modules: We jumped several major versions of old, inaccessible node modules that our UI components depend upon (and we broke many things along the way).
  6. Color: We overhauled our use of color, and contributed a new volume of accessible sequential colors to our design system.
  7. Bugs: We squashed a lot of bugs that had made their way into the dashboard over the years. The most common type of bug we encountered related to incorrect or unsemantic use of HTML elements—for example, using a <div> where we should have used a <td> (table data) or <tr> (table row) element within a table.

Case Study: Accessibility Work On Cloudflare Dashboard Data & Analytics

The Cloudflare dashboard is replete with analytics and data visualizations designed to offer deep insight into users’ websites’ performance, traffic, security, and more. Making those data visualizations accessible proved to be among the most complex and interdisciplinary issues we faced in the remediation work.

An example of a problem we needed to solve related to WCAG success criterion 1.4.1, which pertains to the use of color. 1.4.1 specifies that color cannot be the only means by which to convey information, such as the differentiation between two items compared in a chart or graph.

Our charts were clearly nonconforming with this standard, using color alone to represent different data being compared. For example, a typical graph might have used the color blue to show the number of requests to a website that were 200 OK, and the color orange to show 403 Forbidden, but failed to offer users another way to discern between the two status codes.

Our UI team went to work on the problem, and chose to focus our effort first on the Cloudflare dashboard time series graphs.

Interestingly, we found that design patterns recommended even by accessibility experts created wholly unusable visualizations when placed into the context of real world data. Examples of such recommended patterns include using different line weights, patterns (dashed, dotted or other line styles), and terminal glyphs (symbols set at the beginning and end of the lines) to differentiate items being compared.

We tried, and failed, to apply a number of these patterns; you can see the evolution of this work on our time series graph component in the three different images below.

v.1

Project A11Y: how we upgraded Cloudflare’s dashboard to adhere to industry accessibility standards
Here is an early attempt at using both terminal glyphs and patterns to differentiate data in a time series graph. You can see that the terminal glyphs pile up and become indistinguishable; the differences among the line patterns are very hard to discern. This code never made it into production.

v.2

Project A11Y: how we upgraded Cloudflare’s dashboard to adhere to industry accessibility standards
In this version, we eliminated terminal glyphs but kept line patterns. Additionally, we faded the unfocused items in the graph to help bring highlighted data to the forefront. This latter technique made it into our final solution.

v.3

Project A11Y: how we upgraded Cloudflare’s dashboard to adhere to industry accessibility standards
Here we eliminated patterns altogether, simplified the user interface to only use the fading technique on unfocused items, and put our new, sequentially accessible colors to use. Finally, a visual design solution approved by accessibility and data visualization experts, as well as our design and engineering teams.

After arriving at our design solution, we had some engineering work to do.

In order to meet WCAG success criterion 2.1.1, we rewrote our time series graphs to be fully keyboard accessible by adding focus handling to every data point, and enabling the traversal of data using arrow keys.

Navigating time series data points by keyboard on the Cloudflare dashboard.

We did some fine-tuning, specifically to support screen readers: we eliminated auditory “chartjunk” (unnecessary clutter or information in a chart or graph) and cleaned up decontextualized data (a scenario in which numbers are exposed to and read by a screen reader, but contextualizing information, like x- and y-axis labels, is not).

And lastly, to meet WCAG 1.1.1, we engineered new UI component wrappers to make chart and graph data downloadable in CSV format. We deployed this part of the solution across all charts and graphs, not just the time series charts like those shown above. No matter how you browse and interact with the web, we hope you’ll notice this functionality around the Cloudflare dashboard and find value in it.

Making all of this data available to low vision, keyboard, and assistive technology users was an interesting challenge for us, and a true team effort. It necessitated a separate data visualization report conducted by another, more specialized team of third party experts, deep collaboration between engineering and design, and many weeks of development.

Applying this thorough treatment to all data visualizations on the Cloudflare dashboard is our goal, but still work in progress. Please stay tuned for more accessible updates to our chart and graph components.

Conclusion

There’s a lot of nuance to accessibility work, and we were novices at the beginning: researching and learning as we were doing. We also broke a lot of things in the process, which (as any engineering team knows!) can be stressful.

Overall, our team’s biggest challenge was figuring out how to complete a high volume of cross-functional work in the shortest time possible, while also setting a foundation for these improvements to persist over time.

As a frontend engineering and design team, we are very grateful for having had the opportunity to focus on this problem space and to learn from truly world-class accessibility experts along the way.

Accessibility matters to us, and we know it does to you. We’re proud of our progress, and there’s always more to do to make Cloudflare more usable for all of our customers. This is a critical piece of our foundation at Cloudflare, where we are building the most secure, performant and reliable solutions for the Internet. Stay tuned for what’s next!

Not using Cloudflare yet? Get started today and join us on our mission to build a better Internet.

1All references to “dashboard” in this post are specific to the primary user authenticated Cloudflare web platform. This does not include Cloudflare’s product-specific dashboards, marketing, support, educational materials, or third party integrations.

Lights, Camera, Action! Business and Pro customers get bundled streaming video

Post Syndicated from Zaid Farooqui original https://blog.cloudflare.com/stream-for-pro-biz-customers/

Lights, Camera, Action! Business and Pro customers get bundled streaming video

Lights, Camera, Action! Business and Pro customers get bundled streaming video

Beginning December 1, 2022, if you have a Business or Pro subscription, you will receive a complimentary allocation of Cloudflare Stream. Here’s what this means:

  • All Cloudflare customers with a Biz or Pro domain will be able to store up to 100 minutes of video content and deliver up to 10,000 minutes of video content each month at no additional cost
  • If you need additional storage or delivery beyond the complimentary allocation, you will be able to upgrade to a paid Stream subscription from the Stream Dashboard.

Cloudflare Stream simplifies storage, encoding and playback of videos. You can use the free allocation of Cloudflare Stream for various use cases, such as background/hero videos, e-commerce product videos, how-to guides and customer testimonials.

Upload videos with no code

To upload your first video Stream, simply visit the Stream Dashboard and drag-and-drop the video file:

Lights, Camera, Action! Business and Pro customers get bundled streaming video

Once you upload a video, Stream will store and encode your video. Stream automatically optimizes your video uploads by creating multiple versions of it at different quality levels. This happens behind-the-scenes and requires no extra effort from your side. The Stream Player automatically selects the optimal quality level based on your website visitor’s Internet connection using a technology called adaptive-bit rate encoding.

Your uploaded video will appear on the Dashboard:

Lights, Camera, Action! Business and Pro customers get bundled streaming video

Click on the video in the list of videos to watch a preview, change settings or to grab the embed code:

Lights, Camera, Action! Business and Pro customers get bundled streaming video

Built-in Stream Player

Stream provides an embed code that can be used to place your uploaded videos onto your website. The embed code can be found under the Embed tab:

Lights, Camera, Action! Business and Pro customers get bundled streaming video

To include the video on your website, simply copy-and-paste the embed code.

You’ll notice in the screenshot above that the Embed tab lets you customize the viewing experience. It supports the following optional properties:

  • Poster: The “poster image” is what appears on the video player before the user has started playing the video. By default, the poster image is set to the first frame in the video. However, you can change it by specifying another point in time or by specifying a URL to an image.
  • Start Time: Let’s say you have a 10-minute instructional video and your customer writes in with a question that is answered in that video at the 8-minute mark. You can use the Start Time property to have the video playback begin at the 8-minute mark, so your customer with a specific question does not need to watch 8 minutes of the video wondering “when will it answer my question?”. Instead, you can share a link with the customer that begins the video playback at the 8-minute mark.
  • Default Text Track: You can upload caption files for multiple languages for a given video. By default, captions are turned off. But if you want the captions to always render when the video plays, you can choose the default language from the Default Text Track dropdown.
  • Primary Color: You can choose your brand’s primary color and have it applied to various elements within the player, including the play button and the seek bar. Here is an example of the Stream Player with the Primary Color property configured to the Cloudflare orange:
Lights, Camera, Action! Business and Pro customers get bundled streaming video
Lights, Camera, Action! Business and Pro customers get bundled streaming video

Much, much more…

We live in a video-first world. Many Cloudflare customers already upload their videos to free video hosting services for marketing purposes. However, when you embed a video on your website that is hosted on a free video sharing service, your users often have to engage with unwelcomed ads and pixel trackers. Our hope is that by offering a free tier of Stream to Biz and Pro customers, you can use video to show off your products in a way that respects your users’ privacy and reflects your brand identity.

In addition to the features described in this announcement, Cloudflare Stream includes many more features including:

  • Dynamic Thumbnail Generation
  • Multi-language Captions
  • Live Streaming
  • Analytics

For a comprehensive list of features and how to use them, check out the Cloudflare Stream Docs.

The home page for Internet insights: Cloudflare Radar 2.0

Post Syndicated from Joao Sousa Botto original https://blog.cloudflare.com/radar2/

The home page for Internet insights: Cloudflare Radar 2.0

The home page for Internet insights: Cloudflare Radar 2.0

Cloudflare Radar was launched two years ago to give everyone access to the Internet trends, patterns and insights Cloudflare uses to help improve our service and protect our customers.

Until then, these types of insights were only available internally at Cloudflare. However, true to our mission of helping build a better Internet, we felt everyone should be able to look behind the curtain and see the inner workings of the Internet. It’s hard to improve or understand something when you don’t have clear visibility over how it’s working.

On Cloudflare Radar you can find timely graphs and visualizations on Internet traffic, security and attacks, protocol adoption and usage, and outages that might be affecting the Internet. All of these can be narrowed down by timeframe, country, and Autonomous System (AS). You can also find interactive deep dive reports on important subjects such as DDoS and the Meris Botnet. It’s also possible to search for any domain name to see details such as SSL usage and which countries their visitors are coming from.

Since launch, Cloudflare Radar has been used by NGOs to confirm the Internet disruptions their observers see in the field, by journalists looking for Internet trends related to an event in a country of interest or at volume of cyberattacks as retaliation to political sanctions, by analysts looking at the prevalence of new protocols and technologies, and even by brand PR departments using Cloudflare Radar data to analyze the online impact of a major sports event.

Cloudflare Radar has clearly become an important tool for many and, most importantly, we find it has helped shed light on parts of the Internet that deserve more attention and investment.

The home page for Internet insights: Cloudflare Radar 2.0

Introducing Cloudflare Radar 2.0

What has made Cloudflare Radar so valuable is that the data and insights it contains are unique and trustworthy. Cloudflare Radar shows aggregate data from across the massive spectrum of Internet traffic we see every day, presenting you with datasets you won’t find elsewhere.

However, there were still gaps. Today, on the second anniversary of Cloudflare Radar, we are launching Cloudflare Radar 2.0 in beta. It will address three common pieces of feedback from users:

  • Ease of finding insights and data. The way information was structured on Cloudflare Radar made finding information daunting for some people. We are redesigning Cloudflare Radar so that it becomes a breeze.
  • Number of insights. We know many users have wanted to see insights about other important parts of the Internet, such as email. We have also completely redesigned the Cloudflare Radar backend so that we can quickly add new insights over the coming months (including insights into email).
  • Sharing insights. The options for sharing Cloudflare Radar insights were limited. We will now provide you the options you want, including downloadable and embeddable graphs, sharing to social media platforms, and an API.

Finding insights and data

On a first visit to the redesigned Cloudflare Radar homepage one will notice:

  • Prominent and intuitive filtering capabilities on the top bar. A global search bar is also coming soon.
  • Content navigation on the sidebar.
  • Content cards showing glanceable and timely information.
The home page for Internet insights: Cloudflare Radar 2.0

The content you find on the homepage are what we call “quick bytes”. Those link you to more in-depth content for that specific topic, which can also be found through the sidebar navigation.

At the top of the page you can search for a country, autonomous system number (ASN), domain, or report to navigate to a home page for that specific content. For example, the domain page for google.com:

The home page for Internet insights: Cloudflare Radar 2.0

The navigation sidebar allows you to find more detailed insights and data related to Traffic, Security & Attacks, Adoption & Usage, and Domains. (We will be adding additional topic areas in the future.) It also gives you quick access to the Cloudflare Radar Outage Center, a tool for tracking Internet disruptions around the world and to which we are dedicating a separate blog post, and to Radar Reports, which are interactive deep dive reports on important subjects such as DDoS and the Meris Botnet.

The home page for Internet insights: Cloudflare Radar 2.0

Within these topic pages (such as the one for Adoption & Usage shown above), you will find the quick bytes for the corresponding topic at the top, and quick bytes for related topics on the right. The quick bytes on the right allow you to quickly glance at and navigate to related sections.

In the middle of the page are the more detailed charts for the topic you’re exploring.

Sharing insights

Cloudflare Radar’s reason to exist is to make Internet insights available to everyone, but historically we haven’t been as flexible as our users would want. People could download a snapshot of the graph, but not much more.

With Cloudflare Radar 2.0 we will be introducing three major new ways of using Radar insights and data:

  • Social share. Cloudflare Radar 2.0 charts have a more modern and clean look and feel, and soon you’ll be able to share them directly on the social media platform of your choice. No more dealing with low quality screenshots.
  • Embeddable charts. The beautiful charts will also be able to be embedded directly into your webpage or blog – it will work just like a widget, always showing up-to-date information.
  • API. If you like the data on Cloudflare Radar but want to manipulate it further for analysis, visualization, or for posting your own charts, you’ll have the Cloudflare Radar API available to you starting today.

For example, the last seven days of HTTP traffic data for Portugal can be obtained from https://api.cloudflare.com/client/v4/radar/http/timeseries/device_type?dateRange=7d&location=PT

Note: The API is available today. To use the Cloudflare API you need an API token or key (more details here). Embedding charts and sharing directly to social are new features to be released later this year.

Technology changes

Cloudflare Radar 2.0 was built on a new technology stack; we will write a blog post about why and how we did it soon. A lot changed: we now have proper GraphQL data endpoints and a public API, the website runs on top of Cloudflare Pages and Workers, and we’re finally doing server-side rendering using Remix. We adopted SVG whenever possible, built our reusable data visualization components system, and are using Cosmos for visual TDD. These foundational changes will provide a better UX/UI to our users and give us speed when iterating and improving Cloudflare Radar in the future.

We hope you find this update valuable, and recommend you keep an eye on radar.cloudflare.com to see the new insights and topics we’ll be adding regularly. If you have any feedback, please send it to us through the Cloudflare Community.

The status page the Internet needs: Cloudflare Radar Outage Center

Post Syndicated from David Belson original https://blog.cloudflare.com/announcing-cloudflare-radar-outage-center/

The status page the Internet needs: Cloudflare Radar Outage Center

The status page the Internet needs: Cloudflare Radar Outage Center

Historically, Cloudflare has covered large-scale Internet outages with timely blog posts, such as those published for Iran, Sudan, Facebook, and Syria. While we still explore such outages on the Cloudflare blog, throughout 2022 we have ramped up our monitoring of Internet outages around the world, posting timely information about those outages to @CloudflareRadar on Twitter.

The new Cloudflare Radar Outage Center (CROC), launched today as part of Radar 2.0, is intended to be an archive of this information, organized by location, type, date, etc.

Furthermore, this initial release is also laying the groundwork for the CROC to become a first stop and key resource for civil society organizations, journalists/news media, and impacted parties to get information on, or corroboration of, reported or observed Internet outages.

The status page the Internet needs: Cloudflare Radar Outage Center

What information does the CROC contain?

At launch, the CROC includes summary information about observed outage events. This information includes:

  • Location: Where was the outage?
  • ASN: What autonomous system experienced a disruption in connectivity?
  • Type: How broad was the outage? Did connectivity fail nationwide, or at a sub-national level? Did just a single network provider have an outage?
  • Scope: If it was a sub-national/regional outage, what state or city was impacted? If it was a network-level outage, which one?
  • Cause: Insight into the likely cause of the outage, based on publicly available information. Historically, some have been government directed shutdowns, while others are caused by severe weather or natural disasters, or by infrastructure issues such as cable cuts, power outages, or filtering/blocking.
  • Start time: When did the outage start?
  • End time: When did the outage end?

Using the CROC

Radar pages, including the main landing page, include a card displaying information about the most recently observed outage, along with a link to the CROC. The CROC will also be linked from the left-side navigation bar

The status page the Internet needs: Cloudflare Radar Outage Center

Within the CROC, we have tried to keep the interface simple and easily understandable. Based on the selected time period, the global map highlights locations where Internet outages have been observed, along with a tooltip showing the number of outages observed during that period. Similarly, the table includes information (as described above) about each observed outage, along with a link to more information. The linked information may be a Twitter post, a blog post, or a custom Radar graph.

The status page the Internet needs: Cloudflare Radar Outage Center
The status page the Internet needs: Cloudflare Radar Outage Center

As mentioned in the Radar 2.0 launch blog post, we launched an associated API alongside the new site. Outage information is available through this API as well — in fact, the CROC is built on top of this API. Interested parties, including civil society organizations, data journalists, or others, can use the API to integrate the available outage data with their own data sets, build their own related tools, or even develop a custom interface.

Information about the related API endpoint and how to access it can be found in the Cloudflare API documentation.

We recognize that some users may want to download the whole list of observed outages for local consumption and analysis. They can do so by clicking the “Download CSV” link below the table.

The status page the Internet needs (coming soon)

Today’s launch of the Cloudflare Radar Outage Center is just the beginning, as we plan to improve it over time. This includes increased automation of outage detection, enabling us to publish more timely information through both the API and the CROC tool, which is important for members of the community that track and respond to Internet outages. We are also exploring how we can use synthetic monitoring in combination with other network-level performance and availability information to detect outages of popular consumer and business applications/platforms.

And anyone who uses a cloud platform provider (such as AWS) will know that those companies’ status pages take a surprisingly long time to update when there’s an outage. It’s very common to experience difficulty accessing a service, see hundreds of messages on Twitter and message boards about a service being down, only to go to the cloud platform provider’s status page and see everything green and “All systems normal”.

For the last few months we’ve been monitoring the performance of cloud platform providers to see if we can detect when they go down and provide our own, real time status page for them. We believe we can and Cloudflare Radar Outage Center will be extended to include cloud service providers and give the Internet the status page it needs.

The status page the Internet needs: Cloudflare Radar Outage Center

If you have questions about the CROC, or suggestions for features that you would like to see, please reach out to us on Twitter at @CloudflareRadar.

Gateway + CASB: alphabetti spaghetti that spells better SaaS security

Post Syndicated from Alex Dunbrack original https://blog.cloudflare.com/gateway-casb-in-action/

Gateway + CASB: alphabetti spaghetti that spells better SaaS security

This post is also available in 简体中文 and Español.

Gateway + CASB: alphabetti spaghetti that spells better SaaS security

Back in June 2022, we announced an upcoming feature that would allow for Cloudflare Zero Trust users to easily create prefilled HTTP policies in Cloudflare Gateway (Cloudflare’s Secure Web Gateway solution) via issues identified by CASB, a new Cloudflare product that connects, scans, and monitors your SaaS apps – like Google Workspace and Microsoft 365 – for security issues.

With Cloudflare’s 12th Birthday Week nearing its end, we wanted to highlight, in true Cloudflare fashion, this new feature in action.

Gateway + CASB: alphabetti spaghetti that spells better SaaS security

What is CASB? What is Gateway?

To quickly recap, Cloudflare’s API-driven CASB offers IT and security teams a fast, yet effective way to connect, scan, and monitor their SaaS apps for security issues, like file exposures, misconfigurations, and Shadow IT. In just a few clicks, users can see an exhaustive list of security issues that may be affecting the security of their SaaS apps, including Google Workspace, Microsoft 365, Slack, and GitHub.

Cloudflare Gateway, our Secure Web Gateway (SWG) offering, allows teams to monitor and control the outbound connections originating from endpoint devices. For example, don’t want your employees to access gambling and social media websites on company devices? Just block access to them in our easy-to-use Zero Trust dashboard.

The problems at hand

As we highlighted in our first post, Shadow IT – or unapproved third-party applications being used by employees – continues to be one of the biggest pain points for IT administrators in the cloud era. When employees grant access to external services without the consent of their IT or security department, they risk granting bad actors access to some of the company’s most sensitive data stored in these SaaS applications.

Another major issue affecting the security of data stored in the cloud is file exposure in the form of oversharing. When an employee shares a highly sensitive Google Doc to someone via a public link, would your IT or security team know about it? And even if they do, do they have a way to minimize the risk and block access to it?

With these two products now being used by customers around the world, we’re excited to share how visibility and basic awareness of SaaS security issues doesn’t have to be the end of it. What are admins supposed to do next?

Gateway + CASB: blocking identified threats in three (yes, three) clicks

Now, when CASB discovers a problem (which we call a Finding), it’s now possible to easily create a corresponding Gateway policy in as few as three clicks.

This means users can now automatically generate fine-grained Gateway policies to prevent specific inappropriate behavior from continuing, while still allowing for expected access and usage that meets company policy.

Example 1: Block employees from uploading to their personal Google Drive

Gateway + CASB: alphabetti spaghetti that spells better SaaS security

A common use case we heard during CASB’s beta program was the tendency for employees to upload corporate data – documents, spreadsheets, files, folders,  etc. – to their personal Google Drive (or similar) accounts, presenting the risk of intellectual property making its way out of a secure corporate environment. With Gateway and CASB working together, IT administrators can now directly block upload activity from anywhere other than their corporate tenant of Google Drive or Microsoft OneDrive.

Example 2: Restrict repeat oversharers from uploading and downloading files

Gateway + CASB: alphabetti spaghetti that spells better SaaS security

A great existing use case of Cloudflare CASB has been the ability to identify employees that are habitual oversharers of files in their corporate Google or Microsoft tenants – sharing files to anyone that has the link, sharing files with emails outside their company, etc.

Now when these employees are identified, CASB admins can create Gateway policies to block specific users from further upload and download activity until the behavior has been addressed.

Example 3: Prevent file uploads to unapproved, Shadow IT applications

Gateway + CASB: alphabetti spaghetti that spells better SaaS security

To address the concern of Shadow IT, CASB-originating Gateway policies can be customized, including being able to restrict upload and download events to only the SaaS applications your organization uses. Let’s say your company uses Box as its file storage solution; in just a few clicks, you can use an identified CASB Finding to create a Gateway policy that blocks activity to any file sharing application other than Box. This gives IT and security admins the peace of mind that their files will only end up in the approved cloud application they use.

Get started today with the Cloudflare Zero Trust

Ultimately, the power of Cloudflare Zero Trust comes from its existence as a single, unified platform that draws strength from its combination of products and features. As we continue our work towards bringing these new and exciting offerings to market, we believe that it’s just as important to highlight their synergies and associated use cases, this time from Cloudflare Gateway and CASB.

For those not already using Cloudflare Zero Trust, don’t hesitate to get started today – see the platform yourself with 50 free seats by signing up here.

For those who already know and love Cloudflare Zero Trust, reach out to your Cloudflare sales contact to get started with CASB and Gateway. We can’t wait to hear what interesting and exciting use cases you discover from this new cross-product functionality.

Goodbye, Alexa. Hello, Cloudflare Radar Domain Rankings

Post Syndicated from Celso Martinho original https://blog.cloudflare.com/radar-domain-rankings/

Goodbye, Alexa. Hello, Cloudflare Radar Domain Rankings

Goodbye, Alexa. Hello, Cloudflare Radar Domain Rankings

The Internet is a living organism. Technology changes, shifts in human behavior, social events, intentional disruptions, and other occurrences change the Internet in unpredictable ways, even to the trained eye.

Cloudflare Radar has long been the place to visit for accessing data and getting unique insights into how people and organizations are using the Internet across the globe, as well as those unpredictable changes to the Internet.

One of the most popular features on Radar has always been the “Most Popular Domains,” with both global and country-level perspectives. Domain usage signals provide a proxy for user behavior over time and are a good representation of what people are doing on the Internet.

Today, we’re going one step further and launching a new dataset called Radar Domain Rankings (Beta). Domain Rankings is based on aggregated 1.1.1.1 resolver data that is anonymized in accordance with our privacy commitments. The dataset aims to identify the top most popular domains based on how people use the Internet globally, without tracking individuals’ Internet use.

There are a few reasons why we’re doing this now. One is obviously to improve our Radar features with better data and incorporate new learnings. But also, ranking lists are used all over the Internet in all sorts of systems. One of the most used and trusted sources of domain rankings was Alexa, but that service was recently deprecated. We believe we are in a good position to provide a strong alternative.

Let’s see how we built it.

Differences in domain names

Before we dig into the data science behind Domain Rankings, it’s important to understand what a domain and DNS are. Internet domain names are human-readable dot-separated letters, digits and hyphens that correspond to a network resource, like a server or a website. However, your computer and applications don’t know what to do with a domain name; they need IP addresses to send and receive information over the network. DNS is the system that converts, or resolves, a domain name into an IP address. Think of it as an Internet phonebook for domain names.

Note: This is a simplification. A new standard called Internationalized Domain Names, or IDN, allows using Unicode strings in domain names.

Each dot defines a new hierarchy level, reading right to left. Domains can have multiple levels of depth. The highest level corresponds to country code top-level domains (ccTLDs) like .uk, .fr or .pt, or generic top-level domains (gTLDs) like .com, .org, or .net. These are normally assigned to and managed by either country-level entities or administrative organizations operating a registry.

Then there are the second-level domains like cloudflare.com or google.com. These are normally purchased and registered by individuals or organizations, which are then free to create and manage as many hostnames and hierarchy levels as they want.

Unfortunately, however, there are exceptions. For instance, many countries use second-level domain registration. One such example is the United Kingdom, where commercial domains can only be registered under the .co.uk hierarchy. That’s why Google in the UK isn’t google.uk, but rather google.co.uk.

But that’s not all. Some countries use 3rd level domain registrations. One example is Japan, which offers Regional Domain registration under cities like *.aisai.aichi.jp.

Projects like the Public Suffix List are a good starting point for understanding the variations involved, and how they affect validations and assumptions in other systems, such as cookies in web browsers.

Domain Rankings takes some of this nuance into account to inform the definition of our current ruleset:

  • We boil everything down to second-level domains, such as cloudflare.com or google.com.
  • However, if the second level is .edu, .com, .org, .gov, .net, .gov, .net, .co or .mil, then we use third-level domains.
  • We don’t distinguish between what we think is a website or an infrastructure system. A domain represents an Internet-available resource.
  • We will also semi-automate, curate and maintain a list of domains that map to popular platforms and services in the future. Example: fb.audio, fb.com, fb.watch, all map to a “facebook” platform.

Defining popularity

Definitions are important. We established what we consider a domain, but what does domain popularity mean exactly? Our research showed that the volume of traffic generated to a given domain doesn’t really work as a proxy for what we perceive as popular. Instead, Domain Rankings looks at the size of the population of users that look up a domain per unit of time. The more people who are interested in a domain, the more popular it is.

Sounds pretty straightforward, right? Well, it’s not. Our databases don’t have cookies, IPs, or other tracking artifacts, and we strip information that leads to identifying an individual from all of our data, by design.

The good news, however, is that we do a very good job at identifying automated traffic (for instance, you can read about Bot Management and how we use Machine Learning to detect bots in HTTP traffic in our blog) and we found we could develop a reasonable proxy for the unique users metric without sacrificing privacy (using other data points that we store for a limited period of time, like the ASN and high-level geolocation information of the request or the Cloudflare data center that served it).

Domain Rankings’ popularity metric is best described as the estimated relative size of the user population that accesses a domain over some period of time.

Our approach

We announced 1.1.1.1, our privacy-first consumer DNS resolver in 2018, and over the years it’s grown to become one of the top DNS services in the world. 1.1.1.1 is also part of a Research Agreement with APNIC in which we collaborate with them doing public research and DNS data insights.

The data we collect from it honors our privacy commitments, and is aggregated and stripped of any information that could lead to identifying or tracking users. We conducted a privacy examination by a Big Four accounting firm to determine whether the 1.1.1.1 resolver was effectively configured to meet our privacy commitments. You can read more about it in this blog, and the full report is publicly available on our compliance page.

Even without this personally identifying information, the resulting collection is vast and representative of Internet activity.

The 1.1.1.1 service is used in many ways. Regular (human) Internet users use it as their DNS resolver, either because they explicitly configured it in their devices, or their ISP did, or because they use WARP, or their browser uses 1.1.1.1 under the hood. However, servers and cloud infrastructure, IoT devices, home routers, and bots also use 1.1.1.1 extensively, which creates a lot of challenges for us when trying to identify human traffic.

We’ve been using DNS data to calculate the top and trending domains found on both the global and country pages on Cloudflare Radar. It’s been quite a learning experience trying to improve these lists. We have implemented aggregations, counts, filters, handling exceptions, and tried reducing noise, and yet they’re far from perfect. We felt that there had to be a better way.

We’ve spent the last six months building a variety of machine learning models to help us predict the rank of a domain.

Building the model was no easy feat. We experimented with multiple regression types first, to know exactly what the model was doing, and then more complex algorithms to get better performance. We played with different datasets, changed the population groups, variables (features), and combinations of variables, and used synthetic data.

After evaluation, one of our first conclusions was that building a model that could produce good results for the highest ranked domains and the long tail would be difficult.

The paper “A Long Way to the Top: Significance, Structure, and Stability of Internet Top Lists” describes this problem well. “The ranking of domains in the long tail should be based on significantly smaller and hence less reliable numbers.” Talking to our Research Team who submitted the collaboration paper “Toppling Top Lists: Evaluating the Accuracy of Popular Website Lists” to IMC 2022, got us to the same conclusion: the most popular domains (like google.com and facebook.com) have feature values disproportionally higher than the lower-ranked domains.

Therefore, we selected the two models that performed best. One model was trained on the population with the highest feature values, uses more features, and is used to generate the ordered top 100 domain list. A second model was trained on a more general group of domains, uses fewer features, and is used to get the top one million most popular domains, which we then divide into ranking buckets.

These buckets are ranked, but each bucket’s contents are intentionally unordered. For example, the second bucket of 10,000 most popular domains includes the set of domains that rank from 10,001 to 20,000, but give no further indication of the individual ranking of domains in that bucket. Given the size of some of these buckets and the window of time we use to populate them, they will inherently be exposed to more instability, too. We feel this is a good compromise between the described natural uncertainties of our long tail model and providing a reasonable idea of how close to the top a domain is.

Results

It’s important to mention there is no global view that can establish the perfect rank, and there’s no easy mechanism to confirm if a ranking is, ultimately, good. Data-driven results are always subject to some bias and skewing, related to the context of the organizations and systems that collect them. Sometimes all that can be done is to be transparent about potential sources of bias. The geographical distribution of customers and users, product characteristics, platform features, and behavioral diversity play an essential role in the final result. We are presenting the Cloudflare view, what we see.

Having said this, Cloudflare sits in a privileged position and handles a significant amount of Internet traffic. We have plenty of signals we can extract from our aggregated data, and believe that makes it possible to generate high quality domain rankings.

Domain Rankings are available today. You can head up to the Domains page and check it out:

  • Ordered list of the top 100 most popular domains globally and per country, based on our first model. Last 24 hours, updated daily.
  • Unordered global most popular domains datasets divided into buckets of the following sizes: 200, 500, 1,000, 2,000, 5,000, 10,000, 20,000, 50,000, 100,000, 200,000, 500,000, 1,000,000. Last 7 days, updated weekly.
Goodbye, Alexa. Hello, Cloudflare Radar Domain Rankings

Next steps

We will keep improving Domain Rankings and monitoring the results. Anyone can access them on Cloudflare Radar, read the results, and download the CSV files.

Feel free to explore our Domain Rankings and share feedback with us.

Backups to the rescue!

Post Syndicated from Nathan Liefting original https://blog.zabbix.com/backups-to-the-rescue/23442/

In this blog post, you will learn how to set up backups for your Zabbix environment. There’s a wide variety of different options when it comes to taking backups of our Zabbix environment, for us, it will just be a matter of choosing the right fit.

 

Introduction

Monitoring is an important part of our IT infrastructure and often times when our monitoring isn’t working for a certain period, we feel like we are blind as to what is going on with our different IT components. As such, taking backups of our Zabbix environment is an important part of running a production Zabbix environment, as we do want to be prepared for a possible issue that might corrupt or even lose our data. It’s always a possibility and as such we should be prepared.

For Zabbix, there are a few different methods on how to take backups and it all starts at the database level. Both the Zabbix frontend as well as the Zabbix server write their data into the Zabbix database as we can see in the illustration below:

This means that both our configuration as well as all of our collected values are present in the same Zabbix database and if we take a database backup, we back up (almost) everything we need. So, let’s start there and have a look at how we can make a database backup.

How to

MySQL backups

Let’s start with the most used variant of Zabbix databases: MySQL and it’s forks like MariaDB and Percona. All of them can easily be backed up using built-in functionality like the MySQL Dump command and we can then use other industry standards to get things going. First, we have to understand the tables in our database though. Most of the tables in your Zabbix environment contain configuration data and as such, they are all important to backup. There are a few tables that we need to consider, however, as they can contain Giga or even Terabytes of data. These are the History, Trends and Events tables:

It is possible to omit these tables from your backup and make smaller, more manageable backups. To make the backup we can then start using tools like MySQL Dump:

Once we have taken a backup, we can easily import that back into our environment using the MySQL Import command or simply using the cat command:

Do not forget, taking and importing large backups can take a long time. This completely depends on your MySQL database performance tuning settings as well as the underlying resources like CPU, Memory and Disk I/O. Also, make sure to check out the MySQL documentation:

MySQL Dump:  https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html / https://mariadb.com/kb/en/making-backups-with-mysqldump/

MySQL Import: https://dev.mysql.com/doc/refman/8.0/en/mysqlimport.html / https://mariadb.com/kb/en/mysqlimport/

 

Alternatively, it’s also possible to create backups using tools like xtrabackup and mariadbbackup.

PostgreSQL backups

We can actually use the same kinds of methods for the PostgreSQL backups. Keep the required tables in mind and fire away with the built-in tools:

 

Then we can restore it by loading the file into postgres:

What about the configuration files?

Once we have a database backup, everything is backed up, right? Well, almost everything. With just a database backup we are quite safe, but (and this is oftentimes overlooked) there are a lot of configuration files and perhaps even custom scripts we need to take into account! There are three parts to this story – the Zabbix server, the Zabbix frontend, and also the Zabbix additional components. All of them have their own set of configuration files and locations that are used for storing custom scripts.

The Zabbix frontend location and configuration files can be different, depending on the environment, as we have a few choices to make. Are we running Apache or Nginx? On what Linux distribution? All of these have to be considered when making configuration backups. In general, the locations for the configuration would be:

/etc/nginx/
/etc/httpd/
/etc/apache2

There’s also a symlink to the Zabbix frontend configuration file located in /etc/zabbix/ but we will get to that one in a bit.

Then we have the Zabbix server itself, which keeps its configuration in /etc/zabbix/ and if we’re following best practices any script should be placed in /usr/lib/zabbix. So we need:

/etc/zabbix/
/usr/lib/zabbix

Let’s add them to the list and find a method to back up these files. Crontab is a built-in tool that we can use, but there are definitely other (perhaps better) solutions out there. Let’s add the following to cron:

I also added a find command here, which will serve as our roll-over or rotation toll. It will find files older than 180 days and delete them from /mnt/backup/config_files/. Make sure to pick a good (network) folder to store these files as it’s important to keep these safe. Feel free to change the number of days you’d like to store the files for.

What about the additional components like Zabbix proxy, Zabbix Java gateway and Zabbix web service (used for PDF reporting)?. Well, these have configuration files as well. Make sure to run a backup on the devices running these additional components. As for Zabbix proxies – they have the same file locations as Zabbix server:

For Zabbix Java gateway and Zabbix web service, we can omit the /usr/lib/zabbix/ folder.

Don’t forget the import/export files!

In general, database backups are slow to make, but also slow to import back unless we do not include the history/trends in the backup. But even then, restoring an entire database simply because someone made an error on a single template is a hassle. Zabbix ships with the built-in frontend export functionality, allowing us to export (and then import) entire parts of the configuration instantly! We can use these for a number of different parts of the configuration:

  • Hosts
  • Templates
  • Media types
  • Maps
  • images
  • Host groups (API ONLY)
  • Template groups (API ONLY)

All of these are available through the Zabbix API allowing us to choose whether we do a manual configuration backup from the frontend, as well as providing us with automation options using that API. You could even manage and update your Zabbix configuration from GIT entirely if you write the right scripts for this.

Frontend backups

To run an export from the frontend simply go to one of the supported sections like Configuration | Templates and select the export data format. When selecting multiple entities, keep in mind that they will all be exported to a single file.

We can then make our edits and import files from the frontend as well:

For Templates this will even result in a nice diff pop-up window, detailing all the changes, deletes and additions to the templates:

 

API backups

For the API things get a little more complicated as we need to select a mode of execution. Of course, it’s possible to do a curl command from the CLI or even use something like Postman:

Request body

The response will then look something like this:

But this feature really starts to shine once we combine it with our own automation scripts. Use it wisely!

High availability

So, what about high availability? Isn’t that some form of a backup?

Well yes and no. High availability is not an “IT backup” in the form of making sure we can recover something that is broken. But it is a backup in the way that if a Zabbix server instance fails, another one takes over for it. HA is somewhat out of scope for this blog post, but it’s still worth mentioning. There are several solutions to set up Zabbix as a full high availability cluster. For MySQL we can use a Primary/Primary setup, for the frontend we can use load balancing techniques like HAProxy and for the Zabbix server, we can use the built-in high availability method. Combine all of these together and you’ll definitely be able to serve your every (production ready!) need.

Conclusion

To conclude, there are many options to start taking backups of our Zabbix environment. It all starts at the database and these backups are definitely vital to keep things safe in case of disaster. When making the backups, do not forget about the configuration files and custom scripts as well as the frontend backup option. Combining all of these solutions will safeguard our environment, but if that isn’t enough – do not forget about industry standards like snapshots. Even further safeguarding our environment on multiple levels.

I hope you enjoyed reading this blog post. If you have any questions or need help configuring anything on your Zabbix setup feel free to contact me and the team at Opensource ICT Solutions. We build a ton of cool integrations like this and much more!

Nathan Liefting

https://oicts.com

A close up of a logo Description automatically generated

The post Backups to the rescue! appeared first on Zabbix Blog.

Защо резултатите от изборите ще са служебни

Post Syndicated from Светла Енчева original https://toest.bg/zashto-rezultatite-ot-izborite-shte-sa-sluzhebni/

„Служебна“ се нарича победа, която не е постигната в състезание, а е присъдена по други, формални критерии. Спорно е дали предстоящите парламентарни избори на 2 октомври ще излъчат отчетлив победител, който да е в състояние да сформира правителство, дори в коалиция. Сигурно е обаче, че те ще произведат все някакви резултати. Акцентът тук е върху „все някакви“. Проблемът е в прецизността, с която ще се измерят. По-точно – в липсата ѝ.

Колко и как са разпределени имащите право на глас?

Отговорът на този въпрос е от съществено значение, защото избирателната активност представлява делът на гласувалите от всички, които имат право на глас. А избирателната активност определя легитимността на бъдещите управляващи, или колко от българските граждани са реално представени в парламента. По нея се мери и тежестта на купения и контролирания вот. Броят на имащите право на глас определя и броя на мандатите (депутатските места) от съответния избирателен район. Ако те са изчислени на неточна база, резултатите са „на око“.

По предварителни данни от преброяването на населението през 2021 г. в България живеят около 6,5 милиона души. В тях влизат не само имащите право на глас, а и непълнолетните, чужденците без българско гражданство, лишените от свобода с влязла в сила присъда, поставените под запрещение – все групи, чиито представители не могат да гласуват.

Официалните резултати от преброяването ще се огласят в особен момент.

Националният статистически институт ще обяви първите официални резултати от преброяването „поради засилен интерес“ на 3 октомври. Това е тъкмо в деня след изборите. Едва тогава ще стане ясно какви биха били резултатите, ако организацията на гласуването беше съобразена с актуалните данни за гласоподавателите на България.

Междувременно според Централната избирателна комисия имащите право на глас са 6,6 милиона, тоест повече от цялото население на България според предварителните резултати от преброяването. От ЦИК обясняват несъответствието със следния аргумент:

6 милиона и 600 хиляди са всички български граждани, които имат избирателни права, във всички части на света. Ако сравняваме с преброяването – в него участват всички граждани на територията на България.

Ала откъде ЦИК знае колко са българските граждани в чужбина? И съответно тези от тях с право на глас?

Държавата ни няма инструмент, с който да води точен регистър на броя им. А и като се замислим, това не е лесна работа. Много български граждани може изобщо да не си направят труда да уведомят държавата, че живеят в чужбина. Още по-малко – че се намират извън родната си страна към момента на провеждане на изборите.

България няма как да изиска достоверна информация от властите на всички държави за това колко български граждани се намират на територията им. А такава информация е възможно и да липсва – например ако наш сънародник пребивава в страна членка на ЕС, може до три месеца да не регистрира престоя си. На практика – и доста по-дълго. Ако български гражданин в чужбина почине, българските институции може така и да не разберат.

За броя на гласоподавателите, съобщен от ЦИК, има по-логично обяснение.

Избирателните списъци се съставят на основата на данните на Главна дирекция „Гражданска регистрация и административно обслужване“ (ГРАО) към Министерството на регионалното развитие и благоустройството. Според последните данни на ГРАО от 15 септември 2022 г. с постоянна адресна регистрация в България са 8 258 712 души. Това е сборът на регистрираните по постоянен адрес във всички 28 области на страната. Като извадим от тях „на око“ нямащите право на глас, обявените от ЦИК 6,6 милиона придобиват известен смисъл.

Защо регистрираните в системата на ГРАО са с над 1,7 милиона повече от населението на България според преброяването, като имаме предвид, че то поне на теория обхваща и бездомните, а не само тези с адресна регистрация? НСИ се опитва да установи колко хора действително живеят в България. Мнозина имат постоянен адрес, на който не живеят. Това важи и за немалка част от българите в чужбина. Други от тях обаче не са адресно регистрирани у нас. Така че тези 1,7 милиона не отговарят непременно на броя на сънародниците ни извън страната.

А колко са точни резултатите от преброяването?

В България отдавна не живеят над 8 милиона души, но каква е гаранцията, че при последното преброяване е установен реалният брой на населението? Много хора не са взели участие в преброяването – било от недоверие, било заради пандемията, било защото хазяите са им забранили, защото се страхуват, че така може да ги хванат, че не плащат данъци (това впрочем е широко разпространен проблем), било по други причини.

Според НСИ преброените към предпоследния ден на преброителната кампания са 6,62 милиона, или около 82% от населението. Останалите ще бъдат „обхванати по административен път“, тоест вероятно чрез бази данни като тази на ГРАО. Възниква обаче логичният въпрос – откъде НСИ знае колко е населението, че да може да изчисли 82% от него? Та нали целта на преброяването е именно да се установи колко и какви хора живеят в България? Ако това е предварително известно на НСИ, защо е цялото усилие? (Друг е въпросът за някои от формулировките в преброителната карта, които изначално гарантират некоректни резултати по някои признаци – като например етнос, майчин език или религия.)

И все пак, при цялата си условност, резултатите от преброяването по всяка вероятност са по-коректни от данните на ГРАО, защото има поне опит за връзка с реалното население на страната. С помощта на преброяването се добива по-ясна представа и за разпределението на хората в България по райони – нещо, което тромавата система на адресните регистрации изключително изкривява.

Кой губи от неточната статистика на населението?

Ако разпределението на избирателните мандати отговаря на структурата на населението според последното преброяване, поне 15 мандата ще се разместят, а 11 от тях ще отидат в София, установява разследване на Полина Паунова в „Капитал“. Причината е, че от предишното преброяване през 2011 г. населението в цялата страна е намаляло, освен в София, където се е увеличило. Така, вместо 42-ма, депутатите от трите софийски избирателни района биха били общо 53-ма.

От това губи най-вече „Демократична България“, чиито гласоподаватели са концентрирани основно в София. На изборите за 46-тото Народно събрание ДБ дори е първа политическа сила в два от трите избирателни района в столицата. Губи и „Продължаваме промяната“ – на изборите за 47-мото НС най-големият дял от избирателите ѝ е в София.

От неточната статистика обаче губят и партии, които остават под ръба на 4-процентовата бариера. Понякога и една стотна от процента може да бъде решаваща за това дали една политическа сила ще има парламентарно представителство. Или ще вземе субсидия. Като се има предвид, че бариерата е повече или по-малко условна, обяснимо би било недоволството у некласираните.

Най-големите губещи обаче са избирателите. Защото те ще бъдат неадекватно представени не (само) заради спецификата на политическите партии и в резултат на собствения си избор, а и заради формални математически несъответствия. И колкото и да им се натяква, че сами са си виновни, поне за последното не са.

Заглавна снимка: Marco Verch, CC BY 2.0 / Flickr

Източник

Ексклузивни видео и снимки на “Биволъ” Изнеслите кеш от „Хемус“ в чували фирми гонят Биволъ от Видин – Ботевград

Post Syndicated from Николай Марченко original https://bivol.bg/%D0%B8%D0%B7%D0%BD%D0%B5%D1%81%D0%BB%D0%B8%D1%82%D0%B5-%D0%BA%D0%B5%D1%88-%D0%BE%D1%82-%D1%85%D0%B5%D0%BC%D1%83%D1%81-%D0%B2-%D1%87%D1%83%D0%B2%D0%B0%D0%BB%D0%B8-%D1%84%D0%B8%D1%80.html

петък 30 септември 2022


Компаниите, които са се облагодетелствали от авансите за АМ „Хемус“, без да са завършили строителните работи, гонят екипа на „Биволъ“ от строежа на скоростния път „Видин – Ботевград“. Медията ни…

The collective thoughts of the interwebz