Извън усвояването, съответствието и поносимостта. Оптимистично за ВиК отрасъла

Post Syndicated from Радослав Русев original https://toest.bg/optimistichno-za-vik-otrasula/

Има един сектор в икономиката ни, който рядко генерира положителни новини. По-рядко дори от енергетиката, здравеопазването и образованието, където проблемите са наистина дълбоки. Става дума за водоснабдяването и канализацията, накратко ВиК, като в тази абревиатура влиза и все по-важният (и скъп) компонент „пречистване на отпадъчните води“.

В бранша често цитираме две неприятни азбучни истини: 1) хората се сещат за ВиК-то най-вече когато водата спре, и 2) никой не знае как точно се сформира месечната му сметка за вода, но знае, че е „много висока“ и че „сега пак щели да увеличават цената“.

Ако потребителските нагласи са само една от гледните точки, то нека чуем какво казват и „отговорните институции“. Тук преобладаващите термини са „усвояване“, „съответствие“ и „поносимост“:

„Усвояването“ е процентът от средствата по Оперативна програма „Околна среда“ (ОПОС), тоест колко сме прибрали от ЕС спрямо заложения бюджет, без особено значение за какво сме ги похарчили…

„Съответствието“ е процентът изградени пречиствателни станции за отпадъчни води в градове над 2000 души – нищо че вече имаме областни градове с трайно нарушено питейно водоснабдяване, тоест обективно по-значими инвестиционни приоритети. „Тука водата от чешмата пиете ли я?“ е чест въпрос по тези места.

А „поносимостта“ е съотношението между сметката за вода на едно домакинство и нетния му разполагаем доход за същия период.

И не че тези неща нямат значение (особено последното), но вторачени в клишетата, като че ли забравяме базовите неща – да има вода, тя да става за пиене, да строим качествени тръбопроводи и резервоари с дълъг живот, да прилагаме съвременните технологии и секторът като цяло да е интересен и смислен за 20-те хиляди души (и още поне толкова подизпълнители), пряко ангажирани в него.

Настоящият момент

Нека погледнем и по-тясната конюнктура – какво става във ВиК сектора в края на 2021 г.

Регулаторът (КЕВР) разглежда бизнес планове и тарифи за следващия регулаторен период 2022–2026 г., тоест предложения за инвестиции и нови тарифи.

Навлизаме в новия 7-годишен рамков период на ЕС (2021–2027), но във ВиК отрасъла големите инвестиции по ОПОС за периода 2014–2020 г. тепърва ще се осъществяват заради закъснели подготвителни дейности.

Трети пореден за годината нов парламент ще предлага законодателни промени, като „цената на водата“ е обичаен сюжет за парламентарна активност. Пример: за краткия живот на последния парламент по темата бяха направени три различни законодателни предложения от различни парламентарни групи.

Междувременно рязко повишените цени на електроенергията буквално обричат на фалит част от дружествата, които имат висока консумация на електричество (Добрич, Шумен, Разград, Сливен и други, в някои от които тя може да достигне 50% от всички месечни разходи).

В тази ситуация е логично ВиК отрасълът да не генерира особено положителни новини.

Защо дяволът не е толкова черен

Ако твърде дълго се взираш в бездната, бездната ще се взре в теб.

Ницше

Че пред отрасъла има пропаст, има. Но дали е толкова черна и дълбока, до голяма степен зависи и от аршина на взиращия се. Наскоро говорих с външен за сектора чуждестранен експерт, който смело обобщи: „Абе, не че нямате проблеми, ама какво толкова мрънкате – вода имате (като ресурс в природата), повечето ви големи градове даже си имат язовири, скъпите съоръжения (сондажи, резервоари, станции, довеждащи водопроводи) са изградени, инженерите ви ги знаят по цял свят.“ И е доста прав…

Несъмнено и двете позиции (песимистичната и оптимистичната) имат своите основания. Ако си управител на ВиК дружество и на бюрото ти има непосилна сметка за електроенергия, чийто падеж наближава, логично е да виждаш пропаст. Ако си ВиК консултант или проектен ръководител, който се прибира от Северна Африка или Близкия изток, е лесно да кажеш, че „това не са сериозни проблеми“.

Както обикновено, истината е някъде между двете крайни позиции, но не непременно по средата. Личната ми позиция е, че реалното положение е много по-близко до розовата част на спектъра, поне в сравнение с наистина проблемните сектори в икономиката и обществения живот в България. И ако нещата останат дългосрочно зле, то основната причина е в нашата колективна глупост, алчност и късогледство, а не толкова в климата, бедността и старите тръби.

Какво може – и защо сега

Дори за най-черногледите ни сънародници е ясно, че в момента България има шанс да направи крачки в положителна посока, при това в различни области на обществения живот и икономиката. Бавно и от третия път, но вероятно по Коледа ще имаме ново правителство – и то натоварено със справедливи очаквания за промени.

Във ВиК отрасъла, както навсякъде другаде, е хубаво да внимаваме какво си пожелаваме, защото най-често то се сбъдва. И ако си пожелаем повече „усвояване“, да сме наясно, че това е нещо различно от „чиста вода за пиене“. Някъде тук е време тази статия да завие към няколко предложения. Без претенции, че това са най-важните (или пък спешни) приоритети пред сектора, но те ще ни отклонят от опасното и излишно вторачване в бездната.

  • Да свържем отрасъла с големите теми

От поне десетина години всевъзможни класации (например ежегодната класация на World Economic Forum) поставят водата в топ 3 на световните рискове. Отскоро обаче започват да се виждат някои интересни възможности, които надхвърлят секторната тематика, тоест идентифицират се ВиК решения на проблеми от други сфери на живота. Няколко конкретни примера:

Анализ на отпадъчните води с цел откриване на вируси и бактерии (т.нар. epidemiological surveillance). Тази тема придоби огромна популярност около многобройните проекти за проследяване на разпространението на новия коронавирус и неговите мутации. Истината е, че от години е разпространена технологията за откриване на огнища на други вируси и патогени (причиняващи хепатит, полиомиелит и др.), на прекомерна употреба на опиоиди и антибиотици, и други. Редица страни – Австрия, Холандия, Португалия, Великобритания – въведоха наблюдението на отпадъчни води като основен елемент в националните си планове за управление на пандемията. Вероятно променената Директива за пречистване на градските отпадъчни води (в момента тече обществена консултация) ще наложи подобен мониторинг като задължителен стандарт за страните членки на ЕС.

Гъвкаво управление на консумацията на електричество. Особено в условията на рязко поскъпнала електроенергия ВиК дружествата гледат на доставчиците (включително на т.нар. балансиращи групи) като на основна заплаха. Част от истината е обаче, че ВиК операторите и техните контролни зали могат да бъдат най-добрият приятел на енергийните търговци, като предоставят частична възможност за гъвкаво управление на консумацията. Повечето ВиК дружества разполагат с резервоари, които играят ролята на буфери, тоест потреблението на електричество от сондажи и помпени станции може да бъде интелигентно премествано поне с няколко часа, което да помогне на търговците на електроенергия да управляват своите баланси – ситуация, от която и двете страни могат да спечелят.

Използване на пречистени отпадъчни води за напояване. Идеята не е нова и се прилага в страни със сух климат, включително в Южна Европа (Испания, Португалия и др.). Технологично изглежда просто, но както обикновено, дяволът е в детайлите – да се оставят ли полезни елементи (фосфор и азот) във водата при пречистването, каква дезинфекция да се използва, какви са другите рискове, подходящите поливни култури и т.н. Посоката обаче е ясна и постепенно отпадъчните води (а и утайките) ще стават все по-ценен ресурс за селското стопанство.

Посочените примери са установени практики в редица страни. В ограничен формат подобни пилотни проекти са изпълнявани и у нас, като голяма част от необходимата експертност е налична. Тези примери показват как ВиК секторът помага за попълването на сериозни дефицити в сферата на общественото здраве, енергетиката и селското стопанство, а междувременно решава свои проблеми и генерира приходи. Едно побутване от страна на държавата или браншовите организации – Министерството на регионалното развитие и благоустройството, „Български ВиК холдинг“, Българската асоциация по водите, Съюза на ВиК операторите – ще даде необходимия начален тласък.

  • ВиК секторът като среда за развитие на високи технологии

Водоснабдяването и канализацията са известни като отрасъл с критично ниско, бавно и неадекватно навлизане на технологиите. В по-широката рамка на т.нар. ютилити индустрия експертите в бранша обикновено се шегуваме, че технологиите най-напред навлизат в комуникациите, десетина години след това – в енергетиката, а евентуално след още десет – и при нас.

Тезата е особено вярна в България, но точно сега имаме добър шанс секторът да се превърне в среда за създаване и развиване на високи технологии, а не само да бъде поле за прилагането им. Два са основните фактори за тази възможност: 1) наличието на активна и растяща екосистема от ИТ компании, които жадуват за възможността да прилагат т.нар. интернет на нещата (IoT), анализ на големи масиви данни (big data) и други високотехнологични решения в реална среда, и 2) обективното наличие на огромни масиви неизползвани до момента данни във ВиК отрасъла – за клиенти, активи и процеси.

Накратко: ВиК дружествата са перфектната среда за пилотни проекти на иновативни компании или единични изследователи. Интерес съществува, успешни примери също не липсват, но една заявена държавна подкрепа (или поне позволение) може много да помогне – при това без излишно субсидиране, а изцяло на пазарен принцип. И сега български компании предлагат в чужбина ВиК решения (системи за мониторинг, софтуер, аналитични модули), базирани на пилотни проекти в България. Този прохождащ модел може да стане норма на работа.

  • Повече свобода за експертите

Предвид пословично консервативния сектор с презюмирано високо ниво на корупция, не е пресилено да се каже, че организационната среда в отрасъла е токсична. Чак е учудващо как млади инженери, химици и биолози продължават да постъпват на работа в провинциални ВиК дружества за заплати, по-ниски от тези на касиер в хипермаркет. Но го правят. Било за стаж, било от младежки идеализъм.

Представете си обаче, че институционалната шапка над всички ВиК дружества – „Български ВиК Холдинг“ – създаде програма (не инцидентен проект), в рамките на която ежегодно се одобряват и финансират по няколко десетки предложения, идентифицирани и управлявани от вътрешни за системата кадри. Представете си, че всеки районен началник, главен енергетик или старши лаборант може да предлага решения, с които да се подобри и оптимизира експлоатацията, да се спестят разходи и да се постигне широка полезност за обществото. Именно така ще се задържат в сектора младите инженери, химици и биолози – та дори при заплати, по-ниски от тези на касиер в хипермаркет. (И именно подобни проекти биха им проправили пътя и към по-високи заплати.)

ВиК отрасълът в България може много повече. Първата ни задача обаче е да си го поискаме това много повече. Да си го представим, да го нарисуваме, да го обсъдим помежду си. Политическото и оперативното ръководство на сектора, регулаторът и браншовите организации, както и цялата широка общественост са част от този дебат.

Заглавна снимка: Manki Kim / Unsplash

Източник

Джейкъб Лорънс, или когато историята се превърне в изкуство

Post Syndicated from Йоанна Елми original https://toest.bg/jacob-lawrence/

„В края на 60-те всичко минаваше за изкуство. Всички бяха творци в края на 60-те. Нещо като пънка през 70-те“, споделя спомените си писателят Ричард Прайс в интервю за Paris Review. Разбира се, легендарното десетилетие не започва с отбелязването на новата 1960-та, а се заражда още през следвоенните години.

Това е ерата на авангарда, модернизма, сюрреализма и абстрактните форми. В този контекст някои критици смятат, че едно от най-смелите решения на художника Джейкъб Лорънс (1917–2000) е да откаже да създава странно и новаторско изкуство просто в името на новаторството. На фона на творци като Ханс Хартунг, Франц Клайн, Вилем де Кунинг, Марк Ротко и други свои съвременници Лорънс може да мине дори за леко скучен. Но сякаш именно поради липсата на усилие да провокира или пък да създава изкуство, което умишлено да заобикаля рационалното и да се прицелва в емоционалното, той успява да сътвори нещо ново, което не е имитация и не може да бъде имитирано.

„Докато Лорънс създава трийсетте пана от серията „Американската борба“, всяко с големина 30 х 40 см, хоризонтално или вертикално, неговите съвременници абстракционисти и експресионисти гонят коренно противоположни и монументални мащаби. Този контраст съдържа в себе си въпрос: кое предизвикателство е по-голямо – широтата на по-големия размер и масивните пропорции или дестилирането и кондензацията, които са нужни, за да работиш с толкова интимно тясно пространство? Някак Лорънс разбира, че една степен по-малък мащаб може да е по-ефективен в способността си да комуникира и да подхрани връзка с картината“, пишат Линда Роско Хартигън и Браян П. Кенеди от музея „Пийбоди“ в Есекс, Масачузетс.

Лорънс определя стила си като „динамичен кубизъм“ – техниката му определено е интересна и мигновено привлича вниманието. Но едно от най-важните качества на изкуството му е, че работи директно с исторически извори, построявайки мост между изговорената и неизговорена история (конкретно физическо измерение) и визуалното, което може да бъде видяно и осъзнато, но не и преживяно. Така изкуството на Лорънс е политическо и социално, без задължително да има за цел да бъде такова, както се случва често и в наши дни. И ако опитите за морализаторски и политически нотки в творчеството често завършват някъде между досадата и пропагандата, то изкуството на Лорънс не поучава, а показва и разказва чрез два посредника – писаното историческо слово и картината. И още нещо – Лорънс прави всичко това от позицията на чернокож артист, който твори и говори за бурните епизоди в историята на САЩ в една все още сегрегирана Америка.

Дете на разведени мигранти от Юга, Лорънс израства в Харлем, Ню Йорк, където майка му го записва на вечерни уроци по рисуване. Едва 23-годишен, Джейкъб Лорънс изгрява на сцената със серията „Миграция“ – 60 пана, които проследяват миграцията на афроамериканци от южните към северните щати след Първата световна война. Лорънс работи с бързосъхнещи бои и рисува цвят по цвят, за да няма разлика в нюансите. Разказите в картини с кратки изречения напомнят на комикс.

Серия „Миграция“, пано 1: „Началото на миграцията на чернокожи от юг на север след войната.“
Серия „Миграция“, пано 5 (горе): „На мигрантите се разрешава да пътуват с билети за влак, платени от индустрията на север. Разходите се възстановяват от мигрантите от бъдещите им заплати.“; пано 13 (долу): „Реколтата [на юг] е оставена да изсъхне и да гние. Няма кой да я прибере.“
Серия „Миграция“, пано 51: „Афроамериканците, които търсят по-добри условия за живот, се опитват да се нанесат в нови квартали. Вследствие на това домовете им са бомбардирани.“
Разглеждане на всички пана от серията „Миграция“

В началото на 50-те години Лорънс търси тема за нов проект. В ерата на следвоенния консенсус икономиката на САЩ просперира и „американската мечта“ се превръща в продукт за антисъветска пропаганда в контекста на Студената война. Но не всеки има право да мечтае. Макар да са се били рамо до рамо със своите съграждани във Втората световна война, чернокожите войници са интегрирани едва през 1948 г. и нямат достъп до голяма част от следвоенните програми, които подпомагат ветераните. През 1954 г., когато Лорънс започва работа над следваща серия пана, най-сетне е прекратена сегрегацията в училищата.

Макар Конституцията да гарантира равни права, в южните щати, но не само, важи доктрината „разделени, но равни“. Според този принцип чернокожите и белите трябва да живеят отделно, но да имат равен достъп до образование и работа. Разбира се, това няма нищо общо с реалността – на юг насилието над чернокожи е ежедневие и ключови инфраструктури като училищата са в окаяно състояние. Лорънс рисува в разгара на борбата за граждански права и се обръща към историята, за да попита: „Кой има право на Америка?“

„Картините, които предлагам, ще изобразяват борбата на един народ да създаде нация и опитът му да построят демокрация. Серията ще започне с причините и събитията, довели до Революцията, и ще приключат с околосветското пътешествие на американския флот от 1908 г. Между тези две събития ще бъдат изобразени голямото преселение на запад, което заема по-голямата част от XIX в., и инцидентите, довели до Гражданската война и индустриалната революция“, казва Лорънс на 16 декември 1954 г. Всяка картина е съпроводена с цитат от исторически извори.

Серия „Америкаската борба“, пано 1: „… толкоз ли е скъп животът и толкоз ли е сладка свободата, че да се получат на цената на вериги и робство?“ (Патрик Хенри, 1775 г.)

Патрик Хенри е един от бащите на американската нация, а цитатът е от негова реч в контекста на борбата за свобода от „тиранията“ на британската корона, която завършва с обявяването на независимостта на колониите през 1776 г. Изборът на Лорънс със сигурност е ироничен: нация, освободила се от тирания само за да я наложи на свой ред. Паното е завършено през 1955 г. – годината на бойкота в Монтгомъри.

 

Серия „Американската мечта“, пано 6: „… взаимно се заклеваме в нашия живот, в нашата съдба и в свещената си чест.“ (Из Декларацията за независимост на САЩ, 4 юли 1776 г.)

„Лорънс си представя клетвата от Декларацията за независимост като бреме, което се носи от всяко следващо поколение, точно както земеделецът, който се бори да прибере реколтата и се нуждае от подкрепата на онези, които са готови да хванат косата и вилата заради него. Самотният селянин носи тежестта на сламата, напредва под тежестта на независимостта, въплъщавайки ключовата фигура на човешкото оцеляване във враждебна среда“, пише художникът Дейвид Дрискел в каталога към изложбата.

Интересен е и изборът именно на този цитат: ако декларативното начало на историческия документ е познато не само на американците, но и на целия свят, то финалната фраза е смирено напомняне, че животът, свободата и стремежът към щастие не могат да бъдат постигнати без отговорност към другия.

 

Серия „Американската мечта“, пано 12: „И жена застана на оръдието.“

Маргарет Кохран Корбин придружава съпруга си в една от битките по време на Революцията през 1776 г. и когато той е убит, тя заема неговия пост и стреля с оръдието. Маргарет Корбин губи ръката си по време на сражението и се присъединява към ранените войници до официалното си освобождаване през 1783 г. Поради недъга си тя не може да се издържа и изхранва и живее в нищета. Година преди смъртта ѝ през 1800 г. е наградена с пенсия, чиято сума е наполовина по-малка от тази на мъжете, редом с които се е сражавала.

 

Серия „Американската мечта“, пано 18: „Във всичките си срещи с местните се отнасяйте с тях възможно най-приятелски и мирно, доколкото позволява тяхното собствено поведение…“ (Томас Джеферсън към Луис и Кларк, 1803 г.)

След като Наполеон продава Луизиана през 1803 г., територията на САЩ се увеличава почти двойно. Президентът Джеферсън изпраща Мериуедър Луис и Уилям Кларк на експедиция до непознатата земя, населена с коренни жители. Сакагауеа – жена от племето шошони, женена за френски трапер – става техен спътник и преводач в експедицията, като се смята, че мисията на Луис и Кларк би била провал без нейната съобразителност.

Картината на Лорънс, вдъхновена от цитата, поставя именно Сакагауеа на преден план, в момента, в който тя разбира, че вождът, с когото са се срещнали, всъщност е неин брат. Луис и Кларк са изобразени с оръжия и само едно индианско копие е видимо на снимката. Нещо в атмосферата загатва, че в следващите 150 години коренното население ще бъде многократно премествано, претопявано и насилствено асимилирано, докато от него не остане почти никой.

 

Серия „Американската мечта“, пано 28: „Имигранти, приети от всички страни, от 1820 до 1840 г. – 115 773“

Цитатът е от Енциклопедия на американската история, публикувана през 1953 г. Картината на Лорънс е антипод на романтичния американски мит за имигрантите. Той предполага, че новодошлите са оставили някакво ужасно минало зад гърба си и само малко труд е достатъчен, за да вкусят плодовете на американската мечта. Хората на картината са тъжни, изтощени от пътя, с мръсни, прокъсани дрехи.

През 1952 г., малко преди началото на проекта на Лорънс, е приет Актът „Маккарън–Уолтър“: имиграционен закон, чиято цел според един от авторите е да „защити държавата от комунисти, еврейски интереси и чужди интереси, които застрашават националната сигурност“. Законът продължава наложените през 1924 г. квоти (каквито е имало включително за българи и други източно- и южноевропейци, смятани от нативистите за твърде различни, за да бъдат асимилирани) и въвежда таван от 100 души годишно от различни азиатски държави. До 100 имигранти годишно са допускани и от България според същия закон.

Преди началото на Втората световна война, през 1939 г., Конгресът на САЩ отказва да увеличи имиграционните квоти, за да приеме възможно най-голям брой европейски бежанци. Вместо това са проведени напразни опити да убедят държави в Латинска Америка да приемат евреи. Антиимиграционната реторика и нативизмът са част от историята на САЩ много преди войните – около 1850–1860 г. се отбелязва краят на първата вълна на масова миграция от Европа към САЩ и разпространяването на антикатолически възгледи сред протестантското население, което възприема новодошлите европейци и най-вече ирландците като заплаха за установения ред в държавата.

 

Серия „Американската борба“, пано 30: „Старата Америка сякаш се разпада и отива на Запад…“ (Имигрант от Великобритания, 1817 г.)

Цитатът е от дневника на Морис Биркбек, английски квакер, който описва пътуването си от Вирджиния до Илиной. Записките на Биркбек се възприемат като утопично обещание за по-добър живот в Америка. По това време Европа се лута между опити за налагане на имперски ред и революционни бунтове, подхранвани от наскоро родилата се национална идея, която бързо се разпространява на континента. Радикалните и антиаристократични идеи на Биркбек не са по вкуса на английските консервативни среди и текстовете му предизвикват сериозни полемики.

 

Триизмерна обиколка на паната от серията „Американската борба“

Серията „Американската борба“ обхваща САЩ във всичките им нюанси – не само от гледна точка на гражданските права, робството и расизма спрямо афроамериканците. Ключови исторически моменти са представени в съвсем нова светлина както чрез необичайни цитати от исторически извори, така и визуално – чрез ярки цветове и уж обикновеното, достъпното. От общо 60 пана близо половината са все още в неизвестност.

Творчеството на Лорънс е особено актуално в годините след „края на историята“, когато представата за САЩ на много европейци от двете страни на падналата желязна завеса е многократно поставяна под въпрос. Безкрайните войни в Близкия изток, Гуантанамо, ислямофобията и антимигрантската реторика спрямо латиноамериканци – всички тези нюанси се съдържат в багрите на „Американската борба“, просто в друга форма. Въпросът „Кой има право на Америка?“ е също толкова актуален днес, колкото и преди 70 години. Същото важи и за въпроса как САЩ осмислят и променят собствената си история, как говорят за нея и коя история избират да разкажат на света.

Заглавно изображение: Фрагмент от пано 18 от серията „Американската борба“ на Джейкъб Лорънс. Източник: Музея „Пийбоди“ в Есекс, Масачузетс

Източник

От местопрестъплението: Facebook

Post Syndicated from Йовко Ламбрев original https://toest.bg/csi-facebook/

Когато в началото на март 2020 г. се установихме по домовете си и (които можaхме) заработихме оттам, първите ни разговори с колеги и клиенти се въртяха около това дали науката е способна бързо да намери ваксина срещу COVID-19 и най-вече дали регулаторите ще бъдат достатъчно бързи с одобрението ѝ. За да можем да си върнем нормалния живот и ежедневие. Никога не бих допуснал, че голяма част от същите тези хора днес още няма да са ваксинирани, а с някои от тях ще спорим ожесточено по темата. Какво се случи през тази година и половина?

Със сигурност локалният контекст е важен. Генерализациите не са полезни, но е трудно да се обори наблюдението, че българите сме недоверчиви по природа. Изчакваме първо някой да похвали нещо, преди да си го купим, или съседите да пробват нов продукт или услуга, преди да се решим да тестваме сами. Вярно е също и че институциите ни не заслужават доверие, а управлението на кризата стартира повоенному – с генералски униформи, заплахи и овикване вместо с научни аргументи и човешки обяснения. Всичко това няма как да не е част от причините за най-ниското ниво на ваксинирани у нас. Но дали е само това?

Всъщност антивакс пропагандата онлайн е поредното проявление на проблем, който пуска все по-дълбоки корени сред обществата ни – по целия свят.

Същият проблем, който направи възможен Брекзит и докара на власт опасен палячо като Доналд Тръмп. Технологиите ни свързват повече от всякога, но донесоха едни от най-ужасните противопоставяния в най-новата история на човечеството.

Брекзит стана факт след минимално числено превъзходство на гласовете за напускане на ЕС, в референдум, който в значителна степен бе манипулиран чрез лъжи и дезинформация в социалните мрежи и най-вече във Facebook. Механизмите вече са до болка известни – набелязване на група колебаещи се гласоподаватели, които биват облъчвани с реклами и съдържание, обострящо техните лични страхове или чувствителност по определени теми. И тъй като няма контролен механизъм какво съдържание вижда всеки на стената си и то е индивидуално подбрано за него, няма как дори това да бъде впоследствие разследвано, оборено или съпоставено с друго.

Технологията, експлоатирана до крайност, е в състояние да създаде в главата на един човек паралелна реалност, която няма допирни точки с тази в главите на останалите. Така британците започнаха да се страхуват от бежанци и имигранти дори където ги няма. Или че Турция ще стане част от ЕС, въпреки че дори не се полагат усилия за това.

Но не са само лъжите. Социалните мрежи и платформи се превърнаха в рупор на страх, омраза и разделение. Срещу заплащане. Без никакъв контрол върху произхода на средствата, източника на дезинформация или дори яснота кой е поръчителят, доколкото той може да се скрие зад друга компания или посредник.

По-лошото е, че невинаги виждаме лъжите, на които са подложени другите, защото често не сме таргетирани.

Виждаме малка част от мащаба, който е далеч по-голям от всичко, което можем да допуснем или да си представим. Същият модел и техники бяха приложени и при избирането на Тръмп за президент на САЩ. Иронично, не без помощта и на други създадени в името на демокрацията и свободното слово проекти – като WikiLeaks. Същото се случва и с пропагандата срещу ваксините. Уви, за част от разпространението ѝ този път дори не се заплаща, защото радикализацията на хората по темата спомага за автозахранването ѝ със съдържание. Днес, благодарение на технологичните гиганти, страх и омраза се леят из интернет напълно безнаказано. А в контекста на отричане на рисковете от COVID-19 и ползите от ваксините това вече пряко причинява смърт.

Живеем в глобален онлайн експеримент, в който сме се самопоставили в ролята на лабораторни мишки, докато съпреживяваме разпада на демокрациите си. След повече от век избирателно законодателство и изстрадани права в световен план всичко това се разрушава от технологиите. Без съпротива от наша страна.

Оказва се, че правителствата са неспособни да контролират адекватно транснационални корпорации.

Отделна е темата дали наистина имат желание да го направят. Бунтът трудно ще започне от потребителите, докато те бавно биват сварявани като жаби в яхнията на дезинформацията. Провалът ни е свързан с приемането на демокрацията за даденост и недостатъчното гражданско образование в посока хуманитаризъм. Нужно е спешно загърбване на индивидуализма, в който бяха възпитани поне три поредни поколения, и поне малко завръщане на мисълта за общото и за принадлежността ни (и в тази връзка ангажиментите ни) като част от него. Нужни са усилия и за завръщане на авторитета на науката. А това няма как да стане, ако непрекъснато противопоставяме на научните факти псевдонаучни концепции и даваме думата на опиянени от себелюбие и жадни за внимание псевдоавторитети, борещи се за лидерство върху някакво мнение.

Разпространението на псевдонаука и лъжи, безплатно или платено с пари с неясен произход, няма нищо общо с демокрацията или свободата на словото. Това е саботаж, подривна дейност. Оръжие в тиха война с драматични последствия. Защото най-страшното не е, че Тръмп спечели президентските избори през 2017 г. или че Великобритания напусна ЕС.

Главният въпрос пред нас е как да съхраним обществения си договор.

Ще можем ли някога отново да проведем честни и свободни избори изобщо? Ще можем ли да върнем доверието помежду си? Ще имаме ли отново свободни медии извън хватката на платформите? Ще оцелеем ли?

Време е за осъзнаване и съпротива. Не можем да си позволим да оставим социалните мрежи да бъдат основният източник на новини и информация на хората. Време е да изискваме поемане на адекватна отговорност, когато се окаже, че технологичните гиганти влияят на установените механизми на демокрацията и посредничат за разрухата на обществата ни. Защото докато си споделяме снимки на котенца, покълналите плевели ще превземат всичко, ако не бъдат изкоренени.

Заглавна снимка: Annie Spratt / Unsplash

Източник

Седмицата в „Тоест“ (20–24 септември)

Post Syndicated from Тоест original https://toest.bg/editorial-20-24-september-2021/

Светла Енчева

Актуалното плашило напоследък, което всички очакват да изскочи иззад ъгъла, ако не се е настанило вече в хладилниците, фризерите и мазетата на народонаселението, е инфлацията. Факт е, че някои стоки поскъпват, цените на електроенергията винаги са чувствителна тема, а идва и зима. Но всъщност по-важното е, че идват избори и плашилата са нужни на някои политици да мобилизират електората си. Иначе инфлационните тенденции са глобални, а за България по-скоро привнесени, но и са естествени за възстановяващата се постковид световна икономика. Светла Енчева разглежда темата в статията си „Предизборно плашене с хиперинфлация“.


Емилия Милчева

И в този ред на мисли, когато предприятия затварят, бизнесът е настръхнал срещу властта, по улиците се тълпят гневни хора заради поскъпналия ток, храни и услуги, възможни ли са десни политики за постигане на леви цели, пита Емилия Милчева – и отговаря още със заглавието на новия си вътрешнополитически анализ „Дясното отиде „у лево“.


Венелина Попова

Разбира се, че във фокуса на вниманието в тази предизборна кампания ще продължава да бъде най-новият проект на родната политическа сцена – този на бившите служебни министри Кирил Петков и Асен Василев. Реалност или блян е хипотезата за парламентарно мнозинство около „Продължаваме промяната“, се пита пък Венелина Попова в тазседмичния си коментар.


Радослав Русев

Радослав Русев продължава да споделя с читателите на „Тоест“ своя оптимизъм за развитието на един сектор в икономиката ни, който рядко генерира положителни новини и основно се свързва с липса на свежи идеи. Става дума за водоснабдяването и канализацията, накратко ВиК. Според него реалното положение е всъщност много по-оптимистично и ако нещата останат дългосрочно зле, то основната причина е в нашата колективна глупост, алчност и късогледство, а не толкова в климата, бедността и старите тръби. Прочетете статията му „Извън усвояването, съответствието и поносимостта. Оптимистично за ВиК отрасъла“.


Йоанна Елми

„Когато историята се превърне в изкуство“ е материал на Йоанна Елми за чернокожия художник Джейкъб Лорънс, творил през бурните периоди в историята на сегрегирана Америка и в разгара на борбата за граждански права. Според Йоанна едно от най-важните качества на изкуството му е, че работи директно с исторически извори, построявайки мост между изговорената и неизговорена история, от една страна, и визуалното, от друга. Това прави изкуството му актуално и обговарящо човешките разделения и след падането на желязната завеса, безкрайните войни в Близкия изток, до днешната ислямофобия и антиимигрантски настроения.


Йовко Ламбрев

Завършваме с предупреждението на Йовко Ламбрев, според когото настроенията против ваксините срещу COVID-19 имат благодатна почва не само заради специфики от родния контекст, но и заради стар и познат глобален проблем – съсредоточаването на твърде много безконтролна власт в социалните мрежи и платформи и оставянето им да бъдат основен източник на новини, измествайки медиите и заглушавайки авторитетите. Решенията не са очевидни, но спешни контрадействия са крайно наложителни.

Приятно четене!

Източник

Configure Amazon EMR Studio and Amazon EKS to run notebooks with Amazon EMR on EKS

Post Syndicated from Randy DeFauw original https://aws.amazon.com/blogs/big-data/configure-amazon-emr-studio-and-amazon-eks-to-run-notebooks-with-amazon-emr-on-eks/

Amazon EMR on Amazon EKS provides a deployment option for Amazon EMR that allows you to run analytics workloads on Amazon Elastic Kubernetes Service (Amazon EKS). This is an attractive option because it allows you to run applications on a common pool of resources without having to provision infrastructure. In addition, you can use Amazon EMR Studio to build analytics code running on Amazon EKS clusters. EMR Studio is a web-based, integrated development environment (IDE) using fully managed Jupyter notebooks that can be attached to any EMR cluster, including EMR on EKS. It uses AWS Single Sign-On (SSO) or a compatible identity provider (IdP) to log directly in to EMR Studio through a secure URL using corporate credentials.

Deploying EMR Studio to attach to EMR on EKS requires integrating several AWS services:

In addition, you need to install the following EMR on EKS components:

This post helps you build all the necessary components and stitch them together by running a single script. We also describe the architecture of this setup and how the components work together.

Architecture overview

With EMR on EKS, you can run Spark applications alongside other types of applications on the same Amazon EKS cluster, which improves resource allocation and simplifies infrastructure management. For more information about how Amazon EMR operates inside an Amazon EKS cluster, see New – Amazon EMR on Amazon Elastic Kubernetes Service (EKS). EMR Studio provides a web-based IDE that makes it easy to develop, visualize, and debug applications that run in EMR. For more information, see Amazon EMR Studio (Preview): A new notebook-first IDE experience with Amazon EMR.

Spark kernels are scheduled pods in a namespace in an Amazon EKS cluster. EMR Studio uses Jupyter Enterprise Gateway (JEG) to launch Spark kernels on Amazon EKS. A managed endpoint of type JEG is provisioned as a Kubernetes deployment in the EMR virtual cluster’s associated namespace and exposed as a Kubernetes service. Each EMR virtual cluster maps to a Kubernetes namespace registered with the Amazon EKS cluster; virtual clusters don’t manage physical compute or storage, but point to the Kubernetes namespace where the workload is scheduled. Each virtual cluster can have several managed endpoints, each with their own configured kernels for different use cases and needs. JEG managed endpoints provide HTTPS endpoints, serviced by an Application Load Balancer (ALB), that are reachable only from EMR Studio and self-hosted notebooks that are created within a private subnet of the Amazon EKS VPC.

The following diagram illustrates the solution architecture.

The managed endpoint is created in the virtual cluster’s Amazon EKS namespace (in this case, sparkns) and the HTTPS endpoints are serviced from private subnets. The kernel pods run with the job-execution IAM role defined in the managed endpoint. During managed endpoint creation, EMR on EKS uses the AWS Load Balancer Controller in the kube-system namespace to create an ALB with a target group that connects with the JEG managed endpoint in the virtual cluster’s Kubernetes namespace.

You can configure each managed endpoint’s kernel differently. For example, to permit a Spark kernel to use AWS Glue as their catalog, you can apply the following configuration JSON file in the —configuration-overrides flag when creating a managed endpoint:

aws emr-containers create-managed-endpoint \
--type JUPYTER_ENTERPRISE_GATEWAY \
--virtual-cluster-id ${virtclusterid} \
--name ${virtendpointname} \
--execution-role-arn ${role_arn} \
--release-label ${emr_release_label} \
--certificate-arn ${certarn} \
--region ${region} \
--configuration-overrides '{
    "applicationConfiguration": [
      {
        "classification": "spark-defaults",
        "properties": {
          "spark.hadoop.hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory",
          "spark.sql.catalogImplementation": "hive"
        }
      }
    ]
  }'

The managed endpoint is a Kubernetes deployment fronted by a service inside the configured namespace (in this case, sparkns). When we trace the endpoint information, we can see how the Jupyter Enterprise Gateway deployment connects with the ALB and the target group:

# Get the endpoint ID
aws emr-containers list-managed-endpoints --region us-east-1 --virtual-cluster-id idzdhw2qltdr0dxkgx2oh4bp1
{
    "endpoints": [
        {
            "id": "5vbuwntrbzil1",
            "name": "virtual-emr-endpoint-demo",
            ...
            "serverUrl": "https://internal-k8s-default-ingress5-4f482e2d41-2097665209.us-east-1.elb.amazonaws.com:18888",

# List the deployment
kubectl get deployments -n sparkns -l "emr-containers.amazonaws.com/managed-endpoint-id=5vbuwntrbzil1"

NAME                READY   UP-TO-DATE   AVAILABLE   AGE
jeg-5vbuwntrbzil1   1/1     1            1           4h54m


# List the service
kubectl get svc -n sparkns -l "emr-containers.amazonaws.com/managed-endpoint-id=5vbuwntrbzil1"

NAME                    TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)           AGE
service-5vbuwntrbzil1   NodePort   10.100.172.157   <none>        18888:30091/TCP   4h58m

# List the TargetGroups to get the TargetGroup ARN

kubectl get targetgroupbinding -n sparkns -o json | jq .items | jq .[].spec.targetGroupARN

"arn:aws:elasticloadbalancing:us-east-1:< account id >:targetgroup/k8s-sparkns-servicey-a37caa5e1e/02d10652a64cebd8"

# Get the TargetGroup Port number

aws elbv2 describe-target-groups --target-group-arns arn:aws:elasticloadbalancing:us-east-1:< account id >:targetgroup/k8s-sparkns-servicey-a37caa5e1e/02d10652a64cebd8 | jq .TargetGroups | jq .[].Port

30091


# Get Load Balancer ARN

aws elbv2 describe-target-groups --target-group-arns arn:aws:elasticloadbalancing:us-east-1:< account id >:targetgroup/k8s-sparkns-servicey-a37caa5e1e/02d10652a64cebd8 | jq .TargetGroups | jq .[].LoadBalancerArns | jq .[]

"arn:aws:elasticloadbalancing:us-east-1:< account id >:loadbalancer/app/k8s-sparkns-ingressy-830efa48aa/12199b1a7baee273"

# Get Listener Port number

aws elbv2 describe-listeners --load-balancer-arn arn:aws:elasticloadbalancing:us-east-1:< account id >:loadbalancer/app/k8s-sparkns-ingressy-830efa48aa/12199b1a7baee273 | jq .Listeners | jq .[].Port

18888

To look at how this connects, consider two EMR Studio sessions. The ALB exposes port 18888 to the EMR Studio sessions. The JEG service maps the external port 18888 on the ALB to the dynamic NodePort on the JEG service (in this case, 30091). The JEG service forwards the traffic to the TargetPort 9547, which routes the traffic to the appropriate Spark driver pod. Each notebook session has its own kernel, which has its own respective Spark driver and executor pods, as the following diagram illustrates.

Attach EMR Studio to a virtual cluster and managed endpoint

Each time a user attaches a virtual cluster and a managed endpoint to their Studio Workspace and launches a Spark session, Spark drivers and Spark executors are scheduled. You can see that when you run kubectl to check what pods were launched:

$ kubectl get all -l app=enterprise-gateway
NAME                                  READY   STATUS      RESTARTS   AGE
pod/kb1a317e8-b77b-448c-9b7d-exec-1   1/1     Running     0          2m30s
pod/kb1a317e8-b77b-448c-9b7d-exec-2   1/1     Running     0          2m30s
pod/kb1a317e8-b77b-448c-9b7d-driver   2/2     Running     0          2m38s

$ kubectl get pods -n sparkns
NAME                                  READY   STATUS      RESTARTS   AGE
jeg-5vbuwntrbzil1-5fc8469d5f-pfdv9    1/1     Running     0          3d7h
kb1a317e8-b77b-448c-9b7d-exec-1       1/1     Running     0          2m38s
kb1a317e8-b77b-448c-9b7d-exec-2       1/1     Running     0          2m38s
kb1a317e8-b77b-448c-9b7d-driver       2/2     Running     0          2m46s

Each notebook Spark kernel session deploys a driver pod and executor pods that continue running until the kernel session is shut down.

The code in the notebook cells runs in the executor pods that were deployed in the Amazon EKS cluster.

Set up EMR on EKS and EMR Studio

Several steps and pieces are required to set up both EMR on EKS and EMR Studio. Enabling AWS SSO is a prerequisite. You can use the two provided launch scripts in this section or manually deploy it using the steps provided later in this post.

We provide two launch scripts in this post. One is a bash script that uses AWS CloudFormation, eksctl, and AWS Command Line Interface (AWS CLI) commands to provide an end-to-end deployment of a complete solution. The other uses the AWS Cloud Development Kit (AWS CDK) to do so.

The following diagram shows the architecture and components that we deploy.

Prerequisites

Make sure to complete the following prerequisites:

For information about the supported IdPs, see Enable AWS Single Sign-On for Amazon EMR Studio.

Bash script

The script is available on GitHub.

Prerequisites

The script requires you to use AWS Cloud9. Follow the instructions in the Amazon EKS Workshop. Make sure to follow these instructions carefully:

After you deploy the AWS Cloud9 desktop, proceed to the next steps.

Preparation

Use the following code to clone the GitHub repo and prepare the AWS Cloud9 prerequisites:

# Download script from the repository
$ git clone https://github.com/aws-samples/amazon-emr-on-eks-emr-studio.git

# Prepare the Cloud9 Desktop pre-requisites
$ cd amazon-emr-on-eks-emr-studio
$ bash ./prepare_cloud9.sh

Deploy the stack

Before running the script, provide the following information:

  • The AWS account ID and Region, if your AWS Cloud9 desktop isn’t in the same account ID or Region where you want to deploy EMR on EKS
  • The name of the Amazon Simple Storage Service (Amazon S3) bucket to create
  • The AWS SSO user to be associated with the EMR Studio session

After the script deploys the stack, the URL to the deployed EMR Studio is displayed:

# Launch the script and follow the instructions to provide user parameters
$ bash ./deploy_eks_cluster_bash.sh

...
Go to https://***. emrstudio-prod.us-east-1.amazonaws.com and login using < SSO user > ...

AWS CDK script

The AWS CDK scripts are available on GitHub. You need to checkout the main branch. The stacks deploy an Amazon EKS cluster and EMR on EKS virtual cluster in a new VPC with private subnets, and optionally an Amazon Managed Apache Airflow (Amazon MWAA) environment and EMR Studio.

Prerequisites

You need the AWS CDK version 1.90.1 or higher. For more information, see Getting started with the AWS CDK.

We use a prefix list to restrict access to some resources to network IP ranges that you approve. Create a prefix list if you don’t already have one.

If you plan to use EMR Studio, you need AWS SSO configured in your account.

Preparation

After you clone the repository and checkout the main branch, create and activate a new Python virtual environment:

# Clone the repository
$ git clone https://github.com/aws-samples/aws-cdk-for-emr-on-eks.git
$ cd aws-cdk-for-emr-on-eks/
$ git checkout main

# 
$ python3 -m venv .venv
$ source .venv/bin/activate

Now install the Python dependencies:

$ pip install -r requirements.txt

Lastly, bootstrap the AWS CDK:

$ cdk bootstrap aws://<account>/<region> \
  --context prefix=<prefix list> \
  --context instance=m5.xlarge \
  --context username=<SSO user name>

Deploy the stacks

Synthesize the AWS CDK stacks with the following code:

$ cdk synth \
  --context prefix=<prefix list> \
  --context instance=m5.xlarge \
  --context username=<SSO user name>

This command generates four stacks:

  • emr-eks-cdk – The main stack
  • mwaa-cdk – Adds Amazon MWAA
  • studio-cdk – Adds EMR Studio prerequisites
  • studio-cdk-live – Adds EMR Studio

The following diagram illustrates the resources deployed by the AWS CDK stacks.

Start by deploying the first stack:

$ cdk deploy <stack name> \
  --context prefix=<prefix list> \
  --context instance=m5.xlarge  \
  --context username=<SSO user name> \
  emr-eks-cdk

If you want to use Apache Airflow as your orchestrator, deploy that stack:

$ cdk deploy <stack name> \
  --context prefix=<prefix list> \
  --context instance=m5.xlarge \
  --context username=<SSO user name> \
  mwaa-cdk

Deploy the first EMR Studio stack:

$ cdk deploy <stack name> \
  --context prefix=<prefix list> \
  --context instance=m5.xlarge \
  --context username=<SSO user name> \
  studio-cdk

Wait for the managed endpoint to become active. You can check the status by running the following code:

$ aws emr-containers list-managed-endpoints --virtual-cluster-id <cluster ID> | jq '.endpoints[].state'

The virtual cluster ID is available in the AWS CDK output from the emr-eks-cdk stack.

When the endpoint is active, deploy the second EMR Studio stack:

$ cdk deploy <stack name> \
  --context prefix=<prefix list> \
  --context instance=m5.xlarge \
  --context username=<SSO user name> \
  studio-live-cdk

Manual deployment

If you prefer to manually deploy EMR on EKS and EMR Studio, use the steps in this section.

Set up a VPC

If you’re using Amazon EKS v. 1.18, set up a VPC that also has private subnets and appropriately tagged for external load balancers. For tagging, see: Application load balancing on Amazon EKS and Create an EMR Studio service role.

Create an Amazon EKS cluster

Launch an Amazon EKS cluster with at least one managed node group. For instructions, see Setting up and Getting Started with Amazon EKS.

Create relevant IAM policies, roles, IdP, and SSL/TLS certificate

To create your IAM policies, roles, IdP, and SSL/TLS certificate, complete the following steps:

  1. Enable cluster access for EMR on EKS.
  2. Create an IdP in IAM based on the EKS OIDC provider URL.
  3. Create an SSL/TLS certificate and place it in AWS Certificate Manager.
  4. Create the relevant IAM policies and roles:
    1. Job execution role
    2. Update the trust policy for the job execution role
    3. Deploy and create the IAM policy for the AWS Load Balancer Controller
    4. EMR Studio service role
    5. EMR Studio user role
    6. EMR Studio user policies associated with AWS SSO users and groups
  5. Register the Amazon EKS cluster with Amazon EMR to create the virtual EMR cluster
  6. Create the appropriate security groups to be attached to each EMR Studio created:
    1. Workspace security group
    2. Engine security group
  7. Tag the security groups with the appropriate tags. For instructions, see Create an EMR Studio service role.

Required installs in Amazon EKS

Deploy the AWS Load Balancer Controller in the Amazon EKS cluster if you haven’t already done so.

Create EMR on EKS relevant pieces and map the user to EMR Studio

Complete the following steps:

  1. Create at least one EMR virtual cluster associated with the Amazon EKS cluster. For instructions, see Step 1 of Set up Amazon EMR on EKS for EMR Studio.
  2. Create at least one managed endpoint. For instructions, see Step 2 of Set up Amazon EMR on EKS for EMR Studio.
  3. Create at least one EMR Studio; associate the EMR Studio with the private subnets configured with the Amazon EKS cluster. For instructions, see Create an EMR Studio.
  4. When the EMR Studio is available, map an AWS SSO user or group to the EMR Studio and apply an appropriate IAM policy to that user.

Use EMR Studio

To start using EMR Studio, complete the following steps:

  1. Find the URL for EMR Studio by the studios in a Region:
$ aws emr list-studios --region us-east-1
{
    "Studios": [
        {
            "StudioId": "es-XXXXXXXXXXXXXXXXXXXXXX",
            "Name": "emr_studio_1",
            "VpcId": "vpc-XXXXXXXXXXXXXXXXXXXX",
            "Url": "https://es-XXXXXXXXXXXXXXXXXXXXXX.emrstudio-prod.us-east-1.amazonaws.com",
            "CreationTime": "2021-02-10T14:04:13.672000+00:00"
        }
    ]
}
  1. With the listed URL, log in using the AWS SSO username you used earlier.

After authentication, the user is routed to the EMR Studio dashboard.

  1. Choose Create Workspace.
  2. For Workspace name, enter a name.
  3. For Subnet, choose the subnet that corresponds to one of the subnets associated with the managed node group.
  4. For S3 location, enter an S3 bucket where you can store the notebook content.

  1. After you create the Workspace, choose one that is in the Ready status.

  1. In the sidebar, choose the EMR cluster icon.
  2. Under Cluster type¸ choose EMR Cluster on EKS.
  3. Choose the available virtual cluster and available managed endpoint.
  4. Choose Attach.

After it’s attached, EMR Studio displays the kernels available in the Notebook and Console section.

  1. Choose PySpark (Kubernetes) to launch a notebook kernel and start a Spark session.

Because the endpoint configuration here uses AWS Glue for its metastore, you can list the databases and tables connected to the AWS Glue Data Catalog. You can use the following example script to test the setup. Modify the script as necessary for the appropriate database and table that you have in your Data Catalog:

words='Welcome to Amazon EMR Studio'.split(' ')
wordRDD = sc.parallelize(words)
wc = wordRDD.map(lambda word: (word, 1)).reduceByKey(lambda a,b: a+b)
print(wc.collect())

# Connect to Glue Catalog
spark.sql("""show databases like '< Database Name >'""").show(truncate=False)
spark.sql("""show tables in < Database Name >""").show(truncate=False)
# Run a simple select
spark.sql("""select * from < Database Name >.< Table Name > limit 10""").show(truncate=False)


Clean up

To avoid incurring future charges, delete the resources launched here by running remove_setup.sh:

# Launch the script
$ bash ./remove_setup.sh</p>

Conclusion

EMR on EKS allows you to run applications on a common pool of resources inside an Amazon EKS cluster without having to provision infrastructure. EMR Studio is a fully managed Jupyter notebook and tool that provisions kernels that run on EMR clusters, including virtual clusters on Amazon EKS. In this post, we described the architecture of how EMR Studio connects with EMR on EKS and provided scripts to automatically deploy all the components to connect the two services.

If you have questions or suggestions, please leave a comment.


About the Authors

Randy DeFauw is a Principal Solutions Architect at Amazon Web Services. He works with the AWS customers to provide guidance and technical assistance on database projects, helping them improve the value of their solutions when using AWS.

Matthew Tan is a Senior Analytics Solutions Architect at Amazon Web Services and provides guidance to customers developing solutions with AWS Analytics services on their analytics workloads.

Metasploit Wrap-Up

Post Syndicated from Adam Galway original https://blog.rapid7.com/2021/09/24/metasploit-wrap-up-131/

Vulnerability is in the eye of the beholder

Metasploit Wrap-Up

Exploiting firmware authored by UDP Technology and provided to multiple large OEMs (including Geutebruck), community contributor TrGFxX has authored a neat module that allows RCE as root on machines running the web interface of the Geutebruck G-Cam and G-Code products. For more information on the vulnerability check out the CISA advisory.

OpManager exploit is OP plz nerf

Our very own zeroSteiner authored a module implementing both an exploit and patch bypass for a Java deserialization vulnerability that exists in numerous versions of ManageEngine’s OpManager software. This module allows payload execution as either NT AUTHORITY\SYSTEM on Windows or root on Linux. On top of this new module, zeroSteiner made improvements to help utilize the increasingly essential YSoSerial tool. You should definitely check it out if you’re interested in exploring other Java deserialization vulns.

Putting the Win in WinRM

In a big win for Metasploit, community contributor smashery finished off their month-long effort to get fully functional shells working across WinRM! These new sessions support post modules, NTLMSSP authentication, and are also able to run without a payload in remote memory, making these sessions pretty hard to detect. This is a major improvement over the previous WinRM implementation that only supported execution of a single command, so huge thanks again to smashery.

You can tell a lot about a protocol from its handshake

In one final noteworthy addition, smashery has once again come through with a PR that significantly improves our RDP library. Metasploit users can now capture the NETBIOS computer name, NETBIOS domain name, DNS computer name, DNS domain name, and OS version from the NTLM handshake carried out over RDP, and our rdp_scanner module has been updated to display this info to all the RDP sniffers out there.

New module content (3)

Enhancements and features

  • #15684 from adfoster-r7 – This improves interactive shell performance for pasted user input.
  • #15696 from smashery – This updates the RDP scanner module to extract and show additional information gathered from the NTLM handshake used for Network Level Authentication (NLA).
  • #15632 from smashery – This improves Metasploit’s WinRM capabilities by allowing shell sessions to be established over the protocol. The shell sessions are interactive and are usable with post modules.

Bugs fixed

  • #15600 from agalway-r7 – This fixes an issue with encrypted payloads during session setup. The logic that gathers session info is now located in the bootstrap method, which ensures that this functionality is always carried out before any commands are sent.
  • #15666 from timwr – This fixes an issue found in Meterpreter’s download functionality where downloading a file with a name containing unicode characters would fail due to incompatible encoding.
  • #15679 from nvn1729 – This fixes a bug where the tomcat_mgr_upload module was not correctly undeploying the app after exploitation occurred.
  • #15686 from jmartin-r7 – This fixes a crash in msfrpc that occurs due to the exploit/linux/misc/saltstack_salt_unauth_rce module’s MINIONS option default being a regex instead of a string.
  • #15695 from adfoster-r7 – This fixes a crash in the exploit/unix/local/setuid_nmap module and adds logging to print the result of the exploit’s last command so the user knows what happened in the event of a failure.
  • #15697 from smashery – This updates the HTTP NTLM information enumeration module to use the Net::NTLM library for consistent data processing without a custom parser.

Get it

As always, you can update to the latest Metasploit Framework with msfupdate and you can get more details on the changes since the last blog post from GitHub:

If you are a git user, you can clone the Metasploit Framework repo (master branch) for the latest. To install fresh without using git, you can use the open-source-only Nightly Installers or the binary installers (which also include the commercial edition).

I Am Not Satoshi Nakamoto

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2021/09/i-am-not-satoshi-nakamoto.html

This isn’t the first time I’ve received an e-mail like this:

Hey! I’ve done my research and looked at a lot of facts and old forgotten archives. I know that you are Satoshi, I do not want to tell anyone about this. I just wanted to say that you created weapons of mass destruction where niches remained poor and the rich got richer! When bitcoin first appeared, I was small, and alas, my family lost everything on this, you won’t find an apple in the winter garden, people only need strength and money. Sorry for the English, I am from Russia, I can write with errors. You are an amazingly intelligent person, very intelligent, but the road to hell is paved with good intentions. Once I dreamed of a better life for myself and my children, but this will never come …

I like the bit about “old forgotten archives,” by which I assume he’s referring to the sci.crypt Usenet group and the Cypherpunks mailing list. (I posted to the latter a lot, and the former rarely.)

For the record, I am not Satoshi Nakamoto. I suppose I could have invented the bitcoin protocols, but I wouldn’t have done it in secret. I would have drafted a paper, showed it to a lot of smart people, and improved it based on their comments. And then I would have published it under my own name. Maybe I would have realized how dumb the whole idea is. I doubt I would have predicted that it would become so popular and contribute materially to global climate change. In any case, I did nothing of the sort.

Read the paper. It doesn’t even sound like me.

Of course, this will convince no one who doesn’t already believe. Such is the nature of conspiracy theories.

AWS achieves GSMA security certification for US East (Ohio) Region

Post Syndicated from Janice Leung original https://aws.amazon.com/blogs/security/aws-achieves-gsma-security-certification-for-us-east-ohio-region/

We continue to expand the scope of our assurance programs at Amazon Web Services (AWS) and are pleased to announce that our US East (Ohio) Region (us-east-2) is now certified by the GSM Association (GSMA) under its Security Accreditation Scheme Subscription Management (SAS-SM) with scope Data Center Operations and Management (DCOM). This alignment with GSMA requirements demonstrates our continuous commitment to adhere to the heightened expectations for cloud service providers. AWS customers who provide embedded Universal Integrated Circuit Card (eUICC) for mobile devices can run their remote provisioning applications with confidence in the AWS Cloud in the GSMA-certified US East (Ohio) Region.

As of this writing, 128 services offered in the US East (Ohio) Region are in scope of this certification. For up-to-date information, including when additional services are added, see the AWS Services in Scope by Compliance Program and choose GSMA.

AWS was evaluated by independent third-party auditors chosen by GSMA. The Certificate of Compliance illustrating the AWS GSMA compliance status is available on the GSMA website and through AWS Artifact. AWS Artifact is a self-service portal for on-demand access to AWS compliance reports. Sign in to AWS Artifact in the AWS Management Console, or learn more at Getting Started with AWS Artifact.

To learn more about our compliance and security programs, see AWS Compliance Programs. As always, we value your feedback and questions; reach out to the AWS Compliance team through the Contact Us page.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Janice Leung

Janice is a Security Audit Program Manager at AWS, based in New York. She leads various security audit programs across Europe. She previously worked in security assurance and technology risk management in the financial industry for 10 years.

Author

Karthik Amrutesh

Karthik is a Senior Manager, Security Assurance at AWS, based in New York. He leads a team responsible for audits, attestations, and certifications across the European Union. Karthik has previously worked in risk management, security assurance, and technology audits for over 18 years.

[$] Two security improvements for GCC

Post Syndicated from original https://lwn.net/Articles/870045/rss

It has often been said that the competition between the GCC and LLVM
compilers is good for both of them. One place where that competition
shows up is in the area of security features; if one compiler adds a way to
harden programs, the other is likely to follow suit. Qing
Zhao’s session
at the 2021
Linux Plumbers Conference
told the story of how GCC successfully played
catch-up for two security-related features that were of special interest to
the kernel community.

Netflix Cloud Packaging in the Terabyte Era

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/netflix-cloud-packaging-in-the-terabyte-era-d6869b4b84ae

By Xiaomei Liu, Rosanna Lee, Cyril Concolato

Introduction

Behind the scenes of the beloved Netflix streaming service and content, there are many technology innovations in media processing. Packaging has always been an important step in media processing. After content ingestion, inspection and encoding, the packaging step encapsulates encoded video and audio in codec agnostic container formats and provides features such as audio video synchronization, random access and DRM protection. Our previous tech blog Packaging award-winning shows with award-winning technology detailed our packaging technology deployed on the streaming side.

As Netflix becomes a producer of award winning content, the studio and content creation needs are also pushing the envelope of technology advancements. As an example, cloud-based post-production editing and collaboration pipelines demand a complex set of functionalities, including the generation and hosting of high quality proxy content. Supporting those workflows poses new challenges to our packaging service.

The Terabyte Era

Apple ProRes encoded video and PCM audio with Quicktime container are one of the most popular formats in professional post-production editing. The ProRes codec family provides great editing performance and image quality. ProRes 422 HQ offers visually lossless preservation of the highest quality professional HD video and is a great video format choice for high quality editing proxy.

As described by the white paper Apple ProRes (link), the target data rate of the Apple ProRes HQ for 1920×1080 at 29.97 is 220 Mbps. With the wide adoption of 4K content across the production pipeline, the generation of ProRes 422 HQ at the resolution of 3840×2160 requires our processing pipeline to encode and package content in the order of terabytes. The following table gives us an example of file sizes for 4K ProRes 422 HQ proxies.

Table 1: Movie and File Size Examples

Initial Architecture

A simplified view of our initial cloud video processing pipeline is illustrated in the following diagram. The inspection stage examines the input media for compliance with Netflix’s delivery specifications and generates rich metadata. This metadata includes both file level information such as video encoding format, video frame rate, and resolution, as well as frame level information such as frame offset, frame dependency, and frame active region to facilitate downstream processing stages. After the inspection stage, we leverage the cloud scaling functionality to slice the video into chunks for the encoding to expedite this computationally intensive process (more details in High Quality Video Encoding at Scale) with parallel chunk encoding in multiple cloud instances. Once all the chunks are encoded, they are physically stitched back into a final encoded bitstream. Lastly, the packager kicks in, adding a system layer to the asset, making it ready to be consumed by the clients.

Figure 1: A Simplified Video Processing Pipeline

With this architecture, chunk encoding is very efficient and processed in distributed cloud computing instances. However, assembly and packaging become the processing bottleneck, especially when the file size increases to the terabyte range. From chunk encoding to assembly and packaging, the result of each previous processing step must be uploaded to cloud storage and then downloaded by the next processing step.

Uploading and downloading data always come with a penalty, namely latency. While the input to our encoders, assemblers and packager instances is mounted using MezzFS and therefore read in parallel to be processed, output data is uploaded only after all processing is complete. The following table breaks down the various processing (including download) and uploading phases within an assembler and packager instance operating on large media files. It is worth pointing out that cloud processing is always subject to variable network conditions.

Table 2: Assembler and Packager Processing Time

Additionally, in this architecture, the packager still needs access to local storage for its packaged output (before uploading it to the final cloud destination) and any intermediate output if there are multiple passes in the processing. Since not all projects are terabytes projects, allocating the largest cloud storage to all packager instances is not an efficient use of cloud resources. We took the approach of allocating small or large cloud storage space depending on the actual packaging input size (Figure 2). Jobs processing large files were directed to instances with cloud storage large enough to hold intermediate results and packaged output.

Figure 2: Cloud Resource and Job Sizes

This initial architecture was designed at a time when packaging from a list of chunks was not possible and terabyte-sized files were not considered. It is very clear now that there will be significant processing savings if the physical assembly of the encoded chunks can be avoided. Also, the use of different sizes of local storage is not optimal, given that we can only support a small number of storage configurations, and given that the configurations need periodic updates as the maximum file size in the catalog inevitably grows.

Improved Architecture

In order to address the limitations of our initial architecture, we proceeded to make some optimizations.

Virtual Assembly

Figure 3 describes how a virtual assembly of the encoded chunks replaces the physical assembly used in our previous architecture. In this approach, an index assembler generates an index file, maintaining the temporal order of the encoded chunks. Care has been taken to ensure all chunks are accounted for and are in the right order during the virtual assembly to ensure the consistency of the final packaged stream and the original source. The index file keeps track of the physical location (URL) of each chunk and also keeps track of the physical location (URL + byte offset + size) of each video frame to facilitate downstream processing. The main advantage of using an assembled index is that any processing downstream of the video encoding can be abstracted away from the physical storage of the encoded video. Media processing services downstream of video encoding have intelligent downloaders that consume the assembled index file in order to mount the encoded video as video frames or encoded chunks. This is also the case with the packager, which reads and writes the encoded chunks only when it is generating the packaged output.

Figure 3: Video Processing with Index and Virtual Assembly

Using virtual assembly greatly improves the latency performance of the ProRes 422 HQ proxy generation by removing one round trip of cloud downloading and cloud uploading by the physical assembler.

Writable MezzFS

As described in a previous blog post, MezzFS is a tool developed by Netflix that allows cloud storage objects to be mounted as local files via FUSE. It allows our encoders and packagers to do random access reads of cloud storage objects without having to download an entire object before beginning their processing.

With similar goals in mind for write operations, we set about supporting storage of objects in the cloud without incurring any local storage and before the entire object has been created so that data generation and uploading can occur simultaneously. There are existing distributed file systems for the cloud as well as off-the-shelf FUSE modules for S3. We chose to enhance MezzFS instead of using these other solutions because the cloud storage system where packager stores its output is a custom object store service built on top of S3 with additional security features. Doing so has the added advantage of being able to design and tune the enhancement to suit the requirements of packager and our other encoding applications.

The requirements and challenges for supporting write operations are different from those for read operations. Our previous blog post described how MezzFS addresses the challenges for reads using various techniques, such as adaptive buffering and regional caches, to make the system performant and to lower costs. For write operations, those challenges do not apply. Furthermore, the goal for writes is not to build a general purpose system that supports arbitrary writers, but rather one that maximizes potential packager performance. In order to do so, we started by analyzing the packager’s IO patterns and configuring the packager to make its patterns more friendly to cloud writes.

The problematic pattern of packagers is that they do not always generate data linearly. They sometimes update parts of the file that had been written earlier for various reasons. For example, both ISOBMFF (ISO/IEC 14496–12) and Apple Quicktime use box structures to represent packaged media. The ‘moov’ box represents the metadata header describing the media while the ‘mdat’ box encapsulates the media content. Boxes start with a header which gives size and type of the box before the box content. When a packager is encapsulating the media content into the ‘mdat’ box, the size of the box is not known until all the media data are processed. To optimize the packager for writable MezzFS, we did not utilize the packager features that require multi-pass processing, intermediate storage and frequent update of headers.

With this packager constraint, there are a number of ways to design a writable MezzFS feature, but we wanted a solution that best fit the IO patterns of the packager in terms of latency, network utilization, and memory usage. In order to do that, the storage cloud object is modeled as a number of fixed size parts. MezzFS maintains a pool of upload buffers that correspond to a subset of these parts. As the packager creates and writes data to the object, data fills up the buffers, which are automatically uploaded asynchronously to the cloud. You can think of packaging as creating a stream of output that is stored directly in the cloud.

What happens when the packager references bytes that have already been uploaded (e.g. when it updates the ‘mdat’ size)? Any single read or write operation may involve a mix of previously uploaded and yet-to-be uploaded bytes. MezzFS borrows from how operating systems handle page faults. It downloads the part(s) that contain the referenced, uploaded bytes and keeps them in an LRU active cache. Packager’s read/write operations are then translated into operations on the upload buffers and/or buffers in the active cache. The buffers in the active cache are uploaded to the cloud as the cache becomes full. Just as with virtual memory management systems, locality of reference is important in determining the active cache size and performance. If the packager updates random, unrelated parts of the file within short periods of time, thrashing would occur and performance would be degraded. Our analysis of packager’s IO patterns determined that the packager makes updates with close proximity to each other — at most few parts at a time — thus making this design viable.

Figure 4: Overview of Writable MezzFS Design

Use of MezzFS is not without its cost in terms of performance. Use of FUSE means that file operations must go through MezzFS instead of directly to the kernel. As faster disk technology such as NVMe SSD are adopted, this overhead becomes increasingly noticeable and the time saved by uploading while packaging is counterbalanced by this overhead. As shown in Figure 5, packaging locally using non-NVMe local storage such as AWS Elastic Block Store (EBS) and then uploading takes more time than using MezzFS, but doing the same with NVMe SSD takes less time.

Figure 5: Performance Comparison of Packager Jobs using Writable MezzFS

However, time savings is not the only factor to consider when choosing between different upload techniques. Use of writable MezzFS offers the advantage of not requiring a large disk — larger and faster disks incur higher monetary costs and multiple disk size configurations make resource scheduling and sharing more challenging.

Conclusion

Supporting packaging of media content at terabytes scale is challenging. With innovation from system architecture, platform engineering and underlying packaging tools, processing terabyte-sized media files is now supported with greater efficiency.

The overall ProRes video processing speed is increased from 50GB/Hour to 300GB/Hour. From a different perspective, the processing time to movie runtime ratio is reduced from 6:1 to about 1:1. This significantly improves the Studio high quality proxy generation efficiency and workflow latency.

In addition to speed improvements, there is no longer a need to keep different configurations of local storage for cloud packagers. All the cloud packager instances now share a single scheduling queue with optimized compute resource utilization. There is also no need for multi-terabyte local storage, and no more unexpected out-of-disk processing failures. The single-sized local disk storage is future-proof for movies with longer runtime and higher resolution.

Acknowledgements

We would like to thank Anush Moorthy, Subbu Venkatrav and Chao Chen for their contribution to virtual assembly, Zoran Simic and Barak Alon for their contribution to writable MezzFS.

We’re hiring!

If you are passionate about media processing and platform engineering, come join us at Netflix! The Media Systems team is hiring. Please contact Flavio Ribeiro for more information.


Netflix Cloud Packaging in the Terabyte Era was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

How to Back Up Old Email Accounts

Post Syndicated from Nicole Perry original https://www.backblaze.com/blog/how-to-back-up-old-email-accounts/

Growing up, a common conversation I overheard between my mom and grandma went like this: “Do you have that recipe from our great aunt?”

“Sure, I do. Let me email it to you. Also, I have some funny jokes to forward along.”

My mom, and I’m guessing many others too, have kept every email they’ve ever received from their parents, family, and friends because they don’t want to lose the funny jokes, family recipes, announcements, and more that they’ve sent back and forth over the years. In the moment, our email accounts can feel like a day-to-day concern, or worse, a repository of spam. But for most of us, every email account holds some amount of treasured memories.

Nowadays, my mom has many different email accounts. But, she wanted to find a way to keep all of those emails she loved without having to keep the accounts themselves. She also found that she had so many emails in her inbox that she was running out of storage space.

Buying more storage can become expensive and doesn’t guarantee that those emails are safely backed up and remain accessible. One option is to download the emails, delete them in the client, and back them up somewhere reliable and accessible for the long term.

If you’re looking for a way to keep old emails or just want to clean up your inbox storage because you’re running out of space, this post walks you through the steps of how to download your data from various email platforms.

We’ve gathered a handful of guides to help you protect content across many different platforms—including social media, sync services, and more. We’re working on developing this list—please comment below if you’d like to see another platform covered.

Getting Started: How to Download One Email

If you know the exact email you want to make sure you have a copy of, it’s very easy to download it from any client.

For this example, we are going to use Gmail, but this should work for most email clients. If you run into an email client that it does not work with, feel free to note it in the comments below and we’ll update the guidance.

  1. Log in to the email address you would like to download a copy of the email from. (I’m using Gmail.)
  2. Find the email you would like to download. For this example, I will be downloading a family recipe sent by my mom.
  3. Select “Print” in the top right corner.
  4. When the print screen appears, save the email as a PDF on to your computer.
  5. And presto, you have a copy of that email you would like to save forever.

This process can be a bit tedious as you would have to download each email one at a time. It also can be tough if you don’t remember how to find the email you would like to save. If this is true, there are also ways that you can download all of your email data.

While there are other file formats you can download individual emails in, we strongly recommend that—if you want to be able to manage or search your old emails—you download all of your emails (which we explain how to do below). This provides the data in easily manageable formats and is far more time efficient.

Getting Serious: How to Download All of Your Emails

Below, I explain how to download your email data from two top free email websites. Don’t see the email platform you use? Leave a comment below and we’ll work to add material to help you!

How to Download Outlook Emails

A lot of people use Outlook for various reasons, often for work or school. If you downloaded Microsoft 365, then you also have access to Outlook email. To export your email from Outlook and save it as a PST file (don’t worry about what a PST file is quite yet, we’ll explain below), do the following:

  1. Sign in to your Outlook account.
  2. Click the gear button in the upper right corner.
  3. Scroll down on the settings panel to “View all Outlook settings.”
  4. Click on the button with a gear symbol labeled “General.”
  5. Select “Privacy and data” on the second panel that appears.
  6. On the right side, there will be a button labeled “Export mailbox.” Select this button.
  7. The button will grey out and a status update will appear to let you know the download is in progress.
  8. When the export is complete, we’ve found that Outlook may not notify your inbox. If this is the case, you will need to repeat steps one through five and navigate to the “Download here” button. This button will only appear once your emails are ready to download.
  9. Click “Download here” to download your PST file with all of your email data. (Scroll past the section on downloading Gmail data to learn what to do with this file type.)

How to Download Gmail Emails

In a previous post, we explained how to download all of your data from Google Drive. But, if you are just looking to download your Gmail data, here is a more detailed way to just do that.

  1. Log in to the Google Account you’d like to download your emails from.
  2. Once signed in, you will want to go to: myaccount.google.com.
  3. Go to the “Privacy & personalization” section and select “Manage your data & privacy.”
  4. On the next screen it takes you to, you’ll want to scroll down to a section labeled “Data from apps and services you use.” Here, you’ll select “Download your data” in the “Download or delete your data” section.
  5. From here, it’ll take you to the Google Takeout page. On this page, you’ll be given the option to select to download all of your Gmail emails and also your Google Chrome bookmarks, transactions from various Google services, locations stored in Google Maps, Google Drive contents, and other Google-related products you may use.
  6. If you want to download all your Google data, keep everything selected. If you just want a copy of your emails, deselect all and only select Google Mail to be downloaded.
  7. Click the next step on the bottom of the page.
  8. On the next page, you’ll decide what file type you would like it sent as, the frequency you would like this action to happen (Example: If you would like your data to be downloaded every six months, this is where you can set that to happen.), and the destination you would like your data to be sent to. For this example, I picked a one time download.
  9. Select “Create export” and you’ll see an export in progress page.
  10. An email will appear in a few minutes, hours, or a couple of days (depending on the size of data you are downloading), informing you that your Google data is ready to download. Once you have this email in your inbox, you have a week to download the data. Click the “Download your files” button in the email and you will have a ZIP file or a TGZ file (depending on what type of file you picked) on your computer with your Google data.
  11. When you open the ZIP, you will have all of your emails (including spam and trash) in an MBOX file.

What Is a PST File? What Is a MBOX File? How Do I Open Them?

A PST file is used by Microsoft programs to store data and items such as email messages, calendar events, and contacts. By moving items to an Outlook Data File (also known as a PST file) saved to your computer, you can free up storage space in the mailbox on your mail server. If you would like to make this file usable by other email clients, here’s a guide on how to convert your newly downloaded PST file to a MBOX file type.

An MBOX file is an email mailbox saved in a mail storage format used for organizing email messages in a single text file. It saves messages in a connected order where each message is stored after another, starting with the “From” header.

To open a MBOX file, you will need a third-party email program, such as Apple Mail or Mozilla Thunderbird. We recommend Mozilla Thunderbird, as it’s a free email client and it’s supported by both Macs and PCs.

This step is helpful if you would like to view the emails you downloaded. It also helps if you were looking to take the emails you downloaded and move them to a new inbox. For example, if you are afraid the email account you’ve used to sign up for everything over the past 10 years is vulnerable, you can download the emails from that inbox and move them to a new inbox using Apple Mail or Mozilla Thunderbird.

Great, now you’ve downloaded your emails. You’re not done yet! Read on to learn how to safely back up your emails so that you can hold on to them forever.

Use Backblaze B2 Cloud Storage Buckets to Keep an Organized Archive of Your Emails

Once you have your email data downloaded to your computer, it’s best practice to make sure that you have at least one copy of your data stored off-site in the cloud. Storing it in the cloud alongside two local copies ensures you never lose all those important emails.

A simple way to do this is with Backblaze B2, where you can upload and organize your files in buckets. To upload your files to a bucket, follow the steps below.

  1. Sign in to your Backblaze account.
  2. In the left hand column, select “Buckets” under the section “B2 Cloud Storage.”
  3. Click on the button “Create a bucket.”
  4. In the next step, you will need to create a unique name for your bucket and select some settings for it, like if it will be public or private or if you would like to enable encryption.
  5. Once the bucket is created, it will take you to a page where you can upload your files. You will want to drag and drop the email files you want to upload to it. If the MBOX file is too large to drag and drop into the bucket, you can use a third-party integration like Cyberduck to facilitate the upload. You can read the guide to using Cyberduck for Backblaze B2 bucket uploads here.

Alternatively, if you’re not worried about organizing or working with your email archives and just want to know they’re stored away safely, you can keep your downloaded files on your computer. If you follow this route, remember to sign up for a backup service that makes a copy of all of your computer’s files in the cloud. In the case of any data loss, a service like Backblaze Computer Backup would have a copy of all of your data ready for you to restore. If your email applications are locally stored on your computer, Backblaze will automatically back up your emails. You can learn more about how this works here. This approach will take up more room on your computer, but it’s a simple path to peace of mind.

From here, your MBOX file with all your emails from your family, friends, and reminders to yourself (We all have those!) will be safe in the cloud. If you ever want to pull out the archive and read the emails you saved, remember to use the third-party tools mentioned above. What’s important is that you have all your memories stored, safely with a provider who will ensure their redundancy and reliability.

Have questions or want to see a guide for an email client we didn’t mention above? Feel free to let us know in the comments!

The post How to Back Up Old Email Accounts appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

coreutils-9.0 released

Post Syndicated from original https://lwn.net/Articles/870372/rss

The GNU Core Utilities (coreutils) has announced the release of version 9.0 of “the basic file, shell and text manipulation utilities” used by the GNU operating system and various Linux distributions. In the year and a half or so since the last major release (8.32), various new features were added, including:

cp has changed how it handles data

  • enables CoW [copy on write] by default (through FICLONE ioctl),
  • uses copy offload where available (through copy_file_range),
  • detects holes differently (though SEEK_HOLE)
  • This also applies to mv and install.

Field Notes: Set Up a Highly Available Database on AWS with IBM Db2 Pacemaker

Post Syndicated from Sai Parthasaradhi original https://aws.amazon.com/blogs/architecture/field-notes-set-up-a-highly-available-database-on-aws-with-ibm-db2-pacemaker/

Many AWS customers need to run mission-critical workloads—like traffic control system, online booking system, and so forth—using the IBM Db2 LUW database server. Typically, these workloads require the right high availability (HA) solution to make sure that the database is available in the event of a host or Availability Zone failure.

This HA solution for the Db2 LUW database with automatic failover is managed using IBM Tivoli System Automation for Multiplatforms (Tivoli SA MP) technology with IBM Db2 high availability instance configuration utility (db2haicu). However, this solution is not supported on AWS Cloud deployment because the automatic failover may not work as expected.

In this blog post, we will go through the steps to set up an HA two-host Db2 cluster with automatic failover managed by IBM Db2 Pacemaker with quorum device setup on a third EC2 instance. We will also set up an overlay IP as a virtual IP pointing to a primary instance initially. This instance is used for client connections and in case of failover, the overlay IP will automatically point to a new primary instance.

IBM Db2 Pacemaker is an HA cluster manager software integrated with Db2 Advanced Edition and Standard Edition on Linux (RHEL 8.1 and SLES 15). Pacemaker can provide HA and disaster recovery capabilities on AWS, and an alternative to Tivoli SA MP technology.

Note: The IBM Db2 v11.5.5 database server implemented in this blog post is a fully featured 90-day trial version. After the trial period ends, you can select the required Db2 edition when purchasing and installing the associated license files. Advanced Edition and Standard Edition are supported by this implementation.

Overview of solution

For this solution, we will go through the steps to install and configure IBM Db2 Pacemaker along with overlay IP as virtual IP for the clients to connect to the database. This blog post also includes prerequisites, and installation and configuration instructions to achieve an HA Db2 database on Amazon Elastic Compute Cloud (Amazon EC2).

Figure 1. Cluster management using IBM Db2 Pacemaker

Prerequisites for installing Db2 Pacemaker

To set up IBM Db2 Pacemaker on a two-node HADR (high availability disaster recovery) cluster, the following prerequisites must be met.

  • Set up instance user ID and group ID.

Instance user id and group id’s must be set up as part of Db2 Server installation which can be verified as follows:

grep db2iadm1 /etc/group
grep db2inst1 /etc/group

  • Set up host names for all the hosts in /etc/hosts file on all the hosts in the cluster.

For both of the hosts in the HADR cluster, ensure that the host names are set up as follows.

Format: ipaddress fully_qualified_domain_name alias

  • Install kornshell (ksh) on both of the hosts.

sudo yum install ksh -y

  • Ensure that all instances have TCP/IP connectivity between their ethernet network interfaces.
  • Enable password less secure shell (ssh) for the root and instance user IDs across both instances.After the password less root ssh is enabled, verify it using the “ssh <host name> -l root ls” command (hostname is either an alias or fully-qualified domain name).

ssh <host name> -l root ls

  • Activate HADR for the Db2 database cluster.
  • Make available the IBM Db2 Pacemaker binaries in the /tmp folder on both hosts for installation. The binaries can be downloaded from IBM download location (login required).

Installation steps

After completing all prerequisites, run the following command to install IBM Db2 Pacemaker on both primary and standby hosts as root user.

cd /tmp
tar -zxf Db2_v11.5.5.0_Pacemaker_20201118_RHEL8.1_x86_64.tar.gz
cd Db2_v11.5.5.0_Pacemaker_20201118_RHEL8.1_x86_64/RPMS/

dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y
dnf install */*.rpm -y

cp /tmp/Db2_v11.5.5.0_Pacemaker_20201118_RHEL8.1_x86_64/Db2/db2cm /home/db2inst1/sqllib/adm

chmod 755 /home/db2inst1/sqllib/adm/db2cm

Run the following command by replacing the -host parameter value with the alias name you set up in prerequisites.

/home/db2inst1/sqllib/adm/db2cm -copy_resources
/tmp/Db2_v11.5.5.0_Pacemaker_20201118_RHEL8.1_x86_64/Db2agents -host <host>

After the installation is complete, verify that all required resources are created as shown in Figure 2.

ls -alL /usr/lib/ocf/resource.d/heartbeat/db2*

Figure 2. List of heartbeat resources

Configuring Pacemaker

After the IBM Db2 Pacemaker is installed on both primary and standby hosts, initiate the following configuration commands from only one of the hosts (either primary or standby hosts) as root user.

  1. Create the cluster using db2cm utility.Create the Pacemaker cluster using db2cm utility using the following command. Before running the command, replace the -domain and -host values appropriately.

/home/db2inst1/sqllib/adm/db2cm -create -cluster -domain <anydomainname> -publicEthernet eth0 -host <primary host alias> -publicEthernet eth0 -host <standby host alias>

Note: Run ifconfig to get the –publicEthernet value and replace in the former command.

  1. Create instance resource model using the following commands.Modify -instance and -host parameter values in the following command before running.

/home/db2inst1/sqllib/adm/db2cm -create -instance db2inst1 -host <primary host alias>
/home/db2inst1/sqllib/adm/db2cm -create -instance db2inst1 -host <standby host alias>

  1. Create the database instance using db2cm utility. Modify -db parameter value accordingly.

/home/db2inst1/sqllib/adm/db2cm -create -db TESTDB -instance db2inst1

After configuring Pacemaker, run crm status command from both the primary and standby hosts to check if the Pacemaker is running with automatic failover activated.

Figure 3. Pacemaker cluster status

Quorum device setup

Next, we shall set up a third lightweight EC2 instance that will act as a quorum device (QDevice) which will act as a tie breaker avoiding a potential split-brain scenario. We need to install only corsync-qnetd* package from the Db2 Pacemaker cluster software.

Prerequisites (quorum device setup)

  1. Update /etc/hosts file on Db2 primary and standby instances to include the host details of QDevice EC2 instance.
  2. Set up password less root ssh access between Db2 instances and the QDevice instance.
  3. Ensure TCP/IP connectivity between the Db2 instances and the QDevice instance on port 5403.

Steps to set up quorum device

Run the following commands on the quorum device EC2 instance.

cd /tmp
tar -zxf Db2_v11.5.5.0_Pacemaker_20201118_RHEL8.1_x86_64.tar.gz
cd Db2_v11.5.5.0_Pacemaker_20201118_RHEL8.1_x86_64/RPMS/
dnf install */corosync-qnetd* -y

  1. Run the following command from one of the Db2 instances to join the quorum device to the cluster by replacing the QDevice value appropriately.

/home/db2inst1/sqllib/adm/db2cm -create -qdevice <hostnameofqdevice>

  1. Verify the setup using the following commands.

From any Db2 servers:

/home/db2inst1/sqllib/adm/db2cm -list

From QDevice instance:

corosync-qnetd-tool -l

Figure 4. Quorum device status

Setting up overlay IP as virtual IP

For HADR activated databases, virtual IP provides a common connection point for the clients so that in case of failovers there is no need to update the connection strings with the actual IP address of the hosts. Furtermore, the clients can continue to establish the connection to the new primary instance.

We can use the overlay IP address routing on AWS to send the network traffic to HADR database servers within Amazon Virtual Private Cloud (Amazon VPC) using a route table so that the clients can connect to the database using the overlay IP from the same VPC (any Availability Zone) where the database exists. aws-vpc-move-ip is a resource agent from AWS which is available along with the Pacemaker software that helps to update the route table of the VPC.

If you need to connect to the database using overlay IP from on-premises or outside of the VPC (different VPC than database servers), then additional setup is needed using either AWS Transit Gateway or Network Load Balancer.

Prerequisites (setting up overlay IP as virtual IP)

  • Choose the overlay IP address range which needs to be configured. This IP should not be used anywhere in the VPC or on-premises, and should be a part of the private IP address range as defined in RFC 1918. If the VPC is configured in the range of 0.0.0.0/8 or 172.16.0.0/12, we can use the overlay IP from the range of 192.168.0.0/16.We will use the following IP and ethernet settings.

192.168.1.81/32
eth0

  • To route traffic through overlay IP, we need to disable source and target destination checks on the primary and standby EC2 instances.

aws ec2 modify-instance-attribute –profile <AWS CLI profile> –instance-id EC2-instance-id –no-source-dest-check

Steps to configure overlay IP

The following commands can be run as root user on the primary instance.

  1. Create the following AWS Identity and Access Management (IAM) policy and attach it to the instance profile. Update region, account_id, and routetableid values.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt0",
      "Effect": "Allow",
      "Action": "ec2:ReplaceRoute",
      "Resource": "arn:aws:ec2:<region>:<account_id>:route-table/<routetableid>"
    },
    {
      "Sid": "Stmt1",
      "Effect": "Allow",
      "Action": "ec2:DescribeRouteTables",
      "Resource": "*"
    }
  ]
}
  1. Add the overlay IP on the primary instance.

ip address add 192.168.1.81/32 dev eth0

  1. Update the route table (used in Step 1) with the overlay IP specifying the node with the Db2 primary instance. The following command returns True.

aws ec2 create-route –route-table-id <routetableid> –destination-cidr-block 192.168.1.81/32 –instance-id <primrydb2instanceid>

  1. Create a file overlayip.txt with the following command to create the resource manager for overlay ip.

overlayip.txt

primitive db2_db2inst1_db2inst1_TESTDB_AWS_primary-OIP ocf:heartbeat:aws-vpc-move-ip \
  params ip=192.168.1.81 routing_table=<routetableid> interface=<ethernet> profile=<AWS CLI profile name> \
  op start interval=0 timeout=180s \
  op stop interval=0 timeout=180s \
  op monitor interval=30s timeout=60s

eifcolocation db2_db2inst1_db2inst1_TESTDB_AWS_primary-colocation inf:

db2_db2inst1_db2inst1_TESTDB_AWS_primary-OIP:Started
db2_db2inst1_db2inst1_TESTDB-clone:Master

order order-rule-db2_db2inst1_db2inst1_TESTDB-then-primary-oip Mandatory:

db2_db2inst1_db2inst1_TESTDB-clone db2_db2inst1_db2inst1_TESTDB_AWS_primary-OIP
location prefer-node1_db2_db2inst1_db2inst1_TESTDB_AWS_primary-OIP

db2_db2inst1_db2inst1_TESTDB_AWS_primary-OIP 100: <primaryhostname>
location prefer-node2_db2_db2inst1_db2inst1_TESTDB_AWS_primary-OIP

db2_db2inst1_db2inst1_TESTDB_AWS_primary-OIP 100: <standbyhostname>

The following parameters must be replaced in the resource manager create command in the file.

    • Name of the database resource agent (This can be found through crm config show | grep primitive | grep DBNAME command. For this example, we will use: db2_db2inst1_db2inst1_TESTDB)
    • Overlay IP address (created earlier)
    • Routing table ID (used earlier)
    • AWS command-line interface (CLI) profile name
    • Primary and standby host names
  1. After the file with commands is ready, run the following command to create the overlay IP resource manager.

crm configure load update overlayip.txt

  1. Next, create the VIP resource manager—not in managed state. Run the following command to manage and start the resource.

crm resource manage db2_db2inst1_db2inst1_TESTDB_AWS_primary-OIP

  1. Validate the setup with crm status command.

Figure 5. Pacemaker cluster status along with overlay IP resource

Test failover with client connectivity

For the purpose of this testing, launch another EC2 instance with Db2 client installed, and catalog the Db2 database server using overlay IP.

Figure 6. Database directory list

Establish a connection with the Db2 primary instance using the cataloged alias (created earlier) using overlay IP address.

Figure 7. Connect to database

If we connect to the primary instance and check the applications connected, we can see the active connection from the client’s IP as shown in Figure 8.

Check client connections before failover

Figure 8. Check client connections before failover

Next, let’s stop the primary Db2 instance and check if the Pacemaker cluster promoted the standby to primary and we can still connect to the database using the overlay IP, which now points to the new primary instance.

If we check the CRM status from the new primary instance, we can see that the Pacemaker cluster has promoted the standby database to new primary database as shown in Figure 9.

Figure 9. Automatic failover to standby

Let’s go back to our client and reestablish the connection using the cataloged DB alias created using overlay IP.

Figure 10. Database reconnection after failover

If we connect to the new promoted primary instance and check the applications connected, we can see the active connection from the client’s IP as shown in Figure 11.

Check client connections after failover

Figure 11. Check client connections after failover

Cleaning up

To avoid incurring future charges, terminate all EC2 instances which were created as part of the setup referencing this blog post.

Conclusion

In this blog post, we have set up automatic failover using IBM Db2 Pacemaker with overlay (virtual) IP to route traffic to secondary database instance during failover, which helps to reconnect to the database without any manual intervention. In addition, we can also enable automatic client reroute using the overlay IP address to achieve a seamless failover connectivity to the database for mission-critical workloads.

Field Notes provides hands-on technical guidance from AWS Solutions Architects, consultants, and technical account managers, based on their experiences in the field solving real-world business problems for customers.

The Proliferation of Zero-days

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2021/09/the-proliferation-of-zero-days.html

The MIT Technology Review is reporting that 2021 is a blockbuster year for zero-day exploits:

One contributing factor in the higher rate of reported zero-days is the rapid global proliferation of hacking tools.

Powerful groups are all pouring heaps of cash into zero-days to use for themselves — and they’re reaping the rewards.

At the top of the food chain are the government-sponsored hackers. China alone is suspected to be responsible for nine zero-days this year, says Jared Semrau, a director of vulnerability and exploitation at the American cybersecurity firm FireEye Mandiant. The US and its allies clearly possess some of the most sophisticated hacking capabilities, and there is rising talk of using those tools more aggressively.

[…]

Few who want zero-days have the capabilities of Beijing and Washington. Most countries seeking powerful exploits don’t have the talent or infrastructure to develop them domestically, and so they purchase them instead.

[…]

It’s easier than ever to buy zero-days from the growing exploit industry. What was once prohibitively expensive and high-end is now more widely accessible.

[…]

And cybercriminals, too, have used zero-day attacks to make money in recent years, finding flaws in software that allow them to run valuable ransomware schemes.

“Financially motivated actors are more sophisticated than ever,” Semrau says. “One-third of the zero-days we’ve tracked recently can be traced directly back to financially motivated actors. So they’re playing a significant role in this increase which I don’t think many people are giving credit for.”

[…]

No one we spoke to believes that the total number of zero-day attacks more than doubled in such a short period of time — just the number that have been caught. That suggests defenders are becoming better at catching hackers in the act.

You can look at the data, such as Google’s zero-day spreadsheet, which tracks nearly a decade of significant hacks that were caught in the wild.

One change the trend may reflect is that there’s more money available for defense, not least from larger bug bounties and rewards put forward by tech companies for the discovery of new zero-day vulnerabilities. But there are also better tools.

Ransomware: Is Critical Infrastructure in the Clear?

Post Syndicated from Jen Ellis original https://blog.rapid7.com/2021/09/24/ransomware-is-critical-infrastructure-in-the-clear/

Ransomware: Is Critical Infrastructure in the Clear?

Recently I’ve been getting asked whether I believe ransomware is on the decline, particularly for critical infrastructure. Part of the reason for this question seems to be a recent security briefing from White House deputy national security adviser Anne Neuberger, suggesting that language on the site of a new-but-already-high-profile ransomware gang, BlackMatter, could indicate that President Biden’s comments to President Putin regarding consequences for attacks against US critical infrastructure may have hit their mark. Yet just this week, this same gang demanded a ransom of $5.9 million for an attack on Iowa-based feed and grain cooperative, NEW Cooperative.

So the question remains: Is critical infrastructure in the clear, is it a specific target of ransomware attackers, or is it simply on the same footing as any other organization? As we’ll see — and as current developments confirm — it’s clear that critical infrastructure is indeed at risk from ransomware attacks.

Before I get into the nuances of this, I want to quickly note upfront that much of this is going to be opinion or theories based on discussion with — and anecdotal evidence from — various security experts, ransomware victims, and news stories. I’m not a ransomware attacker, nor am I directly in touch with any, so I can only speculate on their motivations, interests, and plans. Where possible, I provide reference to further reading to provide context, but in general, it’s important to note that broad under-reporting and inconsistent handling of ransomware incident data means that any predictions, projections, or summaries of ransom activity (on this blog or elsewhere) are likely somewhat incomplete.

The BlackMatter at hand

The BlackMatter website indicates that the group is somewhat selective in which organizations it will target for attacks. According to The Record, BlackMatter is particularly interested in organizations with a revenue of over $100 million a year, with networks of 500 to 1,500 hosts located in the US, the UK, Canada, or Australia. They state they are specifically not planning to attack organizations in the following sectors and would in fact decrypt data for free should they infect any organizations in them:

  • Hospitals
  • Critical infrastructure facilities (nuclear power plants, power plants, water treatment facilities)
  • Oil and gas industry (pipelines, oil refineries)
  • Defense industry
  • Nonprofit companies
  • Government sector

Interestingly, they do not include the food and agriculture sector in this list, though it is included in the US government’s list of 16 critical infrastructure sectors. When NEW Cooperative’s representatives pointed this out to BlackMatter, the ransomware group’s response was:

You do not fall under the rules, everyone will only incur losses, everything is tied to the commerce, the critical ones mean the vital needs of a person.

On the surface, it’s funny to think they are saying food isn’t a vital need for people. The JBS attack at the start of June highlighted the importance of the food supply chain. The cost of basic meat food staples is still higher in the US as a result of the attack, which can make a huge difference to those living on or under the poverty line. BlackMatter explains the distinction in terms of the impact — it views loss of money for the company itself as the only real impact of the NEW Cooperative attack.

This may be because NEW Cooperative is a fairly small, regional entity, nowhere near the scale of JBS, and therefore disruption for them is not going to create anywhere close to the same level of impact on the US food supply chain. This leads to the question of whether these types of organizations really count as critical infrastructure on an individual level. That’s a question for the US government to answer as they determine whether to respond to this attack and others like it. If you want to get into this more, Joseph Marks has a great write-up on the different aspects in his coverage for the The Washington Post’s Cybersecurity 202.

In the meantime, it is interesting to see BlackMatter communicate so proactively on the topic of critical infrastructure and what they consider to be in scope. This could, as Anne Neuberger suggested, reflect a heeding of the President’s warning. It could also be somewhat influenced by the lessons learned from DarkSide’s experiences following the Colonial Pipeline attack back in May. BlackMatter states, “The project has incorporated in itself the best features of DarkSide, REvil, and LockBit,” so it’s entirely possible their communications strategy is informed by the blowback DarkSide experienced in the wake of Colonial.

After coming under intense scrutiny and focus following the Colonial Pipeline attack, the DarkSide group published a statement describing themselves as “apolitical” and asserting, “Our goal is to make money and not creating problems for society.” When its infrastructure was then compromised and their bitcoin drained, the group decided it was time to shut up shop and lay low. This prompted a great deal of speculation from security commentators over whether they would reappear under a different name after sufficient time had passed. It didn’t take long after the appearance of BlackMatter for security researchers to start pointing to indicators that the new ransomware group may be the phoenix rising from DarkSide’s ashes.

Hackers with hearts of gold?

DarkSide and BlackMatter are not the only ransomware gangs to draw a line around healthcare and other targets that can impact public safety.

In March 2020, as the pandemic ramped up in ferocity, Bleeping Computer reached out to a number of high-profile ransomware groups and asked if they would lay off healthcare organizations in light of COVID-19. The group behind the CLOP ransomware stated that they have “never attacked hospitals, orphanages, nursing homes, charitable foundations, and we won’t.” They went on to state, “We are not enemies of humanity… our goal is money, not harm,” and they indicated that if a healthcare organization was encrypted by accident, they would provide the decryptor for free.

Four other ransomware groups responded to Bleeping Computer with similar assertions that hospitals are never targets or would not be during the duration of the pandemic. Some even sounded offended by the suggestion that hospitals could ever be considered fair game for attacks.

Critical infrastructure attacks abound

Yet, despite this, attacks against the healthcare sector were prolific throughout 2020. According to the 2021 Unit 42 Ransomware Threat Report, “the healthcare sector… was the most targeted vertical for ransomware in 2020. Ransomware operators were brazen in their attacks in an attempt to make as much money as possible, knowing that healthcare organizations – which needed to continue operating to treat COVID-19 patients and help save lives – couldn’t afford to have their systems locked out and would be more likely to pay a ransom.”

We see the same trend continuing in 2021. The fantastic Black Fog site tracks publicly disclosed ransomware attacks on The State of Ransomware in 2021. Their stats highlight that 2021 continues to be a busy year for ransomware attackers and their victims, with more attacks in every month of 2021 than during their 2020 counterpart. They break down the attacks they track by industry sector, and the top 9 are all covered within the US government’s description of its 16 sectors of critical infrastructure. Healthcare is the fourth most impacted sector according to their analysis, with government and education taking the first and second spots.

So does this mean that these sectors are in fact being highly targeted for attack? The answer is complicated, and there are a number of factors at play.

It’s worth calling out again that ransomware and other cybercrime remains terribly under-reported. It’s possible that one of the reasons we “see” most attacks in the sectors mentioned earlier is because they are very public-facing in nature. Thus, disruptive attacks against their systems may be more visible to the public — and hence more easily tracked and reported. Other sectors may be better able to avoid public disclosure, possibly in the hopes of avoiding a loss of customer confidence or regulatory or legal implications.

This does not mean that these sectors are not also appealing targets for some cybercriminals. Healthcare, government, and educational organizations are often highly vulnerable to attack due to a number of factors including a deficit of resources, reliance on legacy systems, complexity of technical ecosystems and user behavior models, and lack of tolerance for downtime due to the consequences to the public of a halt in operations. This latter point may also mean these sectors are more likely to pay a ransom demand: If an entity can’t tolerate downtime enough to patch their systems, an attacker may speculate that they will also likely want to resolve a ransomware incident as quickly as possible, resulting in a paid ransom.

So, the question comes down to whether attackers think this way and specifically target these sectors.

Targets locked and loaded?

One of the things that most caught my attention about the DarkMatter website information, the responses to Bleeping Computer, and Unit 42’s research was that they all seem to reflect the notion that ransom attacks are targeted. Indeed, in its response to Bleeping Computer, the Nefilim Ransomware group stated, “We work very diligently in choosing our targets.”

Yet the BlackMatter site and a couple of the other responses also alluded to organizations being infected by accident. In its response to Bleeping Computer, the Netwalker group stated:

Hospitals and medical facilities? do you think someone has a goal to attack hospitals? we don’t have that goal -it never was. it coincidence. no one will purposefully hack into the hospital. [sic]

But they then went on to add:

If someone is encrypted, then he must pay for the decryption.

The implication here is that while they may not go out of their way to target hospitals or any other organization, their attacks are opportunistic and whoever is hit is fair game and expected to pay.

So how do these things relate to each other? How can an attack be both targeted and run the risk of accidentally infecting unintended organizations?

First, consider the nature of profit-motivated attacks of this type. While there are profit-driven attacks that are extremely targeted —for example corporate espionage attacks — in the case of ransomware attacks, it is more likely that groups of organized cybercriminals are going to try to maximize their potential profits by orchestrating attacks at scale. By casting their nets wide, they are able to get more bang for their buck/ruble, making the most of their upfront investment to increase the odds of hitting organizations that are willing to pay. They may have an ideal target profile as indicated on BlackMatter’s site, but that doesn’t mean they won’t take a spray-and-pray approach to see what they can hit. Even with a focus on a specific demographic, they are still likely to take a fairly broad approach to maximize the potential for profit.

This is consistent with the most common attack methodologies for extortion-based attacks. According to Digital Defense, phishing, RDP, and vulnerable systems are the top three attack vectors for ransomware attacks. While any of these can be leveraged in highly targeted attacks, it’s more common for them to be used at scale. Phishing emails are sent out to vast lists of potential recipients, and malware to exploit RDP or other exposed systems is automated and set loose on the internet. With this in mind, it’s not surprising that organizations that weren’t being directly targeted will be impacted.

While it’s important to note that the opportunistic nature of these attack methodologies means any organization can fall victim to a ransomware attack, that does not mean that specific sectors or geographies are not more likely to be hit. The majority of profit-motivated attackers may not be targeting specific organizations (unless there is another motivation at play), but that doesn’t mean they can’t target groups or classes of organization, as we see with BlackMatter’s website. The sheer volume of attacks hitting the US indicates that whatever the chosen attack vector, it is often pointed towards specific geographical regions. Likewise, it’s possible or in some cases, likely, that attackers develop phishing target lists with data specific to certain sectors that they believe will be more easily compromised or likely to pay. As already noted, critical infrastructure is viewed by many as sitting firmly in this category.

Critical infrastructure not in the clear

So what does all this mean? The incomplete data we have clearly shows that ransomware attacks are not in decline and critical infrastructure is certainly not in the clear.

We need more consistent reporting of ransom incidents to get a clearer picture of what’s really happening, but we can confidently say healthcare providers, governments, and education are regularly being hit and need greater support to help them tackle the security issues I mentioned earlier.

The good news is that this is a problem that many are scrutinizing, and we’re starting to see more resources and assistance for critical infrastructure. If you work in one of the US critical infrastructure sectors, check out the free tools and services CISA provides to help you protect yourself. If you are working for a government entity (including public education and healthcare providers), you may also qualify for free services from the MS-ISAC.

In addition, the US Senate recently passed infrastructure legislation that would provide federal grants and funding to several critical infrastructure sectors — such as state and local governments, energy, and water — to help them strengthen their cybersecurity postures. We hope this may be extending and that, as Congress considers large spending bills, the healthcare sector should be provided access to federal funding and other resources dedicated to cybersecurity.

The US government has also announced a number of other measures both to address ransomware and to shore up cybersecurity in critical infrastructure. We hope that, over time, we will see these efforts bearing fruit in the form of less successful attacks against critical infrastructure.

The Ransomware Task Force also identified a number of recommendations for governments to better support critical infrastructure, from grant funding (pages 40 and 41) to mandated adoption of cyber hygiene measures (page 39) and provision of emergency response authorities in the event of a successful attack (page 42). The US government is already taking action on some of these priorities, such as requiring greater cyber hygiene for federal agencies and contractors, and including a response and recovery fund for victims of cyberattack in the pending infrastructure legislation.

Although all public data sources agree that far more ransomware attacks are being reported in the US than in any other country, this is not only a US issue. Many other countries are impacted, and we see critical infrastructure being hit around the world. Governments in other affected countries are likely taking or investigating similar measures, though to date, they have mostly been less vocal on it in public.

In an ideal world, governments will work together to amplify the impact of their actions and proactively deter and disrupt the global ransomware market. To that end, I look forward to seeing what will come from the Extraordinary Senior Officials Forum on ransomware that the G7 has committed to holding before the end of 2021.

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

The collective thoughts of the interwebz