Уейн Шортър – кадри от живота на един истински супергерой

Post Syndicated from Таня Иванова original https://www.toest.bg/wayne-shorter-kadri-ot-zhivota-na-edin-istinski-supergeroy/

Време е да си намеря ново тяло и да се върна, за да продължа мисията си.

Уейн Шортър – кадри от живота на един истински супергерой

С тези думи на Уейн Шортър на 2 март семейството на големия музикант оповести новината за смъртта му. Съобщението гласеше още: „Днес в 4 ч. сутринта Уейн мирно продължи необятното си пътешествие в неизвестното. Той беше заобиколен от любящото си семейство.“ Колкото и странно да звучи, в този момент не изпитах тъга от загубата, а голяма благодарност, че Уейн Шортър беше наш съвременник и озаряваше живота ни със своята мъдрост и гений. Вероятно за това успокоение допринесе и будистката перспектива в написаното от роднините на саксофониста, в което се прокрадва идеята, че нищо не свършва със смъртта и че животът е един непрестанен процес на промяна.

Уейн Шортър беше велик композитор и саксофонист, дързък импровизатор, ментор на десетки млади музиканти. Дълбоко в същността си той беше философ и свободен дух. След преселването му във вечността последваха обяснения в любов и прояви на признателност, споделяне на спомени, истории, снимки…

„Свирѝ музика за такъв свят, какъвто искаш да сътвориш.“

Няколко месеца по-късно, по случай рождения му ден, който е на 25 август, Хърби Хенкок и приятели (сред които най-добрите джаз музиканти на планетата) се събраха в амфитеатъра Hollywood Bowl в Лос Анджелис пред 11 000 души, за да отпразнуват 90-тия юбилей на Шортър. По това време в Amazon Prime беше разпространен документалният филм за него „Нулева гравитация“ (Zero Gravity), сниман в продължение на повече от две десетилетия. Филмът е чудесна възможност да се потопим дълбоко в света на Уейн Шортър, тъй като режисьорката Дорсей Алави успява да изгради открит, честен и обаятелен портрет на една магнетична личност.

Алави и саксофонистът се запознават през 90-те години. Тя е режисьорка на музикални видеоклипове и работи основно с независими артисти и групи. Един ден агентът ѝ се обажда с новината, че от Verve Records искат да се направи кратък филм за Уейн Шортър. Така се появява видеото High Life, в което саксофонистът разказва за знакови сътрудничества в кариерата си и разкрива своя поглед към музиката и живота. В онзи момент Алави е имала представа за някои от най-големите имена в джаза, но не е знаела кой е Уейн Шортър. Постепенно, докато го опознава и научава все повече за приноса и влиянието му, тя е очарована от него и между двамата се заражда силно приятелство.

Първите кадри за „Нулева гравитация“ са заснети още през 2002 г., когато Шортър е бил вдъхновен от създаването на симфоничната си музика и от сътрудничеството на квартета му с различни оркестри. За Дорсей Алави това е бил важен момент за документиране. Същинската работа по филма започва през 2013 г., тъй като е било необходимо да се намери финансиране. Продукцията е завършена пет години по-късно и е с продължителност малко над три часа. „Нулева гравитация“ достига до зрителите в три части, всяка от които е като духовен портал за различен период от живота и творчеството на Уейн Шортър.

По време на специалната прожекция в края на 2018 г., събрала саксофониста и семейството му, приятели, музиканти и дарители, в залата сред поканените е и Брад Пит. Още с появата си той уточнява, че ще успее да изгледа само първата част на филма, тъй като след това има друг ангажимент. Но въздействието на видяното е толкова силно, а историята на Уейн Шортър – толкова впечатляваща и необикновена, че Брад Пит остава до края на прожекцията, след което предлага на Дорсей Алави да съдейства по-нататък и да подкрепи филма. Това е причината името на Пит да бъде вписано в „Нулева гравитация“ като изпълнителен продуцент.

„В живота на човека има две велики събития – едното е раждането, другото е да разбереш защо си се родил.“

С това послание, изречено от Уейн Шортър, започва документалният филм за него. Думите всъщност са на американския драматург, сценарист и поет Тенеси Уилямс, които саксофонистът много е харесвал и често е обичал да цитира.

Две важни и определящи събития се случват в живота на Шортър още като дете: той чува за първи път бибоп по радиото и открива света на киното. Бързо става толкова подвластен на образите от филмите, че често бяга от училище, за да се озове в някой киносалон. Години по-късно Шортър ще посочи като свое най-голямо вдъхновение филмовите герои и книгата „Водните дечица“ на Чарлс Кингсли, която го отвежда към идеята, че приказките са като ДНК на вселената. Сред големите му страсти през тези години са и комиксите. На осемгодишна възраст той започва да прерисува любимите си комикс герои, а на петнайсет измисля и създава свой комикс, запазен и до днес.

Уейн Шортър – кадри от живота на един истински супергерой
Кадър от филма „Нулева гравитация“

Уейн Шортър остава свързан с тези ранни влияния до края на живота си. В „Нулева гравитация“ зрителите ще видят как в кабинета му, недалеч от неговите 12 награди „Грами“, са подредени десетки малки пъстри фигури на комикс и фентъзи герои, превърнали го в страстен колекционер, който се грижи добре за детето в себе си. Пример за тази силна любов към рисуваните истории е и решението на саксофониста през 2018 г. да придружи издаването на тройния си албум Emanon с графичен роман, написан от самия него и създаден в сътрудничество с художника на комикси Ранди Дюбърк.

Това, че Уейн Шортър притежава силата и чистотата на детското въображение, не убягва на барабаниста Арт Блеки. Той е една от първите важни личности в кариерата на Шортър, които виждат големия потенциал у него. В края на 50-те години саксофонистът се присъединява към прочутата група Art Blakey and the Jazz Messengers. Той определя Блеки като „цунами“ и „голям шоумен“. Годините, които прекарва във формацията, са особено плодотворен период за Шортър. Той получава възможност да израства, подготвяйки се за самостоятелния си път по-нататък, тъй като, от една страна, Jazz Messengers са имали репутацията на група, създаваща бендлидери, а от друга, Шортър непрекъснато усъвършенства композиторските си умения. Музиката, написана от него, се превръща в скритото му оръжие и става част от репертоара на групата.

Времето на Уейн Шортър в Art Blakey and the Jazz Messengers е предопределено, тъй като той трябва да се придвижи на следващото ниво. Един ден Шортър получава обаждане от Майлс Дейвис с думите: „Когато си готов, кажи ми!“ Така през 1964 г. Шортър става част от Втория велик квинтет на тромпетиста заедно с Хърби Хенкок (пиано), Рон Картър (контрабас) и Тони Уилямс (барабани). Динамиката на професионалните си взаимоотношения с Майлс Дейвис той описва шеговито през призмата на спорта. Твърди, че няма нужда да ходиш на фитнес, щом свириш редом до Майлс, и разкрива, че е трябвало много да работи, за да се почувства уверен на сцената до него.

Не е изненада, че Майлс Дейвис цени високо музиката, създадена от Уейн Шортър. „Ще се видим в студиото. И донеси онази тетрадка“ – обръща се той към саксофониста преди поредните им студийни записи. В потвърждение идват и спомените на Хърби Хенкок:

Майлс винаги е променял композициите на другите, но не и пиесите на Уейн. Те винаги са били идеални за групата.

В продължение на десетилетия Уейн Шортър ще създаде забележителни произведения както за формациите, в които участва, така и за своите проекти като лидер. Богатият му каталог ще го превърне в един от най-великите композитори на нашето време. В дългия списък са заглавия като Capricorn, Elegant People, Endangered Species, Footprints, Infant Eyes, Juju, Mysterious Traveller, Night Dreamer, Nefertiti, Speak No Evil, The Three Marias, Yes or No – те са станали неразделна част от езика на джаза и са вплетени в тъканта на този музикален жанр. Всички те са съхранили любопитството и изумлението на своя автор от света, разказвайки за голямото приключение, което представлява животът.

Няколко момента от първата част на документалния филм „Нулева гравитация“ дават представа за трудностите и трагедиите, които съпътстват личния живот на Уейн Шортър. Но нищо не може да подготви зрителя за това, през което ще ни преведе следващият „портал“ от филма с мотото

„Вярата означава да не се страхуваш от нищо.“

През 60-те години за кратко Уейн Шортър има един неуспешен брак с мистериозната Терука Накагами. От тази връзка е първата дъщеря на саксофониста – Мияко Шортър, на която той посвещава пиеси като Miyako и Infant Eyes.

Няколко години по-късно Шортър се влюбва от пръв поглед, запознавайки се след концерт с втората си съпруга Ана Мария. Тя е обаятелна, умна, владее няколко езика и общуването с нея му носи усещането за нещо специално. Ана Мария бързо се превръща в муза на Шортър – той я рисува и не пропуска да изрази любовта си чрез музика. Връзката им е силна и сякаш непоклатима. Вдъхновяват се един от друг и заедно се радват на живота.

Преди да се роди втората му дъщеря Иска, Уейн Шортър работи по пророческия албум Odyssey of Iska („Одисеята на Иска“), предусещайки, че момичето ще преживее житейска одисея.

Заради увреждане на мозъчни артерии още от бебе Иска започва да получава припадъци, които я съпътстват през целия ѝ кратък живот. Момичето умира на 14-годишна възраст, оставяйки родителите си съкрушени. Годините, прекарани с нея, се превръщат в огромно изпитание за семейството. Докато Шортър пътува и изнася концерти по целия свят без възможност да бъде до най-близките си, Ана Мария и Иска са редовни посетители на спешното отделение в болницата, а след трагедията съпругата на Шортър се чувства сломена и изгубена. В търсене на светлина тя се обръща към будизма и постепенно увлича и своя съпруг към духовното учение.

„Иска се смелост, за да бъдеш щастлив“,

твърди Уейн Шортър в разговор с тромпетиста Терънс Бланчард. Десет години след смъртта на Иска внезапно умира и Ана Мария. Загива в самолетна катастрофа заедно с племенницата си. Двете са на път за Италия, където саксофонистът е на турне по същото време.

Въпреки преживяното Шортър не се затваря в болката си, не допуска загубата да поеме контрола върху живота му. Негови приятели, които се връщат към този период от живота му, разкриват, че в моментите на изпитания той е бил загрижен не толкова за себе си, колкото за това как се чувстват хората около него. Под влиянието на будизма той все повече осъзнава, че животът и смъртта са двете страни на една и съща монета, учейки се как да превръща предизвикателствата във възможности и отровата в лекарство.

Докато преживяваме всяка минута от историята на Уейн Шортър, разказана в „Нулева гравитация“, пред погледа ни през цялото време попадат очите на музиканта, които не се забравят дълго след прожекцията. Топли и дълбоки очи, приютили мъдреца и детето. Очи, проникващи отвъд разбирането ни за сила и уязвимост. Очи като океан, в който се размива всяка болка, отваряйки път за нови възможности и перспективи.

„За мен джазът означава „Предизвиквам те!“

След личното откровение, което представлява втората част на филма, третият „портал“ е свързан с последните години от живота на музиканта. Тук особено силно е застъпена работата на квартета на Шортър. За тази група Хърби Хенкок твърди, че дава нова дефиниция на музиката. Квартетът е с участието на пианиста Данило Перес, басиста Джон Патитучи и барабаниста Брайън Блейд. С тях тримата Шортър си позволява немислимото за много музиканти – пълната свобода да навлезе смело в непознати територии, преодолявайки лични страхове и граници.

По време на отделните концерти Шортър и неговите колеги изграждат композиции от нищото в реално време. За този безценен опит Данило Перес разказва случка, оставяща усещането за съвременна притча. Един ден пианистът се обръща към Шортър с думите:

Маестро, кога ще репетираме?

Последвал отговорът:

Не може да репетираш неизвестното, Данило!

„В будизма казваме hom nim yoh, което означава, че в този момент започва първият ден от живота ти, така че вложи сто процента от себе си в него, защото той е единственото време, в което можеш да промениш миналото и бъдещето“,

казва Шортър в интервю за Националното обществено радио на САЩ.

В музикално отношение „Нулева гравитация“ обръща внимание и на други важни моменти от биографията на Уейн Шортър. Сред тях е участието му в Weather Report – култова група със собствен звук, аура и митология. Музиката им е добре приета от широката публика, свирят на пълни стадиони, а концертите им са с въздействието на трансцендентално преживяване.

Още един вдъхновяващ момент от пътя на Шортър е дългогодишното му приятелство с друг велик джаз музикант – Хърби Хенкок. Двамата са заедно от десетилетия – на сцената и в живота, във всеки момент разтворили обятия за неизвестното. През 2016 г. те написаха отворено писмо, адресирано до младите артисти. Посланията им, изпълнени с мъдрост и човечност, са универсални и прескачат професионалните граници, достигайки с лекота до всеки от нас.

През последните години от живота си Уейн Шортър успява да сбъдне една голяма своя мечта, която датира още от тийнейджърските му години – да напише опера. Шортър е автор на музиката към „Ифигения“, а Есперанца Спалдинг – на либретото. Работата по произведението, чиято история е вдъхновена от гръцката митология, отнема осем години. Сценичният дизайн е поверен на архитекта Франк Гери, дългогодишен приятел на Шортър.

Когато разказваш за необикновен герой, няма как да подходиш конвенционално. Затова в създаването на „Нулева гравитация“ Дорсей Алави използва необичаен подход. Тя разкрива, че не е разполагала с много визуален материал, тъй като Шортър не се е снимал толкова често, но понеже е голям почитател на киното, Алави се е опитала да сътвори специфична кинематографска атмосфера, чрез която да пресъздаде различните епизоди от живота на саксофониста. Във филма са използвани архивни кадри, семейни видеа, анимация, игрални сцени и всеки от трите „портала“ е със собствен визуален облик. За своя увлекателен разказ Алави успява да привлече роднини и приятели на Шортър, критици и музиканти, сред които имена като Хърби Хенкок, Джони Мичъл, Съни Ролинс, Маркъс Милър, Терънс Бланчард, Карлос Сантана, Тери Лин Карингтън, Есперанца Спалдинг и мн.др.

В едно интервю режисьорката споделя наблюдението си, че всички тези хора, с които през годините е разговаряла за музиканта, най-често са споменавали думата „любов“ в комбинация с името му. И наистина, след „Нулева гравитация“ ми се струва невъзможно да не обичаш Уейн Шортър, защото той и музиката му са въплъщение на чистата любов. Тя ще разцъфтява и в следващите десетилетия, тъй като за разлика от други свои колеги, потънали по една или друга причина в забрава, саксофонистът продължава да бъде уважаван и обичан от хора от всякакви възрасти. Достатъчно е да видите в „Нулева гравитация“ опашките от млади фенове за автографи след концерт, както и сълзите в очите им. Достатъчно е да видите блясъка в погледите на студентите по музика, за които Шортър е истински супергерой.

„Твоят инструмент е твоята човечност“

Историята познава и други велики композитори, но хуманизмът на саксофониста ще продължава да дава различни перспективи към сътвореното от него. Силната му любов към хората си остава един от основните ключове в опитите за осмисляне, интерпретация и разгадаване на творчеството му. Неговата музика е пресечна точка на видимия и невидимия свят, на минало и бъдеще, на реалност и въображение, на лично и общочовешко. Това е музика, създадена тук, на Земята, с широк поглед към неизвестното и тайните на Вселената.

Физическото отсъствие на Уейн Шортър след смъртта му не се отрази на присъствието му в живота на меломаните. Документалният филм „Нулева гравитация“ е в Amazon Prime. През последните седмици симфоничната музика на Шортър прозвуча на някои от най-престижните сцени в Европа. A в оповестените неотдавна резултати от годишната анкета на Downbeat читателите на популярното списание избраха Уейн Шортър за артист на годината.

1st and 2nd Gen Intel Xeon to 5th Gen Intel Xeon Consolidation

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/1st-and-2nd-gen-intel-xeon-to-5th-gen-intel-xeon-consolidation-supermicro/

We do a deep dive into a refresh cycle performance, power and features comparison with 2nd Gen and 5th Gen Intel Xeon servers from Supermicro

The post 1st and 2nd Gen Intel Xeon to 5th Gen Intel Xeon Consolidation appeared first on ServeTheHome.

Definitely

Post Syndicated from xkcd.com original https://xkcd.com/2871/

Your MySQL 5.7 and PostgreSQL 11 databases will be automatically enrolled into Amazon RDS Extended Support

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/your-mysql-5-7-and-postgresql-11-databases-will-be-automatically-enrolled-into-amazon-rds-extended-support/

Today, we are announcing that your MySQL 5.7 and PostgreSQL 11 database instances running on Amazon Aurora and Amazon Relational Database Service (Amazon RDS) will be automatically enrolled into Amazon RDS Extended Support starting on February 29, 2024.

This will help avoid unplanned downtime and compatibility issues that can arise with automatically upgrading to a new major version. This provides you with more control over when you want to upgrade the major version of your database.

This automatic enrollment may mean that you will experience higher charges when RDS Extended Support begins. You can avoid these charges by upgrading your database to a newer DB version before the start of RDS Extended Support.

What is Amazon RDS Extended Support?
In September 2023, we announced Amazon RDS Extended Support, which allows you to continue running your database on a major engine version past its RDS end of standard support date on Amazon Aurora or Amazon RDS at an additional cost.

Until community end of life (EoL), the MySQL and PostgreSQL open source communities manage common vulnerabilities and exposures (CVE) identification, patch generation, and bug fixes for the respective engines. The communities release a new minor version every quarter containing these security patches and bug fixes until the database major version reaches community end of life. After the community end of life date, CVE patches or bug fixes are no longer available and the community considers those engines unsupported. For example, MySQL 5.7 and PostgreSQL 11 are no longer supported by the communities as of October and November 2023 respectively. We are grateful to the communities for their continued support of these major versions and a transparent process and timeline for transitioning to the newest major version.

With RDS Extended Support, Amazon Aurora and RDS takes on engineering the critical CVE patches and bug fixes for up to three years beyond a major version’s community EoL. For those 3 years, Amazon Aurora and RDS will work to identify CVEs and bugs in the engine, generate patches and release them to you as quickly as possible. Under RDS Extended Support, we will continue to offer support, such that the open source community’s end of support for an engine’s major version does not leave your applications exposed to critical security vulnerabilities or unresolved bugs.

You might wonder why we are charging for RDS Extended Support rather than providing it as part of the RDS service. It’s because the engineering work for maintaining security and functionality of community EoL engines requires AWS to invest developer resources for critical CVE patches and bug fixes. This is why RDS Extended Support is only charging customers who need the additional flexibility to stay on a version past community EoL.

RDS Extended Support may be useful to help you meet your business requirements for your applications if you have particular dependencies on a specific MySQL or PostgreSQL major version, such as compatibility with certain plugins or custom features. If you are currently running on-premises database servers or self-managed Amazon Elastic Compute Cloud (Amazon EC2) instances, you can migrate to Amazon Aurora MySQL-Compatible Edition, Amazon Aurora PostgreSQL-Compatible Edition, Amazon RDS for MySQL, Amazon RDS for PostgreSQL beyond the community EoL date, and continue to use these versions these versions with RDS Extended Support while benefiting from a managed service. If you need to migrate many databases, you can also utilize RDS Extended Support to split your migration into phases, ensuring a smooth transition without overwhelming IT resources.

In 2024, RDS Extended Support will be available for RDS for MySQL major versions 5.7 and higher, RDS for PostgreSQL major versions 11 and higher, Aurora MySQL-compatible version 2 and higher, and Aurora PostgreSQL-compatible version 11 and higher. For a list of all future supported versions, see Supported MySQL major versions on Amazon RDS and Amazon Aurora major versions in the AWS documentation.

Community major version RDS/Aurora version Community end of life date End of RDS standard support date Start of RDS Extended Support pricing End of RDS Extended Support
MySQL 5.7 RDS for MySQL 5.7 October 2023 February 29, 2024 March 1, 2024 February 28, 2027
Aurora MySQL 2 October 31, 2024 December 1, 2024
PostgreSQL 11 RDS for PostgreSQL 11 November 2023 March 31, 2024 April 1, 2024 March 31, 2027
Aurora PostgreSQL 11 February 29, 2024

RDS Extended Support is priced per vCPU per hour. Learn more about pricing details and timelines for RDS Extended Support at Amazon Aurora pricing, RDS for MySQL pricing, and RDS for PostgreSQL pricing. For more information, see the blog posts about Amazon RDS Extended Support for MySQL and PostgreSQL databases in the AWS Database Blog.

Why are we automatically enrolling all databases to Amazon RDS Extended Support?
We had originally informed you that RDS Extended Support would provide the opt-in APIs and console features in December 2023. In that announcement, we said that if you decided not to opt your database in to RDS Extended Support, it would automatically upgrade to a newer engine version starting on March 1, 2024. For example, you would be upgraded from Aurora MySQL 2 or RDS for MySQL 5.7 to Aurora MySQL 3 or RDS for MySQL 8.0 and from Aurora PostgreSQL 11 or RDS for PostgreSQL 11 to Aurora PostgreSQL 15 and RDS for PostgreSQL 15, respectively.

However, we heard lots of feedback from customers that these automatic upgrades may cause their applications to experience breaking changes and other unpredictable behavior between major versions of community DB engines. For example, an unplanned major version upgrade could introduce compatibility issues or downtime if applications are not ready for MySQL 8.0 or PostgreSQL 15.

Automatic enrollment in RDS Extended Support gives you additional time and more control to organize, plan, and test your database upgrades on your own timeline, providing you flexibility on when to transition to new major versions while continuing to receive critical security and bug fixes from AWS.

If you’re worried about increased costs due to automatic enrollment in RDS Extended Support, you can avoid RDS Extended Support and associated charges by upgrading before the end of RDS standard support.

How to upgrade your database to avoid RDS Extended Support charges
Although RDS Extended Support helps you schedule your upgrade on your own timeline, sticking with older versions indefinitely means missing out on the best price-performance for your database workload and incurring additional costs from RDS Extended Support.

MySQL 8.0 on Aurora MySQL, also known as Aurora MySQL 3, unlocks support for popular Aurora features, such as Global Database, Amazon RDS Proxy, Performance Insights, Parallel Query, and Serverless v2 deployments. Upgrading to RDS for MySQL 8.0 provides features including up to three times higher performance versus MySQL 5.7, such as Multi-AZ cluster deployments, Optimized Reads, Optimized Writes, and support for AWS Graviton2 and Graviton3-based instances.

PostgreSQL 15 on Aurora PostgreSQL supports the Aurora I/O Optimized configuration, Aurora Serverless v2, Babelfish for Aurora PostgreSQL, pgvector extension, Trusted Language Extensions for PostgreSQL (TLE), and AWS Graviton3-based instances as well as community enhancements. Upgrading to RDS for PostgreSQL 15 provides features such as Multi-AZ DB cluster deployments, RDS Optimized Reads, HypoPG extension, pgvector extension, TLEs for PostgreSQL, and AWS Graviton3-based instances.

Major version upgrades may make database changes that are not backward-compatible with existing applications. You should manually modify your database instance to upgrade to the major version. It is strongly recommended that you thoroughly test any major version upgrade on non-production instances before applying it to production to ensure compatibility with your applications. For more information about an in-place upgrade from MySQL 5.7 to 8.0, see the incompatibilities between the two versions, Aurora MySQL in-place major version upgrade, and RDS for MySQL upgrades in the AWS documentation. For the in-place upgrade from PostgreSQL 11 to 15, you can use the pg_upgrade method.

To minimize downtime during upgrades, we recommend using Fully Managed Blue/Green Deployments in Amazon Aurora and Amazon RDS. With just a few steps, you can use Amazon RDS Blue/Green Deployments to create a separate, synchronized, fully managed staging environment that mirrors the production environment. This involves launching a parallel green environment with upper version replicas of your production databases lower version. After validating the green environment, you can shift traffic over to it. Then, the blue environment can be decommissioned. To learn more, see Blue/Green Deployments for Aurora MySQL and Aurora PostgreSQL or Blue/Green Deployments for RDS for MySQL and RDS for PostgreSQL in the AWS documentation. In most cases, Blue/Green Deployments are the best option to reduce downtime, except for limited cases in Amazon Aurora or Amazon RDS.

For more information on performing a major version upgrade in each DB engine, see the following guides in the AWS documentation.

Now available
Amazon RDS Extended Support is now available for all customers running Amazon Aurora and Amazon RDS instances using MySQL 5.7, PostgreSQL 11, and higher major versions in AWS Regions, including the AWS GovCloud (US) Regions beyond the end of the standard support date in 2024. You don’t need to opt in to RDS Extended Support, and you get the flexibility to upgrade your databases and continued support for up to 3 years.

Learn more about RDS Extended Support in the Amazon Aurora User Guide and the Amazon RDS User Guide. For pricing details and timelines for RDS Extended Support, see Amazon Aurora pricing, RDS for MySQL pricing, and RDS for PostgreSQL pricing.

Please send feedback to AWS re:Post for Amazon RDS and Amazon Aurora or through your usual AWS Support contacts.

— Channy

The 6.7 kernel will be released on January 7

Post Syndicated from corbet original https://lwn.net/Articles/955970/

Unsurprisingly, Linus Torvalds has let
it be known
that he will do a 6.7-rc8 release (rather than 6.7 final)
on December 31, thus avoiding opening the 6.8 merge window on New
Year’s Day.

Just FYI – my current plan is that -rc7 will happen this Saturday
(because I still follow the Finnish customs of Christmas _Eve_
being the important day, so Sunday I’ll be off), and then if
anything comes in that week – which it will do, even if networking
might be offline – I’ll do an rc8 the week after.

Then, unless anything odd happens, the final 6.7 release will be Jan
7th, and so the merge window for 6.8 will open Jan 8th.

Case of success: phygital environment monitoring with Zabbix

Post Syndicated from Aurea Araujo original https://blog.zabbix.com/case-of-success-phygital-environment-monitoring-with-zabbix/27108/

When retail needs involve monitoring diverse and complex environments, with digital and physical operations, the tool chosen to meet those needs must be versatile, scalable and capable of collecting and analyzing data to generate insights for managers and support decision-making.

With this in mind, Unirede – a Zabbix Premium Partner – developed a use case consisting of monitoring a client in the retail segment, using Zabbix as the main tool for data collection, consolidation and event management.

The result: a reduction of up to 70% in operational costs and other benefits related to data-based decision-making, following the data driven concept and automation at the technological environment level for rapid responses to incidents.

Continue reading to understand, in detail, how monitoring can support retail needs, based on this case of success.

Retail needs

Currently, stores and brands that offer an omnichannel experience are standing out in the market. This means that they are available 24 hours a day, 7 days a week, not only in physical spaces (such as the stores themselves), but also digitally (through e-commerce and mobile app operations). These retailers also have critical operations in distribution centers that operate without interruption.

As a result, the environment to be monitored becomes what we call phygital – both physical and digital, at the same time. This is a concept the origins of which are closely linked to the Internet and global digitalization.

With this, customers can choose to buy from home, on their cell phone, wherever they are. However, if necessary, they can find the support they need in physical stores, with the same rules and prices across all channels.

Therefore, retailers need to ensure that the operation is able to deliver, full-time, a consistent customer experience on any channel, mitigating or preventing unavailability and loss of service performance. Additionally, they need to provide support to requests for help that may arise from managers who are responsible for the company’s results.

And this is not limited to just one type of retail. Segments such as supermarkets, fast fashion, specialists, fast food and pharmaceutical, among others, can benefit from data monitoring to improve the work carried out in activities such as:

  • Understanding the purchasing journey of omnibuyer customers (on-line/off-line):
  • Complete monitoring of user experience;
  • Maximizing the operation of distribution centers;
  • Monitoring points of sale (POS);
  • Developing technical and executive dashboardswith the main KPIs of the business;
  • Reports with information for making decisions in real time.

So, through monitoring with Zabbix, it is possible to collect data from different points, organize these data as information in visual dashboards and generate knowledge to improve internal and external customer service from end to end.

How monitoring with Zabbix works

We talk about the benefits and needs regarding retailers, but we also need to explain how monitoring with Zabbix works in this type of environment.

Beginning with the basics: Zabbix collects, categorizes, analyzes and generates information from data.

This process divided into 4 stages:

  • Data collection;
  • Problem detection;
  • Visualization;
  • Reaction.
Zabbix activity cycle

In the first stage, Zabbix captures data from different sources, which can be cloud systems, containers, networks, servers, databases, APIs, stocks, payment methods, applications, services and the Internet of Things. At this stage, there is a lot of flexibility in the tool itself, and it is also possible to create completely customized collection methods.

The data are encrypted, as Zabbix follows the premise of Security by Design, and they are analyzed in a processing stage to detect possible problems or behaviors that the business wants to be detected.

At this stage, data processing categorizes information into events by severity, indicates the root cause of the potential problem or anomaly, correlates these events based on predefinitions established by system administrators or business managers, begins self-remediation of this problem, and creates predictions based on metrics behaviors so that the business is ready and prepared to deal with events that have not yet occurred.

Afterwards, the information generated is allocated to dashboards for better visualization and, consequently, administrators choose how to react to what is shown.

Reactions can take the form of alerts via message, e-mail and applications, by generating tickets to the support team, by establishing a connection to other applications and systems, and by automating problem solving – or self-healing.

Main on-line and off-line retail indicators

By monitoring systems and the main resources of the retail environment, in addition to ensuring better availability and performance, it is possible to extract critical indicators for your business in real time.

There are indicators that are found both in physical and digital retail operations. With Zabbix, it is possible to collect and measure each one of these indicators, such as:

  • Gross sales;
  • Average ticket;
  • Sales by product category;
  • Sales by payment method;
  • Number of sales;
  • Accumulated sales in a given period;
  • Inventory value;
  • Sales by M2;
  • Sales by collaborator;
  • Year-over-Year Sales (YoY);
  • Goals achieved;
  • Conversion rate (from visitor to customer);
  • Traffic origin channels;
  • Time spent in e-commerce;
  • New visitors vs. returning visitors;
  • Cart abandonment.

By analyzing the elements mentioned above, also through monitoring, it is possible to understand how the performance of on-line sales is compared to off-line sales, helping business owners to make a decision on which of the means – or all of them – should receive more or less investments to generate more revenue.

We mentioned automating manual processes not long ago.

In retail, this can happen with the discovery of events and the indication of root causes, such as identifying the unavailability of a service or component that impacts the proper operation of a given system and, based on rules defined in Zabbix, triggering a self-recovery command, without human intervention, as in the following example:

Example of self-healing with Zabbix, used by Unirede.

What are the benefits of monitoring for retailers?

How can monitoring become essential for the digital transformation of retailers?

In order to do this, we need to understand the benefits of collecting and analyzing data with Zabbix.

The first and most objective one is the monitoring of support services, both in physical and digital operations. Here, we are talking about networks, connections, and IT infrastructure in general.

But there is also monitoring distribution centers in order to optimize supply chains, and capturing data from stores, points of sale, data centers and clouds.

With this duly adjusted, we move on to how the monitoring and sustainment of basic services helps retailers to have a better view of environments, analyzing performance indicators in real time and managing SLAs.

The result of a monitoring system with Zabbix in retail is having operations focused on customer experience, ensuring cost reductions and gains in operational efficiency.

Lessons learned from retailer monitoring

With so many possibilities and advantages resulting from using Zabbix in retail, it is difficult to choose where to start.

We need to bear in mind that, when implementing Zabbix in this area, it is important to focus on what is essential, that is, monitoring only what is necessary, instead of monitoring data that will not result in any type of action or analysis in case of an event. Avoid standard templates without the necessary adjustments to meet the specificities of your environment and the analysis practices your business requires..

Automating as much as possible is also a crucial practice, as it allows the team to dedicate more time to strategic activities in the area, thus spending less time dealing with incidents and adding new hosts.

And, of course, even if it is possible to have an integration with other tools, it is worth carrying out a thorough review of existing monitoring efforts in other tools to avoid generating events that are irrelevant, that is, that do not require any type of action by the team. This approach ensures that integration is smooth and does not compromise the effectiveness of the system and operations by generating excessive or unnecessary events.

Last but not least: it is important to recognize the essential and crucial role of the people who use the tool. They not only operate Zabbix, but also play an active role in the development and continuous evolution of business monitoring efforts.

By giving these users a voice and promoting training sessions, your company can invest in more meaningful collaborations, contributing to the continuous adaptation of Zabbix to the specific needs of the retail segment.

About Unirede

Unirede is a technology company, with roots in the State of Rio Grande do Sul and headquartered in Porto Alegre. It was created in 1999 and is dedicated to provide its clients with effective consulting services to improve business performance. Its activities aim to increase productivity, minimize downtime and drive the integration of technological innovations through managed services.

With a philosophy centered on simplicity, Unirede focuses on human relationships, both internally and with clients. There is a conscious effort to not only provide services, but also to establish relationships, favoring the delivery of intelligent solutions that add value to clients.

Unirede has achieved a level of excellence and commitment to results that has resulted in the establishment of strategic partnerships with technology market leaders. It stands out as the first Zabbix Premium Partner in Latin America, since 2008, and was the first Zabbix Training Partner in the world, in 2012.

➜ Find out more about the Official Zabbix Partner Program.

The post Case of success: phygital environment monitoring with Zabbix appeared first on Zabbix Blog.

What’s New in Rapid7 Products & Services: 2023 Year in Review

Post Syndicated from Margaret Wei original https://blog.rapid7.com/2023/12/21/whats-new-in-rapid7-products-services-2023-year-in-review/

What’s New in Rapid7 Products & Services: 2023 Year in Review

Throughout 2023 Rapid7 has made investments across the Insight Platform to further our mission of providing security teams with the tools to proactively anticipate imminent risk, prevent breaches earlier, and respond faster to threats. In this blog you’ll find a review of our top releases from this past year, all of which were purpose-built to bring your team a holistic, unified approach to security operations and command of your attack surface.

Proactively secure your environment

Endpoint protection with next-gen antivirus in Managed Threat Complete

To provide protection against both known and unknown threats, we released multilayered prevention with Next-Gen Antivirus in Managed Threat Complete. Available through the Insight Agent, you’re immediately able to:

  • Block known and unknown threats early in the kill chain
  • Halt malware that’s built to bypass existing security controls
  • Maximize your security stack and ROI with existing Insight Agent
  • Leverage the expertise of our MDR team to triage and investigate these alerts

New capabilities to help prioritize risk in your cloud and on-premise environments and effectively communicate risk posture

As the attack surface expands, we know it’s critical for you to have visibility into vulnerabilities across your hybrid environments and communicate it with your executive and remediation stakeholders. This year we made a series of investments in this area to help customers better visualize, prioritize, and communicate risk.

What’s New in Rapid7 Products & Services: 2023 Year in Review
  • Executive Risk View, available as a part of Cloud Risk Complete, provides security leaders with the visibility and context needed to track total risk across cloud and on-premises assets to better understand organizational risk posture and trends.
  • Active Risk, our new vulnerability risk-scoring methodology, helps security teams prioritize vulnerabilities that are actively exploited or most likely to be exploited in the wild. Our approach enriches the latest version of the Common Vulnerability Scoring System (CVSS) with multiple threat intelligence feeds, including intelligence from proprietary Rapid7 Labs research. Active Risk normalizes risk scores across cloud and on-premises environments within InsightVM, InsightCloudSec, and Executive Risk View.
  • The new risk score in InsightCloudSec’s Layered Context makes it easier for you to understand the riskiest resources within your cloud environment. Much like Layered Context, the new risk score combines a variety of risk signals – including Active Risk – and assigns a higher risk score to resources that suffer from toxic combinations or multiple risk vectors that present an increased likelihood or impact of compromise.
  • Two new dashboard cards in InsightVM to help security teams communicate risk posture cross-functionally and provide context on asset and vulnerability prioritization:
  • Vulnerability Findings by Active Risk Score Severity – ideal for executive reporting, this dashboard card indicates total number of vulnerabilities across the Active Risk severity levels and number of affected assets and instances.
  • Vulnerability Findings by Active Risk Score Severity and Publish Age – ideal for sharing with remediation stakeholders to assist with prioritizing vulnerabilities for the next patch cycle, or identifying critical vulnerabilities that may have been missed.

Coverage and expert analysis for critical vulnerabilities with Rapid7 Labs

Rapid7 Labs provides easy-to-use threat intelligence and guidance, curated by our industry-leading attack experts, to the security teams.

Emergent Threat Response (ETR) program, part of Rapid7 Labs, provides teams with accelerated visibility, alerting, and guidance on high-priority threats. Over this past year we provided coverage and expert analysis within 24 hours for over 30 emergent threats, including Progress Software’s MOVEit Transfer solution where our security research team was one of the first to detect exploitation—four days before the vendor issued public advisory. Keep up with future ETRs on our blog here.

Detect and prioritize threats anywhere, from the endpoint to the cloud

Enhanced alert details in InsightIDR Investigations

An updated evidence panel for attacker behavior analytics (ABA) alerts gives you a description of the alert and recommendations for triage, rule logic that generated the alert and associated data, and a process tree (for MDR customers) to show details about what occurred before, during, and after the alert was generated.

What’s New in Rapid7 Products & Services: 2023 Year in Review
Process tree details within alert details in InsightIDR

AI-driven detection of anomalous activity with Cloud Anomaly Detection

Cloud Anomaly Detection provides AI-driven detection of anomalous activity occurring across your cloud environments, with automated prioritization to assess the likelihood that activity is malicious. With Cloud Anomaly Detection, your team will benefit from:

  • A consolidated view that aggregates threat detections from CSP-native detection engines and Rapid7’s AI-driven proprietary detections.
  • Automated prioritization to focus on the activity that is most likely to be malicious.
  • The ability to detect and respond to cloud threats using the same processes and tools your SOC teams are using today with easy API-based ingestion into XDR/SIEM tools for threat investigations and prioritizing remediation efforts.

Detailed views into risks across your cloud environment with Identity Analysis and Attack Path Analysis

We’re constantly working to improve the ways with which we provide a real-time and comprehensive view of your current cloud risk posture. This year, we made some major strides in this area, headlined by two exciting new features:

  • Identity Analysis provides a unified view into identity-related risk across your cloud environments, allowing you to achieve least privileged access (LPA) at scale. By utilizing machine learning (ML), Identity Analysis builds a baseline of access patterns and permissions usage, and then correlates the baseline against assigned permissions and privileges. This enables your team to identify overly-permissive roles or unused access so you can automatically right-size permissions in accordance with LPA.
  • Attack Path Analysis enables you to analyze relationships between resources and quickly identify potential avenues bad actors could navigate within your cloud environment to exploit a vulnerable resource and/or access sensitive information. This visualization helps teams communicate risk across the organization, particularly for non-technical stakeholders that may find it difficult to understand why a compromised resource presents a potentially larger risk to the business.
What’s New in Rapid7 Products & Services: 2023 Year in Review

More flexible alerting with Custom Detection Rules

Every environment, industry, and organization can have differing needs when it comes to detections. With custom detection rules in InsightIDR, you can detect threats specific to your needs while take advantage of the same capabilities that are available for out-of-the-box detection rules, including:

  • The ability to set a rule action and rule priority to choose how you are alerted when your rule detects suspicious activity.
  • The ability to add exceptions to your rule for specific key-value pairs.

A growing library of actionable detections in InsightIDR

In 2023 we added over 3,000 new detection rules. See them in-product or visit the Detection Library for descriptions and recommendations.

Agent-Based Policy supports custom policy assessment in InsightVM

Guidelines from Center for Internet Security (CIS) and Security Technical Implementation Guides (STIG) are widely used industry benchmarks for configuration assessment. However, a benchmark or guideline as-is may not meet the unique needs of every business.

Agent-Based Policy assessment now supports Custom Policies. Global Administrators can customize built-in policies, upload policies, or enable a copy of existing custom policies for agent-based assessments. Learn more here.

Investigate and respond with confidence

Faster containment and remediation of threats with expansion of Active Response for Managed Detection and Response customers

Attackers work quickly and every second you wait to take action can have detrimental impacts on your environment. Enter automation—Active Response enables Rapid7 SOC analysts to immediately quarantine assets and users in a customer’s environment with response actions powered by InsightConnect, Rapid7’s SOAR solution.

Active Response has you covered to quarantine via our Insight Agent, as well as a variety of third-party providers—including Crowdstrike and SentinelOne. And with MDR analyst actions logged directly in InsightIDR, you have more expansive, collaborative detection and response faster than ever before. Read what Active Response can do for your organization—and how it stopped malware in a recent MDR Investigation—here.

What’s New in Rapid7 Products & Services: 2023 Year in Review
Active Response in action: Rapid7 MDR analyst activity logged within InsightIDR Investigations timeline

Velociraptor integrates with InsightIDR for broader DFIR coverage

The attack surface is continually expanding, and so should your visibility into potential threats across it. This year we integrated Velociraptor, Rapid7’s open-source DFIR framework, with our Insight Platform to bring the data you need for daily threat monitoring and hunting into InsightIDR for investigation via our Insight Agent.

This integration brings you faster identification and remediation, always-on monitoring for threat activity across your endpoint fleet, and expanded threat detection capabilities. Read more about what this integration unlocks here.

What’s New in Rapid7 Products & Services: 2023 Year in Review
Velociraptor alert details in InsightIDR

Stay tuned!

As always, we’re continuing to work on exciting product enhancements and releases throughout the year. Keep an eye on our blog and release notes as we continue to highlight the latest in product and service investments at Rapid7. See you in 2024!

The Backblaze Team Recommends: Tech That Saves the Day

Post Syndicated from Molly Clancy original https://www.backblaze.com/blog/the-backblaze-team-recommends-tech-that-saves-the-day/

A decorative image showing a superhero holding a computer with the Backblaze logo showing.

Everyone has their arsenal of indispensable gadgets and apps they absolutely couldn’t live without, and we had a feeling the folks here at Backblaze would have a lot to say about the subject. We tapped the smart, savvy minds that keep our storage cloud up and running, and discovered a treasure trove of insights into the tech essentials that power their daily lives.

From budgeting apps to text editors to humble charging jacks, our staff share the tools they can’t live without. So, without further ado, let’s dig into the gear that keeps our collective gears turning:

Tech for Staying Connected

Solutions Engineering Director, Troy Liljedahl

As a Mac guy, I love my Airpod Pros and the way they work seamlessly with my iPhone, Macbook Pro, and iPad. But things get a little wonky when you try to use them outside of the Apple ecosystem. I tried many different wireless earbuds and settled on the Anker Soundcore Space A40 Earbuds. I’m a big fan of The Wirecutter by the New York Times (and they’re big fans of us) and they had these at the top of their list. I love the sound quality, noise canceling, and excellent battery life. My Airpod Pros are still my go to when I’m out of the house, but when I need a good headset at home for my PC and other non-Apple devices, these have become my go to earbuds.

Senior Director, Marketing, Yev Pusin

You know what I hate? Getting my phone out at inopportune times. You know what I don’t mind so much? Glancing down at my Google Pixel Watch to see that the cold call I’m receiving is being answered by my phone’s call screening and I don’t have to pick it up. Whether it’s the first version or the second, I have grown accustomed to having something on my wrist that acts as an extension of my phone. True digital bliss. 

Chief Technical Evangelist, Pat Patterson

I use my webcam a LOT: Zoom and Google Meet with coworkers, FaceTime with family and friends, webinars with the Backblaze community of developers and admins, and quick-start videos for the Backblaze YouTube channel. Ever since I got my PlexiCam Pro mount about a year ago, it’s been my secret weapon in every one of those interactions. It’s a transparent plexiglass webcam mount that hangs from the top edge of my monitor, allowing me to position my webcam in my eyeline, just above my focus. To anyone on the call, I appear to be looking directly into the camera. 

At $85, it’s not cheap, but it’s well designed and constructed, and feels like it will last forever. Highly recommended for anyone who spends a lot of time flicking their eyes between the screen and the webcam!

Tech for Devs

Senior Site Reliability Engineer, nathaniel wagner

Ah, the age old debate of Emacs versus Vim: the two most widely used editors for Linux operating systems. I solidly planted my flag on team Vim once I learned how to save and exit the program. 😉

I do aspire to one day having a computer that only runs Emacs because byte compiled Emacs is cool to me. Until I graduate to that level of wizardry, I stick to Neovim when I need to quickly edit something from a terminal or want to appear cool in front of my coworkers. I mostly use the Vim extension in Intellij for day-to-day modifying of code and configs. If you would like to also learn Vim, I really enjoyed playing through Vim Adventures, which is a free game that teaches you a lot of the shortcuts and movements in Vim.

Lead Software Engineer, Application Security, Ola Nordstrom

Continuing the discussion in favor of Vim, specifically Neovim. You only have to learn the keybindings once. There is a fantastic set of plugins to customize it to your heart’s content—Visual Studio Code has VSCodeVim, Intellij has IdeaVim, for example. Then you don’t have to relearn keybindings while switching between languages, projects, and code. 

Tmux the terminal multiplexer: like Vim, it may have a steep learning curve but once you learn it you can’t live without it. The tmux wiki has some great getting started guides. I strongly recommend remapping the leader key (mine is Caps + A, or Caps Lock + A on Windows keyboards). You can set up customized tmux scripts to re-create all your environments (one session for server code, another for a different codebase, another for your notes, and so on). Each session then has multiple windows which you can create, split, and close quickly, no need to leave your keyboard.

For web technologies, learn the toolset available in the browser developer tools. Remember to preserve logs filter to specific responses so you won’t be overwhelmed looking at messages.

Stepping away from the browser and back into the terminal, learn Curl and ag or rg. For every “old” unix command there’s likely a modern replacement that’s 100s of times faster with much more customization available. 

But there’s always the middle ground for situations where you may need to initiate a complex series of browser–webapp interactions and you need to modify or test something quickly. To do this, learn how to use Burp. In the long run it’s well worth it. It makes it a breeze to modify data between the browser and your app.

Last tip: for native code, just learn how to use the debugger.

Tusen Tack!

Tech That Makes Home Homier

Principle Site Reliability Engineer, Elliott Sims

One power adapter to rule them all: the Anker 715 Charger (Nano II 65W). This one little power adapter can power my personal laptop, work laptop, headphones, and more. With one cable, one small charger cube, and a few small USB end adapters, I can charge everything.

Senior Product Marketing Manager, James Flores 

A Roku. I use it at home for streaming apps, but I also carry one in my travel bag. There’s nothing worse than flipping through basic cable channels in a hotel when you’re traveling. Wait, yes there is—signing in to Netflix on the hotel TV and forgetting to sign out. If I travel with it, I just plug it in and I’m already signed in to all my apps. 

Senior Technical Editing Manager, Alison McClelland

I got tired of locking myself out of my own house and forgetting who I gave spare keys to, so I really appreciate this Yale Lock with Nest Connect. It works with the Nest cameras that I mostly use to see whether the UPS or DoorDash delivery person has the nerve to ring my doorbell. (Drop it and run, people!) 

It’s secure and easy-to-use; no more locking myself out of my own house in the middle of winter. I can give a code to a friend so they can feed my cats while I’m away, or create temporary passcodes so I don’t have to wait around for the cable guy.

Senior Content Editor, Molly Clancy

I have a hard time turning my brain off at night, so I used to pop in earbuds to listen to something soothing (not comfortable at all!). Then I got this Cozyband as a gift and became 100% addicted to it. I CAN fall asleep without it, but I don’t do it willingly. It’s also good for working out if you hate sweaty earbuds slipping out all the time. 

Chief Executive Officer, Gleb Budman

It gets cold in my home, and I don’t necessarily want to heat the whole place when it’s just me. A good old fashioned heated blanket does the trick. All the tech in the world won’t help you when you’re shivering. 

The Apps Have Entered the Chat

Partner Marketing Director, Jen Newman

As a working mom with two boys, I am always on the go. Both of my boys are now playing for AAA travel teams. I try my best to keep up with all their games, and LiveBarn is how I stay connected to them. It allows me to pull up a live feed or on-demand video of the game. Last weekend was a great example: they were playing at two different rinks across town. I was able to pull up one game on LiveBarn on my phone, and watch both games literally at the same time. When my older son came home and asked me if I saw his goal, I was able to say, “I heard Coach yell, ‘Nice shot, Newy!’” 

Associate Editor and Writer, Stephanie Doyle

My life changed when I adopted a password manager years ago. Before I went back to full-time corporate life in 2020, I freelanced quite a bit—which means an endless series of logins and passwords, depending on how you’re engaging with your clients. And, while I enjoy making up 13–15 character passphrases with a mix of upper and lowercase letters, at least one symbol with some outlawed symbols (but different ones on each site), and then remembering which ones I’ve used for which accounts without reusing them… Oh wait, I actually really don’t like that. I’d rather have a password manager like Bitwarden that can generate passwords, follows me device to device, and allows me to enable biometric controls. And, spoiler alert for any of my family members who diligently read my work (I’m sure): This year the whole family is getting a subscription as a gift, and I can centrally manage it for my non-tech-inclined family members.

Senior Director of Publishing, Patrick Thomas

I’m not going into the story of how I got hooked on this app, but I will tell you that CENTR’s meal planning tool is a life saver. You can set a crazy variety of dietary needs, select your meals and portions for a week, and it spits out a perfectly organized shopping list. Then, when you want to make a meal, you just pop into the app and it tells you exactly how to look like you know what you’re doing in the kitchen. It’s sort of pricey, but the amount of money I save by not ordering in or wasting food that I bought without a plan more than makes up for it.

And, oh yeah, Chris Hemsworth is one of its founders.

Senior Product Marketing Manager, Kari Rivas

Hoping on the app train here. The one I can’t live without is definitely YNAB. I’m trying to get better at budgeting ahead (and get my husband and I on the same page—haha!) and I like their philosophical approach to a typically boring subject.

The “Tech Is a Tool” Answer

Principal Cloud Storage Storyteller, Andy Klein

I don’t have a favorite tech thing. I certainly use lots of them, but I would not be lost without them because I was raised without any of the current tech. I know how to tie my shoes without watching a Youtube video. Just sayin’.

But, you know, just in case.

Thanks, Andy

Leave it to Andy to send us off with a reminder to put the tech down sometimes, as we hope you all get a chance to do this holiday season. But, we also want to know: what’s the tech that you can’t live without? Let us know in the comments.

The post The Backblaze Team Recommends: Tech That Saves the Day appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

От „нова визия за децата“ към демонополизиране на образованието. Разговор с Нора Гавазова

Post Syndicated from Надежда Цекулова original https://www.toest.bg/ot-nova-viziya-za-decata-kum-demonopolizirane-na-obrazovanieto-razgovor-s-nora-gavazova/

От „нова визия за децата“ към демонополизиране на образованието. Разговор с Нора Гавазова

Нора Гавазова е детски психолог, специализира в работата с деца на възраст между 2 и 7 години, с техните семейства и учители. В различни периоди от живота си е работила в образованието и сферата на социалните дейности в Германия, Турция и България. Председател е на Фондация „Калейдоскоп“, която свързва България с Reggio Children – Loris Malaguzzi Center, Италия – една от значимите обучителни организации за качествено и включващо ранно образование.

В началото на декември Фондация „Калейдоскоп“ представи на кръгла маса в Народното събрание своята „Визия за децата“ – едно от малкото ценностно ориентирани изследвания на концепцията за ранното образование у нас.

От „нова визия за децата“ към демонополизиране на образованието. Разговор с Нора Гавазова
Нора Гавазова. Снимка: Личен архив

Докладът Ви „Визия за децата“ представя резултатите от национално изследване на демократичните нагласи по отношение на детството. По време на заключителното събитие обаче остана усещането, че дори самите понятия, с които боравите, звучат революционно в контекста на българската образователна система. Нека започнем с опит да ги обясним. Какво означава „демократична нагласа към детството и образованието“?

Образованието не е случайна система, то буквално оформя тъканта на общество. Ако знанието се налага чрез пасивност, заучаване и послушание, с натрупването на тези ценности в поколенията образованието доказано може да започне да накланя обществото към тоталитарен режим. От една страна, хората свикват да приемат знанието като нещо независещо от тях, което се налага от външни авторитети; от друга – свикват и да бъдат пасивни граждани, да се подчиняват безкритично на авторитети. При определени съчетания на социални, културни и политически обстоятелства знаем, че тази комбинация може да има фатални последствия за човечеството.

Това е причината днес да имаме поредица от международни споразумения, които демократичните държави следват при структурирането на националните образователни системи. Те започват да се публикуват точно около периода на двете световни войни, когато постепенно кристализира тази осъзнатост. В началото са били пожелателни, а след Втората световна война стават задължителни за държавите членки на ООН, впоследствие за ЕС.

Следвоенният период играе много важна роля за общественото и политическо отхвърляне на традиционния образователен модел в Западна Европа, САЩ и в други части от света (за съжаление, България не е сред тях) и преизграждането на националните образователни системи около ценности като избор, активност, критично мислене. Идеята е, че тази система не може да бъде оставена да се развива по традиционния модел, защото в дългосрочен план това застрашава не само местната демократична култура, но и международния мир. За съжаление, както наблягаме и в доклада, България се отнася към международното право изцяло формално: условията се изпълняват на хартия, но структурата остава непроменена, тоест тоталитарна.

Може ли да обвържете този проблем по-конкретно с образователната система? Със съжаление трябва да признаем, че почти 35 години след началото на прехода към демокрация това понятие все още остава абстрактно за много хора у нас и вероятно ще е трудно да се асоциира това, за което говорите, с конкретни дефицити на образователната система.

Нещо доста конкретно е например кой и как определя какво означава „образование“. У нас това е само държавата и за да си образован, трябва да си минал задължително по начертания от нея път.

Образованието, което наричаме „традиционно“, е свързано с индустриалния период. Неговата цел е била да създаде множество хора с унифицирани умения – като поточна линия. Това обаче поставя човека в подчинена позиция и ограничава потенциала му. С развиване на идеите за демократичното общество, според които свободата на личността е ценност, се появяват много други концепции за образованието, които се наричат реформаторски. Тези концепции са на повече от век, не са толкова съвременни, колкото си мислим. За да могат да се осъществят обаче, държавата трябва да пусне контрола върху образованието.

Затова едно от ключовите неща, които се казват в документите, споменати по-рано, е, че не може да има държавен образователен монопол. В България положението е много особено – частното образование не е забранено, но върви с редица рестрикции и съществуването му на практика не води до свобода. Друг елемент за „демонополизиране“ на образованието е възможността на родителите и обществото да участват. В момента у нас държавата е оставила на свободната инициатива в образованието възможността да създаде още една услуга от същия тип, за която обаче да се плаща, вместо да остави възможност за алтернативна образователна услуга.

Ако правилно Ви разбирам, според Вас частната инициатива в образованието трябва да предлага алтернативни концепции и подходи, а не само алтернативен начин на финансиране?

Точно така. Частната инициатива е ценна, когато е източник на многообразие, и това трябва да се защити ясно в закона. Нейната роля се обезсмисля, когато е затворена в унифицирания държавен стандарт. В момента имаме един държавен монопол, който се разпростира и върху частната инициатива.

И оттук произтичат редица проблеми. Ако на ценностно ниво образованието работи срещу демократичните възгледи, се получава следното: тези семейства, които са ценностно ориентирани и могат да си го позволят, започват да се оттичат от държавния в частния сектор или даже извън системата. С това качеството в общинските институции обаче спада до минимума, който наблюдаваме в момента, и се получава феноменът на две паралелно развиващи се системи – „лошо държавно образование за бедните“ (както го наричат в Препоръките към образование, базирано на човешките права) и платено за останалите, които могат да си го позволят. Ние сме точно в тази ситуация и трябва спешно да осъзнаем, че ножицата започва да се отваря още в ранното детство и е много важно да я затворим пак там.

Държавата е длъжна, и то, подчертавам, не само на думи – тези задължения са юридически и произлизат от членството ни в ЕС, – да гарантира превенция на проблема, като направи така, че образованието да отговаря на няколко характеристики: да не е зависимо от покупателната способност на семействата; да достига реално до всички деца; да гарантира многообразие от услуги и подходи, а не да налага унифициран модел; да предвижда активна роля на самите родители и граждани в самоорганизирането и доставянето на такива услуги и да предоставя на практика правото на всяко дете да учи чрез собствената си активност, личен избор и глас още от най-ранна възраст.

Нашата държава обаче продължава да мисли в тоталитарните модели на контрол и управление на масите – тя се държи, все едно въобще не е чувала за тези неща и е достатъчно да направи институциите задължителни и безплатни, ако може и за тригодишните. Пропуска се фактът, че докато не повишим качеството и не се погрижим то да достига до всички, най-вече до рисковите групи; докато не гарантираме свобода и демократични ценности; докато продължаваме да потискаме правата на децата и техните родители, ние реално все повече вредим на собствените си деца, на обществото и на самата крайна цел, която е образованост на масите. Всичко това също е ясно описано в международните документи: ако висококачествените услуги доказано са от полза за децата, то услугите с ниско качество имат значително отрицателно влияние върху децата и обществото като цяло.

В този контекст какво и как измерихте в проекта „Визия за децата“?

Проектът продължи повече от половин година. Проведохме срещи в десет български града и поканихме представители на детски градини, училища, занимални, родители и деца. Поставихме им три ключови въпроса: 1) Какви са децата?; 2) Как учат?; 3) Каква среда им осигуряваме, за да се развиват?

Тъй като изследването е качествено, искахме да видим, ако в България нямаме свързване на ценностно ниво между демократичното общество и образованието, как може да бъде създадено то. От анализа на отговорите на тези три въпроса извлякохме изводи кои са ценностите, които хората най-често посочват. Идеята беше да формулираме заявка за какво искаме да работи образователната система. Ние изхождаме от убеждението, че в България това още не се е случило, и затова е много трудно да продължим нататък.

При представянето на доклада Ваша колежка каза, че в момента очакваме 18 години да отглеждаме деца, които да бъдат послушни, да изпълняват безропотно каквото им се казва, да не си позволяват самоинициатива, но на 19 да се превърнат в критично мислещи и активни граждани. Изследването дава ли някакво обяснение на този парадокс?

Парадоксът се дължи на незавършилия преход в умовете на хората. За да се осъществи преход от индустриална към демократична образователна система, ние първо трябва да преобърнем гледната си точка към детето като личност. Това е бавен процес, ние самите сме плод на тази система и сме естествено склонни да я възпроизвеждаме. Ако няма достатъчно силно осъзнаване, възпроизвеждането върви поколение след поколение, независимо какви са промените около нас.

Една от целите ни беше да провокираме това осъзнаване и да наблюдаваме какви конфликти ще се породят във всяка среща, защото най-вече там се ражда новото учене.

Например може да се обединим около идеята, че детето трябва да е активно, да може да играе свободно. Но се оказва, че в тази идея хората може да вложат много различни тълкувания. Едно от нещата, които видяхме, е, че за хората, работещи с деца, играта може да е едно, а за родителя – нещо коренно различно.

Още на първата среща една детска учителка стана и каза: „Ние това всичкото го правим, не разбирам защо не ви харесва. При нас децата играят, извеждаме ги на двора, какъв е проблемът?“ Един академичен преподавател беше провокиран и ѝ отговори: „Колега, това, което Вие наричате игра, изобщо не е игра.“ Имаше предвид, че когато на децата се каже, че вече може да станат от столчетата си и сега ще скачат или ще играят на зайчета или на жабки, у тях се възпитават съвсем различни разбирания от това, което би възпитала свободната игра.

Тоест ключовият въпрос е къде учителят смята, че е правото на игра – у детето, или у възрастния. В нашите детски градини то често се оказва у възрастния, с което ние отново упражняваме контрол над личността на детето и не насърчаваме и не развиваме умения за свободен избор, нито за вземане на решения и т.н. Все умения, които после ще очакваме да видим при PISA и у едни пораснали граждани.

Може ли от „Визия за децата“ да се извлекат идеи как да се излезе от този капан?

За съжаление, по-скоро не. За целта ще е нужна следваща стъпка. Няма точна рецепта „правим това, получаваме онова като резултат“, тъй като става дума за обществени процеси и влияят много фактори.

Но има няколко стъпки, които според нас може да се изпълняват едновременно, за да се насърчи едновременна промяна в системата и в обществото. Първата стъпка е да приведем нормативната регулация в съответствие с международните документи – без това не може. То означава не само да напишеш в закона, че се насърчава многообразието, както е при нас, а да създадеш среда, в която може да покълва. Ето тук ние се проваляме тотално, защото сега действащият финансов механизъм в образованието цели запазване на държавния образователен монопол. В случая с малките деца това е институционален монопол. Така е и при по-големите, но през първите седем години е особено проблемно за личността на детето. 

Второто е да се разделим със задължителните педагогически ситуации, които се налагат на детето, независимо от неговата готовност и интереси. Нашата система не дава никаква възможност за индивидуален подход, въпреки че непрекъснато говорим за него.

Подготовката на учителите е друг аспект, на който следва да обърнем специално внимание. Трябва да научим учителите да работят през отношенчески процеси, което е изцяло нов подход. Например при малките деца, защото нашият фокус е върху тях, това означава учителят да може да види в играта, която детето само̀ организира с другите деца, какви процеси на учене протичат. Това е обръщане на образователната парадигма – не възрастният да „налива“ знания в детето, а на основата на убеждението ни, че детето има в себе си заложбите, да осигурим условия и среда, в които те да могат да се проявят и развиват.

Необходимо е възрастният да наблюдава, да документира, да познава добре теориите на ученето и развитието, за да може да ги „вижда“ в детето. Това обаче са все умения, които в българския университет не присъстват. Тук се работи в дидактичен стил и е важно да се чуят самите университети, доколко те са отворени за нещо различно.

Учителите, които в момента са в системата, как гледат на идеята за подобна промяна?

Изследването ни носи усещането, че сякаш след няколко години в системата самите учители се разделят със свободата си. Тук дори още не говорим за свободата на детето, а за свободата на учителя. Учителят в България се възприема като някой, на когото му казват какво да прави. Така че ако искаме промяна в системата, тя трябва да започне от еманципиране на самия учител – да се почувства свободен, да се почувства овластен в комуникацията си с децата и с родителите, за да може отношенията да започнат да водят до учене.

На прага на 2024 г. е крайно време да се разделим с идеята, че ученето е процес, в който възрастният казва на детето какво да прави, и то го изпълнява автоматизирано. Ученето е в самото дете и колкото по-малко го потискаш, толкова повече то самò иска да развива знанията и уменията си.


Живеем във време, в което да научиш това, което поискаш, в момент, в който го искаш, е по-лесно от когато и да било преди в човешката история. Въпреки това – или може би имено заради това – формалното образование преживява криза на идентичността. В рубриката „Разговори за образованието“ Надежда Цекулова и нейните събеседници търсят философията, смисъла и формите на онова, което наричаме „образование“ в третата декада на ХХI в.

[$] Data-type profiling for perf

Post Syndicated from corbet original https://lwn.net/Articles/955709/

Tooling for profiling the effects of memory usage and layout has always
lagged behind that for profiling processor activity, so Namhyung Kim’s patch set for data-type profiling
in perf
is a welcome addition. It provides aggregated breakdowns of
memory accesses by data type that can inform structure layout and access
pattern changes. Existing tools have either, like heaptrack, focused on
profiling allocations, or, like perf mem, on accounting memory
accesses only at the address level. This new work builds on the latter,
using DWARF debugging information to correlate memory operations with their
source-level types.

How smava makes loans transparent and affordable using Amazon Redshift Serverless

Post Syndicated from Alex Naumov original https://aws.amazon.com/blogs/big-data/how-smava-makes-loans-transparent-and-affordable-using-amazon-redshift-serverless/

This is a guest post co-written by Alex Naumov, Principal Data Architect at smava.

smava GmbH is one of the leading financial services companies in Germany, making personal loans transparent, fair, and affordable for consumers. Based on digital processes, smava compares loan offers from more than 20 banks. In this way, borrowers can choose the deals that are most favorable to them in a fast, digitalized, and efficient way.

smava believes in and takes advantage of data-driven decisions in order to become the market leader. The Data Platform team is responsible for supporting data-driven decisions at smava by providing data products across all departments and branches of the company. The departments include teams from engineering to sales and marketing. Branches range by products, namely B2C loans, B2B loans, and formerly also B2C mortgages. The data products used inside the company include insights from user journeys, operational reports, and marketing campaign results, among others. The data platform serves on average 60 thousand queries per day. The data volume is in double-digit TBs with steady growth as business and data sources evolve.

smava’s Data Platform team faced the challenge to deliver data to stakeholders with different SLAs, while maintaining the flexibility to scale up and down while staying cost-efficient. It took up to 3 hours to generate daily reporting, which impacted business decision-making when re-calculations needed to happen during the day. To speed up the self-service analytics and foster innovation based on data, a solution was needed to provide ways to allow any team to create data products on their own in a decentralized manner. To create and manage the data products, smava uses Amazon Redshift, a cloud data warehouse.

In this post, we show how smava optimized their data platform by using Amazon Redshift Serverless and Amazon Redshift data sharing to overcome right-sizing challenges for unpredictable workloads and further improve price-performance. Through the optimizations, smava achieved up to 50% cost savings and up to three times faster report generation compared to the previous analytics infrastructure.

Overview of solution

As a data-driven company, smava relies on the AWS Cloud to power their analytics use cases. To bring their customers the best deals and user experience, smava follows the modern data architecture principles with a data lake as a scalable, durable data store and purpose-built data stores for analytical processing and data consumption.

smava ingests data from various external and internal data sources into a landing stage on the data lake based on Amazon Simple Storage Service (Amazon S3). To ingest the data, smava uses a set of popular third-party customer data platforms complemented by custom scripts.

After the data lands in Amazon S3, smava uses the AWS Glue Data Catalog and crawlers to automatically catalog the available data, capture the metadata, and provide an interface that allows querying all data assets.

Data analysts who require access to the raw assets on the data lake use Amazon Athena, a serverless, interactive analytics service for exploration with ad hoc queries. For the downstream consumption by all departments across the organization, smava’s Data Platform team prepares curated data products following the extract, load, and transform (ELT) pattern. smava uses Amazon Redshift as their cloud data warehouse to transform, store, and analyze data, and uses Amazon Redshift Spectrum to efficiently query and retrieve structured and semi-structured data from the data lake using SQL.

smava follows the data vault modeling methodology with the Raw Vault, Business Vault, and Data Mart stages to prepare the data products for end consumers. The Raw Vault describes objects loaded directly from the data sources and represents a copy of the landing stage in the data lake. The Business Vault is populated with data sourced from the Raw Vault and transformed according to the business rules. Finally, the data is aggregated into specific data products oriented to a specific business line. This is the Data Mart stage. The data products from the Business Vault and Data Mart stages are now available for consumers. smava decided to use Tableau for business intelligence, data visualization, and further analytics. The data transformations are managed with dbt to simplify the workflow governance and team collaboration.

The following diagram shows the high-level data platform architecture before the optimizations.

High-level Data Platform architecture before the optimizations

Evolution of the data platform requirements

smava started with a single Redshift cluster to host all three data stages. They chose provisioned cluster nodes of the RA3 type with Reserved Instances (RIs) for cost optimization. As data volumes grew 53% year over year, so did the complexity and requirements from various analytic workloads.

smava quickly addressed the growing data volumes by right-sizing the cluster and using Amazon Redshift Concurrency Scaling for peak workloads. Furthermore, smava wanted to give all teams the option to create their own data products in a self-service manner to increase the pace of innovation. To avoid any interference with the centrally managed data products, the decentralized product development environments needed to be strictly isolated. The same requirement was also applied for the isolation of different product stages curated by the Data Platform team.

Optimizing the architecture with data sharing and Redshift Serverless

To meet the evolved requirements, smava decided to separate the workload by splitting the single provisioned Redshift cluster into multiple data warehouses, with each warehouse serving a different stage. In addition, smava added new staging environments in the Business Vault to develop new data products without the risk of interfering with existing product pipelines. To avoid any interference with the centrally managed data products of the Data Platform team, smava introduced an additional Redshift cluster, isolating the decentralized workloads.

smava was looking for an out-of-the-box solution to achieve workload isolation without managing a complex data replication pipeline.

Right after the launch of Redshift data sharing capabilities in 2021, the Data Platform team recognized that this was the solution they had been looking for. smava adopted the data sharing feature to have the data from producer clusters available for read access on different consumer clusters, with each of those consumer clusters serving a different stage.

Redshift data sharing enables instant, granular, and fast data access across Redshift clusters without the need to copy data. It provides live access to data so that users always see the most up-to-date and consistent information as it’s updated in the data warehouse. With data sharing, you can securely share live data with Redshift clusters in the same or different AWS accounts and across Regions.

With Redshift data sharing, smava was able to optimize the data architecture by separating the data workloads to individual consumer clusters without having to replicate the data. The following diagram illustrates the high-level data platform architecture after splitting the single Redshift cluster into multiple clusters.

High-level Data Platform architecture after splitting the single Redshift cluster in multiple clusters

By providing a self-service data mart, smava increased data democratization by providing users with access to all aspects of the data. They also provided teams with a set of custom tools for data discovery, ad hoc analysis, prototyping, and operating the full lifecycle of mature data products.

After collecting operational data from the individual clusters, the Data Platform team identified further potential optimizations: the Raw Vault cluster was under steady load 24/7, but the Business Vault clusters were only updated nightly. To optimize for costs, smava used the pause and resume capabilities of Redshift provisioned clusters. These capabilities are useful for clusters that need to be available at specific times. While the cluster is paused, on-demand billing is suspended. Only the cluster’s storage incurs charges.

The pause and resume feature helped smava optimize for cost, but it required additional operational overhead to trigger the cluster operations. Additionally, the development clusters remained subject to idle times during working hours. These challenges were finally solved by adopting Redshift Serverless in 2022. The Data Platform team decided to move the Business Data Vault stage clusters to Redshift Serverless, which allows them to pay for the data warehouse only when in use, reliably and efficiently.

Redshift Serverless is ideal for cases when it’s difficult to predict compute needs such as variable workloads, periodic workloads with idle time, and steady-state workloads with spikes. Additionally, as usage demand evolves with new workloads and more concurrent users, Redshift Serverless automatically provisions the right compute resources, and the data warehouse scales seamlessly and automatically, without the need for manual intervention. Data sharing is supported in both directions between Redshift Serverless and provisioned Redshift clusters with RA3 nodes, so no changes to the smava architecture were needed. The following diagram shows the high-level architecture setup after the move to Redshift Serverless.

High-level Data Platform architecture after introducing Redshift Serverless for Business Vault clusters

smava combined the benefits of Redshift Serverless and dbt through a seamless CI/CD pipeline, adopting a trunk-based development methodology. Changes on the Git repository are automatically deployed to a test stage and validated using automated integration tests. This approach increased the efficiency of developers and decreased the average time to production from days to minutes.

smava adopted an architecture that utilizes both provisioned and serverless Redshift data warehouses, together with the data sharing capability to isolate the workloads. By choosing the right architectural patterns for their needs, smava was able to accomplish the following:

  • Simplify the data pipelines and reduce operational overhead
  • Reduce the feature release time from days to minutes
  • Increase price-performance by reducing idle times and right-sizing the workload
  • Achieve up to three times faster report generation (faster calculations and higher parallelization) at 50% of the original setup costs
  • Increase agility of all departments and support data-driven decision-making by democratizing access to data
  • Increase the speed of innovation by exposing self-service data capabilities for teams across all departments and strengthening the A/B test capabilities to cover the complete customer journey

Now, all departments at smava are using the available data products to make data-driven, accurate, and agile decisions.

Future vision

For the future, smava plans to continue to optimize the Data Platform based on operational metrics. They’re considering switching more provisioned clusters like the Self-Service Data Mart cluster to serverless. Additionally, smava is optimizing the ELT orchestration toolchain to increase the number of parallel data pipelines to be run. This will increase the utilization of provisioned Redshift resources and allow for cost reductions.

With the introduction of the decentralized, self-service for data product creation, smava made a step forward towards a data mesh architecture. In the future, the Data Platform team plans to further evaluate the needs of their service users and establish further data mesh principles like federated data governance.

Conclusion

In this post, we showed how smava optimized their data platform by isolating environments and workloads using Redshift Serverless and data sharing features. Those Redshift environments are well integrated with their infrastructure, flexible in scaling on demand, and highly available, and they require minimum administration efforts. Overall, smava has increased performance by three times while reducing the total platform costs by 50%. Additionally, they reduced operational overhead to a minimum while maintaining the existing SLAs for report generation times. Moreover, smava has strengthened the culture of innovation by providing self-service data product capabilities to speed up their time to market.

If you’re interested in learning more about Amazon Redshift capabilities, we recommend watching the most recent What’s new with Amazon Redshift session in the AWS Events channel to get an overview of the features recently added to the service. You can also explore the self-service, hands-on Amazon Redshift labs to experiment with key Amazon Redshift functionalities in a guided manner.

You can also dive deeper into Redshift Serverless use cases and data sharing use cases. Additionally, check out the data sharing best practices and discover how other customers optimized for cost and performance with Redshift data sharing to get inspired for your own workloads.

If you prefer books, check out Amazon Redshift: The Definitive Guide by O’Reilly, where the authors detail the capabilities of Amazon Redshift and provide you with insights on corresponding patterns and techniques.


About the Authors

Blog author: Alex NaumovAlex Naumov is a Principal Data Architect at smava GmbH, and leads the transformation projects at the Data department. Alex previously worked 10 years as a consultant and data/solution architect in a wide variety of domains, such as telecommunications, banking, energy, and finance, using various tech stacks, and in many different countries. He has a great passion for data and transforming organizations to become data-driven and the best in what they do.

Blog author: Lingli ZhengLingli Zheng works as a Business Development Manager in the AWS worldwide specialist organization, supporting customers in the DACH region to get the best value out of Amazon analytics services. With over 12 years of experience in energy, automation, and the software industry with a focus on data analytics, AI, and ML, she is dedicated to helping customers achieve tangible business results through digital transformation.

Blog author: Alexander SpivakAlexander Spivak is a Senior Startup Solutions Architect at AWS, focusing on B2B ISV customers across EMEA North. Prior to AWS, Alexander worked as a consultant in financial services engagements, including various roles in software development and architecture. He is passionate about data analytics, serverless architectures, and creating efficient organizations.


This post was reviewed for technical accuracy by David Greenshtein, Senior Analytics Solutions Architect.

Accelerate analytics on Amazon OpenSearch Service with AWS Glue through its native connector

Post Syndicated from Basheer Sheriff original https://aws.amazon.com/blogs/big-data/accelerate-analytics-on-amazon-opensearch-service-with-aws-glue-through-its-native-connector/

As the volume and complexity of analytics workloads continue to grow, customers are looking for more efficient and cost-effective ways to ingest and analyse data. Data is stored from online systems such as the databases, CRMs, and marketing systems to data stores such as data lakes on Amazon Simple Storage Service (Amazon S3), data warehouses in Amazon Redshift, and purpose-built stores such as Amazon OpenSearch Service, Amazon Neptune, and Amazon Timestream.

OpenSearch Service is used for multiple purposes, such as observability, search analytics, consolidation, cost savings, compliance, and integration. OpenSearch Service also has vector database capabilities that let you implement semantic search and Retrieval Augmented Generation (RAG) with large language models (LLMs) to build recommendation and media search engines. Previously, to integrate with OpenSearch Service, you could use open source clients for specific programming languages such as Java, Python, or JavaScript or use REST APIs provided by OpenSearch Service.

Movement of data across data lakes, data warehouses, and purpose-built stores is achieved by extract, transform, and load (ETL) processes using data integration services such as AWS Glue. AWS Glue is a serverless data integration service that makes it straightforward to discover, prepare, and combine data for analytics, machine learning (ML), and application development. AWS Glue provides both visual and code-based interfaces to make data integration effortless. Using a native AWS Glue connector increases agility, simplifies data movement, and improves data quality.

In this post, we explore the AWS Glue native connector to OpenSearch Service and discover how it eliminates the need to build and maintain custom code or third-party tools to integrate with OpenSearch Service. This accelerates analytics pipelines and search use cases, providing instant access to your data in OpenSearch Service. You can now use data stored in OpenSearch Service indexes as a source or target within the AWS Glue Studio no-code, drag-and-drop visual interface or directly in an AWS Glue ETL job script. When combined with AWS Glue ETL capabilities, this new connector simplifies the creation of ETL pipelines, enabling ETL developers to save time building and maintaining data pipelines.

Solution overview

The new native OpenSearch Service connector is a powerful tool that can help organizations unlock the full potential of their data. It enables you to efficiently read and write data from OpenSearch Service without needing to install or manage OpenSearch Service connector libraries.

In this post, we demonstrate exporting the New York City Taxi and Limousine Commission (TLC) Trip Record Data dataset into OpenSearch Service using the AWS Glue native connector. The following diagram illustrates the solution architecture.

By the end of this post, your visual ETL job will resemble the following screenshot.

Prerequisites

To follow along with this post, you need a running OpenSearch Service domain. For setup instructions, refer to Getting started with Amazon OpenSearch Service. Ensure it is public, for simplicity, and note the primary user and password for later use.

Note that as of this writing, the AWS Glue OpenSearch Service connector doesn’t support Amazon OpenSearch Serverless, so you need to set up a provisioned domain.

Create an S3 bucket

We use an AWS CloudFormation template to create an S3 bucket to store the sample data. Complete the following steps:

  1. Choose Launch Stack.
  2. On the Specify stack details page, enter a name for the stack.
  3. Choose Next.
  4. On the Configure stack options page, choose Next.
  5. On the Review page, select I acknowledge that AWS CloudFormation might create IAM resources.
  6. Choose Submit.

The stack takes about 2 minutes to deploy.

Create an index in the OpenSearch Service domain

To create an index in the OpenSearch service domain, complete the following steps:

  1. On the OpenSearch Service console, choose Domains in the navigation pane.
  2. Open the domain you created as a prerequisite.
  3. Choose the link under OpenSearch Dashboards URL.
  4. On the navigation menu, choose Dev Tools.
  5. Enter the following code to create the index:
PUT /yellow-taxi-index
{
  "mappings": {
    "properties": {
      "VendorID": {
        "type": "integer"
      },
      "tpep_pickup_datetime": {
        "type": "date",
        "format": "epoch_millis"
      },
      "tpep_dropoff_datetime": {
        "type": "date",
        "format": "epoch_millis"
      },
      "passenger_count": {
        "type": "integer"
      },
      "trip_distance": {
        "type": "float"
      },
      "RatecodeID": {
        "type": "integer"
      },
      "store_and_fwd_flag": {
        "type": "keyword"
      },
      "PULocationID": {
        "type": "integer"
      },
      "DOLocationID": {
        "type": "integer"
      },
      "payment_type": {
        "type": "integer"
      },
      "fare_amount": {
        "type": "float"
      },
      "extra": {
        "type": "float"
      },
      "mta_tax": {
        "type": "float"
      },
      "tip_amount": {
        "type": "float"
      },
      "tolls_amount": {
        "type": "float"
      },
      "improvement_surcharge": {
        "type": "float"
      },
      "total_amount": {
        "type": "float"
      },
      "congestion_surcharge": {
        "type": "float"
      },
      "airport_fee": {
        "type": "integer"
      }
    }
  }
}

Create a secret for OpenSearch Service credentials

In this post, we use basic authentication and store our authentication credentials securely using AWS Secrets Manager. Complete the following steps to create a Secrets Manager secret:

  1. On the Secrets Manager console, choose Secrets in the navigation pane.
  2. Choose Store a new secret.
  3. For Secret type, select Other type of secret.
  4. For Key/value pairs, enter the user name opensearch.net.http.auth.user and the password opensearch.net.http.auth.pass.
  5. Choose Next.
  6. Complete the remaining steps to create your secret.

Create an IAM role for the AWS Glue job

Complete the following steps to configure an AWS Identity and Access Management (IAM) role for the AWS Glue job:

  1. On the IAM console, create a new role.
  2. Attach the AWS managed policy GlueServiceRole.
  3. Attach the following policy to the role. Replace each ARN with the corresponding ARN of the OpenSearch Service domain, Secrets Manager secret, and S3 bucket.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "OpenSearchPolicy",
            "Effect": "Allow",
            "Action": [
                "es:ESHttpPost",
                "es:ESHttpPut"
            ],
            "Resource": [
                "arn:aws:es:<region>:<aws-account-id>:domain/<amazon-opensearch-domain-name>"
            ]
        },
        {
            "Sid": "GetDescribeSecret",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": "arn:aws:secretsmanager:<region>:<aws-account-id>:secret:<secret-name>"
        },
        {
            "Sid": "S3Policy",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:GetBucketAcl",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket-name>",
                "arn:aws:s3:::<bucket-name>/*"
            ]
        }
    ]
}

Create an AWS Glue connection

Before you can use the OpenSearch Service connector, you need to create an AWS Glue connection for connecting to OpenSearch Service. Complete the following steps:

  1. On the AWS Glue console, choose Connections in the navigation pane.
  2. Choose Create connection.
  3. For Name, enter opensearch-connection.
  4. For Connection type, choose Amazon OpenSearch.
  5. For Domain endpoint, enter the domain endpoint of OpenSearch Service.
  6. For Port, enter HTTPS port 443.
  7. For Resource, enter yellow-taxi-index.

In this context, resource means the index of OpenSearch Service where the data is read from or written to.

  1. Select Wan only enabled.
  2. For AWS Secret, choose the secret you created earlier.
  3. Optionally, if you’re connecting to an OpenSearch Service domain in a VPC, specify a VPC, subnet, and security group to run AWS Glue jobs inside the VPC. For security groups, a self-referencing inbound rule is required. For more information, see Setting up networking for development for AWS Glue.
  4. Choose Create connection.

Create an ETL job using AWS Glue Studio

Complete the following steps to create your AWS Glue ETL job:

  1. On the AWS Glue console, choose Visual ETL in the navigation pane.
  2. Choose Create job and Visual ETL.
  3. On the AWS Glue Studio console, change the job name to opensearch-etl.
  4. Choose Amazon S3 for the data source and Amazon OpenSearch for the data target.

Between the source and target, you can optionally insert transform nodes. In this solution, we create a job that has only source and target nodes for simplicity.

  1. In the Data source properties section, specify the S3 bucket where the sample data is located, and choose Parquet as the data format.
  2. In the Data sink properties section, specify the connection you created in the previous section (opensearch-connection).
  3. Choose the Job details tab, and in the Basic properties section, specify the IAM role you created earlier.
  4. Choose Save to save your job, and choose Run to run the job.
  5. Navigate to the Runs tab to check the status of the job. When it is successful, the run status should be Succeeded.
  6. After the job runs successfully, navigate to OpenSearch Dashboards, and log in to the dashboard.
  7. Choose Dashboards Management on the navigation menu.
  8. Choose Index patterns, and choose Create index pattern.
  9. Enter yellow-taxi-index for Index pattern name.
  10. Choose tpep_pickup_datetime for Time.
  11. Choose Create index pattern. This index pattern will be used to visualize the index.
  12. Choose Discover on the navigation menu, and choose yellow-taxi-index.


You have now created an index in OpenSearch Service and loaded data into it from Amazon S3 in just a few steps using the AWS Glue OpenSearch Service native connector.

Clean up

To avoid incurring charges, clean up the resources in your AWS account by completing the following steps:

  1. On the AWS Glue console, choose ETL jobs in the navigation pane.
  2. From the list of jobs, select the job opensearch-etl, and on the Actions menu, choose Delete.
  3. On the AWS Glue console, choose Data connections in the navigation pane.
  4. Select opensearch-connection from the list of connectors, and on the Actions menu, choose Delete.
  5. On the IAM console, choose Roles in the navigation page.
  6. Select the role you created for the AWS Glue job and delete it.
  7. On the CloudFormation console, choose Stacks in the navigation pane.
  8. Select the stack you created for the S3 bucket and sample data and delete it.
  9. On the Secrets Manager console, choose Secrets in the navigation pane.
  10. Select the secret you created, and on the Actions menu, choose Delete.
  11. Reduce the waiting period to 7 days and schedule the deletion.

Conclusion

The integration of AWS Glue with OpenSearch Service adds the powerful ability to perform data transformation when integrating with OpenSearch Service for analytics use cases. This enables organizations to streamline data integration and analytics with OpenSearch Service. The serverless nature of AWS Glue means no infrastructure management, and you pay only for the resources consumed while your jobs are running. As organizations increasingly rely on data for decision-making, this native Spark connector provides an efficient, cost-effective, and agile solution to swiftly meet data analytics needs.


About the authors

Basheer Sheriff is a Senior Solutions Architect at AWS. He loves to help customers solve interesting problems leveraging new technology. He is based in Melbourne, Australia, and likes to play sports such as football and cricket.

Shunsuke Goto is a Prototyping Engineer working at AWS. He works closely with customers to build their prototypes and also helps customers build analytics systems.

A return to US net neutrality rules?

Post Syndicated from Zaid Zaid http://blog.cloudflare.com/author/zaid-zaid/ original https://blog.cloudflare.com/a-return-to-us-net-neutrality-rules


For nearly 15 years, the Federal Communications Commission (FCC) in the United States has gone back and forth on open Internet rules – promulgating and then repealing, with some court battles thrown in for good measure. Last week was the deadline for Internet stakeholders to submit comments to the FCC about their recently proposed net neutrality rules for Internet Service Providers (ISPs), which would introduce considerable protections for consumers and codify the responsibility held by ISPs.

For anyone who has worked to help to build a better Internet, as Cloudflare has for the past 13 years, the reemergence of net neutrality is déjà vu all over again. Cloudflare has long supported the open Internet principles that are behind net neutrality, and we still do today. That’s why we filed comments with the FCC expressing our support for these principles, and concurring with many of the technical definitions and proposals that largely would reinstitute the net neutrality rules that were previously in place.

But let’s back up and talk about net neutrality. Net neutrality is the principle that ISPs should not discriminate against the traffic that flows through them. Specifically, when these rules were adopted by the FCC in 2015, there were three bright line rules: (1) that ISPs cannot block subscribers from reaching legal content, applications or services, (2) that ISPs cannot throttle subscribers’ access to content, putting some content in a “slow lane”, and (3) that ISPs can’t engage in “paid prioritization” which means charging websites and services for special access to their subscribers.

Net neutrality has a long history. In 2010, the FCC passed the first set of open Internet rules which were: (1) no blocking; (2) no unreasonable discrimination; and (3) transparency rules. In 2014, after a lawsuit from Verizon, the D.C. Circuit Court invalidated some of the 2010 rules, saying that if the FCC wanted to have these rules, it needed to treat ISPs as “common carriers.” (A common carrier is an entity that offers its services to the general public and will provide its services to anyone willing to pay the fee.) In 2015, the FCC did exactly that: it reclassified ISPs as common carriers, and instituted rules which we now know as net neutrality protections. In 2017, the FCC reversed course and repealed the rules. Now, the FCC again wants to reinstate them. It’s a dizzying chain of events.

And all the while, the Internet has carried on. For most Americans, net neutrality principles are reasonably uncontroversial — surveys show that more than 80% of Americans support them. And for all the lawsuits and regulatory ping-pong, in our view ISPs have largely followed these principles. The Internet has worked and is working.

What is broadband Internet?

In the same way that the delivery of Internet service hasn’t changed much, the underlying rationale for the net neutrality rules hasn’t changed. Broadband Internet is more critical than ever for our day-to-day lives, with more of our healthcare, work, education and entertainment happening over the Internet. ISPs still now, as then, are likely to have a monopoly on how subscribers reach the Internet – there’s only one path in and out of most people’s homes over the Internet, and even where consumers have a choice, they often face onerous switching costs. The FCC is ensuring there are rules for that road by defining the requirements that ISPs are obliged to fulfill.

In late September, the FCC released a public draft of its Notice of Proposed Rulemaking (NPRM) on net neutrality and gave the public about 3 months to review it and submit comments to the agency. The current NPRM asks what has changed about the Internet since 2015, whether the original principles are still the right ones, what should be the definition of an ISP, and many other things. The net neutrality principles proposed by the FCC will be familiar to net neutrality advocates, who have campaigned for similar ideas for years. As always, at Cloudflare we want consumers to have full access to legal content and services on the Internet.

What has changed – or at least become more complicated – is all of the various services that consumers and businesses use on the Internet. At Cloudflare, we know this well because we offer many of these services. We offer a content delivery network that protects and accelerates website delivery to consumers. We have a developer platform that developers use to deploy their code all across the world. And we have a platform that offers large businesses the ability to securely connect their offices and employees. Of course, we’re not alone. The ability of the Internet to foster permissionless innovation is unmatched.

For all the innovation (and some quite complicated services) flowing across the Internet, the ISPs that would be subject to these rules are, in our view, easy to define. In FCC terminology, an ISP is a provider of Broadband Internet Access Services (“BIAS”). As the FCC proposes to define it, a BIAS service is a mass-market Internet service which consumers purchase with the expectation they can reach the whole Internet. One of the main things we said to the FCC in our comments boils down to “you know a BIAS service when you see one.” Once we have a simple definition of BIAS service, we’ve also established that everything else is not BIAS.

As we said in our comments to the FCC:

[The FCC’s] historic definition identifies two primary characteristics of BIAS: (1) “a mass-market retail service” that (2) “provides the capability to transmit data to and receive data from all or substantially all internet endpoints.” The proposed definition of BIAS places the focus where it belongs: the ability of Internet end users to reach and interact with all Internet destinations without interference from their BIAS provider.

Interconnection and traffic exchange between networks

The interconnection section of the FCC’s proposed rules is also worthy of attention. Interconnection is how networks send data to one another on the Internet. Cloudflare is one of the best connected networks in the world (we’re directly connected to over 13,000 other networks, and are present at nearly as many Internet exchanges as any other network) so we know this topic well.

To give a very brief overview of the way interconnection works, assume a user on the network of ISP A requests cloudflare.com in their browser. That request goes out from the subscriber’s home through the ISP’s network. At some point it will reach an interconnection point, which is a data center where lots of networks connect together. If the ISP network and the content network (in this example it’s Cloudflare, since they are requesting cloudflare.com) directly connect (called “peering”) then the request will pass to Cloudflare and Cloudflare will respond, delivering back the HTML, JavaScript, images, and everything else that makes up a website.

Maybe the ISP and Cloudflare aren’t peered directly, but if they are both members of the same Internet Exchange, traffic could be exchanged there. Or, if neither of those are an option, the ISP and Cloudflare might exchange data through an IP transit provider, a 3rd party network that gets paid to deliver traffic on their behalf.

Interconnection is relevant to the FCC’s net neutrality proceeding because an ISP makes a representation to their subscriber that the subscriber can access the whole Internet, and the ISP needs to make interconnection arrangements to make good on that representation.

What the FCC is proposing is that ISPs would be required to make interconnection arrangements as part of their responsibility to deliver the whole Internet to their subscribers without blocking, throttling, or paid prioritization.

Beyond the representation that ISP’s make to their subscribers, the FCC is not proposing to directly impose rules on interconnection. Instead, the FCC is proposing to adopt a “watch, learn, and act as required” case-by-case approach to interconnection challenges. Interconnection disputes between ISPs and content and service providers have happened. Famously, in 2014, Comcast and Netflix didn’t have enough interconnection capacity and thus Comcast subscribers trying to watch Netflix were subject to lots of buffering and a generally bad experience. But they worked it out between themselves. Similar disputes in the United States have been rare since.

Both from the Comcast-Netflix instance, and other issues we see internationally, we know interconnection disputes can arise, and they can affect users. For example, we’re currently monitoring interconnection in Germany, where users on one of the largest networks have had trouble reaching normal websites like GitHub, or just browsing the Internet. It’s likely those issues are caused by insufficient interconnection capacity.

While we don’t have this type of interconnection issue in the United States currently, under the proposed rules the FCC would be set up as an arbiter of last resort for disputes in the United States. With this approach, hopefully we would be able to avoid the type of issues we’re seeing in Germany. And if ever consumers’ Internet experience was being harmed by the interconnection policy of any network, the FCC could adjudicate the matter.

It has been eight years since net neutrality rules were passed in the United States, and six years since they were repealed. During that time the Internet has kept growing. If the FCC does reinstate net neutrality rules, we’re hopeful they will be common sense rules of the road for ISPs, making official the already-widely-followed principles of a free and open Internet.

Announcing `async fn` and return-position `impl Trait` in traits (Rust Blog)

Post Syndicated from corbet original https://lwn.net/Articles/955925/

The Rust Blog announces
the stabilization of a couple of trait features aimed at improving support
for async code:

Ever since the stabilization of RFC #1522 in Rust 1.26, Rust has
allowed users to write impl Trait as the return type of
functions (often called “RPIT”). This means that the function
returns “some type that implements Trait“. This is
commonly used to return closures, iterators, and other types that
are complex or impossible to write explicitly. […]

Starting in Rust 1.75, you can use return-position impl
Trait
in trait (RPITIT) definitions and in trait impls. For
example, you could use this to write a trait method that returns an
iterator: […]

So what does all of this have to do with async functions? Well,
async functions are “just sugar” for functions that return
-> impl Future. Since these are now permitted in
traits, we also permit you to write traits that use async fn.

Три отговора за Конституцията

Post Syndicated from Bozho original https://blog.bozho.net/blog/4187

Три въпроса/критики, които виждам най-често във връзка с изменения в Конституцията.

1. Защо парламентарната квота в прокурорския съвет е по-голяма от професионалната – така не се ли поставя прокуратурата под политически контрол?

Проблемът с прокуратурата към момента е, че главният прокурор (който е началник на всички прокурори и може да осъществява надзор за законност по делата) или не носеше отговорност пред никого („над мен е само Бог“), или получаваше задкулисни указания от различни центрове на власт. Съответно отговорност за действията на прокуратурата не се носеха – „те си го избират прокурорите“.

С измененията, освен че му премахваме правата за намеса, слагаме превес на обществената квота. Това, от една страна, предотвратява капсулирането (прокурори, които зависят кариерно от главния, го поддържат в съвета), а от друга дава отговорност и отчетност. Трябва да отбележим и че обществената квота не е партийна и това ще е водещ принцип при нейното излъчване.

С друго изменение подчертаваме, че водещата роля в съдебната власт е на съда. Прокуратурата и следствието са страна по делата, а съдът е арбитър. Именно независимостта на съда е в центъра на съдебната власт, поради което квотите там са наобратно – 8 съдии избрани от съдии (от общо 15 члена, като само 5 се избират от парламента). Водещото при съда е, че трябва да е независим. Водещото при прокуратурата, която осъществява наказателната политика на държавата, е да бъде отчетна. Именно за това са такива квотите в двата съвета.

2. Защо двойно гражданство? Така не се ли отваря вратата за чуждо влияние?

В рамките на дебата бяха изнесени данни от г-жа Цонева, че България е последната държава и в ЕС, и в Съвета на Европа, която позволява двойно гражданство по принцип, но го забранява за изборни длъжности. С премахването на това ограничение правим по-лесно за българската диаспора (стотиците хиляди емигранти за последните 30 години) да се включи в политическия без бюрократични пречки и усложнения.

Изискването да нямаш двойно гражданство, за да си депутат/министър, не е пречка за чужди агенти на влияние да заемат позиции, както някои се надяват. Пример е Георги Димитров, който преди да се върне и да превърне България в съветска колония, се отказва от съветското си гражданство. Борбата с чуждото влияние е работа на службите, а не на формално конституционно ограничение.

Има няколко вида българско гражданство – по рождение (ако си се родил с българско гражданство или си дете на български гражданин), по произход (от историческите територии, на които е имало българско население) и по натурализация (ако сиsigna живял поне 5 години в България и отговаряш на допълнителни изисквания по закон). Няма хипотеза, в която някой чужденец получава българско гражданство без да е чувал за България, така че тези притеснения са неоснователни.

3. Защо с обикновено Народно събрание приемате изменения, за които се изисква Велико Народно Събрание, съгласно решение на Конституционния съд?

Решение 3 от 2003 г. на Конституционния съд, по искане на тогавашния главен прокурор (който публично се е хвалел с влиянието си в Конституционния съд във връзка с това решение), определя как се тълкува разпоредбата на чл. 158, т. 3 от Констиуцията, а именно, какво значи „формата на държавно устройство и на държавно управление“. Тезата, че обикновено Народно събрание не може да разделя Висшия съдебен съвет на два съвета и да премахва правомощия на главния прокурор, се базира на това решение.

Има обаче и друго решение – номер 8 от 2005 г., което казва, че може да има повече от едно тълкувателно решение по една норма, и че „Промени в Конституцията, концентрирани в рамките на съдебната власт, [..] които са насочени към преструктуриране, оптимизиране от съдържателна гледна точка и прецизиране на отделни функции на нейни органи, поставяне на акценти или уточняване на техни правомощия или наименования, както и взаимодействието им с институции на другите власти, не представляват промяна във формата на държавно управление“

И двете решения са тълкувателни – по искане на главния прокурор и по искане на ВКС. Те дават рамка, но както посочва Решение 8, трябва да се извърши преценка на конкретни изменения, за да се прецени нюанса и баланса. Иначе казано, не може да се твърди, че Решение 3 прави настоящите изменения противоконституционни.


Всички тези аргументи са изказвани в комисия и в пленарна зала. Аз лично смятам, че измененията в Конституцията са добри, водят до по-независим съд и по-отчетна прокуратура. Критиките към проекта бяха отчетени с измененията между първо и второ четене, като бяха обсъждани с професионалните общности в двата месеца между двете първи гласувания. Подхождаме към Конституцията с огромно внимание и с разбиране за отговорността, която носим. Ако някой е прочел някоя критична статия или е чул изказване на някой от опозицията или е извадил едно-две критични изречения от три страници становище на някоя институция, призовавам към по-задълбочен прочит. И на контекста, и на дебатите в комисия и в зала.

Материалът Три отговора за Конституцията е публикуван за пръв път на БЛОГодаря.

How to implement client certificate revocation list checks at scale with API Gateway

Post Syndicated from Arthur Mnev original https://aws.amazon.com/blogs/security/how-to-implement-client-certificate-revocation-list-checks-at-scale-with-api-gateway/

ityAs you design your Amazon API Gateway applications to rely on mutual certificate authentication (mTLS), you need to consider how your application will verify the revocation status of a client certificate. In your design, you should account for the performance and availability of your verification mechanism to make sure that your application endpoints perform reliably.

In this blog post, I demonstrate an architecture that will help you on your journey to implement custom revocation checks against your certificate revocation list (CRL) for API Gateway. You will also learn advanced Amazon Simple Storage Service (Amazon S3) and AWS Lambda techniques to achieve higher performance and scalability.

Choosing the right certificate verification method

One of your first considerations is whether to use a CRL or the Online Certificate Status Protocol (OCSP), if your certificate authority (CA) offers this option. For an in-depth analysis of these two options, see my earlier blog post, Choosing the right certificate revocation method in ACM Private CA. In that post, I demonstrated that OCSP is a good choice when your application can tolerate high latency or a failure for certificate verification due to TLS service-to-OCSP connectivity. When you rely on mutual TLS authentication in a high-rate transactional environment, increased latency or OCSP reachability failures may affect your application. We strongly recommend that you validate the revocation status of your mutual TLS certificates. Verifying your client certificate status against the CRL is the correct approach for certificate verification if you require reliability and lower, predictable latency. A potential exception to this approach is the use case of AWS Certificate Manager Private Certificate Authority (AWS Private CA) with an OCSP responder hosted on AWS CloudFront.

With an AWS Private CA OCSP responder hosted on CloudFront, you can reduce the risks of network and latency challenges by relying on communication between AWS native services. While this post focuses on the solution that targets CRLs originating from any CA, if you use AWS Private CA with an OCSP responder, you should consider generating an OCSP request in your Lambda authorizer.

Mutual authentication with API Gateway

API Gateway mutual TLS authentication (mTLS) requires you to define a root of trust that will contain your certificate authority public key. During the mutual TLS authentication process, API Gateway performs the undifferentiated heavy lifting by offloading the certificate authentication and negotiation process. During the authentication process, API Gateway validates that your certificate is trusted, has valid dates, and uses a supported algorithm. Additionally, you can refer to the API Gateway documentation and related blog post for details about the mutual TLS authentication process on API Gateway.

Implementing mTLS certificate verification for API Gateway

In the remainder of this blog post, I’ll describe the architecture for a scalable implementation of a client certificate verification mechanism against a CRL on your API Gateway.

The certificate CRL verification process presented here relies on a custom Lambda authorizer that validates the certificate revocation status against the CRL. The Lambda authorizer caches CRL data to optimize the query time for subsequent requests and allows you to define custom business logic that could go beyond CRL verification. For example, you could include other, just-in-time authorization decisions as a part of your evaluation logic.

Implementation mechanisms

This section describes the implementation mechanisms that help you create a high-performing extension to the API Gateway mutual TLS authentication process.

Data repository for your certificate revocation list

API Gateway mutual TLS configuration uses Amazon S3 as a repository for your root of trust. The design for this sample implementation extends the use of S3 buckets to store your CRL and the public key for the certificate authority that signed the CRL.

We strongly recommend that you maintain an updated CRL and verify its signature before data processing. This process is automatic if you use AWS Private CA, because AWS Private CA will update your CRL automatically on revocation. AWS Private CA also allows you to retrieve the CA’s public key by using an API call.

Certificate validation

My sample implementation architecture uses the API Gateway Lambda authorizer to validate the serial number of the client certificate used in the mutual TLS authentication session against the list of serial numbers present in the CRL you publish to the S3 bucket. In the process, the API Gateway custom authorizer will read the client certificate serial number, read and validate the CRL’s digital signature, search for the client’s certificate serial number within the CRL, and return the authorization policy based on the findings.

Optimizing for performance

The mechanisms that enable a predictable, low-latency performance are CRL preprocessing and caching. Your CRL is an ASN.1 data structure that requires a relatively high computing time for processing. Preprocessing your CRL into a simple-to-parse data structure reduces the computational cost you would otherwise incur for every validation; caching the CRL will help you reduce the validation latency and improve predictability further.

Performance optimizations

The process of parsing and validating CRLs is computationally expensive. In the case of large CRL files, parsing the CRL in the Lambda authorizer on every request can result in high latency and timeouts. To improve latency and reduce compute costs, this solution optimizes for performance by preprocessing the CRL and implementing function-level caching.

Preprocessing and generation of a cached CRL file

The first optimization happens when S3 receives a new CRL object. As shown in Figure 1, the S3 PutObject event invokes a preprocessing Lambda that validates the signature of your uploaded CRL and decodes its ASN.1 format. The output of the preprocessing Lambda function is the list of the revoked certificate serial numbers from the CRL, in a data structure that is simpler to read by your programming language of choice, and that won’t require extensive parsing by your Lambda authorizer. The asynchronous approach mitigates the impact of CRL processing on your API Gateway workload.

Figure 1: Sample implementation flow of the pre-processing component

Figure 1: Sample implementation flow of the pre-processing component

Client certificate lookup in a CRL

The optimization happens as part of your Lambda authorizer that retrieves the preprocessed CRL data generated from the first step and searches through the data structure for your client certificate serial number. If the Lambda authorizer finds your client’s certificate serial number in the CRL, the authorization request fails, and the Lambda authorizer generates a “Deny” policy. Searching through a read-optimized data structure prepared by your preprocessing step is the second optimization that reduces the lookup time and the compute requirements.

Function-level caching

Because of the preprocessing, the Lambda authorizer code no longer needs to perform the expensive operation of decoding the ASN.1 data structures of the original CRL; however, network transfer latency will remain and may impact your application.

To improve performance, and as a third optimization, the Lambda service retains the runtime environment for a recently-run function for a non-deterministic period of time. If the function is invoked again during this time period, the Lambda function doesn’t have to initialize and can start running immediately. This is called a warm start. Function-level caching takes advantage of this warm start to hold the CRL data structure in memory persistently between function invocations so the Lambda function doesn’t have to download the preprocessed CRL data structure from S3 on every request.

The duration of the Lambda container’s warm state depends on multiple factors, such as usage patterns and parallel requests processed by your function. If, in your case, API use is infrequent or its usage pattern is spiky, pre-provisioned concurrency is another technique that can further reduce your Lambda startup times and the duration of your warm cache. Although provisioned concurrency does have additional costs, I recommend you evaluate its benefits for your specific environment. You can also check out the blog dedicated to this topic, Scheduling AWS Lambda Provisioned Concurrency for recurring peak usage.

To validate that the Lambda authorizer has the latest copy of the CRL data structure, the S3 ETag value is used to determine if the object has changed. The preprocessed CRL object’s ETag value is stored as a Lambda global variable, so its value is retained between invocations in the same runtime environment. When API Gateway invokes the Lambda authorizer, the function checks for existing global preprocessed CRL data structure and ETag variables. The process will only retrieve a read-optimized CRL when the ETag is absent, or its value differs from the ETag of the preprocessed CRL object in S3.

Figure 2 demonstrates this process flow.

Figure 2: Sample implementation flow for the Lambda authorizer component

Figure 2: Sample implementation flow for the Lambda authorizer component

In summary, you will have a Lambda container with a persistent in-memory lookup data structure for your CRL by doing the following:

  • Asynchronously start your preprocessing workflow by using the S3 PutObject event so you can generate and store your preprocessed CRL data structure in a separate S3 object.
  • Read the preprocessed CRL from S3 and its ETag value and store both values in global variables.
  • Compare the value of the ETag stored in your global variables to the current ETag value of the preprocessed CRL S3 object, to reduce unnecessary downloads if the current ETag value of your S3 object is the same as the previous value.
  • We recommend that you avoid using built-in API Gateway Lambda authorizer result caching, because the status of your certificate might change, and your authorization decision would rest on out-of-date verification results.
  • Consider setting a reserved concurrency for your CRL verification function so that API Gateway can invoke your function even if the overall capacity for your account in your AWS Region is exhausted.

The sample implementation flow diagram in Figure 3 demonstrates the overall architecture of the solution.

Figure 3: Sample implementation flow for the overall CRL verification architecture

Figure 3: Sample implementation flow for the overall CRL verification architecture

The workflow for the solution overall is as follows:

  1. An administrator publishes a CRL and its signing CA’s certificate to their non-public S3 bucket, which is accessible by the Lambda authorizer and preprocessor roles.
  2. An S3 event invokes the Lambda preprocessor to run upon CRL upload. The function retrieves the CRL from S3, validates its signature against the issuing certificate, and parses the CRL.
  3. The preprocessor Lambda stores the results in an S3 bucket with a name in the form <crlname>.cache.json.
  4. A TLS client requests an mTLS connection and supplies its certificate.
  5. API Gateway completes mTLS negotiation and invokes the Lambda authorizer.
  6. The Lambda authorizer function parses the client’s mTLS certificate, retrieves the cached CRL object, and searches the object for the serial number of the client’s certificate.
  7. The authorizer function returns a deny policy if the certificate is revoked or in error.
  8. API Gateway, if authorized, proceeds with the integrated function or denies the client’s request.

Conclusion

In this post, I presented a design for validating your API Gateway mutual TLS client certificates against a CRL, with support for extra-large certificate revocation files. This approach will help you align with the best security practices for validating client certificates and use advanced S3 access and Lambda caching techniques to minimize time and latency for validation.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Security, Identity, and Compliance re:Post or contact AWS Support.

Arthur Mnev

Arthur is a Senior Specialist Security Architect for AWS Industries. He spends his day working with customers and designing innovative approaches to help customers move forward with their initiatives, improve their security posture, and reduce security risks in their cloud journeys. Outside of work, Arthur enjoys being a father, skiing, scuba diving, and Krav Maga.

Rafael Cassolato de Meneses

Rafael Cassolato de Meneses

Rafael Cassolato is a Solutions Architect with 20+ years in IT, holding bachelor’s and master’s degrees in Computer Science and 10 AWS certifications. Specializing in migration and modernization, Rafael helps strategic AWS customers achieve their business goals and solve technical challenges by leveraging AWS’s cloud platform.

The collective thoughts of the interwebz