Кая от Голямата къща, или шансовете за „най-безнадеждните“ хора в Германия

Post Syndicated from Светла Енчева original https://www.toest.bg/kaya-ot-golyamata-kushta/

Кая от Голямата къща, или шансовете за „най-безнадеждните“ хора в Германия

Повод за тази статия е публикуваният на 1 март т.г. в „Тоест“ разказ на Юлия Георгиева за Розовата къща – единствения нископрагов център в България, където хора със зависимости могат да получат подкрепа. Същия ден приятелка, която живее в Германия, постъпи на работа в къща (нека я наречем Голямата къща), която според концепцията си предоставя шанс на хора с „особени социални затруднения“. Обитателите ѝ са, освен със зависимости, и с други психиатрични заболявания. Плюс това са бездомни. Отгоре на всичко повечето от тях са били в затвора или в психиатрия от затворен тип за извършители на престъпления.

Наслагването на няколко дискриминационни признака един върху друг се нарича интерсекционалност – това е термин от съвременния феминизъм. Жителите на Голямата къща са въплъщение на интерсекционалността – носители са на по три-четири дискриминационни признака накуп, всеки от които е силно стигматизиран в обществото. Това не са хора, които бихте искали да срещнете на улицата, особено по тъмно, дори да сте толерантни. Всяко помещение от къщата е снабдено с паникбутон. Макар до натискане на някой от бутоните да се стига веднъж на 5–10 години, наличието им вдъхва известна сигурност у работещите в къщата.

Историята на Кая

Искам да ви представя Кая, която отскоро е назначена в Голямата къща. Тя не се казва така, но си избра това име за статията, в която разказвам за работата ѝ. Кая е българка, малко над 30-те, живееща в Германия. Пълна е с енергия и има широка усмивка, която става още по-широка, когато говори за новото си работно място. Преди близо година се дипломира като бакалавър по социални дейности и оттогава работи по специалността си.

Макар че умее да пази здравословна дистанция спрямо хората, с които работи, Кая е способна „да влезе в обувките им“. Защото си дава сметка, че и тя можеше да не е много по-различна от тях, ако беше останала в България. В приятелската ѝ компания от гимназията наркотиците са били всекидневие. За да се откъсне от тази среда, Кая заминава за Германия, където започва от нулата – записва се в университет, учи езика в движение и работи по време на следването си, за да се издържа. Оставя не само наркотиците, а и цигарите и алкохола. Междувременно зависимостта на един член на тийнейджърската ѝ компания от наркотиците става повод за убийството му, а друг от тайфата влиза в затвора.

Кая казва, че ѝ харесва да работи там, където е трудно и има предизвикателства. Като студентка стажува в друга къща за хора със зависимости. В началото на пробната година след дипломирането си, след успешното преминаване на която вече ще е пълноправна социална работничка, постъпва в къща за възрастни с психически и други здравословни проблеми. Последните месеци от изпитателния период ще прекара в Голямата къща, където има шанс да я назначат за постоянно и на пълно работно време. Там е трудно – точно според нейния вкус, което ще рече твърде трудно за повечето социални работници в Германия. Затова и желаещите да работят в къщата не са много.

Голямата къща в системата на „Диакони“

Както и другите социални услуги, споменати в горния параграф, Голямата къща е част от „Диакони“ – голяма организация към Евангелската църква. Може да се каже, че „Диакони“ е евангелският вариант на католическата „Каритас“. За разлика от повечето известни религиозни организации в България обаче, в Германия „Диакони“ и „Каритас“ не се занимават с гонене на „джендъра“, а помагат на нуждаещите се, без да се интересуват от отношението на последните към християнските ценности. (В България „Каритас“ помага на някои групи нуждаещи се, но не пропусна да се „разпише“ срещу Истанбулската конвенция.)

Освен „Диакони“ и „Каритас“, с предоставяне на социални услуги в Германия се занимават множество организации – профсъюзни, частни, общински, Червеният кръст и т.н. В половинмилионния град, в който работи Кая, съществуват различни възможности за социална подкрепа. Конкретно „Диакони“ разполага с над 1150 места за настаняване на хора на всякаква възраст, с всевъзможни проблеми и потребности – от деца с увреждания и младежи с проблемно поведение до възрастни с деменция.

Хората със зависимости могат да получат различен тип подкрепа от „Диакони“ според спецификата на състоянието си. Ако могат да живеят самостоятелно, социален работник ще ги посещава периодично да следи как се справят. Ако никак не се оправят със себе си, ще ги пратят в къща като тази от студентския стаж на Кая, служителите в която дори следят дали обитателите пропускат някоя част от тялото си, докато се къпят, и им напомнят да си я измият, но не им връзват кусур, ако не са се отказали от зависимостите си. А ако са с „особени социални затруднения“, се насочват към Голямата къща.

Пътят към Голямата къща

Клиентите на Голямата къща стигат до нея по различни начини. Някои са научили за мястото от познати или приятели. Други са забелязани и насочени от социални работници. По отношение на трети, от затвора или психиатрията, където са лежали, се свързват с къщата: „Тук имаме един човек като за вас, който излиза на свобода и няма къде да отиде.“ Четвърти са насочени от персонала на приюта за бездомни на гарата. Пети идват от други къщи, за които е преценено, че не са подходящи. И т.н.

Приемането в Голямата къща става с интервю и по определени критерии. Кандидатите трябва да са адресно регистрирани в рамките на съответния регион. Ако нямат адресна регистрация (повечето имат, макар да са бездомни), им се прави, а ако са регистрирани в друг регион, се препращат към тамошното „Диакони“. Задължително условие е да не употребяват наркотици/алкохол и да са минали през абстиненция. Ако не са, се насочват към клиника, през която да преминат, преди да дойдат пак. Потенциалните обитатели трябва да покажат и някакво желание да започнат да се справят сами с живота си.

Престоят в къщата не е безплатен, макар клиентите да не го плащат директно от джоба си. Той струва над 800 евро на месец. В тях се включват социалните помощи, които обитателите ѝ биха получавали, ако живееха сами – 502 евро за 2023 г. плюс пари за наем на жилище. Къщата съдейства на тези от тях, които не са вземали социални помощи до този момент, да получат. Клиентите подписват декларация, че предоставят средствата на къщата.

Структура на къщата

В Голямата къща има място за 55 души. Около 80% от тях са мъже. Осем от клиентите всъщност не живеят в сградата, но за това – след малко. На партерния етаж се настаняват в стаи за двама новодошлите за пробен срок от няколко седмици. Тези от тях, които успеят да вкарат всекидневието си в някакви рамки и да понесат съжителството със състайниците си, се прехвърлят в самостоятелна стая при другите обитатели, минали през тази цедка. Но около половината не издържат още на този етап и напускат.

В основната част на къщата клиентите прекарват средно между шест месеца и година, някои повече, други – по-малко. Целта е да се научат във възможно по-голяма степен да се справят сами. Затова и всички, които могат, работят нещо в къщата. Един отговаря за пералнята, друг изпълнява ролята на офис секретар, трети работи в кафетерията и т.н. За работата си те получават по 1,5 евро на час. Тези пари се добавят към 135-те евро джобни на месец, които се полагат на всеки от обитателите и които им се дават на ръка.

Осемте клиенти, които показват най-голям напредък в социализирането си, живеят в отделна къща с двор, която, макар да е на около километър и половина от Голямата къща, се води част от нея. Те ходят на работа и отговарят за живота в къщата съвсем сами – така, както го правят редовите немски съквартиранти. Това е последната стъпка, преди да се отделят от къщата и да излязат на квартира, и до нея успяват да стигнат единици. Повечето напускат на по-ранен етап, някои са изгонени от къщата, а други просто изчезват.

Какво работят служителите в къщата

Вече стана ясно, че целта на Голямата къща е да помогне на обитателите си да „влязат в коловозите“ на обществото и да започнат да живеят независимо, без да представляват опасност за себе си и околните. В този смисъл къщата е нещо като гара разпределителна към „нормалния“ живот и в нея няма кой знае колко форми на психологическа подкрепа, като изключим работата на ерготерапевта, който предлага например възможности за арт терапия.

Персоналът се грижи клиентите да посещават психиатрите си (които са извън къщата), а ако нямат, да им намерят. Урежда здравните им застраховки, намира им възможности за работа и/или за професионална квалификация, както и квартири. В повечето случаи обитателите отиват сами където трябва, но понякога се налага и някой да бъде придружен. Дори чистачката (която не е от обитателите) няма задължението да почиства абсолютно всичко, а в работата ѝ се включва да напомня на клиентите сами да си измият прозорците или да избършат праха.

Разбира се, работещите в къщата полагат усилия и за потушаването на конфликтите, които неизбежно възникват в подобна среда. Освен конфликти, понякога възниква и любов. В тези случаи влюбените биват помолени да държат връзката си в тайна от останалите, за да не предизвикват напрежение. Предупреждават ги, че ако ги видят да правят секс, ще ги изгонят от къщата. Което е по-скоро намек да си заключват стаите, когато пристъпват към тази дейност, защото в същото време ги съветват да ползват предпазни средства.

Личните истории

„За статията си имаш нужда от лични истории“, сеща се Кая. Изводът, до който е стигнала, прочитайки документацията за клиентите, е, че проблемите им обикновено се коренят в детството.

Например Х страда от шизофрения и когато изпадне в криза, чува гласове, които му заповядват да престане да яде и да пие вода. Щом от персонала видят, че е престанал да се храни и да поема течности, са наясно, че състоянието му се е обострило. Защо ли гласовете му нашепват именно това? Навремето майка му го е наказвала да не яде и да не пие вода, когато преценявала, че „не слуша“.

За Y пък има данни още от детските му години, че нещо не е наред в поведението му. Родителите му така и не са обърнали внимание и не са го завели при специалист. С годините той започва да се държи все по-екстремно – например, опитва се да изгони живеещите в сградата, в която скоро се е нанесъл под наем, поради което самият той е изгонен оттам след дни. А веднъж пребива баща си почти до смърт. След като няколко пъти е лежал в затвора, съдията на поредното му дело решава да го прати на психиатър. Оказва се, че още от детството си е с недиагностицирана шизофрения. Днес отговаря за пералнята в къщата и набляга на фитнеса.

Кая забелязва, че и външният вид понякога може да създаде погрешна представа. Z например е двуметров великан, целият в татуировки. Първоначално видът му плаши Кая, която, както вече стана ясно, изобщо не се плаши лесно. Ала тя скоро научава, че той е много кротък човек в дълбока депресия и е опасен само за себе си.

Повечето от обитателите на Голямата къща имат зависимост от наркотици и/или алкохол, но има и такива със зависимост например от компютърни игри – толкова голяма, че са изгубили жилищата си заради нея.

Не всичко е розово в Голямата къща

Кая не спестява и нещата в Голямата къща, които не ѝ харесват. Основен проблем за нея е липсата на достатъчно персонал, поради което не се обръща необходимото индивидуално внимание на обитателите, а те са „пуснати по течението“.

Според Кая в най-голяма степен се пренебрегва психичното състояние на клиентите. При постъпването на всеки в къщата се определят три (рядко – четири) приоритета, върху които да се работи по време на престоя му в нея. Сред тези приоритети са търсенето на работа, търсенето на жилище, преодоляването на зависимостта, психичното състояние и др. Обикновено тъкмо психичното състояние „изпада“ от приоритетите.

Най-трудно ѝ е да се примири с факта, че Голямата къща гони някои от обитателите си, макар да е наясно, че това понякога се налага. Ако клиент е хванат, че взема наркотици и не се „стегне“ до една седмица, изгонването му е гарантирано. За 90% от тези, за които вече няма място в къщата, могат да се намерят други социални услуги. Но има и такива, които просто остават на улицата. И вероятността някой отново да им подаде ръка е нищожна.

А в България е розова само Розовата къща

Слушайки Кая, мислено сравнявам ситуацията в Германия с тази в България. Да, и в Германия повечето хора не умират от желание да работят със зависими, с психичноболни, с бивши затворници и бездомници, особено пък ако съвместяват всичко това накуп. Затова персоналът не достига. Понякога социалните работници вършат работата през пръсти, не правят достатъчно, за да помогнат на всички.

И все пак в Германия има множество организации и неизброими социални услуги в тази сфера. Има и работеща социална система, която подпомага дейността им. А в България имаме… Юлия Георгиева и Розовата къща.

Проблемът не е в спецификата на социалната система в България, а по-скоро че тази система е плод на определена представа за човека. Представа, според която за „недостойните“ няма място в обществото.

С други думи, на първо място проблемът е хуманистичен, а не структурен.

2022 H2 IRAP report is now available on AWS Artifact for Australian customers

Post Syndicated from Patrick Chang original https://aws.amazon.com/blogs/security/2022-h2-irap-report-is-now-available-on-aws-artifact-for-australian-customers/

Amazon Web Services (AWS) is excited to announce that a new Information Security Registered Assessors Program (IRAP) report (2022 H2) is now available through AWS Artifact. An independent Australian Signals Directorate (ASD) certified IRAP assessor completed the IRAP assessment of AWS in December 2022.

The new IRAP report includes an additional six AWS services, as well as the new AWS Melbourne Region, that are now assessed at the PROTECTED level under IRAP. This brings the total number of services assessed at the PROTECTED level to 139.

The following are the six newly assessed services:

For the full list of services, see the IRAP tab on the AWS Services in Scope by Compliance Program page.

AWS has developed an IRAP documentation pack to assist Australian government agencies and their partners to plan, architect, and assess risk for their workloads when they use AWS Cloud services.

We developed this pack in accordance with the Australian Cyber Security Centre (ACSC) Cloud Security Guidance and Anatomy of a Cloud Assessment and Authorisation framework, which addresses guidance within the Australian Government Information Security Manual (ISM), the Attorney-General’s Protective Security Policy Framework (PSPF), and the Digital Transformation Agency Secure Cloud Strategy.

The IRAP pack on AWS Artifact also includes newly updated versions of the AWS Consumer Guide and the whitepaper Reference Architectures for ISM PROTECTED Workloads in the AWS Cloud.

Reach out to your AWS representatives to let us know which additional services you would like to see in scope for upcoming IRAP assessments. We strive to bring more services into scope at the PROTECTED level under IRAP to support your requirements.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

Patrick Chang

Patrick Chang

Patrick is the APJ Audit Lead based in Hong Kong. He leads security audits, certifications and compliance programs across the APJ region. He is a technology risk and audit professional with over a decade of experience. He is passionate about delivering assurance programs that build trust with customers and provide them assurance on cloud security.

Data ingestion pipeline with Operation Management

Post Syndicated from Netflix Technology Blog original https://netflixtechblog.com/data-ingestion-pipeline-with-operation-management-3c5c638740a8

by Varun Sekhri, Meenakshi Jindal, Burak Bacioglu

Introduction

At Netflix, to promote and recommend the content to users in the best possible way there are many Media Algorithm teams which work hand in hand with content creators and editors. Several of these algorithms aim to improve different manual workflows so that we show the personalized promotional image, trailer or the show to the user.

These media focused machine learning algorithms as well as other teams generate a lot of data from the media files, which we described in our previous blog, are stored as annotations in Marken. We designed a unique concept called Annotation Operations which allows teams to create data pipelines and easily write annotations without worrying about access patterns of their data from different applications.

Goals

Annotation Operations

Lets pick an example use case of identifying objects (like trees, cars etc.) in a video file. As described in the above picture

  • During the first run of the algorithm it identified 500 objects in a particular Video file. These 500 objects were stored as annotations of a specific schema type, let’s say Objects, in Marken.
  • The Algorithm team improved their algorithm. Now when we re-ran the algorithm on the same video file it created 600 annotations of schema type Objects and stored them in our service.

Notice that we cannot update the annotations from previous runs because we don’t know how many annotations a new algorithm run will result into. It is also very expensive for us to keep track of which annotation needs to be updated.

The goal is that when the consumer comes and searches for annotations of type Objects for the given video file then the following should happen.

  • Before Algo run 1, if they search they should not find anything.
  • After the completion of Algo run 1, the query should find the first set of 500 annotations.
  • During the time when Algo run 2 was creating the set of 600 annotations, clients search should still return the older 500 annotations.
  • When all of the 600 annotations are successfully created, they should replace the older set of 500.
  • So now when clients search annotations for Objects then they should get 600 annotations.

Does this remind you of something? This seems very similar (not exactly same) to a distributed transaction.

Typically, an algorithm run can have 2k-5k annotations. There are many naive solutions possible for this problem for example:

  • Write different runs in different databases. This is obviously very expensive.
  • Write algo runs into files. But we cannot search or present low latency retrievals from files
  • Etc.

Instead our challenge was to implement this feature on top of Cassandra and ElasticSearch databases because that’s what Marken uses. The solution which we present in this blog is not limited to annotations and can be used for any other domain which uses ES and Cassandra as well.

Marken Architecture

Marken’s architecture diagram is as follows. We refer the reader to our previous blog article for details. We use Cassandra as a source of truth where we store the annotations while we index annotations in ElasticSearch to provide rich search functionalities.

Marken Architecture

Our goal was to help teams at Netflix to create data pipelines without thinking about how that data is available to the readers or the client teams. Similarly, client teams don’t have to worry about when or how the data is written. This is what we call decoupling producer flows from clients of the data.

Lifecycle of a movie goes through a lot of creative stages. We have many temporary files which are delivered before we get to the final file of the movie. Similarly, a movie has many different languages and each of those languages can have different files delivered. Teams generally want to run algorithms and create annotations using all those media files.

Since algorithms can be run on a different permutations of how the media files are created and delivered we can simplify an algorithm run as follows

  • Annotation Schema Type — identifies the schema for the annotation generated by the Algorithm.
  • Annotation Schema Version — identifies the schema version of the annotation generated by the Algorithm.
  • PivotId — a unique string identifier which identifies the file or method which is used to generate the annotations. This could be the SHA hash of the file or simply the movie Identifier number.

Given above we can describe the data model for an annotation operation as follows.

{
  "annotationOperationKeys": [
    {
      "annotationType": "string",   ❶
      "annotationTypeVersion": “integer”,
      "pivotId": "string",
      "operationNumber": “integer”    ❷
    }
  ],
  "id": "UUID",
  "operationStatus": "STARTED",   ❸
  "isActive": true   ❹
}
  1. We already explained AnnotationType, AnnotationTypeVersion and PivotId above.
  2. OperationNumber is an auto incremented number for each new operation.
  3. OperationStatus — An operation goes through three phases, Started, Finished and Canceled.
  4. IsActive — Whether an operation and its associated annotations are active and searchable.

As you can see from the data model that the producer of an annotation has to choose an AnnotationOperationKey which lets them define how they want UPSERT annotations in an AnnotationOperation. Inside, AnnotationOperationKey the important field is pivotId and how it is generated.

Cassandra Tables

Our source of truth for all objects in Marken in Cassandra. To store Annotation Operations we have the following main tables.

  • AnnotationOperationById — It stores the AnnotationOperations
  • AnnotationIdByAnnotationOperationId — it stores the Ids of all annotations in an operation.

Since Cassandra is NoSql, we have more tables which help us create reverse indices and run admin jobs so that we can scan all annotation operations whenever there is a need.

ElasticSearch

Each annotation in Marken is also indexed in ElasticSearch for powering various searches. To record the relationship between annotation and operation we also index two fields

  • annotationOperationId — The ID of the operation to which this annotation belongs
  • isAnnotationOperationActive — Whether the operation is in an ACTIVE state.

APIs

We provide three APIs to our users. In following sections we describe the APIs and the state management done within the APIs.

StartAnnotationOperation

When this API is called we store the operation with its OperationKey (tuple of annotationType, annotationType Version and pivotId) in our database. This new operation is marked to be in STARTED state. We store all OperationIDs which are in STARTED state in a distributed cache (EVCache) for fast access during searches.

StartAnnotationOperation

UpsertAnnotationsInOperation

Users call this API to upsert the annotations in an Operation. They pass annotations along with the OperationID. We store the annotations and also record the relationship between the annotation IDs and the Operation ID in Cassandra. During this phase operations are in isAnnotationOperationActive = ACTIVE and operationStatus = STARTED state.

Note that typically in one operation run there can be 2K to 5k annotations which can be created. Clients can call this API from many different machines or threads for fast upserts.

UpsertAnnotationsInOperation

FinishAnnotationOperation

Once the annotations have been created in an operation clients call FinishAnnotationOperation which changes following

  • Marks the current operation (let’s say with ID2) to be operationStatus = FINISHED and isAnnotationOperationActive=ACTIVE.
  • We remove the ID2 from the Memcache since it is not in STARTED state.
  • Any previous operation (let’s say with ID1) which was ACTIVE is now marked isAnnotationOperationActive=FALSE in Cassandra.
  • Finally, we call updateByQuery API in ElasticSearch. This API finds all Elasticsearch documents with ID1 and marks isAnnotationOperationActive=FALSE.
FinishAnnotationOperation

Search API

This is the key part for our readers. When a client calls our search API we must exclude

  • any annotations which are from isAnnotationOperationActive=FALSE operations or
  • for which Annotation operations are currently in STARTED state. We do that by excluding the following from all queries in our system.

To achieve above

  1. We add a filter in our ES query to exclude isAnnotationOperationStatus is FALSE.
  2. We query EVCache to find out all operations which are in STARTED state. Then we exclude all those annotations with annotationId found in memcache. Using memcache allows us to keep latencies for our search low (most of our queries are less than 100ms).

Error handling

Cassandra is our source of truth so if an error happens we fail the client call. However, once we commit to Cassandra we must handle Elasticsearch errors. In our experience, all errors have happened when the Elasticsearch database is having some issue. In the above case, we created a retry logic for updateByQuery calls to ElasticSearch. If the call fails we push a message to SQS so we can retry in an automated fashion after some interval.

Future work

In near term, we want to write a high level abstraction single API which can be called by our clients instead of calling three APIs. For example, they can store the annotations in a blob storage like S3 and give us a link to the file as part of the single API.

Data ingestion pipeline with Operation Management was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Supermicro SYS-221H-TNR 2U 4th Gen Intel Xeon Scalable Server Review

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/supermicro-sys-221h-tnr-2u-4th-gen-intel-xeon-scalable-server-review/

In our Supermicro SYS-221H-TNR review we see why the X13 Hyper line with the new 4th Gen Intel Xeon Scalable processors is a big leap forward

The post Supermicro SYS-221H-TNR 2U 4th Gen Intel Xeon Scalable Server Review appeared first on ServeTheHome.

How gaming companies can use Amazon Redshift Serverless to build scalable analytical applications faster and easier

Post Syndicated from Satesh Sonti original https://aws.amazon.com/blogs/big-data/how-gaming-companies-can-use-amazon-redshift-serverless-to-build-scalable-analytical-applications-faster-and-easier/

This post provides guidance on how to build scalable analytical solutions for gaming industry use cases using Amazon Redshift Serverless. It covers how to use a conceptual, logical architecture for some of the most popular gaming industry use cases like event analysis, in-game purchase recommendations, measuring player satisfaction, telemetry data analysis, and more. This post also discusses the art of the possible with newer innovations in AWS services around streaming, machine learning (ML), data sharing, and serverless capabilities.

Our gaming customers tell us that their key business objectives include the following:

  • Increased revenue from in-app purchases
  • High average revenue per user and lifetime value
  • Improved stickiness with better gaming experience
  • Improved event productivity and high ROI

Our gaming customers also tell us that while building analytics solutions, they want the following:

  • Low-code or no-code model – Out-of-the-box solutions are preferred to building customized solutions.
  • Decoupled and scalable – Serverless, auto scaled, and fully managed services are preferred over manually managed services. Each service should be easily replaceable, enhanced with little or no dependency. Solutions should be flexible to scale up and down.
  • Portability to multiple channels – Solutions should be compatible with most of endpoint channels like PC, mobile, and gaming platforms.
  • Flexible and easy to use – The solutions should provide less restrictive, easy-to-access, and ready-to-use data. They should also provide optimal performance with low or no tuning.

Analytics reference architecture for gaming organizations

In this section, we discuss how gaming organizations can use a data hub architecture to address the analytical needs of an enterprise, which requires the same data at multiple levels of granularity and different formats, and is standardized for faster consumption. A data hub is a center of data exchange that constitutes a hub of data repositories and is supported by data engineering, data governance, security, and monitoring services.

A data hub contains data at multiple levels of granularity and is often not integrated. It differs from a data lake by offering data that is pre-validated and standardized, allowing for simpler consumption by users. Data hubs and data lakes can coexist in an organization, complementing each other. Data hubs are more focused around enabling businesses to consume standardized data quickly and easily. Data lakes are more focused around storing and maintaining all the data in an organization in one place. And unlike data warehouses, which are primarily analytical stores, a data hub is a combination of all types of repositories—analytical, transactional, operational, reference, and data I/O services, along with governance processes. A data warehouse is one of the components in a data hub.

The following diagram is a conceptual analytics data hub reference architecture. This architecture resembles a hub-and-spoke approach. Data repositories represent the hub. External processes are the spokes feeding data to and from the hub. This reference architecture partly combines a data hub and data lake to enable comprehensive analytics services.

Let’s look at the components of the architecture in more detail.

Sources

Data can be loaded from multiple sources, such as systems of record, data generated from applications, operational data stores, enterprise-wide reference data and metadata, data from vendors and partners, machine-generated data, social sources, and web sources. The source data is usually in either structured or semi-structured formats, which are highly and loosely formatted, respectively.

Data inbound

This section consists of components to process and load the data from multiple sources into data repositories. It can be in batch mode, continuous, pub/sub, or any other
custom integration. ETL (extract, transform, and load) technologies, streaming services, APIs, and data exchange interfaces are the core components of this pillar. Unlike ingestion processes, data can be transformed as per business rules before loading. You can apply technical or business data quality rules and load raw data as well. Essentially, it provides the flexibility to get the data into repositories in its most usable form.

Data repositories

This section consists of a group of data stores, which includes data warehouses, transactional or operational data stores, reference data stores, domain data stores housing purpose-built business views, and enterprise datasets (file storage). The file storage component is usually a common component between a data hub and a data lake to avoid data duplication and provide comprehensiveness. Data can also be shared among all these repositories without physically moving with features, such as data sharing and federated queries. However, data copy and duplication are allowed considering various consumption needs in terms of formats and latency.

Data outbound

Data is often consumed using structured queries for analytical needs. Also, datasets are accessed for ML, data exporting, and publishing needs. This section consists of components to query the data, export, exchange, and APIs. In terms of implementation, the same technologies may be used for both inbound and outbound, but the functions are different. However, it’s not mandatory to use the same technologies. These processes aren’t transformation heavy because the data is already standardized and almost ready to consume. The focus is on the ease of consumption and integration with consuming services.

Consumption

This pillar consists of various consumption channels for enterprise analytical needs. It includes business intelligence (BI) users, canned and interactive reports, dashboards, data science workloads, Internet of Things (IoT), web apps, and third-party data consumers. Popular consumption entities in many organizations are queries, reports, and data science workloads. Because there are multiple data stores maintaining data at different granularity and formats to service consumer needs, these consumption components depend on data catalogs for finding the right source.

Data governance

Data governance is key to the success of a data hub reference architecture. It constitutes components like metadata management, data quality, lineage, masking, and stewardship, which are required for organized maintenance of the data hub. Metadata management helps organize the technical and business metadata catalog, and consumers can reference this catalog to know what data is available in which repository and at what granularity, format, owners, refresh frequency, and so on. Along with metadata management, data quality is important to increase confidence for consumers. This includes data cleansing, validation, conformance, and data controls.

Security and monitoring

Users and application access should be controlled at multiple levels. It starts with authentication, then authorizing who and what should be accessed, policy management, encryption, and applying data compliance rules. It also includes monitoring components to log the activity for auditing and analysis.

Analytics data hub solution architecture on AWS

The following reference architecture provides an AWS stack for the solution components.

Let’s look at each component again and the relevant AWS services.

Data inbound services

AWS Glue and Amazon EMR services are ideal for batch processing. They scale automatically and are able to process most of the industry standard data formats. Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon Managed Streaming for Apache Kafka (Amazon MSK) enables you to build streaming process applications. These streaming services integrate well with the Amazon Redshift streaming feature. This helps you process real-time sources, IoT data, and data from online channels. You can also ingest data with third-party tools like Informatica, dbt, and Matallion.

You can build RESTful APIs and WebSocket APIs using Amazon API Gateway and AWS Lambda, which will enable real-time two-way communication with web sources, social, and IoT sources. AWS Data Exchange helps with subscribing to third-party data in AWS Marketplace. Data subscription and access is fully managed with this service. Refer to the respective service documentation for further details.

Data repository services

Amazon Redshift is the recommended data storage service for OLAP (Online Analytical Processing) workloads such as cloud data warehouses, data marts, and other analytical data stores. This service is the core of this reference architecture on AWS and can address most analytical needs out of the box. You can use simple SQL to analyze structured and semi-structured data across data warehouses, data marts, operational databases, and data lakes to deliver the best price performance at any scale. The Amazon Redshift data sharing feature provides instant, granular, and high-performance access without data copies and data movement across multiple Amazon Redshift data warehouses in the same or different AWS accounts, and across Regions.

For ease of use, Amazon Redshift offers a serverless option. Amazon Redshift Serverless automatically provisions and intelligently scales data warehouse capacity to deliver fast performance for even the most demanding and unpredictable workloads, and you pay only for what you use. Just load your data and start querying right away in Amazon Redshift Query Editor or in your favorite BI tool and continue to enjoy the best price performance and familiar SQL features in an easy-to-use, zero administration environment.

Amazon Relational Database Service (Amazon RDS) is a fully managed service for building transactional and operational data stores. You can choose from many popular engines such as MySQL, PostgreSQL, MariaDB, Oracle, and SQL Server. With the Amazon Redshift federated query feature, you can query transactional and operational data in place without moving the data. The federated query feature currently supports Amazon RDS for PostgreSQL, Amazon Aurora PostgreSQL-Compatible Edition, Amazon RDS for MySQL, and Amazon Aurora MySQL-Compatible Edition.

Amazon Simple Storage Service (Amazon S3) is the recommended service for multi-format storage layers in the architecture. It offers industry-leading scalability, data availability, security, and performance. Organizations typically store data in Amazon S3 using open file formats. Open file formats enable analysis of the same Amazon S3 data using multiple processing and consumption layer components. Data in Amazon S3 can be easily queried in place using SQL with Amazon Redshift Spectrum. It helps you query and retrieve structured and semi-structured data from files in Amazon S3 without having to load the data. Multiple Amazon Redshift data warehouses can concurrently query the same datasets in Amazon S3 without the need to make copies of the data for each data warehouse.

Data outbound services

Amazon Redshift comes with the web-based analytics workbench Query Editor V2.0, which helps you run queries, explore data, create SQL notebooks, and collaborate on data with your teams in SQL through a common interface. AWS Transfer Family helps securely transfer files using SFTP, FTPS, FTP, and AS2 protocols. It supports thousands of concurrent users and is a fully managed, low-code service. Similar to inbound processes, you can utilize Amazon API Gateway and AWS Lambda for data pull using the Amazon Redshift Data API. And AWS Data Exchange helps publish your data to third parties for consumption through AWS Marketplace.

Consumption services

Amazon QuickSight is the recommended service for creating reports and dashboards. It enables you to create interactive dashboards, visualizations, and advanced analytics with ML insights. Amazon SageMaker is the ML platform for all your data science workload needs. It helps you build, train, and deploy models consuming the data from repositories in the data hub. You can use Amazon front-end web and mobile services and AWS IoT services to build web, mobile, and IoT endpoint applications to consume data out of the data hub.

Data governance services

The AWS Glue Data Catalog and AWS Lake Formation are the core data governance services AWS currently offers. These services help manage metadata centrally for all the data repositories and manage access controls. They also help with data classification and can automatically handle schema changes. You can use Amazon DataZone to discover and share data at scale across organizational boundaries with built-in governance and access controls. AWS is investing in this space to provide more a unified experience for AWS services. There are many partner products such as Collibra, Alation, Amorphic, Informatica, and more, which you can use as well for data governance functions with AWS services.

Security and monitoring services

AWS Identity and Access Management (AWS IAM) manages identities for AWS services and resources. You can define users, groups, roles, and policies for fine-grained access management of your workforce and workloads. AWS Key Management Service (AWS KMS) manages AWS keys or customer managed keys for your applications. Amazon CloudWatch and AWS CloudTrail help provide monitoring and auditing capabilities. You can collect metrics and events and analyze them for operational efficiency.

In this post, we’ve discussed the most common AWS services for the respective solution components. However, you aren’t limited to only these services. There are many other AWS services for specific use cases that may be more appropriate for your needs than what we discussed here. You can reach to AWS Analytics Solutions Architects for appropriate guidance.

Example architectures for gaming use cases

In this section, we discuss example architectures for two gaming use cases.

Game event analysis

In-game events (also called timed or live events) encourage player engagement through excitement and anticipation. Events entice players to interact with the game, increasing player satisfaction and revenue with in-game purchases. Events have become more and more important, especially as games shift from being static pieces of entertainment to be played as is to offering dynamic and changing content through the use of services that use information to make decisions about game play as the game is being played. This enables games to change as the players play and influence what works and what doesn’t, and gives any game a potentially infinite lifespan.

This capability of in-game events to offer fresh content and activities within a familiar framework is how you keep players engaged and playing for months to years. Players can enjoy new experiences and challenges within the familiar framework or world that they have grown to love.

The following example shows how such an architecture might appear, including changes to support various sections of the process like breaking the data into separate containers to accommodate scalability, charge-back, and ownership.

To fully understand how events are viewed by the players and to make decisions about future events requires information on how the latest event was actually performed. This means gathering a lot of data as the players play to build key performance indicators (KPIs) that measure the effectiveness and player satisfaction with each event. This requires analytics that specifically measure each event and capture, analyze, report on, and measure player experience for each event. These KPIs include the following:

  • Initial user flow interactions – What actions users are taking after they first receive or download an event update in a game. Are there any clear drop-off points or bottlenecks that are turning people off the event?
  • Monetization – When, what, and where users are spending money on in the event, whether it’s buying in-game currencies, answering ads, specials, and so on.
  • Game economy – How can users earn and spend virtual currencies or goods during an event, using in-game money, trades, or barter.
  • In-game activity – Player wins, losses, leveling up, competition wins, or player achievements within the event.
  • User to user interactions – Invitations, gifting, chats (private and group), challenges, and so on during an event.

These are just some of the KPIs and metrics that are key for predictive modeling of events as the game acquires new players while keeping existing users involved, engaged, and playing.

In-game activity analysis

In-game activity analysis essentially looks at any meaningful, purposeful activity the player might show, with the goal of trying to understand what actions are taken, their timing, and outcomes. This includes situational information about the players, including where they are playing (both geographical and cultural), how often, how long, what they undertake on each login, and other activities.

The following example shows how such an architecture might appear, including changes to support various sections of the process like breaking the data into separate warehouses. The multi-cluster warehouse approach helps scale the workload independently, provides flexibility to the implemented charge-back model, and supports decentralized data ownership.

The solution essentially logs information to help understand the behavior of your players, which can lead to insights that increase retention of existing players, and acquisition of new ones. This can provide the ability to do the following:

  • Provide in-game purchase recommendations
  • Measure player trends in the short term and over time
  • Plan events the players will engage in
  • Understand what parts of your game are most successful and which are less so

You can use this understanding to make decisions about future game updates, make in-game purchase recommendations, determine when and how your game economy may need to be balanced, and even allow players to change their character or play as the game progresses by injecting this information and accompanying decisions back into the game.

Conclusion

This reference architecture, while showing examples of only a few analysis types, provides a faster technology path for enabling game analytics applications. The decoupled, hub/spoke approach brings the agility and flexibility to implement different approaches to analytics and understanding the performance of game applications. The purpose-built AWS services described in this architecture provide comprehensive capabilities to easily collect, store, measure, analyze, and report game and event metrics. This helps you efficiently perform in-game analytics, event analysis, measure player satisfaction, and provide tailor-made recommendations to game players, efficiently organize events, and increase retention rates.

Thanks for reading the post. If you have any feedback or questions, please leave them in the comments.

About the authors

Satesh Sonti is a Sr. Analytics Specialist Solutions Architect based out of Atlanta, specialized in building enterprise data platforms, data warehousing, and analytics solutions. He has over 16 years of experience in building data assets and leading complex data platform programs for banking and insurance clients across the globe.

Tanya Rhodes is a Senior Solutions Architect based out of San Francisco, focused on games customers with emphasis on analytics, scaling, and performance enhancement of games and supporting systems. She has over 25 years of experience in enterprise and solutions architecture specializing in very large business organizations across multiple lines of business including games, banking, healthcare, higher education, and state governments.

Improve productivity by using keyboard shortcuts in Amazon Athena query editor

Post Syndicated from Naresh Gautam original https://aws.amazon.com/blogs/big-data/improve-productivity-by-using-keyboard-shortcuts-in-amazon-athena-query-editor/

Amazon Athena is a serverless, interactive analytics service built on open-source frameworks, supporting open-table and file formats. Athena provides a simplified, flexible way to analyze petabytes of data where it lives. You can analyze data or build applications from an Amazon Simple Storage Service (Amazon S3) data lake and over 25 data sources, including on-premises data sources or other cloud systems using SQL or Python. Athena is built on open-source Trino and Presto engines and Apache Spark frameworks, with no provisioning or configuration effort required.

Different types of users rely on Athena, including business analysts, data scientists, security, and operations engineers. Athena provides a query editor to enter and run queries on data using structured query language (SQL). The query editor provides features like run, cancel, and save queries or statements. Additionally, it provides keyboard shortcuts for user-friendly operation.

This post discusses the keyboard shortcuts available and how you can use them.

Accessing the Athena console

If you’re new to Athena and don’t know how to access the Athena console and run queries and statements, refer to the following getting started tutorial. This tutorial walks you through using Athena to query data. You’ll create a table based on sample data stored in Amazon S3, query the table, and check the results of the query.

Keyboard shortcuts

The query editor provides keyboard shortcuts for different action types like running a query, formatting a query, line operations, selection, multi-cursor, go to, find/replace, and folding. Compared to reaching for the mouse or navigating a menu, a single keyboard shortcut saves a moment of your time.

With keyboard shortcuts, you can use key combinations to edit your SQL statement without using a mouse. For example, you can use multiple cursors in your editing window for selecting all instances of text you wish to edit, and edit your text, fold or unfold selected text, find and replace text, and perform line operations like remove line, move lines, and more.

You can also find these keyboard shortcuts on the query editor on the bottom right corner, as highlighted in the following screenshot.

The following table shows the keyboards shortcuts for Window/Linux and Mac.

Action Type Action Windows/Linux Mac
Other Execute query Ctrl-Enter Cmd-Enter, Ctrl-Enter
Other Format query Ctrl-Alt-L Opt-Cmd-L
Other Previous query Ctrl-Up Ctrl-Shift-Up
Other Next query Ctrl-Down Ctrl-Shift-Down
Other Close tab Alt-X Opt-X
Other Previous tab Ctrl-, Ctrl-,
Other Next tab Ctrl-. Ctrl-.
Other Indent Tab Tab
Other Outdent Shift-Tab Shift-Tab
Other Save Ctrl-S Cmd-S
Other Undo Ctrl-Z Cmd-Z
Other Redo Ctrl-Shift-Z, Ctrl-Y Cmd-Shift-Z, Cmd-Y
Other Toggle comment Ctrl-/ Cmd-/
Other Transpose letters Ctrl-T Ctrl-T
Other Change to lower case Ctrl-Shift-U Ctrl-Shift-U
Other Change to upper case Ctrl-U Ctrl-U
Other Overwrite Insert Insert
Other Delete Delete –
Line Operations Remove line Ctrl-D Cmd-D
Line Operations Copy lines down Alt-Shift-Down Cmd-Opt-Down
Line Operations Copy lines up Alt-Shift-Up Cmd-Opt-Up
Line Operations Move lines down Alt-Down Opt-Down
Line Operations Move lines up Alt-Up Opt-Up
Line Operations Remove to line end Alt-Delete Ctrl-K
Line Operations Remove to line start Alt-Backspace Cmd-Backspace
Line Operations Remove word left Ctrl-Backspace Opt-Backspace, Ctrl-Opt-Backspace
Line Operations Remove word right Ctrl-Delete Opt-Delete
Line Operations Split line – Ctrl-O
Selection Select all Ctrl-A Cmd-A
Selection Select left Shift-Left Shift-Left
Selection Select right Shift-Right Shift-Right
Selection Select word left Ctrl-Shift-Left Opt-Shift-Left
Selection Select word right Ctrl-Shift-Right Opt-Shift-Right
Selection Select line start Shift-Home Shift-Home
Selection Select line end Shift-End Shift-End
Selection Select to line end Alt-Shift-Right Cmd-Shift-Right
Selection Select to line start Alt-Shift-Left Cmd-Shift-Left
Selection Select up Shift-Up Shift-Up
Selection Select down Shift-Down Shift-Down
Selection Select page up Shift-PageUp Shift-PageUp
Selection Select page down Shift-PageDown Shift-PageDown
Selection Select to start Ctrl-Shift-Home Cmd-Shift-Up
Selection Select to end Ctrl-Shift-End Cmd-Shift-Down
Selection Duplicate selection Ctrl-Shift-D Cmd-Shift-D
Selection Select to matching bracket Ctrl-Shift-P –
Multicursor Add multi-cursor above Ctrl-Alt-Up Ctrl-Opt-Up
Multicursor Add multi-cursor below Ctrl-Alt-Down Ctrl-Opt-Down
Multicursor Add next occurrence to multi-selection Ctrl-Alt-Right Ctrl-Opt-Right
Multicursor Add previous occurrence to multi-selection Ctrl-Alt-Left Ctrl-Opt-Left
Multicursor Move multi-cursor from current line to the line above Ctrl-Alt-Shift-Up Ctrl-Opt-Shift-Up
Multicursor Move multi-cursor from current line to the line below Ctrl-Alt-Shift-Down Ctrl-Opt-Shift-Down
Multicursor Remove current occurrence from multi-selection and move to next Ctrl-Alt-Shift-Right Ctrl-Opt-Shift-Right
Multicursor Remove current occurrence from multi-selection and move to previous Ctrl-Alt-Shift-Left Ctrl-Opt-Shift-Left
Multicursor Select all from multi-selection Ctrl-Shift-L Ctrl-Shift-L
Go to Go to left Left Left, Ctrl-B
Go to Go to right Right Right, Ctrl-F
Go to Go to word left Ctrl-Left Opt-Left
Go to Go to word right Ctrl-Right Opt-Right
Go to Go line up Up Up, Ctrl-P
Go to Go line down Down Down, Ctrl-N
Go to Go to line start Alt-Left, Home Cmd-Left, Home, Ctrl-A
Go to Go to line end Alt-Right, End Cmd-Right, End, Ctrl-E
Go to Go to page up PageUp Opt-PageUp
Go to Go to page down PageDown Opt-PageDown, Ctrl-V
Go to Go to start Ctrl-Home Cmd-Home, Cmd-Up
Go to Go to end Ctrl-End Cmd-End, Cmd-Down
Go to Scroll line down Ctrl-Down Cmd-Down
Go to Scroll line up Ctrl-Up –
Go to Go to matching bracket Ctrl-P –
Go to Scroll page down – Opt-PageDown
Go to Scroll page up – Opt-PageUp
Find/Replace Find Ctrl-F Cmd-F
Find/Replace Replace Ctrl-H Cmd-Opt-F
Find/Replace Find next Ctrl-K Cmd-G
Find/Replace Find previous Ctrl-Shift-K Cmd-Shift-G
Folding Fold selection Alt-L, Ctrl-F1 Cmd-Opt-L, Cmd-F1
Folding Unfold Alt-Shift-L, Ctrl-Shift-F1 Cmd-Opt-Shift-L, Cmd-Shift-F1
Folding Unfold all Alt-Shift-0 Cmd-Opt-Shift-0
Other Autocomplete Ctrl-Space Ctrl-Space
Other Focus out Esc Esc

For illustration, you can perform the Format query action by using the keyboard shortcut (Ctrl-Alt-L for Windows/Linux, Opt-Cmd-L for Mac). It converts unformatted SQL to a well-formatted SQL, as shown in the following screenshots.

Similarly, you can try out the Toggle comment command (Ctrl-/ for Windows/Linux, Cmd-/ for Mac) to comment or uncomment lines of SQL in the Athena query editor. This comes in very handy when you want to quickly comment out specific lines in your query, as shown in the following screenshots.

You can do line operations like Remove line, Copy lines down, Copy lines up, and more. The following screenshots show an example of the Remove line action (Ctrl-D for Windows/Linux, Cmd-D for Mac).

You can do a line selection like Select all, Select left, Select line start, and more. The following screenshots show an example the Select all action (Ctrl-A for Windows/Linux, Cmd-A for Mac).

You can do multi-cursor actions like Add multi-cursor above, Add multi-cursor below, Add next occurrence to multi-selection, Add previous occurrence to multi-selection, Move multi-cursor from current line to the line above, and more. The following example is of the Add multi-cursor above action (Ctrl-Alt-Up for Windows/Linux, Ctrl-Opt-Up for Mac).

You can do go to actions like Go to left, Go to right, Go to word left, and more. The following is an example of the Go to left action (Ctrl-B).

You can do find and replace actions like Find, Replace, Find next, and more. The following is an example of the Replace action (Ctrl-H for Windows/Linux, Cmd-Opt-F for Mac).

You can also do folding actions like Fold selection, Unfold, and Unfold all. The following example is of the Unfold action (Alt-Shift-L or Ctrl-Shift-F1 for Windows/Linux, Cmd-Opt-Shift-L or Cmd-Shift-F1 for Mac).

Conclusion

In this post, we saw how Athena provides an array of native options to help you improve productivity when analyzing your data. You can go to the Athena console and start running SQL statements or querying data using the built-in query editor. The query editor provides key shortcuts to improve your productivity by using key combinations to edit SQL statements, instead of using a mouse.

If you have any questions or suggestions, please leave a comment.

About the Authors

Naresh Gautam is a Data Analytics and AI/ML leader at AWS with 20 years of experience, who enjoys helping customers architect highly available, high-performance, and cost-effective data analytics and AI/ML solutions to empower customers with data-driven decision-making. In his free time, he enjoys meditation and cooking.

Srikanth Sopirala is a Principal Analytics Specialist Solutions Architect at AWS. He is a seasoned leader with over 20 years of experience, who is passionate about helping customers build scalable data and analytics solutions to gain timely insights and make critical business decisions. In his spare time, he enjoys reading, spending time with his family, and road biking.

Harsh Vardhan is an AWS Solutions Architect, specializing in analytics. He has over 5 years of experience working in the field of big data and data science. He is passionate about helping customers adopt best practices and discover insights from their data.

AWS Application Composer Now Generally Available – Visually Build Serverless Applications Quickly

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/aws-application-composer-now-generally-available-visually-build-serverless-applications-quickly/

At AWS re:Invent 2022, we previewed AWS Application Composer, a visual builder for you to compose and configure serverless applications from AWS services backed by deployment-ready infrastructure as code (IaC).

In the keynote, Dr. Werner Vogels, CTO of Amazon.com said:

Developers that never used serverless before. How do they know where to start? Which services do they need? How do they work together? We really wanted to make this easier. AWS Application Composer simplifies and accelerates the architecting, configuring, and building of serverless applications.

During the preview, we had lots of interest and great feedback from customers. Today, I am happy to announce the general availability of AWS Application Composer with new improvements based on customer feedback. I want to quickly review its features and introduce some improvements.

Introduction to AWS Application Composer
To get started with AWS Application Composer, choose Open demo in the AWS Management Console. This demo shows a simple cart application with Amazon API Gateway, AWS Lambda, and Amazon DynamoDB resources.

You can easily browse and search for AWS services in the left Resources panel and drag and drop them onto the canvas to expand your architecture.

In the middle Canvas panel, you can connect resources together by clicking and dragging from one resource port to another. Permissions are automatically composed for these resources to interact with each other using policy template, environment variables, and event subscriptions. Grouping resources is very useful to select one visual organization. For above example, API Compute group is compsite of Lambda functions. When you double-click on a specific resource, you can name and configure your properties in the right Resource properties panel.

As well as featured resources available in the visual resource palette, you can use hidden and read-only resources will populate on the canvas when you load an existing template that includes them.

In this example, the MyHttpApi resource is a hidden resource. It is not available from the resource palette but does appear on the canvas in color. The resource named MyHttpApiRole (in this case, an AWS::IAM::Role resource) is read-only. It grayed out on the canvas greyed out. To learn more about all supported resources, see AWS Application Composer featured resources in the AWS documentation.

When you select the Template menu, you can view, edit or manually download your IaC, such as AWS Serverless Application Model (AWS SAM). Your changes are automatically synced with your canvas.

When you start Connected mode, you can use Application Composer with local tools such as an integrated development environment (IDE). Any changes activate the automatic synchronization of your project template and files between Application Composer and your local project directory.

It is useful to incorporate into your existing team processes, such as local testing with AWS SAM Command Line Interface (CLI), peer review through version control, or deployment through AWS CloudFormation and continuous integration and delivery (CI/CD) pipelines.

This mode is supported on Chrome and Edge browsers and requires you to grant temporary local file system access to your browser.

AWS Application Composer can be used in real-world scenarios such as:

  • Building a prototype of serverless applications
  • Reviewing and collaboratively evolving existing serverless projects
  • Generating diagrams for documentation or Wikis
  • Onboarding new team members to a project
  • Reducing the first steps to deploy something in an AWS account

To learn more real-world examples, see Visualize and create your serverless workloads with AWS Application Composer in the AWS Compute Blog, How I Used AWS Application Composer to Make Analyzing My Meetup Data Easy in BuildOn.AWS, or watch a breakout session video (SVS211) from AWS re:Invent 2022.

Improvements Since Preview Launch
Here is a new feature to improve how you work with Amazon Simple Queue Service (Amazon SQS) queues.

You can now directly connect Amazon API Gateway resources to Amazon SQS without routing requests through AWS Lambda function. You can remove the complexity of the Lambda function’s execution and increase the reliability while reducing lines of code.

For example, you can drag API Gateway and Amazon SQS onto the canvas and connect the two resources. When the user drags the connector from API route to SQS, Send message appears. You can connect the API route to the SQS queue via their choice of integration target.

The new Change Inspector provides a visual diff of template changes made when you connect two resources on the canvas. This information is available as a notification when you make the connection, which helps you understand how Composer manages integration configuration in your IaC template as you build.

Here are some more improvements to your experience in the user interface!

First, we reduced the size of resource cards. The larger cards made it difficult for the users to read and view their template on the canvas. Now, you can arrange more resource cards easily and save space on the canvas.

Also, we added zoom in and out and zoom to fit buttons so that users can quickly view the entire screen or zoom to the desired level. When you load a large template onto the canvas, you can easily see all the resource cards in any size.

Now Available
AWS Application Composer is now generally available in the US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm) Regions, adding three more Regions to the six Regions available during preview. There is no additional cost, and you can start using it today.

To learn more, see the AWS Application Composer Developer Guide and send feedback to AWS re:Post for AWS Application Composer or through your usual AWS support contacts.

– Channy

New Zero Trust navigation coming soon (and we need your feedback)

Post Syndicated from Emily Flannery original https://blog.cloudflare.com/zero-trust-navigation/

New Zero Trust navigation coming soon (and we need your feedback)

We’re updating the Zero Trust navigation

New Zero Trust navigation coming soon (and we need your feedback)

On March 20, 2023, we will be launching an updated navigation in the Zero Trust dashboard, offering all of our Zero Trust users a more seamless experience across Cloudflare as a whole. This change will allow you to more easily manage your Zero Trust organization alongside your application and network services, developer tools, and more.

As part of this upcoming release, you will see three key changes:

Quicker navigation

Instead of opening another window or typing in a URL, you can go back to the Cloudflare dashboard in one click.

New Zero Trust navigation coming soon (and we need your feedback)

Switch accounts with ease

View and switch accounts at the top of your sidebar.

New Zero Trust navigation coming soon (and we need your feedback)

Resources and support

Find helpful links to our Community, developer documentation, and support team at the top of your navigation bar.

New Zero Trust navigation coming soon (and we need your feedback)

Why we’re updating the Zero Trust navigation

In 2020, Gateway was broadly released as the first Cloudflare product that didn’t require a site hosted on Cloudflare’s infrastructure. In other words, Gateway was unconstrained by the site-specific model most other Cloudflare products relied on at the time, while also used in close conjunction with Access. And so, the Cloudflare for Teams dashboard was built on a new model, designed from scratch, to give our customers a designated home—consolidated under a single roof—to manage their Teams products and accounts.

Fast forward to today and Zero Trust has grown tremendously, both in capability and reach. Many of our customers are using multiple Cloudflare products together, including Cloudflare One and Zero Trust products. Our home has grown, and this navigation change is one step toward expanding our roof to cover Cloudflare’s rapidly expanding footprint.

A focus on user experience

We have heard from many of you about the pains you experience when using multiple Cloudflare products, including Zero Trust. Your voice matters to us, and we’re invested in building a world-class user experience to make your time with Cloudflare an easy and enjoyable one. Our user experience improvements are based on three core principles: Consistency, Interconnectivity, and Discoverability.

We aim to offer a consistent and predictable user experience across the entire Cloudflare ecosystem so you never have to think twice about where you are in your journey, whether performing your familiar daily tasks or discovering our new ground-breaking products and features.

What else?

This navigation change we’re announcing today isn’t the only user experience improvement we’ve built! You may have noticed a few more optimizations recently:

User authorization and loading experience

Remember the days of the recurrent loading screen? Or perhaps when your Zero Trust account didn’t match the one you had logged in with to manage, say, your DNS? Those days are over! Our team has built a smarter, faster, and more seamless user and account authorization experience.

New tables

Tables are table stakes when it comes to presenting large quantities of data and information. (Yes, pun intended.) Tables are a common UI element across Cloudflare, and now Zero Trust uses the same tables UI as you will see when managing other products and features.

UI consistency

A slight change in color scheme and page layout brings the Zero Trust dashboard into the same visual family as the broader Cloudflare experience. Now, when you navigate to Zero Trust, we want you to know that you’re still under our one single Cloudflare roof.

We’re as excited about these improvements as you are! And we hope the upcoming navigation and page improvements come as a welcome addition to the changes noted above.

What’s next?

The user experience changes we’ve covered today go a long way toward creating a more consistent, seamless and user-friendly interface to make your work on Cloudflare as easy and efficient as possible. We know there’s always room for further improvement (we already have quite a few big improvements on our radar!).

To ensure we’re solving your biggest problems, we’d like to hear from you. Please consider filling out a short survey to share the most pressing user experience improvements you’d like to see next.

[$] BTHome: An open standard for broadcasting sensor data

Post Syndicated from original https://lwn.net/Articles/925125/

Many wireless sensors broadcast their data using Bluetooth Low
Energy (BLE). Their data is easy to receive, but decoding it can be a
challenge. Each manufacturer uses its own format, often tied to its own
mobile apps. Integrating all of these sensors into a home-automation system
requires a lot of custom decoders, which are generally developed by
reverse-engineering
the protocols. The goal of the BTHome
project is to change this: it offers a standardized format for sensors to
broadcast their measurements using BLE. BTHome is supported by the Home
Assistant
home-automation software and by a few open-firmware and open-hardware
projects.

Subscribe to AWS Daily Feature Updates via Amazon SNS

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/subscribe-to-aws-daily-feature-updates-via-amazon-sns/

Way back in 2015 I showed you how to Subscribe to AWS Public IP Address Changes via Amazon SNS. Today I am happy to tell you that you can now receive timely, detailed information about releases and updates to AWS via the same, simple mechanism.

Daily Feature Updates
Simply subscribe to topic arn:aws:sns:us-east-1:692768080016:aws-new-feature-updates using the email protocol and confirm the subscription in the usual way:

You will receive daily emails that start off like this, with an introduction and a summary of the update:

After the introduction, the email contains a JSON representation of the daily feature updates:

As noted in the message, the JSON content is also available online at URLs that look like https://aws-new-features.s3.us-east-1.amazonaws.com/update/2023-02-27.json . You can also edit the date in the URL to access historical data going back up to six months.

The email message also includes detailed information about changes and additions to managed policies that will be of particular interest to AWS customers who currently manually track and then verify the impact that these changes may have on their security profile. Here’s a sample list of changes (additional permissions) to existing managed policies:

And here’s a new managed policy:

Even More Information
The header of the email contains a link to a treasure trove of additional information. Here are some examples:

AWS Regions and AWS Services – A pair of tables. The first one includes a row for each AWS Region and a column for each service, and the second one contains the transposed version:

AWS Regions and EC2 Instance Types – Again, a pair of tables. The first one includes a row for each AWS Region and a column for each EC2 instance type, and the second one contains the transposed version:

The EC2 Instance Types Configuration link leads to detailed information about each instance type:

Each page also includes a link to the same information in JSON form. For example (EC2 Instance Types Configuration), starts like this:

{
    "a1.2xlarge": {
        "af-south-1": "-",
        "ap-east-1": "-",
        "ap-northeast-1": "a1.2xlarge",
        "ap-northeast-2": "-",
        "ap-northeast-3": "-",
        "ap-south-1": "a1.2xlarge",
        "ap-south-2": "-",
        "ap-southeast-1": "a1.2xlarge",
        "ap-southeast-2": "a1.2xlarge",
        "ap-southeast-3": "-",
        "ap-southeast-4": "-",
        "ca-central-1": "-",
        "eu-central-1": "a1.2xlarge",
        "eu-central-2": "-",
        "eu-north-1": "-",
        "eu-south-1": "-",
        "eu-south-2": "-",
        "eu-west-1": "a1.2xlarge",
        "eu-west-2": "-",
        "eu-west-3": "-",
        "me-central-1": "-",
        "me-south-1": "-",
        "sa-east-1": "-",
        "us-east-1": "a1.2xlarge",
        "us-east-2": "a1.2xlarge",
        "us-gov-east-1": "-",
        "us-gov-west-1": "-",
        "us-west-1": "-",
        "us-west-2": "a1.2xlarge"
    },

Other information includes:

  • VPC Endpoints
  • AWS Services Integrated with Service Quotas
  • Amazon SageMaker Instance Types
  • RDS DB Engine Versions
  • Amazon Nimble Instance Types
  • Amazon MSK Apache Kafka Versions

Information Sources
The information is pulled from multiple public sources, cross-checked, and then issued. Here are some of the things that we look for:

Things to Know
Here are a couple of things that you should keep in mind about the AWS Daily Feature Updates:

Content – The content provided in the Daily Feature Updates and in the treasure trove of additional information will continue to grow as new features are added to AWS.

Region Coverage – The Daily Feature Updates cover all AWS Regions in the public partition. Where possible, it also provides information about GovCloud regions; this currently includes EC2 Instance Types, SageMaker Instance Types, and Amazon Nimble Instance Types.

Region Mappings – The internal data that drives all of the information related to AWS Regions is updated once a day if there are applicable new features, and also when new AWS Regions are enabled.

Updates – On days when there are no updates, there will not be an email notification.

Usage – Similar to the updates on the What’s New page and the associated RSS feed, the updates are provided for informational purposes, and you still need to do your own evaluation and testing before deploying to production.

??? –

— Jeff;

The initial posting of the Apple AGX graphics driver

Post Syndicated from original https://lwn.net/Articles/925503/

Asahi Lina has posted an
initial version of a Rust-based driver for Apple AGX graphics
processors; the posting includes a fair amount of Rust infrastructure for
graphics drivers in general.

While developing the driver, I tried to make use of Rust’s safety
and lifetime features to provide not just CPU-side safety, but also
partial firmware-ABI safety. Thanks to this, it has turned out to
be a very stable driver even though GPU firmware crashes are fatal
(no restart capability, need to reboot!) and the FW/driver
interface is a huge mess of unsafe shared memory structures with
complex pointer chains.

The White House’s National Cybersecurity Strategy asks the private sector to step up to fight cyber attacks. Cloudflare is ready.

Post Syndicated from Zaid Zaid original https://blog.cloudflare.com/the-white-houses-national-cybersecurity-strategy-asks-the-private-sector-to-step-up-to-fight-cyber-attacks-cloudflare-is-ready/

The White House’s National Cybersecurity Strategy asks the private sector to step up to fight cyber attacks. Cloudflare is ready.

The White House’s National Cybersecurity Strategy asks the private sector to step up to fight cyber attacks. Cloudflare is ready.

On Thursday, March 2, 2023, the Biden-Harris Administration released the National Cybersecurity Strategy aimed at securing the Internet. Cloudflare welcomes the Strategy, and congratulates the White House on this comprehensive, much-needed policy initiative. The goal of the Strategy is to make the digital ecosystem defensible, resistant, and values-aligned. This is a goal that Cloudflare fully supports. The Strategy recognizes the vital role that the private sector has to play in defending the United States against cyber attacks.

The Strategy aims to make a fundamental shift and transformation of roles, responsibilities, and resources in cyberspace by (1) rebalancing the responsibility to defend cyberspace by shifting the burden away from individuals, small businesses, and local governments, and onto organizations that are most capable and best-positioned to reduce risks, like data holders and technology providers; and (2) realigning incentives to favor long-term investments by balancing defending the United States against urgent threats today and simultaneously investing in a resilient future. The Strategy envisions attaining these goals through five collaborative pillars:

  • Pillar One: defending critical infrastructure;
  • Pillar Two: disrupting and dismantling threat actors;
  • Pillar Three: shaping market forces to drive security and resilience;
  • Pillar Four: investing in a resilient future; and
  • Pillar Five: forging international partnerships to pursue shared goals.

Through the Strategy, the U.S. Government is committed to preserving and extending the open, free, global, interoperable, reliable, and secure Internet. Cloudflare shares this commitment, and has built tools and products that are easily deployed and accessible to everyone that help make it a reality. Here are a few things that stand out to us in the Strategy, and how Cloudflare has contributed to the goals we share.

Defending Critical Infrastructure: Shields Up and Zero Trust

Importantly, Pillar One of the Strategy is focused on defending critical infrastructure. Critical infrastructure is vital to the functioning of society, and includes things like gas pipelines, railways, utilities, clean water, hospitals, and electricity, among others. In the aftermath of Russia’s invasion of Ukraine, the United States, the United Kingdom, Japan, and others issued warnings about the increased risk of cyber attacks. There was widespread concern by private sector and government cybersecurity experts about potential retaliation in the United States to the sanctions that resulted from the Russian invasion of Ukraine. In response, the Cybersecurity Infrastructure & Security Agency (CISA) announced its Shields Up initiative. When Shields Up was announced, we wrote about the essential tools that Cloudflare offers – for free – for protecting an online presence. We also published a Shields Up reading list.

One way we responded to the increased risk to critical infrastructure was the Critical Infrastructure Defense Project (CIDP), which we launched in partnership with Crowdstrike and Ping Identity, and offered a broad suite of products for free for four months to any United States-based hospital, or energy or water utility. Thankfully, the retaliation did not materialize at the level experts and officials were expecting. But that does not mean that the fear was not well-founded nor that malicious actors do not continue to have designs on critical infrastructure in the United States or around the world.

In addition to Shields Up, the Strategy doubles down on the Zero Trust Framework to guard against cyber attacks, a strategy first announced by the White House in January 2022 when it instructed federal agencies to move towards Zero Trust cybersecurity principles. These principles are rooted in the fundamental principle of “never trust, always verify;” no one is trusted by default from inside or outside of a network, and verification is required from everyone trying to gain access to resources on the network.

We could not agree more with the US government’s decision to modernize by grounding its federal defenses with Zero Trust principles. Zero Trust is not just a buzzword. Cloudflare has been championing Zero Trust for years, and we think it is so important for cybersecurity that we believe that a Chief Zero Trust Officer will become increasingly common over the next year. And because we know how important Zero Trust tools are, we recently announced that civil society and government participants in Project Galileo and the Athenian Project will have free access to Zero Trust products because we believe that qualified vulnerable public interest organizations should have access to Enterprise-level cyber security products no matter their size and budgets.

Disrupting and dismantling threat actors

Pillar Three of the Strategy is focused on disrupting and dismantling threat actors. As a member of the Joint Cyber Defense Collaborative, Cloudflare partners with the US government and cyber defenders from organizations across the Internet ecosystem to help increase visibility of malicious activity and threats, and drive collective action. Our network is large, learns from each attack, and is global, providing the best defense against attacks. The more we deal with attacks, the more we know how to stop them, and the easier it gets to find and deal with new threats. We block an average of 136 billion cyber threats per day. Just last month, Cloudflare mitigated a record-breaking 71 million request-per-second DDoS attack, the largest reported HTTP DDoS attack on record, more than 54% higher than the previous reported record of 46M rps in June 2022.

Privacy Preserving Technologies

Pillar Four focuses on investing in a resilient future, partly through supporting privacy-preserving technologies. The Internet was not built with privacy and security in mind, but a more private Internet is a better Internet. Even with encryption, information about consumer IP addresses and the names of websites they visit leak from protocols that weren’t designed to preserve privacy. We believe that reducing the availability of that information can help consumers regain control over their data.

Cloudflare has therefore worked to develop technologies to help build a more privacy-preserving Internet. We’ve been working on technologies that encourage and enable website operators and app developers to build privacy into their products at the protocol level. We’ve released or support a number of services that deploy state-of-the-art, privacy-enhancing technologies for DNS and other communications to help individuals, large corporations, small-businesses, and governments alike. These products include: Privacy Gateway, a fully managed, scalable, and performant Oblivious HTTP (OHTTP) relay, which is designed so that Internet Service Providers don’t know the websites their subscribers are visiting, and likewise websites don’t know the true IP address of their visitors; Private Relay, a version of Privacy Gateway that includes a second relay server that conveys data to websites and applications which hides a device’s true IP address; Cloudflare WARP, a free proxy application that encrypts traffic on the user’s device, routes it through the Cloudflare network, and then routes it on to its intended destination; and 1.1.1.1, our free, public Domain Name System (DNS) resolver, which helps make Internet traffic more private.

Preparing for the Post-Quantum Future and Safer Internet Protocols

As part of its goal of investing in a resilient future, one of the Strategic Objectives of the Strategy is to prepare for the post-quantum future whereby the government will increase investment in post-quantum. Likewise, the US government encourages the private sector to prepare its systems for the future. Cloudflare is already prepared, and although quantum computers are a future state, Cloudflare is helping to make sure the Internet is ready for when they arrive. Here and here, we describe the impact of quantum computing on cryptography, and how to use stronger algorithms resistant to the power of quantum computing. In October, we announced that by default, all websites and APIs served through Cloudflare now support post-quantum hybrid key agreement. And because we strongly believe that post-quantum security should be the new baseline for the Internet, we offer this post-quantum cryptography free of charge.

We were happy to see some focus in the Strategy on improving Internet protocols, which are important for ensuring that the Internet is functional, safe, and secure. The Strategy envisions a “clean-up effort” of the technical foundations of the Internet including Border Gateway Protocol (BGP) vulnerabilities, unencrypted DNS, and the slow adoption of IPv6. Cloudflare has been a long time supporter of security and privacy improvements to these foundational protocols, and wholeheartedly endorses this clean up effort. We have written about our support for improving the security of these protocols, including securing BGP through the use of RPKI and improving DNS privacy by launching support for DNS over HTTPS, DNS over TLS and Oblivious DNS over HTTPS.

Building International Partnerships and Assisting Allies and Partners

Pillar 5 of the Strategy commits the United States to forging international partnerships to pursue shared goals. Cyber attacks by their very nature are borderless, which means that protecting against cyber attacks cannot mean only protecting entities within one’s borders. Cyber defense is an international effort, and we cannot preserve and extend the open, free, global, interoperable, reliable and secure Internet if we do not help to defend, as well as build the capacity of, other countries through coalition building. The Strategy aims to assist allies and partners. With the invasion of Ukraine, Cloudflare has directly witnessed the importance of private sector collaboration [link to article] in efforts to assist allies and partners. Cloudflare is proud of the role we have played in helping protect Ukraine from cyberattack, which we described here, here, and here. Another way that we are working to provide support to vulnerable infrastructure outside of the United States is through Project Safekeeping, modeled after CIDP. In December, as part of Impact Week, we announced that we would be providing our enterprise-level Zero Trust cybersecurity solution to eligible entities in Australia, Germany, Japan, Portugal, and the United Kingdom, at no cost, with no time limit.

We again congratulate the White House on the National Cybersecurity Strategy. We have partnered with the US government in the past to help the federal government defend itself against cyberattacks, and we look forward to continuing our collaboration with the US government and other private sector entities for a more safe and secure Internet.

Vulnerability Management vs. Vulnerability Assessment

Post Syndicated from Marla Rosner original https://blog.rapid7.com/2023/03/07/vulnerability-management-vs-vulnerability-assessment/

Evolving networks and evolving threats

Vulnerability Management vs. Vulnerability Assessment

When it comes to protecting your cloud or hybrid networks, what you don’t know can most certainly hurt your enterprise. Today’s NetOps teams are tasked with monitoring the health and performance of both on-premises and cloud applications, as well as software, devices, and instances. As if this wasn’t complicated enough, malicious threat actors relentlessly seek to capitalize on the vulnerabilities in an enterprise’s network.

These attacks affect enterprises across all industries. Recently, Gartner predicted that 45% of global organizations will have experienced attacks on their software supply chains by 2025. Statista also reported that approximately 15M data records were exposed worldwide through data breaches in the third quarter of 2022. This staggering figure represented a quarterly increase of over 37%.

Network attacks are costly, too. In fact, the average cost of a data breach increased to $9.44M in the United States in 2022. Keep in mind, this figure doesn’t include the frustration, lost productivity, and negative impact on brand reputation that often accompany cyber attacks.

Vulnerability assessment (VA) and vulnerability management (VM) are two of the best ways to protect your enterprise against threats, but these terms are often used incorrectly and interchangeably. A better understanding of these concepts and how they relate to one another can help you significantly boost the security posture of your hybrid and cloud environments.

What is a vulnerability assessment?

TechTarget defines vulnerability assessment as “the process of defining, identifying, classifying and prioritizing vulnerabilities in computer systems, applications and network infrastructures.” These vulnerabilities usually fall into one of three categories:

  • Hardware: Hardware refers to the physical devices in your network infrastructure, such as servers or routers. These require firmware upgrades and patches to remain secure. Vulnerabilities result from failure to perform upgrades and using outdated devices.
  • Software: Software refers to the applications an organization uses. Software vulnerabilities might be a flaw, glitch, or weakness in the software code. Again, patching and other updates are required to maintain security.
  • Human: These vulnerabilities stem from user security issues like weak (or leaked) passwords, clicking links on malicious websites, and human error such as opening a phishing email. Of the three categories, this is often the hardest for NetOps teams to control and enforce.

Vulnerability assessments scan your network for potential issues in each of these categories, and provide your team with crucial insight into the weaknesses of your IT infrastructure. Ideally, a vulnerability assessment will also prioritize the risks by level of severity, showing your team which to address first.

Enterprises looking to shift from reactive security measures like firewalls to a more proactive security approach look to vulnerability assessment as the first step in building an information security program.

What is vulnerability management?

Vulnerability management is the process of identifying, evaluating, treating, and reporting on security vulnerabilities in systems and the software that runs on them. Sounds a lot like vulnerability assessment, right? The key difference between the two, however, is that vulnerability management is a continuous cycle that includes vulnerability assessment. Where VA identifies and classifies the risks in your network infrastructure, VM goes a step further and includes decisions on whether to remediate, mitigate, or accept risks. VM is also concerned with general infrastructure improvement and reporting.

According to Gartner, vulnerability management runs on a cycle—a five-step process (not including pre-work like selecting vulnerability assessment tools) that most organizations follow.

The vulnerability management cycle

  1. Assess: Here’s where vulnerability assessments come in. In this step of the cycle, NetOps teams will identify assets, scan them, and build a report.
  2. Prioritize: The report generated in the first phase is used to prioritize risks. The NetOps team will also add threat context to the risks, which requires a thorough knowledge of the existing threat landscape as well as consideration of how threats may evolve over time.
  3. Act: The prioritized threats are then sorted into remediate, mitigate, and accept buckets. Remediation calls for removing the threat completely, if possible. Mitigation, on the other hand, reduces the likelihood of a vulnerability being exploited. Mitigation may be used if remediation is too disruptive to the system or if a patch isn’t available yet. You may also have threats that fall under the acceptance category. These may include devices/software soon to be replaced, which wouldn’t require any action.
  4. Reassess: Once the team has processed the risks according to their final recommendations, they’ll need to rescan and validate that the risks have been properly remediated, mitigated, or accepted.
  5. Improve: In this final step, the team should evaluate their metrics, checking that they’re accurate and up to date to ensure that they’re correctly assessing risks. Additionally, this phase should be used to eliminate any other underlying issues that may be contributing to system vulnerabilities.

Benefits of vulnerability management and vulnerability assessment

Vulnerability assessments are an important part of the vulnerability management cycle, and the VM cycle should be a key component of your NetOps team’s security strategy. Organizations today simply can’t afford to ignore the risks in their network infrastructure. As networks grow more complex, teams struggle to maintain visibility into their network. This creates an ideal environment for threat actors looking to exploit system vulnerabilities. Often, risks and attacks go unnoticed until they’ve caused irreparable damage at considerable cost to the organization.

VM has benefits that extend beyond security. For example, regularly evaluating your network’s devices and applications can help your team identify outdated technology or potential patches that will not only improve the general security of the network, but also optimize its performance. VM can also help your organization meet federal and internal compliance requirements. Regularly identifying and resolving risks through vulnerability assessments and the VM cycle can help your organization stay ahead of changing compliance requirements and prevent non-compliance penalties like fines.

Get started with vulnerability assessment and vulnerability management

With the obvious benefits, it should be clear that vulnerability assessment and vulnerability management are crucial to reducing overall risk in an organization’s infrastructure. And yet, many NetOps teams struggle to implement these processes. Whether your team is just getting started with vulnerability management, or looking to optimize your VM cycle to meet the challenges of an increasingly complex network and threat landscape, Rapid7 has the solutions that will empower your team to tackle vulnerabilities head on.

Ready to see the benefits of the vulnerability management cycle in your network?

Our report, Best Practices for Vulnerability Management in an Evolving Threat Landscape, can show you how!

McQueen: Flathub in 2023

Post Syndicated from original https://lwn.net/Articles/925472/

The Flathub organization (in the form of Robert McQueen) has posted a lengthy
update on the state of Flathub and its plans for the coming year.

So far, the GNOME Foundation has acted as an incubator and legal
host for Flathub even though it’s not purely a GNOME product or
initiative. Distributing software to end users along with
processing and forwarding payments and donations also has a
different legal profile in terms of risk exposure and nonprofit
compliance than the current activities of the GNOME
Foundation. Consequently, we plan to establish an independent legal
entity to own and operate Flathub which reduces risk for the GNOME
Foundation, better reflects the independent and cross-desktop
interests of Flathub, and provides flexibility in the future should
we need to change the structure.

Security updates for Tuesday

Post Syndicated from original https://lwn.net/Articles/925469/

Security updates have been issued by Debian (kopanocore), Fedora (golang-github-projectdiscovery-chaos-client, rust-sequoia-octopus-librnp, rust-sequoia-sop, rust-sequoia-sq, and usd), Oracle (libjpeg-turbo and pesign), Red Hat (kernel, kernel-rt, kpatch-patch, osp-director-downloader-container, pesign, rh-mysql80-mysql, samba, and zlib), SUSE (mariadb), and Ubuntu (fribidi, gmp, linux, linux-aws, linux-kvm, linux-lts-xenial, linux-azure, linux-azure-4.15, linux-kvm, linux-raspi2, linux-snapdragon, linux-raspi, nss, python3.6, rsync, systemd, and tiff).

Keeping the Cloudflare API ‘all green’ using Python-based testing

Post Syndicated from Elie Mitrani original https://blog.cloudflare.com/keeping-cloudflare-api-all-green-using-python-based-testing/

Keeping the Cloudflare API 'all green' using Python-based testing

Keeping the Cloudflare API 'all green' using Python-based testing

At Cloudflare, we reuse existing core systems to power multiple products and testing of these core systems is essential. In particular, we require being able to have a wide and thorough visibility of our live APIs’ behaviors. We want to be able to detect regressions, prevent incidents and maintain healthy APIs. That is why we built Scout.

Scout is an automated system periodically running Python tests verifying the end to end behavior of our APIs. Scout allows us to evaluate APIs in production-like environments and thus ensures we can green light a production deployment while also monitoring the behavior of APIs in production.

Why Scout?

Before Scout, we were using an automated test system leveraging the Robot Framework. This older system was limiting our testing capabilities. In fact, we could not easily match json responses against keys we were looking for. We would abandon covering different behaviors of our APIs as it was impossible to decide on which resources a given test suite would run. Two different test suites would create false negatives as they were running on the same account.

Regarding schema validation, only API responses were validated against a json schema and tests would not fail if the response did not match the schema. Moreover, It was impossible to validate API requests.

Test suites were run in a queue, making the delay to a new feature assessment dependent on the number of test suites to run. The queue would as well potentially make newer test suites run the following day. Hence we often ended up with a mismatch between tests and APIs versions. Test steps could not be run in parallel either.

We could not split test suites between different environments. If a new API feature was being developed it was impossible to write a test without first needing the actual feature to be released to production.

We built Scout to overcome all these difficulties. We wanted the developer experience to be easy and we wanted Scout to be fast and reliable while spotting any live API issue.

A Scout test example

Scout is built in Python and leverages the functionalities of Pytest. Before diving into the exact capabilities of Scout and its architecture, let’s have a quick look at how to use it!

Following is an example of a Scout test on the Rulesets API (the docs are available here):

from scout import requires, validate, Account, Zone

@validate(schema="rulesets", ignorePaths=["accounts/[^/]+/rules/lists"])
@requires(
    account=Account(
        entitlements={"rulesets.max_rules_per_ruleset": 2),
    zone=Zone(plan="ENT",
        entitlements={"rulesets.firewall_custom_phase_allowed": True},
        account_entitlements={"rulesets.max_rules_per_ruleset": 2 }))
class TestZone:
    def test_create_custom_ruleset(self, cfapi):
        response = cfapi.zone.request(
            "POST",
            "rulesets",
            payload=f"""{{
            "name": "My zone ruleset",
            "description": "My ruleset description",
            "phase": "http_request_firewall_custom",
            "kind": "zone",
            "rules": [
                {{
                    "description": "My rule",
                    "action": "block",
                    "expression": "http.host eq \"fake.net\""
                }}
            ]
        }}""")
        response.expect_json_success(
            200,
            result=f"""{{
            "name": "My zone ruleset",
            "version": "1",
            "source": "firewall_custom",
            "phase": "http_request_firewall_custom",
            "kind": "zone",
            "rules": [
                {{
                    "description": "My rule",
                    "action": "block",
                    "expression": "http.host eq \"fake.net\"",
                    "enabled": true,
                    ...
                }}
            ],
            ...
        }}""")

A Scout test is a succession of roundtrips of requests and responses against a given API. We use the functionalities of Pytest fixtures and marks to be able to target specific resources while validating the request and responses.  Pytest marks in Scout allow to provide an extra set of information to test suites. Pytest fixtures are contexts with information and methods which can be used across tests to enhance their capabilities. Hence the conjunction of marks with fixtures allow Scout to build the whole harness required to run a test suite against APIs.

Being able to exactly describe the resources against which a given test will run provides us confidence the live API behaves as expected under various conditions.

The cfapi fixture provides the capability to target different resources such as a Cloudflare account or a zone. In the test above, we use a Pytest mark @requires to describe the characteristics of the resources we want, e.g. we need here an account with a flag allowing us to have 2 rules for a ruleset. This will allow the test to only be run against accounts with such entitlements.

The @validate mark provides the capability to validate requests and responses to a given OpenAPI schema (here the rulesets OpenAPI schema). Any validation failure will be reported and flagged as a test failure.

Regarding the actual requests and responses, their payloads are described as f-strings, in particular the response f-string can be written as a “semi-json”:

 response.expect_json_success(
            200,
            result=f"""{{
            "name": "My zone ruleset",
            "version": "1",
            "source": "firewall_custom",
            "phase": "phase_http_request_firewall_custom",
            "kind": "zone",
            "rules": [
                {{
                    "description": "My rule",
                    "action": "block",
                    "expression": "http.host eq \"fake.net\"",
                    "enabled": true,
                    ...
                }}
            ],
            ...
        }}""")

Among many test assertions possible, Scout can assert the validity of a partial json response and it will log the information. We added the handling of ellipsis … as an indication for Scout not to care about any further fields at a given json nesting level. Hence, we are able to do partial matching on JSON API responses, thus focusing only on what matters the most in each test.

Once a test suite run is complete, the results are pushed by the service and stored using Cloudflare Workers KV. They are displayed via a Cloudflare Worker.

Keeping the Cloudflare API 'all green' using Python-based testing

Scout is run in separate environments such as production-like and production environments. It is part of our deployment process to verify Scout is green in our production-like environment prior to deploying to production where Scout is also used for monitoring purposes.

How we built it

The core of Scout is written in Python and it is a combination of three components interacting together:

Keeping the Cloudflare API 'all green' using Python-based testing
  • The Scout plugin: a Pytest plugin to write tests easily
  • The Scout service: a scheduler service to run the test suites periodically
  • The Scout Worker: a collector and presenter of test reports

The Scout plugin

This is the core component of the Scout system. It allows us to write self explanatory tests while ensuring a high level of compliance against OpenAPI schemas and verifying the APIs’ behaviors.

Keeping the Cloudflare API 'all green' using Python-based testing

The Scout plugin architecture can be split into three components: setup, resource allocator, and runners. Setup is a conjunction of multiple sub components in charge of setting up the plugin.

The Registry contains all the information regarding a pool of accounts and zones we use for testing. As an example, entitlements are flags gating customers for using products features, the Registry provides the capability to describe entitlements per account and zone so that Scout can run a test against a specific setup.

As explained earlier, Scout can validate requests and responses against OpenAPI schemas. This is the responsibility of validators. A validator is built per OpenAPI schema and can be selected via the @validate mark we saw above.

@validate(schema="rulesets", ignorePaths=["accounts/[^/]+/rules/lists"])

As soon as a validator is selected, all the interaction of a given test with an API will be validated. If there is a validation failure, it will be marked as a test failure.

Last element of the setup, the config reader. It is the sub component in charge of providing all the URLs and authentication elements required for the Scout plugin to communicate with APIs.

Next in the chain, the resources allocator. This component is in charge of consuming the configuration and objects of the setup to build multiple runners. This is a factory which will make available the runners in the cfapi fixture.

response = cfapi.zone.request(method, path, payload)

When such a line of code is processed, it is the actual method request of the zone runner allocated for the test which is executed. Actually, the resources allocator is able to provide specialized runners (account, zone or default) which grant the possibility of targeting specific API endpoints for a given account or zone.

Runners are in charge of handling the execution of requests, managing the test expectations and using the validators for request/response schema validation.

Any failure on expectation or validation and any exceptions are recorded in the stash. The stash is shared across all runners. As such, when a test setup, run or cleanup is processed, the timeline of execution and potential retries are logged in the stash. The stash contents are later used for building the test suite reports.

Scout is able to run multiple test steps in parallel. Actually, each resource couple (Account Runner, Zone Runner) is associated with a Pytest-xdist worker which runs test steps independently. There can be as many workers as there are resource couples. An extra “default” runner is provided for reaching our different APIs and/or URLs with or without authentication.

Testing a test system was not the easiest part. We have been required to build a fake API and assert the Scout plugin would behave as it should in different situations. We reached and maintained a test coverage confidence which was considered good (close to 90%) for using the Scout plugin permanently.

The Scout service

The Scout service is meant to schedule test suites periodically. It is a configurable scheduler providing a reporting harness for the test suites as well as multiple metrics. It was a design decision to build a scheduler instead of using cron jobs.

We wanted to be aware of any scheduling issue as well as run issues. For this we used Prometheus metrics. The problem is that the Prometheus default configuration is to scrape metrics advertised by services. This scraping happens periodically and we were concerned about the eventuality of missing metrics if a cron job was to finish prior to the next Prometheus metrics scraping. As such we decided a small scheduler was better suited for overall observability of the test runs. Among the metrics the Scout service provides are network failures, general test failures, reporting failures, tests lagging and more.

Keeping the Cloudflare API 'all green' using Python-based testing

The Scout service runs threads on configured periods. Each thread is a test suite run as a separate Pytest with Scout plugin process followed by a reporting execution consuming the results and publishing them to the relevant parties.

The reporting component provided to each thread publishes the report to Workers KV and notifies us on chat in case there is a failure. Reporting takes also care of publishing the information relevant for building API testing coverage. In fact it is mandatory for us to have coverage of all the API endpoints and their possible methods so that we can achieve a wide and thorough visibility of our live APIs.

As a fallback, if there are any thread failure, test failure or reporting failure we are alerted based on the Prometheus metrics being updated across the service execution. The logs of the Scout service as well as the logs of each Pytest-Scout plugin execution provide the last resort information if no metrics are available and reporting is failing.

The service can be deployed with a minimal YAML configuration and be set up for different environments. We can for example decide to run different test suites based on the environment, publish or not to Cloudflare Workers, set different periods and retry mechanisms and so on.

We keep the tests as part of our code base alongside the configuration of the Scout service, and that’s about it, the Scout service is a separate entity.

The Scout Worker

It is a Cloudflare worker in charge of fetching the most recent Worker KVs and displaying them in an eye pleasing manner. The Scout service publishes a test report as JSON, thus the Scout worker parses the report and displays its content based on the status of the test suite run.

For example, we present below an authentication failure during a test which resulted in such a display in the worker:

Keeping the Cloudflare API 'all green' using Python-based testing

What does Scout let us do

Through leveraging the capabilities of Pytest and Cloudflare Workers, we have been able to build a configurable, robust and reliable system which allows us to easily write self explanatory tests for our APIs.

We can validate requests and responses against OpenAPI schemas and test behaviors over specific resources while getting alerted through multiple means if something goes wrong.

For specific use cases, we can write a test verifying the API behaves as it should, the configuration to be pushed at the edge is valid and a given zone will react as it should to security threats. Thus going beyond an end-to-end API test.

Scout quickly became our permanent live tester and monitor of APIs. We wrote tests for all endpoints to maintain a wide coverage of all our APIs. Scout has since been used for verifying an API version prior to its deployment to production. In fact, after a deployment in a production-like environment we can know in a couple of minutes if a new feature is good to go to production and assess if it is behaving correctly.

We hope you enjoyed this deep dive description into one of our systems!

Zabbix 6.4 is out now!

Post Syndicated from Arturs Lontons original https://blog.zabbix.com/zabbix-6-4-is-out-now/25444/

Zabbix team is pleased to announce the release of the latest Zabbix major version – Zabbix 6.4. The release delivers many long-awaited improvements, such as Just-in-time LDAP and SAML user provisioning; support of older Zabbix proxy versions for simplified proxy management and zero–downtime Zabbix upgrades; near-instant configuration sync across Zabbix agents and proxies, and much more! 

New features and improvements

Just-in-time (JIT) user provisioning 

Zabbix 6.4 adds support of JIT user provisioning for LDAP and SAML authentication.

JIT user provisioning can be enabled in LDAP/SAML authentication settings

Zabbix administrators can now configure user provisioning by selecting the LDAP group pattern for matching and automatically assign User groups and User roles to the discovered users. Media types can also be mapped based on LDAP/SAML attributes.

A media can be assigned to the provisioned users based on their LDAP/SAML attributes
A group and role is assigned to the provisioned users

Cause and symptom events 

Zabbix 6.4 adds the ability to mark events as Cause or Symptom events. This allows us to filter events in a way, where we can see only root cause problems instead of being overwhelmed by symptom events. It is also possible to pause action operations for symptom events as to avoid unnecessary noise.

Multiple symptom events can be linked to a single cause event
Any event can be marked as a symptom or converted to a cause event
Action operations can be paused for symptom problems

Instant propagation of configuration changes 

Continuing to build on changes introduced in Zabbix 6.2 (Collecting only configuration change deltas), Zabbix 6.4 introduces instant configuration synchronization across passive and active agents and proxies.

  • Instead of receiving the full configuration copy every 2 minutes (old behavior), in Zabbix 6.4 active agent receives the configuration copy only when changes have been performed
  • RefreshActiveChecks parameter now supports a range 1-86400 (old range: 60-3600)
  • The ProxyConfigFrequency parameter is now used in both Zabbix server (for passive mode) and Zabbix proxy (for active mode) configuration files
  • ConfigFrequency parameter in Zabbix proxy configuration is now deprecated
  • Default ProxyConfigFrequency parameter is 10 seconds (down from 1 hour)

This also improves the performance of Zabbix servers and proxies, since only configuration deltas are synced. As for active agents – the active agent receives a full configuration copy only when any changes are detected in the configuration instead of receiving it every RefreshActiveChecks interval (old behavior)

New SNMP walk item for bulk collection and discovery of SNMP metrics 

A new SNMP agent walk item has been introduced. The item looks at a specified OID or OIDs and polls their indexes by suing the SNMP GetBulk requests. An SNMP GetBulk request can provide better performance and more rapid metric collection and discovery from enterprise-tier SNMP devices.

For example:

walk[1.3.6.1.1,1.3.6.2]

Result:

1.3.6.1.2.1.1 = STRING: "<value1>"
1.3.6.1.2.1.2 = STRING: "<value2>"
1.3.6.1.2.1.3 = STRING: "<value3>"
1.3.6.2.1 = INTEGER: 10
1.3.6.2.2 = INTEGER: 20

Textual values can then be transformed to JSON, which can serve as a master item for low-level discovery rules:

SNMP walk to JSON transforms the obtained data to JSON

Resulting values:

[
{"{#SNMPINDEX}":"7","{#IFALIAS}":"Uplink PT","{#IFTYPE}":"6"},
{"{#SNMPINDEX}": "8","{#IFALIAS}": "Uplink FB","{#IFTYPE}":"6"},
{"{#SNMPINDEX}": "473","{#IFALIAS}":"lag","{#IFTYPE}":"161"}
]

Once the data is converted to JSON, we can use SNMP walk value preprocessing step together with LLD macros, to create dependent item prototypes:

SNMP walk value preprocessing step can be used to specify value for extraction in item prototypes

Support of data collection for outdated proxies

To improve the Zabbix component upgrade workflows (especially for large environments), outdated proxies can still perform data collection with a newer Zabbix server version:

  • Proxy is fully supported if it has the same major version as the Zabbix server
  • Proxy is marked as outdated if its major version is older than the Zabbix server but not older than the previous LTS release
  • Outdated proxies still support data collection and remote command execution
  • In other scenarios, the proxy becomes not supported
Deployed proxy compatibility can be seen in Zabbix frontend
Server version Current proxy version Outdated proxy version Unsupported proxy version
6.4 6.4 6.0, 6.2 Older than 6.0; newer than 6.4
7.0 7.0 6.0, 6.2, 6.4 Older than 6.0; newer than 7.0
7.2 7.2 7.0 Older than 7.0; newer than 7.2

New menu layout 

Zabbix menu layout has been redesigned. The goal of the new menu layout is to provide logical and consistent access to main Zabbix features.

The new menu provides a more consistent and logical layout to Zabbix features

Real-time streaming of metrics and events over HTTP

In addition to streaming collected metrics and events to files, Zabbix 6.4 adds the option to stream metrics and events over HTTP. Zabbix administrators have the option to filter the data for streaming by using tag filters. A new Connectors section has been introduced under Administration – General. Here Zabbix administrators can define an external system where item values and events should be pushed to.

Define a new connector to stream metrics and events over HTTP

Zabbix 6.4 can be used as a source of information for other applications, analytics reports, and AI engines by streaming metrics and events over HTTP in real time. Metrics and events can be streamed to message brokers like Kafka, RabbitMQ, or Amazon Kinesis to adapt the behavior of external systems in real time. 

Template versioning 

Template versioning has been introduced to improve template management and ease of use. Templates are now marked with vendor ar version fields, which are visible in Zabbix frontend; these fields can also be added when writing a custom template.

Template version and vendor fields are visible in the frontend

Development framework for Zabbix widget creation 

Zabbix has a large developer community creating their own custom frontend modules, widgets and Go plugins. In Zabbix 6.4, our goal was to streamline this process by creating a development framework for widget creation. To achieve this, the following changes have been introduced:

  • Widgets have been converted to modules
  • Modules are now fully self-contained and modular
  • Built-in widgets reside in ui/widgets
  • Custom widgets reside in ui/modules/<widget>
  • Adding new widgets is as simple as adding new files without changing the existing files

In addition to these changes, we have also added a new Developer Center section to our documentation. The section contains guides, tutorials and code examples to guide our community in developing Frontend modules and widgets, as well as help with Zabbix agent 2 custom Go plugin development.

The Developer Center section contains guides, tutorials, and code examples for extending Zabbix

Other features and improvements 

The release includes many other changes:

  • Simple check, External check, SSH agent, Telnet agent item types now do not require an interface to be present on the host 
  • Pre-configured email media type settings for Gmail and O365 email providers 
  • Dynamic item value widget thresholds
  • Option to define custom labeled links for hosts and events
  • Ability to label trigger URLs
  • Improved preprocessing performance and thread-based preprocessing workers
  • Ability to label aggregated datasets in Graph widget
  • SQLite3 Zabbix proxies now automatically recreate the SQLite3 database file during an upgrade
  • A host status filter (enabled/disabled) has been added under Data collection – Hosts
  • Additional filtering options have been added to the Action log
  • Action log now supports import to CSV
  • Multiple context menu improvements to Host, Item and Event context menus
  • Old password verification is now required when changing your internal Zabbix user password
  • Value cache performance improvements when working with metrics that get updated less frequently than once per day
  • Added commands to enable profiling of rwlocks/mutexes (for debugging)

The full list of changes, bug fixes, and new features can be found in the Zabbix 6.4 release notes

New templates and integrations

Zabbix 6.4 comes pre-packaged with many new templates and integrations for the most popular vendors and cloud providers. Multiple existing templates have also received improvements:

  • Microsoft Azure MySQL servers 
  • Microsoft Azure PostgreSQL servers 
  • Microsoft Azure virtual machines 
  • Low-level discovery improvements in AWS by HTTP template 
  • Veeam Backup Enterprise Manager 
  • Veeam Backup and Replication 
  • Cisco Nexus 9000 Series 
  • BMC Control-M 
  • Cisco Meraki dashboard 
  • OS processes by Zabbix agent 
  • Improvements to filesystem discovery in official Zabbix OS templates 

Zabbix 6.4 introduces a webhook integration for the Line messaging app, allowing Zabbix events to be forwarded to the Line messenger. 

Zabbix 6.4 adds a variety of new templates and integrations

Zabbix 6.4 packages and images

Official Zabbix packages and images are available for: 

  • Linux distributions for different hardware platforms on RHEL, CentOS, Oracle Linux, Debian, SUSE, Ubuntu, Raspbian 
  • Virtualization platforms based on VMWare, VirtualBox, Hyper-V, XEN 
  • Docker 
  • Packages and pre-compiled agents for the most popular platforms, including macOS and MSI packages for Microsoft Windows 

You can find the download instructions and download the new version on the Download page.

One-click deployments for the following cloud platforms are coming soon: 

  • AWS, Azure, Google Cloud Platform, Digital Ocean 

Upgrading to Zabbix 6.4

In order to upgrade to Zabbix 6.4 you need to upgrade your repository package and download and install the new Zabbix component packages (Zabbix server, proxy, frontend, and other Zabbix components). When you start the Zabbix server, an automatic database schema upgrade will be performed. Zabbix agents are backward compatible; therefore, it is not required to install the new agent versions. You can perform the agent upgrade at a later time. 

If you’re using the official Docker container images – simply deploy a new set of containers for your Zabbix components. Once the Zabbix server container connects to the backend database, the database upgrade will be performed automatically.

You can find detailed step-by-step upgrade instructions on our Upgrade procedure page. 

Join the webinar

If you wish to learn more about the Zabbix 6.4 features and improvements, we invite you to join our What’s new in Zabbix 6.4 public webinar.

During the webinar, you will get the opportunity to:

  • Learn about Zabbix 6.4 features and improvements
  • See the latest Zabbix templates and integrations
  • Participate in a Q&A session with Zabbix founder and CEO Alexei Vladishev
  • Discuss the latest Zabbix version with Zabbix community and Zabbix team members

This is a public webinar – anyone can sign up, attend and have their questions answered by the Zabbix team!

The collective thoughts of the interwebz