Vulnerability in Stalkerware Apps

2022-03-02 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/03/vulnerability-in-stalkerware-apps.html

TechCrunch is reporting — but not describing in detail — a vulnerability in a series of stalkerware apps that exposes personal information of the victims. The vulnerability isn’t in the apps installed on the victims’ phones, but in the website the stalker goes to view the information the app collects. The article is worth reading, less for the description of the vulnerability and more for the shadowy string of companies behind these stalkerware apps.

Migrating a monolithic .NET REST API to AWS Lambda

2022-03-02 James Beswick

Post Syndicated from James Beswick original https://aws.amazon.com/blogs/compute/migrating-a-monolithic-net-rest-api-to-aws-lambda/

This post is written by James Eastham, Cloud Infrastructure Architect.

There are many ways to deploy a .NET application to AWS. From a single process ASP.NET core web API hosted on an EC2 instance to a serverless API backed by AWS Lambda. This post explains key topics to simplify your move from monolith to serverless.

The .NET Framework launched in 2002. This means that there are years’ worth of existing .NET application code that can benefit from moving to a serverless architecture. With the release of the AWS Porting Assistant for .NET and the AWS Microsoft Extractor for .NET, AWS tooling can assist directly with this modernization.

These tools help modernization but don’t migrate the compute layer from traditional servers to serverless technology.

Hexagonal architecture

The hexagonal architecture pattern proposes the division of a system into loosely coupled and interchangeable components. The application and business logic sit at the core of the application.

The next layer up is a set of interfaces that handle bidirectional communication from the core business logic layer. Implementation details are moved to the outside. The inputs (API controllers, UI, consoles, test scripts) and outputs (database implementations, message bus interactions) are at the perimeter.

The chosen compute layer becomes an implementation detail, not a core part of the system. It allows a cleaner process for migrating any integrations, from the frontend, to the compute layer and underlying database engine.

Code examples

The GitHub repo contains the code examples from this post with instructions for deploying the migrated serverless application.

The repository contains a .NET core REST API. It uses MySQL for its database engine and relies on an external API as part of its business logic. It also contains a migrated serverless version of the same application that you can deploy to your AWS account. This uses a combination of the AWS Cloud Development Kit (CDK) and the AWS Serverless Application Model (AWS SAM) CLI.

The architecture of the deployed monolithic application is:

After migrating the application to Lambda, the architecture is:

Integrations

Modern web applications rely on databases, file systems, and even other applications. With first class support for dependency injection in .NET Core, managing these integrations is simpler.

The following code snippet is taken from the BookingController.cs file. It shows how required interfaces are injected into the constructor of the controller. One of the controller methods uses the injected interface to list bookings from the BookingRepository.

    [ApiController]
    [Route("[controller]")]
    public class BookingController : ControllerBase
    {
        private readonly ILogger<BookingController> _logger;
        private readonly IBookingRepository _bookingRepository;
        private readonly ICustomerService _customerService;

        public BookingController(ILogger<BookingController> logger,
            IBookingRepository bookingRepository,
            ICustomerService customerService)
        {
            this._logger = logger;
            this._bookingRepository = bookingRepository;
            this._customerService = customerService;
        }

        /// <summary>
        /// HTTP GET endpoint to list all bookings for a customer.
        /// </summary>
        /// <param name="customerId">The customer id to list for.</param>
        /// <returns>All <see cref="Booking"/> for the given customer.</returns>
        [HttpGet("customer/{customerId}")]
        public async Task<IActionResult> ListForCustomer(string customerId)
        {
            this._logger.LogInformation($"Received request to list bookings for {customerId}");

            return this.Ok(await this._bookingRepository.ListForCustomer(customerId));
        }
}

The implementation of the IBookingRepository is configured at startup using dependency injection in the Startup.cs file.

services.AddTransient<IBookingRepository, BookingRepository>();

This works when using an ASP.NET Core Web API project, since the framework abstracts much of the complexity and configuration. But it’s possible to apply the same practices for .NET core code running in Lambda.

Configuring dependency injection in AWS Lambda

The startup logic is moved to a standalone DotnetToLambda.Serverless.Config library. This allows you to share the dependency injection configuration between multiple Lambda functions. This library contains a single static class named ServerlessConfig.

There is little difference between this file and the Startup.cs file:

public void ConfigureServices(IServiceCollection services)
{
	var databaseConnection =
		new DatabaseConnection(this.Configuration.GetConnectionString("DatabaseConnection"));
	
	services.AddSingleton<DatabaseConnection>(databaseConnection);
	
	services.AddDbContext<BookingContext>(options =>
		options.UseMySQL(databaseConnection.ToString()));

	services.AddTransient<IBookingRepository, BookingRepository>();
	services.AddHttpClient<ICustomerService, CustomerService>();
	
	services.AddControllers();
}

And the configuration method in the ServerlessConfig class:


public static void ConfigureServices()
{
	var client = new AmazonSecretsManagerClient();
	
	var serviceCollection = new ServiceCollection();

	var connectionDetails = LoadDatabaseSecret(client);

	serviceCollection.AddDbContext<BookingContext>(options =>
		options.UseMySQL(connectionDetails.ToString()));
	
	serviceCollection.AddHttpClient<ICustomerService, CustomerService>();
	serviceCollection.AddTransient<IBookingRepository, BookingRepository>();
	serviceCollection.AddSingleton<DatabaseConnection>(connectionDetails);
	serviceCollection.AddSingleton<IConfiguration>(LoadAppConfiguration());

	serviceCollection.AddLogging(logging =>
	{
		logging.AddLambdaLogger();
		logging.SetMinimumLevel(LogLevel.Debug);
	});

	Services = serviceCollection.BuildServiceProvider();
}

The key addition is the manual creation of the ServiceCollection object on line 27 and the call to BuildServiceProvider on line 45. In.NET core the framework abstracts away this manual object initialization. The created ServiceProvider is then exposed as a read-only property of the ServerlessConfig class. All we have done is taken the boilerplate code that an ASP.NET Core web API performs behind the scenes and brought it into the foreground.

This allows you to copy and paste large parts of the startup configuration directly from the web API and re-use it in your Lambda functions.

Lambda API controllers

For the function code, follow a similar process. For example, here is the ListForCustomer endpoint re-written for Lambda:

 public class Function
{
	private readonly IBookingRepository _bookingRepository;
	private readonly ILogger<Function> _logger;
	
	public Function()
	{
		ServerlessConfig.ConfigureServices();

		this._bookingRepository = ServerlessConfig.Services.GetRequiredService<IBookingRepository>();
		this._logger = ServerlessConfig.Services.GetRequiredService<ILogger<Function>>();
	}
	
	public async Task<APIGatewayProxyResponse> FunctionHandler(APIGatewayProxyRequest apigProxyEvent, ILambdaContext context)
	{
		if (!apigProxyEvent.PathParameters.ContainsKey("customerId"))
		{
			return new APIGatewayProxyResponse
			{
				StatusCode = 400,
				Headers = new Dictionary<string, string> { { "Content-Type", "application/json" } }
			};
		}

		var customerId = apigProxyEvent.PathParameters["customerId"];
		
		this._logger.LogInformation($"Received request to list bookings for: {customerId}");

		var customerBookings = await this._bookingRepository.ListForCustomer(customerId);
		
		return new APIGatewayProxyResponse
		{
			Body = JsonSerializer.Serialize(customerBookings),
			StatusCode = 200,
			Headers = new Dictionary<string, string> { { "Content-Type", "application/json" } }
		};
	}
}

The function constructor calls the startup configuration. This allows the initial configuration to be re-used while the Lambda execution environment is still active. Once the services have been configured any required interfaces can be retrieved from the services property of the ServerlessConfig class.

The second key differences are the mapping of the inbound request and response back to API Gateway. The HTTP request arrives as an event and the contents must be manually parsed out of the raw HTTP data. The same applies to the HTTP response, which must be constructed manually. Other than these two differences, it’s a copy from the original BookingController.

Application configuration

An ASP.NET Core Web API contains an appsettings.json file, which contains runtime specific configuration. The framework handles loading the file and exposing it as an injectable IConfiguration interface. It’s also possible to load settings from environment variables.

This is still possible when using Lambda. You can package an appsettings.json file with the compiled code and load it manually at runtime. However, when using Lambda as the compute layer, there are AWS-specific options for managing configuration.

Environment variables

Lambda environment variables are used to add runtime configuration, as shown in the template.yaml file:

 Environment:
	Variables:
		SERVICE: bookings
		DATABASE_CONNECTION_SECRET_ID: !Ref SecretArn

This AWS SAM configuration adds an environment variable named DATABASE_CONNECTION_SECRET_ID. You can access this in Lambda the same way an environment variable is accessed in any C# application:

 var databaseConnectionSecret = client.GetSecretValueAsync(new GetSecretValueRequest()
            {
                SecretId = Environment.GetEnvironmentVariable("DATABASE_CONNECTION_SECRET_ID"),
            }).Result;

This is the simplest way to add runtime configuration. The variables are stored in plaintext and any change requires a redeployment or manual interaction.

External configuration services

AWS has services that allow you to move application configuration outside of the function code. These include AWS Systems Manager Parameter Store, AWS AppConfig and AWS Secrets Manager.

You can use Parameter Store to store plaintext parameters that can also be encrypted using the AWS Key Management Service. The contents of the appsettings.json file from the ASP.NET Core API is directly copied into the parameter string and deployed using the AWS CDK.

 var parameter = new StringParameter(this, "dev-configuration", new StringParameterProps()
{
	ParameterName = "dotnet-to-lambda-dev",
	StringValue = "{\"CustomerApiEndpoint\": \"https://jsonplaceholder.typicode.com/users\"}",
	DataType = ParameterDataType.TEXT,
	Tier = ParameterTier.STANDARD,
	Type = ParameterType.STRING,
	Description = "Dev configuration for dotnet to lambda",
});

This JSON data is loaded as part of the startup configuration. The IConfiguration implementation is then built manually using the parameter string.

 private static IConfiguration LoadAppConfiguration()
{
	var client = new AmazonSimpleSystemsManagementClient();
	var param = client.GetParameterAsync(new GetParameterRequest()
	{
		Name = "dotnet-to-lambda-dev"
	}).Result;
	
	return new ConfigurationBuilder()
		.AddJsonStream(new MemoryStream(Encoding.ASCII.GetBytes(param.Parameter.Value)))
		.Build();

The second configuration mechanism is Secrets Manager. This helps protect secrets and provides easier rotation and management of database credentials.

Amazon RDS is integrated with Secrets Manager. When creating a new RDS instance, the database connection details can be automatically encrypted and persisted as a secret. The details for the MySQL instance are stored in Secrets Manager and are not exposed. These connection details can be accessed as part of the startup configuration using the Secrets Manager SDK.

private static DatabaseConnection LoadDatabaseSecret(AmazonSecretsManagerClient client)
{
	var databaseConnectionSecret = client.GetSecretValueAsync(new GetSecretValueRequest()
	{
		SecretId = Environment.GetEnvironmentVariable("DATABASE_CONNECTION_SECRET_ID"),
	}).Result;

	return JsonSerializer
		.Deserialize<DatabaseConnection>(databaseConnectionSecret.SecretString);
}

The Lambda functions require IAM permissions to access both Secrets Manager and Parameter Store. AWS SAM includes pre-defined policy templates that you can add to the template. Four lines of YAML apply the required Secrets Manager and SSM permissions:

Policies:
	- AWSSecretsManagerGetSecretValuePolicy:
		SecretArn: !Ref SecretArn
	- SSMParameterReadPolicy:
		ParameterName: dotnet-to-lambda-dev

For a full list, see the policy template list.

Networking

The final architectural component is the network. Lambda functions are deployed into a VPC owned by the service. The function can access anything available on the public internet such as other AWS services, HTTPS endpoints for APIs, or services and endpoints outside AWS. The function then has no way to connect to your private resources inside of your VPC.

When deploying an RDS instance into AWS, it’s best practice to place the database in a private subnet with external ingress. If Lambda uses RDS, you must create a connection between the Lambda service VPC and your VPC. The details of this networking component can be found in this blog article.

The AWS SAM template defines this networking configuration:

VpcConfig:
	SubnetIds:
	  - !Ref PrivateSubnet1
	  - !Ref PrivateSubnet2
	SecurityGroupIds:
	  - !Ref SecurityGroup

In this example, the networking configuration is applied globally. This means that the same configuration is applied to all Lambda functions in the template. The functions here are deployed across two subnets and one security group. Learn more about the steps for configuring the subnets and security groups for RDS access in this article.

The specific values for the subnets and security groups are taken from environment variables. When running locally, you can provide these variables manually. When deploying via CICD, these variables can be changed dynamically based on the stage of the pipeline.

 PrivateSubnet1:
	Description: 'Required. Private subnet 1. Output from cdk deploy'
	Type: 'String'
PrivateSubnet2:
	Description: 'Required. Private subnet 2. Output from cdk deploy'
	Type: 'String'
SecurityGroup:
	Description: 'Required. Security group. Output from cdk deploy'
	Type: 'String'

Conclusion

This blog post shows the required considerations for migrating a .NET core REST API to AWS Lambda. You can now start to look at your existing code base and make an informed decision whether Lambda is for you. With the right abstractions and configuration, you can migrate a .NET core API to Lambda compute with copy and paste.

For more serverless learning resources, visit Serverless Land.

Литературата като противоотрова на пропагандата: 10 важни книги за Украйна и Русия

2022-03-02 Йоанна Елми

Post Syndicated from Йоанна Елми original https://toest.bg/10-vazhni-knigi-za-ukrayna-i-rusiya/

Литературата е противоотрова на пропагандата. Колкото и поетично да звучи, дори проучвания в сферата на комуникационните науки стигат до заключенията, че четенето на книги всъщност е един от най-ефективните начини за изграждане на светоглед и преодоляване на естествените предразсъдъци. Докато пропагандата търси прости и емоционални обяснения, книгите дават нюанс и преобразуват първичната емоция в емпатия. Докато пропагандата цели манипулация, литературата дава свобода за разбиране. С тази мисъл подготвихме списък с важни книги, с които читателите да си обяснят света, в който осъмнахме преди няколко дни.

Мисля, че Украйна тъкмо започна да се учи да говори за себе си, да се заявява. Това обаче невинаги ни се получава. Тогава за нас започват да говорят други, което невинаги е в наша полза. Именно в този случай литературата е важен инструмент за комуникация. Много се надявам, че чрез украинските книги ще се появи по-голямо разбиране за Украйна във вашата страна, което не се гради на стереотипи и предразсъдъци. Същото се отнася и за Източна Украйна и нейното възприемане – разбира се, че за украинския Изток се говори много от самото начало на руската агресия и военните действия в Донбас. Жалко е, че Украйна, по-точно източната ѝ част, присъства в информационното поле предимно като територия на война, място, където се водят военни действия. Всъщност това е един интересен, нееднороден, парадоксален район, където живеят много интересни хора. Надявам се, че в бъдеще ще можем да говорим за Източна Украйна не само в контекста на войната.

Това са думи на украинския писател Сергий Жадан пред „Литературен вестник“. Именно негова книга е първата ни препоръка в настоящата статия.

„Химн на демократичната младеж“ от Сергий Жадан

превод от украински Албена Стаменова, изд. „Парадокс“, 2019

Сборникът съдържа шест истории от Украйна през 90-те, разказващи за постсъветската трансформация, която само формално нарича себе си демокрация. „Идеологемата за средната класа като гръбнак на съвременното капиталистическо общество се разбива в посткомунистическата икономическа действителност. В това пространство на рекет, корупция, кражби, далаверки и данъчни измами действат (или по-скоро въобще не действат) странни закони. Те са неразбираеми за другите народи от Европа, понякога са неразбираеми и за славянските народи, а се случва и самите украинци да не ги разбират“, пише в представянето на книгата.

„Зад гърба“ от Гаска Шиян

превод от украински Пенка Кънева, изд. „Колибри“, 2021

„Зад гърба“ също е „роман за съвременна Украйна с нейните национални и морални проблеми, за дълга и патриотизма, за женската съдба в едно традиционно патриархално общество. Роман, който шокира с натуралистичните сексуални сцени, с богоборчеството и бунтарското си отношение към семейните и социалните ценности. Роман, силно оспорван и горещо приветстван от критици и читатели“, четем в рецензията на издателя.

Преди седмица в „Дневник“ беше публикуван откъс от книгата.

„Монументалната пропаганда“ от Владимир Войнович

превод от руски Иван Тотоманов, изд. „Жанет 45“, 2005

Романът проследява историята на Аглая Степановна Ревкина и нейната дълбока привързаност към другаря Йосиф Висарионович Сталин. През 1956 г. обаче култът към личността на Сталин пада, а с него се срива и светът на Аглая Степановна. От черно-белите реалии, в които всеки враг на Сталин е враг на народа, тя трябва да се адаптира към съвсем различна политическа обстановка, която обаче ползва същия речник и етикети.

През по-голямата част действието се развива в руския град Долгов, където главната героиня е важна партийна особа. С настъпването на промените обаче Аглая Степановна отказва да приеме новата партийна линия, което води до изтласкването ѝ от структурите. В своя отчаян бунт на преданост, тя подслонява премахнатата от града статуя на Сталин в дома си. Под историята за съжителството им и за живота на Аглая Степановна откриваме важни прозрения за култа към личността и партията, за крайностите, до които стигат хората в името на идеологията, и за способността на човешкия ум да съществува в доброволен плен на собствените си илюзии. Предлагаме ви да прочетете откъс от книгата, публикуван в „ЛитерНет“.

„Остров Крим“ от Василий Аксьонов

превод от руски Здравка Петрова, изд. „Жанет 45“, 2005

Писан преди повече от четвърт век, романът „Остров Крим“ е издаден за първи път в САЩ, където Василий Аксьонов живее от 1980 г. Оттогава до наши дни произведението се смята за един от шедьоврите на антиутопията и се нарежда до образци на жанра, като „1984“ на Оруел, „Прекрасният нов свят“ на Хъксли и „Ние“ на Замятин. И не само: за огромното мнозинство читатели „Остров Крим“ продължава да звучи пророчески и е култова книга на въплътената антиутопия – особено с оглед на насилственото анексиране на Крим през 2014 г.

Съветското гражданство на Аксьонов е отнето и възстановено по време на Перестройката, а възгледите му се определят като антитоталитарни. „Остров Крим“ разказва алтернативна история, в която белогвардейците успяват да удържат полуостров Крим след Октомврийската революция. „Ню Йоркър“ описва книгата като „трескавия сън на руски интелектуалец – каква би могла да е Русия, ако не беше се превърнала в Съветския съюз“. Ключовото събитие в романа е съветската инвазия в алтернативния Крим – инвазия, която е уж поискана, разбира се, от живеещите там. Защото не е война, ако е „освобождение“.

„Време секънд хенд“ от Светлана Алексиевич

превод от руски Тотка Монова, изд. „Парадокс“, 2016

Това е последната част от петокнижието на Алексиевич и разказва „историите на съветския човек, „изхвърлен от живота“ след рухването на империята СССР“, както пише Георги Грънчаров в ревюто си на книгата:

„Алексиевич дава думата на жертвите и на палачите; на разочарованите идеалисти; на фанатизираните комунисти; на репресираните и техните наследници; на наивните мечтатели и на прагматиците. […] Първата част на книгата е хипнотизиращ колаж от разкази за срива на режима. На идеалистите, чиито идеали се сблъскват с новата действителност. На хората, които не могат да разберат промяната – защо на битпазара се продават портрети на белогвардейски генерали редом с бюст на Ленин. На скептиците, носталгиците, хардлайнерите. На онези, за които Елцин е герой, и на онези, за които е предател. И сред тези разкази витае ужасът. Знаете ли защо? Защото всеки един от разказите добавя парче след парче към кошмарната картина на необратимо деформирания съветски човек. Всеки от разказите – и на разочарованите идеалисти, и на злорадстващите комунисти, оголва истината за жестокия експеримент, състоял се в Съветска Русия в продължение на седем десетилетия. Но и на хаотичната безпътица, нахлула след рухването на режима.“

„Предел на забравата“ от Сергей Лебедев

превод от руски Денис Коробко, изд. „Кръг“, 2020

„След толкова много написани и изговорени думи за съдбата на заточениците в Далечния север по времето на комунизма, нов ракурс към тази тема изглежда невъзможен. Но романът на Сергей Лебедев го открива – той е потъване в мрачните галерии на рудниците, поглед отдолу, отзад, през очите на невинния, омърсен косвено, заразен от злото като дете“, четем в представянето на книгата. За нея писателката Яница Радева казва в рецензията си за „Литературен вестник, че „не е роман за ГУЛАГ, а роман за мястото на човека в света, за отговора на въпроса какво забравяме в предела на забравата, за бъдещето, което не може да настъпи без миналото. А миналото може да намери покой не когато се забрави, а когато срещне мъртвите и живите в словото“.

„Неизбежно си мислех също, че този роман е зловещо елегантен шамар за Путинова Русия, в която открай време текат процеси по пренаписване, изтриване или създаване на историческа памет. При това шамар от 21-ви век, от сегашното поколение на средна възраст, а не от времето на Шаламов и Солженицин“, пише Антония Апостолова в „Литературни разговори“.

„Полеви изследвания на украинския секс“ от Оксана Забужко

превод от руски София Бранц, изд. „Изток-Запад“, 2005

Публикувана за пръв път през 1996 г., тази книга е сред най-противоречивите и днес. Високолитературен роман, привидно написан хаотично и като поток на мисълта, който пласт след пласт ерозира и оголва посланието. Поради това и една история за края на връзката между главната героиня и нейния приятел всъщност разкрива детайли за живота в Украйна, историческото наследство и украинската идентичност. Човешкото тяло може да бъде разглеждано като метафора за тялото на нацията, за територията, и с оглед на събитията човек веднага си задава въпроса как се усещат фантомната болка на анексията, прорезите на инвазията.

„Украйна има дълга история на разделяне и събиране, отново и отново. Части от съвременна Украйна са се считали за руска, полска, немска територия… Забужко е силно чувствителна към психологическия аспект, към тъмната и потисната част от украинската история и идентичност, но тя е също толкова наясно с потенциала за трансформация. В крайна сметка културата винаги се променя, дори обременена с теглото на миналото“, четем в ревю на Бриана Бербенюик в канадския книжен блог Numéro Cinq.

„Русия на Путин“ от Анна Политковская

превод от английски Елена Кодинова, изд. „Бард“, 2007

Едва ли има човек, който да не знае името Анна Политковская. И ако днес все още някои наблюдатели смятат Владимир Путин за рационален руски лидер от ново поколение, то Политковская изгуби живота си вследствие на проницателните си и навременни прозрения за характера на руския диктатор.

„Бившият шпионин от КГБ Владимир Путин е избран за президент на Русия през 2000 г. Още с излизането си на публичната арена той се представя за открит, просветен лидер, който горещо желае разбирателство със Запада. За разлика от много европейски и американски журналисти и политици, Анна Политковская никога не е вярвала на медийния образ на Путин. С преимуществата на погледа от сърцето на Русия тя развенчава както човека Путин, така и търговската марка, в която се е превърнал. Твърди, че е жаден за власт продукт на миналото си и не може да се въздържи да потъпква гражданските права и свободи при всеки удобен случай. Според Политковская не такъв лидер искат повечето от днешните руснаци. За да докаже хипотезата си, тя описва желязната хватка на Путин от позицията на отделния човек, чието положение зависи от този уникален вид тирания. Мафиотски сделки, въоръжени грабителски скандали в провинцията, потресаваща корупция в армията и съдебната система, срив сред интелигенцията, трагични грешки по време на акцията „Норд-Ост“ – това са темите, които Политковская подлага на безпощаден, но неизменно дълбоко човечен анализ“, четем в представянето на книгата.

„Алкохолен делир. Литературни репортажи за Русия от XXI век“ от Яцер Хуго-Бадек

превод от полски Диляна Денчева, изд. „Парадокс“, 2019

Полският журналист прекарва няколко месеца в пътуване из Русия и превръща черната статистика в истории, припомняйки ни, че зад числата стоят хора. По пътя, дълъг 13 000 километра, се срещаме с жителите на градове, в близост до които са се провеждали ядрени тестове, с живота на алкохолици и наркозависими, с работещи в оръжейната индустрия на Русия и с какви ли не други персонажи от най-ниските и тъмни прослойки. Въпреки тежестта на разказаното без нито една спестена истина, Хуго-Бадек открива хумор, топлина и човещина в сибирската степ, която може да се интерпретира като отражение на пустотата, обитавана от много от истинските герои в репортажите.

„Зимата идва“ от Гари Каспаров

превод от английски Христо Димитров, изд. „Сиела“, 2015

„Гари, ти беше прав.“ Това са думи на мнозина от познатите на Каспаров, които самият той сподели, че чува от началото на инвазията в Украйна. „Дългогодишният световен шампион по шах Гари Каспаров е отдаден активист за човешки права и един от основните противници на политиката на руския лидер Владимир Путин. Каспаров е убеден, че трябва да се създаде мощен дипломатически и икономически фронт срещу президента на Русия, който ощетява интересите на страната си за сметка на консолидирането на едноличната си власт“, четем в ревю на книгата в „Култура“, където може да прочетете и откъс.

Каспаров пише за много важна част от руската политика през последните 20 години – приобщаването на цели сектори от западните политически елити към Кремъл чрез финансови облаги, например постове по високите етажи на руския бизнес. Днес думите на Каспаров се четат като пророчество, с което може да завършим нашите препоръки:

Корените на Путин в КГБ мрачно прораснаха в стил на управление, което не беше нито реформистко, нито демократично. Общата черта във вътрешната и външната му политика беше опитът му да търгува със страх – страховете на руснаците, че страната им е подложена на атака от враждебни външни сили (от чеченците, НАТО или свободните пазари, а обикновено и от трите заедно), и страховете на западняците, че ако няма силен и прагматичен лидер, Русия отново ще стане непокорна, нестабилна и потенциално опасна. Петнайсет години по-късно тактиката на Путин на търгуване със страх е все още много сходна и също толкова ефективна.

Припомняме и препоръките на Йовко Ламбрев от миналата седмица за още книги по темата за Украйна и Русия: „Червен глад“ на Ан Апълбаум, както и „Кървави поля“ и „Пътят към несвободата“ на Тимъти Снайдър.

Активните дарители на „Тоест“ получават постоянна отстъпка в размер на 20% от коричната цена на всички заглавия от каталозите на „Парадокс“, „Колибри“, „Жанет 45“, както и на няколко други български издателства в рамките на партньорската програма Читателски клуб „Тоест“. За повече информация прочетете на toest.bg/club.

Заглавна снимка: Alfons Morales / Unsplash

Източник

Хуманизмът е голяма хапка

2022-03-02 Светла Енчева

Post Syndicated from Светла Енчева original https://toest.bg/humanizmut-e-golyama-hapka/

След началото на войната в Украйна сме свидетели на вълна от съпричастност към бягащите от агресията на Путин. И частни лица, и представители на бизнеса предлагат да помогнат с каквото могат – едни предоставят собствените си домове, други – безплатен хотел или работа, някои дори предлагат да осиновят украински дечица. На доста места се събират дарения. За броени дни са събрани много чували с хуманитарни помощи. Публичните институции също оказват подкрепа – държавата организира изпращането на автобуси в Украйна, които да извозват бежанци; някои общини предоставят центрове за настаняване; правителството създаде специален информационен портал за действия във връзка с кризата. Помощ предлагат дори църкви и манастири. Жестовете на подкрепа са твърде много, за да бъдат изброени.

Мейнстрийм медиите излъчват послания, с които насърчават хората да постъпват по този начин – че хуманното отношение е да отвориш дома си за бягащите от война, че в кризисна ситуация сме призвани да се лишим от личния си комфорт, за да помогнем на други. Че така е християнско. На фона на този всенароден ентусиазъм една шепа правозащитници гледат с почуда –

кога станахме такива хуманисти?

Занимавам се с проблемите на бежанците и чужденците в България от 2009 г. В началото това дори не беше тема, по която у нас се говореше, освен от неколцина правозащитници и още по-малко журналисти. За бежанци у нас започна да се говори повече през есента на 2013 година в контекста на войната в Сирия, и то не с добро. Тогава националисти извършиха поредица от акции срещу бежанци в София, за да я „прочистят“ от тях. В една от тези акции беше пребит почти до смърт българският гражданин Георги Димитров – Метин, защото… приличал на бежанец. След няколко месеца в медикаментозна кома Метин оживя, но последиците от побоя за здравето му са тежки.

През 2015 г., по време на т.нар. „бежанска вълна“ и „бежанска криза“, ксенофобията у нас съвсем ескалира. Държавата ни великодушно се съгласи да приеме… 1352 бежанци – ама ако може, да са жени и деца. Тъй като пътят на бягащите от войната нямаше как да се контролира, у нас дойдоха повече хора. Най-високият им брой беше около 11 000, но тогава „бежанското бреме“ се усещаше като „непоносимо“. Макар огромна част от хората в България да не бяха виждали бежанец на живо.

През тези години държавата беше абдикирала от всякакви хуманитарни ангажименти.

Помощта за търсещите убежище се осъществяваше основно благодарение на мрежи от доброволци. Условията в центровете за бежанци бяха нечовешки, достъпът до медицинска помощ, а понякога и до адвокат – често пъти недостижим лукс. Освен това държавата категорично отказваше да прави разлика между центрове за бежанци, които са на отворен режим, и центрове за задържане на чужденци (евфемистично наричано „настаняване“), които на теория би трябвало да бъдат единствено с цел принудително извеждане от страната. Така мнозина от търсещите убежище се оказваха затворени в нещо като концлагер.

Преобладаващото настроение в България беше, че бежанците, ако изобщо успеят да влязат в страната ни, трябва да са задължително под ключ. Само и само да не са сред нас, да не ги виждаме. През 2014 г. казанлъшкото село Розово категорично отказа да приюти няколко сирийски семейства и не миряса, докато не ги изгони. По същия начин село Калище изхвърли деца бежанци от местното училище. Децата бяха приети в училище в Земен. А през 2017 г. отец Кортези от католическата църква в Белене беше принуден да напусне България след заплахи за живота си – само защото искаше да приюти семейство бежанци.

Бежанците впрочем неусетно станаха „мигранти“. Не само в България.

Думата „бежанци“ означава хора, които бягат от нещо и заради това имат нужда от закрила, докато „мигранти“ е етически неутрален термин. Това са просто лица, които се местят от едно място на друго. Когато говорим за мигранти, а не за бежанци, е по-лесно да се оправдаваме защо не ги искаме при нас. По-лесно е да строим огради, за да не ги допускаме в страната си.

Впрочем още преди да възприемем говоренето за мигранти, у нас „бежанци“ беше станала мръсна дума. „Ловци на бежанци“ като Динко и Перата се изживяваха като национални герои. А виновникът за бежанеца, застрелян от граничен полицай, се оказа… рикоширал куршум.

„Това не са бежанците, с които сме свикнали“,

заяви премиерът Кирил Петков по адрес на бягащите от Украйна. И поясни: „Това са европейци, интелигентни, образовани хора. Ние, както и всички други, сме готови да ги посрещнем. Това не е обичайната бежанска вълна от хора с неясно минало. Никоя от европейските страни не се притеснява от тях.“ Кирил Петков просто изговаря общата нагласа у нас, както впрочем и на други места по света – че има бежанци, които заслужават нашето съпричастие, и такива, които не заслужават. Защото са по-долна категория хора.

От това изказване човек може да остане с извода, че „бежанците, с които сме свикнали“, не са „интелигентни и образовани хора“, освен това са с „неясно минало“. Ако си от Украйна, си интелигентен и образован, а ако си от Близкия изток или Африка – не си. Не е ясно с какви бежанци е „свикнал“ Кирил Петков, но думите му са дискриминационни за чужденците с неевропейски произход в България. Сред тях – немалко професионалисти, инженери, медицински специалисти, дори директор на болница.

Не може да се отрече, че профилът на бежанците от Украйна е по-различен.

Не само защото са европейци и православни християни, а и защото са предимно жени и деца. Жените и децата са най-желаните категории бежанци, но обикновено от страните си бягат млади мъже, които пък са най-нежеланите. Тази особеност има своето логично обяснение. Повечето бежанци напускат страни, в които са преследвани. Когато властта е против местното население, в най-голяма опасност са точно мъжете, защото от тях се очаква, че са способни да се бият против въпросната власт. Освен това за младите мъже е по-лесно да преодолеят всевъзможни препятствия, докато се установят в мирна страна и се опитат (без гаранция, че ще успеят) да изтеглят при себе си жените и децата. Жените са в много по-голяма степен заплашени от насилие, включително сексуално.

Що се отнася до Украйна, докато управлението е в ръцете на украинците, то е на страната на населението. Това има своята цена – украинските мъже са мобилизирани във войната. Ето защо те, за разлика от жените и децата, не могат да избягат, освен ако не дезертират.

Всяка война е ужасна, но ужасът има степени.

Хората в Украйна са бомбардирани, губят домовете си, крият се в бомбоубежища и метростанции. И все пак те живеят в държава, която им предоставя възможности да се скрият и коридори да избягат, когато това е възможно. Други страни, включително България, осигуряват транспорт и топъл прием на бежанците. Които са с „ясно минало“ (по Кирил Петков), защото не им се е наложило да си изхвърлят документите, за да не бъдат разпознати от властите на собствената си страна, от която се опитват да се спасят.

Представете си сега тийнейджър от Афганистан, който бяга от талибаните, защото баща му е политик с леви убеждения. Момчето успява да се отърве от тях с един отрязан пръст. Озовава се в България, изпадайки в безсъзнание, защото каналджията просто го изхвърля от влака. Вместо да му предостави убежище, държавата ни шест години го държи затворен в центрове за задържане. След като момчето не издържа и се самозапалва, в болницата не получава обезболяващи, защото няма здравна осигуровка. А после в центъра за задържане е наказан в изолатора, защото… с опита си да се самоубие е нарушил дисциплината. Познавам този човек. Казва се Джавед Нури. Историята със самозапалването му е отпреди петнайсетина години, но

оттогава страната ни не е станала много по-хуманна към търсещите убежище.

Вълната на солидарност към бягащите от Украйна ми припомня историите на бежанци и чужденци, останали без документи не по своя вина, които познавам:

Ола от Нигерия, дошъл в България толкова малък, че не помни родината си. Останал без документи и изхвърлен от училище, защото баща му, протестантски пастор и учител по английски, нямал пари за такси за подновяване на статута. Без право да работи легално или да завърши образованието си, накрая, след трийсетина години в България, Ола е депортиран в Нигерия, която не познава. Ей така, внезапно, без дори да успее да си събере багажа.

Питър от Замбия, католик, завършил у нас журналистика, но останал тук, защото е преследван в родината си. Вместо да си върне инвестицията, вложена в образованието му, и да се възползва от интелекта му, държавата ни десетилетия наред го държи без документи, защото не е станал бежанец по „правилния“ начин – дошъл е да учи, а не е избягал. Питър получава легален статут, когато по-голямата част от активната му работоспособна възраст вече е в миналото.

Аревик – бяла и православна жена от Армения. На 21-годишна възраст е затворена в центъра за задържане в Бусманци бременна чак до седмия месец. Заради единственото ѝ „прегрешение“, че иска да бъде с момчето, което обича, също с арменски произход. То живее в България без документи, защото не иска да става войник в Армения, която не познава. След дългогодишни перипетии двамата получават легален статут и днес отглеждат децата си в България.

Олег Мавромати, избягал от Русия, където е преследван заради изкуството си. След като България повече от десет години не му предоставя убежище, успява да се установи в друга държава.

Обществото не казваше „браво“, ако се опитваш да помогнеш на такива хора.

Напротив – който се опитваше публично да говори за правата им, получаваше обиди, а понякога и директни заплахи. Тези, които приютяваха чужденци без легален статут в домовете си, го правеха с ясното съзнание, че нарушават закона и ще си имат неприятности, ако ги хванат. Помня и как се търсят лекарства за епилепсия за бежанец без легален статут, който нямаше право дори на личен лекар. Няма законен начин да му намериш тези лекарства. Но за него те са животоспасяващи.

Ако днес България отваря сърцето си за украинците, това не означава, че сме станали по-хуманни. Обществото ни няма да каже „добре дошли“ и на бягащите от Афганистан например. А хуманизъм означава да цениш човека, защото е човек. Всеки човек. Ни повече, ни по-малко. Ако помагаш на някого, защото прилича на теб, това, разбира се, е проява на съпричастност, солидарност и съчувствие. И е добре, че поне на това сме способни. От друга страна обаче,

не се ли крие зад тази съпричастност и страхът да не изпаднем и ние в такова положение?

За разлика от други войни, тази ни засяга пряко. Защото става дума не само за Украйна, а за геополитическата ориентация на страните, които Путин смята, че изконно трябва да са под руско влияние. А сред тях е и България. Засяга ни и защото конфликтът ескалира толкова непредвидимо, че не знаем къде ще падат бомби утре. И дали няма да е над нашите глави. Естествено, че ако това ни се случи, ще искаме и ние да получим човешко отношение и топъл прием в някоя безопасна страна.

Бягащите от Украйна имат нужда от нашата съпричастност и е прекрасно, че толкова хора у нас им оказват подкрепа. Но нека не наричаме желанието да им помогнем хуманизъм. Хуманизмът е голяма хапка за нашето общество. Може да ни приседне.

Заглавна снимка: Стопкадър от репортаж на БТА

Източник

Party Quadrants

2022-03-02

Post Syndicated from original https://xkcd.com/2588/

Single-elimination might provide more drama, but I think we can all agree that a comprehensive numerical scoring system will let us better judge the party's winner.

[$] A Debian GR on secret voting—and more

2022-03-02

Post Syndicated from original https://lwn.net/Articles/886403/

Debian has been working on some “constitutional maintenance” of late; a
general resolution (GR) on tweaks to the project’s decision-making processes passed at the end of January. As part of the
discussion surrounding those changes, the question of secret voting came
up; currently, Debian publicly lists every voter for a GR and their ranking of the
options. Another GR has been proposed to change that, but the discussion
has shown that the definition of “secret” is not exactly the same for
everyone. In addition, secret voting is not the only change being proposed.

Queen Latifah | Lift Every Voice | Talks at Google

2022-03-01 Talks at Google

Post Syndicated from Talks at Google original https://www.youtube.com/watch?v=EYau1jze9Ew

Conti Ransomware Group Internal Chats Leaked Over Russia-Ukraine Conflict

2022-03-01 Rapid7

Post Syndicated from Rapid7 original https://blog.rapid7.com/2022/03/01/conti-ransomware-group-internal-chats-leaked-over-russia-ukraine-conflict/

Conti Ransomware Group Internal Chats Leaked Over Russia-Ukraine Conflict

On February 27, Twitter user @ContiLeaks released a trove of chat logs from the ransomware group, Conti – a sophisticated ransomware group whose manual was publicly leaked last year. Ahead of the chat log disclosures, Conti pledged their support for the Russian Government following the Russian invasion of Ukraine. However, a number of members sided with Ukraine, causing strife within the organization. Two days later, Conti posted a second message revising their statement to condemn the war and to strike back only if Russian critical infrastructure is targeted.

At the time of the leak, a file titled `1.tgz` was released on the “AnonFiles” website, containing 14 megabytes of chat logs across 393 JSON files. However, some of the messages were encrypted and could not be read, so the information provided is necessarily incomplete. The remaining files contained internal Conti communications, screenshots of tools, and discussions of their exploits and design processes.

On February 28 and March 1, a bevy of additional files were posted, along with a number of pro-Ukraine tweets. Among both sets of leaked messages, there were a number of usernames and passwords for a variety of accounts. Additionally, user @ContiLeaks shared access details for a number of alleged Conti command and control servers, plus storage servers for stolen files. However, we have not accessed any of the data necessitating access to remote servers or the use of usernames and passwords, and we strongly recommend against doing so.

@ContiLeaks also shared a file that they purport to be the source code for the Conti ransomware but declined to share the password except with “trusted parties.” @ContiLeaks did, however, name one alleged Conti developer, providing their email address and Github. The scale of the leaked information suggests that the leaker is likely either a very senior member of the group or a coalition of disgruntled Conti affiliates.

Conti is a business – and a well-funded one

Much of the discussion within the chat logs concerns fairly mundane things – interviewing potential operators of the group, payment for services, out-of-office messages, gossip, and discussions of products. Based on the leaked chats, the Conti interview process actually looks a lot like a standard technical interview, with coding exercises to be performed hosted on public code repositories, salary negotiations, and the status of ongoing products.

In addition to other financial information related to specific actors, the leaked chats have revealed Conti’s primary Bitcoin address, which contains over two billion USD as of February 28, 2022. Moreover, a conversation on April 9, 2021 between “mango” and “johnyboy77” indicates Russian FSB involvement in some portion of their funding and that the FSB were interested in files from the media outlet Bellingcat on “Navalny” – an apparent reference to Alexei Navalny, the currently imprisoned opposition leader in Russia.

Conti development

Conti seems to operate much like a software company – the chat logs disclose concerns with the development of specific features for targets and a particular difficulty in encrypting very large files. The Conti team also attempted to get demos of popular endpoint detection software with the intent to develop their malware to avoid detection.

Two of the actors, “lemur” and “terry” shared phishing templates (included verbatim in Appendix B at the end of this post) to be used against potential targets. Conti gains initial access in many ways, with phishing a popular line of attack due in part to its relatively high efficacy and low cost. Conti often uses phishing emails to establish a presence on targeted networks.

A screenshot of the Conti control panel was also leaked, showing a number of compromised hosts and a breakdown of the operating systems, antiviruses, user rights, and detailed information about the infected assets.

Further discussions detailed the use of infrastructure against targets, disclosing a number of both known and unknown Conti command and control domains. At the time of this post, only a small number of the previously unknown command and control domains appear to be active. Conversations between two operators, “Stern” and “Bentley” discuss the use of third parties for malicious documents, favoring certain providers over others. They also discuss logistics for how to deliver ransomware without being detected by dynamic analysis. In a conversation between the two back in June of 2021, Stern discloses that Conti wants to start their own cryptocurrency but does not know who to work with. There is no evidence that anything came of this desire, and Conti continues to use Bitcoin for their ransoms.

Other groups assert they are strictly business

In stark contrast to Conti, other groups have made it clear to the public that despite their “business model,” they take no public stance on this crisis. LockBit is remaining aloof from the conflict and made it clear that they intend to operate as usual. Although it is believed that LockBit is a Russian organization, they assert that “we are all simple and peaceful people, we are all Earthlings,” and “for us it is just business and we are all apolitical.” Another ransomware group, ALPHV, claims to be “extremely saddened” by Conti’s pledge of support and condemns Conti. Their message concludes, “The Internet, and even more so its dark side, is not the place for politics.”

Rumors of Conti’s demise have been greatly exaggerated

Conti’s payment and “support” portal is still live, even following the infighting and leaks. Conti has repeatedly proven to be one of the most capable ransomware actors and these chats indicate that the group is well-organized and still very well-funded despite the schism. Any suggestion that these leaks spell the end for Conti is overstated, and we expect that Conti will continue to be a powerful player in the ransomware space.

What you can do

We are keeping an eye on dark web activity related to Conti and other ransomware groups and want to reiterate the following steps for protecting yourself from ransomware:

User education, especially related to well-crafted phishing campaigns
Asset and vulnerability management, including reducing your external attack surface
Multi-factor authentication

Additionally, it is worth ensuring that you are well-guarded against the exploits and malware commonly used by Conti (vulnerabilities provided in Appendix A at the end of this post). Furthermore, security teams should also take some time to review CISA’s recent report on the group. For further discussion on how to protect yourself from ransomware, see our ransomware playbook.

Appendix A – Conti known exploited vulnerabilities

CVE-2017-0143, CVE-2017-0144, CVE-2017-0145, CVE-2017-0146 (MS17-010; EternalBlue/EternalSynergy/EternalChampion)

CVE-2020-1472 (ZeroLogon)

CVE-2021-34527 (PrintNightmare)

CVE-2021-44228 (Log4Shell)

CVE-2021-34473, CVE-2021-34523, CVE-2021-31207 (ProxyShell/ProxyLogon)

Appendix B – Phishing templates

{Greetings|Hello|Good afternoon|Hi|Good day|Greeting|Good morning|Good evening}!
{Here|Right here|In this letter|With this letter} we {send|direct} you {all the|all the necessary|the most important} {documentation|papers|documents|records} {regarding|concerning|relating to} your {payment|deposit payment|last payment} {#|№|No. }НОМЕР ПЛАТЕЖА, right {as we|as we have} {discussed|revealed} {not so long ago|not too long ago|recently|just recently|not long ago}. Please {review the|check the|take a look at} аll {necessary|required|important} {information|data} in the {file attached|attached file}.
Т: {Payment|Deposit payment} {invoice|receipt} {#|№|No. }НОМЕР ИНВОЙСА {prepared|formed}
D: {payment|deposit|dep|paym}_{info|information|data}

{Hello|Greetings|Greetings to you|Good evening|Good morning|Good day|Good afternoon}{!|,|.|}
Your {order|purchase order|online order} was {successfully|correctly|timely} {paid|compensated|covered} by you {yesterday|today|recently}. Your {documentation|docs|papers} and {bank check|receipt|paycheck} {can be found|are listed} in the {attached file|file attached}.
T: {Invoice|Given invoice|Bill} {we|we have|we’ve} {sent|mailed|delivered} to you {is paid|is covered|is processed}.
D: {Purchase order|Order} {verification|approval}

{Hello|Greetings|Greetings to you|Good evening|Good morning|Good day|Good afternoon}{!|,|.|}
{We are contacting you to|This is to|This mail is to} {notify|remind} you {about|regarding} your {debt|unprocessed payment} for {our last|the recent|our recent} {contract|agreement}. All {compensation|payment} {data|information}, {agreement|contract} and prepared legal {documents|documentation} {can be found|are located} in the {file attached|attached file}.
T: {Missing|Additional} payment {information|details|info} reminder
D: {Contract|Agreement} 2815/2 {case|claim}

{Hello|Greetings|Greetings to you|Good evening|Good morning|Good day|Good afternoon}{!|,|.|}
{Your payment|Your advance payment|Your obligatory payment|Payment you sent|Payment you made} was {successfully|correctly|timely|properly} {achieved|accomplished|approved|affirmed|received|obtained|collected|processed}. All {required documentation|necessary documents|important documentation|documents you need|details that can be important|essential documents} {can be found|you can find} in the {attached file|file attached}.
T: {Invoicing|Invoice|Agreement|Contract|Payment} {info|data|information|details}
D: {Receipt|Bill} {id|ID|Number|number|No.|No.|No|#|##} 3212-inv8

{Greetings|Hello|Good day|Good afternoon}{!|,|}
{Thank you for|We are thankful for|We are grateful for|Many thanks for} {your|your recent} {on-line order|purchase order|order}. {We|Our financiers have|Our team has|We have|Our shop has} {received|collected|processed|checked} your {payment|advance payment|money transfer|funds transfer} НОМЕР ПЕРЕВОДА. Now we {are and ready to|begin to} {pack|prepare|compose} your {shipment|order|box}. Your {parcel|packet|shipment|box} {will|is going to|would} {arrive|be delivered} to {you|your residence} within {4|5|6|four|five|six} {days|business days}.
{Total|Full|Whole} {order|purchase|payment} sum: СУММА
You {can find|will find} {all|full} {relative information|order info|order and payment details} and your {receipt|check} НОМЕР ЧЕКА {in|in the} {attached file|file attached}.
{Thank you!|Have a nice day!}
ТЕМЫ: Your {order|purchase|on-line order|last order} НОМЕР ЗАКАЗА payment {processed|obtained|received}
АТТАЧИ:
ord_conf
full.details
compl_ord_7847
buyer_auth_doc
info_summr
customer_docs
spec-ed_info

Additional reading

NEVER MISS A BLOG

Get the latest stories, expertise, and news about security today.

Audit AWS service events with Amazon EventBridge and Amazon Kinesis Data Firehose

2022-03-01 Anand Shah

Post Syndicated from Anand Shah original https://aws.amazon.com/blogs/big-data/audit-aws-service-events-with-amazon-eventbridge-and-amazon-kinesis-data-firehose/

Amazon EventBridge is a serverless event bus that makes it easy to build event-driven applications at scale using events generated from your applications, integrated software as a service (SaaS) applications, and AWS services. Many AWS services generate EventBridge events. When an AWS service in your account emits an event, it goes to your account’s default event bus.

The following are a few event examples:

Amazon EC2 Auto Scaling events
Amazon CloudWatch alarm status change events
Events generated from developer tools on AWS services like AWS CodeCommit and AWS CodeDeploy
AWS Step Functions state change events
Analytics on AWS services like AWS Glue, Amazon Redshift, and Amazon EMR

By default, these AWS service-generated events are transient and therefore not retained. This post shows how you can forward AWS service-generated events or custom events to Amazon Simple Storage Service (Amazon S3) for long-term storage, analysis, and auditing purposes using EventBridge rules and Amazon Kinesis Data Firehose.

Solution overview

In this post, we provide a working example of AWS service-generated events ingested to Amazon S3. To make sure we have some service events available in default event bus, we use Parameter Store, a capability of AWS Systems Manager to store new parameters manually. This action generates a new event, which is ingested by the following pipeline.

Architecture Diagram

The pipeline includes the following steps:

AWS service-generated events (for example, a new parameter created in Parameter Store) goes to the default event bus at EventBridge.
The EventBridge rule matches all events and forwards those to Kinesis Data Firehose.
Kinesis Data Firehose delivers events to the S3 bucket partitioned by detail-type and receipt time using its dynamic partitioning capability.
The S3 bucket stores the delivered events, and their respective event schema is registered to the AWS Glue Data Catalog using an AWS Glue crawler.
You query events using Amazon Athena.

Deploy resources using AWS CloudFormation

We use AWS CloudFormation templates to create all the necessary resources for the ingestion pipeline. This removes opportunities for manual error, increases efficiency, and provides consistent configurations over time. The template is also available on GitHub.

Complete the following steps:

Click here to
Acknowledge that the template may create AWS Identity and Access Management (IAM) resources.
Choose Create stack.

The template takes about 10 minutes to complete and creates the following resources in your AWS account:

An S3 bucket to store event data.
A Firehose delivery stream with dynamic partitioning configuration. Dynamic partitioning enables you to continuously partition streaming data in Kinesis Data Firehose by using keys within the data (for example, customer_id or transaction_id) and then deliver the data grouped by these keys into corresponding S3 prefixes.
An EventBridge rule that forwards all events from the default event bus to Kinesis Data Firehose.
An AWS Glue crawler that references the path to the event data in the S3 bucket. The crawler inspects data landed to Amazon S3 and registers tables as per the schema with the AWS Glue Data Catalog.
Athena named queries for you to query the data processed by this example.

Trigger a service event

After you create the CloudFormation stack, you trigger a service event.

On the AWS CloudFormation console, navigate to the Outputs tab for the stack.
Choose the link for the key CreateParameter.

Create Parameter

You’re redirected to the Systems Manager console to create a new parameter.

For Name, enter a name (for example, my-test-parameter).
For Value, enter the test value of your choice (for example, test-value).

My Test parameter

Leave everything else as default and choose Create parameter.

This step saves the new Systems Manager parameter and pushes the parameter-created event to the default EventBridge event bus, as shown in the following code:

{
  "version": "0",
  "id": "6a7e4feb-b491-4cf7-a9f1-bf3703497718",
  "detail-type": "Parameter Store Change",
  "source": "aws.ssm",
  "account": "123456789012",
  "time": "2017-05-22T16:43:48Z",
  "region": "us-east-1",
  "resources": [
    "arn:aws:ssm:us-east-1:123456789012:parameter/foo"
  ],
  "detail": {
    "operation": "Create",
    "name": "my-test-parameter",
    "type": "String",
    "description": ""
  }
}

Discover the event schema

After the event is triggered by saving the parameter, wait at least 2 minutes for the event to be ingested via Kinesis Data Firehose to the S3 bucket. Now complete the following steps to run an AWS Glue crawler to discover and register the event schema in the Data Catalog:

On the AWS Glue console, choose Crawlers in the navigation pane.
Select the crawler with the name starting with S3EventDataCrawler.
Choose Run crawler.

Run Crawler

This step runs the crawler, which takes about 2 minutes to complete. The crawler discovers the schema from all events and registers it as tables in the Data Catalog.

Query the event data

When the crawler is complete, you can start querying event data. To query the event, complete the following steps:

On the AWS CloudFormation console, navigate to the Outputs tab for your stack.
Choose the link for the key AthenaQueries.

Athena Queries

You’re redirected to the Saved queries tab on the Athena console. If you’re running Athena queries for the first time, set up your S3 output bucket. For instructions, see Working with Query Results, Recent Queries, and Output Files.

Search for Blog to find the queries created by this post.
Choose the query Blog – Query Parameter Store Events.

Find Athena Saved Queries

The query opens on the Athena console.

Choose Run query.

You can update the query to search the event you created earlier.

Apply a WHERE clause with the parameter name you selected earlier:

SELECT * FROM "AwsDataCatalog"."eventsdb-randomId"."parameter_store_change"
WHERE detail.name = 'Your event name'

You can also choose the link next to the key CuratedBucket from the CloudFormation stack outputs to see paths and the objects loaded to the S3 bucket from other event sources. Similarly, you can query them via Athena.

Clean up

Complete the following steps to delete your resources and stop incurring costs:

On the AWS CloudFormation console, select the stack you created and choose Delete.
On the Amazon S3 console, find the bucket with the name starting with eventbridge-firehose-blog-curatedbucket.
Select the bucket and choose Empty.
Enter permanently delete to confirm the choice.
Select the bucket again and choose Delete.
Confirm the action by entering the bucket name when prompted.
On the Systems Manager console, go to the parameter store and delete the parameter you created earlier.

Summary

This post demonstrates how to use an EventBridge rule to redirect AWS service-generated events or custom events to Amazon S3 using Kinesis Data Firehose to use for long-term storage, analysis, querying, and audit purposes.

For more information, see the Amazon EventBridge User Guide. To learn more about AWS service events supported by EventBridge, see Events from AWS services.

About the Author

Anand Shah is a Big Data Prototyping Solution Architect at AWS. He works with AWS customers and their engineering teams to build prototypes using AWS analytics services and purpose-built databases. Anand helps customers solve the most challenging problems using the art of the possible technology. He enjoys beaches in his leisure time.

Virginia Postrel | The Fabric of Civilization | Talks at Google

2022-03-01 Talks at Google

Post Syndicated from Talks at Google original https://www.youtube.com/watch?v=QjPx09ZY3ls

Create a low-latency source-to-data lake pipeline using Amazon MSK Connect, Apache Flink, and Apache Hudi

2022-03-01 Ali Alemi

Post Syndicated from Ali Alemi original https://aws.amazon.com/blogs/big-data/create-a-low-latency-source-to-data-lake-pipeline-using-amazon-msk-connect-apache-flink-and-apache-hudi/

During the recent years, there has been a shift from monolithic to the microservices architecture. The microservices architecture makes applications easier to scale and quicker to develop, enabling innovation and accelerating time to market for new features. However, this approach causes data to live in different silos, which makes it difficult to perform analytics. To gain deeper and richer insights, you should bring all your data from different silos into one place.

AWS offers replication tools such as AWS Database Migration Service (AWS DMS) to replicate data changes from a variety of source databases to various destinations including Amazon Simple Storage Service (Amazon S3). But customers who need to sync the data in a data lake with updates and deletes on the source systems still face a few challenges:

It’s difficult to apply record-level updates or deletes when records are stored in open data format files (such as JSON, ORC, or Parquet) on Amazon S3.
In streaming use cases, where jobs need to write data with low latency, row-based formats such as JSON and Avro are best suited. However, scanning many small files with these formats degrades the read query performance.
In use cases where the schema of the source data changes frequently, maintaining the schema of the target datasets via custom code is difficult and error-prone.

Apache Hudi provides a good way to solve these challenges. Hudi builds indexes when it writes the records for the first time. Hudi uses these indexes to locate the files to which an update (or delete) belongs. This enables Hudi to perform fast upsert (or delete) operations by avoiding the need to scan the whole dataset. Hudi provides two table types, each optimized for certain scenarios:

Copy-On-Write (COW) – These tables are common for batch processing. In this type, data is stored in a columnar format (Parquet), and each update (or delete) creates a new version of files during the write.
Merge-On-Read (MOR) – Stores Data using a combination of columnar (for example Parquet) and row-based (for example Avro) file formats and is intended to expose near-real time data.

Hudi datasets stored in Amazon S3 provide native integration with other AWS services. For example, you can write Apache Hudi tables using AWS Glue (see Writing to Apache Hudi tables using AWS Glue Custom Connector) or Amazon EMR (see New features from Apache Hudi available in Amazon EMR). Those approaches require having a deep understanding of Hudi’s Spark APIs and programming skills to build and maintain data pipelines.

In this post, I show you a different way of working with streaming data with minimum coding. The steps in this post demonstrate how to build fully scalable pipelines using SQL language without prior knowledge of Flink or Hudi. You can query and explore your data in multiple data streams by writing familiar SELECT queries. You can join the data from multiple streams and materialize the result to a Hudi dataset on Amazon S3.

Solution overview

The following diagram provides an overall architecture of the solution described in this post. I describe the components and steps fully in the sections that follow.

You use an Amazon Aurora MySQL database as the source and a Debezium MySQL connector with the setup described in the MSK Connect lab as the change data capture (CDC) replicator. This lab walks you through the steps to set up the stack for replicating an Aurora database salesdb to an Amazon Managed Streaming for Apache Kafka (Amazon MSK) cluster, using Amazon MSK Connect with a MySql Debezium source Kafka connector.

In September 2021, AWS announced MSK Connect for running fully managed Kafka Connect clusters. With a few clicks, MSK Connect allows you to easily deploy, monitor, and scale connectors that move data in and out of Apache Kafka and MSK clusters from external systems such as databases, file systems, and search indexes. You can now use MSK Connect for building a full CDC pipeline from many database sources to your MSK cluster.

Amazon MSK is a fully managed service that makes it easy to build and run applications that use Apache Kafka to process streaming data. When you use Apache Kafka, you capture real-time data from sources such as database change events or website clickstreams. Then you build pipelines (using stream processing frameworks such as Apache Flink) to deliver them to destinations such as a persistent storage or Amazon S3.

Apache Flink is a popular framework for building stateful streaming and batch pipelines. Flink comes with different levels of abstractions to cover a broad range of use cases. See Flink Concepts for more information.

Flink also offers different deployment modes depending on which resource provider you choose (Hadoop YARN, Kubernetes, or standalone). See Deployment for more information.

In this post, you use the SQL Client tool as an interactive way of authoring Flink jobs in SQL syntax. sql-client.sh compiles and submits jobs to a long-running Flink cluster (session mode) on Amazon EMR. Depending on the script, sql-client.sh either shows the tabular formatted output of the job in real time, or returns a job ID for long-running jobs.

You implement the solution with the following high-level steps:

Create an EMR cluster.
Configure Flink with Kafka and Hudi table connectors.
Develop your real-time extract, transform, and load (ETL) job.
Deploy your pipeline to production.

Prerequisites

This post assumes you have a running MSK Connect stack in your environment with the following components:

Aurora MySQL hosting a database. In this post, you use the example database salesdb.
The Debezium MySQL connector running on MSK Connect, ending in Amazon MSK in your Amazon Virtual Private Cloud (Amazon VPC).
An MSK cluster running within in a VPC.

If you don’t have an MSK Connect stack, follow the instructions in the MSK Connect lab setup and verify that your source connector replicates data changes to the MSK topics.

You also need the ability to connect directly to the EMR leader node. Session Manager is a feature of AWS Systems Manager that provides you with an interactive one-click browser-based shell window. Session Manager also allows you to comply with corporate policies that require controlled access to managed nodes. See Setting up Session Manager to learn how to connect to your managed nodes in your account via this method.

If Session Manager is not an option, you can also use Amazon Elastic Compute Cloud (Amazon EC2) private key pairs, but you’ll need to launch the cluster in a public subnet and provide inbound SSH access. See Connect to the master node using SSH for more information.

Create an EMR cluster

The latest released version of Apache Hudi is 0.10.0, at the time of writing. Hudi release version 0.10.0 is compatible with Flink release version 1.13. You need Amazon EMR release version emr-6.4.0 and later, which comes with Flink release version 1.13. To launch a cluster with Flink installed using the AWS Command Line Interface (AWS CLI), complete the following steps:

Create a file, configurations.json, with the following content:

[
    {
      "Classification": "flink-conf",
      "Properties": {
        "taskmanager.numberOfTaskSlots":"4"
      }
    }
]

Create an EMR cluster in a private subnet (recommended) or in a public subnet of the same VPC as where you host your MSK cluster. Enter a name for your cluster with the --name option, and specify the name of your EC2 key pair as well as the subnet ID with the --ec2-attributes option. See the following code:

aws emr create-cluster --release-label emr-6.4.0 \
--applications Name=Flink \
--name FlinkHudiCluster \
--configurations file://./configurations.json \
--region us-east-1 \
--log-uri s3://yourLogUri \
--instance-type m5.xlarge \
--instance-count 2 \
--service-role EMR_DefaultRole \ 
--ec2-attributes KeyName=YourKeyName,InstanceProfile=EMR_EC2_DefaultRole, SubnetId=A SubnetID of Amazon MSK VPC

Wait until the cluster state changes to Running.
Retrieve the DNS name of the leader node using either the Amazon EMR console or the AWS CLI.
Connect to the leader node via Session Manager or using SSH and an EC2 private key on Linux, Unix, and Mac OS X.
When connecting using SSH, port 22 must be allowed by the leader node’s security group.
Make sure the MSK cluster’s security group has an inbound rules that accepts traffic from the EMR cluster’s security groups.

Configure Flink with Kafka and Hudi table connectors

Flink table connectors allow you to connect to external systems when programming your stream operations using Table APIs. Source connectors provide access to streaming services including Kinesis or Apache Kafka as a data source. Sink connectors allow Flink to emit stream processing results to external systems or storage services like Amazon S3.

On your Amazon EMR leader node, download the following connectors and save them in the /lib/flink/lib directory:

Source connector – Download flink-connector-kafka_2.11-1.13.1.jar from the Apache repository. The Apache Kafka SQL connector allows Flink to read data from Kafka topics.
Sink connector – Amazon EMR release version emr-6.4.0 comes with Hudi release version 0.8.0. However, in this post you need Hudi Flink bundle connector release version 0.10.0, which is compatible with Flink release version 1.13. Download hudi-flink-bundle_2.11-0.10.0.jar from the Apache repository. It also contains multiple file system clients, including S3A for integrating with Amazon S3.

Develop your real-time ETL job

In this post, you use the Debezium source Kafka connector to stream data changes of a sample database, salesdb, to your MSK cluster. Your connector produces data changes in JSON. See Debezium Event Deserialization for more details. The Flink Kafka connector can deserialize events in JSON format by setting value.format with debezium-json in the table options. This configuration provides the full support for data updates and deletes, in addition to inserts.

You build a new job using Flink SQL APIs. These APIs allow you to work with the streaming data, similar to tables in relational databases. SQL queries specified in this method run continuously over the data events in the source stream. Because the Flink application consumes unbounded data from a stream, the output constantly changes. To send the output to another system, Flink emits update or delete events to the downstream sink operators. Therefore, when you work with CDC data or write SQL queries where the output rows need to update or delete, you must provide a sink connector that supports these actions. Otherwise, the Flink job ends with an error with the following message:

Target Table doesn't support consuming update or delete changes which is produced by {your query statement} …

Launch the Flink SQL client

Start a Flink YARN application on your EMR cluster with the configurations you previously specified in the configurations.json file:

cd /lib/flink && ./bin/yarn-session.sh --detached

After the command runs successfully, you’re ready to write your first job. Run the following command to launch sql-client:

./bin/sql-client.sh

Your terminal window looks like the following screenshot.

Set the job parameters

Run the following command to set the checkpointing interval for this session:

SET execution.checkpointing.interval = 1min;

Define your source tables

Conceptually, processing streams using SQL queries requires interpreting the events as logical records in a table. Therefore, the first step before reading or writing the data with SQL APIs is to create source and target tables. The table definition includes the connection settings and configuration along with a schema that defines the structure and the serialization format of the objects in the stream.

In this post, you create three source tables. Each corresponds to a topic in Amazon MSK. You also create a single target table that writes the output data records to a Hudi dataset stored on Amazon S3.

Replace BOOTSTRAP SERVERS ADDRESSES with your own Amazon MSK cluster information in the 'properties.bootstrap.servers' option and run the following commands in your sql-client terminal:

CREATE TABLE CustomerKafka (
      `event_time` TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,  -- from Debezium format
      `origin_table` STRING METADATA FROM 'value.source.table' VIRTUAL, -- from Debezium format
      `record_time` TIMESTAMP(3) METADATA FROM 'value.ingestion-timestamp' VIRTUAL,
      `CUST_ID` BIGINT,
      `NAME` STRING,
      `MKTSEGMENT` STRING,
       WATERMARK FOR event_time AS event_time
    ) WITH (
      'connector' = 'kafka',
      'topic' = 'salesdb.salesdb.CUSTOMER', -- created by debezium connector, corresponds to CUSTOMER table in Amazon Aurora database. 
      'properties.bootstrap.servers' = '<PLAINTEXT BOOTSTRAP SERVERS ADDRESSES>',
      'properties.group.id' = 'ConsumerGroup1',
      'scan.startup.mode' = 'earliest-offset',
      'value.format' = 'debezium-json'
    );

CREATE TABLE CustomerSiteKafka (
      `event_time` TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,  -- from Debezium format
      `origin_table` STRING METADATA FROM 'value.source.table' VIRTUAL, -- from Debezium format
      `record_time` TIMESTAMP(3) METADATA FROM 'value.ingestion-timestamp' VIRTUAL,
      `CUST_ID` BIGINT,
      `SITE_ID` BIGINT,
      `STATE` STRING,
      `CITY` STRING,
       WATERMARK FOR event_time AS event_time
    ) WITH (
      'connector' = 'kafka',
      'topic' = 'salesdb.salesdb.CUSTOMER_SITE',
      'properties.bootstrap.servers' = '< PLAINTEXT BOOTSTRAP SERVERS ADDRESSES>',
      'properties.group.id' = 'ConsumerGroup2',
      'scan.startup.mode' = 'earliest-offset',
      'value.format' = 'debezium-json'
    );

CREATE TABLE SalesOrderAllKafka (
      `event_time` TIMESTAMP(3) METADATA FROM 'value.source.timestamp' VIRTUAL,  -- from Debezium format
      `origin_table` STRING METADATA FROM 'value.source.table' VIRTUAL, -- from Debezium format
      `record_time` TIMESTAMP(3) METADATA FROM 'value.ingestion-timestamp' VIRTUAL,
      `ORDER_ID` BIGINT,
      `SITE_ID` BIGINT,
      `ORDER_DATE` BIGINT,
      `SHIP_MODE` STRING,
       WATERMARK FOR event_time AS event_time
    ) WITH (
      'connector' = 'kafka',
      'topic' = 'salesdb.salesdb.SALES_ORDER_ALL',
      'properties.bootstrap.servers' = '< PLAINTEXT BOOTSTRAP SERVERS ADDRESSES>',
      'properties.group.id' = 'ConsumerGroup3',
      'scan.startup.mode' = 'earliest-offset',
      'value.format' = 'debezium-json'
    );

By default, sql-client stores these tables in memory. They only live for the duration of the active session. Anytime your sql-client session expires, or you exit, you need to recreate your tables.

Define the sink table

The following command creates the target table. You specify 'hudi' as the connector in this table. The rest of the Hudi configurations are set in the with(…) section of the CREATE TABLE statement. See the full list of Flink SQL configs to learn more. Replace S3URI OF HUDI DATASET LOCATION with your Hudi dataset location in Amazon S3 and run the following code:

CREATE TABLE CustomerHudi (
      `order_count` BIGINT,
      `customer_id` BIGINT,
      `name` STRING,
      `mktsegment` STRING,
      `ts` TIMESTAMP(3),
      PRIMARY KEY (`customer_id`) NOT Enforced
    )
    PARTITIONED BY (`mktsegment`)
    WITH (
      'connector' = 'hudi',
      'write.tasks' = '4',
      'path' = '<S3URI OF HUDI DATASET LOCATION>',
      'table.type' = 'MERGE_ON_READ' --  MERGE_ON_READ table or, by default is COPY_ON_WRITE
    );

Verify the Flink job’s results from multiple topics

For select queries, sql-client submits the job to a Flink cluster, then displays the results on the screen in real time. Run the following select query to view your Amazon MSK data:

SELECT Count(O.order_id) AS order_count,
       C.cust_id,
       C.NAME,
       C.mktsegment
FROM   customerkafka C
       JOIN customersitekafka CS
         ON C.cust_id = CS.cust_id
       JOIN salesorderallkafka O
         ON O.site_id = CS.site_id
GROUP  BY C.cust_id,
          C.NAME,
          C.mktsegment;

This query joins three streams and aggregates the count of customer orders, grouped by each customer record. After a few seconds, you should see the result in your terminal. Note how the terminal output changes as the Flink job consumes more events from the source streams.

Sink the result to a Hudi dataset

To have a complete pipeline, you need to send the result to a Hudi dataset on Amazon S3. To do that, add an insert into CustomerHudi statement in front of the select query:

INSERT INTO customerhudi
SELECT Count(O.order_id),
       C.cust_id,
       C.NAME,
       C.mktsegment,
       Proctime()
FROM   customerkafka C
       JOIN customersitekafka CS
         ON C.cust_id = CS.cust_id
       JOIN salesorderallkafka O
         ON O.site_id = CS.site_id
GROUP  BY C.cust_id,
          C.NAME,
          C.mktsegment;

This time, the sql-client disconnects from the cluster after submitting the job. The client terminal doesn’t have to wait for the results of the job as it sinks its results to a Hudi dataset. The job continues to run on your Flink cluster even after you stop the sql-client session.

Wait a few minutes until the job generates Hudi commit log files to Amazon S3. Then navigate to the location in Amazon S3 you specified for your CustomerHudi table, which contains a Hudi dataset partitioned by MKTSEGMENT column. Within each partition you also find Hudi commit log files. This is because you defined the table type as MERGE_ON_READ. In this mode with the default configurations, Hudi merges commit logs to larger Parquet files after five delta commit logs occur. Refer to Table & Query Types for more information. You can change this setup by changing the table type to COPY_ON_WRITE or specifying your custom compaction configurations.

Query the Hudi dataset

You may also use a Hudi Flink connector as a source connector to read from a Hudi dataset stored on Amazon S3. You do that by running a select statement against the CustomerHudi table, or create a new table with hudi specified for connector. The path must point to an existing Hudi dataset’s location on Amazon S3. Replace S3URI OF HUDI DATASET LOCATION with your location and run the following command to create a new table:

CREATE TABLE `CustomerHudiReadonly` (
      `_hoodie_commit_time` string,
      `_hoodie_commit_seqno` string,
      `_hoodie_record_key` string,
      `order_count` BIGINT,
      `customer_id` BIGINT,
      `name` STRING,
      `mktsegment` STRING,
      `ts` TIMESTAMP(3),
      PRIMARY KEY (`customer_id`) NOT Enforced
    )
    PARTITIONED BY (`mktsegment`)
    WITH (
      'connector' = 'hudi',
      'hoodie.datasource.query.type' = 'snapshot',
      'path' = '<S3URI OF HUDI DATASET LOCATION>',
     'table.type' = 'MERGE_ON_READ' --  MERGE_ON_READ table or, by default is COPY_ON_WRITE
    );

Note the additional column names prefixed with _hoodie_. These columns are added by Hudi during the write to maintain the metadata of each record. Also note the extra 'hoodie.datasource.query.type' read configuration passed in the WITH portion of the table definition. This makes sure you read from the real-time view of your Hudi dataset. Run the following command:

Select * from CustomerHudiReadonly where customer_id <= 5;

The terminal displays the result within 30 seconds. Navigate to the Flink web interface, where you can observe a new Flink job started by the select query (See below for how to find the Flink web interface). It scans the committed files in the Hudi dataset and returns the result to the Flink SQL client.

Use a mysql CLI or your preferred IDE to connect to your salesdb database, which is hosted on Aurora MySQL. Run a few insert statements against the SALES_ORDER_ALL table:

insert into SALES_ORDER_ALL values (29001, 2, now(), 'STANDARD');
insert into SALES_ORDER_ALL values (29002, 2, now(), 'TWO-DAY');
insert into SALES_ORDER_ALL values (29003, 2, now(), 'STANDARD');
insert into SALES_ORDER_ALL values (29004, 2, now(), 'TWO-DAY');
insert into SALES_ORDER_ALL values (29005, 2, now(), 'STANDARD');

After a few seconds, a new commit log file appears in your Hudi dataset on Amazon S3. The Debezium for MySQL Kafka connector captures the changes and produces events to the MSK topic. The Flink application consumes the new events from the topic and updates the customer_count column accordingly. It then sends the changed records to the Hudi connector for merging with the Hudi dataset.

Hudi supports different write operation types. The default operation is upsert, where it initially inserts the records in the dataset. When a record with an existing key arrives in a process, it’s treated as an update. This operation is useful here where you expect to sync your dataset with the source database, and duplicate records are not expected.

Find the Flink web interface

The Flink web interface helps you view a Flink job’s configuration, graph, status, exception errors, resource utilization, and more. To access it, first you need to set up an SSH tunnel and activate a proxy in your browser, to connect to the YARN Resource Manager. After you connect to the Resource Manager, you choose the YARN application that’s hosting your Flink session. Choose the link under the Tracking UI column to navigate to the Flink web interface. For more information, see Finding the Flink web interface.

Deploy your pipeline to production

I recommend using Flink sql-client for quickly building data pipelines in an interactive way. It’s a good choice for experiments, development, or testing your data pipelines. For production environments, however, I recommend embedding your SQL scripts in a Flink Java application and running it on Amazon Kinesis Data Analytics. Kinesis Data Analytics is a fully managed service for running Flink applications; it has built-in auto scaling and fault tolerance features to provide your production applications the availability and scalability they need. A Flink Hudi application with the scripts from this this post is available on GitHub. I encourage you to visit this repo, and compare the differences between running in sql-client and Kinesis Data Analytics.

Clean up

To avoid incurring ongoing charges, complete the following cleanup steps:

Stop the EMR cluster.
Delete the AWS CloudFormation stack you created using the MSK Connect Lab setup.

Conclusion

Building a data lake is the first step to break down data silos and running analytics to gain insights from all your data. Syncing the data between the transactional databases and data files on a data lake isn’t trivial and involves significant effort. Before Hudi added support for Flink SQL APIs, Hudi customers had to have the necessary skills for writing Apache Spark code and running it on AWS Glue or Amazon EMR. In this post, I showed you a new way in which you can interactively explore your data in streaming services using SQL queries, and accelerate the development process for your data pipelines.

To learn more, visit Hudi on Amazon EMR documentation.

About the Author

Ali Alemi is a Streaming Specialist Solutions Architect at AWS. Ali advises AWS customers with architectural best practices and helps them design real-time analytics data systems which are reliable, secure, efficient, and cost-effective. He works backward from customer’s use cases and designs data solutions to solve their business problems. Prior to joining AWS, Ali supported several public sector customers and AWS consulting partners in their application modernization journey and migration to the Cloud.

Zoë Kooyman is the new FSF executive director

2022-03-01

Post Syndicated from original https://lwn.net/Articles/886490/

The Free Software Foundation has announced that Zoë Kooyman will be the
organization’s new executive director.

Kooyman was appointed by the FSF board following a careful
selection process that included a review by a FSF staff committee
and evaluation criteria such as management, fundraising, business
and finance, legal, and technical skills. She succeeds John
Sullivan, who served as executive director for twelve years.

OpenWrt 21.02.2 and 19.07.9 released

2022-03-01

Post Syndicated from original https://lwn.net/Articles/886489/

Versions 21.02.2
and 19.07.9
of the OpenWrt router distribution are available. Both releases include a
number of security fixes. Additionally, 21.02.2 adds support for a set of
new devices, adds a new rpcapd package, and includes various other
enhancements.

QsrSoft launches Digital Huddle Board in 3 months with AWS serverless and Fire devices

2022-03-01 Sushanth Mangalore

Post Syndicated from Sushanth Mangalore original https://aws.amazon.com/blogs/architecture/qsrsoft-launches-digital-huddle-board-in-3-months-with-aws-serverless-and-fire-devices/

QsrSoft is a software as a service (SaaS) company that develops solutions for clients in the restaurant, hospitality, and retail industries to help them achieve operational excellence. QsrSoft has provided these services for more than two decades and now services over 14,000 locations. QsrSoft started using AWS in 2015 and fully migrated all their workloads to AWS by 2016. QsrSoft can innovate rapidly with AWS and use best-in class technologies for cloud-native solutions for their customers.

In QsrSoft’s target industries, it is important to have a way to focus and motivate employees on common objectives. It can be hard to stay on top of ongoing activities and inspire a team towards operational goals. Through client engagement and data collection, QsrSoft identified this as a pressing business challenge that could be solved with technology. After attending an AWS Digital Innovation Program workshop, QsrSoft conceptualized a digital huddle board that connects teams through gamification, communication, recognition, and excellence in shift management. To bring QsrSoft TV to market in the shortest possible time, QsrSoft decided to build using the AWS serverless suite of services.

QsrSoft successfully lowered the barrier of entry when installing a digital huddle board. Using commodity hardware from Amazon devices such as Fire TV Sticks and Fire Smart TVs, QsrSoft released the product as an app in the Amazon App store. Clients can use existing TV screens when rolling out QsrSoft TV, and only have to plug in a Fire TV Stick and pair it with a five-digit code. This blog post describes QsrSoft TV’s architecture and the AWS services employed in building it.

QsrSoft TV architecture

Building on top of their existing microservices architecture and AWS Amplify, a team of three developers brought QsrSoft TV to market in three months. The solution relies heavily on serverless technologies on AWS. Traditionally, in developing a new product, QsrSoft would need to engage several specialized technical resources. Using the fully managed experience of serverless technologies, QsrSoft can focus on delivering business value for the use case. AWS takes care of managing the technology’s hosting and implementation. With serverless, you only pay for what you use, which makes it possible to correlate your costs with the success of your solution.

Figure 1 illustrates the architecture of this solution:

Figure 1. Architecture diagram of QsrSoft TV solution

Digital huddle board app

The customer-facing component of the solution is the application, which runs on Fire devices in restaurants. AWS Amplify is a service that streamlines development of both web applications and native apps. The app is derived from a single-page application (SPA) developed with Vue.js. AWS Amplify provides features such as integrated authentication, CI/CD, and Web Preview. It also provides GraphQL-based endpoints to access Amazon DynamoDB using AWS AppSync. This enables the QsrSoft development team to function autonomously without dependency on the operations or data integration teams. You can connect to the different Amplify backends from the AWS Management Console or with command line interface (CLI) commands. With the Amplify CLI, you can use default categories for the backends or use the AWS Cloud Development Kit (CDK) to customize them.

GraphQL based API layer

The heart of QsrSoft TV is the API that provides the application with its core functionality. QsrSoft built an AWS AppSync endpoint to power QsrSoft TV’s business logic. AWS Amplify provides an easy way to create a secure AWS AppSync API endpoint through integrated authentication and transport layer security (TLS) for in-transit encryption. The development team can first model the data visually in Amplify Studio. Amplify then creates the queries, subscriptions, and mutations. With a single click from the Studio, you can deploy this model to an AWS AppSync API endpoint. The use of annotations permitted the dev team to customize the model further for the application’s needs, such as indexing by key attributes, authorization, and model relationships. This feature of Amplify Studio saved the development team up to 50% of the total API development effort.

Continuous automated deployments and releases

AWS Amplify abstracts the need for a dedicated operations team. This is enabled by AWS Amplify’s fully managed deployment and hosting for full-stack web applications. The development team connected Amplify to the GitHub repository, and in minutes had a complete CI/CD pipeline in place. There was no need to configure any pipelines or handcraft any YAML files. QsrSoft uses Amplify Web Previews, which enabled the product team and beta testers to preview multiple changes and experiments without releasing code to production. While Amplify deployed the Vue.js application, QsrSoft used fastlane to automate the deployment of the Fire TV application into the Amazon App Store. fastlane is an open-source tool that automates tasks like code signing and releasing the Fire TV binary to the Amazon App Store. This enabled QsrSoft to stay true to its automated deployments and infrastructure as code (IaC) practices.

Simplified password-less authentication

With this command line statement, amplify add auth, QsrSoft laid the groundwork for securing their TV application. Behind the scenes, Amplify uses Amazon Cognito to set up a user pool for the app users. QsrSoft TV provides a password-less login experience to the users, by abstracting the need to log in using a username and password. Instead, you use a five-digit code to pair the Fire TV app with a specific location. Amazon Cognito enables this by securing the app with JSON web tokens (JWT).

Automatic data synchronization

The development team focused on data modeling using the visual tools built into Amplify Studio. Amplify provided an AWS AppSync endpoint, so the developers could use GraphQL to interact with Amplify DataStore. Since the data models in the Amplify Studio support DataStore, the development team can now support an app that works offline. Offline mode is vital to any enterprise application, and it engages QsrSoft TV’s users even during an internet outage. Amplify is used to create an AWS AppSync backend with Amazon DynamoDB tables that match the schema created at the application. As the app interacts with the local DataStore, it starts an instance of its Sync Engine, which publishes the data changes by the application to the DynamoDB backend. An additional AWS Lambda-based backend called QORM, aggregates information from QsrSoft’s custom data warehouse implementation based on Amazon S3 and Amazon Aurora.

The scaling and performance provided by DynamoDB and Lambda allowed QsrSoft to scale without extensive planning, as the number of installs increased. This fully serverless application stack is cost-effective and enables QsrSoft to innovate freely. QsrSoft TV onboarded 100 locations in the first week. QsrSoft projects 7,000+ installs in the first year.

Conclusion

AWS is constantly innovating on behalf of our customers like QsrSoft. By leveraging serverless technologies on AWS, QsrSoft accelerated the go-to-market time for QsrSoft TV. Serverless on AWS provides a low barrier to entry for innovation-focused organizations who want to bring an idea to life quickly to provide business value to their customers.

Amazon Fire devices enable software vendors to make their applications available for large-scale distribution. Fire devices are today being used by several thousand households and workplaces worldwide. Many industries can benefit from developing apps for Amazon Fire devices that can be used on large displays and television screens.

Further reading:

Security updates for Tuesday

2022-03-01

Post Syndicated from original https://lwn.net/Articles/886472/

Security updates have been issued by Debian (thunderbird), Oracle (kernel, kernel-container, and ruby:2.5), Red Hat (rh-ruby26-ruby), Slackware (libxml2 and libxslt), SUSE (htmldoc and SUSE Manager Server 4.2), and Ubuntu (mariadb-10.3, mariadb-10.5, policykit-1, qemu, virglrenderer, and webkit2gtk).

THG Podcast: Weird Wars: China vs Sparrows and Pastry War

2022-03-01 The History Guy: History Deserves to Be Remembered

Post Syndicated from The History Guy: History Deserves to Be Remembered original https://www.youtube.com/watch?v=BVikzIJEz-I

Decrypting Hive Ransomware Data

2022-03-01 Bruce Schneier

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/03/decrypting-hive-ransomware-data.html

Nice piece of research:

Abstract: Among the many types of malicious codes, ransomware poses a major threat. Ransomware encrypts data and demands a ransom in exchange for decryption. As data recovery is impossible if the encryption key is not obtained, some companies suffer from considerable damage, such as the payment of huge amounts of money or the loss of important data. In this paper, we analyzed Hive ransomware, which appeared in June 2021. Hive ransomware has caused immense harm, leading the FBI to issue an alert about it. To minimize the damage caused by Hive Ransomware and to help victims recover their files, we analyzed Hive Ransomware and studied recovery methods. By analyzing the encryption process of Hive ransomware, we confirmed that vulnerabilities exist by using their own encryption algorithm. We have recovered the master key for generating the file encryption key partially, to enable the decryption of data encrypted by Hive ransomware. We recovered 95% of the master key without the attacker’s RSA private key and decrypted the actual infected data. To the best of our knowledge, this is the first successful attempt at decrypting Hive ransomware. It is expected that our method can be used to reduce the damage caused by Hive ransomware.

Here’s the flaw:

The cryptographic vulnerability identified by the researchers concerns the mechanism by which the master keys are generated and stored, with the ransomware strain only encrypting select portions of the file as opposed to the entire contents using two keystreams derived from the master key.

The encryption keystream, which is created from an XOR operation of the two keystreams, is then XORed with the data in alternate blocks to generate the encrypted file. But this technique also makes it possible to guess the keystreams and restore the master key, in turn enabling the decode of encrypted files sans the attacker’s private key.

The researchers said that they were able to weaponize the flaw to devise a method to reliably recover more than 95% of the keys employed during encryption.

New – Customer Carbon Footprint Tool

2022-03-01 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-customer-carbon-footprint-tool/

Carbon is the fourth-most abundant element in the universe, and is also a primary component of all known life on Earth. When combined with oxygen it creates carbon dioxide (CO₂). Many industrial activities, including the burning of fossil fuels such as coal and oil, release CO₂ into the atmosphere and cause climate change.

As part of Amazon’s efforts to increase sustainability and reduce carbon emissions, we co-founded The Climate Pledge in 2019. Along with the 216 other signatories to the Pledge, we are committed to reaching net-zero carbon by 2040, 10 years ahead of the Paris Agreement. We are driving carbon out of our business in a multitude of ways, as detailed on our Carbon Footprint page. When I share this information with AWS customers, they respond positively. They now understand that running their applications in AWS Cloud can help them to lower their carbon footprint by 88% (when compared to the enterprise data centers that were surveyed), as detailed in The Carbon Reduction Opportunity of Moving to Amazon Web Services, published by 451 Research.

In addition to our efforts, organizations in many industries are working to set sustainability goals and to make commitments to reach them. In order to help them to measure progress toward their goals they are implementing systems and building applications to measure and monitor their carbon emissions data.

Customer Carbon Footprint Tool
After I share information about our efforts to decarbonize with our customers, they tell me that their organization is on a similar path, and that they need to know more about the carbon footprint of their cloud infrastructure. Today I am happy to announce the new Customer Carbon Footprint Tool. This tool will help you to meet your own sustainability goals, and is available to all AWS customers at no cost. To access the calculator, I open the AWS Billing Console and click Cost & Usage Reports:

Then I scroll down to Customer Carbon Footprint Tool and review the report:

Let’s review each section. The first one allows me to select a time period with month-level granularity, and shows my carbon emissions in summary, geographic, and per-service form. In all cases, emissions are in Metric Tons of Carbon Dioxide Equivalent, abbreviated as MTCO2e:

All of the values in this section reflect the selected time period. In this example (all of which is sample data), my AWS resources emit an estimated 0.3 MTCO2e from June to August of 2021. If I had run the same application in my own facilities instead of in the AWS Cloud, I would have used an additional 0.9 MTCO2e. Of this value, 0.7 MTCO2e was saved due to renewable energy purchases made by AWS, and an additional 0.2 MTCO2e was saved due to the fact that AWS uses resources more efficiently.

I can also see my emissions by geography (all in America for this time period), and by AWS service in this section.

The second section shows my carbon emission statistics on a monthly, quarterly, or annual basis:

The third and final section projects how the AWS path to 100% renewable energy for our data centers will have a positive effect on my carbon emissions over time:

If you are an AWS customer, then you are already benefiting from our efforts to decarbonize and to reach 100% renewable energy usage by 2025, five years ahead of our original target.

You should also take advantage of the new Sustainability Pillar of AWS Well-Architected. This pillar contains six design principles for sustainability in the cloud, and will show you how to understand impact and to get the best utilization from the minimal number of necessary resources, while also reducing downstream impacts.

Things to Know
Here are a couple of important facts to keep in mind:

Regions – The emissions displayed reflect your AWS usage in all commercial AWS regions.

Timing – Emissions are calculated monthly. However, there is a three month delay due to the underlying billing cycle of the electric utilities that supply us with power.

Scope – The calculator shows Scope 1 and Scope 2 emissions, as defined here.

— Jeff;

UI Notify – Your Product Notification User Interface!

2022-03-01 Crosstalk Solutions

Post Syndicated from Crosstalk Solutions original https://www.youtube.com/watch?v=4ulJJkPwZwU

Home Assistant 2022.3 Release Party

2022-03-01 Home Assistant

Post Syndicated from Home Assistant original https://www.youtube.com/watch?v=vBPx7J2DT4g

Hexagonal architecture

Code examples

Integrations

Configuring dependency injection in AWS Lambda

Lambda API controllers

Application configuration

Environment variables

External configuration services

Networking

Conclusion

Заглавна снимка: Alfons Morales / Unsplash

кога станахме такива хуманисти?

През тези години държавата беше абдикирала от всякакви хуманитарни ангажименти.

Бежанците впрочем неусетно станаха „мигранти“. Не само в България.

„Това не са бежанците, с които сме свикнали“,

Не може да се отрече, че профилът на бежанците от Украйна е по-различен.

Всяка война е ужасна, но ужасът има степени.

оттогава страната ни не е станала много по-хуманна към търсещите убежище.

Обществото не казваше „браво“, ако се опитваш да помогнеш на такива хора.

не се ли крие зад тази съпричастност и страхът да не изпаднем и ние в такова положение?

Заглавна снимка: Стопкадър от репортаж на БТА

Conti is a business – and a well-funded one

Conti development

Other groups assert they are strictly business

Rumors of Conti’s demise have been greatly exaggerated

What you can do

Appendix A – Conti known exploited vulnerabilities

Appendix B – Phishing templates

NEVER MISS A BLOG

Solution overview

Deploy resources using AWS CloudFormation

Trigger a service event

Discover the event schema

Query the event data

Clean up

Summary

About the Author

Solution overview

Prerequisites

Create an EMR cluster

Configure Flink with Kafka and Hudi table connectors

Develop your real-time ETL job

Launch the Flink SQL client

Set the job parameters

Define your source tables

Define the sink table

Verify the Flink job’s results from multiple topics

Sink the result to a Hudi dataset

Query the Hudi dataset

Find the Flink web interface

Deploy your pipeline to production

Clean up

Conclusion

About the Author

QsrSoft TV architecture

Digital huddle board app

GraphQL based API layer

Continuous automated deployments and releases

Simplified password-less authentication

Automatic data synchronization

Conclusion

The collective thoughts of the interwebz