Седмицата (29 май – 3 юни)

Post Syndicated from Боряна Телбис original https://www.toest.bg/sedmitsata-29-may-3-yuni/

Седмицата (29 май – 3 юни)

Измина седмица, в която станаха толкова много неща и в същото време не се случи нищо ново. Изумително е как в България успяваме всекидневно и с лекота да докажем това, което учените се опитват да докажат с години – че времето е произволна човешка конструкция и въобще не е задължително да се движи така, както ние си го представяме. Наскоро дори излезе информация за проучване, според което в един момент времето буквално ще спре.

Е, тук то е спряло отдавна. Или е останало едно късче от него, в което сме принудени да се въртим като добитък в егрек и да преживяваме едно и също отново и отново. Избори, неадекватни главни прокурори, безкрайни служебни правителства, президенти с голям глад за власт и желание за партийни проекти, руско влияние, русофилски партии, протести, опити да се влезе в Шенген и еврозоната, показни убийства на знакови имена от подземния свят, записи, снимки – на пачки, на къщи, на любовници, бой в парламента…

Да, дори боят в парламента се повтаря всяка година по същото време. И тази година на 1 юни депутатите се плюха и се блъскаха, и миналата година на 1 юни пак беше същото. Тогава дори имаше деца в пленарната зала, на които тържествено им честитиха празника, преди да им покажат как възрастни мъже са успели да запазят „детското“ в себе си.

В една от новинарските емисии на 1 юни (вече не си спомням коя, защото, както казах, всичко ми се слива) развълнуваната репортерка определи случващото се в парламента като „безпрецедентно“. Но всъщност няма нищо „небивало“ в това, което става в нашето Народно събрание. То си е така по фабрични настройки още от 90-те години, когато като дете обожавах да гледам по телевизията и „Парламентарен контрол“, защото се биеха и раздърпваха от време на време и беше много вълнуващо, и „Милион и едно желания“, защото даваха детското „Мариан Първата“, в което също се биеха, стреляха и преследваха, само че в контекста на Френската революция, а не на посткомунистически преход (ех, какви филмчета имаше едно време!).

Единственото ново, което видях и прочетох тази седмица, беше броят на „Тоест“. Той започна с анализа на Емилия Милчева (вместо традиционно да завърши с него). „Този юмрук на Радев харесваме ли го?“ се появи в резултат на безпрецедентното (ето тук думата си е намясто) връчване на мандата на ПП–ДБ, когато президентът дори не се ръкува с представителите на коалицията и им съобщи на всеослушание с каква неохота им дава тази папка. Със същия успех можеше просто да им я метне на масата и да излезе от залата.

„Тоест“ продължи с текста на Светла Енчева „Моето „Времеубежище“, в който тя споделя своя личен прочит на романа на Георги Господинов и обяснява защо тази книга се намира на „най-горния рафт“ в библиотеката на душата ѝ. Покрай „Времеубежище“ се случи още нещо „безпрецедентно“ – имаше огромна опашка на Панаира на книгата за среща с автора и автограф от него.

Новото в „Тоест“ е и поредицата за важни и все още непреведени на български език книги от различни национални литератури. В първата и втората ѝ част се говореше за нидерландски и норвежки книги, а в третата Йоанна Елми ни води към Украйна с трима съвременни украински автори, докато следим пътя по картата на общата ни културна и историческа памет.

Още едно приятно пътуване гарантира тазседмичният „Тоест“. Правим го с безкрайно вкусния текст на Атанас Шиников за кухнята на средновековните араби, в която се прескачат вкусове, аромати и цветове, но се говори и за местата за хранене като жест на културен избор и социален статус. Не е препоръчително да четете този текст гладни, защото има опасност на по-чувствителните да им се изостри апетитът и да няма откъде да си набавят агнешка главичка или свинска шунка с ароматни треви (да, свинска шунка и в арабския свят!). Затова подложете нещо и четете текста за вдъхновение.

Нови всеки път са и малките парчета наука, които подбира Михаил Ангелов. Тази седмица той обяснява защо се трансплантират митохондрии, за какво ще ни помогнат генетичните редакции в селското стопанство и как така може да се улавя ДНК от въздуха.

Завършвам бюлетина не със състава на новото правителство, защото вероятно вече всички са наясно с него, а с един кратък епизод от сутрешния блок на bTV в петък. След прекъсване на програмата за включване на Николай Денков от парламента водещият казва: „Сега се връщаме на Централните софийски гробища, за да видим променила ли се е там ситуацията.“ Сега, на вас може и да не ви се струва, че ситуацията на едни гробища може да е особено динамична, но – хей – нека не прибързваме с изводите. Оказва се, че точно преди Черешова задушница се започват ремонтни дейности, за да не е гробищният парк в кал и локви. Той не че е в кал и локви от вчера, просто наесен има местни избори и всички избиратели са ни важни. И ако сме буквалисти, особено важни са ни „мъртвите души“ и тежко неактуализираните избирателни списъци.

Но в един по-философски прочит на това „връщане“ към гробищата щяхме да си дадем сметка, че всички рано или късно свършваме там. И няма да е лошо да си го припомняме всекидневно, както и да назначим по един специален придружител към президенти, премиери и министри, който да им шепне Memento mori. Последното не е ново. Ново ще е, ако паралелно с това сменим надписа на Народното събрание, защото „Съединението прави силата“ изглежда като твърде голяма хапка за нашата уста. Моето предложение е епитафията на гроба на Казандзакис: „Не се надявам на нищо, не се страхувам от нищо, свободен съм.“ Може би децата ни ще го сложат този надпис. Ако изобщо дотогава намират смисъл да им се занимава. Впрочем наша работа е да им осигурим смисъла. Само казвам, не се заяждам.

2023-06-02 Взимане на решения в екип

Post Syndicated from original https://vasil.ludost.net/blog/?p=3466

От няколко места виждам вариации на темата, та ми се стори уместно да напиша нещо.

Ще започна с нещо от Камил Галеев, който на моменти има много добри обяснения. Важният момент в това е, че хората, които са на власт/в позиция да взимат решения/вършат нещо, не могат да имат крайни позиции. Вършенето на работа в екип с някого изисква компромиси, и е далеч по-лесно да работиш с хора, които са съгласни с теб и с които може да си сглобиш обща цел, отколкото с такива, които трябва да бъдат насилвани.

От това има няколко посоки на изводите:

Първо, хората/организациите с крайни позиции са такива, които никога не са стигали до свършване на нещо реално. Неспособността да направиш някакъв компромис и да се промениш/пренасочиш, за да се справиш с някаква промяна в обстановката води до това да си по-зле от тези, които намират вариант и да “отпаднеш” по някакъв начин.
(тук има огромна скоба и малко примери, понеже може да се разбира по грешния начин…

Опциите за компромис и промяна на посоката винаги са повече от една. Първи пример, някой на база на горното може да каже “Това значи ли, че Украйна трябва да направи компромис с Русия?” и отговорът там е, че те са направили много компромиси със западния свят/ЕС, за да могат да продължат да се бият. Дали тези ще са им достатъчни, е отделен въпрос.

Друг пример, на скоро обяснявах нуждата от “черна каса” и бях гледан с огромно учудване, че това се налага в наши дни. Моето твърдение е, че на практика няма организация, в която да няма нещо подобно, просто защото има неща, които не могат да се случат без подобен фонд за плащания, които нямат насрещен документ. Имам някакво количество примери от моята практика, за които по очевидни причини няма да разкажа, и само ще отбележа, че всяка достатъчно голяма организация от едно ниво нагоре има нещо подобно, известно като “discretionary фондове”, с които определени хора разполагат по собствено усмотрение за благото на организацията. С тях се злоупотребява, но ползите от тях надминават загубите.
Чувал съм твърдение, че има организации без такива, но тогава тези неща се случват, като определени хора поемат тези разходи от собствения си джоб. В повечето НПО-та е така и за това в счетоводството на доста от тях може да се намери “и за <нещоси> на тоя член му платихме толкова пари”, като нещото е от типа на “да стои и да изглежда добре”
край на скобата)

Второ, съвсем нормално и здравословно в една организация е да има някакво количество спорове и да се достига до решение с дискусия. Мисля си, че сме супер повредени от много години на “един спасител, който знае всичко”, но очакването да има един-единствен диктатор/ръководител/авторитет и той да е прав за всичко е идиотщина и като цяло сбъркано. Смислените лидери (думата не ми харесва, но не ми хрумва по-добра) не взимат решенията ей-така, без да се консултират, и дори в повечето случаи работата им не е да вземат решението, а например да решат кога има някакъв приблизителен консенсус или достатъчно голямо мнозинство за някакво решение, и да прекратят дискусията.
Много полезна е идеята за rough consensus, дошла от IETF. Може да се каже дори, че организации работещи на еквивалента за тях на “rough consensus and running code” (щото не всички живеят на код) са по-здрави и работещи от тези, които не го правят.

За това и аз лично много се зарадвах на записа от заседанието на ПП, на което обсъждат правителството с ГЕРБ – понеже е пример за всичките неща по-горе и за смислен процес по взимане на решение (въпреки че чисто практически изтичането на записа има неприятни последици). Може би след някакво количество години подобни записи ще се използват за обучителни материали 🙂

(изобщо, може да не съм от ПП, но от време на време Кирил Петков свършва нещо, с което ми се издига в очите, което като цяло е огромна рядкост за български политик)

BWH Hotels scales enterprise business intelligence adoption while reducing costs with Amazon QuickSight

Post Syndicated from Joseph W. Landucci original https://aws.amazon.com/blogs/big-data/bwh-hotels-scales-enterprise-business-intelligence-adoption-while-reducing-costs-with-amazon-quicksight/

This is a guest post by Joseph Landucci from BWH Hotels. In their own words, “BWH Hotels is a leading, global hospitality enterprise comprised of three hotel companies, including WorldHotels, Best Western Hotels & Resorts and SureStay Hotels. Our mission is to deliver trusted guest experiences, drive hotel success and foster a caring, inclusive culture that respects the environment. Technology plays a key role in meeting these standards.

My team supports analytics throughout BWH Hotels, including some affiliate and international organizations. Four years ago, we switched to Amazon QuickSight, a fully managed, cloud-native business intelligence (BI) service, and have not looked back. In March 2021, I wrote about how QuickSight had helped us cut costs by over 70%, while also improving our operations. Since then, we have moved fully onto Amazon Web Services (AWS) from IBM Cognos and Oracle, and improved our BI adoption while navigating the effects of the COVID-19 pandemic.

In this post, I discuss how we are using QuickSight today, how it helped us through the pandemic and how we are using it to build the future of our business.

Migrating to AWS and QuickSight

After a six-month proof of concept, we decided to move fully to AWS for our data and analytics needs. Our previous BI solution, IBM Cognos, was prohibitively expensive because we had to pay for a license for each user. After we made the decision to make QuickSight our BI solution, we were able to move our hotels from Cognos to QuickSight within 10 months. Our affiliate users made the migration four months later, followed by our corporate users.

Having used Cognos since 2008, this was not just a lift-and-shift migration. We determined what dashboards and data we needed to migrate from our previous solution to QuickSight. During the migration, we discovered that a significant portion of what was in Cognos wasn’t being used. With AWS Database Migration Service (AWS DMS), we were able to put our data into Amazon Redshift and then build dashboards with QuickSight.

Navigating the pandemic with consumption-based pricing

Shortly after we switched to AWS and QuickSight, the COVID-19 pandemic hit, which impacted the hospitality industry considerably. If we hadn’t made this migration, we would have been stuck with fixed licensing costs. Thanks to the AWS pay-as-you-go pricing model, we were able to only pay for what we needed when demand was lower in 2020. Then when travel increased again, we were able to seamlessly scale back up with demand.

This consumption-based pricing model was one of the reasons we chose QuickSight, which turned out to be a game changer during an unanticipated event. We didn’t have to do any of the scaling down and then jump back up ourselves.

Increased usage and functionality

Even as we have reduced costs, our enterprise BI adoption and usage has gone up almost tenfold. Our corporate users and hoteliers find QuickSight much more intuitive than Cognos. We have 24,000 registered users and 8,000 monthly active users within BWH Hotels.

With greater usage comes greater responsibility for governance and training. When we began the move to AWS, we had a weekly governance meeting, which we have now moved to biweekly. In this governance meeting, we determined how we were going to standardize data handling and set rules. These meetings also gave us the opportunity to provide training on the new tools we were using, like QuickSight. Finally, we also had learning sessions which team members were able to access virtually. Now, whenever a new team begins to use QuickSight, we have 3–4 years’ worth of training material to help them get up to speed. We have also found the training resources provided by QuickSight to be valuable.

Improving our business with QuickSight

The BWH Hotels Revenue Management team was an early adopter of QuickSight. We have around 20–30 specific revenue management dashboards used both by corporate and hoteliers worldwide. Our sales team has implemented QuickSight to perform recency, frequency and monetary (RFM) analysis to determine which customers it should prioritize. With these dashboards, our sales teams at both the corporate and property levels are able to score customer and corporate accounts so that they can optimize how they use their time. We have also begun to experiment with sentiment analysis based on call center data that is fed into Amazon Transcribe. Gathering a multitude of data and then visualizing it in QuickSight has made us a much nimbler organization.

We are also excited about the value we are able to provide to our hoteliers with QuickSight Paginated Reports for creating billing invoices. With this new feature, our hotel operators no longer need to go onto the credit card processing portal to access transactions. They can access the information they need right within QuickSight.

Building for the future

In the near future, we plan to embed QuickSight into our property management system (PMS), AutoClerk. This will provide BWH Hotels and our hoteliers with analytics within their PMS, saving them time and giving them easy access to the information they need to improve their businesses. By including analytics, AutoClerk will provide incredible value to customers.

By switching to QuickSight, BWH Hotels has provided more value to our hoteliers – increasing adoption, saving money and building for the future.

To learn more about how QuickSight can help your business with dashboards, reports and more, visit Amazon QuickSight.


About the Author
Joseph W. Landucci serves as Director of Technology Management at BWH Hotels where he has spent over eight years building and evolving their data management strategy. In this role he is responsible for the management and strategic direction of six product teams: Database, Revenue Management Systems, Data Services, Loyalty Applications, Loyalty Automation and Enterprise Analytics.

New eBook: 5 Keys to Secure Enterprise Messaging

Post Syndicated from Anne Grahn original https://aws.amazon.com/blogs/security/new-ebook-5-keys-to-secure-enterprise-messaging/

AWS is excited to announce a new eBook, 5 Keys to Secure Enterprise Messaging. The new eBook includes best practices for addressing the security and compliance risks associated with messaging apps.

An estimated 3.09 billion mobile phone users access messaging apps to communicate, and this figure is projected to grow to 3.51 billion users in 2025.

Legal and regulatory requirements for data protection, privacy, and data retention have made protecting business communications a priority for organizations across the globe. Although consumer messaging apps are convenient and support real-time communication with colleagues, customers, and partners, they often lack the robust security and administrative controls many businesses require.

The eBook details five keys to secure enterprise messaging that balance people, process, and technology.

We encourage you to read the eBook, and learn about:

  • Establishing messaging policies and guidelines that are effective for your workforce
  • Training employees to use messaging apps in a way that doesn’t increase organizational risk
  • Building a security-first culture
  • Using true end-to-end encryption (E2EE) to secure communications
  • Retaining data to help meet requirements, without exposing it to outside parties

Download 5 Keys to Secure Enterprise Messaging.

 
If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security news? Follow us on Twitter.

Anne Grahn

Anne Grahn

Anne is a Senior Worldwide Security GTM Specialist at AWS based in Chicago. She has more than a decade of experience in the security industry, and focuses on effectively communicating cybersecurity risk. She maintains a Certified Information Systems Security Professional (CISSP) certification.

[$] Memory-management documentation and development process

Post Syndicated from original https://lwn.net/Articles/932398/

As the 2023 Linux Storage, Filesystem,
Memory-Management and BPF Summit
neared its conclusion, two sessions
were held in the memory-management track on process-oriented topics. Mike
Rapoport ran a session on memory-management documentation (or the lack
thereof), while Andrew Morton talked about the state of the subsystem’s
development process in general. Both sessions were relatively brief and did
not foreshadow substantial changes to come.

Metasploit Weekly Wrap-Up

Post Syndicated from Christopher Granleese original https://blog.rapid7.com/2023/06/02/metasploit-weekly-wrap-up-12/

AD CS certificate templates

Metasploit Weekly Wrap-Up

Our very own Spencer McIntyre has developed a new module that allows for creating, reading, updating and deleting certificate template objects from Active Directory.

ESC4 Exploitation

These changes notably enables the exploitation of the technique identified as ESC4 whereby an attacker that has access to modify the certificate template object in LDAP can change it to set the CT_FLAG_ENROLLEE_SUPPLIES_SUBJECT flag in the mspki-certificate-name-flag field to enable exploitation of ESC1. Exploiting this scenario would be a three step process:

  1. Use this module update a certificate template that you have rights to modify
  2. Use the icpr_cert module to exploit ESC1 by specifying a privileged user in the ALT_UPN field
  3. Restore the certificate template that was replaced in step 1 with the backup that was automatically created

SDDL

When the user updates the certificate template, the nTSecurityDescriptor field is overwritten with one that provides all access to all authenticated users. This means it’s critical that the template be restored when the operator is finished. A backup is created every time the template is read, but it’s not restored automatically because the actions taken once the module has completed will likely involve another module such as icpr_cert.

The existing MsDtypSecurityDescriptor class has a new .from_sddl_text method to create a new instance from Microsoft’s (relatively) human-readable Security Descriptor Definition Language. This means the SID in the ACEs can be specified by copying the included template file and changing it to whatever the user would like. They could for example set it to the SID of the current user, or the domain admins group, etc.

New module content (2)

AD CS Certificate Template Management

Authors: Lee Christensen, Oliver Lyak, Spencer McIntyre, and Will Schroeder
Type: Auxiliary
Pull request: #17965 contributed by zeroSteiner

Description: This adds an auxiliary module that can create, read, update, and delete certificate template objects from Active Directory.

Sudoedit Extra Arguments Priv Esc

Authors: Matthieu Barjole, Victor Cutillas, and h00die
Type: Exploit
Pull request: #17929 contributed by h00die
AttackerKB reference: CVE-2023-22809

Description: This adds an exploit for CVE-2023-22809, an LPE within sudoedit. The exploit currently only supports Ubuntu 22.04 and 22.10.

Enhancements and features (1)

  • #17989 from cgranleese-r7 – The auxiliary/admin/kerberos/inspect_ticket and auxiliary/admin/kerberos/forge_ticket modules have been updated to visually represent the decoded binary values of the Kerberos ticket fields

Bugs fixed (4)

  • #18009 from cgranleese-r7 – This PR updates the msfdb commands to no longer enable the web services as default. The web service will now be enabled with the web service flag: --msf-data-service <NAME>.
  • #18010 from adfoster-r7 – Fix edgecase crash when running smb_login with Kerberos auth activated
  • #18015 from distortedsignal – Deletes a dead link from the Using Metasploit page
  • #18024 from zgoldman-r7 – This PR fixes an issue with credentials being normalized to lowercase inconsistently, causing collisions with uppercase data. Relevant credentials are now automatically normalized to lowercase on insert and lookup.

Documentation

You can find the latest Metasploit documentation on our docsite at docs.metasploit.com.

Get it

As always, you can update to the latest Metasploit Framework with msfupdate
and you can get more details on the changes since the last blog post from
GitHub:

If you are a git user, you can clone the Metasploit Framework repo (master branch) for the latest.
To install fresh without using git, you can use the open-source-only Nightly Installers or the
binary installers (which also include the commercial edition).

Beelink EQ12 Pro Review Intel i3-N305 Alder Lake-N is Amazing

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/beelink-eq12-pro-review-intel-i3-n305-alder-lake-n-is-amazing/

The Beelink EQ12 Pro is our first look at the Intel Core i3 N305. 8-core Alder Lake-N shows just how much E-cores can do, but there is more

The post Beelink EQ12 Pro Review Intel i3-N305 Alder Lake-N is Amazing appeared first on ServeTheHome.

Migrate from Google BigQuery to Amazon Redshift using AWS Glue and Custom Auto Loader Framework

Post Syndicated from Tahir Aziz original https://aws.amazon.com/blogs/big-data/migrate-from-google-bigquery-to-amazon-redshift-using-aws-glue-and-custom-auto-loader-framework/

Amazon Redshift is a widely used, fully managed, petabyte-scale cloud data warehouse. Tens of thousands of customers use Amazon Redshift to process exabytes of data every day to power their analytic workloads. Customers are looking for tools that make it easier to migrate from other data warehouses, such as Google BigQuery, to Amazon Redshift to take advantage of the service price-performance, ease of use, security, and reliability.

In this post, we show you how to use AWS native services to accelerate your migration from Google BigQuery to Amazon Redshift. We use AWS Glue, a fully managed, serverless, ETL (extract, transform, and load) service, and the Google BigQuery Connector for AWS Glue (for more information, refer to Migrating data from Google BigQuery to Amazon S3 using AWS Glue custom connectors). We also add automation and flexibility to simplify migration of multiple tables to Amazon Redshift using the Custom Auto Loader Framework.

Solution overview

The solution provides a scalable and managed data migration workflow to migrate data from Google BigQuery to Amazon Simple Storage Service (Amazon S3), and then from Amazon S3 to Amazon Redshift. This pre-built solution scales to load data in parallel using input parameters.

The following architecture diagram shows how the solution works. It starts with setting up the migration configuration to connect to Google BigQuery, then converts the database schemas, and finally migrates the data to Amazon Redshift.

Architecture diagram showing how the solution works. It starts with setting-up the migration configuration to connect to Google BigQuery, then convert the database schemas, and finally migrate the data to Amazon Redshift.

The workflow contains the following steps:

  1. A configuration file is uploaded to an S3 bucket you have chosen for this solution. This JSON file contains the migration metadata, namely the following:
    • A list of Google BigQuery projects and datasets.
    • A list of all tables to be migrated for each project and dataset pair.
  2. An Amazon EventBridge rule triggers an AWS Step Functions state machine to start migrating the tables.
  3. The Step Functions state machine iterates on the tables to be migrated and runs an AWS Glue Python shell job to extract the metadata from Google BigQuery and store it in an Amazon DynamoDB table used for tracking the tables’ migration status.
  4. The state machine iterates on the metadata from this DynamoDB table to run the table migration in parallel, based on the maximum number of migration jobs without incurring limits or quotas on Google BigQuery. It performs the following steps:
    • Runs the AWS Glue migration job for each table in parallel.
    • Tracks the run status in the DynamoDB table.
    • After the tables have been migrated, checks for errors and exits.
  5. The data exported from Google BigQuery is saved to Amazon S3. We use Amazon S3 (even though AWS Glue jobs can write directly to Amazon Redshift tables) for a few specific reasons:
    • We can decouple the data migration and the data load steps.
    • It offers more control on the load steps, with the ability to reload the data or pause the process.
    • It provides fine-grained monitoring of the Amazon Redshift load status.
  6. The Custom Auto Loader Framework automatically creates schemas and tables in the target database and continuously loads data from Amazon S3 to Amazon Redshift.

A few additional points to note:

  • If you have already created the target schema and tables in the Amazon Redshift database, you can configure the Custom Auto Loader Framework to not automatically detect and convert the schema.
  • If you want more control over converting the Google BigQuery schema, you can use the AWS Schema Conversion Tool (AWS SCT). For more information, refer to Migrate Google BigQuery to Amazon Redshift using AWS Schema Conversion tool (SCT).
  • As of this writing, neither the AWS SCT nor Custom Auto Loader Framework support the conversion of nested data types (record, array and struct). Amazon Redshift supports semistructured data using the Super data type, so if your table uses such complex data types, then you need to create the target tables manually.

To deploy the solution, there are two main steps:

  1. Deploy the solution stack using AWS CloudFormation.
  2. Deploy and configure Custom Auto Loader Framework to load files from Amazon S3 to Amazon Redshift.

Prerequisites

Before getting started, make sure you have the following:

In this example, we named the file bq-mig-config.json

    1. Configure your Google account.
    2. Create an IAM role for AWS Glue (and note down the name of the IAM role).
    3. Subscribe to and activate the Google BigQuery Connector for AWS Glue.

Deploy the solution using AWS CloudFormation

To deploy the solution stack using AWS CloudFormation, complete the following steps:

  1. Choose Launch Stack:

This template provisions the AWS resources in the us-east-1 Region. If you want to deploy to a different Region, download the template bigquery-cft.yaml and launch it manually: on the AWS CloudFormation console, choose Create stack with new resources and upload the template file you downloaded.

The list of provisioned resources is as follows:

    • An EventBridge rule to start the Step Functions state machine on the upload of the configuration file.
    • A Step Functions state machine that runs the migration logic. The following diagram illustrates the state machine.
      Diagram representing the state machine deployed by the solution stack.
    • An AWS Glue Python shell job used to extract the metadata from Google BigQuery. The metadata will be stored in an DynamoDB table, with a calculated attribute to prioritize the migration job. By default, the connector creates one partition per 400 MB in the table being read (before filtering). As of this writing, the Google BigQuery Storage API has a maximum quota for parallel read streams, so we set the limit for worker nodes for tables larger than 400 GB. We also calculate the max number of jobs that can run in parallel based on those values.
    • An AWS Glue ETL job used to extract the data from each Google BigQuery table and saves it in Amazon S3 in Parquet format.
    • A DynamoDB table (bq_to_s3_tracking) used to store the metadata for each table to be migrated (size of the table, S3 path used to store the migrated data, and the number of workers needed to migrate the table).
    • A DynamoDB table (bq_to_s3_maxstreams) used to store the maximum number of streams per state machine run. This helps us minimize job failures due to limits or quotas. Use the Cloud Formation template to customize the name of the DynamoDB table. The prefix for the DynamoDB table is bq_to_s3.
    • The IAM roles needed by the state machine and AWS Glue jobs.
  1. Choose Next.

Screen caption showing the AWS Cloudformation Create stack page.

  1. For Stack name, enter a name.
  2. For Parameters, enter the parameters listed in the following table, then choose Create.
CloudFormation Template Parameter Allowed Values Description
InputBucketName S3 bucket name

The S3 bucket where the AWS Glue job stores the migrated data.

The data will be actually stored in a folder named s3-redshift-loader-source, which is used by the Custom Auto Loader Framework.

InputConnectionName AWS Glue connection name, the default is glue-bq-connector-24 The name of the AWS Glue connection that is created using the Google BigQuery connector.
InputDynamoDBTablePrefix DynamoDB table name prefix, the default is bq_to_s3 The prefix that will be used when naming the two DynamoDB tables created by the solution.
InputGlueETLJob AWS Glue ETL job name, the default is bq-migration-ETL The name you want to give to the AWS Glue ETL job. The actual script is saved in the S3 path specified in the parameter InputGlueS3Path.
InputGlueMetaJob AWS Glue Python shell job name, the default is bq-get-metadata The name you want to give to AWS Glue Python shell job. The actual script is saved in the S3 path specified in the parameter InputGlueS3Path.
InputGlueS3Path S3 path, the default is s3://aws-glue-scripts-${AWS::Account}-${AWS::Region}/admin/ This is the S3 path in which the stack will copy the scripts for AWS Glue jobs. Remember to replace: ${AWS::Account} with the actual AWS account ID and ${AWS::Region} with the Region you plan to use, or provide your own bucket and prefix in a complete path.
InputMaxParallelism Number of parallel migration jobs to run, the default is 30 The maximum number of tables you want to migrate concurrently.
InputBQSecret AWS Secrets Manager secret name The name of the AWS Secrets Manager secret in which you stored the Google BigQuery credential.
InputBQProjectName Google BigQuery project name The name of your project in Google BigQuery in which you want to store temporary tables; you will need write permissions on the project.
StateMachineName

Step Functions state machine name, the default is

bq-to-s3-migration-state-machine

The name of the Step Functions state machine.
SourceS3BucketName S3 bucket name, the default is aws-blogs-artifacts-public

The S3 bucket where the artifacts for this post are stored.

Do not change the default.

Deploy and configure the Custom Auto Loader Framework to load files from Amazon S3 to Amazon Redshift

The Custom Auto Loader Framework utility makes data ingestion to Amazon Redshift simpler and automatically loads data files from Amazon S3 to Amazon Redshift. The files are mapped to the respective tables by simply dropping files into preconfigured locations on Amazon S3. For more details about the architecture and internal workflow, refer to Custom Auto Loader Framework.

To set up the Custom Auto Loader Framework, complete the following steps:

  1. Choose Launch Stack to deploy the CloudFormation stack in the us-east-1 Region:

  1. On the AWS CloudFormation console, choose Next.
  2. Provide the following parameters to help ensure the successful creation of resources. Make sure you have collected these values beforehand.
Parameter Name Allowed Values Description
CopyCommandSchedule cron(0/5 * ? * * *) The EventBridge rules KickoffFileProcessingSchedule and QueueRSProcessingSchedule are triggered based on this schedule. The default is 5 minutes.
DatabaseName dev The Amazon Redshift database name.
DatabaseSchemaName public The Amazon Redshift schema name.
DatabaseUserName demo The Amazon Redshift user name who has access to run COPY commands on the Amazon Redshift database and schema.
RedshiftClusterIdentifier democluster The Amazon Redshift cluster name.
RedshiftIAMRoleARN arn:aws:iam::7000000000:role/RedshiftDemoRole The Amazon Redshift cluster attached role, which has access to the S3 bucket. This role is used in COPY commands.
SourceS3Bucket Your-bucket-name The S3 bucket where data is located. Use the same bucket you used to store the migrated data as indicated in the previous stack.
CopyCommandOptions delimiter '|' gzip

Provide the additional COPY command data format parameters as follows:

delimiter '|' dateformat 'auto' TIMEFORMAT 'auto'

InitiateSchemaDetection Yes The setting to dynamically detect the schema prior to file upload.

The following screenshot shows an example of our parameters.

Screen capture showing the stack detailes page with the input parameters filled with example values

  1. Choose Create.
  2. Monitor the progress of the Stack creation and wait until it is complete.
  3. To verify the Custom Auto Loader Framework configuration, log in to the Amazon S3 console and navigate to the S3 bucket you provided as a value to the SourceS3Bucket parameter.

You should see a new directory called s3-redshift-loader-source is created.

Screen caption of the Amazon S3 console showing the folder you should be able to see in your S3 bucket.

Test the solution

To test the solution, complete the following steps:

  1. Create the configuration file based on the prerequisites. You can also download the demo file.
  2. To set up the S3 bucket, on the Amazon S3 console, navigate to the folder bq-mig-config in the bucket you provided in the stack.
  3. Upload the config file into it.
  4. To enable EventBridge notifications to the bucket, open the bucket on the console and on the Properties tab, locate Event notifications.
  5. In the Amazon EventBridge section, choose Edit.
  6. Select On, then choose Save changes.

  1. On AWS Step Function console, monitor the run of the state machine.
  2. Monitor the status of the loads in Amazon Redshift. For instructions, refer to Viewing Current Loads.
  3. Open the Amazon Redshift Query Editor V2 and query your data.

Pricing considerations

You might have egress charges for migrating data out of Google BigQuery into Amazon S3. Review and calculate the cost for moving your data on your Google cloud billing console. As of this writing, AWS Glue 3.0 or later charges $0.44 per DPU-hour, billed per second, with a 1-minute minimum for Spark ETL jobs. For more information, see AWS Glue Pricing. With auto scaling enabled, AWS Glue automatically adds and removes workers from the cluster depending on the parallelism at each stage or microbatch of the job run.

Clean up

To avoid incurring future charges, clean up your resources:

  1. Delete the CloudFormation solution stack.
  2. Delete the CloudFormation Custom Auto Loader Framework stack.

Conclusion

In this post, we demonstrated how to build a scalable and automated data pipeline to migrate your data from Google BigQuery to Amazon Redshift. We also highlighted how the Custom Auto Loader framework can automate the schema detection, create tables for your S3 files, and continuously load the files into your Amazon Redshift warehouse. With this approach, you can automate the migration of entire projects (even multiple projects at the time) in Google BigQuery to Amazon Redshift. This helps improve data migration times into Amazon Redshift significantly through the automatic table migration parallelization.

The auto-copy feature in Amazon Redshift simplifies automatic data loading from Amazon S3 with a simple SQL command, users can easily automate data ingestion from Amazon S3 to Amazon Redshift using the Amazon Redshift auto-copy preview feature

For more information about the performance of the Google BigQuery Connector for AWS Glue, refer to Migrate terabytes of data quickly from Google Cloud to Amazon S3 with AWS Glue Connector for Google BigQuery and learn how to migrate a large amount of data (1.9 TB) into Amazon S3 quickly (about 8 minutes).

To learn more about AWS Glue ETL jobs, see Simplify data pipelines with AWS Glue automatic code generation and workflows and Making ETL easier with AWS Glue Studio.


About the Authors

Tahir Aziz is an Analytics Solution Architect at AWS. He has worked with building data warehouses and big data solutions for over 13 years. He loves to help customers design end-to-end analytics solutions on AWS. Outside of work, he enjoys traveling and cooking.

Ritesh Kumar Sinha is an Analytics Specialist Solutions Architect based out of San Francisco. He has helped customers build scalable data warehousing and big data solutions for over 16 years. He loves to design and build efficient end-to-end solutions on AWS. In his spare time, he loves reading, walking, and doing yoga.

Fabrizio Napolitano is a Principal Specialist Solutions Architect for DB and Analytics. He has worked in the analytics space for the last 20 years, and has recently and quite by surprise become a Hockey Dad after moving to Canada.

Manjula Nagineni is a Senior Solutions Architect with AWS based in New York. She works with major financial service institutions, architecting and modernizing their large-scale applications while adopting AWS Cloud services. She is passionate about designing big data workloads cloud-natively. She has over 20 years of IT experience in software development, analytics, and architecture across multiple domains such as finance, retail, and telecom.

Sohaib Katariwala is an Analytics Specialist Solutions Architect at AWS. He has over 12 years of experience helping organizations derive insights from their data.

Open-Source LLMs

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/06/open-source-llms.html

In February, Meta released its large language model: LLaMA. Unlike OpenAI and its ChatGPT, Meta didn’t just give the world a chat window to play with. Instead, it released the code into the open-source community, and shortly thereafter the model itself was leaked. Researchers and programmers immediately started modifying it, improving it, and getting it to do things no one else anticipated. And their results have been immediate, innovative, and an indication of how the future of this technology is going to play out. Training speeds have hugely increased, and the size of the models themselves has shrunk to the point that you can create and run them on a laptop. The world of AI research has dramatically changed.

This development hasn’t made the same splash as other corporate announcements, but its effects will be much greater. It will wrest power from the large tech corporations, resulting in both much more innovation and a much more challenging regulatory landscape. The large corporations that had controlled these models warn that this free-for-all will lead to potentially dangerous developments, and problematic uses of the open technology have already been documented. But those who are working on the open models counter that a more democratic research environment is better than having this powerful technology controlled by a small number of corporations.

The power shift comes from simplification. The LLMs built by OpenAI and Google rely on massive data sets, measured in the tens of billions of bytes, computed on by tens of thousands of powerful specialized processors producing models with billions of parameters. The received wisdom is that bigger data, bigger processing, and larger parameter sets were all needed to make a better model. Producing such a model requires the resources of a corporation with the money and computing power of a Google or Microsoft or Meta.

But building on public models like Meta’s LLaMa, the open-source community has innovated in ways that allow results nearly as good as the huge models—but run on home machines with common data sets. What was once the reserve of the resource-rich has become a playground for anyone with curiosity, coding skills, and a good laptop. Bigger may be better, but the open-source community is showing that smaller is often good enough. This opens the door to more efficient, accessible, and resource-friendly LLMs.

More importantly, these smaller and faster LLMs are much more accessible and easier to experiment with. Rather than needing tens of thousands of machines and millions of dollars to train a new model, an existing model can now be customized on a mid-priced laptop in a few hours. This fosters rapid innovation.

It also takes control away from large companies like Google and OpenAI. By providing access to the underlying code and encouraging collaboration, open-source initiatives empower a diverse range of developers, researchers, and organizations to shape the technology. This diversification of control helps prevent undue influence, and ensures that the development and deployment of AI technologies align with a broader set of values and priorities. Much of the modern internet was built on open-source technologies from the LAMP (Linux, Apache, mySQL, and PHP/PERL/Python) stack—a suite of applications often used in web development. This enabled sophisticated websites to be easily constructed, all with open-source tools that were built by enthusiasts, not companies looking for profit. Facebook itself was originally built using open-source PHP.

But being open-source also means that there is no one to hold responsible for misuse of the technology. When vulnerabilities are discovered in obscure bits of open-source technology critical to the functioning of the internet, often there is no entity responsible for fixing the bug. Open-source communities span countries and cultures, making it difficult to ensure that any country’s laws will be respected by the community. And having the technology open-sourced means that those who wish to use it for unintended, illegal, or nefarious purposes have the same access to the technology as anyone else.

This, in turn, has significant implications for those who are looking to regulate this new and powerful technology. Now that the open-source community is remixing LLMs, it’s no longer possible to regulate the technology by dictating what research and development can be done; there are simply too many researchers doing too many different things in too many different countries. The only governance mechanism available to governments now is to regulate usage (and only for those who pay attention to the law), or to offer incentives to those (including startups, individuals, and small companies) who are now the drivers of innovation in the arena. Incentives for these communities could take the form of rewards for the production of particular uses of the technology, or hackathons to develop particularly useful applications. Sticks are hard to use—instead, we need appealing carrots.

It is important to remember that the open-source community is not always motivated by profit. The members of this community are often driven by curiosity, the desire to experiment, or the simple joys of building. While there are companies that profit from supporting software produced by open-source projects like Linux, Python, or the Apache web server, those communities are not profit driven.

And there are many open-source models to choose from. Alpaca, Cerebras-GPT, Dolly, HuggingChat, and StableLM have all been released in the past few months. Most of them are built on top of LLaMA, but some have other pedigrees. More are on their way.

The large tech monopolies that have been developing and fielding LLMs—Google, Microsoft, and Meta—are not ready for this. A few weeks ago, a Google employee leaked a memo in which an engineer tried to explain to his superiors what an open-source LLM means for their own proprietary tech. The memo concluded that the open-source community has lapped the major corporations and has an overwhelming lead on them.

This isn’t the first time companies have ignored the power of the open-source community. Sun never understood Linux. Netscape never understood the Apache web server. Open source isn’t very good at original innovations, but once an innovation is seen and picked up, the community can be a pretty overwhelming thing. The large companies may respond by trying to retrench and pulling their models back from the open-source community.

But it’s too late. We have entered an era of LLM democratization. By showing that smaller models can be highly effective, enabling easy experimentation, diversifying control, and providing incentives that are not profit motivated, open-source initiatives are moving us into a more dynamic and inclusive AI landscape. This doesn’t mean that some of these models won’t be biased, or wrong, or used to generate disinformation or abuse. But it does mean that controlling this technology is going to take an entirely different approach than regulating the large players.

This essay was written with Jim Waldo, and previously appeared on Slate.com.

EDITED TO ADD (6/4): Slashdot thread.

Security updates for Friday

Post Syndicated from original https://lwn.net/Articles/933576/

Security updates have been issued by Debian (cups and netatalk), SUSE (cups, ImageMagick, installation-images, libvirt, openvswitch, and qemu), and Ubuntu (avahi, cups, linux, linux-aws, linux-aws-hwe, linux-azure, linux-azure-4.15, linux-gcp,
linux-gcp-4.15, linux-hwe, linux-kvm, linux-oracle, linux-snapdragon, linux, linux-aws, linux-azure, linux-azure-5.4, linux-gcp, linux-gcp-5.4,
linux-gke, linux-gkeop, linux-hwe-5.4, linux-ibm, linux-ibm-5.4,
linux-kvm, linux-oracle, linux-oracle-5.4, linux-aws-5.4, linux-bluefield, linux-intel-iotg, and linux-intel-iotg-5.15).

Dynamic data collection with Zaraz Worker Variables

Post Syndicated from Tom Klein original http://blog.cloudflare.com/dynamic-data-collection-with-zaraz-worker-variables/

Dynamic data collection with Zaraz Worker Variables

Bringing dynamic data to the server

Dynamic data collection with Zaraz Worker Variables

Since its inception, Cloudflare Zaraz, the server-side third-party manager built for speed, privacy and security, has strived to offer a way for marketers and developers alike to get the data they need to understand their user journeys, without compromising on page performance. Cloudflare Zaraz makes it easy to transition from traditional client-side data collection based on marketing pixels in users’ browsers, to a server-side paradigm that shares events with vendors from the edge.

When implementing data collection on websites or mobile applications, analysts and digital marketers usually first define the set of interactions and attributes they want to measure, formalizing those requirements along technical specifications in a central document (“tagging plan”). Developers will later implement the required code to make those attributes available for the third party manager to pick it up. For instance, an analyst may want to analyze page views based on an internal name instead of the page title or page pathname. They would therefore define an example “page name” attribute that would need to be made available in the context of the page, by the developer. From there, the analyst would configure the tag management system to pick the attribute’s value before dispatching it to the analytics tool.

Yet, while the above flow works fine in theory, the reality is that analytics data comes from multiple sources, in multiple formats, that do not always fit the initially formulated requirements.

The industry accepted solution, such as Google Tag Manager’s “Custom JavaScript variables” or Adobe’s “Custom Code Data Elements”, was to offer a way for users to dynamically invoke custom JavaScript functions on the client, allowing them to perform cleaning (like removing PII data from the payload before sending it to Google Analytics), transformations (extracting specific product attributes out of a product array) or enrichment (making an API call to grab the current user’s CRM id to stitch user sessions in your analytics tool) to the data before dispatch by the third-party manager.

Dynamic data collection with Zaraz Worker Variables
Example of Google Tag Manager custom javascript variable that aggregates individual items prices from a javascript array of product information. 

Having the ability to run custom JavaScript is a powerful feature that offers a lot of flexibility and yet, was a missing part of Cloudflare Zaraz. While some workarounds existed, it did not really fit with Cloudflare Zaraz’s objective of high-performance. We needed a way for our users to provide custom code to be executed fast, server-side. Quite fast, it was clear that Cloudflare Workers, the globally distributed serverless V8-based JavaScript runtime was the solution.

Worker Variables to the rescue

Cloudflare Zaraz Worker Variables is powered by Cloudflare Workers, our platform for running custom code on the edge, but let’s take a step back and work through how Cloudflare Zaraz is implemented.

When making a request to a website proxied by Cloudflare, a few things will run before making it to your origin. This includes the firewall, DDoS mitigation, caching, and also something called First-Party Workers.

These First-Party Workers are Cloudflare Workers with special permissions. Cloudflare Zaraz is one of them. Our Worker is built in a way that allows variables to be replaced by their contents. Those variables can be used in places where you would be reusing hardcoded text, to make it easier to make changes to all places where it would be used. For example, the name of your site, a secret key, etc:

Dynamic data collection with Zaraz Worker Variables

These variables can then be used in any of Cloudflare Zaraz’s components, by selecting them right from the dashboard as a property, or as part of a component’s settings:

Dynamic data collection with Zaraz Worker Variables

When using a Worker Variable, instead of replacing your variable with a hardcoded string, we instead execute the custom code hosted in your own Cloudflare Worker that you have associated with the variable. The response of which is then being used as the variable’s value. Calling one worker from within the Cloudflare Zaraz Worker is done using Dynamic Dispatch.

In our Cloudflare Zaraz Worker, calling Dynamic Dispatch is very similar to how your regular, everyday worker might do it. From having a binding in our wrangler.toml:

...
unsafe.bindings = [
  { name = "DISPATCHER", type = "dynamic_dispatch", id_prefix = "", first_party = true },
]

To having our code responsible for variables actually call your worker:

if (variable.type === 'worker') {
  // Get the persistent ID of the Worker
  const mutableId = variable.value.mutableId
  // Get the binding for the specific Worker
  const workerBinding = context.env.DISPATCHER.get(mutableId)
  ...
  // Execute the Worker and return the response
  const workerResponse = await workerBinding.fetch(
    new Request(String(url || context.url || 'http://unknown')),
    {
      method: 'POST',
      headers: {
        'Content-type': 'application/json',
      },
      body: JSON.stringify(payload),
    }
  )
  ...
  return workerResponse
}

Benefits of Cloudflare Zaraz Worker Variables

Cloudflare Workers is a world-class solution to build complex applications. Together with Cloudflare Zaraz, we feel that makes it the ideal platform to orchestrate your data workflows:

Build with context: Cloudflare Zaraz automatically shares the context as part of the call to the Cloudflare Zaraz Worker, allowing you to use that data as input to your functions. Cloudflare Zaraz offers a Web API which customers can use to track important events in their users' journeys. Along with the Web API, Zaraz offers ways for users to define custom attributes, called “Track properties” and “Variables”, that allow our customers to provide additional context to the event getting sent to a vendor. The Cloudflare Zaraz context holds every attribute that was tracked by Cloudflare Zaraz as part of the current visitor session along with other generic attributes like the visitors’ device cookies for instance.

Speed: In comparison to manually calling a worker from client-side JavaScript, this saves the roundtrip to the Worker’s HTTP endpoint and gives you access to Cloudflare Zaraz properties, allowing you to work with client-side data right from the edge.

Isolated environment: As the function is executed inside the worker, which lives outside of your visitor browser, it cannot access the DOM or JavaScript runtime from the browser, preventing potential bugs in your Worker’s code from affecting the experience of your user.

When combining Worker Variables with the Custom HTML tool, you also get the benefits of offloading client-side JavaScript to a worker. This improves performance for both AJAX network requests, which can then be executed directly from Cloudflare’s global network, as well as the offloading of resource intensive tasks to a Worker, such as data manipulation or computations. All whilst keeping your API secrets and other sensitive data hidden from the clients, allowing you to only send the results that are actually needed by the client.

Examples walkthrough

Now that you are more familiar with the concept, let’s get to some practical use cases!

We will cover two examples: translating a GTM custom JavaScript variable to a Cloudflare Zaraz Worker variable, and enriching user information with data from an external API.

Translate a GTM Custom JavaScript variable into Cloudflare Zaraz Worker Variable: Let’s take our previous example, Google Tag Manager custom javascript variable that aggregates individual items prices from a JavaScript array of product information.

Dynamic data collection with Zaraz Worker Variables

The function makes use of a “GTM Data Layer Variable” (represented with double curly braces, line 2, “{{DLV – Ecommerce – Purchase – Products }}”). That kind of variable is equivalent to Track Properties in Cloudflare Zaraz land: they are a way to access the value of a custom attribute that you shared with the third party manager. When translating from GTM to Cloudflare Zaraz, one should take care of making sure that such variables have also been translated to their Cloudflare Zaraz counterpart.

Back to our example, let’s say that the GTM variable “DLV – Ecommerce – Purchase – Products” is equal to a track property “products” in Cloudflare Zaraz. Your first step is to parse the Cloudflare Zaraz context ($1), that gives you two objects: client, holding all track properties set in the current visitor context and system that gives you access to some generic properties of the visitor’s device.

You can then reference a specific track property by accessing it from the client object. ($2)

The variable code was aggregating price from product information into a comma-separated string. For this, we can keep the same code. ($3)

A major difference between javascript functions executed in the client and Workers is that the worker should “return” the value as part of a Response object. ($4)

export default {
  async fetch(request, env) {
    // $1 Parse the Zaraz Context object
    const { system, client } = await request.json();

    // $2 Get a reference to the products track property
    const products = client.products;

    // $3 Calculate the sum
    const prices = products.map(p => p.price).join();

    return new Response(prices);
  },
};

Enriching user information with data coming from an API: For this second example, let’s imagine that we want to synchronize user activity online and offline. To do so, we need a common key to reconcile the user journeys. A CRM id looks like an appropriate candidate for that use case. We will obtain this id through the CRM solution API (the fictitious “https://example.com/api/getUserIdFromCookie”) Our primary key, that will be used to lookup the user CRM id, will be taken from a cookie that holds the current user session id.

export default {
  async fetch(request, env) {
    // Parse the Zaraz Context object
    const { system, client } = await request.json();

    // Get the value of the cookie "login-cookie"
    const cookieValue = system.cookies["login-cookie"];

    const userId = await fetch("https://example.com/api/getUserIdFromCookie", {
      method: POST,
      body: cookieValue,
    });

    return new Response(userId);
  },
};

Start using Worker Variables today

Worker Variables are available for all accounts with a paid Workers subscription (starting at $5 / month).

Create a worker

To use a Worker Variable, you first need to create a new Cloudflare Worker. You can do this through the Cloudflare dashboard or by using Wrangler.

To create a new worker in the Cloudflare dashboard:

  1. Log in to the Cloudflare dashboard.
  2. Go to Workers and select Create a Service.
  3. Give a name to your service and choose the HTTP Handler as your starter template.
  4. Click Create Service, and then Quick Edit.

To create a new worker through Wrangler:

1. Start a new Cloudflare Worker project

$ npx wrangler init my-project
$ cd my-project

2. Run your development server

$ npx wrangler dev

3. Start coding

// my-project/index.js || my-project/index.ts
export default {
 async fetch(request) {
   // Parse the Zaraz Context object
   const { system, client } = await request.json();

   return new Response("Hello World!");
 },
};

Configure a Worker Variable

With your Cloudflare Worker freshly configured and published, it is straightforward to configure a Worker Variable:

1. Log in to the Cloudflare dashboard

2. Go to Zaraz > Tools configuration > Variables.

3. Click Create variable.

4. Give your variable a name, choose Worker as the Variable type, and select your newly created Worker.

Dynamic data collection with Zaraz Worker Variables

5. Save your variable.

Use your Worker Variable

It is now time to use your Worker Variable! You can reference your variable as part of a trigger or an action. To set it up for a specific action for instance:

  1. Go to Zaraz > Tools configuration > Tools.
  2. Click Edit next to a tool that you have already configured.
  3. Select an action or add a new one.
  4. Click on the plus sign at the right of the text fields.
  5. Select your Worker Variable from the list.
Dynamic data collection with Zaraz Worker Variables

New – AWS DMS Serverless: Automatically Provisions and Scales Capacity for Migration and Data Replication

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/new-aws-dms-serverless-automatically-provisions-and-scales-capacity-for-migration-and-data-replication/

With the vast amount of data being created today, organizations are moving to the cloud to take advantage of the security, reliability, and performance of fully managed database services. To facilitate database and analytics migrations, you can use AWS Database Migration Service (AWS DMS). First launched in 2016, AWS DMS offers a simple migration process that automates database migration projects, saving time, resources, and money.

Although you can start AWS DMS migration with a few clicks through the console, you still need to do research and planning to determine the required capacity before migrating. It can be challenging to know how to properly scale capacity ahead of time, especially when simultaneously migrating many workloads or continuously replicating data. On top of that, you also need to continually monitor usage and manually scale capacity to ensure optimal performance.

Introducing AWS DMS Serverless
Today, I’m excited to tell you about AWS DMS Serverless, a new serverless option in AWS DMS that automatically sets up, scales, and manages migration resources to make your database migrations easier and more cost-effective.

Here’s a quick preview on how AWS DMS Serverless works:

AWS DMS Serverless removes the guesswork of figuring out required compute resources and handling the operational burden needed to ensure a high-performance, uninterrupted migration. It performs automatic capacity provisioning, scaling, and capacity optimization of migrations, allowing you to quickly begin migrations with minimal oversight.

At launch, AWS DMS Serverless supports Microsoft SQL Server, PostgreSQL, MySQL, and Oracle as data sources. As for data targets, AWS DMS Serverless supports a wide range of databases and analytics services, from Amazon Aurora, Amazon Relational Database Service (Amazon RDS), Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon DynamoDB, and more. AWS DMS Serverless continues to add support for new data sources and targets. Visit Supported Engine Versions to stay updated.

With a variety of sources and targets supported by AWS DMS Serverless, many scenarios become possible. You can use AWS DMS Serverless to migrate databases and help to build modern data strategies by synchronizing ongoing data replications into data lakes (e.g., Amazon S3) or data warehouses (e.g., Amazon Redshift) from multiple, perhaps disparate data sources.

How AWS DMS Serverless Works
Let me show you how you can get started with AWS DMS Serverless. In this post, I migrate my data from a source database running on PostgreSQL to a target MySQL database running on Amazon RDS. The following screenshot shows my source database with dummy data:

As for the target, I’ve set up a MySQL database running in Amazon RDS. The following screenshot shows my target database:

Getting starting with AWS DMS Serverless is similar to how AWS DMS works today. AWS DMS Serverless requires me to complete the setup tasks such as creating a virtual private cloud (VPC) to defining source and target endpoints. If this is your first time working with AWS DMS, you can learn more by visiting Prerequisites for AWS Database Migration Service.

To connect to a data store, AWS DMS needs endpoints for both source and target data stores. An endpoint provides all necessary information including connection, data store type, and location to my data stores. The following image shows an endpoint I’ve created for my target database:

When I have finished setting up the endpoints, I can begin to create a replication by selecting the Create replication button on the Serverless replications page. Replication is a new concept introduced in AWS DMS Serverless to abstract instances and tasks that we normally have in standard AWS DMS. Additionally, the capacity resources are managed independently for each replication.

On the Create replication page, I need to define some configurations. This starts with defining Name, then specifying Source database endpoint and Target database endpoint. If you don’t find your endpoints, make sure you’re selecting database engines supported by AWS DMS Serverless.

After that, I need to specify the Replication type. There are three types of replication available in AWS DMS Serverless:

  • Full load — If I need to migrate all existing data in source database
  • Change data capture (CDC) — If I have to replicate data changes from source to target database.
  • Full load and change data capture (CDC) — If I need to migrate existing data and replicate data changes from source to target database.

In this example, I chose Full load and change data capture (CDC) because I need to migrate existing data and continuously update the target database for ongoing changes on the source database.

In the Settings section, I can also enable logging with Amazon CloudWatch, which makes it easier for me to monitor replication progress over time.

As with standard AWS DMS, in AWS DMS Serverless, I can also configure Selection rules in Table mappings to define filters that I need to replicate from table columns in the source data store.

I can also use Transformation rules if I need to rename a schema or table or add a prefix or suffix to a schema or table.

In the Capacity section, I can set the range for required capacity to perform replication by defining the minimum and maximum DCU (DMS capacity units). The minimum DCU setting is optional because AWS DMS Serverless determines the minimum DCU based on an assessment of the replication workload. During replication process, AWS DMS uses this range to scale up and down based on CPU utilization, connections, and available memory.

Setting the maximum capacity allows you to manage costs by making sure that AWS DMS Serverless never consumes more resources than you have budgeted for. When you define the maximum DCU, make sure that you choose a reasonable capacity so that AWS DMS Serverless can handle large bursts of data transaction volumes. If traffic volume decreases, AWS DMS Serverless scales capacity down again, and you only pay for what you need. For cases in which you want to change the minimum and maximum DCU settings, you have to stop the replication process first, make the changes, and run the replication again.

When I’m finished with configuring replication, I select Create replication.

When my replication is created, I can view more details of my replication and start the process by selecting Start.

After my replication runs for around 40 minutes, I can monitor replication progress in the Monitoring tab. AWS DMS Serverless also has a CloudWatch metric called Capacity utilization, which indicates the use of capacity to run replication according to the range defined as minimum and maximum DCU. The following screenshot shows the capacity scales up in the CloudWatch metrics chart.

When the replication finishes its process, I see the capacity starting to decrease. This indicates that in addition to AWS DMS Serverless successfully scaling up to the required capacity, it can also scale down within the range I have defined.

Finally, all I need to do is verify whether my data has been successfully replicated into the target data store. I need to connect to the target, run a select query, and check if all data has been successfully replicated from the source.

Now Available
AWS DMS Serverless is now available in all commercial regions where standard AWS DMS is available, and you can start using it today. For more information about benefits, use cases, how to get started, and pricing details, refer to AWS DMS Serverless.

Happy migrating!
—Donnie

The collective thoughts of the interwebz