Измина седмица, в която станаха толкова много неща и в същото време не се случи нищо ново. Изумително е как в България успяваме всекидневно и с лекота да докажем това, което учените се опитват да докажат с години – че времето е произволна човешка конструкция и въобще не е задължително да се движи така, както ние си го представяме. Наскоро дори излезе информация за проучване, според което в един момент времето буквално ще спре.
Е, тук то е спряло отдавна. Или е останало едно късче от него, в което сме принудени да се въртим като добитък в егрек и да преживяваме едно и също отново и отново. Избори, неадекватни главни прокурори, безкрайни служебни правителства, президенти с голям глад за власт и желание за партийни проекти, руско влияние, русофилски партии, протести, опити да се влезе в Шенген и еврозоната, показни убийства на знакови имена от подземния свят, записи, снимки – на пачки, на къщи, на любовници, бой в парламента…
Да, дори боят в парламента се повтаря всяка година по същото време. И тази година на 1 юни депутатите се плюха и се блъскаха, и миналата година на 1 юни пак беше същото. Тогава дори имаше деца в пленарната зала, на които тържествено им честитиха празника, преди да им покажат как възрастни мъже са успели да запазят „детското“ в себе си.
В една от новинарските емисии на 1 юни (вече не си спомням коя, защото, както казах, всичко ми се слива) развълнуваната репортерка определи случващото се в парламента като „безпрецедентно“. Но всъщност няма нищо „небивало“ в това, което става в нашето Народно събрание. То си е така по фабрични настройки още от 90-те години, когато като дете обожавах да гледам по телевизията и „Парламентарен контрол“, защото се биеха и раздърпваха от време на време и беше много вълнуващо, и „Милион и едно желания“, защото даваха детското „Мариан Първата“, в което също се биеха, стреляха и преследваха, само че в контекста на Френската революция, а не на посткомунистически преход (ех, какви филмчета имаше едно време!).
Единственото ново, което видях и прочетох тази седмица, беше броят на „Тоест“. Той започна с анализа на Емилия Милчева (вместо традиционно да завърши с него). „Този юмрук на Радев харесваме ли го?“ се появи в резултат на безпрецедентното (ето тук думата си е намясто) връчване на мандата на ПП–ДБ, когато президентът дори не се ръкува с представителите на коалицията и им съобщи на всеослушание с каква неохота им дава тази папка. Със същия успех можеше просто да им я метне на масата и да излезе от залата.
„Тоест“ продължи с текста на Светла Енчева „Моето „Времеубежище“, в който тя споделя своя личен прочит на романа на Георги Господинов и обяснява защо тази книга се намира на „най-горния рафт“ в библиотеката на душата ѝ. Покрай „Времеубежище“ се случи още нещо „безпрецедентно“ – имаше огромна опашка на Панаира на книгата за среща с автора и автограф от него.
Новото в „Тоест“ е и поредицата за важни и все още непреведени на български език книги от различни национални литератури. В първата и втората ѝ част се говореше за нидерландски и норвежки книги, а в третата Йоанна Елми ни води към Украйна с трима съвременни украински автори, докато следим пътя по картата на общата ни културна и историческа памет.
Още едно приятно пътуване гарантира тазседмичният „Тоест“. Правим го с безкрайно вкусния текст на Атанас Шиников за кухнята на средновековните араби, в която се прескачат вкусове, аромати и цветове, но се говори и за местата за хранене като жест на културен избор и социален статус. Не е препоръчително да четете този текст гладни, защото има опасност на по-чувствителните да им се изостри апетитът и да няма откъде да си набавят агнешка главичка или свинска шунка с ароматни треви (да, свинска шунка и в арабския свят!). Затова подложете нещо и четете текста за вдъхновение.
Нови всеки път са и малките парчета наука, които подбира Михаил Ангелов. Тази седмица той обяснява защо се трансплантират митохондрии, за какво ще ни помогнат генетичните редакции в селското стопанство и как така може да се улавя ДНК от въздуха.
Завършвам бюлетина не със състава на новото правителство, защото вероятно вече всички са наясно с него, а с един кратък епизод от сутрешния блок на bTV в петък. След прекъсване на програмата за включване на Николай Денков от парламента водещият казва: „Сега се връщаме на Централните софийски гробища, за да видим променила ли се е там ситуацията.“ Сега, на вас може и да не ви се струва, че ситуацията на едни гробища може да е особено динамична, но – хей – нека не прибързваме с изводите. Оказва се, че точно преди Черешова задушница се започват ремонтни дейности, за да не е гробищният парк в кал и локви. Той не че е в кал и локви от вчера, просто наесен има местни избори и всички избиратели са ни важни. И ако сме буквалисти, особено важни са ни „мъртвите души“ и тежко неактуализираните избирателни списъци.
Но в един по-философски прочит на това „връщане“ към гробищата щяхме да си дадем сметка, че всички рано или късно свършваме там. И няма да е лошо да си го припомняме всекидневно, както и да назначим по един специален придружител към президенти, премиери и министри, който да им шепне Memento mori. Последното не е ново. Ново ще е, ако паралелно с това сменим надписа на Народното събрание, защото „Съединението прави силата“ изглежда като твърде голяма хапка за нашата уста. Моето предложение е епитафията на гроба на Казандзакис: „Не се надявам на нищо, не се страхувам от нищо, свободен съм.“ Може би децата ни ще го сложат този надпис. Ако изобщо дотогава намират смисъл да им се занимава. Впрочем наша работа е да им осигурим смисъла. Само казвам, не се заяждам.
От няколко места виждам вариации на темата, та ми се стори уместно да напиша нещо.
Ще започна с нещо от Камил Галеев, който на моменти има много добри обяснения. Важният момент в това е, че хората, които са на власт/в позиция да взимат решения/вършат нещо, не могат да имат крайни позиции. Вършенето на работа в екип с някого изисква компромиси, и е далеч по-лесно да работиш с хора, които са съгласни с теб и с които може да си сглобиш обща цел, отколкото с такива, които трябва да бъдат насилвани.
От това има няколко посоки на изводите:
Първо, хората/организациите с крайни позиции са такива, които никога не са стигали до свършване на нещо реално. Неспособността да направиш някакъв компромис и да се промениш/пренасочиш, за да се справиш с някаква промяна в обстановката води до това да си по-зле от тези, които намират вариант и да “отпаднеш” по някакъв начин.
(тук има огромна скоба и малко примери, понеже може да се разбира по грешния начин…
Опциите за компромис и промяна на посоката винаги са повече от една. Първи пример, някой на база на горното може да каже “Това значи ли, че Украйна трябва да направи компромис с Русия?” и отговорът там е, че те са направили много компромиси със западния свят/ЕС, за да могат да продължат да се бият. Дали тези ще са им достатъчни, е отделен въпрос.
Друг пример, на скоро обяснявах нуждата от “черна каса” и бях гледан с огромно учудване, че това се налага в наши дни. Моето твърдение е, че на практика няма организация, в която да няма нещо подобно, просто защото има неща, които не могат да се случат без подобен фонд за плащания, които нямат насрещен документ. Имам някакво количество примери от моята практика, за които по очевидни причини няма да разкажа, и само ще отбележа, че всяка достатъчно голяма организация от едно ниво нагоре има нещо подобно, известно като “discretionary фондове”, с които определени хора разполагат по собствено усмотрение за благото на организацията. С тях се злоупотребява, но ползите от тях надминават загубите.
Чувал съм твърдение, че има организации без такива, но тогава тези неща се случват, като определени хора поемат тези разходи от собствения си джоб. В повечето НПО-та е така и за това в счетоводството на доста от тях може да се намери “и за <нещоси> на тоя член му платихме толкова пари”, като нещото е от типа на “да стои и да изглежда добре”
край на скобата)
Второ, съвсем нормално и здравословно в една организация е да има някакво количество спорове и да се достига до решение с дискусия. Мисля си, че сме супер повредени от много години на “един спасител, който знае всичко”, но очакването да има един-единствен диктатор/ръководител/авторитет и той да е прав за всичко е идиотщина и като цяло сбъркано. Смислените лидери (думата не ми харесва, но не ми хрумва по-добра) не взимат решенията ей-така, без да се консултират, и дори в повечето случаи работата им не е да вземат решението, а например да решат кога има някакъв приблизителен консенсус или достатъчно голямо мнозинство за някакво решение, и да прекратят дискусията.
Много полезна е идеята за rough consensus, дошла от IETF. Може да се каже дори, че организации работещи на еквивалента за тях на “rough consensus and running code” (щото не всички живеят на код) са по-здрави и работещи от тези, които не го правят.
За това и аз лично много се зарадвах на записа от заседанието на ПП, на което обсъждат правителството с ГЕРБ – понеже е пример за всичките неща по-горе и за смислен процес по взимане на решение (въпреки че чисто практически изтичането на записа има неприятни последици). Може би след някакво количество години подобни записи ще се използват за обучителни материали 🙂
(изобщо, може да не съм от ПП, но от време на време Кирил Петков свършва нещо, с което ми се издига в очите, което като цяло е огромна рядкост за български политик)
This is a guest post by Joseph Landucci from BWH Hotels. In their own words, “BWH Hotels is a leading, global hospitality enterprise comprised of three hotel companies, including WorldHotels, Best WesternHotels & Resorts and SureStay Hotels. Our mission is to deliver trusted guest experiences, drive hotel success and foster a caring, inclusive culture that respects the environment. Technology plays a key role in meeting these standards.
My team supports analytics throughout BWH Hotels, including some affiliate and international organizations. Four years ago, we switched to Amazon QuickSight, a fully managed, cloud-native business intelligence (BI) service, and have not looked back. In March 2021, I wrote about how QuickSight had helped us cut costs by over 70%, while also improving our operations. Since then, we have moved fully onto Amazon Web Services (AWS) from IBM Cognos and Oracle, and improved our BI adoption while navigating the effects of the COVID-19 pandemic.
In this post, I discuss how we are using QuickSight today, how it helped us through the pandemic and how we are using it to build the future of our business.
Migrating to AWS and QuickSight
After a six-month proof of concept, we decided to move fully to AWS for our data and analytics needs. Our previous BI solution, IBM Cognos, was prohibitively expensive because we had to pay for a license for each user. After we made the decision to make QuickSight our BI solution, we were able to move our hotels from Cognos to QuickSight within 10 months. Our affiliate users made the migration four months later, followed by our corporate users.
Having used Cognos since 2008, this was not just a lift-and-shift migration. We determined what dashboards and data we needed to migrate from our previous solution to QuickSight. During the migration, we discovered that a significant portion of what was in Cognos wasn’t being used. With AWS Database Migration Service (AWS DMS), we were able to put our data into Amazon Redshift and then build dashboards with QuickSight.
Navigating the pandemic with consumption-based pricing
Shortly after we switched to AWS and QuickSight, the COVID-19 pandemic hit, which impacted the hospitality industry considerably. If we hadn’t made this migration, we would have been stuck with fixed licensing costs. Thanks to the AWS pay-as-you-go pricing model, we were able to only pay for what we needed when demand was lower in 2020. Then when travel increased again, we were able to seamlessly scale back up with demand.
This consumption-based pricing model was one of the reasons we chose QuickSight, which turned out to be a game changer during an unanticipated event. We didn’t have to do any of the scaling down and then jump back up ourselves.
Increased usage and functionality
Even as we have reduced costs, our enterprise BI adoption and usage has gone up almost tenfold. Our corporate users and hoteliers find QuickSight much more intuitive than Cognos. We have 24,000 registered users and 8,000 monthly active users within BWH Hotels.
With greater usage comes greater responsibility for governance and training. When we began the move to AWS, we had a weekly governance meeting, which we have now moved to biweekly. In this governance meeting, we determined how we were going to standardize data handling and set rules. These meetings also gave us the opportunity to provide training on the new tools we were using, like QuickSight. Finally, we also had learning sessions which team members were able to access virtually. Now, whenever a new team begins to use QuickSight, we have 3–4 years’ worth of training material to help them get up to speed. We have also found the training resources provided by QuickSight to be valuable.
Improving our business with QuickSight
The BWH Hotels Revenue Management team was an early adopter of QuickSight. We have around 20–30 specific revenue management dashboards used both by corporate and hoteliers worldwide. Our sales team has implemented QuickSight to perform recency, frequency and monetary (RFM) analysis to determine which customers it should prioritize. With these dashboards, our sales teams at both the corporate and property levels are able to score customer and corporate accounts so that they can optimize how they use their time. We have also begun to experiment with sentiment analysis based on call center data that is fed into Amazon Transcribe. Gathering a multitude of data and then visualizing it in QuickSight has made us a much nimbler organization.
We are also excited about the value we are able to provide to our hoteliers with QuickSight Paginated Reports for creating billing invoices. With this new feature, our hotel operators no longer need to go onto the credit card processing portal to access transactions. They can access the information they need right within QuickSight.
Building for the future
In the near future, we plan to embed QuickSight into our property management system (PMS), AutoClerk. This will provide BWH Hotels and our hoteliers with analytics within their PMS, saving them time and giving them easy access to the information they need to improve their businesses. By including analytics, AutoClerk will provide incredible value to customers.
By switching to QuickSight, BWH Hotels has provided more value to our hoteliers – increasing adoption, saving money and building for the future.
To learn more about how QuickSight can help your business with dashboards, reports and more, visit Amazon QuickSight.
About the Author Joseph W. Landucci serves as Director of Technology Management at BWHHotels where he has spent over eight years building and evolving their data management strategy. In this role he is responsible for the management and strategic direction of six product teams: Database, Revenue Management Systems, Data Services, Loyalty Applications, Loyalty Automation and Enterprise Analytics.
AWS is excited to announce a new eBook, 5 Keys to Secure Enterprise Messaging. The new eBook includes best practices for addressing the security and compliance risks associated with messaging apps.
An estimated 3.09 billion mobile phone users access messaging apps to communicate, and this figure is projected to grow to 3.51 billion users in 2025.
Legal and regulatory requirements for data protection, privacy, and data retention have made protecting business communications a priority for organizations across the globe. Although consumer messaging apps are convenient and support real-time communication with colleagues, customers, and partners, they often lack the robust security and administrative controls many businesses require.
The eBook details five keys to secure enterprise messaging that balance people, process, and technology.
We encourage you to read the eBook, and learn about:
Establishing messaging policies and guidelines that are effective for your workforce
Training employees to use messaging apps in a way that doesn’t increase organizational risk
Building a security-first culture
Using true end-to-end encryption (E2EE) to secure communications
Retaining data to help meet requirements, without exposing it to outside parties
As the 2023 Linux Storage, Filesystem,
Memory-Management and BPF Summit neared its conclusion, two sessions
were held in the memory-management track on process-oriented topics. Mike
Rapoport ran a session on memory-management documentation (or the lack
thereof), while Andrew Morton talked about the state of the subsystem’s
development process in general. Both sessions were relatively brief and did
not foreshadow substantial changes to come.
Our very own Spencer McIntyre has developed a new module that allows for creating, reading, updating and deleting certificate template objects from Active Directory.
ESC4 Exploitation
These changes notably enables the exploitation of the technique identified as ESC4 whereby an attacker that has access to modify the certificate template object in LDAP can change it to set the CT_FLAG_ENROLLEE_SUPPLIES_SUBJECT flag in the mspki-certificate-name-flag field to enable exploitation of ESC1. Exploiting this scenario would be a three step process:
Use this module update a certificate template that you have rights to modify
Use the icpr_cert module to exploit ESC1 by specifying a privileged user in the ALT_UPN field
Restore the certificate template that was replaced in step 1 with the backup that was automatically created
SDDL
When the user updates the certificate template, the nTSecurityDescriptor field is overwritten with one that provides all access to all authenticated users. This means it’s critical that the template be restored when the operator is finished. A backup is created every time the template is read, but it’s not restored automatically because the actions taken once the module has completed will likely involve another module such as icpr_cert.
The existing MsDtypSecurityDescriptor class has a new .from_sddl_text method to create a new instance from Microsoft’s (relatively) human-readable Security Descriptor Definition Language. This means the SID in the ACEs can be specified by copying the included template file and changing it to whatever the user would like. They could for example set it to the SID of the current user, or the domain admins group, etc.
New module content (2)
AD CS Certificate Template Management
Authors: Lee Christensen, Oliver Lyak, Spencer McIntyre, and Will Schroeder
Type: Auxiliary
Pull request: #17965 contributed by zeroSteiner
Description: This adds an auxiliary module that can create, read, update, and delete certificate template objects from Active Directory.
Sudoedit Extra Arguments Priv Esc
Authors: Matthieu Barjole, Victor Cutillas, and h00die
Type: Exploit
Pull request: #17929 contributed by h00die
AttackerKB reference: CVE-2023-22809
Description: This adds an exploit for CVE-2023-22809, an LPE within sudoedit. The exploit currently only supports Ubuntu 22.04 and 22.10.
Enhancements and features (1)
#17989 from cgranleese-r7 – The auxiliary/admin/kerberos/inspect_ticket and auxiliary/admin/kerberos/forge_ticket modules have been updated to visually represent the decoded binary values of the Kerberos ticket fields
Bugs fixed (4)
#18009 from cgranleese-r7 – This PR updates the msfdb commands to no longer enable the web services as default. The web service will now be enabled with the web service flag: --msf-data-service <NAME>.
#18010 from adfoster-r7 – Fix edgecase crash when running smb_login with Kerberos auth activated
#18024 from zgoldman-r7 – This PR fixes an issue with credentials being normalized to lowercase inconsistently, causing collisions with uppercase data. Relevant credentials are now automatically normalized to lowercase on insert and lookup.
Documentation
You can find the latest Metasploit documentation on our docsite at docs.metasploit.com.
Get it
As always, you can update to the latest Metasploit Framework with msfupdate
and you can get more details on the changes since the last blog post from
GitHub:
If you are a git user, you can clone the Metasploit Framework repo (master branch) for the latest.
To install fresh without using git, you can use the open-source-only Nightly Installers or the binary installers (which also include the commercial edition).
Amazon Redshift is a widely used, fully managed, petabyte-scale cloud data warehouse. Tens of thousands of customers use Amazon Redshift to process exabytes of data every day to power their analytic workloads. Customers are looking for tools that make it easier to migrate from other data warehouses, such as Google BigQuery, to Amazon Redshift to take advantage of the service price-performance, ease of use, security, and reliability.
The solution provides a scalable and managed data migration workflow to migrate data from Google BigQuery to Amazon Simple Storage Service (Amazon S3), and then from Amazon S3 to Amazon Redshift. This pre-built solution scales to load data in parallel using input parameters.
The following architecture diagram shows how the solution works. It starts with setting up the migration configuration to connect to Google BigQuery, then converts the database schemas, and finally migrates the data to Amazon Redshift.
The workflow contains the following steps:
A configuration file is uploaded to an S3 bucket you have chosen for this solution. This JSON file contains the migration metadata, namely the following:
A list of Google BigQuery projects and datasets.
A list of all tables to be migrated for each project and dataset pair.
The Step Functions state machine iterates on the tables to be migrated and runs an AWS Glue Python shell job to extract the metadata from Google BigQuery and store it in an Amazon DynamoDB table used for tracking the tables’ migration status.
The state machine iterates on the metadata from this DynamoDB table to run the table migration in parallel, based on the maximum number of migration jobs without incurring limits or quotas on Google BigQuery. It performs the following steps:
Runs the AWS Glue migration job for each table in parallel.
Tracks the run status in the DynamoDB table.
After the tables have been migrated, checks for errors and exits.
The data exported from Google BigQuery is saved to Amazon S3. We use Amazon S3 (even though AWS Glue jobs can write directly to Amazon Redshift tables) for a few specific reasons:
We can decouple the data migration and the data load steps.
It offers more control on the load steps, with the ability to reload the data or pause the process.
It provides fine-grained monitoring of the Amazon Redshift load status.
The Custom Auto Loader Framework automatically creates schemas and tables in the target database and continuously loads data from Amazon S3 to Amazon Redshift.
A few additional points to note:
If you have already created the target schema and tables in the Amazon Redshift database, you can configure the Custom Auto Loader Framework to not automatically detect and convert the schema.
As of this writing, neither the AWS SCT nor Custom Auto Loader Framework support the conversion of nested data types (record, array and struct). Amazon Redshift supports semistructured data using the Super data type, so if your table uses such complex data types, then you need to create the target tables manually.
An S3 bucket. If you don’t want to use one of your existing buckets, you can create a new one. Note the name of the bucket to use later when you pass it to the CloudFormation stack as an input parameter.
A configuration file with the list of tables to be migrated. This file should have the following structure:
Create an IAM role for AWS Glue (and note down the name of the IAM role).
Subscribe to and activate the Google BigQuery Connector for AWS Glue.
Deploy the solution using AWS CloudFormation
To deploy the solution stack using AWS CloudFormation, complete the following steps:
Choose Launch Stack:
This template provisions the AWS resources in the us-east-1 Region. If you want to deploy to a different Region, download the template bigquery-cft.yaml and launch it manually: on the AWS CloudFormation console, choose Create stack with new resources and upload the template file you downloaded.
The list of provisioned resources is as follows:
An EventBridge rule to start the Step Functions state machine on the upload of the configuration file.
A Step Functions state machine that runs the migration logic. The following diagram illustrates the state machine.
An AWS Glue Python shell job used to extract the metadata from Google BigQuery. The metadata will be stored in an DynamoDB table, with a calculated attribute to prioritize the migration job. By default, the connector creates one partition per 400 MB in the table being read (before filtering). As of this writing, the Google BigQuery Storage API has a maximum quota for parallel read streams, so we set the limit for worker nodes for tables larger than 400 GB. We also calculate the max number of jobs that can run in parallel based on those values.
An AWS Glue ETL job used to extract the data from each Google BigQuery table and saves it in Amazon S3 in Parquet format.
A DynamoDB table (bq_to_s3_tracking) used to store the metadata for each table to be migrated (size of the table, S3 path used to store the migrated data, and the number of workers needed to migrate the table).
A DynamoDB table (bq_to_s3_maxstreams) used to store the maximum number of streams per state machine run. This helps us minimize job failures due to limits or quotas. Use the Cloud Formation template to customize the name of the DynamoDB table. The prefix for the DynamoDB table is bq_to_s3.
The IAM roles needed by the state machine and AWS Glue jobs.
Choose Next.
For Stack name, enter a name.
For Parameters, enter the parameters listed in the following table, then choose Create.
CloudFormation Template Parameter
Allowed Values
Description
InputBucketName
S3 bucket name
The S3 bucket where the AWS Glue job stores the migrated data.
The data will be actually stored in a folder named s3-redshift-loader-source, which is used by the Custom Auto Loader Framework.
InputConnectionName
AWS Glue connection name, the default is glue-bq-connector-24
The name of the AWS Glue connection that is created using the Google BigQuery connector.
InputDynamoDBTablePrefix
DynamoDB table name prefix, the default is bq_to_s3
The prefix that will be used when naming the two DynamoDB tables created by the solution.
InputGlueETLJob
AWS Glue ETL job name, the default is bq-migration-ETL
The name you want to give to the AWS Glue ETL job. The actual script is saved in the S3 path specified in the parameter InputGlueS3Path.
InputGlueMetaJob
AWS Glue Python shell job name, the default is bq-get-metadata
The name you want to give to AWS Glue Python shell job. The actual script is saved in the S3 path specified in the parameter InputGlueS3Path.
InputGlueS3Path
S3 path, the default is s3://aws-glue-scripts-${AWS::Account}-${AWS::Region}/admin/
This is the S3 path in which the stack will copy the scripts for AWS Glue jobs. Remember to replace: ${AWS::Account} with the actual AWS account ID and ${AWS::Region} with the Region you plan to use, or provide your own bucket and prefix in a complete path.
InputMaxParallelism
Number of parallel migration jobs to run, the default is 30
The maximum number of tables you want to migrate concurrently.
InputBQSecret
AWS Secrets Manager secret name
The name of the AWS Secrets Manager secret in which you stored the Google BigQuery credential.
InputBQProjectName
Google BigQuery project name
The name of your project in Google BigQuery in which you want to store temporary tables; you will need write permissions on the project.
StateMachineName
Step Functions state machine name, the default is
bq-to-s3-migration-state-machine
The name of the Step Functions state machine.
SourceS3BucketName
S3 bucket name, the default is aws-blogs-artifacts-public
The S3 bucket where the artifacts for this post are stored.
Do not change the default.
Deploy and configure the Custom Auto Loader Framework to load files from Amazon S3 to Amazon Redshift
The Custom Auto Loader Framework utility makes data ingestion to Amazon Redshift simpler and automatically loads data files from Amazon S3 to Amazon Redshift. The files are mapped to the respective tables by simply dropping files into preconfigured locations on Amazon S3. For more details about the architecture and internal workflow, refer to Custom Auto Loader Framework.
To set up the Custom Auto Loader Framework, complete the following steps:
Choose Launch Stack to deploy the CloudFormation stack in the us-east-1 Region:
On the AWS CloudFormation console, choose Next.
Provide the following parameters to help ensure the successful creation of resources. Make sure you have collected these values beforehand.
Parameter Name
Allowed Values
Description
CopyCommandSchedule
cron(0/5 * ? * * *)
The EventBridge rules KickoffFileProcessingSchedule and QueueRSProcessingSchedule are triggered based on this schedule. The default is 5 minutes.
DatabaseName
dev
The Amazon Redshift database name.
DatabaseSchemaName
public
The Amazon Redshift schema name.
DatabaseUserName
demo
The Amazon Redshift user name who has access to run COPY commands on the Amazon Redshift database and schema.
RedshiftClusterIdentifier
democluster
The Amazon Redshift cluster name.
RedshiftIAMRoleARN
arn:aws:iam::7000000000:role/RedshiftDemoRole
The Amazon Redshift cluster attached role, which has access to the S3 bucket. This role is used in COPY commands.
SourceS3Bucket
Your-bucket-name
The S3 bucket where data is located. Use the same bucket you used to store the migrated data as indicated in the previous stack.
CopyCommandOptions
delimiter '|' gzip
Provide the additional COPY command data format parameters as follows:
delimiter '|' dateformat 'auto' TIMEFORMAT 'auto'
InitiateSchemaDetection
Yes
The setting to dynamically detect the schema prior to file upload.
The following screenshot shows an example of our parameters.
Choose Create.
Monitor the progress of the Stack creation and wait until it is complete.
To verify the Custom Auto Loader Framework configuration, log in to the Amazon S3 console and navigate to the S3 bucket you provided as a value to the SourceS3Bucket parameter.
You should see a new directory called s3-redshift-loader-source is created.
Test the solution
To test the solution, complete the following steps:
Create the configuration file based on the prerequisites. You can also download the demo file.
To set up the S3 bucket, on the Amazon S3 console, navigate to the folder bq-mig-config in the bucket you provided in the stack.
Upload the config file into it.
To enable EventBridge notifications to the bucket, open the bucket on the console and on the Properties tab, locate Event notifications.
In the Amazon EventBridge section, choose Edit.
Select On, then choose Save changes.
On AWS Step Function console, monitor the run of the state machine.
Monitor the status of the loads in Amazon Redshift. For instructions, refer to Viewing Current Loads.
Open the Amazon Redshift Query Editor V2 and query your data.
Pricing considerations
You might have egress charges for migrating data out of Google BigQuery into Amazon S3. Review and calculate the cost for moving your data on your Google cloud billing console. As of this writing, AWS Glue 3.0 or later charges $0.44 per DPU-hour, billed per second, with a 1-minute minimum for Spark ETL jobs. For more information, see AWS Glue Pricing. With auto scaling enabled, AWS Glue automatically adds and removes workers from the cluster depending on the parallelism at each stage or microbatch of the job run.
Clean up
To avoid incurring future charges, clean up your resources:
Delete the CloudFormation solution stack.
Delete the CloudFormation Custom Auto Loader Framework stack.
Conclusion
In this post, we demonstrated how to build a scalable and automated data pipeline to migrate your data from Google BigQuery to Amazon Redshift. We also highlighted how the Custom Auto Loader framework can automate the schema detection, create tables for your S3 files, and continuously load the files into your Amazon Redshift warehouse. With this approach, you can automate the migration of entire projects (even multiple projects at the time) in Google BigQuery to Amazon Redshift. This helps improve data migration times into Amazon Redshift significantly through the automatic table migration parallelization.
The auto-copy feature in Amazon Redshift simplifies automatic data loading from Amazon S3 with a simple SQL command, users can easily automate data ingestion from Amazon S3 to Amazon Redshift using the Amazon Redshift auto-copy preview feature
Tahir Aziz is an Analytics Solution Architect at AWS. He has worked with building data warehouses and big data solutions for over 13 years. He loves to help customers design end-to-end analytics solutions on AWS. Outside of work, he enjoys traveling and cooking.
Ritesh Kumar Sinha is an Analytics Specialist Solutions Architect based out of San Francisco. He has helped customers build scalable data warehousing and big data solutions for over 16 years. He loves to design and build efficient end-to-end solutions on AWS. In his spare time, he loves reading, walking, and doing yoga.
Fabrizio Napolitano is a Principal Specialist Solutions Architect for DB and Analytics. He has worked in the analytics space for the last 20 years, and has recently and quite by surprise become a Hockey Dad after moving to Canada.
Manjula Nagineni is a Senior Solutions Architect with AWS based in New York. She works with major financial service institutions, architecting and modernizing their large-scale applications while adopting AWS Cloud services. She is passionate about designing big data workloads cloud-natively. She has over 20 years of IT experience in software development, analytics, and architecture across multiple domains such as finance, retail, and telecom.
Sohaib Katariwala is an Analytics Specialist Solutions Architect at AWS. He has over 12 years of experience helping organizations derive insights from their data.
In February, Meta released its large language model: LLaMA. Unlike OpenAI and its ChatGPT, Meta didn’t just give the world a chat window to play with. Instead, it released the code into the open-source community, and shortly thereafter the model itself was leaked. Researchers and programmers immediately started modifying it, improving it, and getting it to do things no one else anticipated. And their results have been immediate, innovative, and an indication of how the future of this technology is going to play out. Training speeds have hugely increased, and the size of the models themselves has shrunk to the point that you can create and run them on a laptop. The world of AI research has dramatically changed.
This development hasn’t made the same splash as other corporate announcements, but its effects will be much greater. It will wrest power from the large tech corporations, resulting in both much more innovation and a much more challenging regulatory landscape. The large corporations that had controlled these models warn that this free-for-all will lead to potentially dangerous developments, and problematic uses of the open technology have already been documented. But those who are working on the open models counter that a more democratic research environment is better than having this powerful technology controlled by a small number of corporations.
The power shift comes from simplification. The LLMs built by OpenAI and Google rely on massive data sets, measured in the tens of billions of bytes, computed on by tens of thousands of powerful specialized processors producing models with billions of parameters. The received wisdom is that bigger data, bigger processing, and larger parameter sets were all needed to make a better model. Producing such a model requires the resources of a corporation with the money and computing power of a Google or Microsoft or Meta.
But building on public models like Meta’s LLaMa, the open-source community has innovated in ways that allow results nearly as good as the huge models—but run on home machines with common data sets. What was once the reserve of the resource-rich has become a playground for anyone with curiosity, coding skills, and a good laptop. Bigger may be better, but the open-source community is showing that smaller is often good enough. This opens the door to more efficient, accessible, and resource-friendly LLMs.
More importantly, these smaller and faster LLMs are much more accessible and easier to experiment with. Rather than needing tens of thousands of machines and millions of dollars to train a new model, an existing model can now be customized on a mid-priced laptop in a few hours. This fosters rapid innovation.
It also takes control away from large companies like Google and OpenAI. By providing access to the underlying code and encouraging collaboration, open-source initiatives empower a diverse range of developers, researchers, and organizations to shape the technology. This diversification of control helps prevent undue influence, and ensures that the development and deployment of AI technologies align with a broader set of values and priorities. Much of the modern internet was built on open-source technologies from the LAMP (Linux, Apache, mySQL, and PHP/PERL/Python) stack—a suite of applications often used in web development. This enabled sophisticated websites to be easily constructed, all with open-source tools that were built by enthusiasts, not companies looking for profit. Facebook itself was originally built using open-source PHP.
But being open-source also means that there is no one to hold responsible for misuse of the technology. When vulnerabilities are discovered in obscure bits of open-source technology critical to the functioning of the internet, often there is no entity responsible for fixing the bug. Open-source communities span countries and cultures, making it difficult to ensure that any country’s laws will be respected by the community. And having the technology open-sourced means that those who wish to use it for unintended, illegal, or nefarious purposes have the same access to the technology as anyone else.
This, in turn, has significant implications for those who are looking to regulate this new and powerful technology. Now that the open-source community is remixing LLMs, it’s no longer possible to regulate the technology by dictating what research and development can be done; there are simply too many researchers doing too many different things in too many different countries. The only governance mechanism available to governments now is to regulate usage (and only for those who pay attention to the law), or to offer incentives to those (including startups, individuals, and small companies) who are now the drivers of innovation in the arena. Incentives for these communities could take the form of rewards for the production of particular uses of the technology, or hackathons to develop particularly useful applications. Sticks are hard to use—instead, we need appealing carrots.
It is important to remember that the open-source community is not always motivated by profit. The members of this community are often driven by curiosity, the desire to experiment, or the simple joys of building. While there are companies that profit from supporting software produced by open-source projects like Linux, Python, or the Apache web server, those communities are not profit driven.
And there are many open-source models to choose from. Alpaca, Cerebras-GPT, Dolly, HuggingChat, and StableLM have all been released in the past few months. Most of them are built on top of LLaMA, but some have other pedigrees. More are on their way.
The large tech monopolies that have been developing and fielding LLMs—Google, Microsoft, and Meta—are not ready for this. A few weeks ago, a Google employee leaked a memo in which an engineer tried to explain to his superiors what an open-source LLM means for their own proprietary tech. The memo concluded that the open-source community has lapped the major corporations and has an overwhelming lead on them.
This isn’t the first time companies have ignored the power of the open-source community. Sun never understood Linux. Netscape never understood the Apache web server. Open source isn’t very good at original innovations, but once an innovation is seen and picked up, the community can be a pretty overwhelming thing. The large companies may respond by trying to retrench and pulling their models back from the open-source community.
But it’s too late. We have entered an era of LLM democratization. By showing that smaller models can be highly effective, enabling easy experimentation, diversifying control, and providing incentives that are not profit motivated, open-source initiatives are moving us into a more dynamic and inclusive AI landscape. This doesn’t mean that some of these models won’t be biased, or wrong, or used to generate disinformation or abuse. But it does mean that controlling this technology is going to take an entirely different approach than regulating the large players.
This essay was written with Jim Waldo, and previously appeared on Slate.com.
Since its inception, Cloudflare Zaraz, the server-side third-party manager built for speed, privacy and security, has strived to offer a way for marketers and developers alike to get the data they need to understand their user journeys, without compromising on page performance. Cloudflare Zaraz makes it easy to transition from traditional client-side data collection based on marketing pixels in users’ browsers, to a server-side paradigm that shares events with vendors from the edge.
When implementing data collection on websites or mobile applications, analysts and digital marketers usually first define the set of interactions and attributes they want to measure, formalizing those requirements along technical specifications in a central document (“tagging plan”). Developers will later implement the required code to make those attributes available for the third party manager to pick it up. For instance, an analyst may want to analyze page views based on an internal name instead of the page title or page pathname. They would therefore define an example “page name” attribute that would need to be made available in the context of the page, by the developer. From there, the analyst would configure the tag management system to pick the attribute’s value before dispatching it to the analytics tool.
Yet, while the above flow works fine in theory, the reality is that analytics data comes from multiple sources, in multiple formats, that do not always fit the initially formulated requirements.
The industry accepted solution, such as Google Tag Manager’s “Custom JavaScript variables” or Adobe’s “Custom Code Data Elements”, was to offer a way for users to dynamically invoke custom JavaScript functions on the client, allowing them to perform cleaning (like removing PII data from the payload before sending it to Google Analytics), transformations (extracting specific product attributes out of a product array) or enrichment (making an API call to grab the current user’s CRM id to stitch user sessions in your analytics tool) to the data before dispatch by the third-party manager.
Example of Google Tag Manager custom javascript variable that aggregates individual items prices from a javascript array of product information.
Having the ability to run custom JavaScript is a powerful feature that offers a lot of flexibility and yet, was a missing part of Cloudflare Zaraz. While some workarounds existed, it did not really fit with Cloudflare Zaraz’s objective of high-performance. We needed a way for our users to provide custom code to be executed fast, server-side. Quite fast, it was clear that Cloudflare Workers, the globally distributed serverless V8-based JavaScript runtime was the solution.
Worker Variables to the rescue
Cloudflare Zaraz Worker Variables is powered by Cloudflare Workers, our platform for running custom code on the edge, but let’s take a step back and work through how Cloudflare Zaraz is implemented.
When making a request to a website proxied by Cloudflare, a few things will run before making it to your origin. This includes the firewall, DDoS mitigation, caching, and also something called First-Party Workers.
These First-Party Workers are Cloudflare Workers with special permissions. Cloudflare Zaraz is one of them. Our Worker is built in a way that allows variables to be replaced by their contents. Those variables can be used in places where you would be reusing hardcoded text, to make it easier to make changes to all places where it would be used. For example, the name of your site, a secret key, etc:
These variables can then be used in any of Cloudflare Zaraz’s components, by selecting them right from the dashboard as a property, or as part of a component’s settings:
When using a Worker Variable, instead of replacing your variable with a hardcoded string, we instead execute the custom code hosted in your own Cloudflare Worker that you have associated with the variable. The response of which is then being used as the variable’s value. Calling one worker from within the Cloudflare Zaraz Worker is done using Dynamic Dispatch.
In our Cloudflare Zaraz Worker, calling Dynamic Dispatch is very similar to how your regular, everyday worker might do it. From having a binding in our wrangler.toml:
...
unsafe.bindings = [
{ name = "DISPATCHER", type = "dynamic_dispatch", id_prefix = "", first_party = true },
]
To having our code responsible for variables actually call your worker:
if (variable.type === 'worker') {
// Get the persistent ID of the Worker
const mutableId = variable.value.mutableId
// Get the binding for the specific Worker
const workerBinding = context.env.DISPATCHER.get(mutableId)
...
// Execute the Worker and return the response
const workerResponse = await workerBinding.fetch(
new Request(String(url || context.url || 'http://unknown')),
{
method: 'POST',
headers: {
'Content-type': 'application/json',
},
body: JSON.stringify(payload),
}
)
...
return workerResponse
}
Benefits of Cloudflare Zaraz Worker Variables
Cloudflare Workers is a world-class solution to build complex applications. Together with Cloudflare Zaraz, we feel that makes it the ideal platform to orchestrate your data workflows:
Build with context: Cloudflare Zaraz automatically shares the context as part of the call to the Cloudflare Zaraz Worker, allowing you to use that data as input to your functions. Cloudflare Zaraz offers a Web API which customers can use to track important events in their users' journeys. Along with the Web API, Zaraz offers ways for users to define custom attributes, called “Track properties” and “Variables”, that allow our customers to provide additional context to the event getting sent to a vendor. The Cloudflare Zaraz context holds every attribute that was tracked by Cloudflare Zaraz as part of the current visitor session along with other generic attributes like the visitors’ device cookies for instance.
Speed: In comparison to manually calling a worker from client-side JavaScript, this saves the roundtrip to the Worker’s HTTP endpoint and gives you access to Cloudflare Zaraz properties, allowing you to work with client-side data right from the edge.
Isolated environment: As the function is executed inside the worker, which lives outside of your visitor browser, it cannot access the DOM or JavaScript runtime from the browser, preventing potential bugs in your Worker’s code from affecting the experience of your user.
When combining Worker Variables with the Custom HTML tool, you also get the benefits of offloading client-side JavaScript to a worker. This improves performance for both AJAX network requests, which can then be executed directly from Cloudflare’s global network, as well as the offloading of resource intensive tasks to a Worker, such as data manipulation or computations. All whilst keeping your API secrets and other sensitive data hidden from the clients, allowing you to only send the results that are actually needed by the client.
Examples walkthrough
Now that you are more familiar with the concept, let’s get to some practical use cases!
We will cover two examples: translating a GTM custom JavaScript variable to a Cloudflare Zaraz Worker variable, and enriching user information with data from an external API.
Translate a GTM Custom JavaScript variable into Cloudflare Zaraz Worker Variable: Let’s take our previous example, Google Tag Manager custom javascript variable that aggregates individual items prices from a JavaScript array of product information.
The function makes use of a “GTM Data Layer Variable” (represented with double curly braces, line 2, “{{DLV – Ecommerce – Purchase – Products }}”). That kind of variable is equivalent to Track Properties in Cloudflare Zaraz land: they are a way to access the value of a custom attribute that you shared with the third party manager. When translating from GTM to Cloudflare Zaraz, one should take care of making sure that such variables have also been translated to their Cloudflare Zaraz counterpart.
Back to our example, let’s say that the GTM variable “DLV – Ecommerce – Purchase – Products” is equal to a track property “products” in Cloudflare Zaraz. Your first step is to parse the Cloudflare Zaraz context ($1), that gives you two objects: client, holding all track properties set in the current visitor context and system that gives you access to some generic properties of the visitor’s device.
You can then reference a specific track property by accessing it from the client object. ($2)
The variable code was aggregating price from product information into a comma-separated string. For this, we can keep the same code. ($3)
A major difference between javascript functions executed in the client and Workers is that the worker should “return” the value as part of a Response object. ($4)
export default {
async fetch(request, env) {
// $1 Parse the Zaraz Context object
const { system, client } = await request.json();
// $2 Get a reference to the products track property
const products = client.products;
// $3 Calculate the sum
const prices = products.map(p => p.price).join();
return new Response(prices);
},
};
Enriching user information with data coming from an API: For this second example, let’s imagine that we want to synchronize user activity online and offline. To do so, we need a common key to reconcile the user journeys. A CRM id looks like an appropriate candidate for that use case. We will obtain this id through the CRM solution API (the fictitious “https://example.com/api/getUserIdFromCookie”) Our primary key, that will be used to lookup the user CRM id, will be taken from a cookie that holds the current user session id.
export default {
async fetch(request, env) {
// Parse the Zaraz Context object
const { system, client } = await request.json();
// Get the value of the cookie "login-cookie"
const cookieValue = system.cookies["login-cookie"];
const userId = await fetch("https://example.com/api/getUserIdFromCookie", {
method: POST,
body: cookieValue,
});
return new Response(userId);
},
};
Start using Worker Variables today
Worker Variables are available for all accounts with a paid Workers subscription (starting at $5 / month).
Create a worker
To use a Worker Variable, you first need to create a new Cloudflare Worker. You can do this through the Cloudflare dashboard or by using Wrangler.
To create a new worker in the Cloudflare dashboard:
Log in to the Cloudflare dashboard.
Go to Workers and select Create a Service.
Give a name to your service and choose the HTTP Handler as your starter template.
4. Give your variable a name, choose Worker as the Variable type, and select your newly created Worker.
5. Save your variable.
Use your Worker Variable
It is now time to use your Worker Variable! You can reference your variable as part of a trigger or an action. To set it up for a specific action for instance:
Go to Zaraz > Tools configuration > Tools.
Click Edit next to a tool that you have already configured.
Select an action or add a new one.
Click on the plus sign at the right of the text fields.
With the vast amount of data being created today, organizations are moving to the cloud to take advantage of the security, reliability, and performance of fully managed database services. To facilitate database and analytics migrations, you can use AWS Database Migration Service (AWS DMS). First launched in 2016, AWS DMS offers a simple migration process that automates database migration projects, saving time, resources, and money.
Although you can start AWS DMS migration with a few clicks through the console, you still need to do research and planning to determine the required capacity before migrating. It can be challenging to know how to properly scale capacity ahead of time, especially when simultaneously migrating many workloads or continuously replicating data. On top of that, you also need to continually monitor usage and manually scale capacity to ensure optimal performance.
Introducing AWS DMS Serverless Today, I’m excited to tell you about AWS DMS Serverless, a new serverless option in AWS DMS that automatically sets up, scales, and manages migration resources to make your database migrations easier and more cost-effective.
Here’s a quick preview on how AWS DMS Serverless works:
AWS DMS Serverless removes the guesswork of figuring out required compute resources and handling the operational burden needed to ensure a high-performance, uninterrupted migration. It performs automatic capacity provisioning, scaling, and capacity optimization of migrations, allowing you to quickly begin migrations with minimal oversight.
With a variety of sources and targets supported by AWS DMS Serverless, many scenarios become possible. You can use AWS DMS Serverless to migrate databases and help to build modern data strategies by synchronizing ongoing data replications into data lakes (e.g., Amazon S3) or data warehouses (e.g., Amazon Redshift) from multiple, perhaps disparate data sources.
How AWS DMS Serverless Works Let me show you how you can get started with AWS DMS Serverless. In this post, I migrate my data from a source database running on PostgreSQL to a target MySQL database running on Amazon RDS. The following screenshot shows my source database with dummy data:
As for the target, I’ve set up a MySQL database running in Amazon RDS. The following screenshot shows my target database:
Getting starting with AWS DMS Serverless is similar to how AWS DMS works today. AWS DMS Serverless requires me to complete the setup tasks such as creating a virtual private cloud (VPC) to defining source and target endpoints. If this is your first time working with AWS DMS, you can learn more by visiting Prerequisites for AWS Database Migration Service.
To connect to a data store, AWS DMS needs endpoints for both source and target data stores. An endpoint provides all necessary information including connection, data store type, and location to my data stores. The following image shows an endpoint I’ve created for my target database:
When I have finished setting up the endpoints, I can begin to create a replication by selecting the Create replication button on the Serverless replications page. Replication is a new concept introduced in AWS DMS Serverless to abstract instances and tasks that we normally have in standard AWS DMS. Additionally, the capacity resources are managed independently for each replication.
On the Create replication page, I need to define some configurations. This starts with defining Name, then specifying Source database endpoint and Target database endpoint. If you don’t find your endpoints, make sure you’re selecting database engines supported by AWS DMS Serverless.
After that, I need to specify the Replication type. There are three types of replication available in AWS DMS Serverless:
Full load — If I need to migrate all existing data in source database
Change data capture (CDC) — If I have to replicate data changes from source to target database.
Full load and change data capture (CDC) — If I need to migrate existing data and replicate data changes from source to target database.
In this example, I chose Full load and change data capture (CDC) because I need to migrate existing data and continuously update the target database for ongoing changes on the source database.
In the Settings section, I can also enable logging with Amazon CloudWatch, which makes it easier for me to monitor replication progress over time.
As with standard AWS DMS, in AWS DMS Serverless, I can also configure Selection rules in Table mappings to define filters that I need to replicate from table columns in the source data store.
I can also use Transformation rules if I need to rename a schema or table or add a prefix or suffix to a schema or table.
In the Capacity section, I can set the range for required capacity to perform replication by defining the minimum and maximum DCU (DMS capacity units). The minimum DCU setting is optional because AWS DMS Serverless determines the minimum DCU based on an assessment of the replication workload. During replication process, AWS DMS uses this range to scale up and down based on CPU utilization, connections, and available memory.
Setting the maximum capacity allows you to manage costs by making sure that AWS DMS Serverless never consumes more resources than you have budgeted for. When you define the maximum DCU, make sure that you choose a reasonable capacity so that AWS DMS Serverless can handle large bursts of data transaction volumes. If traffic volume decreases, AWS DMS Serverless scales capacity down again, and you only pay for what you need. For cases in which you want to change the minimum and maximum DCU settings, you have to stop the replication process first, make the changes, and run the replication again.
When I’m finished with configuring replication, I select Create replication.
When my replication is created, I can view more details of my replication and start the process by selecting Start.
After my replication runs for around 40 minutes, I can monitor replication progress in the Monitoring tab. AWS DMS Serverless also has a CloudWatch metric called Capacity utilization, which indicates the use of capacity to run replication according to the range defined as minimum and maximum DCU. The following screenshot shows the capacity scales up in the CloudWatch metrics chart.
When the replication finishes its process, I see the capacity starting to decrease. This indicates that in addition to AWS DMS Serverless successfully scaling up to the required capacity, it can also scale down within the range I have defined.
Finally, all I need to do is verify whether my data has been successfully replicated into the target data store. I need to connect to the target, run a select query, and check if all data has been successfully replicated from the source.
Now Available AWS DMS Serverless is now available in all commercial regions where standard AWS DMS is available, and you can start using it today. For more information about benefits, use cases, how to get started, and pricing details, refer to AWS DMS Serverless.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.