The Universal Blue
project, which produces operating system images based on Fedora’s Atomic Desktops,
has issued an announcement
that manual steps are required to continue receiving updates. Jorge
Castro wrote:
If you use Bazzite, Bluefin, Aurora, or any other Universal Blue
image (including our toolboxes) then you need to follow the
instructions in this announcement in order to ensure that your device
is getting updates. We were rotating our cosign keypairs this morning,
which is the method that we use to sign our images.
During this process I made a critical error which has resulted in
forcing you to take manual steps to migrate to our newly signed
images.
This applies to all Universal Blue images released before July 2,
2024. See the full announcement for instructions. LWN covered Bluefin in
December, 2023.
In 2016, Oliver Smith reached a point of frustration with the short
lifespan of updates for his Android phone. Taking matters into his own
hands, he began developing postmarketOS, a Linux distribution for
mobile phones. Eight years later, the core team and
trusted contributors have grown to twenty individuals, while the latest
release, v24.06,
now shows support for over 250 devices. Although postmarketOS isn’t
usable as a day-to-day phone operating system on all of them, it can also enable repurposing devices into compact servers or kiosk machines.
Version 4.10.0 of GNU findutils has been released. Notable changes
include allowing find -name / as a valid
pattern, and accepting larger UIDs/GIDs for find -user and find -group. It is also once again possible to build
findutils on systems with musl-libc.
David Rosenthal looks
back at 40 years of the X Window System:
A major reason for Sun’s early success was that they in effect
open-sourced the Network File System. X11 was open source under the
MIT license. I, and some of the other Sun engineers, understood
that NeWS could not displace X11 as the Unix standard window system
without being equally open source. But Sun’s management looked at
NeWS and saw superior technology, an extension of the PostScript
that Adobe was selling, and couldn’t bring themselves to give it
away.
Между табелите Exit и Luggage на летище „Бен Гурион“ има опънат голям транспарант, на който пише: BRING THEM HOME. NOW (Върнете ги у дома. Сега).
Последната дума е написана в червено. И това червено е като крясък, като клетва. Като кръв.
Тeзи четири думи диктуват системното кръвообращение на цялата страна.
Има ги навсякъде: по огради, по фасади, по площади. Там са и снимките на отвлечените заложници: стоят закачени върху колони, върху пътни знаци и билбордове, по пейки и витрини. Усмихнатите физиономии те съпровождат денем и нощем. Блокират собствената ти система за самосъхранение, която не иска да си представяш къде в момента са тези хора, живи ли са, кой ги наблюдава, пият ли си хапчетата, как се къпят, къде спят, до кого се събуждат. Говорят ли, шепнат ли, плачат ли. На какъв език мълчат.
Те са приятели, сестри, съседи. Майки, братя, колеги. Те са деца.
Най-малкото сред тях е червенокосото бебе с беззъба усмивка, което се превърна в символ на бруталността на „Хамас“.
Кфир Бибас. Снимката е от личния архив на семейството, предоставена за целите на движението Bring Them Home
Кфир Бибас е роден на 18 януари 2023 г. Когато го отвличат, е на 9 месеца.
Между 9 и 12 месеца бебетата се научават да стоят, поникват им първите зъбки, правят първите си стъпки, произнасят първата си дума.
Ако Кфир е жив, сега е на 18 месеца.
Между 12 и 18 месеца бебето започва да променя отношението си към хората. То не само се опитва да играе с тях, но и да им подражава.
Ако Кфир е жив, може би започва да подражава на своите похитители.
Ако Кфир е жив, ще е живял по-дълго като заложник, отколкото като свободно човешко същество. Този най-малък от всички отвлечени от терористите човек е живял на свобода само 262 дни.
Вземането на цивилни за заложници е военно престъпление според Женевските конвенции. „Но деца почти никога не се вземат за заложници“, пише в своята студия, публикувана онлайн от Cambridge University Press, изследователката Даниел Гилбърт от Северозападния университет в Чикаго. „Мотивите са брутални: децата не стават за разменна монета, защото трудно оцеляват след изпитанието“, обяснява тя.
2
След 7 октомври животът в Държавата Израел е като имитация на живот в Държавата Израел. Тече застинал, забавен, задръстен, замислен.
Привидно град Тел Авив функционира нормално. Шуми, излъчва, бърза, диша, расте. Расте нависоко със стъписващи темпове.
Снимка: Еми Барух
Долу кафенетата продължават да завземат улиците, климатиците разпръскват лъжлив комфорт, тапите в час пик са епични.
Над плажовете се стеле мараня. По улиците се разминават хора, тротинетки, кучета, велосипеди. Навалица…
Гневни графити гарнират стените в синхрон с една вибрираща енергия на сподавено очакване за промяна. Усеща се, ако поговориш с някого. Ако поспреш. Хората не спират. Блъскат се. Бързат. Поддържат онзи ритъм, който не ти оставя много време за споделяне.
Надписът под изображението на Нетаняху гласи: „Опасност“. Снимка: Еми Барух
3
Ние бяхме два пъти изоставени от нашето правителство. Веднъж на 7 октомври и сега, когато децата ни са още пленници.
Това са думи на един баща, един от хилядите, които всяка седмица участват в протести с призиви за преговори за освобождаване на заложниците, за оставка на правителството на Нетаняху, за край на окупацията, за избори, за мир между араби и евреи, за изход от мъртвата хватка, в която крайнодесните държат страната.
Събота вечер е. Ден за протести.
С моите приятели си проправяме път между групи, носещи израелски знамена. Мнозина са облекли тениски, всяка с лицето на един от сто двайсет и четиримата заложници, които все още се държат от „Хамас“ в Газа.
Снимка: Еми Барух
Това, което ни обединява, е знамето и неистовото желание Нетаняху да си тръгне.
Те са тук всяка седмица от две години. Изтощени са. Но не могат да се откажат. Нямат друг избор.
Когато приятелите ми ме посрещнаха на летището, дори не ги попитах „Как сте? Добре ли сте?“, защото знам: никой не е добре.
Напредваме към кръстовището между „Бегин“ и улица „Каплан“, което от миналата година Общината е преименувала на „Площад на демокрацията“ в чест на нестихващите протести срещу съдебната реформа на правителството. „Демокрацията не е даденост – е написал тогава кметът на Тел Авив в Х, за да обоснове смяната на името. – Трябва да я защитаваме всеки ден“.
Минаваме покрай импозантната сграда на Военния щаб, срещу която екран отброява дните, часовете, минутите и секундите от 7 октомври. Емоционалният часовник на хората обаче е спрял. Всяка сутрин те отново се събуждат на 7 октомври. Медиатизирането на погрома не помага. Новините се прекъсват от профила на поредния загинал войник.
Предната вечер сме говорили дълго колко взаимносвързани са жителите на тази страна, колко всъщност е малка тя. Как всеки е в контакт с всеки. Как всеки или е загубил някого, или познава някого, който е загубил някого на 7 октомври.
Докато израелските заложници и цивилното население на Газа са държани като живи щитове, раната стои отворена. Рационалното мислене е в плен на националната травма.
Доближаваме сцена с огромни екрани, около която се сгъстява гневът на протестиращите. Думата, която всички повтарят в транс, е עכשיו – „сега“. „Освободете ги СЕГА!“
Снимка: Еми Барух
Отчаянието е по-силно от гнева.
4
„Аз съм част от всичко това“, казва Етгар Керет, този най-любим съвременен бард на израелската литература. И продължава:
„Но когато крещя: „Освободете ги сега!“, на кого го крещя? На Синуар ли го крещя? Ако извикам достатъчно силно, дали той ще каже:
– Мохамед, млъкни за момент. – Какво? – Освободете ги сега!… – Добре. Мохамед, пусни ги. Има хора, които ме молят да ги освободя сега.“
Мисля ли, че ще проработи? Или скандирам „Освободете ги сега!“, така че Байдън да разбере, че трябва да направи нещо по въпроса?
Или може би крещя това на Нетаняху? И той ще каже: „О, боже мой, да, трябва да ги освободим сега. Не бях мислил за това…“
Какво точно постигам? Когато крещя СЕГА, не е ли равносилно на молитва? Не е ли това инфантилен сблъсък с реалността?“
Молитва. Заклинание. Клетва. Отсъства „публичната употреба на разума“ (по Имануел Кант).
5
Множеството от улица „Каплан“ е нарицателно за онази колективна енергия, в която е концентриран ураган от непоносимост към религиозната мегаломания на фанатичните екстремисти в коалицията на премиера Нетаняху. Две имена предизвикват яростно освиркване – Бен-Гвир и Смотрич. Известни със своята агресивна и расистка реторика, и двамата са ревностни поддръжници на идеята за завръщане на еврейските заселници в Газа и за разселване на палестинското население. Първият е министър на националната сигурност, вторият е министър на финансите.
Речи, освирквания, скандирания.
Един след друг на сцената се качват артисти, родители на отвлечени деца, политици, дипломати, музиканти. Сцените са няколко. Драматургията на протеста следва емоцията на множеството.
Стигаме до Kikar HaChatufim. „Площадът на заложниците“ се намира пред Музея на изкуствата в Тел Авив. „Директна линия между публиката и изкуството“, написа един от кураторите на музея. Изкуството е излязло на площада. Той е отрупан с инсталации, интерактивни експонати, постери, картини, книги, предмети.
Хората преминават ритуално покрай топоси на травмата.
Около площада има палатки на най-тежко пострадалите кибуци. Разпънати са шатри, където се събират семействата на заложниците; на специални постаменти са подредени фигурки, играчки, писма; има сергии, откъдето може да се снабдиш с жълта панделка, с жълта птица, с жълто пинче, с жълта гривна, на която пише: „Върнете ги. СЕГА.“ Жълтото е символ на солидарност със семействата на отвлечените, призив за тяхното освобождаване. Жълтото (тук и сега) е отправна точка на идентичността.
Снимки: Еми Барух
В средата на площада е поставена дълга маса с празни чинии и празни чаши пред празни столове, подредени за всеки от изчезналите заложници.
Срещу нея е построен тунел – бутафорна реплика в реален размер на тунелите, прокопани под ивицата Газа.
Проходът е висок над човешки ръст. По дъговидните стени има текстове, рисунки, молитви, снимки, имена, карикатури, надписи на различни езици. Високоговорителите възпроизвеждат звук от далечни изстрели. Симулацията в слабоосветеното пространство е клаустрофобична. Минаваме през тунела. До изхода са само 25 метра…
Отвън са инструментите, които чакат собствениците им да се върнат.
Снимка: Еми Барух
6
В един от репортажите, написани по времето, когато аз бях там, на въпрос дали хората са простили на армията провала на 7 октомври, анкетираните отговарят, че няма време за това.
Няма време да се занимаваме с миналото, да го съдим, да го преоценяваме. Ние сме в капана на едно настояще, което продължава от 7 октомври и оставя място само за една дума: СЕГА. Тази дума е обединителен вик, политическо искане и изразява много повече: трябва да се сложи край на безвремието, в което живее страната.
Снимка: Еми Барух
Два дни по-късно присъствам на среща в Кнесета с членове на парламента. Говорят за противоречията, компромисите, контекстите, нюансите.
„Това е най-дългата война, която някога сме водили“, обръща се към присъстващите Мерав Коен, депутатка от центристката либерална партия „Йеш Атид“.
„Няма победа, без заложниците да бъдат върнати. А това не е приоритет за Нетаняху. Той е циничен политик. Не го интересува човешкият живот. Спешно ни трябват избори. Нетаняху разбира, че това ще е неговият край, и точно затова протака войната…“
В капана на едно безкрайно настояще човек се пита: как да останеш цял в този счупен свят?
7
P.S.
Погледната отвън, израелската държава се възприема като монолитно единство на хора, обединени от обща библейска история. За евреите от диаспората тя е някаква неясна амалгама от предания, идеи и спомени, от обичаи и чувства.
Но тази идилична картина е далече от действителността.
Все по-агресивният диктат на ултраортодоксалните евреи отдавна се е превърнал в заплаха за демокрацията в страната. Именно техните партии удържат коалицията на Нетаняху. И ако на тях не им се угоди, правителството може и да падне.
Точно това е политическото земетресение, което започна съвсем скоро с едно решение на Върховния съд на Израел. То сложи край на привилегированата позиция на ултрасите, които изучават Светите писания, и това ги освобождава от армията. И нека да е ясно: несъгласието с отбиването на задължителна военна служба не е поради пацифистки възгледи. По-голямата част от войниците в израелската армия (IDF) не са в бойни части, а изпълняват административни задачи, работят в офиси. А еврейските студенти, които изучават Талмуда и Тората, не работят и се издържат от данъчни постъпления. За огромната част от данъкоплатците те са нещо като търтеи в кошера.
Съдиите от Върховния съд на Израел постановиха с единодушие, че военната служба става задължителна и за ултраортодоксалните евреи. Че те също трябва да участват в националния проект Израел с всички произтичащи от това права и задължения.
Един от ултрарелигиозните евреи заяви по този повод, че решението може да доведе до „две държави“, разделени не между израелци и палестинци, а между религиозни и нерелигиозни евреи.
Стар виц гласи, че Израел е страна, в която една трета от населението отива в армията, една трета работи и една трета плаща данъци. Проблемът е, гласи шегата, че става дума за една и съща една трета.
Израел е държава, която 75 години след създаването си все още няма писмена конституция, държава, която все още се ръководи от еврейските религиозни закони.
Израел е държава на изгнаници. За някои това е избор. За други е съдба.
France is currently electing a new government through early legislative elections that began on Sunday, June 30, 2024, with a second round scheduled for July 7. In this blog, we show how Cloudflare blocked DDoS attacks targeting three different French political parties.
2024 has been dubbed “the year of elections,” with elections taking place in over 60 countries, as we have mentioned before (1, 2, 3). If you regularly follow the Cloudflare blog, you’re aware that we consistently cover election-related trends, including in South Africa, India, Iceland, Mexico, the European Union and the 2024 US presidential debate. We also continuously update our election report on Cloudflare Radar.
Recently in France, as in the early stages of the war in Ukraine and during EU elections in the Netherlands, political events have precipitated cyberattacks. In France, several DDoS (Distributed Denial of Service attack) attacks targeted political parties involved in the elections over the past few days, with two parties hit just before the first round and another on election day itself.
The first political party, shown in yellow in the previous chart, experienced a DDoS attack on June 23, 2024, peaking at 68,000 requests per second (rps); it also endured a second DDoS attack on June 29, the day before the election, peaking at 20,000 rps. Although these rates are small on Cloudflare’s scale, they can be devastating for unprotected websites unaccustomed to such levels of traffic.
The second party, represented by the blue line, was targeted on June 24, June 27, and June 29, 2024, with the most severe attack occurring on June 27, reaching 118,000 rps during a day marked by frequent DDoS spikes that had in total 610 million daily requests.
The third party was attacked on the evening of June 29 in France, with several attempts blocked by Cloudflare on election day, June 30, between 10:00 and 23:00 UTC (12:00 and 01:00 local time). The peak activity targeting this party hit nearly 40,000 rps at 19:00 UTC (21:00 local time), with a total of 620 million daily DDoS requests on election day.
Modest drops and clear traffic increases after voting ends
During the first round of the election this past Sunday, June 30, 2024, Internet traffic was initially higher than the previous week but dropped by as much as 3% at 11:30 local time (09:30 UTC) after the polls opened. Traffic began to increase again after 17:45 local time (15:45 UTC) and peaked at 20:00 local time (18:00 UTC) when the polls closed and the first projections were announced.
We will provide a trends update on the French election after the runoff scheduled for July 7, 2024.
If you want to follow more trends and insights about the Internet and elections in particular, you can check Cloudflare Radar, and more specifically our new 2024 Elections Insights report, which will be updated as elections take place throughout the year.
To help preserve a safe Internet for content creators, we’ve just launched a brand new “easy button” to block all AI bots. It’s available for all customers, including those on our free tier.
The popularity of generative AI has made the demand for content used to train models or run inference on skyrocket, and, although some AI companies clearly identify their web scraping bots, not all AI companies are being transparent. Google reportedly paid $60 million a year to license Reddit’s user generated content, Scarlett Johansson alleged OpenAI used her voice for their new personal assistant without her consent, and most recently, Perplexity has been accused of impersonating legitimate visitors in order to scrape content from websites. The value of original content in bulk has never been higher. Last year, Cloudflare announced the ability for customers to easily block AI bots that behave well. These bots follow robots.txt, and don’t use unlicensed content to train their models or run inference for RAG applications using website data. Even though these AI bots follow the rules, Cloudflare customers overwhelmingly opt to block them.
We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly. To help, we’ve added a brand new one-click to block all AI bots. It’s available for all customers, including those on the free tier. To enable it, simply navigate to the Security > Bots section of the Cloudflare dashboard, and click the toggle labeled AI Scrapers and Crawlers.
This feature will automatically be updated over time as we see new fingerprints of offending bots we identify as widely scraping the web for model training. To ensure we have a comprehensive understanding of all AI crawler activity, we surveyed traffic across our network.
AI bot activity today
The graph below illustrates the most popular AI bots seen on Cloudflare’s network in terms of their request volume. We looked at common AI crawler user agents and aggregated the number of requests on our platform from these AI user agents over the last year:
When looking at the number of requests made to Cloudflare sites, we see that Bytespider, Amazonbot, ClaudeBot, and GPTBot are the top four AI crawlers. Operated by ByteDance, the Chinese company that owns TikTok, Bytespider is reportedly used to gather training data for its large language models (LLMs), including those that support its ChatGPT rival, Doubao. Amazonbot and ClaudeBot follow Bytespider in request volume. Amazonbot, reportedly used to index content for Alexa’s question-answering, sent the second-most number of requests and ClaudeBot, used to train the Claude chat bot, has recently increased in request volume.
Among the top AI bots that we see, Bytespider not only leads in terms of number of requests but also in both the extent of its Internet property crawling and the frequency with which it is blocked. Following closely is GPTBot, which ranks second in both crawling and being blocked. GPTBot, managed by OpenAI, collects training data for its LLMs, which underpin AI-driven products such as ChatGPT. In the table below, “Share of websites accessed” refers to the proportion of websites protected by Cloudflare that were accessed by the named AI bot.
AI Bot
Share of Websites Accessed
Bytespider
40.40%
GPTBot
35.46%
ClaudeBot
11.17%
ImagesiftBot
8.75%
CCBot
2.14%
ChatGPT-User
1.84%
omgili
0.10%
Diffbot
0.08%
Claude-Web
0.04%
PerplexityBot
0.01%
While our analysis identified the most popular crawlers in terms of request volume and number of Internet properties accessed, many customers are likely not aware of the more popular AI crawlers actively crawling their sites. Our Radar team performed an analysis of the top robots.txt entries across the top 10,000 Internet domains to identify the most commonly actioned AI bots, then looked at how frequently we saw these bots on sites protected by Cloudflare.
In the graph below, which looks at disallowed crawlers for these sites, we see that customers most often reference GPTBot, CCBot, and Google in robots.txt, but do not specifically disallow popular AI crawlers like Bytespider and ClaudeBot.
With the Internet now flooded with these AI bots, we were curious to see how website operators have already responded. In June, AI bots accessed around 39% of the top one million Internet properties using Cloudflare, but only 2.98% of these properties took measures to block or challenge those requests. Moreover, the higher-ranked (more popular) an Internet property is, the more likely it is to be targeted by AI bots, and correspondingly, the more likely it is to block such requests.
Top N Internet properties by number of visitors seen by Cloudflare
% accessed by AI bots
% blocking AI bots
10
80.0%
40.0%
100
63.0%
16.0%
1,000
53.2%
8.8%
10,000
47.99%
8.92%
100,000
44.53%
6.36%
1,000,000
38.73%
2.98%
We see website operators completely block access to these AI crawlers using robots.txt. However, these blocks are reliant on the bot operator respecting robots.txt and adhering to RFC9309 (ensuring variations on user against all match the product token) to honestly identify who they are when they visit an Internet property, but user agents are trivial for bot operators to change.
How we find AI bots pretending to be real web browsers
Sadly, we’ve observed bot operators attempt to appear as though they are a real browser by using a spoofed user agent. We’ve monitored this activity over time, and we’re proud to say that our global machine learning model has always recognized this activity as a bot, even when operators lie about their user agent.
Take one example of a specific bot that others observed to be hiding their activity. We ran an analysis to see how our machine learning models scored traffic from this bot. In the diagram below, you can see that all bot scores are firmly below 30, indicating that our scoring thinks this activity is likely to be coming from a bot.
The diagram reflects scoring of the requests using our newest model, where “hotter” colors indicate more requests falling in that band, and “cooler” colors meaning fewer requests did. We can see the vast majority of requests fell into the bottom two bands, showing that Cloudflare’s model gave the offending bot a score of 9 or less. The user agent changes have no effect on the score, because this is the very first thing we expect bot operators to do.
Any customer with an existing WAF rule set to challenge visitors with a bot score below 30 (our recommendation) automatically blocked all of this AI bot traffic with no new action on their part. The same will be true for future AI bots that use similar techniques to hide their activity.
We leverage Cloudflare global signals to calculate our Bot Score, which for AI bots like the one above, reflects that we correctly identify and score them as a “likely bot.”
When bad actors attempt to crawl websites at scale, they generally use tools and frameworks that we are able to fingerprint. For every fingerprint we see, we use Cloudflare’s network, which sees over 57 million requests per second on average, to understand how much we should trust this fingerprint. To power our models, we compute global aggregates across many signals. Based on these signals, our models were able to appropriately flag traffic from evasive AI bots, like the example mentioned above, as bots.
The upshot of this globally aggregated data is that we can immediately detect new scraping tools and their behavior without needing to manually fingerprint the bot, ensuring that customers stay protected from the newest waves of bot activity.
If you have a tip on an AI bot that’s not behaving, we’d love to investigate. There are two options you can use to report misbehaving AI crawlers:
1. Enterprise Bot Management customers can submit a False Negative Feedback Loop report via Bot Analytics by simply selecting the segment of traffic where they noticed misbehavior:
2. We’ve also set up a reporting tool where any Cloudflare customer can submit reports of an AI bot scraping your website without permission.
We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection. We will continue to keep watch and add more bot blocks to our AI Scrapers and Crawlers rule and evolve our machine learning models to help keep the Internet a place where content creators can thrive and keep full control over which models their content is used to train or run inference on.
превод от украински Павлина Мартинова, изд. „Ерго“, 2023
Първото нещо, за което ме подсети заглавието на този сборник с разкази, е „За какво мислите“ – подканата на Facebook, когато още не си започнал да пишеш публикация. Украинският писател Васил Габор обаче е възможно най-далеч от света на социалните мрежи, от суетнята, суматохата и поредните скандали на деня. Писането му е лаконично и прозрачно, лишено от големи думи и обобщения, взряно в детайли, дреболии и чудеса. Писането му напомня на Радичковото, защото не се наблюдава умора от чудото, напротив, удоволствието от очарованието на света е невинно и сърцато.
Габор казва:
… всеки писател си има своя карма: едни пишат много, други – малко. Аз пиша много малко. Първата си оригинална „Книга на екзотичните сънища и реалните събития“ писах двайсет години и изписах в нея всички думи, които знаех, а ето към тази, втората, вървях почти петнайсет години, събирайки за нея нови думи, образи и впечатления. Но не бързах да пиша. Защото не ми се искаше да прилича на първата, а и впоследствие ми стана безинтересно да пиша разкази или новели, като спазвам строго изискванията на жанра и измислям нови герои.
Нямам усещането, че тази проява на скромност, за разлика от толкова много други, прикрива раздуто его и безмерни амбиции. Постепенно Габор стига до решението книгата му да бъде в японския жанр „дзуйхицу“, който се характеризира със свободно и асоциативно, фрагментарно и есеистично писане. Родоначалничката на жанра е легендарната и добре позната в България придворна дама Сей Шонагон със своите неподражаеми и красиви „Записки под възглавката“, създадени преди хилядолетие. Парадоксално, писането на Шонагон е изключително съвременно и днес заради списъците си, кратките истории и анекдоти, описанието на всекидневни случки и пейзажи. Някои от тях са точно като разговори с близък от днес или вчера, в чат или на живо.
Габор по спокоен, заземяващ и медитативен начин разкрива личния си и непосредствен свят. Той изрежда за какво мисли човек, докато отива на работа – за улицата, за тъжните неща, за света на детството, за това кое носи удоволствие и радва. Но мисли и за неща, за които не му се иска. Натрапчивите мисли могат да пробият всяка защита. Друг автор, с който спонтанно откривам близост, е Кенет Уайт, също заслужено популярен в България. Неговата особена смесица между източна уединена естетика и западна ирония и отказ от конформизъм също са в родословното дърво на поетиката на Габор.
На пръв поглед изглежда лесно или ненужно фокусът върху очевидното да е толкова интензивен акт, но в това няма нищо самоцелно. Популярни и болезнено актуални са думите на Оруел, че
вече сме потънали до дълбочина, в която потвърждаването на очевидното е първото задължение на интелигентните хора. Ако свободата изобщо означава нещо, то е правото да казваш на хората това, което не искат да чуят. Във времена на всеобща измама да казваш истината ще бъде революционен акт.
Когато Габор пише тази книга, в Лвов и в Украйна все още няма война, но тя от години е на прага. Неговият писателски и човешки опит е да реабилитира очевидното и простото, да създава противоотрова по един безхитростен и честен начин, да свидетелства за това, което е и каквото е, а не за това какво може да бъде. Далечни са му абстракцията и плакатната идеология. В писането на украинския автор има воля да види и разбере тайните неща, трагедията на птичетата, които са паднали от гнездото, да чуе човека в ъгъла, да види сградата на живота му, да види ангела в спящото дете. И не, това не са сантиментални и трогателни по кичозен начин разкази.
Споменах, че се родее с Радичков и с неговия поглед, взрян в неопетненото чудо. Габор също е от страната на светлата и добронамерена наивност. В нея има призраци и върколаци, магия и блаженство, съдба и случайност.
Това е проза, която е старовремска, класическа и ретро по възможно най-добрия начин. В нея го има палавия смях на джаза, но го има и оправдания плач и скръб по един свят, който вече е друг.
Във време на виртуална реалност да пишеш за това, което наистина живееш и виждаш, е важно. Важно е, защото светът не се превръща и няма да се превърне в метареалност, която има някакво отношение към интелектуални, културни или материални натрупвания. Светът няма да се побере в телефон или приложение. Не може да бъде опростен до черно и бяло. И това трябва да се припомня, защото самото нещо постепенно е измествано от символа, територията е замествана от картата и това отдалечаване от естеството на всекидневието започва още от детството.
Книгата на Габор е като гръмоотвод, който отвежда електричеството на тревогата и стреса под земята, обезопасява ги, за да не се превърнат фантазмите в реални болки и причини за крах.
Крахът е обратим. Сборникът няма такъв залог и амбиция, но създава пространство на силни емоции и екзистенциални размисли. Те резонират с масово вроденото днес безпокойство, но прочитът на тази книга е трансформиращо преживяване. Освежава и вълнува.
Няма да се въздържа да споделя част от биографията и възгледите на Габор, защото във времена на носталгици и популисти трябва да се припомнят и очевидни неща. Той е писател, литературовед и издател. Роден в Закарпатието през 1959 г. Завършва Факултета по журналистика в Лвовския университет през 1986 г. и работи в редакционните екипи на местни вестници и списания. От 1993 г. е в академичната библиотека в града. Там се съсредоточава върху изучаването на украинската периодика от XIX и XX век.
В независима Украйна аз, просто момче от планинско село, постигнах това, което някога изглеждаше невъзможно. Получих докторска степен и публикувах няколко научни книги, романи, есета, разкази. Освен това в продължение на две десетилетия ръководех издателската си инициатива „Частна колекция“. В рамките на този проект на бял свят се появиха над 200 книги и антологии на съвременни украински автори.
Габор е допринесъл и за възраждането на интереса към творчеството на забравени и забранени писатели и учени.
По времето на тоталитаризма подобни начинания биха били немислими. Ето защо за мен свободата е най-ценното понятие както в Украйна, така и в световен мащаб. За новите поколения, родени в независима Украйна, свободата е толкова естествена, колкото и въздухът, който дишат, и това е правилно. И все пак днес сме изправени пред сериозна заплаха от страна на Путинова Русия. Тя има за цел да ни лиши от най-ценните ни активи – нашата земя и нашата свобода. Твърдо вярвам, че творческото изразяване на всеки украински писател сега служи като оръжие в тази борба.
Сърцето на книгата, поне за мен, е в разказа за помориеца Стефан Кърнов и за черния пясък на плажа. От него раните зараствали, но не всички. Това обаче не отменя болезнено красивата нужда да се говори за тях.
Активните дарители на „Тоест“ получават постоянна отстъпка в размер на 20% от коричната цена на всички заглавия от каталога на издателство „Ерго“, както и на няколко други български издателства в рамките на партньорската програма Читателски клуб „Тоест“. За повече информация прочетете на toest.bg/club.
Никой от нас не чете единствено най-новите книги. Тогава защо само за тях се пише? „На второ четене“ е рубрика, в която отваряме списъците с книги, публикувани преди поне година, четем ги и препоръчваме любимите си от тях. Рубриката е част от партньорската програма Читателски клуб „Тоест“. Изборът на заглавия обаче е единствено на авторите – Стефан Иванов и Антония Апостолова, които биха ви препоръчали тези книги и ако имаше как веднъж на две седмици да се разходите с тях в книжарницата.
Tuesday 11 June 2024 will be remembered as one of the most important days in the history of Raspberry Pi.
At the London Stock Exchange on 11 June 2024.
The successful introduction of the Raspberry Pi Foundation’s commercial subsidiary on the London Stock Exchange is a genuinely remarkable achievement. I want to put on record my huge congratulations and thanks to Eben Upton, Martin Hellawell, and the whole team at Raspberry Pi Holdings plc for everything they have done to make this possible.
The purpose of the IPO was to secure the next stage of growth and impact for both the Foundation and the company. We have huge ambitions and the IPO has provided both organisations with the capital we need to pursue those ambitions at pace and scale. Our Chief Executive Philip Colligan has already explained what it means for the Raspberry Pi Foundation and our mission to empower young people all over the world.
In this post, I wanted to take a moment to acknowledge the significant contribution that others have made over the years, particularly all of the Trustees who have been so generous with their time, energy, and expertise.
Founding Trustees
The Raspberry Pi Foundation was established in 2008 by six founding Trustees: Alan Mycroft, David Braben, Eben Upton, Jack Lang, Pete Lomas, and Rob Mullins. All of them deserve credit and thanks for setting us off on this incredible journey.
Alan, Eben, Jack, and Rob were all involved with the Computer Lab at the University of Cambridge. They were dealing with a decline in applications to study the computer science undergraduate course, which was a symptom of the much wider challenge that far too many young people weren’t getting access to opportunities to learn computer science, or getting hands-on with programming and electronics.
David Braben brought an industry perspective, drawing on the challenges he was experiencing with recruiting engineers and programmers at the world-leading games company that he had founded, Frontier Developments.
Back in 2012 at the Sony factory that produces Raspberry Pi computers in Pencoed, Wales.
For Pete Lomas, he was paying forward the support and inspiration that he received from a college technician who gave him the opportunity and encouragement to experiment with programming a DEC PDP-8. That experience ultimately led Pete to establish Norcott Technologies, an electronics design and manufacturing business that he still runs today.
The founding Trustees’ original idea was to create a low-cost programmable computer — available for the price of a textbook — that would remove price as a barrier to owning a computer and inspire young people to take their first steps with computing. It took four years for the first Raspberry Pi computer to be launched, an achievement for which Eben and Pete were rightly honoured, along with other members of the team, as recipients of the prestigious MacRobert Prize for engineering.
Combining social impact and commercial success
What none of our founding Trustees could have predicted was the enormous commercial success of Raspberry Pi computers. In realising their vision of a low-cost programmable computer for education, the team created a new category of single-board computers that found a home with enthusiasts and industry, enabling the team to evolve — through hard work and creativity — into a business that is now entering a new phase as a listed company.
They also delivered on the original mission, with computer science at the University of Cambridge now being one of the most oversubscribed undergraduate courses in the country and many applicants citing Raspberry Pi computers as part of their introduction to programming.
The commercial success of Raspberry Pi has enabled the Foundation to expand its educational programmes to the point where it is now established as one of the world’s leading nonprofits focused on democratising access to computing education, and is benefiting the lives of tens of millions of young people already.
It takes a village
While no-one really knows the origin of the proverb ‘It takes a village to raise a child’, we can all recognise the truth in that simple statement. It applies just as much for endeavours like Raspberry Pi.
Over the years, Raspberry Pi has been a genuine team game. Employees in the Foundation and our commercial subsidiary, advisers, partner organisations and supporters, volunteers and community members have all played a crucial role in the success of both the company and the Foundation.
At a Raspberry Pi birthday celebration circa 2017.
Over the years there have been 21 Trustees of the Foundation, bringing an incredible range of skills and experience that has elevated our ambitions and supported the teams in both the Foundation and the company.
All of our Trustees have provided their time and expertise for free, never receiving any financial benefit for their contribution as Trustees.
Serving as a Trustee of a charity is a serious business, with significant responsibility and accountability. While many charities have commercial operations, there is no doubt that the scale and complexity of Raspberry Pi’s commercial business has placed significant additional responsibilities on all of our Trustees.
I especially want to pay tribute to my predecessors as chair of the Board of Trustees: Jack Lang, one of our founding Trustees, who sadly passed away this year; and David Cleevely, who continues to support our work as a Member of the Foundation. Both Jack and David played a particularly important part in the success of Raspberry Pi.
Welcoming our new Trustees
As we enter this new phase for the Foundation’s relationship with Raspberry Pi Holdings Ltd, we are delighted to welcome three new Trustees to the Board:
Andrew Sliwinski is a VP at Lego Education, formerly co-director of Scratch @ MIT, ex-Mozilla, and founder of DIY.org. Andrew is a technologist and maker with a deep understanding of education systems globally.
Laura Turkington leads global partnerships and programmes at EY, and was previously at Vodafone Foundation. Laura has extensive global experience (including Ireland and India), including supporting large-scale initiatives on digital skills, computing education, and AI literacy.
Stephen Greene is the founder and CEO of Rockcorps and the former chair of the National Citizen Service. Stephen brings huge experience of building global volunteer movements, social enterprise, marketing (especially to young people), government relations, and education of disadvantaged youth.
Many organizations have critical legacy Java applications that are increasingly difficult to maintain. Modernizing these applications is a necessary, daunting, and risky task that takes the focus off of creating new value or features. This includes undocumented code, outdated frameworks and libraries, security vulnerabilities, a lack of logging and error handling, and a lack of input validation. Amazon Q Developer simplifies and accelerates the modernization of existing Java applications. It can analyze code to highlight areas for potential improvements, assist with resolving technical debt, suggest code optimizations, and facilitate the transition to current frameworks and libraries.
This blog post explores how to modernize legacy Java applications using Amazon Q Developer. We will take an example of Unicorn Store API, a Java application with Java 8 running on Amazon Elastic Compute Cloud (Amazon EC2). First, we will upgrade the underlying runtime from Java 8 to Java 17 and other common dependencies, including Spring. Then, we will reduce technical debt within the code by improving modularity and logging. Finally, we will redeploy this application in a container image using a modern computing option, AWS Fargate.
The Unicorn Store API provides CRUD operations to manage Unicorn records in a database. It is built with Maven.
You will follow the below steps to modernize this application and bring it to Fargate using Amazon Q Developer.
Upgrade the application to Java 17 to leverage the latest features.
Reduce existing technical debt in the codebase.
Make the application cloud native and deploy it to AWS.
Outdated applications require increased effort to maintain security and stability. As a developer, you must continually relearn framework changes and optimizations that others have discovered in previous upgrades. The effort required to maintain the application makes it difficult to balance necessary updates with adding new features.
With Amazon Q Developer agent for code transformation, you can keep applications updated and supported in just a few steps. This removes vulnerabilities from unsupported versions, improves performance, and frees up time to focus on adding new features. Amazon Q Developer agent for code transformation accelerates application maintenance, upgrades, and migration in minutes. It enables developers to remove much of the undifferentiated work out of the tedious task of maintaining, upgrading and migrating existing application workloads, saving up to days’ or months’ worth of the undifferentiated work involved in moving from older language versions.
Let’s upgrade our Unicorn Store API from Java 8 to Java 17 using Amazon Q Developer agent for code transformation to leverage the latest features and optimization. In IntelliJ IDE, you enter /transform in the Amazon Q chat panel and provide the necessary details for Amazon Q Developer to start upgrading the project.
Amazon Q Developer agent for code transformation automatically analyzes the existing code, generates a transformation plan, and completes the transformation tasks suggested by the plan. While doing so, it upgrades popular libraries and frameworks to a version compatible with Java 17, including Spring, Spring Boot, JUnit, JakartaEE, Mockito, Hibernate, and Log4j to their latest available major versions. It also updates deprecated code components according to Java 17 recommendations. To start with the Amazon Q Developer agent for code transformation capability, you can read and follow the steps at Upgrade language versions with Amazon Q Developer agent for Code Transformation.
Once complete, you can review the transformed code, complete with build and test results, before accepting the changes.
Reduce technical debt in the codebase
Technical debt accumulates in any codebase over time. Some technical debt may be unavoidable to meet deadlines, but must be tracked and prioritized to pay back later. If left unmanaged, compounding technical debt will make development slower and expensive. Reducing technical debt should be an ongoing team effort, but often falls behind other priorities. Amazon Q Developer streamlines modernizing legacy Java code by identifying and remediating technical debt. Amazon Q Developer reduces the time and resources it takes to analyze the code by providing a list of issues that contribute to technical debt in a codebase. This makes it easy for software development teams to prioritize technical debt items and make informed decisions about which technical debt to address first.
Let’s find the list of technical debt in our Unicorn Store API. In IntelliJ IDE, use Send to Prompt option to send the highlighted code to the Amazon Q chat panel and prompt to provide a list of all technical debt. Amazon Q Developer lists all technical debt in detail.
Once you identify the technical debt, the next step is to gradually remediate them. Amazon Q Developer reduces the time it takes to implement the code to remediate the technical debt. As a developer, you can interact with Amazon Q Developer agent for software development within your IDE to get help with code suggestions for a specific task that you are trying to accomplish. It uses the code in whole project as context and provides an implementation plan that includes code updates it plans to make across all the files in the project. You can review the plan, and once you are satisfied with the plan, you can ask Amazon Q Developer to generate the code based on the proposed plan. This saves developers’ effort compared to manual updates.
For the technical debt identified for Unicorn Store API in the above step, let’s use Amazon Q Developer to address the missing logging technical debt. In IntelliJ IDE, enter /dev in the Amazon Q chat panel with the details on the logging technical debt. Amazon Q Developer generates an implementation plan and code to add logging based on the full project context. To get started with Amazon Q Developer agent for software development, you can refer to the steps at Develop software with the Amazon Q Developer agent for software development.
Modernizing legacy Java code requires continuous refactoring to incrementally enhance quality and avoid accumulating technical debt over time. Amazon Q Developer simplifies this iterative process through its Refactor capability. Amazon Q Developer provides a refactored version of the selected code, alongside explanations of each change and its coding benefit. It helps you to understand the changes by explaining each change and the benefit of making the change in the existing code. You can read further about this capability at Explain and update code with Amazon Q Developer.
Let’s leverage this feature to refine methods in the UnicornController class in our Unicorn Store API project. Amazon Q Developer furnishes the updated code with better code readability or efficiency, among other improvements, for you to review.
Make the application cloud native and deploy to AWS
The final step in the modernization journey is to make the application cloud-native and deploy to AWS. Cloud native is the software approach of building, deploying, and managing modern applications in cloud computing environments. These cloud native technologies support fast and frequent changes to applications without impacting service delivery, providing adopters with an innovative, competitive advantage. Let’s see how Amazon Q Developer can assist in making our Unicorn Store API project cloud native.
In IntelliJ IDE, open the Amazon Q Chat, and prompt Amazon Q Developer to provide a recommended approach to make the project cloud native and deploy to AWS.
Let’s ask Amazon Q Developer to implement the steps outlined in the previous chat conversation. First, ask Amazon Q Developer to create a docker file to containerize the application. The containerization process streamlines application development by decoupling the software from the underlying hardware and other dependencies. This approach enhances speed, efficiency, and security by isolating different components within the containerized environment.
Having successfully developed a container-based application, let’s leverage Amazon Q Developer’s capabilities to generate an AWS CloudFormation template. This template will enable us to deploy the required resources to AWS using Infrastructure as Code (IaC). IaC allows us to programmatically provision and manage our computing infrastructure, eliminating the need for manual processes and configurations. Manual infrastructure management can be time-consuming and error-prone, especially when dealing with large-scale applications.
To facilitate the creation of the CloudFormation template, let’s revisit the suggestions from our previous conversation and compile a list of the resources that need to be provisioned in AWS. Once you have this list, you can ask Amazon Q Developer to generate the CloudFormation template based on these resource requirements.
Amazon Q Developer can generate the CloudFormation template with all the required resources as outlined in the steps to deploy the container in AWS in a secure, reliable, and scalable manner.
Now that we have the CloudFormation template, once CloudFormation is deployed, let’s push the local docker image of our Unicorn Store API to Amazon ECR and start the Fargate tasks required to run the application in AWS.
In this way, you can use Amazon Q Developer to make your application cloud native by designing the steps to deploy to the cloud, helping migrate your application to container-based solution and even writes Infrastructure as code scripts to deploy your application to AWS.
Conclusion
Amazon Q Developer empowers developers to simplify and accelerate the modernization of legacy Java applications. By leveraging Amazon Q Developer, developers can bring outdated applications up to current frameworks and deploy them to AWS in a cloud-native architecture. This streamlines the process, reducing the effort, risk, and maintenance required. Developers save significant time and resources, which can now be used to focus on building new features and enhancing modernized applications rather than managing technical debt.
To learn more about Amazon Q Developer, see the following resources:
Fine-grained access control is a crucial aspect of data security for modern data lakes and data warehouses. As organizations handle vast amounts of data across multiple data sources, the need to manage sensitive information has become increasingly important. Making sure the right people have access to the right data, without exposing sensitive information to unauthorized individuals, is essential for maintaining data privacy, compliance, and security.
Today, Amazon DataZone has introduced fine-grained access control, providing you granular control over your data assets in the Amazon DataZone business data catalog across data lakes and data warehouses. With the new capability, data owners can now restrict access to specific records of data at row and column levels, instead of granting access to the entire data asset. For example, if your data contains columns with sensitive information such as personally identifiable information (PII), you can restrict access to only the necessary columns, making sure sensitive information is protected while still allowing access to non-sensitive data. Similarly, you can control access at the row level, allowing users to see only the records that are relevant to their role or task.
In this post, we discuss how to implement fine-grained access control with row and column asset filters using this new feature in Amazon DataZone.
Row and column filters
Row filters enable you to restrict access to specific rows based on criteria you define. For instance, if your table contains data for two regions (America and Europe) and you want to make sure that employees in Europe only access data relevant to their region, you can create a row filter that excludes rows where the region is not Europe (for example, region != 'Europe'). This way, employees in America won’t have access to Europe’s data.
Column filters allow you to limit access to specific columns within your data assets. For example, if your table includes sensitive information such as PII, you can create a column filter to exclude PII columns. This makes sure subscribers can only access non-sensitive data.
The row and column asset filters in Amazon DataZone enable you to control who can access what using a consistent, business user-friendly mechanism for all of your data across AWS data lakes and data warehouses. To use fine-grained access control in Amazon DataZone, you can create row and column filters on top of your data assets in the Amazon DataZone business data catalog. When a user requests a subscription to your data asset, you can approve the subscription by applying the appropriate row and column filters. Amazon DataZone enforces these filters using AWS Lake Formation and Amazon Redshift, making sure the subscriber can only access the rows and columns that they are authorized to use.
Solution overview
To demonstrate the new capability, we consider a sample customer use case where an electronics ecommerce platform is looking to implement fine-grained access controls using Amazon DataZone. The customer has multiple product categories, each operated by different divisions of the company. The platform governance team wants to make sure each division has visibility only to data belonging to their own categories. Additionally, the platform governance team needs to adhere to the finance team requirements that pricing information should be visible only to the finance team.
The sales team, acting as the data producer, has published an AWS Glue table called Product sales that contains data for both Laptops and Servers categories to the Amazon DataZone business data catalog using the project Product-Sales. The analytic teams in both the laptop and server divisions need to access this data for their respective analytics projects. The data owner’s objective is to grant data access to consumers based on the division they belong to. This means giving access to only rows of data with laptop sales to the laptops sales analytics team, and rows with servers sales to the server sales analytics team. Additionally, the data owner wants to restrict both teams from accessing the pricing data. This post demonstrates the implementation steps to achieve this use case in Amazon DataZone.
The steps to configure this solution are as follows:
The publisher creates asset filters for limiting access:
We create two row filters: a Laptop Only row filter that limits access to only the rows of data with laptop sales, and a Server Only row filter that limits access to the rows of data with server sales.
We also create a column filter called exclude-price-columns that excludes the price-related columns from the Product Sales
Consumers discover and request subscriptions:
The analyst from the laptops division requests a subscription to the Product Sales data asset.
The analyst from the servers division also request a subscription to the Product Sales data asset.
Both subscription requests are sent to the publisher for approval.
The publisher approves the subscriptions and applies the appropriate filters:
The publisher approves the request from the analysts in the laptops division, applying the Laptop Only row filter and the exclude-price-columns columns filter.
The publisher approves the request from the consumer in the servers division, applying the Server Only row filter and the exclude-price-columns columns filter.
Consumers access the authorized data in Amazon Athena:
After the subscription is approved, we query the data in Athena to make sure that the analyst from the laptops division can now access only the product sales data for the Laptop
Similarly, the analyst from the servers division can access only the product sales data for the Server
Both consumers can see all columns except the price-related columns, as per the applied column filter.
The following diagram illustrates the solution architecture and process flow.
Prerequisites
To follow along with this post, the publisher of the product sales data asset must have published a sales dataset in Amazon DataZone.
Publisher creates asset filters for limiting access
In this section, we detail the steps the publisher takes to create asset filers.
Create row filters
This dataset contains the product categories Laptops and Servers. We want to restrict access to the dataset that is authorized based on the product category. We use the row filter feature in Amazon DataZone to achieve this.
Amazon DataZone allows you to create row filters that can be used when approving subscriptions to make sure that the subscriber can only access rows of data as defined in the row filters. To create a row filter, complete the following steps:
On the Amazon DataZone console, navigate to the product-sales project (the project to which the asset belongs).
Navigate to the Data tab for the project.
Choose Inventory data in the navigation pane, then the asset Product Sales, where you want to create the row filter.
You can add row filters for assets of type AWS Glue tables or Redshift tables.
On the asset detail page, on the Asset filters tab, choose Add asset filter.
We create two row filters, one each for the Laptops and Servers categories.
Complete the following steps to create a laptop only asset row filter:
Enter a name for this filter (Laptop Only).
Enter a description of the filter (Allow rows with product category as Laptop Only).
For the filter type, select Row filter.
For the row filter expression, enter one or more expressions:
Choose the column Product Category from the column dropdown menu.
Choose the operator = from the operator dropdown menu.
Enter the value Laptops in the Value field.
If you need to add another condition to the filter expression, choose Add condition. For this post, we create a filter with one condition.
When using multiple conditions in the row filter expression, choose And or Or to link the conditions.
You can also define the subscriber visibility. For this post, we kept the default value (No, show values to subscriber).
Choose Create asset filter.
Repeat the same steps to create a row filter called Server Only, except this time enter the value Servers in the Value field.
Create column filters
Next, we create column filters to restrict access to columns with price-related data. Complete the following steps:
In the same asset, add another asset filter of type column filter.
On the Asset filters tab, choose Add asset filter.
For Name, enter a name for the filter (for this post, exclude-price-columns).
For Description, enter a description of the filters (for this post, exclude price data columns).
For the filter type, select Column to create the column filter. This will display all the available columns in the data asset’s schema.
Select all columns except the price-related ones.
Choose Create asset filter.
Consumers discover and request subscriptions
In this section, we switch to the role of an analyst from the laptop division who is working within the project Sales Analytics - Laptop. As the data consumer, we search the catalog to find the Product Sales data asset and request access by subscribing to it.
Log in to your project as a consumer and search for the Product Sales data asset.
On the Product Sales data asset details page, choose Subscribe.
For Project, choose Sales Analytics – Laptops.
For Reason for request, enter the reason for the subscription request.
Choose Subscribe to submit the subscription request.
Publisher approves subscriptions with filters
After the subscription request is submitted, the publisher will receive the request, and they can approve it by following these steps:
As the publisher, open the project Product-Sales.
On the Data tab, choose Incoming requests in the left navigation pane.
Locate the request and choose View request. You can filter by Pending to see only requests that are still open.
This opens the details of the request, where you can see details like who requested the access, for what project, and the reason for the request.
To approve the request, there are two options:
Full access – If you choose to approve the subscription with full access option, the subscriber will get access to all the rows and columns in our data asset.
Approve with row and column filters – To limit access to specific rows and columns of data, you can choose the option to approve with row and column filters. For this post, we use both filters that we created earlier.
Select Choose filter, then on the dropdown menu, choose the Laptops Only and pii-col-filter
Choose Approve to approve the request.
After access is granted and fulfilled, the subscription looks as shown in the following screenshot.
Now let’s log in as a consumer from the server division.
Repeat the same steps, but this time, while approving the subscription, the publisher of sales data approves with the Server only The other steps remain the same.
Consumers access authorized data in Athena
Now that we have successfully published an asset to the Amazon DataZone catalog and subscribed to it, we can analyze it. Let’s log in as a consumer from the laptop division.
In the Amazon DataZone data portal, choose the consumer project Sales Analytics - Laptops.
On the Schema tab, we can view the subscribed assets.
Choose the project Sales Analytics - Laptops and choose the Overview
In the right pane, open the Athena environment.
We can now run queries on the subscribed table.
Choose the table under Tables and views, then choose Preview to view the SELECT statement in the query editor.
Run a query as the consumer of Sales Analytics - Laptops, in which we can view data only with product category Laptops.
Under Tables and views, you can expand the table product_sales. The price-related columns are not visible in the Athena environment for querying.
Next, you can switch to the role of analyst from the server division and analyze the dataset in similar way.
We run the same query and see that under product_category, the analyst can see Servers only.
Conclusion
Amazon DataZone offers a straightforward way to implement fine-grained access controls on top of your data assets. This feature allows you to define column-level and row-level filters to enforce data privacy before the data is available to data consumers. Amazon DataZone fine-grained access control is generally available in all AWS Regions that support Amazon DataZone.
Try out the fine-grained access control feature in your own use case, and let us know your feedback in the comments section.
About the Authors
Deepmala Agarwal works as an AWS Data Specialist Solutions Architect. She is passionate about helping customers build out scalable, distributed, and data-driven solutions on AWS. When not at work, Deepmala likes spending time with family, walking, listening to music, watching movies, and cooking!
Leonardo Gomez is a Principal Analytics Specialist Solutions Architect at AWS. He has over a decade of experience in data management, helping customers around the globe address their business and technical needs. Connect with him on LinkedIn.
Utkarsh Mittal is a Senior Technical Product Manager for Amazon DataZone at AWS. He is passionate about building innovative products that simplify customers’ end-to-end analytics journeys. Outside of the tech world, Utkarsh loves to play music, with drums being his latest endeavor.
Amazon Redshift is a fast, scalable, secure, and fully managed cloud data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing ETL (extract, transform, and load), business intelligence (BI), and reporting tools. Tens of thousands of customers use Amazon Redshift to process exabytes of data per day and power analytics workloads such as BI, predictive analytics, and real-time streaming analytics.
As more and more data is being generated, collected, processed, and stored in many different systems, making the data available for end-users at the right place and right time is a very important aspect for data warehouse implementation. A fully automated and highly scalable ETL process helps minimize the operational effort that you must invest in managing the regular ETL pipelines. It also provides timely refreshes of data in your data warehouse.
You can approach the data integration process in two ways:
Full load – This method involves completely reloading all the data within a specific data warehouse table or dataset
Incremental load – This method focuses on updating or adding only the changed or new data to the existing dataset in a data warehouse
This post discusses how to automate ingestion of source data that changes completely and has no way to track the changes. This is useful for customers who want to use this data in Amazon Redshift; some examples of such data are products and bills of materials without tracking details at the source.
We show how to build an automatic extract and load process from various relational database systems into a data warehouse for full load only. A full load is performed from SQL Server to Amazon Redshift using AWS Database Migration Service (AWS DMS). When Amazon EventBridge receives a full load completion notification from AWS DMS, ETL processes are run on Amazon Redshift to process data. AWS Step Functions is used to orchestrate this ETL pipeline. Alternatively, you could use Amazon Managed Workflows for Apache Airflow (Amazon MWAA), a managed orchestration service for Apache Airflow that makes it straightforward to set up and operate end-to-end data pipelines in the cloud.
AWS DMS publishes the replicationtaskstopped event to EventBridge when the replication task is complete, which invokes an EventBridge rule.
EventBridge routes the event to a Step Functions state machine.
The state machine calls a Redshift stored procedure through the Redshift Data API, which loads the dataset from the staging area to the target production tables. With this API, you can also access Redshift data with web-based service applications, including AWS Lambda.
The following architecture diagram highlights the end-to-end solution using AWS services.
In the following sections, we demonstrate how to create the full load AWS DMS task, configure the ETL orchestration on Amazon Redshift, create the EventBridge rule, and test the solution.
Prerequisites
To complete this walkthrough, you must have the following prerequisites:
Use the following INSERT statements to load sample data into the sales staging table:
insert into dbo.fact_sales_stg(order_number,cust_id,order_amt) values (100,1,200);
insert into dbo.fact_sales_stg(order_number,cust_id,order_amt) values (101,1,300);
insert into dbo.fact_sales_stg(order_number,cust_id,order_amt) values (102,2,25);
insert into dbo.fact_sales_stg(order_number,cust_id,order_amt) values (103,2,35);
insert into dbo.fact_sales_stg(order_number,cust_id,order_amt) values (104,3,80);
insert into dbo.fact_sales_stg(order_number,cust_id,order_amt) values (105,3,45);
Create the stored procedures
In the Redshift query editor, create the following stored procedures to process customer and sales transaction data:
Sp_load_cust_dim() – This procedure compares the customer dimension with incremental customer data in staging and populates the customer dimension:
CREATE OR REPLACE PROCEDURE dbo.sp_load_cust_dim()
LANGUAGE plpgsql
AS $$
BEGIN
truncate table dbo.dim_cust;
insert into dbo.dim_cust(cust_key,cust_id,cust_name,cust_city) values (1,100,'abc','chicago');
insert into dbo.dim_cust(cust_key,cust_id,cust_name,cust_city) values (2,101,'xyz','dallas');
insert into dbo.dim_cust(cust_key,cust_id,cust_name,cust_city) values (3,102,'yrt','new york');
update dbo.dim_cust
set cust_rev_flg=case when cust_city='new york' then 'Y' else 'N' end
where cust_rev_flg is null;
END;
$$
sp_load_fact_sales() – This procedure does the transformation for incremental order data by joining with the date dimension and customer dimension and populates the primary keys from the respective dimension tables in the final sales fact table:
CREATE OR REPLACE PROCEDURE dbo.sp_load_fact_sales()
LANGUAGE plpgsql
AS $$
BEGIN
--Process Fact Sales
insert into dbo.fact_sales
select
sales_fct.order_number,
cust.cust_key as cust_key,
sales_fct.order_amt
from dbo.fact_sales_stg sales_fct
--join to customer dim
inner join (select * from dbo.dim_cust) cust on sales_fct.cust_id=cust.cust_id;
END;
$$
Create the Step Functions state machine
Complete the following steps to create the state machine redshift-elt-load-customer-sales. This state machine is invoked as soon as the AWS DMS full load task for the customer table is complete.
On the Step Functions console, choose State machines in the navigation pane.
Choose Create state machine.
For Template, choose Blank.
On the Actions dropdown menu, choose Import definition to import the workflow definition of the state machine.
Open your preferred text editor and save the following code as an ASL file extension (for example, redshift-elt-load-customer-sales.ASL). Provide your Redshift cluster ID and the secret ARN for your Redshift cluster.
Choose Choose file and upload the ASL file to create a new state machine.
For State machine name, enter a name for the state machine (for example, redshift-elt-load-customer-sales).
Choose Create.
After the successful creation of the state machine, you can verify the details as shown in the following screenshot.
The following diagram illustrates the state machine workflow.
The state machine includes the following steps:
Load_Customer_Dim – Performs the following actions:
Passes the stored procedure sp_load_cust_dim to the execute-statement API to run in the Redshift cluster to load the incremental data for the customer dimension
Sends data back the identifier of the SQL statement to the state machine
Wait_on_Load_Customer_Dim – Waits for at least 15 seconds
Check_Status_Load_Customer_Dim – Invokes the Data API’s describeStatement to get the status of the API call
is_run_Load_Customer_Dim_complete – Routes the next step of the ETL workflow depending on its status:
FINISHED – Passes the stored procedure Load_Sales_Fact to the execute-statement API to run in the Redshift cluster, which loads the incremental data for fact sales and populates the corresponding keys from the customer and date dimensions
All other statuses – Goes back to the wait_on_load_customer_dim step to wait for the SQL statements to finish
The state machine redshift-elt-load-customer-sales loads the dim_cust, fact_sales_stg, and fact_sales tables when invoked by the EventBridge rule.
As an optional step, you can set up event-based notifications on completion of the state machine to invoke any downstream actions, such as Amazon Simple Notification Service (Amazon SNS) or further ETL processes.
Create an EventBridge rule
EventBridge sends event notifications to the Step Functions state machine when the full load is complete. You can also turn event notifications on or off in EventBridge.
Complete the following steps to create the EventBridge rule:
On the EventBridge console, in the navigation pane, choose Rules.
Choose Create rule.
For Name, enter a name (for example, dms-test).
Optionally, enter a description for the rule.
For Event bus, choose the event bus to associate with this rule. If you want this rule to match events that come from your account, select AWS default event bus. When an AWS service in your account emits an event, it always goes to your account’s default event bus.
For Rule type, choose Rule with an event pattern.
Choose Next.
For Event source, choose AWS events or EventBridge partner events.
For Method, select Use pattern form.
For Event source, choose AWS services.
For AWS service, choose Database Migration Service.
For Event type, choose All Events.
For Event pattern, enter the following JSON expression, which looks for the REPLICATON_TASK_STOPPED status for the AWS DMS task:
For AWS service, choose Step Functions state machine.
For State machine name, enter redshift-elt-load-customer-sales.
Choose Create rule.
The following screenshot shows the details of the rule created for this post.
Test the solution
Run the task and wait for the workload to complete. This workflow moves the full volume data from the source database to the Redshift cluster.
The following screenshot shows the load statistics for the customer table full load.
AWS DMS provides notifications when an AWS DMS event occurs, for example the completion of a full load or if a replication task has stopped.
After the full load is complete, AWS DMS sends events to the default event bus for your account. The following screenshot shows an example of invoking the target Step Functions state machine using the rule you created.
We configured the Step Functions state machine as a target in EventBridge. This enables EventBridge to invoke the Step Functions workflow in response to the completion of an AWS DMS full load task.
Validate the state machine orchestration
When the entire customer sales data pipeline is complete, you may go through the entire event history for the Step Functions state machine, as shown in the following screenshots.
Limitations
The Data API and Step Functions AWS SDK integration offers a robust mechanism to build highly distributed ETL applications within minimal developer overhead. Consider the following limitations when using the Data API and Step Functions:
To avoid incurring future charges, delete the Redshift cluster, AWS DMS full load task, AWS DMS replication instance, and Step Functions state machine that you created as part of this post.
Conclusion
In this post, we demonstrated how to build an ETL orchestration for full loads from operational data stores using the Redshift Data API, EventBridge, Step Functions with AWS SDK integration, and Redshift stored procedures.
Ritesh Kumar Sinha is an Analytics Specialist Solutions Architect based out of San Francisco. He has helped customers build scalable data warehousing and big data solutions for over 16 years. He loves to design and build efficient end-to-end solutions on AWS. In his spare time, he loves reading, walking, and doing yoga.
Praveen Kadipikonda is a Senior Analytics Specialist Solutions Architect at AWS based out of Dallas. He helps customers build efficient, performant, and scalable analytic solutions. He has worked with building databases and data warehouse solutions for over 15 years.
Jagadish Kumar (Jag) is a Senior Specialist Solutions Architect at AWS focused on Amazon OpenSearch Service. He is deeply passionate about Data Architecture and helps customers build analytics solutions at scale on AWS.
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
Deepmala Agarwal works as an AWS Data Specialist Solutions Architect. She is passionate about helping customers build out scalable, distributed, and data-driven solutions on AWS. When not at work, Deepmala likes spending time with family, walking, listening to music, watching movies, and cooking!
Leonardo Gomez is a Principal Analytics Specialist Solutions Architect at AWS. He has over a decade of experience in data management, helping customers around the globe address their business and technical needs. Connect with him on 
Utkarsh Mittal is a Senior Technical Product Manager for Amazon DataZone at AWS. He is passionate about building innovative products that simplify customers’ end-to-end analytics journeys. Outside of the tech world, Utkarsh loves to play music, with drums being his latest endeavor.
Ritesh Kumar Sinha is an Analytics Specialist Solutions Architect based out of San Francisco. He has helped customers build scalable data warehousing and big data solutions for over 16 years. He loves to design and build efficient end-to-end solutions on AWS. In his spare time, he loves reading, walking, and doing yoga.
Praveen Kadipikonda is a Senior Analytics Specialist Solutions Architect at AWS based out of Dallas. He helps customers build efficient, performant, and scalable analytic solutions. He has worked with building databases and data warehouse solutions for over 15 years.
Jagadish Kumar (Jag) is a Senior Specialist Solutions Architect at AWS focused on Amazon OpenSearch Service. He is deeply passionate about Data Architecture and helps customers build analytics solutions at scale on AWS.