At NVIDIA’s GTC conference, Adobe’s CTO Ely Greenfield walked the audience through the company’s three-year journey building frontier generative AI models from scratch. And the AI training data pipeline required to keep thousands of GPUs productive.
Along the way, he showed a profiler readout from their early training runs—a visualization of exactly how much time each GPU was spending on actual computation versus sitting idle. It revealed that roughly two-thirds of GPU time was spent simply waiting for data. And that idle time had a price tag.
“If we were putting a million dollars into training,” he told the room, “that was $600,000 we were burning away on GPUs sitting and doing nothing.” In other words, roughly sixty cents of every dollar spent on GPU compute was being wasted.
Greenfield’s team traced the waste to two culprits, neither of which is unique to Adobe: whether data can reach your GPUs fast enough, and whether the work it represents is distributed evenly once it arrives.
One culprit is a storage problem. The other isn’t, but you won’t be able to fix it until you’ve solved the first one.
Culprit #1: Storage and retrieval speed
Adobe’s training data lived in petabytes of distributed cloud storage and had to be shipped out to thousands of GPUs constantly, over standard Ethernet. The dataset itself was enormous and varied—images and video, low-res and high-res, simple formats and expensive codecs, all moving at once. Standard networking, which was built for retrieving individual files on request, wasn’t designed for that kind of sustained, parallel, petabyte-scale movement. It became a massive bottleneck.
Checkpointing compounded the problem. As a safeguard, the training run would periodically write a complete copy of the model back to storage. Most of those checkpoints were never needed again. But writing and reading them still consumed real GPU time, which cost a lot of money whether they were used or not.
How Adobe fixed it
Adobe’s solution had two parts.
First, they replaced standard Ethernet with a high-performance networking fabric designed for the petabyte-scale traffic distributed AI training generates. Now, data could finally move at the pace their GPUs needed.
Second, they changed how they saved checkpoints. Instead of writing one giant file containing the entire model, they began breaking the model into smaller pieces and saving pipeline fragments to many places at once. Saving and loading checkpoints now takes significantly less time than it used to. If a checkpoint ever needs to be reassembled, that’s slightly slower, but it’s a rare event. The savings on every other write happen continuously, across thousands of GPUs, around the clock.
The underlying lesson is that for model training, parallel access and sustained high throughput are baseline requirements. Training data has to be immediately accessible at the pace your GPUs consume it, not tucked away in a storage tier that takes minutes or hours to retrieve.
Culprit #2: The data loader problem
A balanced data loader can only do its job if the data it’s balancing is actually available the moment it’s needed. That makes fast storage a precondition for everything else in the pipeline. But even after fixing the storage and retrieval speed culprit, Adobe still had a problem: the way training data was divided across the cluster meant some GPUs were doing far more work than others.
Their pipeline used a straightforward data-parallelism approach: slice the training data into equal-sized chunks and assign one chunk to each GPU in the cluster. But equal-sized chunks weren’t equal work. Some GPUs got simple, low-resolution assets that processed in seconds; others got large, complex files that took minutes. The fast GPUs finished early and sat idle waiting for the slow ones to catch up. And then all of them waited again while their results were merged into one updated model before the next round could begin.
How Adobe fixed it
Adobe stopped treating all data as equivalent. They custom-built a balanced data loader that understood the processing cost of each asset and distributed work so every GPU finished at roughly the same time. Then they restructured how computation was divided across the cluster to make the merging step dramatically cheaper.
A perfectly balanced data loader still idles if the data it’s waiting on hasn’t arrived yet. This is why storage can’t be an afterthought. The loader optimizes what happens once data is there, but storage determines whether it’s there at all.
And building something like Adobe’s balanced data loader takes real engineering investment, including time spent profiling workloads, testing distribution strategies, and tuning until every GPU finishes at roughly the same pace. Teams still fighting storage bottlenecks rarely get to that work. Their engineers are busy figuring out why GPUs are idle in the first place, not optimizing how work gets distributed once data arrives. Removing the storage bottleneck frees up the engineering time needed to tackle the data loader problem properly.
From 40 to 80 cents
After addressing both culprits, Adobe’s GPUs ran at roughly 80% utilization. After accounting for the coordination overhead inherent to running thousands of machines together, 80% is close to the practical ceiling. That’s the difference between a GPU cluster that’s mostly waiting and one that’s mostly working.
The Storage Side of the Solution
Backblaze B2 Overdrive addresses the storage side of exactly the problem Adobe ran into. B2 is always-hot object storage, with no tiering, no retrieval delays, and no waiting for data to be promoted from a cold tier before training can begin. Training data stays immediately accessible whether it was written an hour ago or six months ago.
But availability alone isn’t enough if data can’t move fast enough to keep up with the cluster. B2 Overdrive adds the throughput layer: at up to 1Tbps, it’s designed to keep petabyte-scale GPU clusters fed continuously, not just handle occasional bursts of traffic. Both layers are S3-compatible, so they drop into existing PyTorch or TensorFlow pipelines without a rewrite. And because egress is free, moving training data between storage and GPU compute—across regions or providers—doesn’t add a cost penalty on top of a performance one.
Checkpointing benefits from the same foundation. Adobe’s solution of breaking the model into fragments and saving them to many places at once only works if the underlying storage is fast and parallel enough to make it pay off. Always-hot, high-throughput storage makes checkpoint writes and recoveries faster across the board, so they cost less GPU time whether they happen rarely or often.
Backblaze can’t write your data loader for you, but it can make sure that once you’ve built one, it isn’t waiting on storage to do its job.
Remind is a
command-line calendar and alarm program, with an optional Tk-based graphical
interface, for Linux and Unix-like operating systems. It has its own scripting
language that allows users to create reminders that are difficult (if not
impossible) to specify in other calendaring programs. It is wholly unsuitable for use in
corporate environments that require calendar sharing and exchanging meeting
invitations; however, it may be precisely the calendaring tool for users who
prefer the command line and fast, flexible tools that can help keep track of
messy schedules.
Vanilla
OS 3, an immutable desktop Linux distribution, has been released. Notable
changes in this release include support for Arm64, introduction of a Vanilla OS SDK,
a rewrite of the Apx package
manager using the new SDK, and much more. LWN covered Vanilla OS 2 in
2024.
Всички знаем, че в строителството има нередности. Не са нужни дори пет минути в който и да е град в България без да се сблъскаш с пресни примери разпознаващи се с просто око. Всички знаем, че има изисквания за озеленяване при строежи, както и знаем, че повсеместно не се спазват. На места липсва съвсем озеленяване, а на повечето от останалите е бутафорно или вече заменено с паркоместа и кафенета.
Озеленяването не е просто желание или за красота. Има критична роля в чистотата на въздуха, намаляване на шумовото замърсяване, защитата от наводнения и биоразнообразието в града, което от своя страна помага за намаление на вредители. Специално за наводненията видяхме ярък пример наскоро, където наистина сериозно количество дъжд веднага потече по улиците вместо да се задържи от огромното количество сгради построени наскоро. Запечатването на повърхностния слой е сериозен проблеми прави озеленяването все по-важно.
Да започнем от началото
Писал съм и преди за озеленяването, изискванията и защо не се спазва. Описах и колко е мъчно да се накара районна администация да направи проверка и да защити дори няколко квадрата трева и едно дърво. Писах какво обществото има предвид когато говори за презастрояване и че зелените площи са съществен компонент от това. Последната статия, впрочем, беше подбудена от среща с тогавашния в.и. главен архитект. Това той сподели личното си мнение, че изискванията за озеленяване били неадекватни и затова той и други не следели за тях много-много. Възможно е и да има връзка с оценката, която други негови колеги дадоха, че при строго спазване на дори старите изисквания интензивността на новото застрояване на София би намаляла с 20 до 30%, а доста сгради построени в последните пет години биха загубили я паркоместата си, я цял вход или крило. Всъщност, изискванията за озеленяването са дори твърде занижени и дават предпоставки за злоупотреби.
От доста време се опитвам да разбера този проблем от нормативна, административна, практическа и дори корупционна гледна точка. Затова съм пускал десетки искания за достъп до обществена информация до различни институции. В повечето случаи получавам входящи номера или части от документи, които хем нарочно не ми дават никаква информация, хем номинално се водят за отговор. Затова подобрявах исканията ми по ЗДОИ и питах отново. В други случаи получавах отговор, че такава информация или документи липсват, което само по себе си е полезно, защото знам, че лъжат и ми помага в следващото искане. В трети случаи се налагаше да изчаквам шест месеца да пусна ново искане с надеждата, че ще забравят с какви извинения са отказали преди или какво са скрили с справките. Последното работи учудващо добре.
След четири години игра на котка и мишка се фокусирах върху четири сгради като примери за подобни практики. Не защото са специални, а просто защото често минавам покрай тях и виждам какво се случва. Отделно са добри примери за наглост в контраст със заявките за плувнали в зеленина и устойчивост сгради. В крайна сметка едва тази година получих достъп до документите на проектите им и мога да говоря с числа и конкретни примери за нарушенията.
Ще ги събера във втората част на тази статия. Първо трябва да обсъдим защо беше толкова труден процесът. Подобно на промените в имотния регистър ще покаже отново лобистки текстове в закона и нарочна липса на прозрачност целяща опазването на именно такива схеми. Надявам се, че ще внесе и повече разбиране какво виждате в снимките в следващата част.
Преди това обаче важно уточнение: знам, че далеч не всички термини, които използвам са точни и обясненията изчерпателни. Целта ми тук е да внеса яснота на разговорен език, а не да цитирам наредби и лекции по архитектура. Ще се радвам на всяко допълнение и корекция и ще го обсъдим в коментарите.
Какво точно търсех?
Обичам да казвам, че отворените данни ни помагат да задаваме по-добри въпроси. Същото може да се каже и тук. Всички гледаме в разрешението за строеж, а то е само последната фаза от дълъг процес понякога продължаващ над десетилетие. Преди него има виза за проектиране, често промяна на ПУП, серия от градоустройствени заповеди, обсъждания в специализирани комисии и всичко това на база инвестиционни проекти. Нерядко има и промени и заповеди след разрешението за строеж като намаляване броя на паркоместата, промени в разпределението, а в един случай, на който попаднах тия дни – цялостна промяна на предназначението, конструкцията, мястото и всичко по сградата независимо, че оригиналното разрешение вече няма правно действие.
Всички тези стъпки носят риск за нарушения, пропуски, умишлено забавяне и корупция, особено предвид, че процесът упорито не се дигитализира, за да е лесно проследим. Част от документите като искания за промени на ПУП, издадените визи и заповеди са публични. Строителните книжа заедно с всички тези административни документи следва да се качват в регистъра по ЗУТ, който обаче все още не е видял бял свят – мина обсъждане и от началото на мандата чака само подписа на Шишков.
Час от тези строителни книжа и инвестиционното намерение са и плановете за озеленяване. Това са няколко скици, които в детайли показват къде ще са зелените площи, колко ще е дълбок почвения слой, къде и колко дървета и храсти ще има и какъв процент от изискванията ще бъде постигнат.
Защо е важно?
Плановете за озеленяване са неразделна част от проекта и на тяхна база се издава разрешение за строеж. В последствие приемателната комисия сравнява тези и други документи с това, което на практика виждат на място и решават дали да издадат акт 16. Това е поне на теория. Конкретно за София и озеленяването подписът е на служител на районната община. По принцип трябва да е ландшафт архитект, но в практиката често самият районен кмет или трето лице, на което е делегирал, се подписва. Длъжни са обаче и потвърждават с подписва си, че довършеният имот отговаря на описаното в плана.
В статията си за бутафорното озеленяване описах някои от изискванията в наредбата за зелената система в София. Някои ключови аспекти са:
Почвеният слой трябва да е минимум 120 см. дълбочина при дърветата, а ако са в кашпа – поне 1 м3 обем на почвата
За храсти почвения слой трябва да е поне 60 см, а за трева – поне 40 см.
При строежи над 2 декара – задължителен резервоар за дъждовна вода и използването ѝ за поливане на зелените площи
Отстояние на дървета от сгради – 1.5 м. или и 3 м., ако дървото стига 5 метра.
Отстояние от бордюри – 70 см.
Отстояние от пътни платна – 2 м.
Отстояние от откоси и тераси – 1м.
Отстояние от стълбове – 4м.
Пример за липса на отстояние и достатъчно открита почва – само 30 см. от бордюра и под 50 см. страна. Дървото никога няма да се развие и има нужда от напояване, за да е умре съвсем.
Тези изисквания не значат, например, че не може да се засади дърво точно до бордюра на пътя без достатъчно почва. Безсмислено би било, защото ще изсъхне, но също така не следва да се брои при смятането на озеленяването. Това е важно, защото освен дялът земя като зелена площ има изискване и какъв процент от нея е във висока дървесна растителност. За широколистни дървета с височина между 3 и 5 м. на 12 годишна възраст броят 12 кв.м. площ за това изчисление. Над 5 м. – 20. При иглолистните е съответно 5 и 7 кв.м.
Ключов аспект тук е, че доста от тези изисквания влизат в сила през юли 2023, т.е. за разрешения за строеж издадени преди това важат старите разпоредби за отстояния и почвен слой. През 2019 г., например, изискват да има почвен слой от поне 60 см., а когато е не по-малко от 30 см. площта важи с индекс 50% към общия коефициент на озеленяване. Отстоянията от бордюри, стени и тераси обаче са били същите.
Трябва да се разбере също, че плановете не са абсолютни. Това е просто илюстрация как си представят, че ще стане озеленяването, но на терен може да се окаже друго. Аналогично скиците къде ще бъдат сградите почти никога не отговарят в мащаб – отстоянията не са същите, височината не е тази и прочие. Затова ключовия момент е при приемане на обект на груб строеж да не се гледа по документи, а да се измерват реалната височина и отстояния. Аналогично при премателната комисия не се гледа дали точно в това пространство има дърво, а колко са на брой, дали са същия вид (има значение колко големи ще станат) и дали почвения слой и отстоянията са правилните.
Защо не виждаме тези планове?
Най-честата причина за отказ при поискване на тези скици, е че се засягали правата на трети лица. Това са инвеститорските компании и архитектите. Те от своя страна изрично са искали да не получавам плановете за озеленяване. Доколкото не изясняват никога в отговорите си точно какви права биха били засегнати и по какъв начин, може само да гадаем, че всъщност замесените се притесняват от разкриване на нарушения.
Подобни откази са доста спорни, защото става въпрос за документи неразделна част от строителните книжа. ЗУТ изрично посочва, че поне част от тях трябва да се публикуват в регистъра (когато види бял свят), т.е. са публични по дефиниция. Отделно свободно може да се използват в рамките на административен процес и производство, част от който спокойно да изиска публичността им какъвто вече в случая с исканията за промяна на ПУП и други аналогични скици от същите инвестиционни проекти. Няма основателна причина да се прилагат тези правила за публичност за едни чертежи, а не за други. Най-вече законът за достъп до обществена информация предвижда, че дори да има съмнение за засегнати права, при надделяващ обществен интерес може да се предостави достъп при минимизиране на предполагаемата щета. Т.е. могат да редактират имена и елементи, които ги притесняват и пак да предоставят скиците.
Експлоатиране на авторски права за прикриване
Попаднах и на друг аргумент – авторски права. Законът третира архитектурните планове и всичко свързано с градоустройството по любопитен начин. През последните 10 години са вкарани множество стратегически промени, които силно ограничават разпространението на архитектурни планове освен, ако изрично не се изисква от друг закон – например както ЗУТ дава възможност това да се определели с наредба. Тук ограничението сериозно надвишава опазване на имуществените и неимуществените права. Забранили са притежанието на копия от архитектурни планове дори за лична употреба без търговска цел.
Както се сещате, в тази хипотеза попадат и плановете за озеленяване като част от архитектурния проект. Те съдържат лични данни като кой ги е изготвил и подписал. Тези данни стандартно се заличават по GDPR. Това от своя страна създава интересна хипотеза – ако все пак получа тези скици със скрити имена и подписи, мога ли да спазя Закона за авторските права и да опиша както съм задължен източникът и авторът му, когато не го знам.
Голяма част от възможностите за т.е. „безкористно използване“ на авторски материали като журналистически материали, критика, сатира и прочие нарочно и изрично изключват тези свързани с архитектурата и градоустройството. Трудно ми е да определя тези точки като лобистки опити да се скрие ключова за обществото информация злоупотребявайки с принципите на авторското право. Аналогичен случай имах преди години, когато направих карта на разрешителните за сеч в България и няколко лесовъда ме заплашиха със съд, тъй като имали авторски права върху актовете и никой нямал право да ги използва без тяхно съгласие.
Административни документи в квантова суперпозиция
Всичко описаното до тук не важи за официалните актове и документи издадени от различните институции като част от административния процес. По принцип всички те са по подразбиране публични и специално тези по ЗУТ трябва вече да се публикуват от общини, ДНСК и министерства. Някои, които засягат вътрешни процедури и актове стават публични и обект на поискване по ЗДОИ две години след издаването им. Дори за тях никой държавен служител няма право да откаже предоставяне на вътрешните номера на такива документи, както и да признае съществуването им, освен ако не са засекретени. Тук няма значение дали същите се отнасят или засягат трети лица.
Това значи, че документ като протокола от държавната приемателна комисия, разрешението за ползване и всички документи издадени от общината и ДНСК следва да са публични и то най-късно две години след датата. Отделно копия от протоколите се дават на подписалите ги страни и членове на комисията.
Именно заради тези важни аспекти обжалвах два отказа на ДНСК да ми предоставят точно такива актове издадени от тях, както и протоколи по приемане. В тях се съдържа ключова информация какво са видели, какви забележки са имали и кой се е подписал, че всичко е наред. Делата са насрочени в рамките на следващия месец и се надявам до края на годината да имаме резултат.
Компромисният път
Авторските права в България са доста строги и подобни тенденциозни ограничения специално за архитектурата съвсем естествено притесняват всеки. Дори когато няма зла умисъл и опит за прикриване, това механично води до отказ от действие „за всеки случай“. Както споменах, ЗДОИ дава възможност да се търсят алтернативи при надделяващ обществен интерес. В случая несъмнено има такъв предвид изложеното в началото на този текст. Затова на няколко поредни искания за достъп до обществена информация от Столична община ми дадоха възможност да видя все пак плановете за озеленяване. Условието бе нямам дигитално копие, да ги чета само физически на място и да не правя снимки.
Това ми позволи да видя истинските параметри, разпределени на озеленяването, колко дървета, вертикално озеленяване и отстояния са обещали, както и как тези планове са се променяли с времето. Това е важно, защото често не знаем какво търсим докато не го видим. Не стига просто да се искат няколко числа от документите. Пробвал съм и дори такива са ми били отказвани с аргумента, че не им била работа да ровят по дигиталните архиви. Тази седмица получих такъв отказ по свързана тема, за която ще пиша също скоро. Нямах възможност да видя също протоколите от приемателната комисия, защото те са били при ДНСК, а районната администрация, където са разглежданите сгради и е била страна, отрича да има копие.
Затова в следващата част на тази статия няма да мога да ви покажа самите планове. Първо, защото години наред получавах откази от вече бивши главни архитекти. Второ заради ограниченията в закона за авторското право целящи именно избягване на подобна прозрачност. Трето, защото все още нямаме публичен регистър по ЗУТ, който да се надгради да включва документи защитаващи обществения интерес заедно с други подобни като транспортния анализ и становищата за пожарна безопасност.
Този начин на предоставяне е силно ограничаващ, най-малкото защото мога само да преразкажа какво съм видял на база няколко числа в наредбата. Ландшафт архитект би хванал с един погледа всички нередности, ако документите бяха публични. Все пак, Столична община направи една крачка повече от всички други институции и предишни администрации и ми даде поне някакъв достъп. Следващата стъпка би била довършването на регистъра по ЗУТ от Шишков и изискване да се качват този тип документи там.
В близките дни ще публикувам следващата част от темата, че която ще покажа с конкретни примери какво видях, какво липсва и защо е толкова нужна прозрачност и безкомпромисност.
Andy Ellis has a roundup of the security vendors at Black Hat this year.
Key Takeaways: We have entered into an AI world. While nearly half of booths didn’t directly mention AI or agents in their taglines, the effects of AI are everywhere. Multiple spaces (Identity, SaaS, AppSec, Data) have almost every vendor leading with AI; existing unsolved problem areas just got worse.
At the same time, there’s a clear trichotomy in the market: tools that tell you how bad things are; tools that stop adversaries, and tools that prevent problems from occurring. While you’d suspect that the tools that fix things would dominate, the tools that merely tell you how bad things are seem to be frustratingly plentiful.
Young people see and use artificial intelligence (AI) systems frequently, and hear a lot about how they are applied across many areas of life. That makes them understandably eager to find out more about how AI works. For some, this interest extends to wanting to experiment with the technology to investigate their own questions and build something themselves. How can schools support learners who want to take their interest further and gain recognition for the knowledge and skills they develop along the way?
For learners in the UK, there is currently no established qualification that develops technical AI skills. This can make it difficult to create space in the curriculum for dedicated AI learning, especially for learners from age 14, for whom preparation for high-stakes exams is prioritised over non-examined learning.
In the meantime, we are starting to support students in England who choose an established route for extending their skills and getting that effort recognised: the Extended Project Qualification (EPQ).
An established qualification with room to explore AI
The EPQ is a well-established qualification for 16- to 19-year-olds offered in England, Northern Ireland, and Wales by several exam boards (including AQA, Pearson Edexcel, OCR, Eduqas/WJEC, and City & Guilds). It is completed by around 1 in 10 learners at this stage of education.
Learners pursuing an EPQ undertake an extended, self-directed investigation or project. They choose an area of interest where they want to develop their own project and take responsibility for planning, completing, and evaluating the project work.
Its self-directed nature makes the EPQ particularly well suited to learners who want to explore emerging areas, such as artificial intelligence and data science. For students fascinated by AI, an EPQ offers an opportunity to do more than simply learn about the technology. With the right foundations and support, they can investigate a question of their own, build a machine learning model, and use that work to gain a recognised qualification.
We are now working to offer those foundations through a new introductory ‘Data Science and AI’ course that will enable learners to build a machine learning model as their EPQ artefact.
Building the foundations for an independent AI investigation
Creating a machine learning model involves much more than choosing a statistical technique and using easily available training data. It involves understanding the problem being addressed, deciding what data to use, exploring and preparing that data, building and evaluating a model, and interpreting the model’s outputs.
Learners who want to do this work for their EPQ need not only technical knowledge and skills, but also skills for critical thinking, research, decision-making, evaluation, and reflection — skills that are valuable for any further study and future career.
To help learners develop these foundations, we are creating an introductory ‘Data Science and AI’ course, based on our curriculum framework for data science and designed to be completed before learners begin their AI-focused EPQ.
The course will guide students through a structured investigation based on a recognised data science lifecycle. By working through it, students will:
Learn about each stage of the data science lifecycle
Develop the technical understanding they need to make independent decisions during an AI investigation
Gain practical experience of working with data and building a machine learning model
Learn about key concepts behind the techniques they use, through theoretical content that supports them to understand not just what to do, but why
The course will consist of 10 units, each involving 2 to 3 hours of independent study for learners. We are designing the activities with a no-code approach to allow students to focus on understanding data science concepts and processes.
Supporting independent project work
Importantly, the work learners produce during this course won’t be the EPQ investigation itself. Working through ‘Data Science and AI’ will give learners the knowledge and skills they need to undertake an AI-focused project with greater independence and confidence.
After they have completed the course, when they begin their EPQ project pursuing a question that interests them, students can apply their new skills and knowledge to make decisions about their AI investigation and document their progress.
We are looking forward to supporting learners in England to turn their interest in one of today’s most significant technologies into the opportunity to develop independent research and project skills through a recognised qualification.
What’s next?
The ‘Data Science and AI’ course will be available to selected schools in England from September 2026, where students and teachers will test the materials and we will learn from their feedback to improve the course materials. We plan to make the course freely available in 2027.
Young people see and use artificial intelligence (AI) systems frequently, and hear a lot about how they are applied across many areas of life. That makes them understandably eager to find out more about how AI works. For some, this interest extends to wanting to experiment with the technology to investigate their own questions and build something themselves. How can schools support students who want to take their interest further and gain recognition for the knowledge and skills they develop along the way?
Currently, there is no widely available subject qualification in AI for students in the UK. This can make it difficult to create space in the curriculum for dedicated AI learning, especially for students from age 14, for whom preparation for high-stakes exams is prioritised over non-examined learning.
In the meantime, we are starting to support students in England who choose an established route for extending their skills and getting that effort recognised: the Extended Project Qualification (EPQ).
An established qualification with room to explore AI
The EPQ is a well-established qualification for 16- to 19-year-olds offered in England, Northern Ireland, and Wales by several exam boards (including AQA, Pearson Edexcel, OCR, Eduqas/WJEC, and City & Guilds). It is completed by around 1 in 10 learners at this stage of education.
Learners pursuing an EPQ undertake an extended, self-directed investigation or project. They choose an area of interest where they want to develop their own project and take responsibility for planning, completing, and evaluating the project work.
Its self-directed nature makes the EPQ particularly well suited to learners who want to explore emerging areas, such as artificial intelligence and data science. For students fascinated by AI, an EPQ offers an opportunity to do more than simply learn about the technology. With the right foundations and support, they can investigate a question of their own, build a machine learning model, and use that work to gain a recognised qualification.
We are now working to offer those foundations through a new introductory ‘Data Science and AI’ course that will enable learners to build a machine learning (ML) model as their EPQ artefact.
Building the foundations for an independent AI investigation
Creating an ML model involves much more than choosing a statistical technique and using easily available training data. It involves understanding the problem being addressed, deciding what data to use, exploring and preparing that data, building and evaluating a model, and interpreting the model’s outputs.
Learners who want to do this work for their EPQ need not only technical knowledge and skills, but also skills for critical thinking, research, decision-making, evaluation, and reflection — skills that are valuable for any further study and future career.
To help learners develop these foundations, we are creating an introductory ‘Data Science and AI’ course, based on our curriculum framework for data science and designed to be completed before learners begin their AI-focused EPQ.
The course will guide students through a structured investigation based on a recognised data science lifecycle. By working through it, students will:
Learn about each stage of the data science lifecycle
Develop the technical understanding they need to make independent decisions during an AI investigation
Gain practical experience of working with data and building a machine learning model
Learn about key concepts behind the techniques they use, through theoretical content that supports them to understand not just what to do, but why
The course will consist of 10 units, each involving 2 to 3 hours of independent study for learners. We are designing the activities with a no-code approach to allow students to focus on understanding data science concepts and processes.
Supporting independent project work
Importantly, the work learners produce during this course won’t be the EPQ investigation itself. Working through ‘Data Science and AI’ will give learners the knowledge and skills they need to undertake an AI-focused project with greater independence and confidence.
After they have completed the course, when they begin their EPQ project pursuing a question that interests them, students can apply their new skills and knowledge to make decisions about their AI investigation and document their progress.
We are looking forward to supporting learners in England to turn their interest in one of today’s most significant technologies into the opportunity to develop independent research and project skills through a recognised qualification.
What’s next?
The ‘Data Science and AI’ course will be available to selected schools in England from September 2026, where students and teachers will test the materials and we will learn from their feedback to improve the course materials. We plan to make the course freely available in 2027.
More than one in four images on the web’s most popular home pages have alt text that’s missing, vague, or copied from adjacent images.
That’s from WebAIM’s 2026 WebAIM Million report, which found that alt text,an HTML attribute containing text describing the content of an image, was missing on 16.2% of images across the top million home pages. Among the images that did have alt text, another 10.8% provided an undescriptive attribute, such as alt="image", a raw filename, or a description duplicated from a neighbor.
While automated tooling reliably flags missing alt text, it isn’t as good at fixing poorly written alt text. Most alt text checkers test whether an accessible name for an image exists, not whether the provided alt text says anything useful about the associated image, and that’s a deliberate design choice: a quality-oriented rule with false positives is a rule teams switch off. So alt="IMG_2847.png" passes. So does the same alt="3/5 stars" on five different star-shaped icons.
We built an alt text plugin for the GitHub Accessibility Scanner to help improve your alt text. This post covers where we drew the line between what a checker can prove and what it can only suspect, why our worst bug turned out to be a layout problem rather than a parsing one, and what changed once we let a model into the loop.
If you’re building automated checks of your own, for accessibility or otherwise, the tradeoffs should transfer.
Proving a string is wrong without seeing the picture
Presence of alt text is an objective fact; the attribute is there or it isn’t. Quality is often a judgment call. A machine can’t prove whether a sentence adequately describes a picture in context from markup.
However, not all quality is subjective. There’s several checks you can perform based on the alt text alone, with no need to consult the image content:
The attribute is absent (not empty) or whitespace-only.
The alt is a filename, such as hero.png, IMG_2847.jpg.
The alt is a placeholder somebody meant to replace, such as TODO, tbd.
The alt is one generic word naming the medium instead of the content, such as image, logo, chart.
The same alt repeats across adjacent images.
Every one of those is a claim about a string, and that became our dividing line. Five deterministic rules run by default which need no credentials for running AI models or network calls. One opt-in rule calls a model with provided image content and surrounding context, for judgments an alt text string can’t support on its own.
First, we had to determine which images to judge on a scanned webpage. We use Playwright’s role-based locator rather than querySelectorAll('img'), so anything not included in the browser’s accessibility tree drops out, including anything carrying alt="". That last exclusion matters most. An empty alt is the author explicitly saying the image is decorative, and flagging it would punish exactly the behavior you want to encourage.
So, how strict should it be? A quality checker lives or dies on false positives, so we chose closed sets over clever heuristics. The vague-alt rule normalizes a string, then checks it against a curated list of words that carry no information on their own. It fires only on an exact match:
alt="image" gets flagged.
alt="image of the login screen with the SSO button highlighted" doesn’t.
Rules this literal miss plenty of bad alt text. We took the miss over the false positive, because a reliable checker that developers enable beats one that gets switched off.
Repetition is a layout problem, not a DOM problem
Repeated alt text presented an interesting problem. Picture a row of five star-shaped icons that each say "3/5 stars". A screen reader user hears the same thing five times and learns nothing new from four of them.
Our first version walked the images in document order and flagged any run sharing the same normalized alt. It caught things it shouldn’t have. For example, a footer “GitHub” logo and a header “GitHub” logo might sit next to each other in the extracted list but nowhere near each other on screen, so nobody experiences them as a group.
What matters is where images land on screen, not where they sit in the markup. So the rule now checks page layout, and only extends a run when the gap between two bounding boxes is small compared to the boxes themselves:
const gap = Math.max(horizontalGap, verticalGap)
const largerDim = Math.max(a.boundingBox.width, a.boundingBox.height,
b.boundingBox.width, b.boundingBox.height)
return gap > GAP_MULTIPLIER * largerDim
Two details worth noting:
The multiplier is a judgment call, not a number we derived from anything. It’s the kind of value you tune against real pages instead of trusting from a spec.
When either image has no measurable box, the check fails open and the run continues. A missing finding is invisible; a wrong one isn’t.
Getting a model to act like a reviewer, not a critic
Deterministic rules only need the alt string. Anything smarter needs to know what the page is about, and none of that is tracked by the image element. Whether alt="a smiling person" is fine depends entirely on what surrounds it: on a generic mood shot, it’s probably works. But under a heading where a specific person is named, it doesn’t provide enough detail.
In our optional alt-text-qualitycheck, we extract page context alongside each image: the nearest heading, the page title, any <figcaption>, whether the image sits inside a link or button, and up to 600 characters of nearby prose.
The link signal matters most, because when an image is a link’s only content, its alt becomes the link’s accessible name. The right alt then names the destination instead of describing the picture.
One caution: The plugin only records that an image sits inside a link. We don’t check whether it’s the link’s only content, which is the part that actually turns alt into a link name. So right now both cases look identical to the model.
That context, the alt, and the image go to a vision model through GitHub Models. Our failure modes were rarely the model misreading a picture. They were the model having opinions. Given perfectly good alt text, our first version of the checker would suggest different alt text, because “could this be better?” is a question a language model always answers yes to. Every image becomes a finding, so the signal disappears.
Three changes fixed it:
A decision procedure instead of an instruction. The prompt walks four ordered steps, stops at the first that matches, and emits that step’s verdict: decorative, redundant with a caption, functional, or informative.
Explicit anti-nitpick rules. Trust the author’s framing. Separate redundant prefixes (“Image of…”) from semantic ones (“Photograph of…”). Treat a short alt as correct when the surrounding prose already analyzes the image.
Structured output with a forced field order, so reasoning is generated before verdict and the model has to build an argument before it picks a label.
None of that makes the model unfailingly correct. It makes it consistent enough to iterate against. The repository carries an offline grading harness built from published teaching material: WebAIM, the W3C images tutorial, and POET. The rule and the harness share one prompt, so what you tune offline is what runs in CI. That harness only tests the model’s judgment, though, not the whole pipeline. A case can score perfectly there and never reach the model in a real scan.
Sending images to a model is a privacy and cost decision
The moment a check calls an external model with webpage data, it stops being just a lint rule and requires careful data flow design. A few things follow from that:
The rule is off by default. It won’t run unless you deliberately enable it in your plugin configuration, and it needs a token with access to GitHub Models.
URLs get redacted. Image URLs and link hrefs often carry signed CDN tokens or session identifiers, so query and fragment are stripped from anything entering the model context or the rule’s error logs. For the same reason, src and srcset are replaced with (omitted) in the markup we send.
Everything in that context window is untrusted input. Titles, headings, and prose all come from the page being scanned, and a page can contain text written to steer a model. Structured output constrains the shape of a response, not the reasoning behind it.
One caution, because that list is easy to over-read: findings still carry the real page URL and original HTML into the scanner’s normal reporting pipeline. That’s on purpose, since you can’t fix an image you can’t locate. Redaction narrows what reaches the model and the logs, not what lands in your own issues. And if you set up Azure AI Vision credentials, an optional OCR pre-pass sends image bytes to a second place. Nothing requires Azure, but a data-flow review needs to cover both paths.
Cost follows the same shape. In the common case this is one model call per image per scan, which on an image-heavy site dominates the cost of the whole run. That’s reason enough to put it on a schedule rather than on every commit.
What this still can’t do
The deterministic rules are literal. They catch alt text that’s obviously unwritten, not alt text that’s fluent and wrong. They also read the alt attribute rather than the computed accessible name, so an aria-label that fixes the problem won’t stop the finding.
The model-backed rule produces false positives. Every finding is a prompt for human attention, not a verdict.
Silence isn’t coverage. That rule re-fetches images outside the browser session, so anything behind authentication can fail to load. Fetch and model errors are logged and skipped, which means a page can come back clean because nothing got checked.
Suggested alt text is a draft. A model that sees the image and a few nearby words can’t account for your audience, your house style, or the job that image is doing on the whole page.
Some findings double up with the scanner’s built-in checks, since our missing-alt rule covers the same ground.
We only check HTML<img>tags. SVG, role="img" containers, CSS backgrounds, and canvas aren’t covered yet.
This is new code with limited real-world feedback. Rules like these improve when they meet the variety of markup and content found across real sites. This plugin hasn’t had that yet, so treat early findings accordingly.
Passing isn’t conformance. Automated checks are a floor. Testing with people who use assistive tech is the goal.
What we’d tell you if you’re building something similar
Separate what you can prove from what you can only suspect, and give them different defaults. Checks that prove something should be cheap, predictable, and on by default. Checks that only suspect something should be opt-in, and should read as a suggestion rather than a verdict. Then, ask what the user experiences rather than what the DOM says. Every gap still open in this plugin has that second shape. We record that an image is inside a link, not that it is the link. We read an attribute, not a computed name.
That distance is the real boundary, and a better model doesn’t close it. Deciding what the functionality of an image is for a user who can’t see it still requires human judgment. What automation buys you is making sure that human is giving the right images a second examination.
Organizations running large data processing pipelines want lower costs, faster job runtimes, and dependable support for open table formats, without adding operational overhead. AWS Glue, a serverless, scalable data integration service that you can use to discover, prepare, move, and integrate data from multiple sources, has now launched AWS Glue 6.0, the new version of AWS Glue that addresses these needs. This version upgrade lowers AWS Glue pricing by 30% and improves performance with AWS optimized Apache Spark 4.1. It also augments developer experience with new features and adds support for Apache Iceberg V3 specifications that are suitable for enterprise adoption. The newly available AWS Glue 6.0 makes data processing workloads more manageable, faster to run, and easier to operate.
In this post, we cover the key capabilities of AWS Glue 6.0 and their performance benefits. We share code examples to help you take full advantage of the release, and we show you how to get started.
AWS Glue 6.0 highlights
AWS Glue 6.0 brings together four major improvements designed to transform how you build and run data integration workloads.
First, it upgrades the underlying runtime to Apache Spark 4.1.1, Python 3.13, Scala 2.13, and AWS SDK for Java 2.x, delivering performance improvements that can help with faster job completion times and lower costs.
Second, this release reduces current AWS Glue usage rate by 30%, and when combined with the performance improvements, you may realize an even lower effective cost.
Third, AWS Glue 6.0 introduces support for more capabilities of Apache Iceberg V3. This includes the VARIANT data type with automatic shredding, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types. With these capabilities, you can build modern lakehouse architectures on the latest open table format standards.
Finally, new features like Spark Declarative Pipelines, Real-Time Mode for streaming and Python virtual environments with S3 caching are designed to further improve performance and developer experience. The following sections dive deeper into each of these areas.
Runtime upgrades
AWS Glue 6.0 upgrades the core runtime stack across the board, bringing newer versions of Apache Spark, Python, Scala, and the AWS SDK to your serverless data integration workloads.
Apache Spark 4.1.1 – AWS Glue 6.0 runs an AWS optimized build of Apache Spark 4.1.1, a major generational leap from Spark 3.5 on AWS Glue 5.1. This release introduces improvements focused on intent-driven data engineering, real-time streaming with sub-second latencies down to single-digit milliseconds for stateless tasks, faster PySpark performance, and expanded SQL features.
Python 3.13 – Supports Python 3.13, a stable release that brings interpreter changes, Python data model enhancements, standard library updates, and security updates.
Scala 2.13 – Upgrades to Scala 2.13 which includes a collections library overhaul, language and syntax feature changes, standard library additions, and compiler performance updates.
Reduced Pricing
AWS Glue 6.0 cuts current AWS Glue pricing by 30%. This means every job you run on AWS Glue 6.0 costs 30% less per DPU-hour compared to AWS Glue 5.1, with no changes required to your workload configuration. When you combine this pricing reduction with the performance improvements delivered by runtime upgrades, your effective cost savings can compound because jobs can complete faster and consume fewer DPU-hours on a lower price point. If you run large-scale Extract, Transform, and Load (ETL) pipelines or recurring batch workloads, this compounding effect can help reduce your monthly spend.
To quantify the comparison, we ran the industry-standard TPC-DS benchmark at 3 TB scale on Parquet data stored in Amazon Simple Storage Service (Amazon S3), using 30 G.2X workers on AWS Glue. The following table compares the results we obtained in our tests for AWS Glue 6.0 and AWS Glue 5.1. Thus, based on TPC-DS benchmark at 3 TB scale, AWS Glue 6.0 delivers up to 36% better price performance than AWS Glue 5.1.
.
AWS Glue 6.0
AWS Glue 5.1
Estimated Cost ($)
USD 5.61
USD 8.87
Table 1: 3TB TPC-DS benchmark comparison between AWS Glue 6.0 and AWS Glue 5.1
Updated Open Table Format (OTF) support
AWS Glue 6.0 ships with updated versions of all three major open table formats – Iceberg 1.11.0, Hudi 1.1.1, and Delta Lake 4.2.0 – providing better performance, improved merge-on-read capabilities, streamlined concurrency control, and expanded SQL compatibility.
Besides supporting the latest open table format versions, AWS Glue 6.0 delivers Apache Iceberg V3 specification that is suitable for enterprise use. The highlight is Variant shredding, which AWS Glue uses to automatically decompose semi-structured data into physically optimized, columnar sub-fields, which should result in faster query read performance. Combined with deletion vectors for efficient row-level updates, UNKNOWN column types, default column values, and richer data type support, AWS Glue 6.0 is designed to make your open data lakes faster, more flexible, and more cost-efficient. AWS Glue 6.0 also adds support for geospatial data types (Geometry and Geography) and nanosecond-precision timestamps from the Apache Iceberg V3 specification, neither of which are currently supported in open-source Apache Spark 4.1. Additional features like row lineage tracking round out the Apache Iceberg V3 capabilities available on AWS Glue 6.0.
In the following sections, we illustrate select capabilities from Apache Iceberg V3 specification on AWS Glue 6.0.
VARIANT column type
Apache Iceberg V3 introduces the Variant type to store semi-structured data (think JSON, XML, logs, and deeply nested event data) in a compact binary format. Variant shredding is designed to automatically decompose VARIANT columns into physically optimized, columnar sub-fields, facilitating predicate pushdowns and reducing scan overhead. It aims to provide simpler management of semi-structured data, without the need for complex flattening logic. With Variant type, you get the flexibility of embedding a JSON data type in your table columns while shredding is designed to help accelerate read queries and reduce costs.
UNKNOWN column type
The UNKNOWN type in Apache Iceberg V3 acts as a flexible placeholder for columns where the data type is not yet determined at the time of table creation or data ingestion. Tables can accept all-null data initially, and the column type can be upgraded later without breaking ingestion pipelines or consuming applications. This can simplify schema evolution for rapidly changing data sources. Apache Iceberg V3’s UNKNOWN column type maps to Spark 4.1’s VOID type.
DEFAULT column values
Apache Iceberg V3’s DEFAULT column values allow specifying a default value for a column in the table metadata. When you add a new column, the query engine is designed to automatically apply this default to older rows, without rewriting data or running manual backfill operations.
The following code demonstrates creating an Apache Iceberg V3 table that uses VARIANT and UNKNOWN types, and DEFAULT values for a column.
Prerequisites
To get started with this code example, make sure you have the following prerequisites.
Start fresh, Create script and copy-paste the following code.
Replace the demo S3 bucket name with your bucket name in the code.
# Example pySpark script for testing few Iceberg v3's new data types
from pyspark.sql import SparkSession
CATALOG = "glue_catalog"
DATABASE = "sample_glue6_iceberg_db"
TABLE_NAME = "sample_glue6_table"
TABLE = f"{CATALOG}.{DATABASE}.{TABLE_NAME}"
TABLE_LOCATION = "s3://amzn-s3-demo-table-bucket/glue6blog-newdatatypes/"
# Configure Spark to use Apache Iceberg with the AWS Glue Data Catalog.
spark = (
SparkSession.builder
.appName("Glue6NewDataTypes")
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.config(f"spark.sql.catalog.{CATALOG}", "org.apache.iceberg.spark.SparkCatalog")
.config(f"spark.sql.catalog.{CATALOG}.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog")
.config(f"spark.sql.catalog.{CATALOG}.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
.config(f"spark.sql.catalog.{CATALOG}.warehouse", "s3://amzn-s3-demo-table-bucket/glue6blog-newdatatypes")
.config("spark.sql.defaultColumn.enabled", "true")
.getOrCreate()
)
spark.sql(f"CREATE DATABASE IF NOT EXISTS {CATALOG}.{DATABASE}")
# Create an Iceberg v3 table with VARIANT, unknown, and a default value.
# Spark's VOID type is stored as the Iceberg v3 unknown type.
spark.sql(
f"""
CREATE TABLE {TABLE} (
record_id BIGINT,
payload VARIANT,
reserved_field VOID,
status STRING DEFAULT 'active'
)
USING ICEBERG
LOCATION '{TABLE_LOCATION}'
TBLPROPERTIES ('format-version' = '3')
"""
)
# Insert two rows. The omitted columns use null and the declared default.
spark.sql(
f"""
INSERT INTO {TABLE} (record_id, payload)
VALUES
(1, parse_json('{{"event_type":"created","score":98.5}}')),
(2, parse_json('{{"event_type":"processed","score":87.2}}'))
"""
)
# Query the row and extract values from the VARIANT column.
spark.sql(
f"""
SELECT
record_id,
variant_get(payload, '$.event_type', 'string') AS event_type,
variant_get(payload, '$.score', 'double') AS score,
reserved_field,
status
FROM {TABLE}
"""
).show(truncate=False)
spark.stop()
Provide the following details in the Job details tab.
A Name for the job.
The IAM role you have from Prerequisites (2) for the IAM role of the job.
Choose Glue 6.0 for the Glue version.
Leave the rest as defaults.The following screenshot shows the Job details tab with illustrated values in the AWS Glue console.
Figure 1: Job details tab with the Glue version set to Glue 6.0
Scroll down. Under Advanced properties, for Job parameters, add the following additional Job parameter key-value pair:--datalake-formats=icebergThe following screenshot shows the Job parameters with the illustrated key-value pair in the AWS Glue console.
Figure 2: Job parameters with the datalake-formats key set to iceberg
Save the job and choose Run.
After the job is completed successfully, from the Runs tab – Run details, you can inspect the Output logs that take you to the logs in the Amazon CloudWatch console. The following shows the sample output for the SELECT query in the script.
Notice that we inserted two rows with values only in the record_id and the variant column. Variant column inserts were done using parse_json(). The reserved_field is of VOID type, hence returns NULL values. The status column is declared with a default active value and returns active, since the column was omitted during the insert operation.
Deletion Vectors Apache Iceberg V3 replaces the traditional positional delete files used in Apache Iceberg V2 to deletion vectors. This change can help improve Merge-on-Read (MoR) performance. This shift replaces heavy, multi-file Parquet reads with highly compressed, direct binary bitmaps that can provide lower storage overhead and faster reads on delete-heavy tables. In scenarios with heavy table updates, such as streaming change data capture (CDC) from operational databases, Apache Iceberg V3 can offer read performance advantage over Apache Iceberg V2.
To validate the performance of deletion vectors, we created two identical AWS Glue streaming jobs and ingested the events into two different Iceberg tables, one in Apache Iceberg V2 and another in Apache Iceberg V3 format. The streaming CDC events were approximately 150,000 events per second, merge-on-read, update-heavy. Every micro-batch writes row-level deletes. We froze both tables at the same delete-heavy state and disabled compaction, leaving the tables with roughly 1.7 million rows in valid state out of the 26.4 million physical rows. The following table summarizes the read performance latency of the two Iceberg tables. We observed in this testing that reading from the delete-heavy Apache Iceberg V3 is at least 1.5 times faster than the reading from a similar Apache Iceberg V2 table. For larger enterprise scale Apache Iceberg V3 tables, the read performance could improve further.
Table 2 – Read latency comparison between Apache Iceberg V2 delete files and Apache Iceberg V3 deletion vectors
New ETL features
AWS Glue 6.0 introduces several additional capabilities designed to simplify how you build and manage data pipelines, some of which are discussed in the following list.
Spark Declarative Pipelines (SDP) where you define the outcomes you want for your entire data pipelines in a declarative fashion with SQL statements or Python decorators while AWS Glue handles execution flow, dependency resolution, parallelism, checkpointing, CDC, and recovery automatically. This helps you focus on business logic rather than orchestration plumbing.
Real-Time Mode (RTM) for streaming delivers continuous execution for Structured Streaming with sub-second latencies, down to single-digit milliseconds for stateless tasks. This can help support real-time use cases like fraud detection, live dashboards, and event-driven architectures without managing dedicated streaming infrastructure.
Arrow-Native UDFs/UDTFs execute Python functions directly on PyArrow batches without Pandas conversion overhead, which can result in faster performance for custom transformation logic at scale.
Recursive Common Table Expressions (CTE) adds WITH RECURSIVE queries natively, allowing graph traversals and hierarchical queries without workarounds or external libraries.
Python data source filter pushdown evaluates filters at the data source designed to minimize data movement, reduce the volume of data scanned, and improve job performance.
With Python virtual environments and S3 caching, you can provide pre-built Python dependencies, which should result in reduced startup latency for AWS Glue jobs by eliminating runtime dependency resolution. For existing jobs that use --additional-python-modules, no action is required. AWS Glue automatically handles the conversion to virtual environments when your job runs on AWS Glue 6.0.
Dependent library upgrades
The following table summarizes the key runtime and library version upgrades on AWS Glue 6.0.
Feature
Glue 6.0
Glue 5.1
Spark
4.1.1-amzn-0
3.5.6-amzn-1
Python
3.13.14
3.11.15
Scala
2.13.17
2.12.18
AWS SDK for Java
2.44.6 (Version 1.x removed)
2.35.5
Boto3
1.42.84
1.40.61
Java
17.0.20
17.0.19
Iceberg
1.11.0
1.10.0
Hudi
1.1.1
.0.2
Delta Lake
4.2.0
3.3.2
AWS Glue Data Catalog client
4.11.0
4.9.0
EMR DynamoDB connector
6.1.0
5.7.0
Arrow
18.3.0
2.0.1
Hive
2.3.10-amzn-1
2.3.9-amzn-4
Table 3: Runtime and library version comparison across AWS Glue 6.0 and AWS Glue 5.1
Getting started
To get started with AWS Glue 6.0, you can use one of the following methods.
Migrate your existing AWS Glue jobs to AWS Glue 6.0 using the Spark Upgrade Agent.
Clean up
To avoid incurring costs, clean up the resources you created for this post.
Delete the Data Catalog database and the Iceberg table.
Delete the data and metadata folders of the Iceberg table from your S3 bucket.
Delete the AWS Glue job and the AWS Glue job execution IAM role.
Conclusion
AWS Glue 6.0 is designed to be faster, more cost-effective, and easier to use for building your open data lakehouse architectures and orchestrating your data pipelines. In this post, we discussed the key highlights of AWS Glue 6.0 and illustrated usage of Apache Iceberg V3 features with code samples. You can create new AWS Glue jobs on AWS Glue 6.0 or migrate your existing AWS Glue jobs to benefit from these improvements.
With Apache Spark 4.1.1, Apache Iceberg V3, Python 3.13, upgraded open table format libraries, and new streaming capabilities, AWS Glue 6.0 aims to help you build new data applications or to operate your existing data pipelines more efficiently and with less maintenance overhead.
We encourage you to test AWS Glue 6.0 in your development environment today. Check out this blog that talks about upgrading your AWS Glue jobs to AWS Glue 6.0. Also, in the coming days and weeks, look out for blogs on individual topics illustrating various features of Spark 4.1.1 and Apache Iceberg V3 on AWS Glue 6.0.
Acknowledgements: We thank the numerous engineers and leaders who helped build AWS Glue 6.0 to support customers with a highly performant Spark runtime and other value-added capabilities.
Upgrading PySpark jobs to a new Apache Spark major version can introduce breaking changes. Removed configuration keys, stricter type casting, and Python library incompatibilities can cause runtime failures or silent behavior differences. With AWS Glue 6.0 now running Apache Spark 4.1 and Python 3.13, you need a reliable way to migrate your existing jobs while validating correctness.
In this post, we walk through upgrading a PySpark ETL job from AWS Glue 5.1 to AWS Glue 6.0. We use the generative AI upgrades for Apache Spark in the AWS Glue console. The upgrade analysis automatically identifies incompatibilities, iteratively resolves them, validates the result with data quality checks, and presents recommended changes for your review. AWS Glue 6.0 also delivers up to 36% better price performance* along with Iceberg v3, Spark Declarative Pipelines, Real-Time Mode, and Arrow-native Python UDFs.
What changes with AWS Glue 6.0
AWS Glue 6.0 runs Apache Spark 4.1, which introduces several behavioral changes from the Spark 3.5 runtime used in AWS Glue 5.1:
Behavior
Spark 3.5 (AWS Glue 5.1)
Spark 4.1 (AWS Glue 6.0)
ANSI SQL mode
Disabled by default
Enabled by default
Legacy Parquet datetime configs
Supported
Removed (renamed)
Python runtime
3.11
3.13
Beyond version compatibility, AWS Glue 6.0 also introduces:
Apache Iceberg v3 with VARIANT Shredding for efficient semi-structured data handling.
Arrow-native Python UDFs (PyArrow) for improved performance.
Built-in observability with structured metrics and enhanced Spark UI.
Up to 36% better price performance compared to AWS Glue 5.1*.
These runtime changes mean your existing AWS Glue jobs might encounter removed configuration keys, stricter type casting behavior, or Python package version incompatibilities when running on AWS Glue 6.0. Fixing these manually is time-consuming and error-prone. The following sections show how the generative upgrade analysis handles this automatically.
The sample job
Our example is a daily ecommerce order analytics pipeline running on AWS Glue 5.1:
What the job does:
Ingests 10,000 orders from Parquet files with INT96 timestamps (including pre-1900 historical dates from a legacy system migration).
Computes revenue metrics by casting string prices to numeric values and calculating line totals with discounts and tax.
Segments customers using recency, frequency, and monetary (RFM) scoring through mapInPandas with pandas and scikit-learn.
This job runs successfully on AWS Glue 5.1. The following sections walk through how the upgrade analysis identifies and resolves incompatibilities when upgrading this job to AWS Glue 6.0. Before starting, confirm you have the prerequisites in place.
Prerequisites
An AWS account with access to the AWS Glue console.
An existing AWS Glue job on version 5.1 or earlier with at least one successful run.
An Amazon S3 path for storing the upgrade analysis results.
Running the upgrade analysis from the console
The following steps walk through the upgrade analysis workflow using the AWS Glue console.
Step 1: Select your job
Navigate to your job in the AWS Glue Studio console. Confirm the job has a successful run history on AWS Glue 5.1 before starting the upgrade analysis.
Figure 1: Job run status for the job on AWS Glue 5.1
Step 2: Start the upgrade analysis
From the job’s Actions menu, select Upgrade with generative AI. Configure the following:
Target AWS Glue version: 6.0.
Results S3 path: An S3 location where the analysis stores its artifacts and recommendations.
Figure 2: The Upgrade with generative AI option in the Actions menu
Configure the target AWS Glue version and the S3 results path, then choose Run.
Figure 3: The Upgrade with generative AI window for setting the target AWS Glue version and results path
Choose Run. The analysis begins by running your job on AWS Glue 5.1 to establish a baseline. It then iteratively tests the job on AWS Glue 6.0, identifies failures, applies recommended fixes, and validates the job. If the upgrade analysis cannot resolve an incompatibility within its attempt budget, the analysis stops and reports the unresolved issue for manual review. Your original job remains unchanged.
Note: The upgrade analysis executes your job multiple times (one baseline run plus one or more validation attempts), and each run consumes Data Processing Units (DPUs). For large or long-running jobs, consider using the run configuration option to specify fewer workers or a smaller dataset to optimize analysis cost.
Step 3: Monitor progress
The console displays the analysis progressing through multiple validation attempts. Each attempt either succeeds or fails with a specific error, and the upgrade analysis uses that error signal to determine and apply the appropriate fix for the next attempt.
Figure 4: Upgrade analysis progress across multiple validation attempts
What the upgrade analysis found and fixed
The analysis completed in four validation attempts, identifying and resolving three distinct incompatibilities. The upgrade uses deterministic migration rules for known config changes, and automated diagnosis for runtime or code errors.
Iteration 1: Removed Parquet legacy configuration
The analysis first sanitizes any Spark configurations that were removed in Spark 4.1. Our job used spark.sql.legacy.parquet.datetimeRebaseModeInWrite and spark.sql.legacy.parquet.int96RebaseModeInWrite, which no longer exist.
Migration rule applied: The SQL configs with the spark.sql.legacy prefix were removed in Spark 4.1. They have been renamed to their non-legacy equivalents, preserving the original values.
The read-side configs (datetimeRebaseModeInRead, int96RebaseModeInRead) already used the correct non-legacy names and required no changes.
However, with this fix applied, the validation run still failed because the Python module installation encountered an error on the AWS Glue 6.0 image.
Iteration 2: Python module version incompatibility
The pinned module versions (pandas==2.2.2, scikit-learn==1.5.0, numpy==1.26.4) could not be installed in the AWS Glue 6.0 Python 3.13 environment.
Error:
LAUNCH ERROR | Installation of Additional Python Modules failed
Recommended change: The upgrade analysis updated the version specifications from exact pins to minimum version constraints, allowing pip to resolve compatible versions for Python 3.13:
With modules installing successfully, the job launched on AWS Glue 6.0 but encountered a runtime error.
Iteration 3: ANSI mode strict type casting
Spark 4.1 enables ANSI SQL mode by default (spark.sql.ansi.enabled=true). Our revenue calculation casts string prices to double, but approximately 1.8% of records contain non-numeric placeholder values such as “N/A”, “pending”, or “null” from the upstream system. From a business perspective, this meant 1.8% of revenue orders were silently excluded from revenue metrics. This data quality issue was invisible to the original pipeline.
On AWS Glue 5.1 (ANSI mode off), cast("N/A" as double) silently returns null. On AWS Glue 6.0 (ANSI mode on), this throws an exception:
Error:
NumberFormatException: [CAST_INVALID_INPUT] The value 'null' of the type
"STRING" cannot be cast to "DOUBLE" because it is malformed. Correct the
value as per the syntax, or change its target type. Use try_cast to
tolerate malformed input and return NULL instead. SQLSTATE: 22018
Migration rule applied: As of Spark 4.1, spark.sql.ansi.enabled is on by default. Casting a malformed value now raises CAST_INVALID_INPUT instead of returning NULL. The upgrade analysis resolved this by updating the script to use try_cast(), which safely returns NULL for malformed input while preserving ANSI mode protections for the rest of the job.
Recommended change:
Before (AWS Glue 5.1):
F.col("unit_price").cast("double")
After (AWS Glue 6.0, fixed by the upgrade analysis):
F.expr("try_cast(unit_price as double)")
This is a targeted fix that handles the known dirty data without disabling ANSI mode globally, keeping overflow detection and type safety active throughout the job.
Final validation and data quality check
After applying all three fixes, the analysis ran the job on AWS Glue 6.0 one final time and performed a data quality comparison between the AWS Glue 5.1 baseline output and the AWS Glue 6.0 output.
Result: The job completed successfully and all data validations passed with no mismatches detected between the source and target outputs.
Figure 5: Final analysis status with links to the results output path in Amazon S3
Reviewing the upgrade summary
The analysis produces a detailed summary stored in your S3 results path. This summary documents each validation attempt, the errors encountered, the migration rules applied, and the recommended configuration changes:
The following is a snippet from the upgrade summary (summary.md) showing the recommended changes and validation attempt details:
The summary documents each validation attempt, the changes applied, and the data quality results.
Figure 6: Upgrade summary snippet showing validation attempt details, data quality, and analysis results
After reviewing the recommendations, accept the changes to upgrade your job to AWS Glue 6.0. This updates your job definition with the recommended configuration, including the renamed Spark configs, updated module versions, and any script modifications. Because the analysis has already validated the job on AWS Glue 6.0 and confirmed data quality parity with the original, your job is ready for production.
After reviewing the recommendations, you can apply the upgraded script to your job.
Figure 7: The option to apply the upgraded script to the job
Choose Apply to confirm the upgrade.
Figure 8: The Apply button that confirms upgrading the job to AWS Glue 6.0
After applying, the job definition reflects the new AWS Glue version.
Figure 9: The AWS Glue version for the job after applying the upgrade
Python virtual environments in AWS Glue 6.0
AWS Glue 6.0 introduces --python-virtual-env-storage-prefix, a service-managed virtual environment with S3 caching that simplifies Python dependency management.
For existing jobs that use --additional-python-modules, no action is required. AWS Glue automatically handles the conversion to virtual environments when your job runs on AWS Glue 6.0. Your jobs continue to work without any changes.
For new jobs on AWS Glue 6.0, we recommend using the virtual environment approach:
On the first run, AWS Glue installs your modules into a virtual environment, packages it, and caches the result to your specified S3 path (approximately 15–30 seconds of additional startup time).
On subsequent runs, AWS Glue downloads and extracts the cached virtual environment instead of running pip install.
The cache is automatically invalidated when your module list, versions, or AWS Glue version changes.
This approach provides faster cold starts after the first run, requires no Docker image management (unlike --python-virtual-env), and is entirely service-managed with no maintenance burden.
Conclusion
The generative upgrade analysis identified and resolved three distinct compatibility issues in our AWS Glue 5.1 job, so the job now runs successfully on AWS Glue 6.0 with Apache Spark 4.1:
The upgrade analysis renamed legacy Parquet datetime configuration keys (removed in Spark 4.1) to their current equivalents.
The upgrade analysis updated Python module version specifications that were incompatible with Python 3.13 to use flexible minimum version constraints.
The upgrade analysis addressed the new ANSI SQL mode default (which causes runtime failures on malformed data) with a targeted fix using try_cast() to safely handle non-numeric values while preserving ANSI mode protections.
The analysis validated that the upgraded job produces output consistent with the original, and presented all changes as recommendations for review before applying them to your job.
To reproduce this walkthrough, start with any existing AWS Glue job running on version 5.1 or earlier with a successful run history. No additional sample code or CloudFormation template is required.
After you have reviewed and accepted the upgrade changes, you can delete the analysis results stored in your S3 results path.
*Based on 3TB TPC-DS benchmark comparing AWS Glue 6.0 to AWS Glue 5.1.
About the authors
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
