Местни данъци в помощ на семействата, а не на инвеститорите

Post Syndicated from Боян Юруков original https://yurukov.net/blog/2026/mestni-danaci-2/

На 18-ти юни премиерът Радев споделил, че се обмисля прогресивен данък сгради. Това мимолетно изказване се разви в следващите дни с протестни становища на анализатори, строителни инвеститори и финансисти. Беше допълнено от хора в кабинета му с това, че не е ясно как точно ще стане, но се обсъжда актуализирането на данъчните оценки. Преглед както всъщност каза и какво е състоянието в момента може да прочетете от Цветелина Соколова в Медиапул и Таня Петрова в Сега.

Опипване със задни помисли

По същността си, това е политически ход. Никой няма съмнение, че е опит за ухажване на кметове и общински съветници като подготовка за изборите догодина. Години преди да подаде оставка водеше активна кампания за събиране на метни кандидати на постове. С това, както и кадруването в съдебната система се опитваше да следва моделът на идване на власт на Орбан. Не се получи, но сега отново има нужда от тях, за да оцелее това, което ще е не само ефективно втори вот на доверие към кабинета в края на 2027-ма, но и следващата голяма битка за страната с Борисов и Пеевски. Повече доходи за общините значи повече средства за преразпределяне от общинските съвети на фона на все по-трудни за крадене еврофондове.

Това изказване също е и опипване на почвата. Колко ще се приеме, какви идеи ще има, каква политическа цена ще трябва да плати. Аналогичното видяхме при идеите му за въвеждане на прогресивен подоходен данък. Там се опита да внуши, че ще се прави заради задължение от Европейския съюз прехвърляйки негативите върху лошия Брюксел. Доколкото има препоръчки за такава стъпка от международни наблюдатели като МВФ и Световната банка, същото има и за имотните данъци. Не му се получи особено и почти универсално беше отхвърлено от икономистите. Тук препоръчвам да прочетете коментара на Лъчезар Богданов по темата.

Моето мнение за подоходните данъци е, че наистина не е честна сегашната система и следва да са прогресивни. Против съм въвеждането им обаче заради негативните ефекти от самата промяна. Понякога се взираме твърде много в желаният резултат от промените в регулации и данъци – било то намаление или увеличение. Не се разбира и внимава обаче как точно се правят тези промени и дали пътят до този желан резултат не носи повече краткосрочни и дългосрочни щети. В случая – при липса на добър контрол над събираемостта, лесно пране на пари и укриване на доходи, анемични вътрешни инвестиции и лесна мобилност на капитали ще видим намаление на данъчните приходи при отлив на работни места с висока добавена стойност или регистрирането им под друга форма.

При имотните данъци обаче съм на мнение, че е отдавна закъсняла мярка и може да се направи повече, за да не се повишат наемите и цените на имотите. Дори обратното – има шанс да се намалят.

Данъчната оценка като политическа дъвка

Последната реална актуализация в данъчната оценка на имотите в България е през 2007 г. – преди почти 20 години. За този период пазарните цени в цяла България са скочили няколкократно. Когато се върнах в България през 2020-та, купих апартамента си на цена три пъти и половина по-висока от данъчната оценка. Шест години по-късно разликата е вече пет пъти и половина. Актуализацията на тези оценки към пазарни цени е предлагана в парламента многократно, но отпада по политически причини. От една страна никой не иска да изглежда, че вдига данъците, но от друга – строителното лоби работи активно срещу такава мярка, защото ефективно им бърка в джоба.

В момента има хиляди сгради в страната, които стоят на груб строеж или са готови и в тях се живее, но нямат акт 16. Последното понякога е заради бюрокрация в общините, но по-често е заради нарушения при строежа, фалит на строителя или нарочно нежелание да се довърши процеса. В миналото това беше начин да се избягват данъци, но тази дупка беше затворена и след две години на акт 14 вече се дължи местен данък. Това е особен проблем за строителите, които са взели парите на купувачи и са зарязали строежите си. Наскоро получих справка, която показва, че само в София има поне 2500 сгради – предимно жилищни блокове – които са на акт 14 повече от две години, но не са пуснати в експлоатация. Дали са изоставени или се живее в тях не може да знаем – чакам още данни, за да разбера.

Актуализирането на данъчната оценка обаче не е нещо спорно. Винаги се е знаело, че се блокира нарочно. Въпросът е единствено дали някой ще има смелостта и гласовете в парламента, както и дали ще се направи формула за автоматично актуализиране, за да е предвидимо за купувачите и бизнеса, а не зависимо от политически прищявки.

Данък сгради като социален инструмент

Вторият компонент тук е данъчната ставка. В момента законът позволява общините да го слагат между 0.1 и 4.5 на 1000 като за основно жилище има намаление от 50%. София, например, слагат данък сгради на 1.875 на 1000. Не е ясна каква е дефиницията на „основно жилище“ и съществува проблем с декларирането на такова в различни общини. Това означава, че ако сте купили апартамент за 100 хиляди евро преди 10 години, за да живеете в него, то данъчната му оценка ще е не по-малко от 50 хиляди и данъка на година ще е около 45 евро на година. Ако сте купили още няколко такива апартамента като инвестиция, годишният данък ще е по-малко от вечеря в ресторант в София със семейството ви. Това важи както за семейства с деца, така и за инвеститори купуващи десетки апартаменти и цели комплекси.

Смятам, че това е не е редно и трябва да се промени. Предложението ми цели запазване на данъчната тежест при първо жилище и на жилища на частни лица отдадени законно под наем, но драстичен скок при едрите инвеститори, схемите с жилища и това, което аз наричам криптобетон. Има три компонента:

  • Драстично вдигане на границата на ставката не в промили, а в проценти – от 0.3% до 1.5% от данъчната оценка
  • Намалението за основно жилище скача на 90%.
  • Местният данък платен за жилище отдадено под наем се зачита като данъчно облекчение към данъците на физическите лица, но не повече от 75% от дължимите данъци за приходите от наем

С първата точка се вдига между два и три пъти ставката. С втората точка не само се запазва сегашната ефективна ставка за основно жилище, но и в някои случаи се намаля. Например, в момента София е избрала 1.875 на 1000 или 0.1875%. Дори да реши да вдигне ставката пет пъти на .9375%, при новото облекчение няма да има промяна в платения данък за това семейство.

Третата точка позволява да не се вдигне данъка на физическите лица, които отдават апартаменти под наем. При тях ставката ще остане висока, тъй като не е основно жилище, но ще могат да я приспаднат от данъците си. Разбира се, тук доста ще посочат, че мнозинството от хазаите не плащат данъци и няма от какво да приспадат. Тази мярка ще помогне да се изсветли сектора, което е също в полза на наемателите. Не следва да се прави данъчна политика, която пази или по някакъв начин се опитва да не засегне сивата икономика и укриването на доходи.

Ще забележите, че говоря само за собственици физически лица. Тази мярка цели неколкопластов ефект. Първо, много често се пишат имоти на фирми независимо, че се използват изцяло за лични цели, с цел избягване на данъци и изписване на фактури за тях. Второ, много жилища са необитавани – между 25 и 40 по различни оценки – с цел паркиране на капитали. Трето, вече настъпват едри капитали, които строят или изкупуват жилища с цел отдаване под наем. В доста страни в Европа бизнес моделът на последните е довел до пагубни последствия за имотния пазар.

Всичко това води до концентрация на жилища в малък кръг от лица и влошава значително възможността на млади семейства да се сдобият с жилище. Последното води до отлив на свободен финансов ресурс за друга дейност като образование и пътуване. По-важното обаче е че доказано липсата на достъп до по-голяма жилищна площ и добри условия влияе негативно на демографската обстановка. Семействата се решават по-трудно на деца като не могат да си позволят наем или жилище.

Тези мерки правят неизгодно да се държат жилища на мазилка или да се концентрират такива за наем в големи конгломерати диктуващи пазара. В същото време физически лица наемодатели, каквито са огромното мнозинство в България, няма да видят особена промяна в данъчната ставка. Тази мярка ще създаде стимул многото празни апартаменти да бъдат пуснати на пазара, което би качило предлагането и свалило цената на наемите. Основна пречка пред това е недостигът на майстори, които да преведат тези жилища във вид за живеене. Това и сега е проблем покрай бума на пуснатите в експлоатация жилища. Затова някои жилища ще бъдат пуснати в продажба, което ако не да свали, то ще забави и дори спре ръста на цените на имотите.

Отдавна чуваме от архитекти, строители и инвеститори, че цените на имотите били високи, защото не се строяло достатъчно. Настояват, че в градовете ни имало предостатъчно междублокови пространства и градинки, където нямало проблем да се уплътнява повече. В същото време броят на жилищата пуснати в експлоатация надвишава не само населението, но и броят на домакинствата. В този контекст и предвид броят на започналите строеж жилища виждаме не само цените, но и наемите да скачат с бързи темпове. Проблемът тук не е, че няма достатъчно жилища на пазара, а че последният не се ръководни от пазарни принципи. Пратено на пари, липсата на защита на инвестициите и като цяло лошата бизнес среда в България, както и прилива на спекулативни капитали изкривяват пазара до степен, в която създава трудност за семейства да намерят адекватно жилище.

Това създава социален проблем и следва да се разглежда данъчното облагане като инструмент за влияние върху тази социална функция. Виждаме, че свободния пазар, свободията в строителството, неефективността на регулаторите и липсата съдебен контрол над нарушителите довежда до огромни проблеми в градската среда. Предложената горе промяна не цели и не може да адресира това. Инвеститори и спекуланти обаче в немалка степен се радват на същите привилегии замислени като социални, както младо семейство очакващо първото си дете. Това не следва да е така и ако декларираната сега воля не е просто политическа пиратка, имат шанс да се случи.

Разбира се, говорейки за спекула и пране на пари следва да се осъждат с лека ръка всички, които са намерили за добре да инвестират спестяванията си във втори или трети апартамент. Точно обратното – присъдата тук е за държавата, която не е осигурила предвидимост, сигурност на инвестициите, намалена бюрокрация и корупционен риск. В такава среда и липса на прозрачност за рисковете в имотния пазар, подобен избор си е логичен. Не следва обаче подобни инвестиции, дори когато става дума за физически лица, да се приравняват към жилище за обитаване. В единия случай говорим за рискова капиталова инвестиция, а в другия – критичен за съществуването актив. Инвестициите следва да се третират като такива данъчно и ефектът им върху градската среда е достатъчно значителен, за да обоснове предложените мерки.

Възможни ефекти и мерки

Всяка такава промяна – дори само актуализация на данъчните ефекти – могат да имат неочаквани ефекти. Всяка намеса в пазара на строителството, наемите и имотите може да има нежелани ефекти. Дори с по-голяма сила важи същото, когато не се прави нищо. Виждаме го в момента. Нямам съмнение, че дори така въведени мерки ще бъдат намерени начини за заобикаляне на закона, укриване на данъци и други схеми. Такива има и в момента.

В изказванията на хора от кабинета стана ясно, че виждат техническа трудност в липсата на свързаност в местните данъци на общините. Тяхната идея е за прогресивен данък, което значи, че трябва да има споделени данни за броя жилища и кой какво е декларирал някъде. Затова моето предложение не прави такова разграничение и не изисква никаква синхронизация между общините. Последното би било изключително трудно предвид опита от подобни инициативи в миналото. Дори техническата част да се случи, ще има нужда от значителен административен ресурс да се следи къде какво се декларира и проверки след това. Отделно недостатък на мярката е, че поставя юридическите лица на същото ниво като семействата с деца, което е нередно.

Затова за основно жилище следва да се води там където човек има настоящ адрес. Знам, че системата за адресна регистрация не работи добре и причината е контрола. Понякога има и злоупотреби – например се сменя настоящ адрес, за да се гласува другаде или за да се трупат точки за различно училище и детска градина. Това ще стане неизгодно, ако като резултат се увеличи данъчната ставка за жилището. Ако то е съсобственост на няколко души, то всеки ще получи намалението от 90% за жилището, където живее основно. Ако живее под наем на едно място, но притежава жилище на друго, то не може да се каже, че второто е основно жилище така или иначе. Ако имота е приписан на юридическо лице като формалност, то отново говорим за взаимоотношения аналогични на наемодател независимо дали собственик на фирмата живее в него.

Има аргумент, че следва да се направи облекчение за второ жилище. Причината за това е, че има много хора с наследствени такива или вили. Това би било по-сложно административно и изисква отново централизация на информацията, тъй като трябва да се декларира кое жилище се води като почивно. Възможно е да се осъществи и общините биха имали стимул да обменят информация дали другаде няма така подадени декларации, което би било укриване на данъци.

В миналото се обсъждаше друг вариант – данъчната ставка да зависи от това дали някой живее в жилището било то като собственик или като наемател. Това би спестило въртележката с данъчните облекчения, би осветлило пазара на наемите и би извадило много празни жилища на пазара. Тази промяна обаче беше отхвърлена от Конституционния съд с доста спорно решение, че не може да се обвързва данък върху имущество с това как се използва. Всъщност, именно Радев обжалва тогава промените в местните данъци, част от които вероятно ще предложи сам сега. Описаните горе мерки дават възможност за данъчно облекчение, което не попада в обсега на това решение.

Очаквам съпротива срещу тази или подобни мерки, тъй като засягат пряко пазара на нови жилища. В момента излиза практически безплатно да се държат неизползвани вече построените или ако се нанесе в тях – това, което са освободили. Строителите разчитат на това, защото така на пазара излизат предимно това, което са започнали да строят. Разчитат и че остават скрити измеренията на фалити в сектора, изоставените строежи, невъведени в експлоатация и въобще рисковете в имотния пазар. Конкретни предложените данъчни промени няма да променят особено последното, но имат потенциал да раздвижат пазара на имоти създавайки стимул да се освободи паркирания в бетон капитал.

Финансово ще има повече постъпления за общините, от където започна предложението на Радев. Ще има и повече приходи за държавата покрай изсветляването на наемите, ако то се случи. Ако не – поради данъчното облекчение ще има намаление на приходите към хазната. Това обаче е в рамките на децентрализацията, за която Радев говори. Разбира се, ще значи увеличена данъчна тежест най-вече за хората и фирмите инвестирали в множество имоти. Няма да има засегне обаче мнозинството от хората, които притежават жилището си или тези отдаващи няколко такива. Общините могат да изберат ставка, с която дори на намалят данъчната тежест за мнозинството докато в същото време увеличат приходите си от това, което наричам криптобетон.

Технически и административно нищо от тези промени не изисква свързаност на системите на общините. Такава е добра идея по принцип, но би блокирала мерки като споменатите от Радев с години. Една промяна би била справка за настоящ адрес при изчисляването на данъците от общините. Това обаче вече се прави от тях постоянно и всъщност именно те отговарят за този процес. От страна на НАП ще трябва да се предвиди ново данъчно облекчение и промяна на декларацията каква част от приходите са от наеми в страната.

Добре за семействата, зле за инвеститорите

Убеден съм, че много хора ще са срещу това предложение и като цяло срещу всяка актуализация. Чуват се вече аргументи как един и същи доход се облага многократно, как не получаваме нищо срещу тези данъци, как се крадат и как трябва да бъде оправена събираемостта, а не да се вдигат ставките за всички останали. Има различни философии как да се облага икономическата активност и собствеността. Част е подоходна, част е на потреблението, а част е на акумулираните активи. Колкото и абстрактно и наивно да звучи думата, целта винаги е била да е честно.

Събираемостта на данъци, такси и глоби, сивата икономика, неработещата прокуратура са проблеми, за които говорим всеки ден. С местните данъци не може да се решат, но това не значи, че данъчните оценки не са крайно остарели, а спекула, тъмни капитали, а в последно време и хищнически бизнес практики не създават социален проблем допринасящ към демографската криза. Предложените мерки имат потенциал да раздвижат пазара достатъчно, за да променят поведението на играчите в нея, да направят оценката за риска по-консервативна и първото жилище по-достъпно.

Надявам се, че сред псевдо-либертарианските възгласи ще се чуят достатъчно разумни аргументи против предложението ми. Не се съмнявам, че има недостатъци както всяка друга промяна или настоящото положение. Да не се прави нищо често е по-зле, а привидната ненамеса е форма на заемане на страна. В настоящия случай – тази на едрия капитал и инвеститорите.

MiTAC Computex 2026 Booth Tour: Diamond Cooling, 52U Racks, and More

Post Syndicated from Ryan Smith original https://www.servethehome.com/mitac-computex-2026-booth-tour/

While wandering the Computex 2026 showfloor, we visited MiTAC’s booth. The prolific server vendor had numerous products on display, including racks of GPU systems, storage, and even diamond cooled servers

The post MiTAC Computex 2026 Booth Tour: Diamond Cooling, 52U Racks, and More appeared first on ServeTheHome.

Изкуственият интелект и пазара на труда в ИТ сектора

Post Syndicated from Bozho original https://blog.bozho.net/blog/4595

Изкуственият интелект навлиза в много сектори, но един от тези, в които повишаването на ефективността има най-голям потенциал и технологията е най-развит вид, е ИТ секторът. Изкуственият интелект пише код сам, и макар писането на код да е само част от работата, това може да има сериозен ефект върху пазара на труда. Има и едно усещане сред хора в сектора, че тъсренето се охлажда и има съкращения извън обичайното.
Затова поисках от НАП данни за динамиката на трудовия пазар в ИТ сектора (можете да ги свалите тук) – брой трудови договори по месеци, за всяка от многото длъжности в ИТ сферата.

Резултатът е, че поне на този етап не се вижда ясна негативна тенденция – общият брой на заетите във всички длъжности за последните 3 години расте – от 84 хиляди през юни 2023 до 90 хиляди през май 2026 г., като ръст има всеки месец спрямо предходния. С изключение на последния – май спрямо април 2026 г, с 200 заети по-малко. Ще следя дали това е начало на негативен тренд или временна флуктуация.

Когато влезем на ниво отделни длъжности, картината е същата – ръст навсякъде в последните три години, и спад в последните 1-2 месеца. Спадът в последните 2 месеца на основните 2 позиции: „програмист софтуерни приложения“ и „разработчик софтуер“ е около 100 заети (от общо над 22 хиляди). Разбира се, това не са всички длъжности за програмисти – има „програмист уеб сайтове“, „експерт по проектиране и програмиране“, „програмист-аналитик“ и др. – можете да ги видите в качения по-горе файл.

Разбира се, данните не включват някои други начини на наемане в ИТ сектора – напр. консултанти чрез B2B договор, в който случай експертът е управител, и съответно не влиза в статистиката. Не влизат и граждански договори, и някои freelance отношения през платформи като upwork. Данните не включват и нетехнически персонал на ИТ компании (т.е. самият сектор е по-голям от 90 хиляди). Но дори без тези данни, трудовите договори са достатъчни, за да се отбележи тенденция. Разбира се, гражданските, freelance и консултантските договори в повечето компании се прекратяват първо (тъй като е по-лесно), а чак след това – трудовите. Така че кривата вече може и да е тръгнала надолу, просто през трудовите договори това се отчита със закъснение.

Изкуственият интелект е и възможност, и риск. Но секторът е гъвкав и силен и смятам, че ролята на държавата на този етап трябва да е единствено в облекчаването на регулаторната тежест, за да могат компаниите да са по-конкурентни на тези от други държави (дългосрочните политки по подготовка на кадрите са отделна важна тема)

Материалът Изкуственият интелект и пазара на труда в ИТ сектора е публикуван за пръв път на БЛОГодаря.

Minisforum’s Upcoming MS-03 SFF PC Brings More of a Good Thing

Post Syndicated from Ryan Smith original https://www.servethehome.com/minisforums-upcoming-ms-03-sff-pc-brings-more-of-a-good-thing/

At Computex 2026 Minisforum was showing off their upcoming MS-03 mini-PC, the successor to their popular MS-01 and aiming to bring more of a good thing with Intel’s new Panther Lake platform

The post Minisforum’s Upcoming MS-03 SFF PC Brings More of a Good Thing appeared first on ServeTheHome.

Upgrading Lambda function runtimes at scale with AWS Transform custom

Post Syndicated from Brian Krygsman original https://aws.amazon.com/blogs/compute/upgrading-lambda-function-runtimes-at-scale-with-aws-transform-custom/

When you create an AWS Lambda function, you choose the runtime that Lambda will use to run your code. This includes the base language version and supporting libraries. Lambda runtimes follow a published deprecation schedule. This means that you must periodically upgrade your function’s runtime.

Running on a deprecated runtime means potential security exposure, loss of AWS Support, and compliance challenges. For teams managing dozens of functions, this is a manageable maintenance task. For teams managing hundreds or thousands, it becomes a significant engineering effort that competes with feature work.

You can modernize your code and configurations with AWS Transform custom, an Agentic AI service purpose-built for code modernization. It fits into each stage of a runtime upgrade: surfacing risk, confirming test coverage, code transformation, and validation. The same workflow scales from a single function to an entire organization. You can use AWS-provided transformations or create your own, for compliance or compatibility. You can give it feedback to enforce your standards. You’re charged only for active agent work during server-side operations, not for user idle time or client-side processing.

This post addresses two audiences. If you work in an application team, you will learn how to use AWS Transform custom to upgrade your functions with confidence. If you’re part of a centralized platform team, you will see how to orchestrate Lambda upgrade campaigns at enterprise scale.

The upgrade challenge

Python and Node.js are two of the most widely used Lambda runtimes, and both have important recent or upcoming deprecation timelines.

Runtime Deprecation date
Node.js 20 April 30, 2026
Node.js 22 April 30, 2027
Python 3.9 December 15, 2025
Python 3.10 October 31, 2026

Sometimes a runtime upgrade requires changing your functions’ configuration in your infrastructure-as-code template or in the Lambda console. Other times, you also need to upgrade dependencies or even make code changes.

For example, in Node.js 24 AWS removed support for callback-based function handlers, in favor of the more modern async/await pattern which Lambda has supported since Node.js 8. Functions using the old pattern must be refactored. This is a behavioral change which affects every callback-based handler in the code base.

Before:

exports.handler = function(event, context, callback) {
    const result = processEvent(event);
    callback(null, result);
};

After:

exports.handler = async function(event) {
    const result = await processEvent(event);
    return result;
};

Applying this type of transformation across multiple Lambda functions used to require manual code changes. With AWS Transform custom, you can automate the upgrade to free your team’s capacity and focus for differentiated work.

AWS Transform custom for application teams

We assume you have AWS Transform custom already set up. For guidance, see the AWS Transform custom documentation. You can also use AWS Transform custom through the Kiro Power.

Prerequisites

Make sure you have the following configured locally:

Run a documentation transform

For your first transform, you can run the AWS-provided “AWS/comprehensive-codebase-analysis” transformation on a representative function or code base. This produces a prioritized view of the upgrade effort before a single line of code is changed, helping you plan your upgrade. Better-documented functions are easier to assess, maintain, and hand off. Running a documentation transform is a low-risk first step: it doesn’t change function behavior and lets you build familiarity with the AWS Transform custom workflow.

When you run the code analysis transformation, add additionalPlanContext to inform AWS Transform custom that you plan to upgrade your Lambda function runtimes. It can flag functions most likely to require code changes. For example, functions with callback-based handlers, complex async/callback code, or low test coverage.

atx custom def exec \
    --code-repository-path . \
    --transformation-name AWS/comprehensive-codebase-analysis \
    --configuration additionalPlanContext="Include analysis of Lambda function runtime upgrade to Node.js 24"

The following figure is a screenshot from running the preceding command on a sample code base.

Example AWS Transform output from documentation transform

Validation planning

Before an upgrade, you must verify correctness. This provides the confidence that you haven’t introduced new issues by upgrading. Test coverage from unit and integration tests helps with verification. A passing test suite can enforce the behavioral contract for the transformed code and help prevent problems.

Observability tools like metrics and alarms can help you validate your changes after they’ve been deployed. They can help you detect when breaks happen and are critical for finding the underlying cause.

If you’re not comfortable with your test or monitoring coverage, you can use AI agents to help. You can create a custom transformation definition in Transform custom to add or improve your tests or add alarms to your infrastructure as code (IaC) template. You can also use Kiro or other agents to generate tests from function specs, covering expected inputs, outputs, and error paths.

Transform

Now that you’ve used the documentation transformation to familiarize yourself with the tool and confirmed you have a way to validate your upgrade, you can use AWS Transform custom to upgrade your functions to a new runtime.

To apply the transform, use the AWS Transform custom CLI or Kiro Power. The example command below runs the “AWS/lambda-nodejs-runtime-upgrade” transformation against the code in the current directory. You can use additional switches to automatically trust all tools and run non-interactively.

atx custom def exec \
    --code-repository-path . \
    --transformation-name AWS/lambda-nodejs-runtime-upgrade \
    --configuration additionalPlanContext="Target Node.js 24"

Transform custom follows the instructions in the transform definition and additional plan context you specify. You can tell it to focus on a specific Lambda function in your code repository or upgrade all the functions it finds. Transform custom identifies callback-based handlers and refactors them to async/await. It handles edge cases including callbackWaitsForEmptyEventLoop and mixed async/callback patterns.

Dependency analysis flags packages with known incompatibilities with Node.js 24 and replaces them. Configuration updates change the Lambda runtime from nodejs22.x to nodejs24.x. AWS Transform custom self-debugs on build or test errors and commits changes to git incrementally on a separate transformation branch. You can also share feedback along the way, which is captured as Knowledge Items that can be applied to future transformations.

The following figures are screenshots from running the preceding command on a sample code base.

Screenshot of AWS Transform custom CLI output. It shows a sequence of tasks relating to Node.js upgrade. AWS Transform explains each task in natural language and states which tools are being used.

Screenshot of AWS Transform custom CLI output. It shows a transformation summary report at the end of the documentation transformation run. The report describes the status of each stage in the process (all ‘Yes’) and the summary summarizes the files to be upgraded.

Validate

AWS Transform custom validates defined exit criteria before marking the transformation complete.

Exit criteria can include:

  • All handlers run without errors on Node.js 24.
  • All tests pass, including generated callback behavior tests.
  • All dependencies confirmed compatible with Node.js 24.
  • Runtime configuration updated to nodejs24.x.
  • Additional requirements added with additionalPlanContext.

The newly transformed code remains in the transformation branch until you’re ready to merge and deploy. You can review logs of the transformation process captured by Transform. You can also run additional validation on the new code, including security scans or more complex test suites like performance or penetration tests. Because the changes are on a separate git branch, you can follow your standard code review, testing, and deployment processes. For extra safety, you can deploy using Lambda traffic shifting with Versions and Aliases, which you can use to roll back.

AWS Transform custom for platform teams

The preceding workflow works well for application teams managing tens or hundreds of functions across a few repositories. But what if you’re a platform team coordinating upgrades across thousands of functions in multiple AWS accounts?

In that case, you must orchestrate upgrades across teams and repositories. In some cases, you might apply the upgrades yourself. In other organizations, you focus on coordination and keep ownership of the upgrades distributed. In both approaches you need visibility to the breadth of the challenge, and tools to monitor progress. Transform custom campaigns can help.

Initiating and tracking an upgrade campaign

Platform teams create campaigns through the AWS Transform custom web application. Log in to the web application, create a workspace, and describe your goal. For example, “I want to upgrade all Lambda functions from Node.js 22 to Node.js 24.” AWS Transform custom displays matching transformation definitions and generates a campaign with a unique campaign ID and CLI command. Note: the command includes --trust-all-tools and --non-interactive switches, meaning it will run without tool prompts or user assistance.

atx custom def exec \
    --code-repository-path <path-to-repo> \
    --non-interactive \
    --trust-all-tools \
    --campaign <campaign-id> \
    --repo-name <repo-name> \
    --add-repo

You can identify candidate functions in your organization with AWS Trusted Advisor, the AWS CLI, Amazon CloudWatch, or AWS Config. To distribute upgrade responsibility, map the functions to owners using Tags or deployment metadata in AWS CloudTrail or your continuous integration and delivery (CI/CD) pipeline. Then share the campaign command with them.

Run the command against each target repository. When the command runs, it automatically registers the repository with the campaign. It then begins the upgrade based on the configuration the platform team chose when creating the campaign.

The AWS Transform web application dashboard tracks campaign progress at a glance. It shows total repositories registered in the campaign and how many are completed, in progress, or not started. It also reports success and failure rates along with transformation results and validation summaries.

The following figures show examples of dashboard visualizations.

AWS Transform console screenshot showing progress of a transformation campaign. The pie chart shows 10 of 10 repositories upgraded. The data shows 73 files and 407 lines of code modified, and the validation rate is 100%.

AWS Transform console screenshot showing a breakdown of files changed and lines of code modified for each of 10 repositories in the upgrade campaign.

AWS Transform report showing estimated saved time of 326 hours.

Scaling with cloud infrastructure

AWS also provides Open Source infrastructure that can automate parallel transform execution using AWS Batch and AWS Fargate. This solution moves processing to the cloud from individual developer machines to help you move more quickly, and includes:

  • REST API: submit single transformations or batches of thousands.
  • Serverless compute: AWS Batch with Fargate runs transformation jobs in parallel.
  • Automatic credential management: AWS Identity and Access Management (IAM) credentials auto-refresh, avoiding long-lived access keys.
  • Multi-language container: pre-built container supporting Java, Python, and Node.js with build tools included.

The default configuration supports up to 128 concurrent transformation jobs, with automatic queuing and resource management. For detailed implementation guidance, cost information, and code, see Building a scalable code modernization solution with AWS Transform custom.

Note: AWS Batch and Fargate incur additional charges beyond AWS Transform custom. See README for cost details.

Clean up

AWS Transform custom charges for active agent work during server-side operations. To avoid ongoing charges, stop any running transformations. See the AWS Transform pricing page for details.

If you deployed the scaling infrastructure, follow the cleanup instructions.

Conclusion

You can streamline Lambda runtime upgrades with AWS Transform custom, an Agentic AI service purpose-built for code modernization.

Customers with a backlog of existing functions to upgrade can use Transform custom to coordinate and streamline bulk upgrades across their organization. Transform custom also helps you move from the tail of the release cycle to the leading edge. By making runtime upgrades faster and more straightforward, you can stay ahead of the challenges of deprecation and take advantage of better performance and new features from newer runtimes.

AWS Transform custom fits into each stage of the software development lifecycle: surface risk early, confirm validation coverage, transform, validate, deploy. It can work with your existing code management, build, test, and deployment, giving you control over changes using your existing processes and tools.

Start with the documentation transform on a function today to get hands-on with AWS Transform custom. Review the currently-deprecated runtimes and make a plan to upgrade.

For more information, see AWS Transform custom documentation and Getting Started topic in the AWS Transform User Guide.

For more serverless learning resources, visit Serverless Land.

About the authors

Brian Krygsman

Brian Krygsman

Brian is a Senior Solutions Architect at Amazon Web Services. He has an application development background and technical depth in event-driven architectures and serverless development. He works with enterprise customers to effectively leverage cloud when building scalable, fault-tolerant, high-performant, cost-effective solutions.

Jonathan Tuliani

Jonathan Tuliani

Jonathan is a Principal Product Manager with AWS Lambda. Based in Dublin, Ireland, Jonathan is responsible for Lambda’s programming language runtimes. He bridges between customers and engineering teams to define strategy, prioritize investments, and design features that solve real-world customer problems.

Седмицата (15–20 юни)

Post Syndicated from Йовко Ламбрев original https://www.toest.bg/sedmitsata-15-20-yuni/

Седмицата (15–20 юни)

Въпросът на седмицата, който вълнува политическите наблюдатели в целия свят, е загуби ли Тръмп войната в Иран. Оценките са далеч от еднозначни, но дори сред привържениците на президента в Републиканската партия се чуха твърдения, че това развитие е най-тежката външнополитическа грешка на САЩ от десетилетия насам. Разбира се, тези оценки щяха да имат значение, ако и подписаният меморандум имаше значение – а в това никой не изглежда да е сигурен.

У нас също малко неща изглеждат сигурни. Все още няма представен бюджет, а почти преполовихме годината. Затова пък парламентът позволи на правителството да изтегли до 3,8 млрд. евро допълнителен държавен дълг за финансиране на бюджетен дефицит. Съвсем несигурни изглеждат и реформите, очаквани от кабинета Радев – и тези в държавната администрация, и тези в съдебната система. За сметка на това обаче доскорошният президент е пределно ясен по отношение на новия пакет санкции срещу Русия… с православни аргументи. Приоритети!

И Делян Пеевски изглежда непоклатим, макар и черупката на лидерството му в ДПС да се понапука напоследък. Рано е да проличи дали усилията му да обедини партията със законодателни предложения по теми, които са емоционални за мюсюлманската общност, ще спрат разпадането на локални структури и ще поправят слабия електорален резултат. Зависи от това дали „Прогресивна България“ ще успее да измести ДПС в местното управление, или българските турци ще продължават да гласуват както досега. Засега Пеевски олеква, но не достатъчно, за да потъне, смята Емилия Милчева.

Пеевски олеква. Не съвсем, не съвсем

Пеевски изглежда непоклатим, но под повърхността се виждат първите пукнатини. Отцепления в ДПС, битка за смесените райони и амбициите на Радев очертават възможно пренареждане на политическите мрежи, които години наред крепят едни и същи играчи на власт. Но нека не бързаме… От Емилия Милчева.

Управляващата партия обаче не за първи път демонстрира, че прогресивността е само куха част от названието ѝ. Личи от декларацията ѝ в подкрепа на т.нар. Шествие за семейството. Това е онзи тип „подкрепа“, която избира „правилното“ и зачерква „неправилното“. Според Светла Енчева България навлиза в нов етап на отношение към човешките права и демокрацията – и по-точно заглавие на текста ѝ от „Добре дошли в държавния регрес!“ едва ли може да се намери.

Добре дошли в държавния регрес!

Днес езикът на „традиционните ценности“ все по-често измества езика на демокрацията. Как стигнахме дотук и защо отношението към ЛГБТИ+ хората се оказва показателно за по-дълбока промяна в държавата и в обществото? От Светла Енчева.

Съвременните палиативни грижи не са „медицински грижи в края на живота“, а комплексен подход, който цели да направи живота на хората с тежки заболявания по-пълен и достоен, колкото и да остава от него. В България обаче това остава извън фокуса на публичния дебат и обществените политики. Липсват думи като „радост“ и „игра“, когато се обсъжда темата за детските палиативни грижи, а гласовете на самите деца и техните семейства почти не се чуват. Според Надежда Цекулова точно думите, които избираме да не включим в този разговор, показват в коя посока като общество сме решили да гледаме и какво остава извън полезрението ни. Непременно прочетете статията ѝ „Големият отсъстващ. Детските палиативни грижи в публичните политики и в публичния дебат“.

Големият отсъстващ. Детските палиативни грижи в публичните политики и в публичния дебат

Детските палиативни грижи в България остават почти невидими – и в законите, и в обществения разговор. Политиките свеждат темата до болници и терминални състояния, а медиите рядко говорят за качество на живот, достойнство, радост и игра. Цената я плащат децата и семействата им. От Надежда Цекулова.

Как архитектурата се превръща в средство за инструментализиране на властта, е въпросът, който пулсира от всеки абзац на статията на арх. Анета Василева, която има късмета да наблюдава от първо лице началото на вълната от граждански протести против застрояването в Албания. Във фокуса на обществения гняв е лично Еди Рама, министър-председател и бивш кмет на Тирана, превърнал столицата в лаборатория за авангардни архитектурни проекти, но на фона на хаотично градоустройство и изключване на местните професионалисти от процеса. А т.нар. Фламингова революция е, от една страна, отчаян ход на съпротива на обикновените хора, а от друга, разкрива лицемерието на статукво, в което архитектурата е оръжие, но и жертва на политически амбиции.

Протестите в Албания, или как архитектурата отново излезе на терена на политическото

Със сигурност сте чули за масовите протести в Албания срещу инвестиционните апетити на зетя на Тръмп, който иска да застрои защитена територия. Анета Василева беше на място, разкрива ни контекста в дълбочина и споделя защо в Тирана се е срамувала, че е архитектка от чужбина.

За началото на лятото в рубриката ни на „Второ четене“ Антония Апостолова ни връща към един доста различен сборник с разкази. „Да си мъж“ на Никол Краус само на пръв поглед изглежда равен и сякаш безсъбитиен, докато дълбочината на прозата е скрита именно в липсите, празнините и неизказаните неща. Краус майсторски превръща микрокосмоса на семейното и интимното в универсални размисли за паметта, идентичността и времето. „Да си мъж“ е книга за това как човек свиква с непознатото, превръщайки го в част от себе си, и как паметта – и личната, и колективната – оформя същността ни. Според Антония това е сборник на човешкото оцеляване.

На второ четене: „Да си мъж“

Дни преди лятното слънцестоене Антония Апостолова ни препоръчва книга с разкази, която на пръв поглед е безсъбитийна и се занимава с липсите, празнините и пукнатините в историите. Вторият поглед и прочит обаче ще ви отведе на неподозирани места. Никол Краус умее това.

А като говорим за оцеляване чрез изкуство, за мнозина точно с музиката по-лесно се преглъща ежедневието. Как обаче да стане това, ако живееш в свят, в който музиката (особено западната) е „суетно забавление“, че даже отваря и врата към пороците и разврата. В рубриката си „Ориент кафе“ Атанас Шиников сблъска хевиметъла с исляма – два свята, осъдени на вечна раздяла. В първата част на „Къде е шейтанът тук?“ Аллах и тежка музика“ ще научим за корените на това негативно отношение на ислямските авторитети към музиката, но и защо дори в най-консервативните общности тя си пробива път. За втората част Атанас обещава да е запазил повече от същинското „главотръскане“. Нямаме търпение…

„Къде е шейтанът тук?“ Аллах и тежка музика

В сезона на „Хилс ъв рок“ и изобщо на всякакви благини „он дъ рокс“ Атанас Шиников набива спирачка по хлъзгавия терен на исляма и метъла – тези „две субстанции, осъдени на вечна раздяла“. В първата част на статията звучат гласовете на ислямските авторитети; другата седмица е ред на жиците.

Не знам дали вече се разчу, но след множество прожекции и срещи с публиката в цяла България филмът на Лина Кривошиева „Какво е да остарееш в България“ вече е достъпен за свободно гледане в YоuТube канала на „Тоест“.

Качествената документалистика обаче не може да се прави без пари. Този филм беше финансиран по проект, но за следващата тема (и своеобразно продължение на разговора) с Лина имаме нужда от вашата подкрепа. Документалните филми са отделна част от нашата дейност и за тях винаги търсим самостоятелно финансиране, така че да не отклоняваме средства от всекидневната журналистическа работа. Включете се с дарение за краудфъндинг кампанията на „Тоест“ за новия ни филм „Какво е да си млад в България“. Всички средства от нея ще отидат директно за заснемането, озвучаването, историята и екипа, който ще я разкаже.

Краудфъндинг кампания за следващия филм на „Тоест“

Всяко евро, което дарите в тази кампания, отива директно за заснемането, озвучаването, историята и екипа, който ще я разкаже. Подкрепете следващия ни документален филм „Какво е да си млад в България“!

Част от вас може би вече дарявате всеки месец на „Тоест“. Тези средства осигуряват издръжката на медията – журналистическата работа, редакционния процес, комуникацията с публиката, развиването и поддръжката на сайта.

А ако все още не сте регистрирали редовно дарение на „Тоест“, направете го, за да може екипът ни да продължи да прави качествена журналистика за всички.

Станете месечен дарител

The Data Canary: How Netflix Validates Catalog Metadata

Post Syndicated from Netflix Technology Blog original https://medium.com/netflix-techblog/the-data-canary-how-netflix-validates-catalog-metadata-18b699d58e36

By Celina Amados

At Netflix, our catalog metadata is crucial to our member experience, and a single corrupted data state can impact millions of viewers immediately. To protect streaming reliability, we built an automated data canary system that validates data transformations using production traffic. This canary detects issues in under 10 minutes, and blocks bad data from reaching our members.

Intro

Catalog metadata is what makes Netflix functional. It defines what titles exist, where they’re available, whether they can be played, and more. This data gets transformed and distributed across our vast infrastructure near-continuously, powering everything that helps members find what they want to watch. Accurate catalog data delivers moments of joy. Corrupted catalog data breaks streaming.

What Went Wrong

A production incident revealed a critical gap in our resilience strategy. No code had been deployed. No configuration had changed. But, a manual mitigation action taken during a previous incident had inadvertently corrupted a data feed, rendering it empty for a subset of titles.

The impact was immediate: missing metadata prevented manifest generation, causing failures in our catalog service and playback issues.

Engineers were alerted immediately, but identifying the root cause took time. After intense triaging, responders pinpointed the corrupted data feed and pinned services back to a known-good state, restoring playback.

The problem? Our sophisticated code canary deployments had caught nothing. No code had changed — the data had.

This incident exposed a fundamental gap in our resiliency capabilities: we can validate code deployments, but we had no equivalent for our high-velocity data pipelines. Our catalog metadata, consisting of titles, artwork, availability, and more, was continuously transformed from multiple upstream sources and published at a regular cadence. Each upstream source had its own validation, but these checks didn’t catch corruption in the final transformed output.

We needed to treat data deployments with the same rigor as code deployments.

The Challenge: Validating Data at Short Intervals

Our catalog metadata service operates as a high-velocity data pipeline: it processes multiple input feeds, transforms them, and publishes the final catalog state that gets distributed across our infrastructure.

This creates unique validation challenges that our traditional canary analysis tools aren’t designed to handle:

Time Constraints: Our existing canary analysis tools require 30–60 minutes to reach statistical confidence. We had a much shorter window between data cycles; we needed to detect issues, make a decision, and block publishing all within a single cycle.

Emergent Issues: While each upstream data source has independent validation, problems often only manifest in the final transformed state. We needed to validate the actual output that clients would consume, not just the inputs, as close to the clients as possible.

Production Traffic is Essential: We initially considered shadow traffic, but quickly realized it was insufficient. Shadow traffic can only replay requests to our catalog metadata service; it can’t simulate the entire playback lifecycle across multiple services and domains. To detect real customer impact, we needed real production traffic.

Limit Blast Radius: Despite using production traffic for validation, we couldn’t allow customers to experience widespread issues during the validation process. Any regression needed to be detected and contained immediately.

Our Solution: The Data Canary Orchestrator Pattern

After evaluating several architectural approaches, we developed a solution built around three key innovations:

1. Dedicated Orchestrator Pattern

We created a dedicated cluster for the purposes of canarying new catalog metadata that separates concerns, avoids self-testing, and provides a pattern for extensibility. Here’s how it works:

Orchestrator Instance: A dedicated orchestrator instance of our catalog metadata service coordinates the data canary flow. When a new catalog version is published to the canary environment, the orchestrator validates that both baseline and canary clusters are healthy and version-synchronized, then triggers a chaos experiment.

Permanent Baseline & Canary Clusters: Two dedicated service clusters run continuously in our canary region. The baseline cluster always serves the latest production catalog version, while the canary cluster receives new versions for validation.

Generic Integration Point: Upon chaos experiment completion, the orchestrator reports results back to the transformer service via a REST endpoint. This generic interface means new data sources can implement their own orchestrator patterns without requiring transformer code changes.

This pattern can now be adopted by other teams at Netflix for validating different data sources, which is exactly the kind of extensibility we designed for.

Data Canary workflow

2. Utilizing and Extending our Chaos Platform

Meeting the 10-minute constraint required not only leaning on our chaos platform, but also extending it to meet our needs:

Custom Threshold Tuning: We worked with our Resilience team to customize experiment thresholds for our use case. Standard chaos experiment thresholds were too conservative for our time constraints.

Multi-Tenant Testing: Our catalog service supports multiple client types with different traffic patterns and downstream dependencies. We ran separate experiments for major client types and discovered that running traffic through the tenant that handles playback requests consistently identified failures fastest.

Sticky Canaries: To isolate experiment traffic, sticky canaries use session affinity to guarantee that once a user’s traffic is routed to the baseline or canary clusters, it stays there for the duration of the experiment window. This prevents cross-contamination from concurrent chaos experiments, ensuring a clean apples-to-apples comparison between data versions.

Behavioral Metrics Over Technical Metrics: We focused on Starts Per Second (SPS), or actual customer playback attempts, as our primary signal. SPS proved more reliable than latency or error rates for detecting catalog corruption because it directly measures customer impact, and data errors may not always manifest as application errors to our catalog metadata service.

Immediate Abort on Regression: Instead of collecting data for post-hoc analysis, we stream metrics in real-time and abort experiments the moment we detect regression. This trades some statistical confidence for speed, but our tight thresholds and clear signal make this not only acceptable, but necessary.

3. Production-Hardened Edge Case Handling

Building a system that runs in production every 10 minutes taught us that the devil is in the details:

In-Flight Experiments During Redeployment: When the orchestrator restarts, it must detect and continue polling any ongoing experiments, as we can’t abandon a validation cycle mid-flight.

Leader Election: During orchestrator deployments, multiple instances might be running simultaneously. We implemented safeguards to ensure only one experiment is triggered per version announcement.

Version Synchronization: In a multi-tenant service where different clients consume data at different cadences, we track version state to ensure baseline and canary clusters are properly aligned before triggering experiments.

Validating the Validator: Controlled Failure Injection

To prove the system worked, we needed to break things on purpose. We ran a series of controlled experiments where we deliberately corrupted catalog data — denylisting high-profile titles and simulating real data corruption scenarios — to validate that the canary could detect issues and block publication.

These experiments were coordinated as proactive incidents during business hours, with product operations teams on standby. We routed approximately 0.2% of global traffic through the validation flow, minimizing blast radius while still generating meaningful signal.

Key Results:

  • Detection Speed: Issues identified in 2.5–4 minutes depending on client type
  • Clear Signal: 10x error differential between canary and baseline
  • Automatic Blocking: Publishing workflow blocked as designed when regressions detected

The experiments validated our end-to-end workflow and revealed important operational insights: different client traffic patterns detect failures at different speeds, and threshold tuning requires careful refinement based on the magnitude of impact we want this system to detect. Most importantly, they proved that even with a 10-minute validation window, far shorter than traditional 30–60 minute canary analysis, we had sufficient signal to catch high-impact catalog corruption.

Bringing Code Validation Principles to Data

This effort wasn’t just about building a validation system, it was about recognizing that data deployments deserve the same rigor as code deployments. Just because something isn’t a binary doesn’t mean it can’t break production. The patterns we landed on aren’t specific to catalog metadata, and can be applied to systems with high-velocity data pipelines more broadly.

If you’re working with data that changes frequently and impacts customers directly, ask yourself:

  • What’s your MTTD for data corruption?
  • Can you validate with production traffic safely?
  • How would you detect emergent issues in transformed data?
  • What behavioral metric most closely indicates customer impact in your domain?

Today, the failure mode that caused the aforementioned incident would be caught and mitigated in under 10 minutes. We all know outages aren’t a question of if, but when. The next time you find yourself faced with bad data, how fast will you be able to respond?

Acknowledgments

This work was a collaborative effort across multiple teams at Netflix. Special thanks to Jongyoon Lee, David Su, and Zubeen Lalani of the Catalog Foundations & Distribution team for their contributions to the design, and to Ales Plsek of the Resilience team for their support in customizing our chaos platform for our unique use case.

The Data Canary: How Netflix Validates Catalog Metadata was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Data Projects: Managing Data Assets at Netflix Scale

Post Syndicated from Netflix Technology Blog original https://medium.com/netflix-techblog/data-projects-managing-data-assets-at-netflix-scale-7ca25888591e

By Amer Hesson, Marcelo Mayworm, James Mulcahy, and Brittany Truong

The Problem: Managing Assets at Netflix Scale

Netflix’s Data Platform is vast. We have millions of tables in our data warehouse and tens of thousands of scheduled workloads running across our orchestration systems. Behind each of these assets sits an engineer, a team, or an initiative — and behind each of those sits a set of decisions about who can access what, and how those workloads execute day after day.

For years, the tools we used to manage access and identity for these assets operated at the granularity of the individual asset. Every table had its own Access Control List (ACL). Every workflow ran under the identity of the engineer who authored it. In a workforce that is fluid, where people change teams, change roles, and occasionally leave the company, this fine-grained model broke down in two persistent, painful ways.

Problem 1: Permissions that can’t keep up with organizational changes

Imagine you’re on a team that owns a few hundred tables. Your org restructures, a neighboring team merges into yours, and you inherit another few hundred. Now you have to find every ACL on every table, figure out who should still have access, and update them one by one. Multiply that by every reorg across every team across the company. The result? Two failure modes:

  1. The support team gets flooded. A significant and outsized share of support threads were requests to update table permissions en masse in response to org changes. While self-service tooling and best practices are in place to manage this, adherence is inconsistent. Data Projects addresses this by promoting the solution from optional tooling to a foundational part of the data platform.
  2. Access gets granted far too broadly. Rather than maintain fine-grained ACLs, teams would often open up table access to the whole company. This defeated the purpose of having ACLs in the first place.

Problem 2: Workloads tied to human identities

Scheduled and asynchronous workloads — Maestro workflows, data movement jobs, Spark pipelines — need an identity to run as. Historically, that was a human: whoever authored the workflow.

Human identities are not durable. People change teams, get new responsibilities, and leave the company. When they do, their permissions change, and the workflows running under their identity start to fail. The only fix was to swap in a colleague’s identity, which inevitably had different permissions, kicking off a “permissions whack-a-mole” as each fix surfaced the next missing grant. And then, eventually, that colleague would also move on, and the cycle would repeat.

Enter Data Projects

We introduced Data Projects to tackle both problems head-on. At its core, a Data Project is two things:

  1. A container to manage and view a set of related assets in aggregate: tables, workflows, and other data assets grouped under a single logical umbrella.
  2. A synthetic, durable, and assumable identity: one that asynchronous and scheduled workloads can execute under, independent of any human’s lifecycle.

You can think of it as hoisting the granularity of management up from the individual asset to a meaningful container: the project. Instead of managing permissions on 500 tables, you manage them on one project that contains those 500 tables.

While the initial focus has been access and identity, the abstraction has applications well beyond those concerns. That broader potential is part of what makes it worth investing in.

Figure 1a. Individual assets, each managed in isolation, with per-asset access controls and per-person ownership.
Figure 1b. These assets are logically grouped into projects for easier management.

Grants and Roles

Each Data Project has a set of grants managed by the owning team. Different identity types can be added as grants: users, groups, applications, and continuous integration (CI) jobs. Each grant has a role that determines what the grantee can do within the project. For example, a Contributor has read/write access to the project’s assets, while a Viewer has read-only access. These roles roll up neatly — instead of rewriting hundreds of ACLs when someone joins or leaves a team, you update a single project grant.

The Identity Umbrella: Netflix and IAM

Every Data Project is provisioned with a Netflix application identity, and optionally an AWS IAM role. This is the “identity umbrella” that makes workloads durable:

  • The project’s Netflix identity is what executes the project’s async workloads (e.g. Maestro workflows). It belongs to the project, not to any person.
  • The project’s IAM role supports specialized use cases in AWS like Spark jobs on Amazon EMR. Crucially, the IAM role can be exchanged for the project’s Netflix identity in a cryptographically secure way.

Members with privileged roles can also assume the project’s Netflix identity. This is enormously useful for testing and troubleshooting from a development context like a laptop or a notebook — you get to run commands as the project, exactly as the scheduled workload would.

Gravity

One of the more elegant properties of Data Projects is what we call gravity. When a workload running under a project’s identity creates a new asset — say a Maestro workflow creates three tables — those assets are automatically added to the project as contained assets. The project becomes the center of mass for everything produced under its identity. You get organization for free as a side effect of how the platform already works, eliminating future challenges of discovering relevant assets and gaining access to them.

Securing Data Workflows with Data Projects

Maestro is Netflix’s primary workflow orchestrator for batch analytics, covering scheduled ETL pipelines, data movement jobs, ML training, and much more. Because workflows can run on schedules without the original user present, Maestro is designated a Trusted Workload Manager (TWM), formally authorized to mint fresh identity tokens on behalf of the workloads it manages.

That identity matters everywhere. A single workflow execution may be checked against table ACLs in the Secure Data Warehouse, authorization policies for Netflix resources, and IAM policies for AWS — all in a single run. If the identity is fragile, the whole workflow is fragile.

The Problem with User-Tied Identity

The standard pattern was to run workflows under an On-Behalf-Of (OBO) credential — for example, maestro OBO [email protected]. This gave the workflow the union of Maestro’s and the human’s permissions, but in doing so it also bound the workflow’s permissions to that person’s. When they changed teams or left Netflix, the workflow broke. A colleague might take over ownership, but they rarely had the same access as the previous owner, so the workflow would stay broken for days while permissions were sorted out. At Netflix’s scale, with tens of thousands of scheduled workloads, many of them business-critical, this was unsustainable.

Data Projects: Durable Identity

Data Projects solves this by replacing user-tied identity with a durable, team-owned Netflix application identity: one that doesn’t change teams, go on vacation, or leave the company. Each project groups related workflows, tables, secrets, and other assets under a single consistent identity, and Maestro validates the caller’s access to the project before executing any workflow under it.

The downstream improvements are as follows:

  • Tables created during execution are automatically associated with the project’s identity through gravity, inheriting its access controls without additional configuration.
  • Secrets are scoped to project policies, so ownership transfers no longer strand credentials.
  • Access is managed once at the project level, replacing fragmented per-user grants across every asset the workflow touches.

The result is a workflow identity model that is stable, auditable, and built to survive the organizational changes inevitable at any company operating at this scale.

Success Stories

Many Data Projects have already grown to contain tens of thousands of assets in production. A couple examples are highlighted below:

  • Streaming Quality of Experience: A core observability pipeline tracking quality of experience (QoE) metrics whose continuity used to depend on whichever engineer happened to own the underlying workflows. Now it runs under the project’s identity, stable regardless of team membership changes.
  • Member Analytics: Analytical models and ETL workflows for member data products. A concentrated set of business-critical analytics whose access is managed at the project level rather than across hundreds of individual tables and workflows.

More broadly, we’ve seen Data Projects adopted as the organizing principle for entire analytics domains. Where teams previously maintained their own access policies, ad-hoc grant lists, and tribal knowledge about “who should have access to what,” the project is now the single answer.

Using Data Projects

Onboarding workflows onto Data Projects is a matter of:

  1. Creating a project for the logical grouping of assets (or using an existing suitable one).
  2. Granting the right people and groups the appropriate roles.
  3. Configuring the workflow to run with the project’s identity.

Thanks to gravity, new assets produced by project workflows land in the project automatically. Migrating existing workflows can be a challenge as it requires setting up the Data Project with the appropriate permissions before changing its execution identity. We are actively working on infrastructure to track the access patterns of existing workflows so that we can recommend precise permission updates for the destination project. Our goal is to make the Data Project the de facto option for executing any kind of asynchronous workload.

What’s Next

Data Projects started as an Analytics Platform initiative, a response to specific pains in the data warehouse, but the underlying ideas are not unique to data. We see a potential future where Projects (not just Data Projects) are a first-class platform concept spanning data assets, software assets (GitHub repositories, Spinnaker applications, Docker images), and even studio assets (production content, pipelines, and transformations).

We’re also investing in:

  • Rightsizing: we’re integrating a layer on top of our authorization policies that automatically rightsizes permissions based on actual usage patterns, proactively eliminating unnecessary access and preventing “permission creep”.
  • Hoisting beyond access and identity: the project is a natural unit for surfacing other concerns at the aggregate level — cost attribution, health indicators, and more.
  • Ad-hoc use case integrations: extending project identities beyond scheduled workloads to cover interactive, on-demand actions like running a query through the Data Portal.
  • Activity logs and audits: a unified timeline of grant changes, asset changes, and workflow versions at the project level.

Conclusion

Data Projects is an answer to a simple observation: at Netflix’s scale, the unit of identity and access management can’t be the individual asset or the individual human. It has to be something larger, something durable, something that matches the way teams actually think about the work they own.

A project is that unit. And as we continue to generalize the concept beyond the data warehouse, we expect it to become one of the foundational primitives of how engineering at Netflix is organized, not just how data is organized.

Acknowledgments

We would like to express our gratitude to the following individuals for their contributions to this effort: Ryan Bordo, Doug Clark, Luke Fernandez, Sarrah Figueroa, Ankit Gupta, Brian Hoying, Ye Ji, Abhishek Kapatkar, Anmol Khurana, Matheus Leão, Hechao Li, Raymond Liu, Alice Naghshineh, David Noor, Anjali Norwood, Javier Garcia Palacios, Kunaal Parekh, Brandon Quan, Andrew Seier, Jason Seo, and Ethan Zhang.

If you are interested in helping us solve these types of problems and helping entertain the world, please take a look at some of our open positions on the Netflix jobs page.

Data Projects: Managing Data Assets at Netflix Scale was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Predicting Risk in Content Launches: How Data-Driven Insights can Transform Launch Planning

Post Syndicated from Netflix Technology Blog original https://medium.com/netflix-techblog/predicting-risk-in-content-launches-how-data-driven-insights-can-transform-launch-planning-587b1f2de928

by Emily Gill

Each year, we bring the Analytics Engineering community together for an Analytics Summit — a multi-day internal conference to share analytical deliverables across Netflix, discuss analytic practice, and build relationships within the community. This post is one of several topics presented at the Summit highlighting the breadth and impact of Analytics work across different areas of the business.

Understanding Risk in Content Launches

Every title you see on Netflix goes through several key phases: Development, Pre-Production, Production/Principal Photography, Post-Production, and finally, Launch Preparation, all leading up to the Title Launch. Once Principal Photography wraps, the focus shifts in Post-Production from content creation to quality assurance and visual effects (if needed).

At the end of Post Production, Netflix receives the final audio and video files — often delivered as an IMF (Interoperable Master Format) — which triggers a flurry of Launch Preparation activities, focused on tasks such as the development of artwork and trailers, creation of subtitles, maturity ratings & quality control, that happen within a tight window and rely on having the finalized media assets in hand.

Some of this work can be kicked off earlier using a non-final version of the media called the Locked Cut, but since it’s not the absolute final deliverable, this presents a tradeoff: should our teams who prepare content for service wait for the more finalized IMF to begin their work, or start sooner with the unfinal Locked Cut? Waiting for the IMF risks a compressed timeline if it arrives late, while starting with the Locked Cut means teams may need to do additional conformance work if there are significant changes between the Locked Cut and the final IMF.

Identifying Gaps in Schedule Accuracy

To help navigate the decision of when to start launch preparation, our teams rely on estimated delivery dates for both the Locked Cut and IMF media assets, which are manually provided by content partners in production schedules. However, these schedules often have gaps in coverage and lack accuracy for both asset types (see Figure 1).

Figure 1. At an asset-level we generally see that scheduled date accuracy and coverage are lower at horizons further from asset delivery. As we approach delivery (moving towards the right on this plot) schedules become more accurate (errors decrease) adn coverage improves.

This isn’t unexpected — productions are dynamic, facing frequent changes, scheduling conflicts, and unforeseen obstacles that can shift timelines without warning. As a result, there’s a clear opportunity to leverage the wealth of production data we collect to predict the risk of schedule slips. By developing a predictive model, we aim to both fill in ETA gaps (providing asset delivery estimates when none exist) and improve the accuracy of existing ETAs compared to traditional manual schedules.

Correlation between Schedule Accuracy and Launch Misses

Our analysis reveals a strong correlation between scheduled inaccuracies and launch misses — instances where a title experiences delays. To quantify schedule inaccuracy, we created a metric called Accumulated Error Days (AED), which measures the cumulative deviation between estimated (scheduled or predicted) delivery dates and actual delivery dates over time. AED is calculated retrospectively as the area between the scheduled (grey line) or predicted (blue line) delivery dates and the actual delivery date (green line).

When we compare titles with at least one launch miss to those without, we find that mean AED is significantly higher in the group with launch misses. Notably, this effect is even more pronounced when we focus on the period closer to delivery — indicating that high AED (i.e., inaccurate schedules) in the final stretch before launch is especially correlated with launch misses, more so than AED accumulated over a longer timeline. These findings further motivate our efforts to improve schedule accuracy and reduce AED by leveraging rich production data and predictive modeling.

Modeling Time-to-Delivery

Our predictive models are designed as boosted tree regression models that predict the “days until” either media asset delivery for in-progress productions.

To power these models, we leverage a range of upstream data sources including production-level signals of progress, title metadata, and seasonal signals. We are able to predict the days until media asset delivery using daily update snapshots, allowing us to generate up-to-date predictions that reflect the latest state of each in-progress production. This means that we have each feature and what its value was as of each day of a production. Modeling with this snapshotted data enables us to generate up-to-date predictions as new information becomes available, build a flexible model that works across all production phases, and seamlessly incorporate dynamic features that evolve over time (Figure 2).

Figure 2. Hypothetical illustration of the evolving nature of production-related signals used in our models. Some signals are present throughout but dynamic, others are present at single moments in time during specific production phases. By capturing data in a snapshotted form, we’re able to build a flexible phase-agnostic model that leverages many different types of progress signals. This figure is illustrative only and does not depict actual Netflix financial or production data.

Evaluating Our Approach

Building a Comprehensive Metrics Suite

When evaluating the performance of the predictive models, we look across a suite of metrics to try to understand where and when predicted dates outperform scheduled dates. Among these are mean and median absolute error, relative to actual delivery, to understand the accuracy of our estimated dates. We also consider bias metrics, such as mean and median error, to understand if we are consistently over- or under-predicting the actual delivery. We calculate the standard deviation of our errors to understand if there are large shifts in the bulk of the distribution of errors. For the tails of our error distributions, we calculate the percentage of our absolute errors that are greater than x days to delivery.

For scheduled dates, we calculate coverage across various horizons to delivery. This is a value prop of the model; we’ve built the model in such a way that we can always provide a predicted date and recoup any coverage gaps that exist from scheduled dates alone.

Benchmarking Against Manual Scheduling

In a backtest, we observed significant improvements across all of our metrics and across most horizons from delivery. As an example, see Figure 3 which plots global mean absolute error (MAE) and shows large reductions in errors (greater accuracy) in predicted IMF and Locked dates as compared to scheduled dates. Additionally, we see large reductions in outliers from scheduled to predicted dates as well.

Figure 3. This plot compares accuracy (measured as Mean Absolute Error) between predicted and scheduled dates. The horizontal axis plots time prior to delivery, which decreases from left to right until you reach the moment of delivery at the bottom right. For this particular asset, the predicted delivery dates on average are much more accurate than manually scheduled delivery dates throughout the full horizon to delivery.

Since our teams use these dates over a period of time and not at a single point in time, there is an additional benefit that we’re describing as an Earlier Accuracy Signal. By leveraging predictive dates, our teams benefit from a level of accuracy that they would otherwise have to wait x amount of time for if using scheduled dates. As an example, 6 months out from Locked Cut delivery the predicted dates are better than scheduled dates on 76% of titles and have a level of accuracy (6.1 wks MAE) that scheduled dates don’t reach until 11 weeks later.

Circling back to AED, which we mentioned earlier is correlated to launch misses, we find that in our backtested titles globally, and across most buying orgs and content types (i.e., series versus standalones), predicted IMF and Locked Cut dates reduce AED from scheduled dates when calculated across the 6 months leading up to delivery. We see similar patterns when we repeat this for shorter horizons to delivery as well.

Streamlining Workflows with Improved Scheduling

A key advantage of this predictive model is that estimated delivery dates are already integral to our stakeholders’ workflows — meaning we can introduce predictive dates without overhauling existing processes. However, this creates a new challenge: with both scheduled and predicted dates available, teams need to determine which is more reliable. While predictive dates are often more accurate on average, there are situations where scheduled dates perform better. To address this, we’ve built serving logic that defaults to scheduled dates in buying orgs where the model underperforms. Elsewhere, teams can view both dates side by side in dashboards, allowing them to apply their own judgment. Additionally, our predictive models leverage features that are tied to scheduled dates, which has emphasized the need and impact of ensuring our upstream teams continue to input and update scheduled dates even in the presence of our predictions. We’re piloting these predictive signals in multiple ways, tailoring the approach to fit the diverse needs and tools of our various launch prep functions.

Predicting Risk in Content Launches: How Data-Driven Insights can Transform Launch Planning was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Evolution of Cassandra Data Movement at Netflix

Post Syndicated from Netflix Technology Blog original https://medium.com/netflix-techblog/the-evolution-of-cassandra-data-movement-at-netflix-6e13329c80a1

By Guil Pires, Jennifer Prince, Jose Camacho, Ken Kurzweil, Phanindra Chunduru

Background

In a previous post, we introduced Data Bridge, a unified management plane for batch Data Movement at Netflix. Historically, several bespoke Data Movement connectors were developed across different engineering organizations to fulfill their specific requirements. Over the last few years, the Data Movement team has started centralizing these offerings through an abstraction that provides a catalog of connectors, along with simple UI and APIs to initiate Data Movement jobs.

One such case is the Cassandra to Iceberg connector. Apache Cassandra powers mission critical applications at Netflix, including Member, Billing, Recommendations, Subscriptions and many more. These use cases heavily leverage Data Movement to Apache Iceberg for many analytics and operational tasks, and central to this movement was a connector for Cassandra to Iceberg built in-house named Casspactor. As many Cassandra based Data Abstractions emerged, such as Key Value, Time Series and Graph — the need for larger and more complex Data Movement with transformations became more critical to the business.

Data movements are fundamentally fulfilled by leveraging the existing Cassandra backup infrastructure. Regularly scheduled backups are performed directly on the Apache Cassandra nodes, via a sidecar process managing the upload of all necessary SSTables and associated Metadata files directly into Amazon S3. When a Data Movement job is initiated, the job constructs the specific backup structure it needs by referencing the S3 based metadata, allowing it to precisely locate the SSTable files. The engine then downloads these files, performs the required mutation compaction and processing, and finally writes the fully transformed, compacted data directly into the target Apache Iceberg tables.

Image 1: Cassandra Cluster Backups to S3

Casspactor: The Engine We Outgrew

Casspactor processed roughly 1,200 data movements per day, transferring approximately 3 PB of data from Apache Cassandra into Apache Iceberg tables. It served some of the most critical workloads at Netflix. For years, it worked. Then, two compounding challenges made it clear we needed a fundamentally different architecture.

Fragile Metadata Dependencies

Before Casspactor could move a single record, it needed to answer a deceptively simple question: which backup exists, is it complete, and what does it contain?

Casspactor assembled this answer from multiple independent systems:

Image 2: Casspactor’s Composite View of a Backup

Each system had its own failure modes, update cadences, and accuracy guarantees. Casspactor’s view of the world was a composite, and composites diverge from reality.

Metadata fell out of sync with actual backups, causing Casspactor to read stale or incorrect data silently. Routine maintenance on the Cassandra Clusters triggered uncoordinated snapshots, and because Casspactor required all nodes in a region to snapshot at the same clock second, a single node replacement could break data movement for an entire region.

The fix was hiding in plain sight. The answer to “which backup exists and is it complete?” already lived in the backup storage layer (Amazon S3) itself. By reading metadata directly from the backup files, we could replace the entire dependency chain with a single source of truth.

Every Connector Inherited Casspactor’s Limitations

Cassandra at Netflix does not just store raw tables. It backs higher level data abstractions, such as Key Value, Time Series, and others, each with its own data model, access patterns, and semantics. When any of these abstractions needed to move data to Iceberg, they all funneled through Casspactor.

Every abstraction inherited Casspactor’s constraints:

  • Skewed partition failures: Casspactor could not handle tables with large partitions, a common pattern in Key Value and Time Series workloads. Jobs crashed with out-of-memory errors on some of Netflix’s largest datasets.
  • No data model awareness: Casspactor moved raw Cassandra tables as is. Connectors for Key Value and other abstractions had to bolt on post processing to reconstruct their data models from the raw output — extra cost, extra complexity, and an extra surface for failures.
  • Intermediate table bloat: Casspactor wrote to an intermediate Iceberg table before producing the final output. The Key Value connector added another intermediate table and a snapshots table. Connectors for abstractions on top of Key Value added even more. This compounded into significant storage cost overhead.
  • Inability to Time Travel: by relying on multiple services to compose a backup unit, Casspactor was unable to restore prior backups in the event of cluster Topology or Keyspace schema changes.
  • Monolithic design: Casspactor was built as a single connector, not as an engine. There was no way to build a family of purpose built connectors on a shared foundation.

We needed something fundamentally different: an engine that reads directly from backups in S3, produces standard Spark DataFrames, and lets each data abstraction build its own connector with full awareness of its data model. One foundation, many connectors.

The New Stack: A Layered Architecture

The new architecture, built upon the foundation of Apache Cassandra Analytics and the in-house Move Data framework, represents a fundamental shift toward a layered, purpose-built stack designed for reuse and maintainability. This new engine was conceived with clear separation of concerns, moving away from Casspactor’s monolithic design. The architecture is intentionally layered with the foundation being a core S3 reading capability: the Cassandra Analytics Wrapper, which is built on top of the Open Source Cassandra Analytics with Netflix’s internal backup representation and an S3 Client.

This layer handles the raw data retrieval from backups, translating it into standard Spark DataFrames. Sitting atop this foundation is a “Connector Factory” model, via both Java UDFs and transforms which allows individual data abstractions (Key Value, Time Series, others) to build highly optimized, data model aware connectors that process the generic Spark DataFrames, avoiding the need for complex, expensive, and failure-prone post-processing steps. This layered approach ensures that improvements to the core reading engine benefit all connectors, while the connectors themselves are focused solely on data transformation.

Image 3: The new Connector layered stack
  • Handles Skewed Partitions: By moving the mutation compaction and processing to the Executor level within Spark, the new engine can efficiently handle tables with highly skewed or wide partitions, a major pain point for Casspactor. Crucially, this processing occurs without excessive data shuffling, preventing out-of-memory errors and enabling reliable movement of Netflix’s largest datasets.
  • Operates at Spark DataFrames (No Intermediary Tables): The new architecture directly generates standard Spark DataFrames from the Cassandra backups. This eliminates the need for Casspactor’s costly, multi-stage intermediate Iceberg tables, which led to storage bloat and operational complexity. This native DataFrame operation enables the “Connector Factory” by providing a universal, easily consumable interface for building diverse, model specific connectors.
  • Jobs Auto Size: The engine integrates intelligent auto-sizing capabilities, allowing jobs to dynamically adjust resource consumption based on the source table’s characteristics. This removes the burden of manual tuning from engineering teams, ensuring optimal performance and cost efficiency without sacrificing reliability.
  • Reduced Dependencies: By reading metadata directly from the backup files stored in S3, the new stack removes the fragile, multi-service dependency chain that plagued Casspactor. S3 becomes the single, authoritative source of truth for backup existence and completeness, vastly improving data movement reliability and consistency.
  • Time Travel: A critical feature of the new stack is the ability to process the schema, cluster topology, and data as a cohesive unit at a specific point in time. This capability provides robust time travel functionality, essential for auditing, debugging, disaster recovery and reproducing past data states.
  • Performance: Collectively, these architectural improvements, including native DataFrame processing, optimized partition handling, and streamlined metadata retrieval have resulted in notable performance gains, reducing overall data movement execution runtime and cost compared to the legacy Casspactor system.
  • Cost: by eliminating intermediary Iceberg tables and efficient SSTable compaction on Executors, the new stack needs a significantly smaller storage and compute footprint leading to significant cost savings in the order of USD millions.

The Journey Towards a Safe Migration

The successful validation of the new stack was the critical first step, but it only marked the beginning of the most challenging phase: the migration. Large scale data migrations are inherently complex, high-risk undertakings that can be time consuming and often result in customer frustration and service disruption. To navigate the high stakes of decommissioning a mission-critical system like Casspactor and seamlessly replacing it, we needed a strategy that prioritized reliability and transparency above all else.

The migration was fundamentally enabled by a Like-for-Like strategy, which served as the cornerstone of our Platform Engineering philosophy, abstracting complexity. The core tenet was to maintain absolute consistency across the user-facing interface, the output contract, and the final data artifact. This meant ensuring that the data movement parameters defined via the Data Bridge abstraction remained unchanged, and, critically, the schema, metadata, and data within the destination Iceberg tables were identical to the legacy output. By preserving these external contracts, we eliminated the need for complex, time-consuming coordination with dozens of internal teams who relied on these data pipelines. This approach transformed the migration from a distributed, high-risk, multi-team effort into an internal platform implementation detail, allowing us to achieve a transparent, zero-impact transition and accelerate the retirement of the legacy system without requiring any code changes or validation from downstream users.

To navigate this migration, we developed a strategy anchored by three core pillars that serve as a blueprint for successful, large-scale data migrations:

  1. Validation: Establishing and maintaining absolute confidence in data consistency through rigorous, ongoing validation.
  2. Visibility: Instrumenting every part of the system to provide a clear, real-time understanding of migration progress and system health.
  3. Safety: Ensuring user impact is minimized or eliminated, despite the inevitable system failures, by leveraging abstractions and robust fallbacks.

The next section will provide a detailed exploration of these key pillars.

Pillar 1: Validation

Trust is earned, and in data migration, it is earned one row at a time. The first pillar is the most critical: providing a measurable guarantee to users and partners that the data produced by the new system is an exact, row-by-row replica of the data produced by the old one.

Our foundational tactic was deploying the new Move Data connector in a “shadow” testing that ran in parallel with the production Casspactor jobs. This allowed us to validate the new system with real-world, production workloads without any customer impact.

Image 4: Shadow job structure leveraged for data validation
  • Let C be the set of rows in the legacy Casspactor output (Iceberg table).
  • Let M be the set of rows in the new Move Data output (Iceberg table).

The test for trust: prove that C = M. This required continuously checking for two conditions:

  1. Rows in C but not in M (C-M): The new system missed data.
  2. Rows in M but not in C (M-C): The new system introduced phantom or erroneous data.

Any result where the cardinality of these difference sets (the number of differing rows) was greater than zero triggered an immediate, high-priority investigation. The target was 100% similarity.

Uncovering and Resolving Disparities

The shadow mode quickly became a powerful forensic tool, exposing “unknown unknowns”, subtle discrepancies that were not bugs in the new system but rather differences in behavior between the new and old systems. Resolving these was the core work of building trust. For each problem we initiated an investigation log where we captured the details, logs, queries that allowed us to diagnose. Based on the assessment the issues were categorized so that similar differences on other datasets were later resolved affecting many of the shadow pipelines.

Maintaining an investigation log was critical to organize the outstanding issues and effectively communicate to stakeholders the progress and confidence of the new connector so that we effectively measure the appropriate level of “confidence” to initiate the migration.

We observed differences in how connectors leverage reference timestamps for Time-to-Live, Consistency Levels, backup selection, and various internal business logic. This continuous, data-driven cycle of discovery and resolution was the mechanism by which we built confidence in the new architecture.

Pillar 2: Visibility

Trust is built in the background, but an active migration requires real-time insight: Visibility. The second pillar involves instrumenting the system to provide an unambiguous, clear understanding of operational health and migration progress.

We extended our instrumentation to the overall migration workflow and its dependencies:

  • Dashboards: We created centralized dashboards to track migration status, visualizing the total number of data movements migrated versus those remaining. The dashboards tracked execution status, average runtime, and cost comparisons between the two connectors.
  • Dependency Tracking: Since the new system relied on a new set of APIs to fetch backup metadata, we implemented detailed metrics for failures to keep track of the APIs or dependencies failed.
  • Alerting: Proactive alerts were set up for job failures (Move Data or Casspactor), failures on Move Data that triggered a fallback to Casspactor or any data discrepancy being detected.

This comprehensive instrumentation allowed the team to be proactive, fix issues as they emerged during the migration, and gain the necessary confidence to accelerate the migration timeline.

Pillar 3: Safety

Even with perfect data correctness and enhanced visibility, the third pillar, Safety is required for a zero-impact migration. The challenge is ensuring that when a system inevitably fails, the user experience is uninterrupted. Our strategy centered on decoupling the user’s workflow from the underlying connector implementation.

Leveraging Abstraction: The Decider Pattern

To achieve a transparent swap, we leveraged the Maestro workflow orchestration platform to implement the Decider pattern:

  1. Data Movement Abstraction: From a user’s perspective, their Data Movement job definition remained the same.
  2. The Decider Step: Internally the workflow responsible to execute the job was modified to include a Decider step. This step took the data movement parameters (source cluster, table name, destination) and invoked a control plane: Connector Controller.
  3. Connector Controller as the Registry: The control plane served as the dynamic registry. Based on the migration cohort and the data movement attributes, it determined and reported the appropriate connector to use either Casspactor (legacy) or Move Data (new).

This abstraction gave our team complete control. We could upgrade or rollback any connector for any data movement instantly by simply updating a configuration in the controller, with zero modification required to the thousands of downstream customer workflows. Crucially, this abstraction guaranteed the critical safety net: a conditional step in the Maestro workflow logic ensured that if the Move Data step fails, it would immediately execute the Casspactor step.

This pattern would increase the chances that the user’s data movement completes successfully, even if the new connector encountered a bug or transient failure during the initial rollout phases. User impact was completely eliminated; they might see a slightly longer runtime in the event of a failure and fallback, but they would never see a migration failure or suffer from stale data.

Image 5: The Decider Pattern Implementation via Maestro

Beyond the workflow, the new system architecture itself was inherently more resilient. By building the new data movement connector on Cassandra Analytics and reading backups directly from S3, we removed fragile dependencies on deprecated internal services.

Conclusion

The migration from Casspactor to the new, layered architecture built on Cassandra Analytics and the Move Data connector was more than a typical “tech debt” project; it was a fundamental shift in our approach to data movement reliability and scalability at Netflix.

The legacy system, while serving us well for years, was ultimately constrained by monolithic design, fragile metadata dependencies, and an inability to handle the complexity of modern data abstractions. The new stack resolves these issues by delivering a robust, cost-efficient, and inherently more resilient solution that reads directly from S3, handles wide partitions gracefully, and eliminates costly intermediate tables.

Our blueprint for the migration, anchored by the three pillars of Validation, Visibility, and Safety, ensured a transparent and high-confidence transition. Through rigorous shadow testing and a data-driven audit framework, we achieved the desired data consistency. Enhanced dashboards and alerting provided the real-time operational insight necessary to manage risk. Most critically, the implementation of the Decider pattern within our workflow abstraction minimized the impact for all downstream users.

This successful migration validates a core philosophy: by abstracting complexity at the platform level, we can perform large system migrations without burdening our product engineering partners. The new foundation is now ready to support the next generation of Netflix’s data abstractions.

Looking ahead

This foundational work on the Cassandra Data Movement stack has done more than just replace a legacy system: it has become an accelerator for innovation across the entire Data Movement organization. By providing a reliable, performant engine that standardizes data retrieval into Spark DataFrames, we’ve enabled the rapid development of new, highly optimized connectors. This new “Connector Factory” approach has already delivered a dedicated Key-Value to Iceberg and Time Series connectors, both of which are fully aware of their respective data models, eliminating costly post-processing. This architecture is also paving the way for ambitious new initiatives, including the development of a solution for bulk loading data into Cassandra itself, effectively completing the data movement cycle, and enabling safer fleetwide connector rollout with canaries inspired by the Decider Pattern.

We are incredibly grateful for the extensive collaboration among the Data Movement, Data Bridge, Online Data Stores, Membership, Billing, Subscriber and Ads platform teams at Netflix; this work simply couldn’t have been accomplished without their partnership!

The Evolution of Cassandra Data Movement at Netflix was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The collective thoughts of the interwebz