Last month, the Swiss non-profit foundation Quad9 announced that it will withdraw its free public DNS resolver from France. This difficult decision followed months of litigation and a French court ruling that imposed daily financial penalties on Quad9 for failing to block domains associated with alleged copyright infringement.
Quad9’s departure from France is both disheartening and understandable. The court’s ruling put the organization in the impossible bind of having to choose between crippling litigation and penalties, or fundamentally altering its service in a way that would undermine the privacy and security of its users. Left with no good option, Quad9 is choosing the exit door instead.
The withdrawal of a basic infrastructure service from France fundamentally harms the free and open Internet, and it should serve as a warning of the damage that will result if rightsholders and policymakers continue demanding maximal blocking across every layer of the Internet. It not only deprives French Internet users of a trustworthy and important tool; it creates a real risk of disruption to their daily lives. Quad9’s resolver is used by a wide variety of individuals and organizations in France — including government bodies, educational institutions, hospitals, and utilities — and Quad9 has no way to directly notify those individuals and institutions that their resolver will soon stop working. If the affected parties do not change their resolver before Quad9 removes its service, their Internet will fail with no explanation.
Quad9 is not the first public DNS resolver to withdraw from a country following blocking orders, and it almost certainly won’t be the last. Cisco previously withdrew its OpenDNS resolver from France, Portugal, and Belgium (temporarily) under similar circumstances. And a Belgian court recently issued an order that would impose substantial penalties if Cloudflare, Google, and Cisco do not begin blocking websites through their public resolvers by early January 2027. That’s something Cloudflare has never done.
Cloudflare has spent years advocating against orders of this kind. We are concerned that rightsholders’ unrelenting demands for blocking by public DNS resolvers will end up causing real harm to the Internet.
A refresher on DNS, the Domain Name System
To understand why this is such a problem, it helps to understand how the Internet works under the hood. DNS resolvers are a foundational piece of the Internet’s infrastructure. DNS is the Internet’s address book: when you type a website name such as cloudflare.com, the DNS system translates that domain name into the IP address your device needs to connect to it. A DNS resolver is the service that looks up that address and sends it back to your device, much like an operator finding a phone number in a directory.
DNS resolvers can be operated by anyone, but they are commonly run by Internet service providers (ISPs), mobile networks, businesses, and universities. If you’re reading this blog post at home and have never thought about DNS before, chances are that you’re using your ISP’s default resolver. Because you use your DNS resolver every time that you visit a website, it’s important for it to be reliable and trustworthy.
And because the DNS resolver has access to all of your browsing activity, which resolver you use has real privacy implications.
In addition to DNS resolvers that are provided by your local ISP, there are a variety of public DNS resolvers available. Cloudflare launched our public DNS resolver 1.1.1.1 in April 2018 to give back to the Internet. Our goal was to make a fast, reliable, privacy-first consumer DNS service available for free to everyone. We succeeded. 1.1.1.1 is among the fastest DNS resolvers in the world, including in Europe. Cloudflare's resolver doesn’t collect any user data, let alone sell it to advertisers. 1.1.1.1 has always been one of our favorite examples of Cloudflare fulfilling its mission to help build a better Internet.
1.1.1.1 is one of many public resolvers that anyone can use. In addition to 1.1.1.1 and Quad9’s 9.9.9.9, some of the other best-known public resolvers include Google’s 8.8.8.8 and Cisco’s OpenDNS. Just because these services are offered for free doesn’t mean they’re easy to provide. A well-run global public DNS resolver is an extraordinary engineering feat, built to securely and privately resolve hundreds of billions or even trillions of queries a day across a globally distributed network of hundreds of server locations.
As Quad9 notes, there are many other DNS resolvers available to users, many of them offered by essentially anonymous services whose practices are unknown or undocumented. It’s easy to change your DNS resolver, and people do so for a variety of reasons including performance, reliability, or to access specific security features.
Why public DNS resolvers should not be required to block
Rightsholders have a legitimate interest in preventing online piracy. But trying to address online infringement through public DNS resolvers is fundamentally the wrong approach.
Among other things, such blocking is ineffective. Think back to the analogy of an operator finding a phone number in a directory. If a court wants to shut down an illegal business, forcing the directory operator to cross the business's name out of the phone book doesn't actually close the business. The storefront is still open, the products are still there, and anyone who already knows the address, or uses a different phone book, can walk in.
The same is true for the Internet. DNS resolvers are just one link in the transmission chain. The only way a DNS resolver can “block” content is by declining to return the IP address for a particular website. But a public DNS resolver cannot remove the actual infringing content from the Internet, nor can it take down the servers hosting it.
Because of this, blocking by public DNS resolvers is remarkably easy to bypass. Any Internet user who wants to reach a blocked site can circumvent the block in less than a minute simply by switching to one of the thousands of other DNS resolvers available online. In fact, users don’t even need a resolver at all to reach infringing content if they already know the underlying IP address. Unsurprisingly, while blocking via public resolvers may inject some friction for some users when accessing content, there is no evidence that it actually reduces the total amount of online infringement that occurs.
In addition to being ineffective, as we’ve written previously, DNS blocking is a blunt instrument that causes collateral damage. Because DNS operates at the domain level, a resolver cannot block a specific infringing video or file; it must block the entire domain. This affects all content on that domain, whether legal or illegal. It is also entirely opaque to the user: when a resolver is forced to block a domain, the website simply fails to load. The user receives no explanation or legal notice, making it impossible to know if the site is down for maintenance, if their network is broken, or if the site was blocked by a court order.
The problem is substantially worse with respect to public DNS resolvers. Because public resolvers operate globally, any blocking will naturally cross borders, meaning that a single country could dictate what content is accessible around the world. For example, while the EU-dedicated DNS resolver DNS4EU operated by Whalebone blocks in response to country-specific orders, it does so across the entire EU. Efforts to reengineer public DNS resolvers to allow for more localized blocking would fundamentally alter the service, harming performance and resilience, undermining user privacy, and fragmenting the open Internet.
For these reasons, Cloudflare, Google, Cisco, and Quad9 have all opposed efforts to mandate blocking by public DNS resolvers, with some success. Several years ago, courts in Germany ruled in favor of Cloudflare and Quad9, rejecting requests to require DNS blocking. More recently, a French court rejected penalties against Cloudflare for failing to block through 1.1.1.1, finding that other actions we had taken through our CDN and reverse-proxy service were sufficient to comply with the underlying order.
Undeterred, rightsholders have continued to press for blocking by public resolvers with a level of determination that is hard to explain. A recent study found that the majority of online piracy in Europe is hosted in Europe. If rightsholders want to genuinely address online infringement, there are far better places to focus their efforts — most notably by requiring the hosting providers who actually store the pirated files to take them down.
Forcing public DNS resolvers out of markets hurts the Internet
The French court’s penalty order against Quad9 represents the culmination of these rightsholder efforts. As a small non-profit that only offers a public DNS resolver, Quad9 had no alternative methods for demonstrating compliance with the order. The French court therefore granted rightsholders’ request to issue penalties for non-compliance, forcing Quad9 to undermine its own service or leave the country.
Quad9’s withdrawal from France represents a genuine loss for the Internet. In the short term, removal of Quad9’s resolver is likely to cause immediate and potentially severe disruptions to Internet access. Because of the way their service is set up to preserve privacy, Quad9 has no way to know who exactly is using its resolver, let alone to directly communicate to them about the upcoming withdrawal of services. That means that individuals, hospitals, universities, and government offices all may be caught unaware when the resolver stops working. Again, because of how DNS resolvers work, those users will not receive a detailed error message explaining what happened — their Internet will simply stop working. The problem will be even worse when devices like security systems or thermostats are hardwired to use Quad9’s resolver. Fixing these issues may be slow or even impossible.
More generally, driving Quad9 out of France does absolutely nothing to prevent online infringement, but it does leave everyday Internet users with fewer safe, reliable options, particularly ones like Quad9 that also protect users from phishing and malware. Taking established public DNS resolvers away forces Internet users to fall back on the default resolvers offered by their ISPs (which may monetize their browsing data), try lesser-known resolvers operated by anonymous or less trustworthy organizations, or run their own resolver. The irony is that Quad9 and Cloudflare are being pulled into court precisely because we are transparent, reputable organizations. Penalizing transparent services only pushes users toward less accountable corners of the web.
The way forward
If nothing else, we hope that Quad9’s announcement may serve as a wake-up call to rightsholders and policymakers that seeking to block infringement at every layer of the Internet has real costs.
We believe there are far better ways to work together to fight piracy without compromising the integrity of the Internet, and Cloudflare continues to pursue innovative collaborations to that end. Our collective efforts in addressing online infringement would be put to better use in seeking to have such content removed at the hosting provider, which would both achieve a more durable solution and avoid collateral damage to the Internet.
In the meantime, we will keep resisting misguided efforts to require blocking by public DNS resolvers. Along with Google and Cisco, we have appealed the Belgian order that would impose penalties on us if we do not start DNS blocking by January 2027. We hope to get a resolution before the end of the year that would avoid another situation like the one Quad9 faces now in France.
We have grave concerns about the disruption that would be caused if additional resolvers are forced to withdraw from markets, and we will continue to advocate against that possible result.
На редакционната поща пристигна имейл, към който бяха прикачени първоначален вариант на настоящата статия и файл в Excel. Взехме решение да публикуваме статията (след обичайната за нас редакция), защото представлява сериозен анализ на проверими факти. Файлът в Excel съдържаше заглавията на всички над 4000 дописки на сайта на БТА, които са предмет на анализа, с датите на публикуването им и разделени по категории. Основата на графиката по-долу също е от въпросния файл.
Евгени Соколов не е истинското име на автора. Уважаваме основанията му да запази своята анонимност и сме убедени, че анализът на фактите, който той представя, е по-важен от името му.
От редакцията
Когато преди няколко години започнах работа в Българската телеграфна агенция (БТА), забелязах, че едно име се появява в нейни статии отново и отново. Не на Бойко Борисов, нито на Румен Радев или Делян Пеевски – обичайните заподозрени, които сме свикнали да доминират в медийния поток. Говоря за името на главния директор на БТА Кирил Вълчев.
Този факт ме накара да се запитам: работата на журналиста не е ли да стои в сянка и да представя събитията, вместо да бъде част от тях? Кой друг журналист се среща толкова често в статии не като автор, а като действащо лице? Със сигурност не шефовете на световни новинарски агенции като Reuters, Associated Press и BBC.
Като се замислих за скорошен пример за журналист, станал център на внимание, се сетих единствено за Джамал Хашоги. Причината той да се превърне в новина обаче беше бруталното покушение срещу него, вероятно наредено от лидера на Саудитска Арабия. През 1978 г. името на журналиста Георги Марков също се оказва сензация, когато той става жертва на прословутия „български чадър“.
Общото между Марков и Хашоги е, че и двамата ги сполита ненавременна смърт, след като с критиките си стават прекалено неудобни за определен тоталитарен режим. Това подсказва, че когато журналист се е превърнал в център на медийното внимание, значи статуквото е фундаментално сбъркано, защото властимащите възприемат отразяването на фактите като заплаха.
Но пък Кирил Вълчев е център на внимание не заради покушение. Може би тогава нямаме повод за притеснения?
Покрай скорошните избори и кандидатурата на Вълчев за вицепрезидент мнозина се запитаха с какво е заслужил доверието на Илияна Йотова, която го издигна за свой заместник. Поставен беше и въпросът не наблюдаваме ли фрапантен конфликт на интереси, когато един и същи човек е кандидат в изборите и шеф на националната новинарска агенция, която отразява неговата кампания и тази на опонентите му. Обяснението на Вълчев беше следното:
Не е конфликт на интереси, понеже не е незаконно. И също така аз съм в отпуск по време на кампанията.
Е, няма проблем, значи – всички знаем, че когато шефът излезе в отпуск, той моментално губи всички телефони на служителите си и влиянието му над тях рязко се занулява.
Бърза справка на официалния сайт на агенцията, управлявана от Вълчев, показва, че той е обиколил „всички 193 държави, които са членове на Организацията на обединените нации“. Този факт накара хора като бившия директор на Висшето училище по мениджмънт във Варна Николай Райчев да обвинят Вълчев в разточителство и безсмислен туризъм на гърба на българския данъкоплатец.
Този грандиозен проект на суетата изисква торби с пари и много свободно време. Хората, които колекционират държави като пощенски марки, го правят само за да могат на всеки коктейл да доминират разговора с репликата: „Ах, навремето, когато бях в Руанда…“
За мен подобни заключения понякога се оказват прибързани. Може пък бъдещият ни вицепрезидент да е един съвременен Алеко, който попива впечатления от всяко ново място и след това превръща натрупаното знание в стойностна литература?
За да тествам тази хипотеза, реших да прегледам (чрез вътрешната система на сайта, защото тя предоставя повече възможности за търсене и обработка на информацията) всички статии, публикувани в bta.bg с името Кирил Вълчев, било то в полето за автор или в самия текст. Статии от първия вид, изглежда, не съществуват, тоест Вълчев не е публикувал нито един текст на сайта на БТА под собственото си име. При статиите от втория вид положението е малко по-различно.
Към 25.10.2026 г. официалният сайт на БТА съдържа 4359 публикации с името Кирил Вълчев в тях. За сравнение, името на Христос се появява в Новия завет малко над 1000 пъти.
В резултат на четенето и категоризирането на всички 4359 текста научих две неща:
Когато в БТА се обнови статия, не се променя първоначално публикуваната, а се добавя нова статия със същото заглавие и нов час на публикуване. Затова понякога една и съща новина бива отразена с две или повече статии с едно и също заглавие.
Някой в БТА (дали не самият директор?) изпитва болезнена нужда името на Вълчев да бъде вкарвано навсякъде. В някои статии неговото присъствие е абсолютно излишно, но въпреки това авторът е намерил повод да го спомене. Но нека не прибързваме.
Посочете сектор с мишката, за да видите какво означаваНатиснете сектор с пръст, за да видите какво означава
Събития, при които за присъствието на Вълчев няма видимо обяснение. Авторът се пита: защо това събитие, защо е отразено и защо е споменат директорът?
„Повече от 90 процента от задържаните при опит да влязат у нас нелегални мигранти през 2021 г. са мъже“
Новини за самата агенция и/или за нейния директор: пресклубове, списание ЛИК, годишнини, поздравителни адреси
„БТА успешно подмени асансьора в западната част на сградата си и подобри достъпа до агенцията“
Събития, организирани от Агенцията или провеждани в нея. Присъствието на директора тук е очаквано.
БТА съобщава, че е сключила договор за обмен на новини с друга институция или за отразяване на лице, институция или събитие.
Публикации, посветени на изказвания на Вълчев
„Светът няма да може да има истински новини без истински журналисти, заяви генералният директор на БТА Кирил Вълчев“
Мисиите на кораба „Св. св. Кирил и Методий“ до българската база на остров Ливингстън
Публикации от 3 септември 2026 г. нататък, когато Илияна Йотова обявява Вълчев за свой кандидат за вицепрезидент
Материали от поредицата „Събитията днес“, които обобщават събития отпреди година или повече или са планирани за близкото бъдеще
Вълчев се изказва за исторически личности и български градове.
„Благоевград е пример в Европа за защита на свободата, каза генералният директор на БТА Кирил Вълчев“
Статии за друг Кирил Вълчев (например експерт в природен парк „Шуменско плато“) или с грешно поставен термин за търсене.
Данни към 25.10.2026 г., след премахване на повтарящите се заглавия (4075 статии)
(function () {
var root = document.getElementById(‘btaPie’);
if (!root || root.dataset.ready) return;
root.dataset.ready = ‘1’;
// Всички текстове (имена, числа, обяснения, примери) се четат от HTML-а в легендата.
var TXT = {
articles: root.getAttribute(‘data-txt-articles’) || ”,
all: root.getAttribute(‘data-txt-all’) || ”,
close: root.getAttribute(‘data-txt-close’) || ”
};
var DATA = [].map.call(root.querySelectorAll(‘.bta-pie__legend > li’), function (li) {
var info = li.querySelector(‘.bta-pie__info-text’), ex = li.querySelector(‘.bta-pie__info-ex’), dot = li.querySelector(‘i’);
return {
name: li.querySelector(‘.bta-pie__name’).textContent.trim(),
n: parseInt(li.querySelector(‘b’).textContent, 10),
color: dot.style.backgroundColor,
label: li.getAttribute(‘data-label’) || ‘#fff’,
text: info ? info.innerHTML : ”,
ex: ex ? ex.innerHTML : ”,
btn: li.querySelector(‘button’)
};
});
var NS = ‘http://www.w3.org/2000/svg’;
var CX = 300, CY = 300, R = 262, TAU = Math.PI * 2;
var svg = root.querySelector(‘.bta-pie__svg’);
var stage = root.querySelector(‘.bta-pie__stage’);
var tip = root.querySelector(‘.bta-pie__tip’);
var total = DATA.reduce(function (s, d) { return s + d.n; }, 0);
var acc = 0;
DATA.forEach(function (d, i) {
d.i = i;
d.pct = d.n / total * 100;
d.a0 = acc / total * TAU;
acc += d.n;
d.a1 = acc / total * TAU;
d.mid = (d.a0 + d.a1) / 2;
d.done = false;
});
function fmtPct(p) { return p.toFixed(1).replace(‘.’, ‘,’) + ‘%’; }
function pt(a, r) { return [CX + r * Math.sin(a), CY – r * Math.cos(a)]; }
function arc(a0, a1) {
if (a1 – a0 Math.PI ? 1 : 0) + ‘ 1 ‘ + p1[0].toFixed(2) + ‘ ‘ + p1[1].toFixed(2) + ‘Z’;
}
function el(name, attrs) {
var e = document.createElementNS(NS, name);
for (var k in attrs) e.setAttribute(k, attrs[k]);
return e;
}
// Static, invisible hit areas on top: they never move, so the pointer cannot flicker between two neighbouring slices
var hitLayer = el(‘g’, {});
svg.appendChild(hitLayer);
DATA.forEach(function (d) {
d.hit = el(‘path’, { d: ”, ‘class’: ‘bta-pie__hit’ });
hitLayer.appendChild(d.hit);
});
// Tooltip
var current = null;
function fill(d) {
tip.innerHTML =
‘‘ +
‘
‘ + d.name + ‘
‘ +
‘
‘ + d.n + ‘
‘ +
‘
‘ + fmtPct(d.pct) + ‘ ‘ + TXT.all + ‘
‘ +
‘
‘ + d.text + ‘
‘ +
(d.ex ? ‘
‘ + d.ex + ‘
‘ : ”);
}
function place(x, y) {
var sw = stage.clientWidth, sh = stage.clientHeight;
var w = tip.offsetWidth, h = tip.offsetHeight;
var left = x + 16, top = y + 16;
if (left + w > sw + 8) left = x – w – 16;
if (left sh) top = y – h – 16;
if (top < 0) top = Math.max(0, Math.min(sh – h, y – h / 2));
tip.style.left = left + 'px';
tip.style.top = top + 'px';
}
function centroid(d) {
var r = svg.getBoundingClientRect(), s = stage.getBoundingClientRect();
var p = pt(d.mid, R * 0.7), k = r.width / 600;
return [r.left – s.left + p[0] * k, r.top – s.top + p[1] * k];
}
function show(d, xy) {
if (current && current !== d) { current.g.classList.remove('is-on'); current.g.style.transform = ''; current.btn.classList.remove('is-on'); }
current = d;
root.classList.add('is-active');
d.g.classList.add('is-on');
d.btn.classList.add('is-on');
var o = pt(d.mid, 22); // push-out along the bisector (SVG units)
d.g.style.transform = 'translate(' + (o[0] – CX).toFixed(1) + 'px,' + (o[1] – CY).toFixed(1) + 'px)';
fill(d);
tip.setAttribute('aria-hidden', 'false');
tip.classList.add('is-open');
place(xy[0], xy[1]);
}
function hide() {
if (!current) return;
current.g.classList.remove('is-on');
current.g.style.transform = '';
current.btn.classList.remove('is-on');
current = null;
root.classList.remove('is-active');
tip.classList.remove('is-open');
tip.setAttribute('aria-hidden', 'true');
}
function local(e) {
var s = stage.getBoundingClientRect();
return [e.clientX – s.left, e.clientY – s.top];
}
DATA.forEach(function (d) {
d.hit.addEventListener('pointerenter', function (e) { if (e.pointerType === 'mouse') show(d, local(e)); });
d.hit.addEventListener('pointermove', function (e) { if (e.pointerType === 'mouse' && current === d) place.apply(null, local(e)); });
d.hit.addEventListener('pointerleave', function (e) { if (e.pointerType === 'mouse' && current === d) hide(); });
d.hit.addEventListener('click', function (e) {
e.stopPropagation();
if (current === d && e.pointerType !== 'mouse') hide(); else show(d, local(e));
});
d.g.addEventListener('focus', function () { show(d, centroid(d)); });
d.g.addEventListener('blur', function () { if (current === d) hide(); });
d.g.addEventListener('keydown', function (e) { if (e.key === 'Escape') hide(); });
d.btn.addEventListener('pointerenter', function (e) { if (e.pointerType === 'mouse' && d.done) show(d, centroid(d)); });
d.btn.addEventListener('pointerleave', function (e) { if (e.pointerType === 'mouse' && current === d) hide(); });
d.btn.addEventListener('click', function (e) {
e.stopPropagation();
if (!d.done) return;
if (current === d && e.pointerType !== 'mouse') hide(); else show(d, centroid(d));
});
d.btn.addEventListener('focus', function () { if (d.done) show(d, centroid(d)); });
d.btn.addEventListener('blur', function () { if (current === d) hide(); });
});
// any click/tap outside a slice or legend item closes the box (slices and legend buttons stop propagation)
document.addEventListener('click', hide);
document.addEventListener('keydown', function (e) { if (e.key === 'Escape') hide(); });
// Animation: slices are drawn one after another, each sweeping clockwise from its own start angle
var STEP = 170, DUR = 700;
function ease(t) { return 1 – Math.pow(1 – t, 3); }
function finish(d) {
d.path.setAttribute('d', arc(d.a0, d.a1));
d.g.style.opacity = '1';
d.hit.setAttribute('d', arc(d.a0, d.a1));
if (d.text_el) d.text_el.style.opacity = '1';
d.done = true;
}
function run() {
var reduce = window.matchMedia && window.matchMedia('(prefers-reduced-motion: reduce)').matches;
if (reduce) { DATA.forEach(finish); return; }
DATA.forEach(function (d) { if (d.text_el) d.text_el.style.transition = 'opacity .3s ease'; });
var t0 = null;
function frame(ts) {
if (t0 === null) t0 = ts;
var el_ = ts – t0, alive = false;
DATA.forEach(function (d) {
if (d.done) return;
var p = (el_ – d.i * STEP) / DUR;
if (p = 1) { finish(d); return; }
alive = true;
d.g.style.opacity = ‘1’;
d.path.setAttribute(‘d’, arc(d.a0, d.a0 + (d.a1 – d.a0) * ease(p)));
});
if (alive) requestAnimationFrame(frame);
}
requestAnimationFrame(frame);
}
var started = false;
function start() { if (started) return; started = true; run(); }
if (‘IntersectionObserver’ in window) {
var io = new IntersectionObserver(function (entries) {
if (entries.some(function (x) { return x.isIntersecting; })) { io.disconnect(); start(); }
}, { threshold: 0.35 });
io.observe(stage);
} else { start(); }
})();
Много бързо отделих първата категория – „Неважно“. В нея включих статиите от поредицата „Събитията днес“, които обобщават събития, станали преди година или повече или пък планирани за близкото бъдеще. В „Неважно“ влязоха 85 текста, или малко над 2% от всички.
Една от първите сериозни категории нарекох „За БТА“.
В категорията „За БТА“ попаднаха 1001 публикации, или близо 25% от общия брой. Огромна част от тях са посветени на пресклубовете на агенцията („БТА официално откри Национален пресклуб в Скопие“), на нейното списание („Списание ЛИК ще има предаване в Българското национално радио“) и на безкрайни поздравителни адреси по повод годишнината на институцията. Поздрави за преизбирането и за рождения ден на нейния директор („Поздравителен адрес от Мариан Бачев, министър на културата“) също влизат в тази категория. Тук човек би могъл да се запита израз на какво е да използваш поверената ти държавна институция, за да разпространяваш честитките за рождения си ден, но нека не се превръщаме в хейтъри.
Следващите две категории – „Организирано от/провеждано в БТА“ (819 статии, 20,1%) и „Договори“ (282 статии, 6,9%), може да се разглеждат и като подкатегории на „За БТА“. От названието на първата се разбира за какво са публикациите. Във втората вкличих всички статии, в които БТА се хвали, че е сключила договор за обмен на новини с друга институция или за отразяване на дадено лице, институция или събитие. Присъствието на Вълчев в тези категории може да се очаква, предвид че той е директорът на БТА.
Категория „Антарктида“ е интересен феномен.
Статиите в нея (210 на брой, 5,2%) отразяват мисиите на кораба „Св. св. Кирил и Методий“, с който нашите антарктици ежегодно пресичат Атлантическия океан, за да стигнат до българската база на остров Ливингстън. На пръв поглед Вълчев не би трябвало да има много общо с тези новини, но списание ЛИК от юни 2023 г. разкрива друга картина. Още като водещ в Дарик радио през 2018 г. Вълчев е използвал трибуната си, за да убеждава политици, че България има нужда от собствен презокеански кораб за антарктическата си мисия.
Инвестирането в наука само по себе си е достойно за уважение. Нека България има собствен кораб. Да видим обаче и един цитат от Вълчев в споменатия брой на списание ЛИК:
Господ беше на наша страна, защото заради Ковид-19 кризата се оказа, че на пазара има кораби, които се продаваха евтино.
Нашият бъдещ вицепрезидент де факто казва: „Слава богу, че светът беше поразен от смъртоносна пандемия, която изби милиони хора, че да може България да спести някой лев.“ Даже да разтълкуваме Вълчевите думи като екстремен национализъм (един вид, „Не ми пука колко хора ще умрат по света, стига България да намаже нещо“), България също беше поразена от вируса. От началото на пандемията до края на 2024 г. от COVID-19 в България са починали 38 855 души според данни от Националния център по заразни и паразитни болести.
Преди да премина към трите ми любими категории, ще спомена останалите две набързо.
Категорията „Президент“ съдържа 129 новини, 3,2% от общия брой. Предвид това, че на 3 септември Йотова обяви Вълчев за свой кандидат-вицепрезидент, сами се досещате, че всички статии тук са публикувани от тази дата нататък. Няма да се спирам върху категорията, защото предизборната кампания зае достатъчен обем от медийното пространство и без моя принос.
„Друг Кирил“ е най-дребната категория – с едва пет материала, или 0,1% от всички статии. Тази мижава петица дава отговор на въпроса „Ти сигурен ли си, че във всички тези 4000+ статии се говори за един и същи Кирил Вълчев?“. Разумен въпрос – все пак не едно магаре се казва Марко. Петте статии са:
Патриот е – пара дава и пътува на изгода, че не свойта пара харчи, а парата на народа!
Благодаря ви, че сте моя муза, г-н Вълчев!
Категорията „Изказвания“
е подобна на „Name-drop“ и съдържа 211 статии (5,2%). Тук намираме всички публикации, посветени на изказвания на Вълчев. В „Изказвания“ има брилянтни умотворения, като:
Тя съдържа 1278 статии, или 31,4%. Всяка от тях ме накара да се питам защо. Защо това събитие е било нужно? Защо някой си е направил труда да го отразява? Защо Кирил Вълчев е присъствал на събитието? И защо БТА е преценила, че неговото присъствие трябва да бъде отразено? Преди да дам конкретни примери, нека още веднъж си припомним, че БТА е държавна информационна агенция, която служи на българския народ – тя не е лична PR агенция, чиято цел е да обслужва интересите на конкретна персона. Или поне така се води на хартия.
Статия в БТА от 22.02.2022 г. е озаглавена „Изложба показва Балканската война през погледа на художниците“. О, каква изненада – Вълчев пак присъства! Защо? Не е куратор на събитието, не е художник. Ветеран от Балканската война също не е, доколкото ми е известно. Може би просто обича да гледа красиви картини, както БТА обича да споменава името му по повод и без повод (предимно без повод).
От първия си ден на поста директор на БТА (27.01.2021 г.) до обявяването, че ще се включи в президентската кампания (03.09.2026 г.), Вълчев е бил начело на Агенцията в продължение на 2045 дни. През този период тя го е включила в 4209 публикации, или средно в по 2,2 публикации на ден. От януари 2021 г. досега българският читател всеки ден е ставал и си е лягал с името Кирил Вълчев.
Когато започнах да работя по този текст, все още имах надежда, че зад думите на Вълчев ще прозират моралът и родолюбието на Алеко. Алеко за жалост не намерих, но пък героят му Бай Ганьо се виждаше навсякъде:
Many teams already run an OpenID Connect (OIDC) identity provider and want their Amazon API Gateway REST APIs to accept its security tokens. Until now, the built-in authorizer for API Gateway REST APIs accepted only Amazon Cognito tokens. For other identity providers, you wrote an AWS Lambda authorizer that downloaded the provider’s signing keys, verified each token, and returned an AWS Identity and Access Management (IAM) policy.
Today, AWS announced JSON Web Token (JWT) authorizers for API Gateway REST APIs, which validate tokens in API Gateway itself. You declare the issuers you trust, the audiences you accept from each, the signing algorithms you permit, and any additional claims you require the token to carry. JWT authorizers work on Regional, edge-optimized, and private REST APIs.
In this post, you learn how JWT authorizers work. You then build a REST API that accepts machine-to-machine tokens from your own identity provider and enforces different scopes on read and write operations. Finally, you add an Amazon Cognito user pool as a second trusted issuer on the same authorizer.
Solution overview
Lambda authorizers remain the right choice when authorization needs custom logic, token introspection, or credentials that are not JWTs. For token validation alone, a JWT authorizer validates the token in API Gateway itself, with no function to maintain and no invocation in the request path, as the following diagram shows.
Figure 1: New architecture. API Gateway validates the token inline and forwards only authorized requests to the integration
Each request goes through the following steps:
The client sends a security token as a bearer token in the header that the identity source names, typically Authorization.
API Gateway matches the token’s iss claim to a configured issuer and rejects a token from an unknown issuer before any network call.
API Gateway checks that the token’s signing algorithm is one you configured, then validates the signature against the issuer’s public keys, which it locates through OIDC discovery and caches.
API Gateway checks the audience and the expiry and not-before times.
If the method requires authorization scopes, you must use an access token that carries at least one of them in its scope or scp claim.
API Gateway checks any additional claim rules you configured for that issuer.
API Gateway passes the validated claims to the integration, and the request proceeds.
API Gateway stops processing the request at the first failed check. API Gateway returns 401 Unauthorized for a missing, malformed, expired, or untrusted token. It returns 403 Forbidden for a valid token that lacks a required scope or fails a claim rule. If API Gateway cannot reach an issuer’s key endpoint and has no cached keys, it returns a 5xx response rather than letting the request through.
Walkthrough
In this walkthrough, you create an orders REST API in which GET /orders requires the orders:read scope and POST /orders requires orders:write.
Prerequisites
To follow this walkthrough, you need:
An AWS account and an IAM principal with permissions to manage API Gateway resources, and Amazon Cognito resources for step 7. Follow the principle of least privilege.
An OIDC-compatible identity provider that supports the OAuth 2.0 client credentials flow and publishes a discovery document at /.well-known/openid-configuration.
In your identity provider, create an API (or resource server) for the orders service with two scopes, orders:read and orders:write. Then create two machine-to-machine applications: a reporting application with only orders:read and a fulfillment application with both scopes.
Then collect the following values, shown here for three common providers.
Value
Okta
Auth0
Microsoft Entra ID
ISSUER
https://{yourOktaDomain}/oauth2/default for the default custom authorization server
https://{yourTenant}.auth0.com/, including the trailing slash. Note that URL might include country code.
https://login.microsoftonline.com/{tenantId}/v2.0
AUDIENCE
The authorization server’s Audience, for example api://default
The API Identifier
The application ID URI or client ID of the API app registration
TOKEN_URL
{ISSUER}/v1/token
https://{yourTenant}.auth0.com/oauth/tokenNote that URL might include country code.
Permissions on the API, granted to each application
See Considerations. Client credentials tokens carry app roles, not scopes.
For the example walkthrough, set the values as environment variables (production clients can load client secrets from AWS Secrets Manager or an equivalent store).
REGION=<your AWS Region, for example us-east-1>
ISSUER=<issuer URL, exactly as it appears in the token's iss claim>
AUDIENCE=<audience value>
TOKEN_URL=<token endpoint>
READER_ID=<reporting application client ID>
READER_SECRET=<reporting application client secret>
WRITER_ID=<fulfillment application client ID>
WRITER_SECRET=<fulfillment application client secret>
The issuer must match the token’s iss claim exactly, including any trailing slash, so copy it from a decoded token.
Step 2: Create the JWT authorizer
Create a REST API, then describe the authorizer in a file that lists the trusted issuers, the audiences accepted from each, and the signing algorithms you configure on your issuer:
When you create or update the authorizer, API Gateway reads each issuer’s discovery document and checks that its issuer value matches. If your provider signs with an elliptic curve key, add ES256 or another supported algorithm to defaultAlgorithms. Keep jwt-config.json for step 7.
Step 3: Attach the authorizer to methods with different scopes
Create the /orders resource and two methods that share the authorizer but require different scopes. A mock integration returns the issuer and subject from the validated token:
The request returns 401, because API Gateway rejects it before the integration runs.
Next, use a token for the reporting application, which has only orders:read, to create an order. The get_token helper sends an OAuth 2.0 client credentials request:
An authorizer can trust up to 10 issuers, for example while you migrate between identity providers. In this step, you add a Cognito user pool to the same authorizer.
Create a user pool, a domain for its token endpoint, a resource server with the two scopes, and one machine-to-machine client:
Add the user pool to jwt-config.json and send the full file back, because an update replaces the whole configuration. Cognito access tokens carry the app client ID in client_id rather than aud, so use the client ID as the audience:
The following screenshot shows the updated authorizer in the API Gateway console, with the key set that API Gateway resolved for each issuer.
Figure 2: JWT authorizer with two trusted issuers
If you no longer have the jwt-config.json file, rebuild it from get-authorizer and remove the jwksUri fields, because API Gateway rejects an update that includes them:
Cognito names scopes {resource server}/{scope}, so its token carries orders/orders:read rather than orders:read. Add that form to GET /orders, then redeploy, because authorizer and method changes take effect only in a new deployment:
The request returns 200 with the Cognito issuer, and tokens from your original provider keep working. For Cognito issuers, a method with scopes accepts only access tokens and a method without scopes accepts only ID tokens. Give each method that client credentials callers use at least one scope.
To finish a migration, remove the retired issuer from jwt-config.json, send the same replace operation (using aws apigateway update-authorizer --patch-operations command), and redeploy.
Optional: Require an additional claim
To accept only tokens whose department claim equals eng, add a claim rule to the issuer entry. Matching is exact and case-sensitive, and API Gateway returns a 403 for other tokens:
Defining JWT authorizers as infrastructure as code
In AWS CloudFormation, the AWS::ApiGateway::Authorizer resource has a JwtConfiguration property that mirrors the API, and methods set AuthorizationType: JWT and AuthorizationScopes:
You can also convert an existing Lambda or Cognito authorizer in place with one update-authorizer command that changes /type, replaces /jwtConfiguration, and clears the old type’s fields. Clear Lambda fields with a replace that has no value, and remove each Cognito provider ARN with remove. Then switch each method that uses the authorizer to authorizationType: JWT, add its scopes, and redeploy.
Using claims in your integration
API Gateway passes every JWT claim to your backend as JSON as it appeared in the token. Strings stay quoted and escaped, numbers stay numbers, and arrays and objects stay intact, so your backend never has to guess what was sent. Mapping templates read validated claims as $context.authorizer.claims.name, as with Cognito authorizers, and Lambda proxy integrations receive them in event.requestContext.authorizer.claims.
HTTP proxy integrations don’t forward claims automatically, so map the ones you need to headers, for example integration.request.header.x-caller to context.authorizer.claims.sub.
Observability and troubleshooting
When API Gateway rejects a request, the response body is Unauthorized or The incoming token has expired for a 401, and Access Denied for a 403. A WWW-Authenticate header that follows RFC 6750 carries the details, such as the failed check or, for a missing scope, the scopes the method accepts.
With execution logging on, API Gateway logs a claims summary for each request, without the raw token. Data trace logging masks the Authorization header but not custom headers, so if your identity source is a header such as X-Token, keep data trace logging off for that stage.
To test an authorizer before you deploy, choose Test authorizer on the authorizer page or run test-invoke-authorizer. The test skips method scopes and accepts Cognito identity tokens, so call the deployed method to test scopes.
Considerations
Before you add a JWT authorizer, consider the following:
An authorizer trusts 1–10 issuers. Each issuer accepts 1–100 audiences and up to 10 additional claim rules, each with at most one expected value.
Supported signing algorithms are RS256, RS384, RS512, ES256, ES384, and ES512, and API Gateway rejects HMAC and alg: none tokens. The token header must carry a kid, which some providers omit by default.
API Gateway allows 60 seconds of clock skew when it checks exp, nbf, and iat.
API Gateway caches each issuer’s signing keys for about 60 minutes, so a key your provider removes can still verify tokens for that long. If the key endpoint is unavailable, API Gateway keeps using the last key set for up to 1 hour.
A token passes a method’s scope check when its scope or scp claim carries one of the method’s scopes. Microsoft Entra ID client credentials tokens carry permissions in roles instead, so use a dedicated audience per caller or a Lambda authorizer for role checks.
An authorizer reads the token from a single request header.
JWT authorizers don’t check revocation, so a token stays valid until it expires. Use short token lifetimes for sensitive operations.
The issuer URL must be publicly accessible.
Clean up
To avoid incurring future charges, delete the resources you created:
In your identity provider, delete the two applications and the orders API or resource server if you no longer need them.
Conclusion
With JWT authorizers, you can validate tokens from OIDC-compatible identity providers in API Gateway REST APIs without a function in the request path. In this post, you built an API that accepted machine-to-machine tokens from your own provider, enforced separate read and write scopes, and added a Cognito user pool as a second issuer. Lambda authorizers remain the right tool for custom authorization logic. For token validation alone, we recommend the JWT authorizer.
JWT authorizers for REST APIs are available in all commercial AWS Regions, the AWS GovCloud (US) Regions, and the China Regions. To get started, refer to Controlling access to a REST API using JWT authorizers in the Amazon API Gateway Developer Guide. To learn more about serverless architectures, visit Serverless Land.
With biopharma commercial teams facing increasingly complex big data challenges, Veeva used Amazon EMR on EKS to increase speed and efficiency in delivering performance data to biopharma customers, supporting insight generation to find and treat more patients. Therapies designated as “orphan” in the United States that treat rare diseases affecting fewer than 200,000 Americans made up half of U.S. novel drug approvals in 2025, and the continuing shift toward developing therapies for rare and complex disease states only amplifies the need for scalable analytics infrastructure.
Veeva Systems, a leader in cloud-based software for the global life sciences industry, addresses this changing data landscape with Veeva Compass and the Compass Data Network. Compass unifies prescriptions, in-office procedures, and diagnoses into a suite of data products customers use to segment economies with greater precision, prioritize the right healthcare providers, and bring the right therapies to more patients, faster.
Veeva is continuously evaluating and adding new data partners to the Compass network, using both closed and open sources to fill the gaps left by traditional patient data products and meet the biopharma industry’s current and future data needs. To support this ever-growing network, the company undertook an ambitious architecture transformation with Amazon EMR on EKS. Veeva Compass uses faster, more efficient compute to deliver data from suppliers to customers within hours while drastically increasing efficiency by accelerating analytics and supporting AI workflows. This system flags new patient and healthcare provider (HCP) events earlier so that medical reps can engage with healthcare providers at the right time with more precise information.
This post explores the AWS analytics architecture that powers the petabyte-scale elastic big data processing Veeva Compass implemented with Amazon EMR on EKS. Specifically, it explores how Compass avoided execution slowdowns while continuing to increase data volumes to meet customer needs.
The challenges of extracting data insights at petabyte scale
The foundation of Veeva Compass is an analytics lake that was originally designed to process hundreds of terabytes daily. Through years of consistent growth fueled by global customer expansion and the growing gap between legacy data products and industry needs, the Veeva Compass analytics lake grew. It increased support for the number of data sources and Veeva products. This resulted in an exponential growth of data volumes. With the quantity and size of batch jobs becoming more variable, Veeva decided to revisit the original Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2) architecture.
The batch job variability challenges manifested in a few important ways. Whenever Spark executors ran out of memory, the shuffle stages in Spark were unable to compensate, and executor tasks would spill to disk. This would add hours to overall job execution time with executors relying on disk storage to complete their tasks. A significant portion of the analytics budget was allocated to several provisioned Amazon EMR clusters that were idle during the afternoon waiting for the next series of job runs. To better use these clusters, Veeva distributed jobs across resources. But that led to noisy neighbor challenges where one team’s large join operation would monopolize the cluster and block another team’s service level agreement (SLA)-critical pipeline.
Veeva needed a more scalable solution that could scale out to hundreds of Spark executors in minutes when jobs arrived, then scale in when jobs complete, delivering both performance and cost efficiency. Veeva needed to rethink how it provisioned compute resources and managed Spark workloads.
The solution: Amazon EMR on EKS with Karpenter
Let’s explore how Veeva used Amazon EMR on EKS with Karpenter to scale its analytics capabilities and reduce Spark run times. Amazon EMR on EKS runs each Spark application as an ephemeral job inside Kubernetes, decoupling compute from storage. Data lives in Amazon Simple Storage Service (Amazon S3), durable and accessible. Spark executors run as pods that are created and terminated with each job. With Amazon EMR on EKS, Veeva is no longer dependent on always-on clusters and consumes compute only when needed.
The diversity of data types in the analytics workloads requires fine-grained control to provision the optimal compute nodes for each job. Veeva’s original cluster design relied on pre-configured node groups, where you define instance types and sizes upfront. This approach worked for other Veeva workload profiles, but Spark workloads are different. One job might need memory-optimized instances for wide joins, while another needs compute-optimized nodes for CPU-intensive transformations.
To address this challenge, Veeva added real-time EC2 instance type provisioning using Karpenter. For example, a Spark job might request 800 executors with 16 GB memory and 4 cores each. Karpenter reacts to pending pod requests in 60–90 seconds, analyzes the requirements, selects optimal instance types from across over 20 families, and launches nodes. When the job is complete, those nodes terminate within minutes. Karpenter also handles bin-packing, which is a placement strategy where pods are consolidated into fewer EKS nodes to optimize cost and improve resource utilization.
Amazon EMR on EKS with Karpenter architecture overview
Veeva’s analytics architecture follows a medallion data design pattern. This pattern progressively refines data in stages: bronze (raw), silver (cleaned), and gold (aggregated) layers within the same storage system.
Figure 1: Amazon EMR on EKS architecture spanning the storage, compute, orchestration, and query layers
The architecture comprises four layers:
Storage layer: Amazon S3 stores data across bronze (raw), silver (cleaned), and gold (business-ready aggregated) layers. S3 organizes the data in Delta Lake format with daily partitions and continuous compaction to help prevent small-file proliferation.
Compute layer: Amazon EMR on EKS runs Spark applications as Kubernetes jobs. Each application gets its own isolated set of pods that exist only for the duration of that job.
Orchestration layer: Karpenter watches for pending Spark executor pods and provisions EC2 nodes to run them. It selects instance types based on pod resource requests, Availability Zone (AZ) capacity, and Spot pricing, then terminates nodes when they’re no longer needed.
Query layer: An Amazon EMR on EKS cluster running Trino (formerly PrestoSQL) supports interactive queries against the transformed Delta tables, so analysts can explore petabyte-scale datasets with sub-second response times.
This separation of concerns delivers the elasticity required for petabyte-scale operations.
Spark application layer instance types
Veeva selected three distinct node profiles optimized for different Spark workload characteristics:
Driver nodes: Spark drivers and Airflow workers run on On-Demand Instances for reliability. Drivers coordinate job execution and can’t tolerate interruptions because of Spot instance reclamations. These nodes run continuously but represent less than 5 percent of total compute cost.
Executor nodes: General Extract, Transform, Load (ETL) and memory-intensive joins run on approximately 70 percent Spot Instances. Karpenter spreads across multiple instance families to maximize Spot availability. When interruptions occur, Spark’s checkpointing and Adaptive Query Execution handle recovery automatically.
Shuffle-heavy nodes: Workloads with high shuffle I/O run on NVMe-backed instances. The local SSD storage improves performance when executors spill to disk during wide joins. These run at approximately 50 percent Spot because the NVMe instance families have slightly lower Spot availability.
Partition strategy and tuning for large datasets
With the compute and orchestration layers optimized, Veeva turned its attention to improving the partition strategy, data layout, and Spark configuration.
Partition strategy: Veeva’s data is partitioned by date (YYYY-MM-DD) plus a high-cardinality dimension like region or customer segment. This simplifies partition pruning while avoiding the small-file problems that degrade query performance. Target file size is around 256–512 MB per Parquet file. Nightly compaction jobs use Delta Lake’s OPTIMIZE command with Z-ordering on frequently filtered columns, which improved query performance by 3–5x.
Shuffle and query tuning: Using Adaptive Query Execution (AQE) as a query optimization capability transformed Veeva’s most problematic Spark workload. AQE allows Spark to dynamically adjust the query plan based on runtime statistics rather than relying on static estimates. To enable these dynamic adjustments, enable AQE, configure initial partition counts for wide joins, handle data skew automatically, and enable speculative execution to recover from stragglers.
Additionally, Veeva caches some of the dimension tables in memory to avoid repeated reads from S3. For jobs that reference the same lookup tables multiple times, this optimization reduced runtime by 20–30 percent.
The combination of proper partitioning, continuous compaction, and AQE tuning reduced the skewed join stage run time from 7 hours to 55 minutes.
Observability as the foundational component for Amazon EMR on EKS operations
An observability architecture that monitors the entire stack and delivers operational insights is essential for maintaining SLAs and troubleshooting performance issues before they impact production pipelines.
The observability stack has four layers:
Spark History Server runs on EKS with S3-backed event logs. With Spark History Server, you can replay the entire execution graph, examine stage-level metrics, and identify bottlenecks.
Prometheus and Grafana track real-time metrics. Key metrics include executor CPU and memory utilization, pending-pod queue depth, shuffle-spill ratios, and Karpenter provisioning latency.
Amazon CloudWatch Logs Insights aggregates driver and executor logs with automated alerting. When a job fails, Amazon CloudWatch sends an immediate notification with stack traces and error context.
OpenTelemetry traces link Spark jobs to the broader orchestration layer (Airflow, AWS Step Functions). This end-to-end visibility provides insights into how individual job performance impacts downstream dependencies.
To help proactively identify bottlenecks before they cascade into SLA violations, Veeva configured critical alerts such as pending pods exceeding thresholds indicating capacity constraints, high shuffle-spill ratios signaling memory pressure, and excessive repartitioning indicating significant data skew.
Securing multi-tenant data analytics
Datasets that include prescription and claims data require robust security controls. Veeva’s architecture follows AWS best practices for least-privilege access and strict tenant isolation.
IAM Roles for Service Accounts (IRSA) provide each Spark job with temporary credentials scoped to specific S3 prefixes and AWS Lake Formation permissions. Jobs can read from bronze and silver zones but only write to their designated output paths. Using Kubernetes namespaces with taints and tolerations clearly defines workload placement. Each team operates in their own namespace with dedicated service accounts and resource quotas.
Encryption is enforced at rest with AWS Key Management Service (AWS KMS) and in transit using TLS. Amazon S3 uses customer-managed keys to encrypt all data. Spark shuffle data is encrypted using Spark’s built-in encryption. Lineage and auditing through AWS Lake Formation and OpenLineage track data transformations across the entire pipeline.
Measurable improvements
The migration from Amazon EMR on EC2 to Amazon EMR on EKS with Karpenter produced measurable improvements across every key metric:
Metric
Amazon EMR on EC2
Amazon EMR on EKS + Karpenter
Improvement
Bronze→Silver ETL (1 PB)
9h 12m
4h 07m
55% faster
Daily 20 TB compaction
3h 40m
1h 55m
48% faster
Scale-out 40→520 nodes
18 min
5m 42s
68% faster
Idle EC2 cost/month
$42,000.00
$0.00
100% reduction
Spot instance share
0%
~70%
~63% cost savings
Bin-packing failures
Frequent
Near-zero
—
The largest improvement came from scale-to-zero capability. The previous Amazon EMR cluster design ran 24/7 to avoid the 15–20-minute startup time, costing $42,000 monthly in idle compute. With Karpenter, nodes launch in under 6 minutes and terminate automatically, alleviating idle costs. This reduced launch time matters when you have SLA-critical pipelines that need to start processing immediately. For the largest bronze-to-silver ETL job processing approximately 1 PB of data, Karpenter scaled the cluster from 40 to 520 nodes in 5 minutes 42 seconds. It then spun everything down minutes after completion. That level of elasticity is difficult to achieve with traditional automated scaling approaches. By diversifying across over 20 Spot instance types, Karpenter achieves higher Spot availability and lower interruption rates. The executor fleet runs approximately 70 percent Spot, delivering approximately 63 percent cost savings compared to On-Demand pricing.
For Veeva Compass’s business, the speed improvements are as impactful as the cost savings. Several of their most common customer use cases are highly dependent on data delivery speed. For example, medical field reps at a biopharma company rely on alerts from patient data to guide their engagement strategies day to day. The faster Veeva can deliver a new diagnosis or procedure to a customer, the faster that customer’s field team can act.
Lessons learned along the implementation journey
After a year of operating petabyte-scale analytics on Amazon EMR on EKS, here are the lessons that delivered the most value:
Partition hygiene matters more than hardware. Veeva spent weeks tuning executor memory and instance types before realizing the real bottleneck was small-file proliferation. Implementing nightly compaction reduced job runtime by 40 percent without changing cluster configuration.
Compaction must be frequent. Waiting for monthly compaction windows creates trillions of tiny Parquet files that degrade query performance. Daily compaction on hot partitions and weekly compaction on cold partitions keeps file counts manageable.
Design for Spot interruptions. Assume 5–10 percent hourly churn in your executor fleet. Use Spark checkpointing for long-running jobs, enable AQE for query optimization, and rely on Spark retries and re-computation to handle executor failures.
Synthetic benchmarking pays off. Before migrating production workloads, Veeva ran synthetic tests at 1 PB, 2 PB, and 5 PB scale. This exposed Garbage Collection (GC) pressure and shuffle limits that would have caused production incidents.
Speculative execution saves hours. For skew-heavy wide joins, enabling speculative execution allows Spark to launch backup tasks for stragglers. This single configuration change reduced the skewed join stage run time from 7 hours to 55 minutes.
Karpenter consolidation compounds savings. Beyond Spot pricing, Karpenter’s bin-packing and consolidation features deliver another 8–12 percent EC2 savings.
Conclusion
Veeva Compass’s journey from Amazon EMR on EC2 to Amazon EMR on EKS with Karpenter demonstrates how modern cloud-native architectures can transform petabyte-scale analytics workloads. By embracing ephemeral compute, intelligent autoscaling, and continuous optimization, Veeva achieved measurable improvements in both performance and cost efficiency. They reduced critical ETL runtimes by over 50 percent while alleviating $42,000 in monthly idle compute costs.
These cost and performance improvements are important for Veeva Compass’s business practice. The ability to quickly serve large amounts of data to biopharma commercial teams enables faster and more precise analytics, especially as biopharma companies continue to drive increased data demand by exploring AI use cases. These improvements help Veeva Compass customers act with speed and precision to quickly deliver the right therapies to patients in need.
If you’re facing similar challenges with growing data volumes and complex Spark workloads, consider exploring Amazon EMR on EKS and Karpenter as a foundation for elastic, cost-effective big data processing. To learn more about building scalable analytics on AWS, visit the Amazon EMR documentation and explore Karpenter best practices.
On November 17, 2026, public reports described CVE-2026-114938 as a memory corruption vulnerability in Java 17 and above that could allow a remote attacker to execute code without authentication. The reported severity is Critical, with a CVSSv4 score of 9.2. These CVE-specific details remain preliminary, pending confirmation in Oracle or OpenJDK advisories.
Rapid7 is actively monitoring the situation and will update this blog as new information arrives.
Exposure Command, Vulnerability Management, and Nexpose
Exposure Command, Vulnerability Management, and Nexpose customers can assess exposure to CVE-2026-114938 with an authenticated vulnerability check available in the November 17 content release (1.1.4115).
When the agent can’t reach data you need during an incident investigation, you extend it with new tools. You also manage the security and observability overhead of each tool you add. As you connect more Model Context Protocol (MCP) tools to AWS DevOps Agent, your security team ends up tracking a separate authentication scheme, credential rotation schedule, and audit or revocation procedure for each one. This post shows how to extend AWS DevOps Agent with MCP tools securely, using Amazon Bedrock AgentCore, so that new tools can inherit consistent security, access control, private connectivity, and observability.
AWS DevOps Agent is an AI-powered, always-available operations teammate that autonomously investigates incidents, identifies root causes, and prevents future issues across AWS, multicloud, and on-premises environments. It correlates Amazon CloudWatch metrics, logs, traces, and deployment history to produce root-cause analyses in minutes instead of hours. In Announcing General Availability of AWS DevOps Agent, AWS reported results from customers and partners who used AWS DevOps Agent in preview. They saw up to 75 percent lower mean time to resolution (MTTR), 80 percent faster investigations, and 94 percent root-cause accuracy.
By default, AWS DevOps Agent integrates with a broad set of observability, source control, continuous integration and continuous delivery (CI/CD), ticketing, and collaboration services. Real-world operational environments often extend further: proprietary configuration management databases (CMDBs), internal runbook APIs, application logs in OpenSearch, custom deployment tracking systems, or self-hosted tools behind private networks. When you need data from a source without a native integration, you can use MCP tools to give the agent a standardized way to call external services during an investigation.
You will learn when an MCP tool is the right answer, where to host it, how to connect it to AWS DevOps Agent, and how AgentCore Gateway, a capability of Amazon Bedrock AgentCore, provides a unified management layer for security, policy, and observability.
Where MCP tools fit in the incident lifecycle
Before examining tool infrastructure, it helps to see where MCP tools enter the picture. When your Amazon CloudWatch alarms, ServiceNow tickets, or PagerDuty alerts trigger an incident, AWS DevOps Agent begins an autonomous investigation. The following diagram shows where MCP tools extend that investigation beyond the agent’s native integrations.
Figure 1: MCP tools extend the investigation reach of AWS DevOps Agent beyond its native and built-in integrations
These tools can connect to internal operational tools, third-party platforms, on-premises systems, or any custom data source you want to expose to an agent space. AWS DevOps Agent provides logging and the ability to disconnect any MCP server from the console, but you configure authentication, rotate credentials, and maintain the audit trail for each server. As the number of MCP servers grows, managing them independently creates compounding overhead: separate credential lifecycles, no cross-tool policy enforcement, and no unified view for observability across all tools.
AgentCore Gateway helps reduce this per-tool overhead. The agent connects to one endpoint, and tools behind that endpoint share the same security, policy, and observability configuration.
Use this decision framework to choose between a native integration, an existing MCP server, and building a custom MCP server when you need a new data source or workflow.
Figure 2: Decision framework for extending AWS DevOps Agent: native integration, existing MCP server, or custom MCP server
Start with what AWS DevOps Agent already covers. Before building anything, check the native capability list on the Capability Providers page. These integrations need no custom server to build or host. AWS DevOps Agent Tools catalogs additional MCP servers built for use with the agent. It is also worth browsing the AWS DevOps Agent ACP/MCP samples repository for reference implementations, and awslabs/mcp for AWS-published open source MCP servers such as the Amazon Managed Service for Prometheus MCP server and the Amazon Elastic Kubernetes Service (Amazon EKS) MCP server.
If a source is not natively covered, check whether a vendor already publishes an MCP server before building your own. Many do. For example, Atlassian’s Rovo MCP Server covers Jira, Confluence, Jira Service Management, and Bitbucket, and vendors such as Datadog and PagerDuty publish official servers.
Build a custom MCP tool when your source does not fall into any of these categories. The most common scenarios are:
Logs that need domain-specific parsing. AWS DevOps Agent can read log files in Amazon Simple Storage Service (Amazon S3) and query them with Amazon Athena once you grant it access. To do this, add the S3 read and Athena query permissions to its role as an inline policy, scoped to the buckets it needs. For more information, see Extending permissions beyond the default. For high-volume formats such as Application Load Balancer (ALB) access logs and VPC Flow Logs, a custom tool is often the better fit. An AWS Lambda-based MCP tool that parses and pre-filters these logs gives the agent the context to correlate a 502 spike with a specific backend failure. Application logs in Amazon OpenSearch Service follow the same pattern, because they need domain-specific language (DSL) query logic rather than raw document retrieval. For reference patterns, see Six Patterns for Connecting Amazon OpenSearch Service Logs to AWS DevOps Agent.
Internal operational tools. Runbook APIs, proprietary CMDBs, service ownership registries, and custom deployment tracking systems are not reachable through any native integration. An MCP tool wraps the internal API and exposes it as a callable tool during investigations.
On-premises or other-cloud telemetry. Data from on-premises monitoring systems or GCP infrastructure that is not captured in native AWS services requires tools that can reach those environments. For network-specific patterns, see Automated network incident response with AWS DevOps Agent.
AgentCore Gateway also provides one-click integrations for tools such as Jira, Salesforce, Slack, Asana, and Zendesk, as well as native support for Amazon Bedrock Knowledge Bases as a Gateway target. If your tool is available as a one-click target, you can skip building a custom MCP server and register it directly on the Gateway, which you then connect to AWS DevOps Agent as an MCP server through the agent’s capabilities.
Where to host your custom MCP server
After you decide to build a custom MCP tool, you need to choose where it runs. The right choice depends on statefulness, latency tolerance, connection requirements, and how much infrastructure you want to manage. The following table compares the four most common hosting options.
Hosting option
Best for
Advantages
Trade-offs
AWS Lambda (direct IAM invoke)
Stateless, read-heavy tools that execute quickly, such as Amazon S3 log parsers, OpenSearch query wrappers, runbook search, and internal API or ticket lookups
No infrastructure to manage, pay-per-invocation, and auto-scales to zero. AgentCore Gateway invokes Lambda directly through IAM, without needing the HTTP layer, which removes Amazon API Gateway latency for Gateway-connected tools
15-minute execution limit, cold-start latency on first invoke, and no persistent connections
AWS Lambda with Amazon API Gateway
Shared tools used by multiple teams or systems, and tools that require request throttling or AWS WAF protection
All Lambda benefits, plus HTTP-level controls such as throttling, WAF protection, custom domains, and per-caller access logging
Added per-request latency, Amazon API Gateway per-request cost, and more configuration
Amazon Bedrock AgentCore runtime
Teams who want zero-ops MCP hosting with native multi-turn session support
No infrastructure to manage, built-in session isolation, native MCP protocol support, both stateless and stateful modes, and support for your own container image. Works with any MCP framework, including LangGraph, CrewAI, and Strands Agents
Cold-start latency when a new session starts, and less control over the underlying compute than Amazon Elastic Container Service (Amazon ECS)
Amazon ECS on AWS Fargate
Database query tools that maintain connection pools, and tools with expensive initialization or large in-memory caches
No cold starts, persistent connections, and full control over the runtime environment
Always-on cost even when idle, and container image management overhead
As a rule of thumb, start with Lambda for most read-only tools, move to AgentCore runtime, a capability of Amazon Bedrock AgentCore, when you need multi-turn sessions or zero-ops hosting, and use Amazon ECS when you need persistent connections or large in-memory state.
Connecting MCP servers to AWS DevOps Agent
There are two ways to connect MCP servers to AWS DevOps Agent. The right choice depends on how many tools you have, whether tool access needs to be consistent across multiple accounts, and whether you need central policy enforcement. The following diagram compares a direct per-account connection with a single connection through AgentCore Gateway.
Figure 3: Connecting MCP servers to AWS DevOps Agent: direct connection compared to AgentCore Gateway
Path 1: Direct connection
With a direct connection, you register each MCP server at the AWS account level from the Capability Providers page. To register a server:
On the Capability Providers page, register the MCP server once with its endpoint URL and authentication credentials.
In each agent space’s Capabilities tab, select the registered MCP server and choose which specific tools to enable for that space.
Classify each tool as read-only, mutating, or destructive so that AWS DevOps Agent applies the correct safety controls during investigations.
AWS DevOps Agent supports private connections natively. When you register an MCP server, select Connect to endpoint using private connection and choose or create a private connection. AWS DevOps Agent uses Amazon VPC Lattice to route traffic to your private endpoint entirely over the AWS network, with no public endpoint needed. Private connections work with any integration that needs to reach a private endpoint, including self-hosted Grafana or Splunk instances and source control systems such as GitHub Enterprise Server and GitLab Self-Managed. For a detailed walkthrough, see Securely connect AWS DevOps Agent to private services in your VPCs.
Tool safety with directed actions
When you connect an MCP server, you classify each of its tools as read-only, mutating, or destructive. The console prompts you for each tool, and API or CLI registrations set this with toolDetails. Read-only tools run without operator approval. Mutating tools are available only after you enable directed actions on the agent space, and each call requires operator approval in chat. AWS DevOps Agent does not invoke tools classified as destructive. If you register a server through the API or CLI without classifying its tools, every tool defaults to read-only, so classify any tool that can change state. For more information, see Working with directed actions.
Limit prompt-injection exposure
Custom MCP servers can introduce additional prompt-injection risk. Allow only the tools each agent space needs, prefer read-only tools, and give each server’s credentials read-only access. For more information, see the security considerations in Connecting MCP Servers.
Path 2: Amazon Bedrock AgentCore Gateway
AgentCore Gateway is a fully managed service that provides a single, secure entry point for agentic traffic, connecting agents to tools, to other agents, and to models. Instead of registering each MCP server independently in the AWS DevOps Agent account, you register all of them as targets on the Gateway. AWS DevOps Agent registers the Gateway URL once at the account level, and all tools from all registered targets become available through that single registration.
Gateway also consolidates authentication. AWS DevOps Agent connects to a single Gateway endpoint with AWS SigV4. It signs requests with either a dedicated IAM role or, if you register without one, the IAM role of the AWS account associated with the agent space. Grant that role bedrock-agentcore:InvokeGateway, scoped to the Gateway ARN. You still configure per-target credentials on the Gateway, because each backend MCP server has its own authentication. But the Gateway manages those credentials centrally, handling token refresh, secure storage, and credential lifecycle in one place rather than scattering them across individual agent space registrations.
With Policy in Amazon Bedrock AgentCore, you can enforce Cedar rules across every tool behind the Gateway with a single policy, for example blocking any tool that writes to production resources regardless of which MCP server exposes it. Because AWS DevOps Agent already refuses tools classified as destructive, Gateway policy is most valuable as an independent backstop in case someone misclassifies a tool.
Gateway also provides a unified audit trail and built-in observability automatically, with no instrumentation code and no per-target logging configuration. Amazon CloudWatch Logs records every tool call, including tool name, caller identity, request arguments, response, latency, and status, across all registered targets in one log group. AWS X-Ray provides end-to-end distributed traces across the MCP client, Gateway, and Lambda or runtime layers. The CloudWatch generative AI observability dashboard reports per-tool metrics, including invocation count, error rate, and latency percentiles (p50, p90, and p99). These metrics appear regardless of whether the tool runs on Lambda, Amazon ECS, AgentCore runtime, or a third-party server.
Gateway can also route to private MCP servers through VPC Lattice egress. If you need private connectivity together with the authentication, policy, and observability capabilities of the Gateway, register the tool as a private Gateway target. For connectivity patterns, see Private connectivity patterns for AgentCore Gateway Targets. If you only need private connectivity for a single tool without central policy, a direct private connection requires less configuration.
You can also route native MCP integrations through Gateway. Some AWS DevOps Agent native capabilities, such as the Grafana integration, connect to external MCP servers (Grafana’s own MCP server, in this case). If you want unified policy enforcement, observability, and authentication across both your native MCP-based integrations and your custom MCP servers, you can route them through AgentCore Gateway as well. This gives you a single control plane regardless of whether the tool came from the Capability Providers page or a custom registration.
Use a direct connection when you have one or two MCP servers in a single AWS account, when each agent space configures its own tool set manually, and when you do not need cross-cutting policy enforcement. Choose AgentCore Gateway when you have multiple MCP servers and want to manage them through a single registration, when agent spaces across multiple AWS accounts need to share the same tools, when you want one authentication configuration, when you need Cedar policy enforcement across all tools, or when you want a unified audit trail and built-in observability. For full instructions, see Connecting MCP Servers in the AWS DevOps Agent User Guide.
Connectivity options at a glance
This table maps common scenarios to the recommended connection approach.
Scenario
Recommended approach
One or two MCP servers, single account
Direct connection
Multiple MCP servers, single registration preferred
AgentCore Gateway
Agent spaces across multiple AWS accounts sharing the same tools
AgentCore Gateway
Private VPC tool, no central policy needed
Direct connection with DevOps Agent private connection
Private VPC tool with Cedar policy or unified auth
Gateway with VPC Lattice target
Cedar policy enforcement across all tools
AgentCore Gateway
Unified audit trail across all MCP servers
AgentCore Gateway
Managing tool access with Gateway targets and policy
Beyond choosing a connection path, Gateway gives you finer control over which tools an agent space can reach.
Keep the per-space tool ceiling in mind as you scope access: a single agent space can have at most 500 MCP tools on the allowlist across all of its MCP servers, and this limit is not adjustable. Scoping each space to only the tools it needs keeps you well under that ceiling. For the full list of limits, see AWS DevOps Agent quotas.
When AWS DevOps Agent connects to an MCP server, its tools appear on the agent space console, where an admin selects which tools to allow and classifies each one as read-only, mutating, or destructive. This workflow exists with or without Gateway.
Gateway adds a mechanism on top: separate targets per tool category. Register read-only tools and write or destructive tools as separate Gateway targets, then associate AWS DevOps Agent with only the read-only target. The write tools don’t appear in that agent space, because they are not in the target. For how to register targets, see Add targets to an existing AgentCore Gateway.
For example, if an MCP server exposes list_runbooks, get_runbook, search_runbooks, upload_runbook, and delete_runbook, register a target named readonly-runbook-tools with only the first three. Connect the production agent space to the readonly-runbook-tools target, and connect development and staging agent spaces to a target that exposes all five tools.
Even when tools are visible and enabled in an agent space, Gateway policy, which uses Cedar, can block execution. This is the enforcement boundary below the admin workflow, and it holds regardless of what an admin configures on the console, which helps guard against configuration mistakes. For more on Cedar policy patterns, see Secure AI agents with Policy in Amazon Bedrock AgentCore.
Conclusion
AWS DevOps Agent covers a broad set of AWS services and third-party tools by default. When investigations need to reach further, into OpenSearch, internal runbook APIs, proprietary CMDBs, or tools in private VPCs, MCP tools extend that reach.
For one or two tools in a single account, a direct connection with account-level registration is sufficient. The picture changes when MCP servers multiply, when agent spaces span multiple AWS accounts, or when security teams need consistent enforcement and a unified audit trail. AgentCore Gateway collapses that to one registration per account pointing to a single Gateway URL. You configure authentication once, enforce Cedar policies consistently, get built-in CloudWatch observability across your tools, and revoke access in one place instead of across each tool registration.
To try this pattern, register one read-only MCP server as an AgentCore Gateway target, connect the Gateway to a non-production agent space, and classify each tool before you enable it. The following resources cover each step.
Models are now good enough to work for hours or even days on their own. You don’t have to watch over every thing they do, but you still need to be able to check in, no matter where you are. An agent can’t live only inside one application. The work should continue even If you close a window or switch devices.
In GitHub Copilot you can start an agent session in one place and pick it up in another. This session continuity runs on the Agent Host Protocol (AHP), an open standard built first in VS Code and adopted across GitHub. Here’s how it works, what we learned building it, and how you can use it to bring your agents into your own apps and workflows.
Why one integration per client stopped working
GitHub Copilot was first built in VS Code as an extension that bundled its own harness. This gave you access to your agents in VS Code, but it tied their lifetime to the editor. Extensions are attached to a single VS Code window. Close that window, and the agent is gone. Update VS Code, and every agent stops as the application restarts. Connect to a remote machine, and you get an entirely new extension host commit-locked against the version of the VS Code workbench you have.
At the same time, the Copilot app was being developed, alongside the existing GitHub mobile apps for iOS and Android. Each product was doing its own distinct integration with cloud-based agents to work towards delivering session continuity.
Each of these integrations solved the problem for one product, but developers also still want to be able to pull their agents into their own workflows–whether that’s an editor, a terminal, a phone, or a tool they built themselves–and they can’t do that if every client needs its own one-off integration. That’s why developers needed a protocol: a standard, independently versioned language across versions of VS Code, the GitHub Copilot app, mobile apps, and any other client that might host an agent. A shared protocol means every client sees the same agents, history, and state without bespoke integrations, and you aren’t tied to one app’s idea of what an agent session is.
Designing a protocol for agents on every client
The protocol needed to allow multiple clients to interact with the same set of agents seamlessly. Start a session in the cloud from your phone, for example, then continue it from your laptop. It should tolerate network interruptions, reconnections, and concurrent client mutations. And it needed to be full-featured enough for you to do real work from each of your clients.
At the same time, the protocol needed to stay out of agent behavior. Unlike the domains that produced the Language Server Protocol and Debug Adapter Protocol, the world of agents is evolving quickly. That’s why AHP avoids encoding how agents work, and instead focuses on how you want to interact with them.
You also need the protocol to be simple enough to implement on your own client or server. While the full feature set would necessarily be extensive, the core protocol needed to be minimal.
The result of these requirements is the Agent Host Protocol (AHP). It is oriented around URI-identified “channels”, which are the coarsest unit of functionality in the protocol. Every server must implement the root, session, and chat channels, but can also implement more channels that provide terminal support, MCP additions, or OTel reporting, for example.
Within each channel is a reducer-based state object, which drives most of the logic for clients. The server is the arbiter of the state, and each client sends a sequence of actions that either mutate the state or trigger side effects, like sending a chat message. This provides a few very nice properties. Reconnection and replay in poor network conditions are easy: a client sends all queued actions, presents the right ‘loading’ state, and waits for the server to reconcile and acknowledge the actions before confirming them. Clients and servers are straightforward to debug. And reducer-based state also plays nicely with popular UI frameworks like React.
Shipping five clients and two hosts
Adopting the Agent Host Protocol in VS Code required building a new harness and agentic stack underneath it. The Copilot Chat extension, which used to provide all of VS Code’s AI and agentic capabilities, had been growing for over three years, reaching over 400,000 lines of TypeScript, not counting tests. But, as a VS Code extension, it was structured around the lifecycle of a VS Code window and depended entirely on the VS Code extension API. Neither that lifetime nor those APIs would be available in a standalone agent host process.
As we started building the Agent Host Protocol in early 2026, the Copilot SDK was also being developed, and it made sense to leverage it as the foundation for the new agentic stack. This was a big transition. It’s easy to build a harness implementation that says hello and can edit files—we were using the agent host to develop the agent host after the first couple weeks—but it’s another thing to build one that covers the long tail of features developers had come to expect from Copilot Chat.
GitHub has had Copilot cloud agents for a couple of years, allowing developers to run tasks in the cloud from any device. While VS Code was quietly developing the Agent Host Protocol, GitHub was looking to expand that capability to coordinate sessions running on your own devices via the Copilot app, or in cloud sandboxes. This was exactly the N:M problem that AHP was built to solve, so adopting the same technology made perfect sense.
From the outset, the GitHub Agent Host had different requirements than VS Code’s implementation. VS Code runs the host locally and can drop its host on other compute over SSH, containers, WSL, or Dev Tunnels, and it is actually included as part of the VS Code Server bundle. The GitHub Agent Host needed to run in the cloud and across a much wider variety of infrastructure, environments, and security constraints.
Those differing requirements are why they’re separate implementations written in different languages: VS Code’s in TypeScript and GitHub’s in Rust. For now, having two agent hosts has been a good acid-test for the protocol itself. How well does it work when two entirely different teams and sets of agents implement the same protocol independently?
Pretty well, it turns out. Like MCP, the Agent Host Protocol provides escape hatches via custom methods and _metadata on objects that consumers can use for out-of-band signaling. After several months heads-down on our respective implementations, our first task as we worked toward unification was inventorying the ways each implementation had extended the protocol. Many of these extensions reasonably stay GitHub-specific, such as the GitHub Agent Host using metadata to link to project boards. In other cases, variance was a sign of a gap in the protocol; early versions, for example, could not accurately represent shell-type tool output that wasn’t backed by a PTY. And some were bad assumptions, like VS Code’s client assuming a particular structure for the session URIs it received from the agent host. A protocol with only one implementation isn’t truly portable; as we taught our clients and servers to talk to each other, AHP became portable in practice, not just in theory.
Today, the Agent Host Protocol provides that same continuity across all VS Code and GitHub surfaces. Agent sessions running locally in the GitHub Copilot app, in cloud sandboxes, and in VS Code all share the same underlying protocol. To see how it’s used, read about how you can work from anywhere with the new connected Copilot cloud experience.
Using the Agent Host Protocol
AHP was designed to be easy to use in clients and servers. Agent Host Protocol libraries are available in Rust, TypeScript, Kotlin, Swift, .NET, and Go, and agents are able to build clients and servers with little prompting. I have a macropad sitting on my desk that reflects my agent’s state in real time, and others have built everything from smart watch apps to Kanban-style task management tools against the protocol. Whether you want a companion app for your own agents, a way to bring agent sessions into your team’s tools, or AHP support in your own editor or agent, the protocol is open for you to build on.
You can connect to an agent host you implement in the Copilot app (enable Experimental > Direct Connections, then go to Environments > Add Agent Host Protocol Host) and in VS Code (via Add Remote Agent Host in the Agents window). You can spawn a standalone version of VS Code’s agent host by running code agent host in your terminal, or see the endpoints for the one in your editor by running code agent endpoints, and samples can help you and your agent get started.
The Agent Host Protocol is an open standard, and questions and suggestions are welcome on GitHub.
AI coding agents have changed the economics of writing software. Tools like Kiro turn specifications into working code, tests, and documentation, and a single agent can open dozens of pull requests across your repositories in an afternoon. What they have not changed is the number of senior engineers available to review those pull requests. The AI Productivity Paradox research from Faros AI drew on telemetry from more than 10,000 developers. It found that teams with high AI adoption merged 98% more pull requests while review time rose 91%. The gains did not show up in company-level delivery metrics. The constraint moved downstream, from writing code, to reviewing it.
The intuitive fix is to add an AI reviewer to every pull request. The evidence says that alone is not enough. Human-Centric to Agentic Code Review, a study of 1.02 million pull requests across 207 open source projects, found that reviews involving AI agents reached decisions faster but did not improve quality. After AI reviewers took part, the strongest factor explaining review efficiency was not the tooling, but how humans and agents were wired together in the review flow.
This post presents a pattern for wiring together humans and agents as part of code review. It introduces a gated review pipeline with six controls that pairs deterministic checks with verified AI review, applies your policy through risk-tiered gates, and keeps humans in charge of every consequential decision. It describes the core components of each control, how to implement them with AWS services, and how to measure the effect on defect and rework rates.
Two kinds of controls, and one wiring pattern
The AWS Blog Balancing speed and safety: a control framework for AI coding agents establishes a control taxonomy for what agents produce, through author-time controls (steering documents, specifications, and agent hooks) and pipeline scanning. It also introduces an AI-assisted review control. In this post, we expand that control into an operational design pattern. The pattern includes a review panel that produces verified findings, a deterministic gate that applies policy to findings, risk tiers that set agent autonomy, and metrics that show whether defect and rework rates fell.
This pipeline adds two design rules to that foundation. The first is a deliberate pattern for human and agent collaboration, because the research cited earlier found that collaboration patterns shape review efficiency. The second is risk-tiered autonomy. A documentation fix and a payments change carry different risk, so they shouldn’t pass through the same gate.
Assign the initial tier deterministically from what the change touches, not from what the author declares. Certain critical paths and files should be assigned the highest risk tier regardless of the change, for example changes to build and pipeline definitions, the Control 4 policy bundle, steering files, and the panel reviewer prompts. Changes such as a new dependency or an AWS Identity and Access Management (IAM) policy change should also escalate the risk tier. Evaluate related changesets together so a high-risk change can’t be split into low-risk pieces.
Tier
Change risk
Autonomy
Review depth
Evidence required
T1
Low (docs, internal tooling)
Agent drafts, human approves
Light spot-check
Pull request + provenance tag (agent, model, requester)
T2
Medium (standard features)
Agent drafts, engineer edits
Full review + tech-lead sign-off
Pull request + tests + approved spec or design notes
T3
High (payments, auth, data, changes to pipeline configuration)
Agent assists, engineer implements
Two human reviewers, plus architecture review when the design changes
Full audit package
Figure 1: An overview of the six controls in the gated review pipeline
The framework is methodology-agnostic. It composes naturally with structured approaches such as the AI-Driven Development Life Cycle (AI-DLC), but requires none of them. Where a team approves the specification before generation, that review settles whether this is the right change before any diff exists, leaving the pull request review to confirm it was built correctly.
Control 1: Standards consumable from one place
Code quality rules, architectural preferences, security standards, and compliance requirements each have an authoritative home where they’re owned, versioned, and published. Those homes can differ. A security team might keep a policy repository, compliance teams a wiki, and architects a decision log. All of these should feed a single consumption endpoint, so that every repository, pipeline, and agent pulls the same standards through the same mechanism instead of integrating with each source directly. Curating organizational knowledge into a form agents can consume is an easily underestimated investment. Funneling it through one consumption point means you make that investment once, not once for each repository and each source.
A single consumption point doesn’t require a single delivery mechanism. Retrieve situational standards on demand. Small, non-negotiable ones, such as production-safety guardrails, inclusive-language rules, and conventions for working with internal systems, belong in context on every interaction. Push those as baseline configuration, such as a global steering file or default agent configuration, so the agent doesn’t need to fetch them.
On AWS: Keep each standard in its authoritative home and put a single consumption layer in front of them. AgentCore Gateway, a capability of Amazon Bedrock AgentCore, can serve as that consumption point. It exposes each source as Model Context Protocol (MCP) tools from configuration alone, whether the source is a Git repository behind its provider’s MCP server, an HTTP endpoint, or Amazon Bedrock Knowledge Bases built over documents in Confluence or Amazon Simple Storage Service (Amazon S3). Every agent and tool then pulls the same versioned standards at runtime. The knowledge base is a retrieval layer for prose-style guidance that benefits from semantic search. It isn’t the system of record, and the owning team’s repository or document store remains authoritative. If you need evidence of which standard was in force for a given change, either sync a pinned copy into the repository as a pipeline step and record the version, or write the consumed standard version to an audit log at time of use. Deterministic consumers, such as linter configurations and the Control 4 policy bundle, should take the pinned path. A standards change then arrives as a version bump instead of failing every repository’s next build.
In the developer’s environment, the consumption format is whatever the agent reads, such as Kiro steering files or agent skills. Generate or sync those files from the gateway instead of writing them for each repository, so a local project becomes one more consumer of the same standards. For the always-on class of standards, generate the global steering files or default agent configuration from the gateway’s sources, and push them out through whatever provisions your developer environments. The gateway remains the on-demand path for situational standards.
Control 2: Deterministic gates first, with agent self-correction
The first gates to run are fast, deterministic, and hard-fail. These include formatting, lint, type checks, static application security testing (SAST), and secrets scanning. The remediation agent then self-corrects in a build, fail, fix loop until the stage passes or hits a maximum iteration cap, which hands the change to human review. Don’t use AI to review what a rule can catch. When the deterministic gates run first, the AI review stage sees only mechanically clean code and spends its reasoning on semantics instead of style. It also keeps most mechanical defects out of human review.
The self-correction loop is told to make the gates pass, and the shortest path to a passing gate is often a weaker gate. Treat suppression annotations, threshold changes, and removed or skipped tests in agent commits as change risk escalations.
The blog CI/CD Pipelines with agentic AI describes how Elastic does this. When a build fails, an agent steps in as a pipeline stage. It reads the failure logs, applies fixes from a deterministic tool allowlist, verifies the build, and commits, with auto-merge turned off whenever agent commits are present. In its first month, covering under half of Elastic’s dependencies, it fixed 24 broken pull requests and saved 20 days of development work.
On AWS: Run the same gates in two places. In the developer’s environment, Kiro agent hooks run them on file save and task completion. That catches defects before a pull request exists, where the fix loop is cheapest to run. The pipeline is the enforcement backstop. The same gates run in AWS CodeBuild on every pull request as a required status check, so a failing gate blocks the merge and nothing reaches review on the strength of a local run alone. Order them by cost and severity, as the predecessor post describes: secrets detection, SAST, software composition analysis (SCA), and then infrastructure as code (IaC) scanning.
When a pipeline gate fails, a remediation agent takes over. Because this agent commits to the repository, host it as a harness on AgentCore runtime, a capability of Amazon Bedrock AgentCore, outside the build, and call it from the failing CodeBuild action. The harness runs the agent loop in its own managed workspace. It treats the failure logs as untrusted data, applies fixes constrained by a tool allowlist, re-runs the gates, and commits to the source branch, reporting progress back to the pipeline. Authenticate that call with the pipeline’s IAM role so the automation runs on a workload identity you control, with no developer credential embedded in the build. Bound the loop with the harness’s execution limits. Cap fix attempts, set a wall-clock timeout, and give the agent a token budget. Count them per pull request, because each agent commit starts a new build. When any limit trips, the build stops and hands the failure to a human. Configure branch protection to dismiss approvals on new commits, so an agent fix always gets a fresh human review. For standing up the harness itself, see Hosting coding agents on Amazon Bedrock AgentCore.
Control 3: AI review as a verified panel
Run AI review as a panel of reviewers with distinct focus areas, for example, correctness, security, operational readiness, and, where the change was built from an approved specification, scope. Each seat on the panel is an agent that starts from its own clean context, which keeps it from inheriting assumptions about the code. Give each seat the diff as its focus, plus the full changed files and the dependencies they touch, so it reviews the change in context. An agent that sees only the changed lines can’t judge cross-file or architectural impact, and those are the defects most likely to survive the deterministic gates. From a security perspective, the code is untrusted input, so seats should be limited to sandbox environments with read-only tools.
Two sessions of the same model share the same blind spots. For each focus area on the panel, assign at least one agent to use a different model family than the author recorded in the provenance tag. For high-risk changes, you might run multiple agents with different models for the same focus area. Choose models by measurement. Evaluate candidates against past pull requests with known findings and prioritize models that score best for that focus area.
A two-stage verification step decides what reaches a human at the review gate. The first stage is deterministic. It confirms that the cited file and line exist and that the quoted code matches what’s there. A deterministic check can’t tell whether an anchored finding describes a real problem, which is a semantic judgment. The second stage can use a decision model, which returns one bounded answer (valid, invalid, or escalate to a human) with a confidence score. Because the answer is bounded, you can measure it. Evaluate the model against past findings with known outcomes, calibrate its thresholds, and route low-confidence cases to a human. Without this filter, volume works against you, as seen in one industrial deployment of a large language model (LLM)-based review tool. Average pull request closure time rose by more than 40%, even though developers acted on most of the automated comments.
On AWS: Run the panel seats as a single AWS CodeBuild batch build graph, one build per seat, each invoking an agentic coding harness in a fresh session and writing its findings to Amazon S3 under the commit’s key. Mount the checked-out repository so each seat has that context. The agents run inside CodeBuild because they only read code the build has already checked out, in an environment discarded after the run. The remediation agent in Control 2 writes to the repository, so it runs on AgentCore runtime under its own identity and limits.
A verification build that depends on the agents then reads their findings and runs the two-stage check. The deterministic pass checks each finding against the checked-out code and discards any finding whose file, line, or quoted code doesn’t match. A validity judgment then runs on what remains. The open source Strands Decider models fit that judgment, returning a bounded valid, invalid, or escalate classification with a confidence score. The verified, high-confidence set passes to the Control 4 gate, and low-confidence cases go to a human. For the security seat, you can use an open source harness such as the Visa Vulnerability Agentic Harness, which applies the same pattern to vulnerabilities. It uses deterministic stages for repeatable work and agentic stages for judgment, verifies each finding adversarially, and produces Static Analysis Results Interchange Format (SARIF) output that a gate can read. Run your chosen harness in a detection-only mode, so it emits findings without editing code, and feed its output to the same verification step and gate as the other agents.
Control 4: A deterministic policy gate on structured findings
The AI review panel in Control 3 never approves or blocks. It emits structured findings, each with a severity (important or minor), whether this change introduced it or it was pre-existing, and its evidence. Findings use a standard format such as SARIF, so deterministic and AI findings reach the gate in the same shape. A deterministic gate then applies your policy to that signal. It blocks on important findings, fast-paths clean changes, forces oversized changesets to split, and scales all of it by risk tier. Keeping policy in a deterministic component, apart from the model judgment it acts on, is the core of this pipeline. It makes the merge gate auditable, reproducible, and tunable for each repository. An empty finding set means the panel found nothing, not that nothing is there. Fast-path rules might shorten the queue for T1 changes but should never reduce human review depth at T2 or T3.
On AWS: Have the verification build write the verified findings to Amazon S3, keyed to the commit, as the record of that run. Then make the gate itself policy as code. A CodeBuild step or an AWS Lambda function evaluates the findings against rules written in Open Policy Agent or Cedar. The rules encode the preceding policy, scaled by risk tier. Oversized changes reach the gate less often when the specification breaks work into reviewable tasks. The gate reports to the pull request as a status check and posts a severity summary, for example “2 important, 4 minor,” before the verified findings. The rules ship in the pinned standards bundle from Control 1, so changing the merge bar means publishing a new policy version.
Control 5: Human judgment on a pre-triaged change
Human review remains a key pipeline component. What changes is what the human reads. Instead of a raw diff, the reviewer gets a change that the gates have made mechanically clean and the panel has triaged into ranked, evidence-backed findings. The reviewer no longer needs to catch what rules and agents already catch, and can focus on the judgments only a person can make. Is this change worth its risk? Does it fit the system’s design? Does it carry business context the agents never had? Are the important findings resolved, or only acknowledged?
Scale the depth to the tier: a T1 change needs one approver doing a light spot-check, a T2 change a full review with tech-lead sign-off, a T3 change two human reviewers plus architecture review when it changes a design boundary. The person who directed the agent is the change’s author and doesn’t count toward its reviewers. Review depth scales with risk, and the human decision is never delegated.
On AWS: Your existing pull request approval flow stays the baseline human gate. The panel’s verified findings land as review comments with a severity summary at the top, so the approver starts from a pre-triaged change. For T3 changes, require a second human reviewer before merge, and use your source provider’s approval rules to block approval by the requester. After merge, an AWS CodePipeline manual approval action adds a second recorded decision before production. It notifies approvers through an Amazon Simple Notification Service (Amazon SNS) topic routed to wherever they already work. Have the tooling that runs the agent, not the agent itself, write the provenance tag as commit trailers that record the agent, model, session, specification, standards version, and requester.
Figure 2: The pre-merge pipeline, where standards, deterministic gates, the verified review panel, and the policy gate run as pull request checks and branch protection enforces the gate and the tiered human review
Control 6: Post-merge verification and operations
The same logic extends after merge. Tests that ship with a change go through Controls 2 and 3 with the change. This control covers suite-improvement tests that an agent generates against existing code. Meta’s TestGen-LLM is an example. It admits a generated test only after the test clears filters showing that it measurably improves the existing suite. Admit these tests as you would untrusted contributions. They must build, pass reliably across repeated runs, and measurably improve the suite, through a coverage delta or a mutation score, before a human spends time on them. The filter removes tests that add nothing, but it can’t tell whether the asserted behavior is correct, because a test generated against existing code records what the code does today. That judgment belongs to a person, and it matters more here. A test that pins wrong behavior into the regression pack blocks the future change that fixes it. Tests that clear the bar join the regression pack that the deterministic gates run on every later change, so post-merge generation keeps strengthening the pre-merge gates. In operations, agents investigate on their own, reading telemetry, correlating signals, and proposing mitigation plans, while a person approves any remediation.
On AWS: Enforce the coverage or mutation threshold in a CodeBuild project that runs as an action in a dedicated stage of your CodePipeline. A generated test that doesn’t measurably improve the suite fails the build and never reaches the regression pack. This is the same mechanism as Control 2, pointed at a different signal. There, the build fails on lint, type, or SAST violations. Here, it fails on a coverage delta or mutation score that doesn’t clear the bar. For operations, AWS DevOps Agent fits this split. It’s read-only by default, investigating pipeline and operational failures and proposing mitigation plans for a person to approve.
Measuring whether it works
The research cited earlier found that AI review made decisions faster without making them better. Cycle time is the number most readily moved, and it can improve while quality degrades. Track review cycle time, because review is the constraint this approach exists to relieve, but only alongside the following quality measures. Avoid lines of code and pull request counts entirely, because they measure activity, not outcomes.
Measure instead:
Defect escape rate and rework rate – Bugs found after merge, and changes that need follow-up fixes.
Change failure rate and time to restore – The DevOps Research and Assessment (DORA) stability metrics, which show whether stability held as change volume grew.
Finding acceptance rate per panel seat – A falling rate means the seat is getting noisier, or that reviewers have stopped engaging with its findings. Check the seat first.
Panel recall – The share of escaped defects the panel would have flagged, found by replaying them through it. Acceptance shows what the panel gets right, and recall shows what it misses.
Changeset size drift – Agents will absorb ever-larger diffs if you let them. Keep changes small even as more of them are automated.
Cost per accepted finding – Model and compute spend divided by the findings that reviewers acted on, by seat and tier. It shows which seats earn their place and where tier policy should run fewer.
For the executive rollup, cost to serve software, defined as total delivery cost divided by units of value delivered, captures the whole system. Faster generation that creates more review burden shows up as a worse cost to serve. Pair cost to serve with tension metrics, such as security incidents and rollback rate, so the number can’t be improved by cutting controls. Developer experience work at Amazon reported a 15.9% year-over-year reduction on this measure in 2024.
On AWS: Measure from the signals the pipeline already emits instead of building a parallel system. Route CodePipeline and AWS CodeDeploy deployment events, plus your incident records, through Amazon EventBridge to AWS Lambda and into Amazon CloudWatch for the change failure rate and time-to-restore metrics. The DORA metrics sample is a useful reference for wiring this up. The review-quality metrics come from artifacts the pipeline already writes. Query the Control 4 findings store in Amazon S3 with Amazon Athena for acceptance rate per seat and changeset size drift. For that, the gate records each change’s tier and size in its report, and each finding’s outcome on the pull request (resolved or dismissed) is written back to the store. Defect escape and rework rates need one deliberate link. Tag post-merge defects and their follow-up fixes back to the commit that caused them, so the numbers attribute to a change.
Roll the executive view up in Amazon QuickSight, showing cost to serve software alongside the tension metrics that keep it from being gamed.
Figure 3: The post-merge pipeline re-runs the gates, admits generated tests, adds the T3 approval before production, and feeds delivery and review-quality metrics
An incremental adoption path
Roll the pipeline out in stages instead of all at once. Each stage produces the evidence you need to trust the next. A baseline tells you whether anything improved, airtight deterministic gates show what the panel adds beyond rules, and an advisory panel shows whether its findings deserve to block.
Baseline first. Capture defect escape and rework for four weeks before you turn anything on, or you will have no evidence story. Expect escape rates to lag, because defects surface weeks after merge. Rework moves sooner.
Deterministic gates next. Get Control 2 airtight with the agent self-correction loop. This step alone keeps most mechanical defects out of human review.
Add the panel selectively. Pick starter repositories with high pull request volume and moderate (T2) risk, where you get signal fast without affecting a critical system. In this phase, the panel is advisory. It posts findings but doesn’t block anything, because the policy gate isn’t wired yet. Use this window to confirm its value. Watch the finding acceptance rate, and check whether the panel surfaces real issues your reviewers were missing before you let severity block anything.
Wire the policy gate only after the panel has earned trust. When acceptance rates are stable and engineers act on the findings, let severity block merges.
Extend to tests and operations last.The same evidence can widen agent autonomy over time, for example letting the remediation agent fix verified findings in categories where the panel has proven reliable. Teams with long-established trust in their pipeline sometimes go further and auto-merge low-risk changes, as the Kiro Crew team describes.
Figure 4: The staged rollout, from a measurement baseline through deterministic gates, a selective AI panel, the policy gate, and finally tests and operations, aligned to rising risk tiers
The same pipeline, two changes
The following table follows a T1 change and a T3 change through all six controls. The pipeline is the same for both. What changes is autonomy and review depth.
Control
T1: fix a broken link in the README
T3: change refund-retry logic in the payments service
1. Standards
Pulls the same standards, and the always-on security and compliance guardrails that are baseline config for every change
2. Deterministic gates
Formatting and link-check run, pass, nothing to self-correct
SAST, secrets, SCA, and IaC scans run. The agent self-corrects a flagged dependency, re-runs, and passes within the bounded loop
3. AI review panel
Skipped by T1 policy, because the Control 2 link check already verified the fix
Full panel runs with repository context. The security and correctness seats flag an auth-scope concern on the retry path, which is verified and passed to the gate
4. Policy gate
T1 policy: no important findings, fast-path to a single approver
T3 policy: an important finding blocks, oversized-diff check passes, routed for full review
5. Human review
One approver, light spot-check, merge
Two human reviewers, and the auth-scope finding must be resolved, not only acknowledged. T3 adds the CodePipeline manual approval before production
6. Post-merge
Nothing further
Agent-generated suite-improvement tests on the touched module must clear the coverage bar before a human looks. Operational signals are watched after deploy
Conclusion
AI coding agents are accelerating code generation and making code review the constraint. Adding an AI reviewer doesn’t remove that constraint on its own. Outcomes depend on how humans and agents are wired together. In this post, we showed a pipeline that wires them together to speed up review while reducing risk. Deterministic rules catch the first issues. A verified panel reviews what remains and emits a structured signal, a deterministic gate applies your risk-tiered policy to that signal, and a human makes the final call on a pre-triaged change. The same pattern extends to generated tests and operations.
The AWS Workbook for BNM RMiT, available through AWS Artifact, maps each RMiT provision to AWS Control Objectives (AWS responsibilities under the AWS Shared Responsibility Model) and AWS Well-Architected Framework best practices (your responsibilities). It tells you, for every RMiT requirement, what AWS is responsible for and what you implement.
The compliance guides published by AWS SAS over the past year provide implementation guidance for International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC) standards that complements the workbook. These guides show how to implement controls that meet the requirements of specific standards and frameworks using AWS services, giving your technical and compliance teams a clear path from regulatory expectation to deployed architecture.
Malaysian financial institutions can now trace from BNM regulatory requirement, through AWS control implementation, to the evidence that demonstrates it: AWS third-party attestations for AWS-side controls, and your own workload evidence for customer-side controls. BNM expects this specificity in notification documentation.
AWS Workbook for BNM RMiT
The workbook is a structured Excel document available through AWS Artifact. For each of the 394 RMiT provisions, it identifies:
Whether the requirement is an AWS responsibility or a customer responsibility under the Shared Responsibility Model
The specific AWS Control Objective that addresses the requirement (where AWS is responsible)
The relevant AWS Well-Architected Framework best practice for customer implementation (where the customer is responsible)
Supporting information and references for each mapping
The following is a sample entry from the workbook:
RMiT Requirement
Responsibility
Source
AWS supporting information
10.49: A financial institution should ensure network resilience…
AWS
AWS Control Objectives
AWS Control Section: Network Security. AWS maintains redundant network connectivity with multiple tier-1 transit providers across each Availability Zone.
10.49: (Customer responsibility)
Customer
Well-Architected
REL 2: How do you plan your network topology? Implement multi-AZ deployments. Configure VPC flow logs and use AWS Network Firewall.
The workbook currently includes 394 rows across all RMiT appendix requirements, with 208 mapped to AWS responsibilities and 186 to customer responsibilities.
Why this matters now
BNM revised RMiT guidelines effective 28 November 2025, and updated them on 25 September 2026 to add requirements for National Critical Information Infrastructure (NCII) entities. Financial institutions operating under both BNM Outsourcing and BNM RMiT must demonstrate that their cloud service providers maintain controls aligned with regulatory expectations. Institutions preparing notification submissions for material cloud workloads need auditable evidence of this alignment.
The revised guidelines set higher expectations for evidence quality. Institutions submitting notifications without clear, structured control mapping evidence might face extended review cycles and requests for additional information from BNM. Workloads that could be deployed in weeks remain in governance queues for months when supporting documentation lacks the specificity regulators expect.
Compliance guides from AWS Security Assurance Services
Start with the RMiT workbook to identify which provisions AWS addresses and which you implement. Then use the following compliance guides to design and implement your customer-side controls. The guides are implementation guidance for your teams. Audit evidence comes from two places. For AWS-side controls, use AWS third-party attestations, such as ISO certificates and System and Organization Controls (SOC) reports, available in AWS Artifact. For your workload, use evidence from your own environment, including configurations, logs, and AWS Config conformance pack results. Auditors can also use the guides as a reference for how a control is typically implemented on AWS.
AWS SAS has published five compliance guides over the past year. Each guide provides control-level implementation guidance on the customer side of the Shared Responsibility Model, showing how AWS services can be configured to meet the requirements of specific standards and frameworks. These guides were written by AWS SAS practicing security assessors and auditors holding Payment Card Industry Qualified Security Assessor (PCI QSA), Certified Information Systems Security Professional (CISSP), Certified Information Security Manager (CISM), ISO 27001 Lead Auditor, and Certified Cloud Security Professional (CCSP) certifications, drawing on experience from hundreds of cloud security assessments across financial services globally.
The following is the list of guides AWS SAS practicing auditors have published, which include control mapping information relevant to BNM RMiT:
A Malaysian financial institution preparing a BNM notification can use these resources together:
The RMiT workbook identifies which RMiT provisions are AWS responsibilities and which are customer responsibilities.
For AWS responsibilities, the workbook references the specific AWS Control Objective.
AWS third-party reports in AWS Artifact, such as the AWS ISO/IEC 27001 certificate and SOC reports, attest to those AWS Control Objectives. For customer responsibilities, the ISO compliance guides show how to implement the corresponding controls using AWS services.
The ISO 31000 guide can inform the risk management methodology you apply to your cloud workloads.
Compliance teams across Association of Southeast Asian Nations (ASEAN) financial services have already begun referencing the AWS Guide to BNM RMiT and associated resources in their cloud risk assessment processes and BNM notification documentation.
Looking ahead: AI governance
For institutions deploying generative AI and machine learning, ISO/IEC 42001:2023 (Artificial Intelligence Management Systems) is relevant to BNM technology governance expectations. In November 2024, AWS became the first major cloud provider to announce ISO/IEC 42001 certification for AI services. AWS passed its first surveillance audit in November 2025 with zero findings. The ISO 42001 compliance guide published in May 2026 supports institutions building their own responsible AI systems on their side of the AWS Shared Responsibility Model.
How to get started
Access the RMiT workbook through AWS Artifact and review with your compliance lead to identify which provisions apply to your material cloud workloads.
Download the relevant ISO compliance guides to plan how you implement your customer-side controls. Download AWS ISO certificates and SOC reports from AWS Artifact for AWS-side evidence.
Map your workload-specific controls using the workbook as the RMiT baseline and the ISO guides as implementation guidance.
Engage your AWS account team or AWS SAS for additional guidance on implementing customer-side controls referenced in the workbook.
As BNM updates its regulatory expectations, AWS will update these resources to reflect revised requirements. Subscribe to the AWS Security Blog for update notifications.
As AWS re:Invent is around the corner, we’re proud to recognize a new group of AWS Heroes. Every Hero finds their own way to give back, whether through sharing what they know, bringing people together, or building tools that help others succeed. What they have in common is a genuine commitment to supporting the builders around them learn, grow, and lead.
Let us take a moment to meet them!
Andres Moreno – El Paso, United States
Serverless Hero Andres Moreno leads the Cloud del Norte AWS User Group and is a former AWS Community Builder. On his blog, And More Dev, and at conferences and community events, he explains serverless, cloud native architectures, generative AI, and agentic systems in ways that make new technology easier to try. Andres puts special energy into supporting builders across Latin America and helping them explore what’s possible on AWS.
Geethika Guruge – Auckland, New Zealand
Community Hero Geethika Guruge is a Principal Consultant at Mantel Group whose work spans serverless, DevOps, cloud infrastructure, and agentic AI. Geethika has shared real lessons from building agentic applications and leading large migration and modernization programs at AWS re:Invent, AWS Summits, and AWS Community Days. At home in Auckland, Geethika organizes local AWS meetups that encourage grassroots innovation and mentors the next generation of cloud architects and leaders.
Jéssica Cristina Coelho – Belo Horizonte, Brazil
Data Hero Jéssica Cristina Coelho is a Database Reliability Engineer and a former AWS Community Builder with more than 15 years of experience in databases, cloud, and data engineering. She leads the AWS User Group in Minas Gerais, hosts DataTalks Brasil, teaches MBA courses, and supports Women in Cloud, DevOpsDays, and AWS Community Day events. She speaks at events such as AWS Summit São Paulo and Web Summit Rio on database modernization and generative AI. At AWS re:Invent 2025, she was recognized as a LATAM Diversity & Inclusion Champion.
Kazuho Cryer Shinozuka – Tokyo, Japan
DevTools Hero Kazuho Cryer Shinozuka is an AWS CDK Top Contributor and Trusted CDK Community Reviewer. With 200 merged pull requests, he ranks 18th among more than 1,700 contributors. He has reviewed more than 120 pull requests, has been involved in about 2% of all CDK commits, and has published 10 construct libraries, along with tools such as cfn idle cost. Through talks at JAWS DAYS and CDK Conference Japan and through hands on contribution workshops, he helps more developers in Japan start contributing to open source.
We recently introduced AI Discover, a new tab in our TV app where a small group of members can try new ways to find something to watch. For years, our recommendations have worked in the background, inferring what a member might enjoy from their viewing history. AI Discover moves from that silent ranking toward steerable discovery. A member picks a theme, say “Quick Watches,” and gets a row of matching titles, a short note on why they fit and a few narrower themes to try next. From there, they can keep narrowing, head somewhere new or tune what they see.
Building AI Discover raised a question many teams will recognize: how do you deliver a better member experience with generative AI on TV without sacrificing speed, reliability or safety? Our answer is a new intelligence layer, the agent harness behind AI Discover. For each request, it decides how to combine LLMs with our recommendation systems. It builds on years of work understanding what members enjoy, and on our recent work applying LLMs to recommendations and generative experiences. The layer itself plans each request, runs it across those systems, serves the result and evaluates it offline.
This post covers the challenges we designed for and the key decisions we made along the way.
Why an intelligence layer?
Every Netflix page combines recommendations with rules. We pick titles a member may enjoy, and eligibility, maturity and safety rules decide what can appear. An experience built on LLMs needs the same rules, plus code that decides when to call an LLM and how to handle its output. That code is called an agent harness, and it’s the core of our intelligence layer.
The simplest implementation hands the entire request to an LLM and lets it pick the titles, write the labels and decide what happens next. However, that leaves us relying on probabilistic output where we need governed results, because every turn in AI Discover ends on a screen in someone’s living room. At our scale, even rare failures happen many times a day. A member sees every slow call, unavailable title or off-brand line. Seven challenges shaped our design:
Fitting LLMs into a product and architecture designed without them. Traditional recommenders rank titles based on past behavior. LLMs can do more: understand what a member is asking for and plan how to respond. Our existing experience would leave most of that unused.
Keeping recommendations grounded. Every pick must be a real, available title from our changing catalog. Retrieval and rankers work in tandem with LLMs to choose titles that fit the member’s taste and intent.
Working with slow, unpredictable LLMs. LLM calls are the slowest and most expensive part of the system. Tail latencies can spike, and the output may not be what you asked for. So we use LLMs selectively, treat what they return as untrusted input and test performance and quality before anything ships.
Adapting to fast-changing models, tools and prompts. Modeling and engineering solutions keep evolving, so the product can’t depend on any single model, tool or prompt.
Building for TV. Typing on a TV is hard, so members rely on the remote, and the app should respond to every press right away on widely varying hardware. TV apps also update slowly, so changes have to ship without an app release. A typical TV request returns in moments, while multi-step generation can take several seconds.
Holding a safety and reliability bar at scale. Members always get a page, even when an LLM call fails or yields poor results. Their choices and our content standards hold on every request, and when something breaks, structured records show why.
Learning from what members say alongside what we already know. Recommendations have always learned from viewing behavior. AI Discover also lets members tell us what they want, and that has to shape what they see next.
System Architecture
When a member picks a theme, the TV app sends the request to the Page Server, which builds pages for all Netflix apps and hands the request asynchronously to the intelligence layer. At a high level, this new layer has two components: a controller and an executor.
The controller decides what to run. It gathers the context needed for each request, including real-time signals such as the time of day. It then settles on a plan, either an established one or a new one drafted by its planner, validates it and retrieves only the member details that this plan requires from Memory Hub, our per-member memory store. These details may include member taste summaries generated by MemberLLM, our in-house language model for personalized recommendations, or feedback shared by the member. We talk more about Memory Hub and member taste summaries later.
A plan is a graph of approved steps, each running a capability. A capability is a named unit of work that can have multiple versions and belongs to a family such as retrieval, ranking, generation or packaging. It declares its inputs and outputs, the context it may see and what happens if it fails.
Capability declarations make a plan verifiable. Before execution, the controller validates the graph: every step is approved, every input has a producer, nothing loops and the plan ends in exactly one page. Afterward, the controller checks the result before it’s served.
The executor runs the plan. Steps pass typed outputs to each other through the declared contracts, so each receives exactly what it asked for. Independent steps run in parallel, each with its own deadline and appropriate fallback behaviors. Some steps are optimized LLM calls with one task each, such as writing row titles or suggesting next themes, and they act only within the validated plan. Checks before the run and contracts during it give us governed execution, where every action stays within limits set in code.
Both parts write to the request records, the traces we use offline to evaluate and improve the system.
System Architecture Diagram
Key Decisions
Each challenge led to one decision. Here’s what we chose and what it bought us.
1. Design the product and architecture around LLMs
We reimagined the experience around LLMs. The easier path would have been to drop them in as rankers, keep today’s latency budgets and precompute where necessary. AI Discover is built on a back-and-forth around themes a member can steer, which gives LLMs work only they can do: generating themes, explaining rows, summarizing taste and helping choose the titles that fit the member and the moment. Members pick themes with the remote, so they never have to type. On the TV, those themes and explanations get their own native UI elements. Behind the screen, LLMs can draft plans, and asynchronous serving gives multi-step agentic orchestration the time it needs.
This opens up experiences our traditional stack couldn’t support on its own.
2. Make existing systems composable
Keeping recommendations grounded meant building on the systems that already know our catalog and members. Most capabilities wrap complex, proven Netflix systems such as search, large-scale ranking and catalog metadata access. Some of them may run their own agents in the future.
Reshaping them took careful work, because agentic orchestration challenges many long-standing assumptions of distributed systems. A capability now has to work in combinations no one designed, for a caller that chooses to call it at runtime. Each one needs a clear purpose and a semantic interface, and shaping it means deciding how much to combine into a single step and how much to leave to the planner. Too fine, and the planner has more ways to go wrong; too coarse, and it loses room to compose richer plans.
In return, the planner gets building blocks it can mix in new ways, and connecting services takes far less manual wiring. Because plans can only use these blocks, even one drafted on the fly stays grounded in our catalog and member taste.
3. Plan first, then execute
To contain LLM latency and errors, we plan every request before anything runs. The familiar way to let an agent use tools is a loop: reason, call a tool, read the result, reason again. It’s flexible, but you can’t tell up front how many steps it will take or whether it has gone down a wrong path. In agent terms, ours is a plan-and-execute agent: an LLM can propose the path, but code decides whether it runs.
Plans come from two places: established plans we’ve already tested, and plans an LLM drafts on the fly. A router we control decides which one is served. Either way, the plan passes the same checks before it runs. If the planner is slow or its plan fails a check, we fall back to a tested plan.
This allows for adaptive planning. The planner can find new ways to serve a request, and we can balance cost, latency and quality for each task, from a lean single-call plan to a richer multi-step one.
4. Stay model and capability agnostic
The best model or capability for a task today may be second-best, or overkill, in a few months, so we built the system to switch between them. We assign each task to the model or capability version that fits it. Today that’s a per-task setting we can override live. Only some steps in a request use an LLM.
That only works if switching is easy. All the LLMs we use sit behind one common interface. Before we switch, each candidate is scored on the same tasks by automated checks and LLM judges, so every swap is a measured change.
Much of the system’s behavior lives in server-side prompts and configuration, so we can swap models, tools and capabilities without new code or a TV app release.
Upgrades get cheap and experiments get easy. Moving to a better model, ranker or retrieval system is a plan or setting change, not a migration, so each component can keep improving without a large investment.
5. Make page generation asynchronous
Inference keeps getting faster, but member expectations rise just as quickly: richer responses, more context and more of the page shaped by LLMs. The line between what feels instant and what’s worth waiting for will keep shifting, so we built the asynchronous path as a first-class way of serving.
Most of a TV app is synchronous: press a button, get a page. Generative work breaks that pattern. Holding one request open for many seconds across services would tie up too many threads. So requests that involve generation are asynchronous. The press returns right away with an acknowledgement, and the TV switches to a thinking state that explains what’s being prepared. Meanwhile, the executor runs the plan in the background. When the result is ready, the server notifies the TV, which then fetches it.
Slow generation no longer slows anything else. The serving path stays healthy however long a turn takes, and every press still gets immediate feedback on screen.
Async Sequence Diagram
6. Enforce safety in code
Some rules must hold no matter what an LLM writes: content and maturity standards, what each step can access, and what a member has asked us not to show them. Putting those rules in a prompt makes them probabilistic, so we enforce them in code. Generated text is cleaned up before it reaches the screen. The planner sees a stripped-down request, and member details are limited to what the validated plan needs.
For example, the planner decides which inputs each step needs, such as the profile’s maturity rating for retrieval. Instead of writing parameter values directly, it uses placeholders. After the plan passes validation, the controller fills them with values from the request, matched by declared type. This also protects against prompt injection.
Each request also leaves a structured record: the plan, each step and its outcome, and which rules applied. When a page misses, we can tell whether the plan, the data, a rule or the model was at fault. The same records feed our offline evaluation and replay.
Together, these rules and records let an experiment ship at the same safety and privacy bar as the rest of the product.
7. Remember what members tell us
A taste summary is a short profile of what a member enjoys, such as favorite genres, tone and recurring themes. MemberLLM generates it ahead of time from the member’s viewing history, which keeps that LLM work off the request path. It helps us further personalize their page, and steps that write themes and explanations receive it as context, so their wording reflects the member. We’ll cover MemberLLM in more detail in an upcoming post.
In AI Discover, members can also tune their recommendations and tell us directly what they want. Those choices apply right away and carry across visits, and they give us a new signal alongside what we already know about their taste. This is a first step toward remembering more about each member, from their preferences to the habits that show up over time and the corners of the catalog they like to explore. We keep it all in Memory Hub, our persistent, low-latency memory store, next to their taste summaries, and we can add new kinds of memory as we need them. We apply these memories to later requests where they’re relevant, together with real-time context.
The member holds the remote
In this version, members interact by selecting themes and using tuning options such as “don’t recommend.” There’s no free-text input or open-ended chat. That keeps the LLM’s job narrow by design.
The usual protections still apply. Profile settings and our safety and privacy policies hold here as everywhere else.
Generated text was tested before launch. We ran safety evaluations on theme labels, row titles and explanations to make sure they meet our standards.
What’s next
It would have been faster to let an LLM do everything and stop there. The intelligence layer gave us an AI-native product, grounded building blocks, adaptive planning, quicker experimentation, asynchronous generation, governed execution and agent memory for tunable personalization. While AI Discover is its first experience, we designed it with more in mind.
We’ve organized our efforts to answer two questions:
How much do members want to shape their recommendations? Whether they’d rather steer or lean back, and how much, which kinds of themes help, and how we should blend what members tell us with our existing personalization systems.
What tradeoffs does this technology bring? How cost and latency hold up in production as we ramp traffic, how long members will wait for generated results, and whether errors came from the model, the harness or our existing systems. Multi-turn sessions also need different evaluation and observability than single requests.
Longer term, our goal is a system that improves itself, starting from the record each request leaves behind. Today we replay past requests against a new prompt, model or rule to see what changes, and we judge each change with automated checks, LLM judges and human evaluations. Eventually, we want the system to propose improvements, test them safely and keep only what proves better, within our privacy and data-use policies. As LLMs improve, more of the work may move into them, and the intelligence layer lets us move that line deliberately.
This is part of a bigger shift. Recommendation systems have long learned quietly from what members watch and answered with ranked rows. LLMs let them listen and respond, and members may increasingly expect that. We’ll keep exploring what that means for discovery on Netflix.
Acknowledgements
We would like to thank Adi Suresh, Akash Katipally, Allan Johns, Becks Wood, Bhargav Mangipudi, Carrie Santiago, Chris Martin, Claire Campbell, Corey Waldin, Dana Sittler, Daneo Zhang, David Leen, Devon Bryant, Dharamendra Kumar, Dharini Chandrasekaran, Ebony Turner, Federico Roa, Gianfranco Cuoco, Guru Tahasildar, Haley Clanton, Haresh Chudgar, Ivan Provalov, James Schultz, JingXu Yang, Josie Shaw, Kamelia Aryafar, Kate Cummings, Kathleen Garrity, Kelley Robinson, Kevin Du, Kevin O’Connor, Kyla Moore, Kyle Lind, Laura Matsuura, Leah Cusella, Lingyi Liu, Lingyong Wang, Mallory Walkup, Matthew Pereira, Michael Galassi, Minsu Jang, Moumita Bhattacharya, Paran Jain, Paul Lee, Qusai Al-Rabadi, Scarlet Chen, Sergi Perez, Sriram Rao, Stephen Shaw, Sura Elamurugu, Varad Pathak, Yesu Feng, Zainab Mahar Mir, and Zhongli Cheng for their contributions to this work.
Engineering AI Discover was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Drew DeVault has announced that Redict, a fork of Redis that was created in response to Redis’s 2024 license change, has been deprecated. He recommends switching to Valkey, the permissively licensed Redis fork supported by the Linux Foundation. The two projects are mutually compatible, so switching should be as simple as changing which package is installed and renaming the configuration file.
I had hoped to rally the Redis diaspora around a copyleft initiative, to protect ourselves from future rug-pulls, and made Redict to serve this purpose. However, with the valkey project’s success, and Redis’s return to a copyleft license, Redict was not able to build a sustainable community.
We recommend that Redict users switch to valkey, which uses a permissive license but has better long-term prospects. We advise against switching back to Redis; though it is once again open source, and copyleft, it uses a contributor license agreement which will facilitate another rug-pull when Redis has a bad financial quarter.
In this post, we discuss how Rakbank retail digital banking platform ingests approximately 150 GB a day of telemetry across a cross-account, Multi-Availability Zone (Multi-AZ) Amazon OpenSearch Service design, with hot/warm/cold tiering, 7-year compliance retention backed by Amazon Simple Storage Service (Amazon S3), and approximately 55 percent cost savings over an all-hot cluster.
Context and workload profile
Rakbank’s retail digital banking platform runs as containerized microservices on Amazon Elastic Kubernetes Service (Amazon EKS), and its observability requirements are driven by a few hard numbers. The platform now generates roughly 150 GB of logs per day on average, up from about 5 GB/day before the migration. That produces around 9 TB retained in OpenSearch Service over a 60-day window (150 GB/day × 60 days), with the most recent 30 days directly searchable. Retention runs to seven years for compliance, aligned with Central Bank of the UAE (CBUAE) requirements and Rakbank’s internal data-retention policy. At an average of 150 GB per day, that archives approximately 54 TB of log data per year, or approximately 383 TB over seven years. On the consumption side, 25 dashboards serve about 25 concurrent users. The data itself has an important property: logs are append-only. After written, a log document is never updated or deleted, so there is no update or delete-by-query churn. (Where regulatory write-once-read-many immutability is required, the S3 archive bucket can enforce it with Amazon S3 Object Lock.)
The append-only pattern is architecturally significant: because indices are no longer written to once they roll over, force-merging them to a single segment per shard before warm migration is safe, and there is no update overhead as indices roll down the tiers.
Telemetry pipeline: Fluent Bit
Instrumentation follows an instrument once, standardize everywhere model on a Fluent Bit standard across all banking domains (retail, business, wholesale). Concretely:
Fluent Bit runs as a DaemonSet on the EKS cluster, tailing container stdout/stderr on each node and enriching records with Kubernetes metadata (pod, namespace, labels) before routing.
Fluent Bit performs a dual-write/parallel archive: one path ships to the OpenSearch Service ingest endpoint (hot), a parallel path lands raw log objects directly in Amazon Simple Storage Service (Amazon S3) for durable backup independent of the search tier.
Cross-account topology and private connectivity
The design separates the workload from its observability plane using two accounts:
Application account
Amazon EKS cluster running the app pods in the Application virtual private cloud (VPC).
Fluent Bit DaemonSet for container-log collection and routing.
A parallel archive path to Amazon S3 (Log Archive & Backup).
Observability account
A dedicated OpenSearch Service centralized logging domain in the OpenSearch Service VPC, spanning three Availability Zones.
OpenSearch Service Dashboards as the access/visualization layer, routed to the hot data nodes.
Connectivity. Cross-account log flow traverses an AWS Transit Gateway, keeping all communication private within AWS. No traffic egresses to the public internet. In practice, Transit Gateway route tables provide reachability between the application and OpenSearch Service VPCs. Access is scoped at the VPC boundary by security groups and network ACLs that permit only the required ingest traffic (HTTPS/443) to the domain.
The following diagram shows the full cross-account path: application pods write logs through a Fluent Bit DaemonSet, which both forwards them privately over AWS Transit Gateway to the OpenSearch Service domain and archives a parallel copy to Amazon S3.
Figure 1: Centralized observability platform showing cross-account log flow from the EKS application account across AWS Transit Gateway into the Multi-AZ OpenSearch Service domain with hot, UltraWarm, and cold tiering and Amazon S3 archival
In the Application account, three application pods on the Amazon EKS cluster write logs that a Fluent Bit DaemonSet collects. Fluent Bit ships one copy over AWS Transit Gateway to the Observability account and archives a parallel copy to the Amazon S3 Log Archive and Backup bucket. In the Observability account, the OpenSearch Service centralized logging domain spans three Availability Zones. Each zone has a dedicated cluster-manager node (m8g.large.search) and a hot data node (m8g.2xlarge.search) that ages out to an UltraWarm node (ultrawarm1.medium.search) and then to a shared, node-free S3-backed cold storage tier. OpenSearch Service Dashboards reads from the hot data nodes for visualization.
Identity and cross-account authorization flow
The platform spans two accounts, with access governed by the layered security model of OpenSearch Service: network isolation (VPC and private Transit Gateway connectivity), the domain’s resource-based access policy, and fine-grained access control (FGAC) on the VPC domain. End users reach OpenSearch Service Dashboards through Security Assertion Markup Language (SAML) 2.0 single sign-on federated to Rakbank’s corporate Active Directory. Access follows existing enterprise identities and group memberships, with no separate OpenSearch Service credentials. Log ingestion from the application account is authorized separately from user access, keeping the write path and the read path on distinct, least-privilege roles. Together these layers keep all telemetry flowing privately within AWS while access stays centrally governed in Active Directory.
Cluster topology (per-AZ node roles)
The domain is a 3-AZ deployment with dedicated cluster-manager nodes separated from data nodes:
Role
Per-AZ instance
Count
Purpose
Cluster manager
m8g.large.search
3 (1/AZ)
Cluster state, shard allocation, no data
Hot data node
m8g.2xlarge.search
3 (1/AZ)
Active indexing + real-time query
UltraWarm node
ultrawarm1.medium.search
3 (1/AZ)
Read-only, S3-backed warm segments
Cold tier
No nodes
—
Detached S3-backed storage, managed by OpenSearch Service
The hot data and cluster-manager nodes are Graviton (m8g) for better price/performance. Dedicated cluster-manager nodes isolate cluster-state management from indexing and query load, which matters after you’re pushing 150 GB/day through the hot tier.
Storage tiering: right data · right store · right cost
The core mechanism is OpenSearch Service multi-tier storage, with indices aging through tiers using Index State Management (ISM) policies keyed off index age:
Tier
Backing
Retention
Access pattern
Instance/store
Hot
Amazon Elastic Block Store (Amazon EBS) (gp3) volumes on data nodes
0–15 days
Real-time dashboards, active troubleshooting, alerting
m8g.2xlarge.search
Warm (UltraWarm)
Amazon S3 + local cache
15–30 days
Read-only historical analysis. Cost-optimized, still fully searchable
ultrawarm1.medium.search
Cold
Amazon S3 (detached, no nodes)
30–60 days
Older logs for on-demand investigation. Attach to UltraWarm when needed
S3-backed, 0 compute at rest
How the lifecycle behaves:
Hot → UltraWarm: ISM transitions indices at approximately 15 days. UltraWarm keeps primary shard data in S3 while caching hot blocks locally, so queries still run, only at higher latency and far lower cost per GB. Because rolled-over indices are no longer written to, they are force-merged before migration for faster warm-tier read performance.
UltraWarm → Cold: At approximately 30 days, indices detach to the cold tier, which is pure S3 storage with no dedicated compute. Cold indices are not directly searchable: to query them, you selectively attach them back to UltraWarm nodes on demand. (Querying the raw S3 archive without reattaching is a separate path that uses Amazon Athena.)
Compliance archive (S3 + Athena): Everything also lands in a separate Amazon S3 bucket for the full 7-year window, and is queried with Amazon Athena, at the lowest cost per GB. This decouples how long data must be kept from how much hot compute is paid for.
Durability details called out in the design: The durability of Amazon S3 means the design can run 0 replicas on the warm/cold tiers and remain durable (S3 handles redundancy), which further cuts cost. UltraWarm is a read-only warm tier, and Graviton (m8g) powers the hot and cluster-manager nodes.
Alerting. Hot-tier data drives OpenSearch Service alerting with email and Microsoft Teams destinations for real-time operational alarms.
Production sizing and cost model
Sizing target: 150 GB/day average, 60-day retention (30 days directly searchable) (the cluster is provisioned with headroom for an approximately 200 GB/day peak, which is the basis for the node sizing that follows). Rakbank evaluated two topologies:
Option 1 — Hot + UltraWarm (recommended)
3 × m8g.2xlarge hot data nodes.
3 × ultrawarm1.medium.search + cold tier.
3 × m8g.large dedicated cluster-manager nodes.
Approximately 55 percent savings compared to all-hot.
Option 2 — All Hot
3 × r8g.4xlarge, approximately 8,320 GB each.
3 × m8g.large admin, no warm tier.
Higher cost, all data resident hot.
Pricing is On-Demand. Reserved Instances reduce it further. The takeaway: for a 9 TB, 60-day window, the bulk of data is rarely queried after the first couple of weeks. Pushing older indices to an S3-backed warm/cold tier removes the need to keep the full working set on hot r8g nodes, which is where the approximately 55 percent delta comes from.
Scaling the observability architecture with OpenSearch Service
As the platform grew from approximately 5 GB/day to approximately 150 GB/day with 25 dashboards and approximately 25 concurrent users, the observability architecture had to meet three requirements:
Predictable cost at high ingestion volume, repeated dashboard queries without per-query cost coupling, and multi-tier retention spanning real-time access through to 7-year archival. OpenSearch Service meets these requirements with an instance-based model and hot/warm/cold tiering: ingestion and queries run on provisioned capacity, and older indices age into S3-backed tiers for long-term retention at a low cost per GB. This gives Rakbank a predictable, capacity-based cost model as data volume and dashboard usage grow.
Resilience: Multi-AZ and the Standby trade-off
Deployment option
SLA
Failover
Nodes
Multi-AZ (no Standby)
99.9% out-of-the-box
automatic
3 AZ · 3 data nodes + 3 dedicated cluster-manager nodes
Multi-AZ with Standby
99.99%
< 1 min automatic on AZ failure
3 AZ · 2 data active + 1 standby
The trade-off to understand: with Multi-AZ with Standby enabled, only two of three data nodes serve search requests. The third is reserved for failover. That is a 33 percent reduction in active search capacity in exchange for the higher service level agreement (SLA). Standby also requires at least two replicas per index, which increases storage, and is supported only on specific instance families, so check the current list before planning it. If a 99.99 percent SLA is not a hard requirement, running without Standby keeps all data nodes serving search. Rakbank runs its production domain without Standby, with data nodes across all three Availability Zones and replica shards for redundancy.
AI-powered observability capabilities
Rakbank is evaluating newer, machine learning (ML) and agent-driven features of OpenSearch Service that can layer on top of the tiered store:
AI Assistant – Natural-language → query generation (for example, “Show error spikes in the last 24h”), backed by Amazon Bedrock.
MCP Server + Agent investigator – DevOps agents autonomously execute PPL queries for root-cause analysis. A Memory Store provides persistent context across agent conversations.
Direct Query – Query Amazon S3 data without ingesting it into OpenSearch Service, an option alongside Amazon Athena for the S3 archive.
Trace Correlation – Available as trace collection is added to the pipeline: unify logs, traces, and metrics in one view, with automatic trace/log correlation and service maps.
Query surface – PPL, DSL, SQL, PromQL. Integrates with Amazon Managed Service for Prometheus and Amazon OpenSearch Ingestion for high-performance ingest.
Outcomes
Running on the Fluent Bit standard, the platform now serves three audiences from a single pane of glass. Support operations get faster triage and guided troubleshooting, engineering teams use it for root-cause analysis and debugging across the request path, and the business gains journey trends and actionable insights.
Net results: lower mean time to resolution (MTTR), higher reliability, data-driven insights, and improved operational efficiency. In their own environment, Rakbank measured faster performance after upgrading across OpenSearch Service versions (1.3 → current), with single-digit-millisecond latencies for lexical and vector queries.
Engineering lessons learned
Multi-tier from day one – Design hot/warm/cold up front. Retrofitting tiering onto a live 150 GB/day cluster is significantly harder than starting with ISM policies in place.
Size for peak, not today – Ingestion grows as more microservices come online. Provision headroom in the hot tier and shard count.
Model the Standby trade-off explicitly – 99.99 percent costs you 33 percent of active search capacity and requires two replicas per index. Decide per workload whether that SLA is worth it.
Exploit the append-only advantage – Append-only logs are the recommended candidate for multi-tier storage: no update overhead, clean force-merge before warm migration, and predictable rollover.
Marking their return to the HPC market, at SC26 NVIDIA has announced the Rubin HPC chip. The science and engineering-focused GPU will have over 50,000 CUDA cores and will be released in 2028
Zellij is a relatively new entrant in the terminal-multiplexer
category. Like GNU
Screen and tmux it gives
users the ability to manage a workspace inside a terminal with split panes,
provides session management, and so forth; but Zellij also includes a web server for
browser access to a terminal session, is extensible via a plugin API, and is likely
more beginner-friendly than its predecessors. Its most recent release, version 0.46,
adds a standalone terminal emulator, a built-in inbox for notifications, and a settings
screen so users can configure the application without fussing with
configuration files. While the low version number might suggest that it is not
yet ready for serious use, it seems stable and full-featured enough to
meet the needs of many users.
Approving an AI service is only the starting point. As employees adopt Claude and their responsibilities change, security teams need to know who has access, which users hold administrative privileges, and whether those roles and group memberships still reflect the work they do.
IBM’s 2025 Cost of a Data Breach Report found that, among studied organizations reporting an AI-related breach, 97% lacked proper AI access controls. The finding highlights a broader challenge: AI adoption needs security practices that extend beyond the initial approval, supported by information teams can use to review access with administrators and service owners.
Rapid7 Surface Command now integrates with Claude’s Compliance API, bringing supported organization, user, role, group, and settings information into the same platform teams use to understand other assets and identities. The integration helps security teams make enterprise Claude access part of their existing review practices.
Bring enterprise AI access into your security view
An account list tells security teams who is present, while roles and group memberships help explain how access is organized and where further review may be needed. Surface Command’s integration with Claude’s Compliance API brings together:
Organizations and users: Visibility into the Claude organization and its associated user accounts.
Roles and groups: Context to understand users’ assigned roles and group memberships.
Organizational settings: Available configuration information to support review of the enterprise Claude environment.
With those details in Surface Command, practitioners can examine Claude identities alongside the other assets and identities they manage. That gives them a clearer basis for working with service owners to evaluate whether access remains appropriate as teams and responsibilities change.
The integration focuses on administrative visibility. It does not ingest Claude conversation content, provide threat detection on Claude activity, or control Claude’s behavior.
From a user record to an informed access review
Consider a security practitioner reviewing administrative access to Claude. In Surface Command, they can inspect an imported user’s details, review their organization role and administrative status, and examine their group memberships. The relationship graph provides a visual view of how that user connects to the organization and its groups, making those associations easier to follow.
The practitioner can then work with the service owner to confirm whether administrative privileges are still needed, whether group memberships reflect current responsibilities, and whether those assignments align with the organization’s access model.
Surface Command provides information to support the review, while the team applies its policies and judgment to determine whether a change is needed.
Support AI adoption with clearer accountability
As Claude becomes part of everyday work, reviewing its users and access relationships becomes part of managing the environment. Bringing that information into Surface Command helps security practitioners and service owners work from a shared understanding of assigned roles and group memberships. It supports more informed access decisions and helps security leaders incorporate enterprise AI into established oversight practices.
Get started with Claude and Surface Command today
The integration is available through the Anthropic connector in the Rapid7 Extension Library. Using Claude’s Compliance API requires a Compliance API key and the appropriate entitlement for your Claude organization.
Visit the connector page to get started and see how Surface Command brings Claude users, roles, and group relationships into view.
Greg Kroah-Hartman has announced the 7.3.5, 6.18.64, and 6.12.118 stable kernels. Each contains quite a few
important fixes.
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.