Бели пълномощни за черни дни

Post Syndicated from Светла Енчева original https://www.toest.bg/beli-pulnomoshtni-za-cherni-dni/

Бели пълномощни за черни дни

Преди трийсетина години застрахователна агентка разяснявала на моя роднина ползите от застраховката „Живот“. „Ако настъпи застрахователното събитие…“, започнала агентката. Роднината ми я прекъснала насред изречението с въпроса кое е „застрахователното събитие“. „Ами ако умрете“, бил отговорът. Възмутена, роднината ми станала и си тръгнала: „Ще ми казва тя на мене, че ще умра!“

Ако и вие като тази ми роднина не искате да ви се напомня, че сте смъртни и може да ви сполети тежко увреждане на здравето, тук е мястото да престанете да четете. Защото ще става дума за смърт, тежки болести и прочее неприятни сценарии.

Ако обаче за вас е важно да знаете как да се погрижите за най-близките си хора, в случай че се споминете или излезете от строя, и как да направите така, че волята ви да бъде изпълнена, тази статия е за вас.

„Да, човекът е смъртен, но това не е най-страшното. Лошото е, че той понякога е внезапно смъртен, това е неприятното!“, казва героят от „Майстора и Маргарита“ на Михаил Булгаков. Може да се поспори дали внезапната смърт е по-неприятно нещо от продължаващите с години палиативни грижи, особено на фона на данните, че ¼ от хората вероятно ще развият деменция. Възможни са и други неочаквани сценарии.

Ето два примера как всичко може да се обърка

Мой познат почина. Той беше изразил волята си как да се постъпи с тленните му останки, но не я беше оформил с необходимите юридически атрибути. Човекът, който имаше право да реши дали да се съобрази с последната му воля, беше майка му. А тя не приемаше факта, че синът ѝ е гей. Дори след смъртта му. Затова не само не изпълни желанието му какво да се направи с тялото, ами не допусна партньора и приятелите му, които го обичаха такъв, какъвто е, да се простят с него.

В подобно положение може да се окаже човек не само ако е мъртъв гей, а и ако е хетеросексуален и жив. И то не само в България. Мой скъп приятел от друга европейска страна тъкмо се беше събрал отново с дългогодишната си приятелка и ми пишеше колко е щастлив… а след няколко месеца научих, че лежи в болница в много тежко състояние, след мозъчна операция. Право на достъп до него има само майка му, която отказва всякаква информация на партньорката и на приятелите му. Така че не мога да ви кажа дали този човек, на чийто блестящ ум съм се възхищавала близо 20 години, днес изобщо знае кой е.

Въпроси, на които да си отговорим

Идеята на тази статия не е да ви плаша, а да дам идеи как човек може да защити себе си и близките си, ако различни ситуации станат реалност. Като начало можем да се опитаме да си отговорим на следните въпроси:

В случай на смърт:

  • Какво искам да стане с тленните ми останки?
  • Кой и как искам да ме наследи?

В случай на неспособност да се грижа за себе си:

  • Кой да има право да получава медицинска информация за мен?
  • На кого имам доверие да решава дали лечението ми да се продължи, или прекрати, дали да ме оперират, интубират и пр.?
  • Кой да има достъп до парите ми?
  • Кой да може да се разпорежда с договора ми с телекомуникационната компания, с битовите ми сметки, да подава декларации от мое име и т.н.?

В България много хора не се женят: някои – защото са в хомосексуални връзки и нямат право, но повечето просто не вярват в институцията на брака. Родителите на близо две трети (61,9%) от родените през 2024 г. деца не са женени. Ако единият партньор почине, а семейното жилище е на негово име, другият може да бъде изгонен от законните наследници на половинката си.

Ако живеете в брак, ситуацията е една идея по-лека – в общия случай поне няма да имате грижа за наследството си. Съпругът или съпругата ви има правото да получава информация за вас в болницата и да взема решения в критични ситуации. Ако не сте в брак, решенията се вземат от най-близките ви роднини. И в двата случая обаче никой не получава автоматичен достъп до финансите ви, договорите ви за телефон, ток, вода и прочее, ако сте на легло и не сте в състояние да се разпореждате с тях.

Съществуват различни начини да подсигурите себе си и близките си

Човек може да остави например имуществото си на най-близките си хора (в случаите, когато няма автоматично да ви наследят – било изобщо, било както вие искате) по два начина – със завещание и дарение (във втория случай е желателно да си запазите т.нар. вещно право на ползване). При дарение обаче получателят плаща данък, а при завещание плаща само ако не ви е близък роднина. За разлика от дарението обаче, завещанието по-лесно може да се оспори – още повече че някои ваши роднини имат запазени части от наследството ви. Затова е добре човек да се погрижи завещанието му да е юридически изпипано, за да се сведат до минимум рисковете някой да го оспори.

Всички останали въпроси, зададени по-горе, може да се уредят с пълномощни. Може да предоставите достъп до парите си и с банкова карта, но той ще важи само докато изтече срокът на валидност на картата.

Тук не засягам сериозната тема за родителските права. Ако те са неуредени (например единият родител не фигурира в акта за раждане на детето, нито го е осиновил), проблемът може да се реши само по съдебен път. Съдът обаче може и да откаже да ги присъди. А ако родителите са еднополова двойка, понастоящем в България няма начин този, който не е биологичен родител и не е осиновител, да получи права върху детето, каквото и участие да е вземал в отглеждането му. Не е разрешено и двама души от един пол да осиновят дете.

Доверие, доверие, доверие

Когато даваме пълномощно на някого да се разпорежда с парите ни или да взема съдбовни решения за живота и смъртта ни, желателно е да го познаваме достатъчно добре. Винаги има доза риск – случва се някой, на когото сме вярвали безусловно, да не оправдае доверието ни. А и връзки се разпадат, бракове се разтрогват.

Съществува вариант да оттеглим пълномощното си, но той не е безпроблемен. На първо място, упълномощеното лице трябва да разбере за оттеглянето. Ако междувременно обаче то е прехвърлило апартамента ни на свое име и е избягало с парите ни, ще е трудна задача да го намерим, за да го уведомим. Изобщо, лесно е да се ожените и може да е много трудно да се разведете. Така е и с пълномощните.

Възможно е и да направите пълномощно, което е за определен срок и/или само за конкретни неща – например някой да има право да получава информация за вас в болницата, но да няма достъп до банковите ви сметки. Или да може да сключи една имотна сделка от ваше име, но не и да придобие семейния ви дом. На теория може да предоставите пълномощно и при определени условия – примерно, тежка болест. Но после упълномощените ще трябва да доказват здравословното ви състояние, а това е трудна задача, понякога невъзможна – особено ако става въпрос за деменция.

Колко генерално е генералното пълномощно?

Да допуснем, че в живота ви има човек, на когото основателно имате пълно доверие, че няма да злоупотреби, а ще се погрижи за вас и ще се съобрази с волята ви, ако например се разболеете тежко, не можете да плащате сметки и т.н. Може би и вие като мен сте чували словосъчетанието „генерално пълномощно“ и ще решите, че е добра идея да снабдите въпросния човек с него.

Да си призная, до неотдавна си представях въпросното генерално пълномощно като нещо съвсем просто и кратко: Аз, долуподписаната еди-коя си, упълномощавам лицето еди-кое си да ме представлява навсякъде и във всичко. Заверявам при нотариус – и готово.

Да, ама не. В пълномощното всичко трябва да бъде описано възможно най-подробно. Затова е добре човек да изреди всички места, където би имало нужда от пълномощно, и да уточни какво точно може да прави упълномощеното лице във всяка една ситуация. Например в болница – да получава информация, да взема решения за продължаване, прекратяване или вида на лечението ви, да има достъп до медицинската ви документация и т.н.

Добра идея е да се консултирате с юрист

Опитът на позната дългогодишна адвокатка например я е научил, че много институции и фирми държат да видят собственото си име в пълномощното. Затова е желателно в него да изредите конкретни телекомуникационни компании, водоснабдителни, топло- и електроразпределителни дружества, общински администрации и изобщо каквото се сетите (с риска, че все може да забравите нещо). Ако междувременно някоя компания смени името си, може само да се надявате, че служителите ѝ няма да се заядат.

Юридическата помощ е важна и за оформянето на структурата на завещанието, както и за използването на определени ключови фрази, за които неспециалист трудно ще се сети сам. Добре е например в прав текст да кажете, че лекарската тайна не важи пред упълномощеното лице. Ако пък искате да ви кремират след смъртта ви, желателно е да подчертаете, че давате изричното си съгласие за това.

Ако пълномощното е безсрочно, то важи и ако си смените личната карта и/или адреса, вписани в него. Не е изключено обаче някой да се заяде, че данните не отговарят.

Банките са отделен случай – повечето изискват специални пълномощни

За да избегна непрекъснатото повтаряне на упълномощител и упълномощено лице, ви представям двама души с полово неспецифични имена – Тони упълномощава Краси. Ако Тони състави генералното пълномощно по всички правила, спомене в него банката си и го завери нотариално, по всяка вероятност банката няма автоматично да го признае. Различните банки си имат свои критерии за пълномощното, но при всички случаи е желателно да разполагат с него предварително, а не то да им се представи чак когато има нужда от това.

По-долу обобщавам личния опит на няколко души с три различни банки. Тъй като става дума за конкретни случаи, информацията за правилата на тези финансови институции може и да не е изчерпателна.

Пощенска банка изисква Тони и Краси да се явят заедно в неин клон. Тя има свои вътрешни пълномощни, които се вкарват в системата ѝ и не са нотариално заверени. Възможно е Тони да представи и нотариално пълномощно. В такъв случай плаща такса от 5 лв. за достъп до базата данни на нотариусите, а Краси трябва да носи пълномощното със себе си, ако се наложи да го използва.

Тони и Краси трябва да отидат заедно и в Обединена българска банка, която също има свои вътрешни пълномощни. Те съдържат дълъг списък, в който Тони трябва да отбележи за какво упълномощава Краси и за какво – не. Ако Тони обаче иска да даде на Краси достъп до депозитите или инвестиционните си фондове, пълномощното трябва да се завери нотариално.

Първа инвестиционна банка разполага с различни модели пълномощни, от които Тони да избере подходящото, да го попълни и да го завери нотариално. И Краси може да занесе пълномощното в банката, но ще трябва да плати такса, а ако го направи Тони, е безплатно. Не се изисква и двамата да отидат заедно в клона, но ако Краси не е клиент на банката, ще трябва да представи копие от личната си карта.

Ако Тони разполага с банков сейф обаче, пълномощното за достъп на Краси до него може да се издава по съвсем различен ред от обичайните пълномощни на съответната банка.

Защо банките си имат собствени правила и не може просто да им представите нотариално заверено генерално пълномощно, колкото и подробно да е то – не знам. И не ми се вижда редно да сте заявили волята си по юридически коректен начин, а определени частни фирми да решат, че това не е достатъчно. Но все пак банките са подобрили практиките си в сравнение с 2019 г. – поне тези, за които имам информация, не изискват ново пълномощно всяка година. А служителите им казват, че преиздаване не се налага, ако си смените личната карта.

С едно наум за бъдещето

„Ние безгрижно тичаме право към пропастта, изпречили нещо пред себе си, за да не я виждаме“, пише Блез Паскал в своите „Мисли“. Каква по-точна метафора на живота? Ако човек непрекъснато мисли, че е смъртен и какви неприятни неща очакват него и скъпите му хора, всекидневието ще се превърне в кошмар. Отказът да си дадем сметка за тези перспективи обаче може да доведе до куп проблеми.

Все пак има начини екзистенциалният ужас от смъртта, болестите и злополуките да бъде поне в известна степен опитомен. Един от тях е да се опитаме да направим каквото зависи от нас, за да се погрижим за себе си и близките си, докато е време. С ясното съзнание, че не можем всичко да предвидим. Не знаем дори дали слънцето ще изгрее утре. Но междувременно можем поуспокоени да продължим да се радваме на живота.

Young creators build a bright future at Coolest Projects India 2025

Post Syndicated from Vasu Srivastava original https://www.raspberrypi.org/blog/young-creators-build-a-bright-future-at-coolest-projects-india-2025/

Coolest Projects India is officially a wrap, and we are absolutely thrilled with the results! The event, held on 20 September 2025 in Hyderabad, Telangana, was a fantastic showcase of the incredible talent and passion of young creators. With over 250 participants, young creators showcased 118 projects, demonstrating the thriving digital making community across India.

The projects on display were truly inspirational, tackling important topics such as environmental sustainability and the responsible use of AI. From hardware to Scratch to AI, creators showcased an incredible variety of projects that brought their ideas to life, from simple creations to complex inventions.

Here are just a few of the standout projects from the day.

Young creators at Coolest Projects India.
Avyukt, 9 years old | The EchoCharm Spellbox, Hardware category

A truly magical project was The EchoCharm Spellbox, an interactive creation that used a Raspberry Pi computer and a speaker to bring the famous Harry Potter artifact to life. This project was a perfect blend of creativity and coding, showcasing how technology can be used for fun and entertainment.

Young creators at Coolest Projects India.
Aahil, 11 | Automatic Seed Sowbot, Hardware category

We were also blown away by the Automatic Seed Sowbot, a practical invention that used a tiny camera and AI to quickly and accurately sort seeds. This clever hardware project tackles a real-world problem and has fantastic applications in agriculture and farming.

Young creators at Coolest Projects India.
Pranamya, Medha, Manvi, Aradhna — these 13-year-old-girls all come from Karnataka state | Unspoken Truths, Mobile apps category

Another brilliant project that we saw was Unspoken Truths, a fun mobile app for interesting conversations to reveal hidden truths and secrets about yourself and your friends. If you’re ever looking to break the ice or deepen your connections, Unspoken Truth provides an entertaining way to start a conversation.

A community of creators

From first-time coders to seasoned programmers, every young person showcasing their work had the chance to celebrate their ideas and feel part of a vibrant community. The event created a unique atmosphere where participants were able to proudly share their projects with family, friends, and the wider coding community. They all received certificates to celebrate their fantastic achievements.

This year’s showcase brought together creators from different parts of the country, highlighting the incredible passion for digital making in India. It was a powerful reminder that when you put technology into the hands of young people with curiosity, there is no limit to what they can create.

A massive thank you to our organisers, volunteers, and the dedicated mentors who made the event possible. And most importantly, thank you to the young creators of India for sharing their ingenuity and creativity with us! We can’t wait to see what you’ll build next.

Thank you to our sponsors, Qube Research and Technologies, for sponsoring the hardware category.

Want to get involved?

Coolest Projects will be back online and with more in-person events in India and around the world in 2026. Sign up to the newsletter to be the first to hear about dates, deadlines, and exciting updates.

The post Young creators build a bright future at Coolest Projects India 2025 appeared first on Raspberry Pi Foundation.

[$] LWN.net Weekly Edition for October 9, 2025

Post Syndicated from corbet original https://lwn.net/Articles/1040456/

Inside this week’s LWN.net Weekly Edition:

  • Front: Kernel Rust features; systemd v258, part 2; Cauldron kernel hackers; BPF for GNU tools; 6.18 merge window, part 1; Lifetime-end pointer zapping; Robot Operating System.
  • Briefs: OpenSSH 10.1; Firefox profiles; Python 3.14; U-Boot v2025.10; FSF presidency; Quotes; …
  • Announcements: Newsletters, conferences, security updates, patches, and more.

How to configure and verify ACM certificates with trust stores

Post Syndicated from Chris Morris original https://aws.amazon.com/blogs/security/how-to-configure-and-verify-acm-certificates-with-trust-stores/

In this post, we show how to configure customer trust stores to work with public certificates issued through AWS Certificate Manager (ACM). Organizations can encounter challenges when configuring trust stores for ACM certificates and incorrect trust store configuration can lead to SSL/TLS errors and application downtime. While most modern web browsers and operating systems trust ACM certificates by default, understanding how this trust is established and verifying proper configuration is important for IT professionals and developers. We also describe the relationship between public certificates issued through ACM and Amazon Trust Services. Whether you’re developing applications that connect to endpoints using ACM certificates or managing systems with customer trust stores that need to trust ACM certificates, this guide will provide you with insight regarding ACM certificate trust.

Background

ACM is a managed service that you can use to provision, manage, and deploy public and private SSL/TLS certificates. When you visit a website over HTTPS that has an ACM certificate, most modern web browsers will show a Connection is secure message in the address bar. This indicates that the web browser trusted the certificate. ACM certificates are trusted by popular browsers such as Chrome, Firefox, and Safari because they are issued by Amazon Trust Services, a public certificate authority (CA) managed by Amazon, whose root CA certificates are included by default in most web browsers’ and operating systems’ trust stores.

What is a trust store?

Web browsers, devices, and applications trust a collection of certificates known as CA certificates. These collections of CA certificates are called trust stores. Most often, the CA certificates in a trust store are root CA certificates. Root CA certificates are CA certificates that act as the foundation of trust. It’s best practice that root CAs issue intermediate CA certificates, which then issue end-entity certificates to minimize interaction with the root CA. When navigating to a website protected with HTTPS using a web browser, the website will present the end-entity certificate and the certificate chain. The certificate chain is a series of certificates, each issued by the next, leading back to a root CA certificate. The web browser will then check the end-entity certificate. It will make sure it’s derived from a root certificate that is in its trust store. It is important to note that trust store configurations can vary depending on the web browser, device or application.

Amazon Trust Services

Amazon Trust Services is a publicly trusted CA that is managed by Amazon. Amazon Trust Services root CA certificates are included in the trust stores of most web browsers and operating systems. As shown in Figure 1, when you request a public ACM certificate through DNS, Email, or HTTP validation, it will be issued by one of the multiple intermediate CAs that Amazon manages. These intermediate CAs are issued by one of the five Amazon Trust Services root CAs. Therefore, by trusting the Amazon Trust Services root CAs, you will be trusting ACM certificates. It’s important to note that ACM uses a dynamic intermediate CA model. This means you cannot predict which specific intermediate CA will issue an ACM certificate. The issuing intermediate CA is selected dynamically from a group of intermediate CAs at the time of certificate issuance. This means that the intermediate CA that issues ACM certificates is non-deterministic. In summary, we recommend customer trust stores include the five Amazon Trust Services root CA certificates. This includes Amazon Root CA 1, Amazon Root CA 2, Amazon Root CA 3, Amazon Root CA 4 and Starfield Services Root Certificate Authority – G2.

Figure 1 – ACM certificate chain

Figure 1 – ACM certificate chain

Best practices

To help establish reliable HTTPS connections to endpoints using ACM certificates, we recommend that your trust stores include the five Amazon root CAs.

Distinguished name of Amazon root CA SHA-256 hash of subject public key information URL to root CA certificate in DER or PEM format
CN=Amazon Root CA
1,O=Amazon,C=US
fbe3018031f9586bcbf41727e417b7d1c45c2f47f93be372a17b96b50757d5a2 DER, PEM
CN=Amazon Root CA
2,O=Amazon,C=US
7f4296fc5b6a4e3b35d3c369623e364ab1af381d8fa7121533c9d6c633ea2461 DER, PEM
CN=Amazon Root CA
3,O=Amazon,C=US
36abc32656acfc645c61b71613c4bf21c787f5cabbee48348d58597803d7abc9 DER, PEM
CN=Amazon Root CA
4,O=Amazon,C=US
f7ecded5c66047d28ed6466b543c40e0743abe81d109254dcf845d4c2c7853c5 DER, PEM
CN=Starfield Services Root Certificate Authority – G2,O=Starfield Technologies\, Inc.,L=Scottsdale,ST=Arizona,C=US 2b071c59a0a0ae76b0eadb2bad23bad4580b69c3601b630c2eaf0613afa83f92 DER, PEM

Adding the five Amazon root CAs provide maximum compatibility for trusting ACM certificates. If you must use certificate pinning in your application, we recommend that you pin to the public key of the mentioned root CAs.

While addressing the best practices, it is important to review how trust stores should not be configured.

Don’t limit your trust stores to only the intermediate CA certificates that issue ACM certificates. Examples of such intermediate CAs include Amazon RSA 2048 M01, Amazon RSA 2048 M02, Amazon RSA 2048 M03. Adding only these intermediate CA certificates to your trust store will introduce risk to your application. This is because of the dynamic intermediate CA (ICA) model. When an ACM certificate is issued or when it’s renewed, it will be from one of the many intermediate CAs. Furthermore, they are non-deterministic. If an ACM certificate was first issued by Amazon RSA 2048 M01, there is no guarantee that it will renew from that same intermediate CA.

In summary, here are the best practices for trusting ACM certificates.

How do I verify that the Amazon root CAs are in my trust store?

As mentioned in the previous section, most modern web browsers and operating systems already include the five Amazon root CAs in their respective trust stores by default. It’s still recommended to verify that the Amazon root CAs are installed correctly. It’s important to note that many applications have different trust store locations. For example, an application might use the Windows trust store location—Trusted Root Certification Authorities—as its trust store or it might use a PEM trust store in a custom directory. This is why we recommend that you review your application’s trust store documentation.

To verify, check your system’s trust store for existing Amazon root CA certificates. If they are not present, you can proceed with adding the five Amazon root CA certificates.

Windows: Check for the Amazon root CAs in Windows operating systems (GUI)

  1. Press Windows + R, enter certmgr.msc , then press Enter.
  2. Go to Trusted Root Certification Authorities and choose Certificates.
Figure 2: Windows certificate store: Trusted Root Certification Authorities

Figure 2: Windows certificate store: Trusted Root Certification Authorities

Check for the Amazon root CAs in Windows operating systems (CLI)

You can use Powershell to check for the Amazon root CAs. Use the certutil command.

  • Open Windows Powershell and use the following certutil commands. These will search for the five Amazon root CAs.
> certutil -store AuthRoot | findstr /i "Amazon" 
Issuer: CN=Amazon Root CA 4, O=Amazon, C=US 
Subject: CN=Amazon Root CA 4, O=Amazon, C=US 
Issuer: CN=Amazon Root CA 1, O=Amazon, C=US 
Subject: CN=Amazon Root CA 1, O=Amazon, C=US 
Issuer: CN=Amazon Root CA 2, O=Amazon, C=US 
Subject: CN=Amazon Root CA 2, O=Amazon, C=US 
Issuer: CN=Amazon Root CA 3, O=Amazon, C=US 
Subject: CN=Amazon Root CA 3, O=Amazon, C=US

> certutil -store AuthRoot | findstr /i "Starfield Services Root Certificate Authority - G2" 
Issuer: CN=Starfield Services Root Certificate Authority - G2, O=Starfield Technologies, Inc., L=Scottsdale, S=Arizona, C=US
Subject: CN=Starfield Services Root Certificate Authority - G2, O=Starfield Technologies, Inc., L=Scottsdale, S=Arizona, C=US

Add Amazon root CAs to the default trust store using the UI

Download each Amazon Trust Services root CA. You can select the DER or PEM versions.

  1. Open Certmgr: Press Windows + R, enter certmgr.msc, and press Enter.
  2. Add to the trusted root:
    1. Choose Trusted Root Certification Authorities.
    2. Right-click Certificates.
    3. Select All Tasks and choose Import.
    4. Follow the Certificate Import Wizard:
      1. Choose Next.
      2. Browse to the root CA certificate file location. You might need to select All Files(*.*) to view the root CA certificate files.
      3. Select Place all certificates in the following store.
      4. Verify Trusted Root Certification Authorities is selected and choose Next.
      5. Choose Finish.

Add Amazon root CAs to the default trust store using the CLI

  1. Download each Amazon Trust Services root CA. You can select the DER or PEM versions.
  2. In Powershell, add a CA certificate to AuthRoot using certutil.
    > certutil -addstore AuthRoot AmazonRootCA1.cer
  3. In Powershell, verify that the certificate has been added.
    > certutil -store AuthRoot | findstr /i "Amazon"

Amazon Linux 2023: Check for the Amazon root CAs in default trust store

The following is the default location for the system trust store in Amazon Linux 2023:

/etc/pki/tls/certs/ca-bundle.crt

1. Using OpenSSL, search for Amazon root CA certificates in the ca-bundle.crt bundle:

openssl crl2pkcs7 -nocrl -certfile /etc/pki/tls/certs/ca-bundle.crt | openssl pkcs7 -print_certs -noout | grep -i "Amazon\|Starfield Services" 

subject=C=US, O=Amazon, CN=Amazon Root CA 1 
issuer=C=US, O=Amazon, CN=Amazon Root CA 1 
subject=C=US, O=Amazon, CN=Amazon Root CA 2 
issuer=C=US, O=Amazon, CN=Amazon Root CA 2 
subject=C=US, O=Amazon, CN=Amazon Root CA 3 
issuer=C=US, O=Amazon, CN=Amazon Root CA 3 
subject=C=US, O=Amazon, CN=Amazon Root CA 4 
issuer=C=US, O=Amazon, CN=Amazon Root CA 4 
subject=C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2 
issuer=C=US, ST=Arizona, L=Scottsdale, O=Starfield Technologies, Inc., CN=Starfield Services Root Certificate Authority - G2

To add the Amazon root CAs to the default trust store

1. Navigate to the following directory for adding CA certificates
$ cd /etc/pki/ca-trust/source/anchors/

2. Using cURL, download each Amazon Trust Services root CA in the preceding folder. Do this for each of the Amazon root CAs replacing the name of the PEM file as needed.

$ sudo curl -O
https://www.amazontrust.com/repository/AmazonRootCA1.pem

3. Add the root CAs by updating the system trust store.
$ sudo update-ca-trust extract

4. Verify that the bundle has been updated with OpenSSL.
$ openssl crl2pkcs7 -nocrl -certfile /etc/pki/tls/certs/ca-bundle.crt | openssl pkcs7 -print_certs -noout | grep -i "Amazon\|Starfield Services"

Java: Check for the Amazon root CAs in a Java trust store (Java Keystore)

Many custom Java applications use Java Keystore (JKS) as a trust store. You can use the keytool CLI tool to verify if the Amazon root CAs exist in your JKS trust store.

keytool -list -keystore custom_truststore.jks -storepass mypassword

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 5 entries

amazonrootca1, Jun 27, 2025, trustedCertEntry, Certificate fingerprint (SHA-256): 8E:CD:E6:88:4F:3D:87:B1:12:5B:A3:1A:C3:FC:B1:3D:70:16:DE:7F:57:CC:90:4F:E1:CB:97:C6:AE:98:19:6E 
amazonrootca2, Jun 27, 2025, trustedCertEntry, Certificate fingerprint (SHA-256): 1B:A5:B2:AA:8C:65:40:1A:82:96:01:18:F8:0B:EC:4F:62:30:4D:83:CE:C4:71:3A:19:C3:9C:01:1E:A4:6D:B4 
amazonrootca3, Jun 27, 2025, trustedCertEntry, Certificate fingerprint (SHA-256): 18:CE:6C:FE:7B:F1:4E:60:B2:E3:47:B8:DF:E8:68:CB:31:D0:2E:BB:3A:DA:27:15:69:F5:03:43:B4:6D:B3:A4 
amazonrootca4, Jun 27, 2025, trustedCertEntry, Certificate fingerprint (SHA-256): E3:5D:28:41:9E:D0:20:25:CF:A6:90:38:CD:62:39:62:45:8D:A5:C6:95:FB:DE:A3:C2:2B:0B:FB:25:89:70:92 
starfieldg2, Jun 27, 2025, trustedCertEntry, Certificate fingerprint (SHA-256): 56:8D:69:05:A2:C8:87:08:A4:B3:02:51:90:ED:CF:ED:B1:97:4A:60:6A:13:C6:E5:29:0F:CB:2A:E6:3E:DA:B5

The output should show the Amazon root CAs listed as “trustedCertEntry” with those exact certificate fingerprints.

To add the Amazon root CAs to a Java trust store (Java Keytool)

1. Download each Amazon Trust Services root CA in PEM or DER format. Use the PowerShell command Invoke-WebRequest if you’re using Windows, or use cURL if you’re using a Linux-based operating system or MacOS.

> Invoke-WebRequest -Uri "https://www.amazontrust.com/repository/AmazonRootCA1.pem" -OutFile "AmazonRootCA1.pem"

$ curl -O https://www.amazontrust.com/repository/AmazonRootCA1.pem

2. Import the Amazon root CAs to the trust store—custom_truststore.jks. Replace changeit with your JKS password. Do this command for each of the Amazon root CAs, replacing the name of the root CA as needed.

$ keytool -importcert -alias "AmazonRootCA1" -file "AmazonRootCA1.pem" -keystore custom_truststore.jks -storepass changeit -trustcacerts -noprompt

Test your trust store configuration

After you have set up your trust store with the five Amazon root CA certificates, you can perform tests to confirm that the installed root CAs are correctly providing trust. Remember that your custom application might be sourcing its trust from a store other than the stores mentioned in this article. For custom applications, we recommend checking your testing documentation.

PEM

For operating systems or applications that use PEM certificate bundles, such as Amazon Linux 2023, you can use OpenSSL or cURL to test. For additional test URLs, see the Amazon Trust Services website. Replace CAbundle.pem with your certificate bundle.

$ openssl s_client -connect valid.rootca1.demo.amazontrust.com:443 -CAfile CAbundle.pem

$ curl -iv --cacert CAbundle.pem https://valid.rootca1.demo.amazontrust.com

Windows

Because Windows doesn’t use PEM certificate bundles, but a trust store in certmgr called Trusted Root Certification Authorities, you can use PowerShell to test.

1. Copy the following PowerShell script and save it in a file named ssl-connect.ps1.


param (
[string]$url = "https://valid.rootca1.demo.amazontrust.com"
)

$sslStream = $null
$tcpClient = $null

try {
$uri = [System.Uri]$url
$hostname = $uri.Host
$port = if ($uri.Port -eq -1) { 443 } else { $uri.Port }

# Connect to the server
$tcpClient = New-Object System.Net.Sockets.TcpClient
$tcpClient.Connect($hostname, $port)

# Define the certificate validation callback
$callback = {
param($sender, $certificate, $chain, $sslPolicyErrors)

Write-Host "Server Certificate:`nSubject : $($certificate.Subject)`nIssuer : $($certificate.Issuer)`n"

Write-Host "Certificate Chain:"
foreach ($c in $chain.ChainElements) {
Write-Host ("Subject : {0}`nIssuer : {1}`nThumbprint : {2}`n" -f
$c.Certificate.Subject,
$c.Certificate.Issuer,
$c.Certificate.Thumbprint)
}


if ($sslPolicyErrors -eq 'None') {
Write-Host "Certificate is valid and trusted."
} else {
Write-Host "Certificate error(s): $sslPolicyErrors"
}

return $true
}

# Create the SSL stream using the callback
$sslStream = New-Object System.Net.Security.SslStream($tcpClient.GetStream(), $false, $callback)

# Initiate TLS handshake
$sslStream.AuthenticateAsClient($hostname)
}
catch {
Write-Host "ERROR: $($_.Exception.Message)"
}
finally {
if ($sslStream) { $sslStream.Dispose() }
if ($tcpClient) { $tcpClient.Close() }
}

2. Run the PowerShell script with the following command:

  • > .\ssl-connect.ps1

You can test with the other test URLs by passing them in -url:

  • > .\ssl-connect.ps1 -url https://s3.amazonaws.com

3. After running the command, you should see the subject and issuer of the end-entity certificate and the full trust chain, including the intermediate CA and root CA. If the command returns Certificate is valid and trusted, the certificate is trusted. If it returns an error with Certificate error, the error should tell you what went wrong.

Java

To test your Java applications that use JKS as a trust store, you can make HTTPS connections to endpoints that use Amazon Trust Services certificates.

1. Copy the Java code and name the file SSLTester.java.

  • In the code, you can replace the urls variable with additional URLs to test HTTPS. See the Amazon Trust Services website for additional test URLs.
  • Update your_keystore.jks and your password with your JKS file path and password.
import javax.net.ssl.SSLContext; 
import javax.net.ssl.TrustManagerFactory; 
import java.io.FileInputStream; 
import java.net.URL; 
import java.security.KeyStore;
import java.security.cert.Certificate; 
import java.security.cert.X509Certificate; 

public class SSLTester {
     public static void main(String[] args) {
         // Enable revocation checking
         System.setProperty("com.sun.net.ssl.checkRevocation", "true");
         System.setProperty("com.sun.security.enableCRLDP", "true");   
         System.setProperty("com.sun.security.enableAIAcaIssuer", "true");
         // Define your HTTPS URLs here
         String[] urls = {
              "https://valid.rootca1.demo.amazontrust.com/",  // Use an Amazon Trust Services Valid test URL (Example: https://valid.rootca1.demo.amazontrust.com/)
              "https://revoked.rootca1.demo.amazontrust.com/", // Use an Amazon Trust Services Revoked test URL (Example: https://revoked.rootca1.demo.amazontrust.com/)
              "https://expired.rootca1.demo.amazontrust.com/", // Use an Amazon Trust Services Expired test URL (Example: https://expired.rootca1.demo.amazontrust.com/)
              "https://ec2.amazonaws.com" // AWS Service Endpoint
		  };
          String keystorePath = "your_keystore.jks"; // Define your .jks file
          String keystorePassword = "your password"; // Pass your keystore password

          try {
             // Load the JKS
             KeyStore trustStore = KeyStore.getInstance("JKS");
             FileInputStream fis = new FileInputStream(keystorePath);
             trustStore.load(fis, keystorePassword.toCharArray());
             fis.close();

             // Initialize TrustManagerFactory with JKS
             TrustManagerFactory tmf = 
TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
             tmf.init(trustStore);
            // Initialize SSLContext
            SSLContext sslContext =
SSLContext.getInstance("TLS");
             sslContext.init(null, tmf.getTrustManagers(), null);

             // Test SSL connections to URLs
             for (String urlStr : urls) {
                 System.out.println("Testing URL: " + urlStr);
                 try {
                     URL url = new URL(urlStr);
                     javax.net.ssl.HttpsURLConnection conn = (javax.net.ssl.HttpsURLConnection) url.openConnection();                    conn.setSSLSocketFactory(sslContext.getSocketFactory());
                     conn.connect();

                     // Get server certificate
                     Certificate[] certs = 
conn.getServerCertificates();
                     for (Certificate cert : certs) {
                         if (cert instanceof X509Certificate) {
                             X509Certificate x509Cert = (X509Certificate) cert;
                             System.out.println("Certificate: " + x509Cert.getSubjectDN());
                         }
                     }
                     System.out.println("Connection successful for " + urlStr);
                     conn.disconnect();
                 } catch (Exception e) {
                     System.err.println("Failed for " + urlStr + ": " + e.getMessage());
                 }
             }
         } catch (Exception e) {
             e.printStackTrace();
         }
     }
 }

2. After you save the file, compile it and run.

javac SSLTester.java
java SSLTester.java

3. Check the output after it’s finished running.

  • For Valid URLs, you should see Connection successful:
  • Connection successful for https://valid.rootca1.demo.amazontrust.com/

  • For Revoked URLs, you should see Certificate has been revoked:
  • failed: java.security.cert.CertPathValidatorException: Certificate has been revoked, reason: UNSPECIFIED

  • For Expired URLs, you should see Validity check failed:

Failed for https://expired.rootca1.demo.amazontrust.com/: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

Conclusion

When your web browser, device, or application performs HTTPS connections, it validates the certificate presented by the server using its trust store. A trust store is a collection of trusted CA certificates, primarily consisting of root CA certificates. When trusting endpoints using public certificates issued through ACM, best practice recommends installing the five Amazon Trust Services root CA certificates into your trust store. Be aware that trusting only the Amazon Trust Services intermediate CA certificates, such as Amazon RSA 2048 M01 and Amazon RSA 2048 M02, increases your application’s risk for outages. This is because of the non-deterministic nature of the dynamic intermediate CA (ICA) model. It’s worth noting that trust store configurations can vary across different applications. Furthermore, applications can also source their trust store from different locations. For example, you can have a Java application hosted on a Windows-based operating system that sources its trust store from a Java Keystore (JKS) file rather than the default Windows trust store location Trusted Root Certification Authorities. This means that you should thoroughly test your application after installing the Amazon Trust Services root CA certificates in your trust store. This will help to sustain reliable HTTPS connections to endpoints using ACM certificates.


If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Chris Morris

Chris Morris

Chris is a Sr. Cloud Support Engineer at AWS. He specializes in a variety of security topics, including cryptography and data protection. He focuses on helping AWS customers use AWS security services to strengthen their security posture in the cloud. Public key infrastructure and key management are some of his favorite security topics.

Feng Chen

Feng Chen

Feng is an AWS Cloud Support Engineer based in Melbourne, Australia. He specializes in AWS security services, with deep expertise in ACM, IAM, and AWS Identity Center. He is passionate about helping customers protect their cloud infrastructure. He is also an AWS Golden Jacket owner with all AWS certifications.

Nikhil Kalra

Nikhil Kalra

Nikhil is an AWS Cloud Support Engineer based in Hyderabad, India. He is a subject matter expert in AWS Certificate Manager with expertise in core security services such as Amazon Cognito and IAM. Holding the prestigious AWS Certified Security Specialty certification, he is committed to helping customers implement robust security solutions and protect their cloud infrastructure.

New general-purpose Amazon EC2 M8a instances are now available

Post Syndicated from Betty Zheng (郑予彬) original https://aws.amazon.com/blogs/aws/new-general-purpose-amazon-ec2-m8a-instances-are-now-available/

Today, we’re announcing the availability of Amazon Elastic Compute Cloud (Amazon EC2) M8a instances, the latest addition to the general-purpose M instance family. These instances are powered by the 5th Generation AMD EPYC (codename Turin) processors with a maximum frequency of 4.5GHz. Customers can expect up to 30% higher performance and up to 19% better price performance compared to M7a instances. They also provide higher memory bandwidth, improved networking and storage throughput, and flexible configuration options for a broad set of general-purpose workloads.

Improvements in M8a
M8a instances deliver up to 30% better performance per vCPU compared to M7a instances, making them ideal for applications that require benefit from high performance and high throughput such as financial applications, gaming, rendering, application servers, simulation modeling, midsize data stores, application development environments, and caching fleets.

They provide 45% more memory bandwidth compared to M7a instances, accelerating in-memory databases, distributed caches, and real-time analytics.

For workloads with high I/O requirements, M8a instances provide up to 75 Gbps of networking bandwidth and 60 Gbps of Amazon Elastic Block Store (Amazon EBS) bandwidth, a 50% improvement over the previous generation. These enhancements support modern applications that rely on rapid data transfer and low-latency network communication.

Each vCPU on an M8a instance corresponds to a physical CPU core, meaning there is no simultaneous multithreading (SMT). In application benchmarks, M8a instances delivered up to 60% faster performance for GroovyJVM and up to 39% faster performance for Cassandra compared to M7a instances.

M8a instances support instance bandwidth configuration (IBC), which provides flexibility to allocate resources between networking and EBS bandwidth. This gives customers the flexibility to scale network or EBS bandwidth by up to 25% and improve database performance, query processing, and logging speeds.

M8a is available in ten virtualized sizes and two bare metal options (metal-24xl and metal-48xl), providing deployment choices that scale from small applications to large enterprise workloads. All of these improvements are built on the AWS Nitro System, which delivers low virtualization overhead, consistent performance, and advanced security across all instance sizes. These instances are built using the latest sixth generation AWS Nitro Cards, which offload and accelerate I/O for functions, increasing overall system performance.

M8a instances feature sizes of up to 192 vCPU with 768GiB RAM. Here are the detailed specs:

M8a vCPUs Memory (GiB) Network bandwidth (Gbps) EBS bandwidth (Gbps)
medium 1 4 Up to 12.5 Up to 10
large 2 8 Up to 12.5 Up to 10
xlarge 4 16 Up to 12.5 Up to 10
2xlarge 8 32 Up to 15 Up to 10
4xlarge 16 64 Up to 15 Up to 10
8xlarge 32 128 15 10
12xlarge 48 192 22.5 15
16xlarge 64 256 30 20
24xlarge 96 384 40 30
48xlarge 192 768 75 60
metal-24xl 96 384 40 30
metal-48xl 192 768 75 60

For a complete list of instance sizes and specifications, refer to the Amazon EC2 M8a instances page.

When to use M8a instances
M8a is a strong fit for general-purpose applications that need a balance of compute, memory, and networking. M8a instances are ideal for web and application hosting, microservices architectures, and databases where predictable performance and efficient scaling are important.

These instances are SAP certified and also well suited for enterprise workloads such as financial applications and enterprise resource planning (ERP) systems. They’re equally effective for in-memory caching and customer relationship management (CRM), in addition to development and test environments that require cost efficiency and flexibility. With this versatility, M8a supports a wide spectrum of workloads while helping customers improve price performance.

Now available
Amazon EC2 M8a instances are available today in US East (Ohio) US West (Oregon) and Europe (Spain) AWS Regions. M8a instances can be purchased as On-Demand, Savings Plans, and Spot Instances. M8a instances are also available on Dedicated Hosts. To learn more, visit the Amazon EC2 Pricing page.

To learn more, visit the Amazon EC2 M8a instances page and send feedback to AWS re:Post for EC2 or through your usual AWS support contacts.

— Betty

StackSets Deployment Strategies: Balancing Speed, Safety, and Scale to Optimize Deployments for Different Organizational Needs

Post Syndicated from Amar Meriche original https://aws.amazon.com/blogs/devops/stacksets-deployment-strategies-balancing-speed-safety-and-scale-to-optimize-deployments-for-different-organizational-needs/

AWS CloudFormation StackSets enables organizations to deploy infrastructure consistently across multiple AWS accounts and regions. However, success depends on choosing the right deployment strategy that balances three critical factors: deployment speed, operational safety, and organizational scale. This guide explores proven StackSets deployment strategies specifically designed for multi-account infrastructure management.

Understanding StackSets Deployment Fundamentals

What are StackSets Actually Used For?

Unlike single-account AWS CloudFormation templates, StackSets are specifically designed for multi-account infrastructure governance. Common use cases include Security baselines (deploying IAM policies, security groups, and access controls across all accounts), Compliance controls (rolling out AWS Config rules, AWS CloudTrail configurations, and audit requirements), Organizational standards (establishing consistent VPC configurations, tagging policies, and naming conventions), Shared services (deploying monitoring solutions, logging infrastructure, and backup policies) or Cost management (implementing budget controls, cost allocation tags, and resource optimization policies)

The Multi-Account Challenge

Managing infrastructure across dozens or hundreds of AWS accounts presents unique challenges:

Single Account (CFN Template)     Multi-Account (StackSets)
      App A                           Org Unit A (50 accounts)
        |                                     |
   [Deploy Once]               [Deploy consistently across all]
        |                                     |
    Success/Fail                Complex success/failure matrix

Multi account and multi region Cloudformation deployment complexity

The Speed-Safety-Scale Triangle

Every StackSets deployment strategy involves trade-offs: Speed (how quickly changes propagate across your organization), Safety (risk mitigation and failure containment) and Scale (ability to manage hundreds of accounts efficiently)

Prerequisites

Before implementing any of the deployment strategies described in this guide, ensure you have:

  1. AWS CLI Installation
    1. Install the latest version of AWS CLI by following the AWS CLI installation guide
    2. Verify installation with: aws –version
  2. AWS Profile Configuration
    1. Configure your AWS credentials using: aws configure
    2. For details on configuration, see AWS CLI configuration basics
    3. Ensure your profile has appropriate permissions for CloudFormation StackSets operations as described in AWS StackSets prerequisites
  3. Proper Account Access The commands in this guide must be executed from either:
    1. The management account of your AWS Organization
    2. OR a delegated administrator account for CloudFormation

For information on setting up a delegated administrator, see Register a delegated administrator

Note: StackSets deployments using service-managed permissions cannot be performed from standalone accounts.

Verify you’re using the correct account with:

bash
# For management account
aws organizations describe-organization
# For delegated admin
aws cloudformation list-stack-sets —call-as DELEGATED_ADMIN

AWS CLI to check the usage of an Organization and not a Standalone account

Core Deployment Strategies

As explained in the StackSet documentation:

  • “For a more conservative deployment, set Maximum Concurrent Accounts to 1, and Failure Tolerance to 0. Set your lowest-impact region to be first in the Region Order Start with one region.”
  • “For a faster deployment, increase the values of Maximum Concurrent Accounts and Failure Tolerance as needed. ”

Based on the above, we are proposing below several deployment strategies, depending on the speed, safety and scale you want to achieve.

1. Sequential Deployment: Maximum Safety

Use Case : Critical security updates, compliance requirements, first-time organizational rollouts

Below are listed some possible use cases:

  • Security baseline updates: New IAM policies affecting root access
  • Compliance rollouts: SOX, HIPAA, or PCI-DSS control implementations
  • Critical infrastructure changes: VPC security group modifications
  • Organizational policy changes: New AWS Config rules for audit compliance

Implementation Example:

For this example, we will download the following template ConfigRuleCloudtrailEnabled.yml from the Cloudformation sample library in the AWS documentation to configure an AWS Config rule to determine if AWS CloudTrail is enabled and follow the next steps:

Step 1: Create the StackSet

With the AWS CLI:

# Create Stackset for security baseline
# StackSet operation managed from us-east-1
aws cloudformation create-stack-set \
  --stack-set-name security-baseline \
  --template-body file://ConfigRuleCloudtrailEnabled.yml \
  --capabilities CAPABILITY_NAMED_IAM \
  --permission-model SERVICE_MANAGED \
  --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \
  --region us-east-1

AWS CLI to create a security-baseline Stackset

The expected response should be similar to the following :

{"StacksetId": "security-baseline: ...."}

Step 2: Create Stack Instances

Before you launch the below command, you need to adjust the values of the following parameters:

  • OrganizationalUnitIds: you must change the value “ou-test” in the below command line to the name of the target OU you want to deploy to. I recommend creating a new test OU in the console or via the CLI for the purpose of this test.
  • regions: if needed, change the “us-east-1 eu-west-1” value, here you need to list all the regions you want to deploy to. AWS Config must be active in the accounts/regions that you choose, otherwise you’ll get an error when deploying the Stack.

# Deploy security baseline to production accounts
# StackSet operation managed from us-east-1
# Deployed to regions us-east-1 and eu-west-1
# SEQUENTIAL = One region at a time, sequentially
# MaxConcurrentPercentage = Deploy to 5% of accounts at once
# FailureTolerancePercentage = Stop on first failure
aws cloudformation create-stack-instances \
  --stack-set-name security-baseline \
  --deployment-targets OrganizationalUnitIds=ou-test\
  --regions us-east-1 eu-west-1 \
  --region us-east-1 \
  --operation-preferences RegionConcurrencyType=SEQUENTIAL,MaxConcurrentPercentage=5,FailureTolerancePercentage=0

AWS CLI to create security-baseline Stack Instances sequentially for maximum safety

The CLI output should look like the following:

{"OperationId": ....}

Or create the StackSet and add the Stacks with the AWS Console:

In the CloudFormation Console, click “Create StackSet”

AWS CloudFormation Console: create a security-baseline Stackset

AWS CloudFormation Console: create a security-baseline Stackset

Upload your template from S3 or from your computer and click Next:

AWS CloudFormation Console: specify a template

AWS CloudFormation Console: specify a template

Specify the StackSet name and parameters and click Next:

AWS CloudFormation Console: specify the StackSet name and parameters

AWS CloudFormation Console: specify the StackSet name and parameters

Configure StackSet options and click Next:

AWS CloudFormation Console: configure the StackSet options

AWS CloudFormation Console: configure the StackSet options

Set deployment options and click Next:

AWS CloudFormation Console: set deployment options

AWS CloudFormation Console: set deployment options

AWS CloudFormation Console: set deployment options

AWS CloudFormation Console: set more deployment options

Then Review and Submit.

Not to overweight this blog, we’ll provide only this example of CLI output and Console screenshot, but the “Parallel Deployment” and “Balanced Approach” will be similar to this example. You just need to update the parameters for the different StackSet Operations options.

A real-world example would be a financial services company deploying new MFA requirements across 200 production accounts. They could use sequential deployment with 5 concurrency to ensure each batch was validated before proceeding.

2. Parallel Deployment: Maximum Speed

The Parallel Deployment is best for non-critical updates, development environments, routine maintenance

Here are some possible use cases:

  • Development account standardization: Rolling out new development tools
  • Monitoring infrastructure: Deploying Amazon CloudWatch dashboards and alarms
  • Cost optimization: Implementing automated resource cleanup policies
  • Non-production updates: Updating development and staging environments

Implementation Example:

For this example, we will copy paste the .yml template from this Re:Post article about monitoring IAM events in a file called “monitoring-baseline.yml”, and use it in the following command lines.

Step 1: Create the StackSet

# Create Stackset for monitoring baseline
# StackSet operation managed from us-east-1
aws cloudformation create-stack-set \
--stack-set-name monitoring-baseline \
--template-body file://monitoring-baseline.yml \
--capabilities CAPABILITY_NAMED_IAM \
--permission-model SERVICE_MANAGED \
--auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \
--region us-east-1

AWS CLI to create a monitoring-baseline Stackset

Step 2: Create Stack Instances

Just like in the previous example, before you launch the below command, you need to adjust the values of the OrganizationalUnitIds and regions parameters.

# Deploy monitoring baseline to dev and sandbox accounts
# StackSet operation managed from us-east-1
# Deployed to regions us-east-1 and eu-west-1
# PARALLEL = Deployment in parallel
# MaxConcurrentPercentage = Deploy to 80% of accounts at once
# FailureTolerancePercentage = Tolerate failures in 20% of accounts
aws cloudformation create-stack-instances \
--stack-set-name monitoring-baseline \
--deployment-targets OrganizationalUnitIds=ou-development,ou-sandbox \
--regions us-east-1 eu-west-1 \
--region us-east-1 \
--operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=80,FailureTolerancePercentage=20

AWS CLI to create monitoring-baseline Stack Instances in parallel with high value for max concurrent percentage for maximum speed

3. Progressive Deployment: Balanced Approach or Multi Phase Approach (Recommended)

For most production scenarios with moderate risk tolerance, it is recommended to use a Balanced Approach, or Multi-Phase Implementation.

Balanced Approach

For this example, to make it easier, you can create a copy of “monitoring-baseline.yml” created previously, and name it “balanced-template.yml”.

cp monitoring-baseline.yml balanced-template.yml

bash command to copy the monitoring-baseline.yml file to balanced-template.yml

Then you can use it in the following command lines.

Step 1: Create the StackSet

# Create Stackset for a balanced creation
# StackSet operation managed from us-east-1
aws cloudformation create-stack-set \
--stack-set-name balanced-deployment \
--template-body file://balanced-template.yml \
--capabilities CAPABILITY_NAMED_IAM \
--permission-model SERVICE_MANAGED \
--auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \
--region us-east-1

AWS CLI to create a balanced-deployment Stackset

Step 2: Create Stack Instances

You need to adjust the values of the OrganizationalUnitIds and regions parameters.

# Deploy monitoring baseline to production accounts
# StackSet operation managed from us-east-1
# Deployed to regions us-east-1, eu-west-1 and ap-southeast-1
# PARALLEL = Deployment in parallel
# MaxConcurrentPercentage = Deploy to 25% of accounts at once
# FailureTolerancePercentage = Tolerate failures in 8% of accounts
aws cloudformation create-stack-instances \
--stack-set-name balanced-deployment \
--deployment-targets OrganizationalUnitIds=ou-development,ou-sandbox \
--regions us-east-1 eu-west-1 ap-southeast-1 \
--region us-east-1 \
--operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=25,FailureTolerancePercentage=8

AWS CLI to create balanced-deployment Stack Instances in parallel with low max concurrent percentage for a balanced deployment

Multi-Phase Implementation:

Step 1: Create the StackSet

# Create Stackset for a balanced creation
# StackSet operation managed from us-east-1
aws cloudformation create-stack-set \
--stack-set-name balanced-deployment \
--template-body file://balanced-template.yml \
--capabilities CAPABILITY_NAMED_IAM \
--permission-model SERVICE_MANAGED \
--auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \
--region us-east-1

AWS CLI to create a balanced-deployment Stackset

Phase 1: Pilot Accounts (10% of target)

Phase 1: Create Pilot Stack Instances

You need to adjust the values of the OrganizationalUnitIds and regions parameters.

# Deploy monitoring baseline to production accounts
# StackSet operation managed from us-east-1
# Deployed to regions us-east-1
# SEQUENTIAL = Deployment in sequence
# MaxConcurrentPercentage = 100% Deploy full speed for small pilot
# FailureTolerancePercentage = Zero tolerance in pilot
aws cloudformation create-stack-instances \
--stack-set-name balanced-deployment \
--deployment-targets Accounts=pilot-account-1,pilot-account-2 \
--regions us-east-1 \
--region us-east-1 \
--operation-preferences RegionConcurrencyType=SEQUENTIAL,MaxConcurrentPercentage=100,FailureTolerancePercentage=0

AWS CLI to create balanced-deployment Stack Instances sequentially for maximum safety in Pilot accounts

Wait for Pilot validation before proceeding to Phase 2

Phase 2: Early Adopter OUs (30% of target)

Phase 2: Create Early Adopter Stack Instances

You need to adjust the values of the OrganizationalUnitIds and regions parameters.

# Deploy monitoring baseline to production accounts
# StackSet operation managed from us-east-1
# Deployed to regions us-east-1, eu-west-1
# PARALLEL = Deployment in parallel
# MaxConcurrentPercentage = Deploy to 25% of accounts at once
# FailureTolerancePercentage = Tolerate failures in 5% of accounts
aws cloudformation create-stack-instances \
--stack-set-name balanced-deployment \
--deployment-targets OrganizationalUnitIds=ou-early-adopter \
--regions us-east-1 \
--region us-east-1 eu-west-1 \
--operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=25,FailureTolerancePercentage=5

AWS CLI to create balanced-deployment Stack Instances in parallel with low max concurrent percentage for a balanced deployment in Early Adopter OU

Wait for Early Adopter validation before proceeding to Phase 3

Phase 3: Full Deployment (Remaining 60%)

Phase 3: Full Deployment

You need to adjust the values of the OrganizationalUnitIds and regions parameters.

# Deploy monitoring baseline to production accounts
# StackSet operation managed from us-east-1
# Deployed to regions us-east-1, eu-west-1 and ap-southeast-1
# PARALLEL = Deployment in parallel
# MaxConcurrentPercentage = Deploy to 40% of accounts at once for higher speed after validation
# FailureTolerancePercentage = Tolerate failures in 10% of accounts for moderate tolerance
aws cloudformation create-stack-instances \
--stack-set-name balanced-deployment \
--deployment-targets OrganizationalUnitIds=ou-standard-prod,ou-legacy-prod \
--regions us-east-1 \
--region us-east-1 eu-west-1 ap-southeast-1 \
--operation-preferences RegionConcurrencyType=PARALLEL,MaxConcurrentPercentage=25,FailureTolerancePercentage=5

AWS CLI to create balanced-deployment Stack Instances in parallel with low max concurrent percentage for a balanced deployment in the remaining OUs

Using Step Functions for Orchestration

AWS Step Functions provides a serverless workflow service that can orchestrate StackSets deployments with advanced control flow, error handling, and state management capabilities. This approach enhances your multi-account deployments with features not available through standard StackSets operations alone.

Some of the Key Benefits include:

  • Advanced Deployment Orchestration: Coordinate multi-phase rollouts with validation gates
  • Human Approval Workflows: Implement manual approval steps for critical changes
  • Enhanced Error Handling: Define sophisticated retry policies and fallback mechanisms
  • Visual Monitoring: Track deployment progress through the Step Functions visual console

Real-World Use Case: Compliance Control Rollout

In regulated industries, AWS Step Functions enables a phased approach that combines automation with necessary governance. For instance, you can:

  1. Deploy compliance controls to test accounts
  2. Run automated validation and generate compliance reports
  3. Obtain manual approval from compliance team
  4. Deploy to production accounts with comprehensive monitoring

This approach ensures consistent governance while maintaining the complete audit trail required for regulatory compliance.

Monitoring and Optimization

AWS CloudFormation StackSets do not have extensive built-in Amazon CloudWatch metrics specifically designed for monitoring StackSet operations and health. This is actually why the monitoring implementation in our blog post is valuable.

Here’s what AWS does and doesn’t provide out of the box:

What AWS provides natively:

  • Basic AWS API call metrics via AWS CloudTrail (which show that operations happened but don’t track success rates or performance)
  • General service quotas and throttling metrics for CloudFormation as a whole
  • CloudFormation provides some metrics for individual stacks, but not consolidated StackSet-specific metrics

What requires custom implementation (as in our blog post):

  • Success rate metrics for StackSet operations across accounts
  • Deployment completion time tracking
  • Configuration drift detection and monitoring
  • Account-specific failure analysis
  • Comprehensive dashboards that show StackSet health across your organization

The code in our blog post demonstrates how to implement the success rate custom metrics by:

  1. Gathering data from the CloudFormation API about StackSet operations
  2. Calculating the success rate metrics for StackSet deployments
  3. Creating custom Amazon CloudWatch metrics in a custom namespace (like “StackSetMonitoring”)
  4. Setting up alerts for issues

This explains why organizations need to implement custom monitoring solutions like the one shown in our blog post rather than relying solely on built-in metrics.

Automated Monitoring Implementation: example of a custom metric to monitor the StackSet operations success rate

The following AWS Cloudformation template provides real-time monitoring and alerting for AWS CloudFormation StackSet operations through automated infrastructure deployment. This solution creates a complete monitoring system using a AWS Lambda function, Amazon EventBridge rules, Amazon SNS notifications, and Amazon CloudWatch dashboards to track StackSet success and failure rates. The core Lambda function named StackSetMonitor continuously monitors all active StackSets in your account, calculating success rates and publishing custom metrics to Amazon CloudWatch under the StackSetMonitoring namespace.

Below you’ll find a few example of possible custom metrics that could be implemented based on this AWS Cloudformation template:

  • Count of all operations (CREATE, UPDATE, DELETE) per StackSet over time periods
  • Number of stack instances with configuration drift (requires additional API calls)
  • Average time taken for StackSet operations to complete
  • Rate of StackSet operations to identify peak usage times
  • Number of individual stack instances that failed during operations
  • Number of retried operations (indicates infrastructure issues)
  • …

Here’s the StackSetMonitor.yml CloudFormation Template:

# StackSetMonitor.yml 
# CFN template for monitoring AWS CloudFormation StackSet operations with real-time alerts, metrics, and dashboards.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'CloudFormation template for StackSet operation monitoring using CloudWatch and SNS'

Parameters:
  StackSetName:
    Type: String
    Description: 'Name of the StackSet to monitor'
    Default: 'security-baseline'
    MinLength: 1
    MaxLength: 128
    AllowedPattern: '[a-zA-Z][-a-zA-Z0-9]*'
    ConstraintDescription: 'Must be a valid StackSet name (1-128 characters, alphanumeric and hyphens, must start with a letter)'
  
  VpcId:
    Type: String
    Description: 'VPC ID where the Lambda function will be deployed (leave empty to create new VPC)'
    Default: ''
  
  SubnetIds:
    Type: CommaDelimitedList
    Description: 'List of subnet IDs for the Lambda function (leave empty to create new subnets)'
    Default: ''
    
  SecurityGroupIds:
    Type: CommaDelimitedList
    Description: 'List of security group IDs for the Lambda function (leave empty to create new security group)'
    Default: ''

Conditions:
  CreateVPC: !Equals [!Ref VpcId, '']
  CreateVPCAndSubnets: !And [!Equals [!Ref VpcId, ''], !Equals [!Join [',', !Ref SubnetIds], '']]
  HasCustomSecurityGroups: !Not [!Equals [!Join [',', !Ref SecurityGroupIds], '']]
  
Resources:
  # KMS Key for CloudWatch Logs encryption
  LogsKMSKey:
    Type: AWS::KMS::Key
    DeletionPolicy: Delete
    UpdateReplacePolicy: Delete
    Properties:
      Description: 'KMS Key for StackSet Monitor CloudWatch Logs and Lambda environment variable encryption'
      EnableKeyRotation: true
      KeyPolicy:
        Version: '2012-10-17'
        Statement:
          - Sid: Enable IAM User Permissions
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:${AWS::Partition}:iam::${AWS::AccountId}:root'
            Action: 'kms:*'
            Resource: '*'
          - Sid: Allow CloudWatch Logs
            Effect: Allow
            Principal:
              Service: !Sub 'logs.${AWS::Region}.amazonaws.com'
            Action:
              - 'kms:Encrypt'
              - 'kms:Decrypt'
              - 'kms:ReEncrypt*'
              - 'kms:GenerateDataKey*'
              - 'kms:DescribeKey'
            Resource: '*'
            Condition:
              ArnEquals:
                'kms:EncryptionContext:aws:logs:arn': 
                  - !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/StackSetMonitor'
                  - !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/cloudformation/stacksets'
          - Sid: Allow Lambda Service
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action:
              - 'kms:Encrypt'
              - 'kms:Decrypt'
              - 'kms:ReEncrypt*'
              - 'kms:GenerateDataKey*'
              - 'kms:DescribeKey'
            Resource: '*'

  LogsKMSKeyAlias:
    Type: AWS::KMS::Alias
    Properties:
      AliasName: alias/stackset-monitor-logs
      TargetKeyId: !Ref LogsKMSKey

  # VPC Resources (created when no existing VPC is provided)
  StackSetMonitorVPC:
    Type: AWS::EC2::VPC
    Condition: CreateVPC
    Properties:
      CidrBlock: 10.0.0.0/16
      EnableDnsHostnames: true
      EnableDnsSupport: true
      Tags:
        - Key: Name
          Value: StackSetMonitor-VPC
        - Key: Purpose
          Value: VPC for StackSet Monitor Lambda function


  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      CidrBlock: 10.0.1.0/24
      AvailabilityZone: !Select [0, !GetAZs '']
      Tags:
        - Key: Name
          Value: StackSetMonitor-Private-Subnet-1
        - Key: Purpose
          Value: Private subnet for StackSet Monitor Lambda

  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      CidrBlock: 10.0.2.0/24
      AvailabilityZone: !Select [1, !GetAZs '']
      Tags:
        - Key: Name
          Value: StackSetMonitor-Private-Subnet-2
        - Key: Purpose
          Value: Private subnet for StackSet Monitor Lambda

  PrivateRouteTable1:
    Type: AWS::EC2::RouteTable
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      Tags:
        - Key: Name
          Value: StackSetMonitor-Private-RT-1

  PrivateRouteTable2:
    Type: AWS::EC2::RouteTable
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      Tags:
        - Key: Name
          Value: StackSetMonitor-Private-RT-2

  PrivateSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Condition: CreateVPC
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      SubnetId: !Ref PrivateSubnet1

  PrivateSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Condition: CreateVPC
    Properties:
      RouteTableId: !Ref PrivateRouteTable2
      SubnetId: !Ref PrivateSubnet2

  # VPC Endpoints for AWS Services (no internet access needed)
  CloudFormationVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.cloudformation
      VpcEndpointType: Interface
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref VPCEndpointSecurityGroup
      PrivateDnsEnabled: true
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - cloudformation:ListStackSets
              - cloudformation:ListStackSetOperations
              - cloudformation:ListStackInstances
              - cloudformation:DescribeStackInstance
              - cloudformation:DescribeStacks
              - cloudformation:GetTemplate
            Resource: '*'

  CloudWatchVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.monitoring
      VpcEndpointType: Interface
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref VPCEndpointSecurityGroup
      PrivateDnsEnabled: true
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - cloudwatch:PutMetricData
            Resource: '*'

  SNSVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.sns
      VpcEndpointType: Interface
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref VPCEndpointSecurityGroup
      PrivateDnsEnabled: true
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - sns:Publish
            Resource: '*'

  EventsVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.events
      VpcEndpointType: Interface
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref VPCEndpointSecurityGroup
      PrivateDnsEnabled: true
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - events:PutEvents
            Resource: '*'

  LogsVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.logs
      VpcEndpointType: Interface
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref VPCEndpointSecurityGroup
      PrivateDnsEnabled: true
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
            Resource: '*'

  SQSVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.sqs
      VpcEndpointType: Interface
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref VPCEndpointSecurityGroup
      PrivateDnsEnabled: true
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - sqs:SendMessage
            Resource: '*'

  STSVPCEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Condition: CreateVPC
    Properties:
      VpcId: !Ref StackSetMonitorVPC
      ServiceName: !Sub com.amazonaws.${AWS::Region}.sts
      VpcEndpointType: Interface
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2
      SecurityGroupIds:
        - !Ref VPCEndpointSecurityGroup
      PrivateDnsEnabled: true
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action:
              - sts:AssumeRole
              - sts:GetCallerIdentity
              - sts:AssumeRoleWithWebIdentity
            Resource: '*'

  # Security Group for Lambda function
  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for StackSet Monitor Lambda function
      VpcId: !If
        - CreateVPC
        - !Ref StackSetMonitorVPC
        - !Ref VpcId
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 10.0.0.0/16
          Description: HTTPS to VPC Endpoints
        - IpProtocol: tcp
          FromPort: 53
          ToPort: 53
          CidrIp: 10.0.0.0/16
          Description: DNS TCP to VPC for name resolution
        - IpProtocol: udp
          FromPort: 53
          ToPort: 53
          CidrIp: 10.0.0.0/16
          Description: DNS UDP to VPC for name resolution
      Tags:
        - Key: Name
          Value: StackSetMonitor-Lambda-SG
        - Key: Purpose
          Value: Security group for StackSet Monitor Lambda

  VPCEndpointSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Condition: CreateVPC
    Properties:
      GroupDescription: Security group for VPC Endpoints
      VpcId: !Ref StackSetMonitorVPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          SourceSecurityGroupId: !Ref LambdaSecurityGroup
          Description: HTTPS from Lambda security group
        - IpProtocol: tcp
          FromPort: 53
          ToPort: 53
          SourceSecurityGroupId: !Ref LambdaSecurityGroup
          Description: DNS TCP from Lambda security group
        - IpProtocol: udp
          FromPort: 53
          ToPort: 53
          SourceSecurityGroupId: !Ref LambdaSecurityGroup
          Description: DNS UDP from Lambda security group
      SecurityGroupEgress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: 10.0.0.0/16
          Description: HTTPS outbound within VPC
        - IpProtocol: tcp
          FromPort: 53
          ToPort: 53
          CidrIp: 10.0.0.0/16
          Description: DNS TCP outbound within VPC
        - IpProtocol: udp
          FromPort: 53
          ToPort: 53
          CidrIp: 10.0.0.0/16
          Description: DNS UDP outbound within VPC
      Tags:
        - Key: Name
          Value: StackSetMonitor-VPCEndpoint-SG
        - Key: Purpose
          Value: Security group for VPC Endpoints

  # Dead Letter Queue for Lambda function
  StackSetMonitorDLQ:
    Type: AWS::SQS::Queue
    DeletionPolicy: Delete
    UpdateReplacePolicy: Delete
    Properties:
      QueueName: StackSetMonitor-DLQ
      MessageRetentionPeriod: 1209600  # 14 days
      KmsMasterKeyId: alias/aws/sqs
      Tags:
        - Key: Purpose
          Value: Dead Letter Queue for StackSet Monitor Lambda

  StackSetAlertsTopic:
    Type: AWS::SNS::Topic
    Properties: 
      TopicName: StackSetAlerts
      DisplayName: StackSet Monitoring Alerts
      KmsMasterKeyId: alias/aws/sns
  
  StackSetLogGroup:
    Type: AWS::Logs::LogGroup
    DeletionPolicy: Delete
    UpdateReplacePolicy: Delete
    Properties: 
      LogGroupName: /aws/cloudformation/stacksets
      RetentionInDays: 30
      KmsKeyId: !GetAtt LogsKMSKey.Arn

  LambdaLogGroup:
    Type: AWS::Logs::LogGroup
    DeletionPolicy: Delete
    UpdateReplacePolicy: Delete
    Properties:
      LogGroupName: /aws/lambda/StackSetMonitor
      RetentionInDays: 30
      KmsKeyId: !GetAtt LogsKMSKey.Arn
  
  StackSetMonitoringDashboard:
    Type: AWS::CloudWatch::Dashboard
    Properties:
      DashboardName: StackSetMonitoring
      DashboardBody: !Sub |
        {
          "widgets": [
            {
              "type": "metric",
              "width": 24,
              "height": 8,
              "properties": {
                "metrics": [
                  [ "StackSetMonitoring", "SuccessRate", "StackSetName", "${StackSetName}" ]
                ],
                "region": "${AWS::Region}",
                "title": "StackSet Operations",
                "period": 300,
                "stat": "Average"
              }
            },
            {
              "type": "log",
              "width": 24,
              "height": 6,
              "properties": {
                "query": "SOURCE '/aws/lambda/StackSetMonitor' | fields @timestamp, @message\n| sort @timestamp desc\n| limit 20",
                "region": "${AWS::Region}",
                "title": "Latest StackSet Monitor Logs",
                "view": "table"
              }
            }
          ]
        }
  
  # Consolidated rule to catch ALL StackSet events for comprehensive monitoring
  AllStackSetOperationsRule:
    Type: AWS::Events::Rule
    Properties:
      Name: AllStackSetOperationsRule
      Description: "Rule for monitoring all CloudFormation StackSet operations with failure notifications"
      EventPattern: {source: ["aws.cloudformation"], detail-type: ["CloudFormation StackSet Operation Status Change"]}
      State: ENABLED
      Targets:
        - Id: ProcessAllEvents
          Arn: !GetAtt StackSetMonitorLambda.Arn
        - Id: NotifyFailure
          Arn: !Ref StackSetAlertsTopic
          InputTransformer:
            InputPathsMap:
              "stackSetId": "$.detail.stack-set-id"
              "operationId": "$.detail.operation-id"
              "status": "$.detail.status"
              "time": "$.time"
            InputTemplate: '"StackSet Event: ID: <stackSetId>, Op: <operationId>, Status: <status>, Time: <time>"'

  StackSetMonitorLambda:
    Type: AWS::Lambda::Function
    DependsOn: LambdaLogGroup
    Properties:
      FunctionName: StackSetMonitor
      Handler: index.lambda_handler
      Role: !GetAtt StackSetMonitorRole.Arn
      Runtime: python3.12
      Timeout: 300
      MemorySize: 512
      ReservedConcurrentExecutions: 1
      DeadLetterConfig:
        TargetArn: !GetAtt StackSetMonitorDLQ.Arn
      VpcConfig:
        SecurityGroupIds: !If
          - HasCustomSecurityGroups
          - !Ref SecurityGroupIds
          - - !Ref LambdaSecurityGroup
        SubnetIds: !If
          - CreateVPCAndSubnets
          - - !Ref PrivateSubnet1
            - !Ref PrivateSubnet2
          - !Ref SubnetIds
      KmsKeyArn: !GetAtt LogsKMSKey.Arn
      Code:
        ZipFile: |
          import boto3
          import json
          import os
          import logging
          import time
          import datetime
          from typing import Dict, Any, Optional
          
          # Custom JSON encoder to handle datetime objects
          class DateTimeEncoder(json.JSONEncoder):
              def default(self, obj):
                  if isinstance(obj, datetime.datetime):
                      return obj.isoformat()
                  return super().default(obj)
          
          # Set up logging with more details
          logger = logging.getLogger()
          logger.setLevel(logging.INFO)
          
          # Log initialization to verify Lambda is loading correctly
          print("StackSetMonitor Lambda initializing...")
          
          def validate_event(event: Dict[str, Any]) -> bool:
              """Validate the incoming event structure"""
              if not isinstance(event, dict):
                  logger.error("Event must be a dictionary")
                  return False
              
              # If it's an EventBridge event, validate required fields
              if 'detail' in event:
                  detail = event.get('detail', {})
                  if not isinstance(detail, dict):
                      logger.error("Event detail must be a dictionary")
                      return False
                  
                  # Validate StackSet event structure
                  if 'stack-set-id' in detail:
                      stack_set_id = detail.get('stack-set-id')
                      if not isinstance(stack_set_id, str) or not stack_set_id.strip():
                          logger.error("stack-set-id must be a non-empty string")
                          return False
                      
                      # Validate operation-id if present
                      operation_id = detail.get('operation-id')
                      if operation_id is not None and not isinstance(operation_id, str):
                          logger.error("operation-id must be a string if provided")
                          return False
                      
                      # Validate status if present
                      status = detail.get('status')
                      if status is not None and not isinstance(status, str):
                          logger.error("status must be a string if provided")
                          return False
              
              return True
          
          def validate_context(context: Any) -> bool:
              """Validate the Lambda context object"""
              if context is None:
                  logger.error("Context cannot be None")
                  return False
              
              # Check for required context attributes
              required_attrs = ['function_name', 'function_version', 'invoked_function_arn', 'memory_limit_in_mb']
              for attr in required_attrs:
                  if not hasattr(context, attr):
                      logger.error(f"Context missing required attribute: {attr}")
                      return False
              
              return True
          
          def sanitize_string(value: str, max_length: int = 255) -> str:
              """Sanitize and truncate string inputs"""
              if not isinstance(value, str):
                  return str(value)[:max_length]
              return value.strip()[:max_length]
          
          def lambda_handler(event: Dict[str, Any], context: Any) -> Dict[str, Any]:
              """Main Lambda handler function for StackSet monitoring with input validation"""
              
              # Input validation
              if not validate_event(event):
                  return {
                      "statusCode": 400,
                      "body": json.dumps({
                          "status": "error",
                          "message": "Invalid event structure"
                      }, cls=DateTimeEncoder)
                  }
              
              if not validate_context(context):
                  return {
                      "statusCode": 400,
                      "body": json.dumps({
                          "status": "error",
                          "message": "Invalid context object"
                      }, cls=DateTimeEncoder)
                  }
              
              # Log the validated event for debugging
              logger.info(f"Event received: {json.dumps(event, cls=DateTimeEncoder)}")
              logger.info(f"Function: {context.function_name}, Version: {context.function_version}")
              
              try:
                  cf = boto3.client('cloudformation')
                  cw = boto3.client('cloudwatch')
                  
                  # Log that we're starting processing
                  logger.info(f"Starting StackSet monitoring at {time.time()}")
                  
                  # Check if this is an event from EventBridge
                  if 'detail' in event and 'stack-set-id' in event.get('detail', {}):
                      detail = event['detail']
                      stack_set_id = sanitize_string(detail['stack-set-id'])
                      operation_id = sanitize_string(detail.get('operation-id', 'N/A'))
                      status = sanitize_string(detail.get('status', 'N/A'))
                      
                      # Validate stack_set_id format
                      if not stack_set_id or len(stack_set_id) > 128:
                          logger.error(f"Invalid stack_set_id: {stack_set_id}")
                          return {
                              "statusCode": 400,
                              "body": json.dumps({
                                  "status": "error",
                                  "message": "Invalid stack_set_id format"
                              }, cls=DateTimeEncoder)
                          }
                      
                      # Log the StackSet operation with additional context
                      logger.info(f"Processing StackSet event - ID: {stack_set_id}, Op: {operation_id}, Status: {status}")
                      
                      # Extract stack set name from the ID
                      stack_set_name = stack_set_id.split('/')[-1] if '/' in stack_set_id else stack_set_id
                      stack_set_name = sanitize_string(stack_set_name, 128)
                      logger.info(f"Extracted StackSet name: {stack_set_name}")
                  
                  # Always gather metrics regardless of event type
                  # Get all active StackSets
                  stack_sets_response = cf.list_stack_sets(Status='ACTIVE')
                  stack_sets = stack_sets_response.get('Summaries', [])
                  
                  if not isinstance(stack_sets, list):
                      logger.error("Invalid response from list_stack_sets")
                      return {
                          "statusCode": 500,
                          "body": json.dumps({
                              "status": "error",
                              "message": "Invalid CloudFormation API response"
                          }, cls=DateTimeEncoder)
                      }
                  
                  logger.info(f"Found {len(stack_sets)} active StackSets")
                  
                  for stack_set in stack_sets:
                      if not isinstance(stack_set, dict) or 'StackSetName' not in stack_set:
                          logger.warning(f"Skipping invalid stack_set entry: {stack_set}")
                          continue
                      
                      stack_set_name = sanitize_string(stack_set['StackSetName'], 128)
                      logger.info(f"Processing StackSet: {stack_set_name}")
                      
                      try:
                          operations = cf.list_stack_set_operations(StackSetName=stack_set_name, MaxResults=5)
                          
                          # Validate operations response
                          if not isinstance(operations, dict):
                              logger.error(f"Invalid operations response for {stack_set_name}")
                              continue
                          
                          # Calculate success rate
                          successes = 0
                          operations_list = operations.get('Summaries', [])
                          
                          if not isinstance(operations_list, list):
                              logger.error(f"Invalid operations list for {stack_set_name}")
                              continue
                          
                          total_ops = len(operations_list)
                          logger.info(f"Found {total_ops} recent operations for {stack_set_name}")
                          
                          for op in operations_list:
                              if isinstance(op, dict) and op.get('Status') == 'SUCCEEDED':
                                  successes += 1
                          
                          success_rate = (successes / total_ops * 100) if total_ops > 0 else 100
                          
                          # Validate success_rate is within expected bounds
                          if not (0 <= success_rate <= 100):
                              logger.error(f"Invalid success_rate calculated: {success_rate}")
                              continue
                          
                          # Publish metrics to CloudWatch
                          cw.put_metric_data(
                              Namespace='StackSetMonitoring',
                              MetricData=[
                                  {'MetricName': 'SuccessRate', 'Value': success_rate, 
                                   'Dimensions': [{'Name': 'StackSetName', 'Value': stack_set_name}]}
                              ]
                          )
                          
                          logger.info(f"Published metrics for {stack_set_name}: Success Rate = {success_rate}%")
                      except Exception as e:
                          logger.error(f"Error processing StackSet {stack_set_name}: {str(e)}")
                  
                  return {
                      "statusCode": 200,
                      "body": json.dumps({
                          "status": "completed",
                          "message": f"Processed {len(stack_sets)} StackSets"
                      }, cls=DateTimeEncoder)
                  }
                  
              except Exception as e:
                  logger.error(f"Error in Lambda function: {str(e)}")
                  # Return a proper response even on error
                  return {
                      "statusCode": 500,
                      "body": json.dumps({
                          "status": "error",
                          "message": str(e)
                      }, cls=DateTimeEncoder)
                  }
  
  # Managed IAM Policies
  CloudFormationAccessPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: 'Policy for CloudFormation and CloudWatch access for StackSet Monitor'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - cloudformation:ListStackSets
              - cloudformation:ListStackSetOperations
              - cloudformation:ListStackInstances
              - cloudformation:DescribeStackInstance
            Resource: 
              - !Sub "arn:${AWS::Partition}:cloudformation:${AWS::Region}:${AWS::AccountId}:stackset/*"
              - !Sub "arn:${AWS::Partition}:cloudformation:${AWS::Region}:${AWS::AccountId}:stackset-target/*"
          - Effect: Allow
            Action:
              - cloudwatch:PutMetricData
            Resource: "*"
            Condition:
              StringEquals:
                "cloudwatch:namespace": "StackSetMonitoring"
          - Effect: Allow
            Action:
              - sns:Publish
            Resource: !Ref StackSetAlertsTopic

  EventsAccessPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: 'Policy for EventBridge access for StackSet Monitor'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - events:PutEvents
            Resource: !Sub "arn:${AWS::Partition}:events:${AWS::Region}:${AWS::AccountId}:event-bus/default"

  LogsAccessPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: 'Policy for CloudWatch Logs access for StackSet Monitor'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - logs:CreateLogGroup
              - logs:CreateLogStream
              - logs:PutLogEvents
            Resource: 
              - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/StackSetMonitor"
              - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/StackSetMonitor:*"
              - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/cloudformation/stacksets"
              - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/cloudformation/stacksets:*"

  DLQAccessPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      Description: 'Policy for Dead Letter Queue access for StackSet Monitor'
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - sqs:SendMessage
            Resource: !GetAtt StackSetMonitorDLQ.Arn

  StackSetMonitorRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
        - !Ref CloudFormationAccessPolicy
        - !Ref EventsAccessPolicy
        - !Ref LogsAccessPolicy
        - !Ref DLQAccessPolicy

  # Permissions for event rules to invoke Lambda
  AllOperationsRuleLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref StackSetMonitorLambda
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt AllStackSetOperationsRule.Arn
  
  # Using a one minute schedule for testing, but you can change this value
  StackSetMonitorSchedule:
    Type: AWS::Events::Rule
    Properties:
      Name: RegularStackSetMonitoring
      Description: "Triggers Lambda function every 1 minute to check StackSet operations"
      ScheduleExpression: "rate(1 minute)"
      State: ENABLED
      Targets:
        - Id: RunMonitor
          Arn: !GetAtt StackSetMonitorLambda.Arn
  
  ScheduleLambdaInvokePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref StackSetMonitorLambda
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt StackSetMonitorSchedule.Arn
  
  StackSetSuccessRateAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: "Alarm when StackSet operation success rate is low"
      MetricName: SuccessRate
      Namespace: "StackSetMonitoring"
      Statistic: Average
      Period: 300
      EvaluationPeriods: 3
      DatapointsToAlarm: 2
      Threshold: 80
      ComparisonOperator: LessThanThreshold
      AlarmActions: [!Ref StackSetAlertsTopic]
      Dimensions: [{Name: StackSetName, Value: !Ref StackSetName}]

Outputs:
  SNSTopicArn: 
    Description: The ARN of the SNS topic for alerts
    Value: !Ref StackSetAlertsTopic
  DashboardURL: 
    Description: URL to the CloudWatch Dashboard
    Value: !Sub https://console.aws.amazon.com/cloudwatch/home?region=${AWS::Region}#dashboards:name=StackSetMonitoring
  LambdaLogGroupName:
    Description: Name of the CloudWatch Log Group for Lambda logs
    Value: !Ref LambdaLogGroup
  DeadLetterQueueArn:
    Description: ARN of the Dead Letter Queue for Lambda function failures
    Value: !GetAtt StackSetMonitorDLQ.Arn
  DeadLetterQueueURL:
    Description: URL of the Dead Letter Queue for monitoring failed Lambda executions
    Value: !Ref StackSetMonitorDLQ
  TestLambdaCommand:
    Description: Command to manually test the Lambda function
    Value: !Sub "aws lambda invoke --function-name ${StackSetMonitorLambda} --payload '{}' response.json && cat response.json"
  LambdaFunctionArn:
    Description: ARN of the Lambda function configured with VPC
    Value: !GetAtt StackSetMonitorLambda.Arn
  LambdaSecurityGroupId:
    Description: Security Group ID created for the Lambda function
    Value: !Ref LambdaSecurityGroup
  VpcConfiguration:
    Description: VPC configuration summary for the Lambda function
    Value: !Sub 
      - "VPC: ${VpcId}, Subnets: ${SubnetList}, Security Groups: ${LambdaSecurityGroup}"
      - SubnetList: !Join [',', !Ref SubnetIds]

You need to run the following CLI command to deploy the CloudFormation stacks. You can change the ParameterValue of StackSetName“your-stackset-name” by the name of the StackSet you want to monitor. The default value is “security-baseline”. Your CLI profile should use region=“us-east-1“.

aws cloudformation create-stack --stack-name stackset-monitor --template-body file://StackSetMonitor.yml --parameters ParameterKey=StackSetName,ParameterValue="security-baseline" --capabilities CAPABILITY_IAM

AWS CLI to deploy the StackSetMonitor.yml CloudFormation template

The CLI output should look like the following:

{"StackId": "arn:aws:cloudformation:...."}

Here’s the expected output for the CloudFormation template:

StackSetMonitor Console output

StackSetMonitor Console output

And an example of Amazon CloudWatch Dashboard and Alarm screen:

Amazon CloudWatch Dashboard screenshot for StackSetMonitor stack to track StackSet operations success rate

Amazon CloudWatch Dashboard screenshot for StackSetMonitor stack to track StackSet operations success rate

Amazon CloudWatch Alarm screenshot for StackSetMonitor stack to track StackSet operations success rate

Amazon CloudWatch Alarm screenshot for StackSetMonitor stack to track StackSet operations success rate

SNS subscription setup involves retrieving the topic ARN from stack outputs and configuring notifications for email or SMS endpoints (below example CLI for email subscription):

aws sns subscribe --topic-arn $SNS_TOPIC_ARN --protocol email --notification-endpoint [email protected]

AWS CLI to subscribe to the topic providing the user email

Cost:

The estimated monthly expenses ranges between 5 and 15 USD depending on StackSet activity levels, with approximately 2,880 Lambda executions per day (each minute) under the default monitoring schedule.

The solution supports customization of monitoring frequency by modifying the ScheduleExpression from the default one-minute interval. The cost will decrease if the monitoring is less frequent.

Cleanup:

For cleanup, you can run the following command lines:

  • To cleanup the Stack Instances and StackSets created in the Core Deployment Strategies section:

aws cloudformation delete-stack-instances --stack-set-name security-baseline --deployment-targets OrganizationalUnitIds=ou-xxx --regions us-east-1 eu-west-1 --region us-east-1 --no-retain-stack

AWS CLI to delete the Stack Instances

You need to change the parameter OrganizationalUnitIds value with the name of the OU, the parameter regions with the list of regions where you want to delete your stack instances, and the value of the stack-set-name parameter (security-baseline, monitoring-baseline, balanced-deployment…).

Then you can delete the StackSet:

aws cloudformation delete-stack-set --stack-set-name security-baseline

AWS CLI to delete the StackSet

You can change the value of the stack-set-name parameter.

  • To cleanup the stackset-monitor stack

aws cloudformation delete-stack --stack-name stackset-monitor

AWS CLI to delete the stackset-monitor Stack

You can also remove any IAM roles/policies that you specifically created for this blog that you might not need anymore

Conclusion

Throughout this guide, we’ve explored the nuanced approaches to AWS CloudFormation StackSets deployments across large-scale environments. The key takeaways include:

  • Balance is Critical: Every deployment strategy requires careful consideration of the trade-offs between speed, safety, and scale based on your organizational needs.
  • Progressive Adoption Works: For most organizations, a progressive deployment approach with validation gates provides the optimal balance of safety and efficiency.
  • Organizational Context Matters: Enterprise, startup, and regulated industry patterns demonstrate that deployment strategies should be tailored to your specific business requirements and risk tolerance.
  • Monitoring is Essential: As organizations scale to hundreds of accounts, comprehensive monitoring becomes critical for maintaining visibility and ensuring compliance.

These different approaches will help you adopt the right strategy for your AWS CloudFormation Stacksets deployments in your AWS Organization.

You can now test these different approaches on your sandbox environment, before adapting them for your specific needs, in order to balance Speed, Safety and Scale to optimize your deployments.

Amar Meriche

Amar is a Sr Cloud Operations Architect at AWS in Paris. He helps his customers improve their operational posture through advocacy and guidance, and is an active member of the DevOps and IaC community at AWS. He’s passionate about helping customers use the various IaC tools available at AWS following best practices. When he’s not working with customers, Amar can be found on the mountain trails with his family or playing basketball with his team.

Idriss Laouali Abdou

Idriss is a Sr. Product Manager Technical for AWS Infrastructure-as-Code based in Seattle. He focuses on improving developer productivity through StackSets and CloudFormation Infrastructure provisioning experiences. Outside of work, you can find him creating educational content for thousands of students, cooking, or dancing.

Better profile management coming to Firefox

Post Syndicated from jzb original https://lwn.net/Articles/1041266/

Firefox has long had support for multiple profiles
to store personal information such as bookmarks, passwords, and user
preferences. However, Firefox did not make profiles particularly
discoverable or easy to manage. That is about to change; Mozilla has
announced
that it is launching a profile-management feature that will make it
easier to create and switch between profiles. According to the support
page
for the feature, it will be rolled out to users gradually
beginning on October 14.

[$] Upcoming Rust language features for kernel development

Post Syndicated from daroc original https://lwn.net/Articles/1039073/

The

Rust for Linux
project has been good for Rust, Tyler Mandry, one of the
co-leads of Rust’s language-design team, said. He
gave a talk at

Kangrejos 2025
covering upcoming Rust language features and thanking
the Rust for Linux developers for helping drive them forward. Afterward, Benno Lossin and Xiangfei Ding
went into more detail about their work on the three most important language
features for kernel development: field projections, in-place initialization, and arbitrary self types.

Flok License Plate Surveillance

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2025/10/flok-license-plate-surveillance.html

The company Flok is surveilling us as we drive:

A retired veteran named Lee Schmidt wanted to know how often Norfolk, Virginia’s 176 Flock Safety automated license-plate-reader cameras were tracking him. The answer, according to a U.S. District Court lawsuit filed in September, was more than four times a day, or 526 times from mid-February to early July. No, there’s no warrant out for Schmidt’s arrest, nor is there a warrant for Schmidt’s co-plaintiff, Crystal Arrington, whom the system tagged 849 times in roughly the same period.

You might think this sounds like it violates the Fourth Amendment, which protects American citizens from unreasonable searches and seizures without probable cause. Well, so does the American Civil Liberties Union. Norfolk, Virginia Judge Jamilah LeCruise also agrees, and in 2024 she ruled that plate-reader data obtained without a search warrant couldn’t be used against a defendant in a robbery case.

Да преведеш пари по заръката на Аллах

Post Syndicated from Атанас Шиников original https://www.toest.bg/da-prevedesh-pari-po-zarukata-na-allah/

Да преведеш пари по заръката на Аллах

Някои явления и практики, колкото и да са гъвкави спрямо контекста, като че ли проявяват удивителна устойчивост в исторически план. Протягат пръста си през вековете към нас. Ако сте чели „Името на розата“ от Умберто Еко, може да си го представите. Ето например, ако тук съм в ролята на един инфлуенсър или корпоративен коуч, мога да ви припомня историята за задните части на коня. Знаете я, върти се из социалните медии и свързва измеренията на конската задница с космическата програма и совалката в САЩ. 

Според тази история широчината на оста между колелата на древноримските колесници се определяла от широчината на задниците на два бойни коня. Впоследствие стандартът за ширина на оста залегнал в определянето на широчината на междуосието на британските фургони и каруци. Това пък определило и разстоянието между релсите на британските, а после и на американските железници. Оттам и транспортната инфраструктура, включително размерът на тунелите, са определени от този стандарт. Което пък на свой ред диктува размера на ракетоносителите на космическата совалка в американската програма. 

Ето, твърди историята, конските задници имат решаваща роля в дизайна на една от най-напредналите технологии, познати на човечеството. 

Тази история, разбира се, носи белезите на изкуствено украсено и напомпано представяне, чиито фактологични недостатъци могат да бъдат разгледани като предмет на отделно упражнение в стила на предаването „Ловци на митове“. Днес обаче съм решил да ви напомня за нещо по-прагматично в исторически план. Нещо, което не е толкова познато. Не е и толкова инфлуенсърско, с привкус на жълтевина, но пък има връзка със съвремието и би могло да се мисли като злободневно.

Нека сега прибегнем до мисловна спекулация и се опитаме да направим връзка между трафикантите на бежанци през България, ИДИЛ, бившия главен прокурор Иван Гешев, философа от XII век Ибн Рушд, известен при нас като Авероес, и самия Пророк на исляма, „прекрасния образец“ за човечеството – Мохамед. 

Знам, че за много от нашите съвременници възможността нещо, писано през седмото столетие по нашето летоброене, да породи размисъл някъде около дванайсетото столетие, което пък да продължи да се обговаря и практикува до днес, е скандална. Да, знам и че прокарването на паралели между на пръв поглед толкова раздалечени исторически и културни реалии може да изглежда, като да насилваме мисловния процес. Но пък защо да не го насилим – може да излезе нещо забавно или полезно. А и в класическата мюсюлманска образователна литература се казва, че невинаги става „со кротце, со благо“. Там при „акуширането“ на мисловния процес, известно в средите на философите като сократическа майевтика, пръчката и чепикът са незаменими помощници. 

Да започнем с по-скорошните свидетелства. 

В началото на юни тази година Европол излезе с интересна новина. След координирана акция под ръководството на българските власти е разбита криминална мрежа, която се занимава с трафик на предимно сирийски мигранти от Турция към Западна Европа. Според информацията от Европол всеки, който иска да стигне по този канал нелегално от Сирия до Европейския съюз, плаща сума, възлизаща на около 8500 евро. Обикновено мигрантите дават между 2000 и 2500 евро, за да бъдат превозени от Сирия до Турция. След няколкомесечен престой в Турция те плащат между 5000 и 6000 евро, за да влязат в Европейския съюз.

Връщаме се малко по-назад и четем прессъобщение от 2023 г. на Министерството на финансите на САЩ и тяхната служба за контрол на чуждестранните активи. САЩ и Турция предприемат съвместни действия, за да прекъснат финансирането на ИДИЛ. Защото, съгласете се, всичките канали, по които вече бившето образувание, издигащо претенции за халифат по смисъла на ислямския свещен закон (шари‘а), продава археологически паметници, за да се самофинансира, няма как да са прозрачни. Но са реални.

През 2019 г. пък в България са задържани шестима души за финансиране на тероризъм, които се занимават с изпращане на коли за продажба в Близкия изток. От тях петима са сирийци, твърди се в нашенските медии, а една – българска гражданка. Нейната задача според арабиста проф. Чуков, когото често гледаме тези дни покрай войната в Газа, е да търси посредници и купувачи, а останалите участници способстват за транспорта на автомобилите.

Какво е общото между тези няколко новини?

Имах един английски директор от Нюкасъл, който на подобен въпрос винаги отговаряше с „много“. И толкова. Това беше много дразнещо. Може да направим лесно формални паралели. Например по какво си приличат Земята и портокалът? По кръглата форма. По какво си приличат мюсюлманското религиозно училище (мадраса) и западният университет? И двете са образователни институции, възникнали в Средновековието. По какво си приличат Коранът и Библията? Ами и двете са свещени писания. Но както казваше и друга моя бивша шефка от Венецуела в духа на корпоративното клише, „Хора, нека сравняваме ябълки с ябълки“.

Естествено, може да кажем, че става въпрос за дейности отвъд позволеното от закона. Това е лесна аналогия. Може да кажем, че става въпрос и за транснационални криминални мрежи. Това също не е трудно да се заключи. Може да ги свържем и с Близкия изток. Не е толкова интересно. Но тук предлагам да мислим за тях през нещо по-конкретно, не толкова отвлечено. Проблемът за прехвърлянето на парите. Нали не си представяме как сирийските бежанци прилежно попълват платежно нареждане със сумата от 8500 евро, на което в основание за превода пише: „Такса трафик Сирия–ЕС“? Или пък някой си британски търговец на исторически старини превежда няколко десетки хиляди долара от банковата си сметка на получател на име Абу Бакр ал-Багдади, халиф, в Рака, Сирия, с основание „Трансфер покупка на антична статуя“? 

Затова си има друг механизъм, много удобен и достатъчно непрозрачен. За него ислямското право и практика са измислили термина хауала. И за него не е нужна формална банкова система.

Хауала на арабски идва от тройния корен, съставен от буквите ха’-уау-лам. Обозначава неща като „прехвърляне“, „трансформация“, „превръщане“, „преобръщане“.

Почти като „джуркането“ на делата и „превъртането на файловете“ на бившия правосъден министър Данаил Кирилов. Оттук и паричен „трансфер“, но не точно. Защото в терминологичното си значение според теорията и практиката на мюсюлманското право и финанси означава следното.

Да речем, че съм Хасан, новоприел исляма български гражданин от ромската махала в Пазарджик, който възприема себе си като част от глобализираната мюсюлманска общност (умма) по света. Използвам социални медии, ходил съм на курс по класически арабски, Коран, Сунна и право (фикх) в Йордания, смятам традиционния ислям в България за невернишки, защото мюсюлманите в България пият ракия и ядат свинско, и имам своя мрежа от международни контакти. Искам да изпратя пари на Ахмед, преподавател в религиозно училище (медресе) в Кандахар, Афганистан, за да купи нови корани или да финансира кухня за бедни, т.нар. имарет, откъдето идва и името на втората джамия в Пловдив – Имарет джамия. Или пък искам да пратя пари за калашници, произведени в Казанлък, за благочестивата духовна борба и върховно усилие (джихад) на талибаните срещу неверниците от САЩ на терен. 

Тогава тук в играта влиза фигурата на Муса от Пазарджик, примерно. Известен в неформалните среди като хауаладар, т.е. посредник на хауала. Брокер. Агент. Отивам при Муса и му казвам, че искам да пратя 250 долара на Ахмед в Кандахар. 

А Муса, представете си, върти пари от малък бизнес – магазин за халално месо и млечни продукти. (Като онзи на входа на махалата в Пазарджик, в който преди няколко години под пластмасовия часовник с цитат от сура 112 на Корана попитах продавачката: „Имате ли телешки дроб?“ А тя ми отговори: „Имаме, даже е от теле!“) Но междувременно въобразеният за целта на примера Муса се оказва изпечен международник. Познава друг хауаладар в Кандахар на име Мустафа. Давам 250 долара на Муса и му казвам кодова дума. Пращам кодовата дума и на Ахмед. Муса се свързва с Мустафа и му казва кодовата дума. Ахмед отива при Мустафа и срещу кодовата дума получава 250 долара. За услугата се плаща и малка комисиона. 

След това се оказва, че Муса дължи на Мустафа 250 долара. И тъй като са добри познати, знаят се от много години, изяли са много хумус, пилета и дюнери заедно, си уреждат задълженията когато и както могат по линията Пазарджик–Кандахар. Няма реален физически трансфер на пари. Доверието е определящо. Защото Муса и Мустафа поддържат контакти с други свои „колеги“ в Мумбай, Карачи, Истанбул, Кайро, Маракеш и на още десетки други места. Няма следи, няма запис от превода. Най-много някой тефтер на хауаладар да съдържа в таблица бележки за кодови имена на клиенти, други колеги, дати, часове, суми. Да речем, „Абу Мазен (кодовото име за Мустафа от измислената ни история), 6 октомври 2025 г., 250 долара“. 

А как хауаладарите после ще си уредят сметките, си е тяхна работа. Дали по някое време в неопределеното бъдеще ще си пращат пари в плик, пъхнати между страниците на Корана по поща, дали ще втъкнат банкноти в замразен телешки шол, по камили, гълъби, патици (като във филма „Мисия Лондон“), или пък ще действат с нормален банков трансфер с основание „превод“ по друг повод, дали единият ще купи на другия стоки или услуги на съответната стойност – не ни интересува. И Хасан от Пазарджик, и Ахмед от Кандахар не ги интересува. Точно както в момента, в който пиша този текст, не ме интересува как работят софтуерът на текстообработващата програма, сървърите на социалните мрежи, операционната система на мобилния ми телефон или мрежата SWIFT за междубанкови операции. 

Така работи международният обмен на пари според традиционната система на хауала. Типичен пример за трансфер на парично задължение. А това, както е ясно от горния спекулативен и анекдотичен пример, отваря пространство за много транзакции на тъмно и за пране на пари. Примерите са безброй – вижте само доклада на службата по наркотици и престъпления към ООН от 2023 г. за търговията с наркотици в Афганистан¹.

Хауала дори попада в полето на академичните занимания на самия (вече бивш) главен прокурор на България Иван Гешев.

През 2020 г. той, тогава докторант по международно право и отношения, заедно със съавтор Николай Марин, публикува статията „Системата „хавала“ – между обичайното право и организираната престъпност“. След като е предпочетено хавала, другото разпространено четене на оригиналното арабско хауала, „брокерите“ се превръщат в хаваладари. Любопитно е, че практиката е сравнена с блокчейн технологиите и биткойн заради децентрализацията, връзката тип потребитeл-към-потребител и анонимизацията на транзакцията. Дава се пример с разкриването през януари 2019 г. на мрежа от лица, които участват в хауала „за прикриване на особено тежки престъпления“. Това се смятало за първия сблъсък на правозащитните органи с явлението, и то в контекста на организираната престъпност. Като утежняващо обстоятелство се отчита фактът, че по този начин се предоставяла логистична подкрепа на терористични организации. Очевидно е, че така се нарушават не само банковото законодателство, законите за кредитните институции, за плащането в брой и за пренасянето на парични средства през граница. Затова и практиката е определена като „явление с висока степен на обществена опасност“. 

Признавам си, изпитвам известни трудности да мисля за бившия главен прокурор в неговото чисто академично амплоа, но пък с любопитство изчитам докторантската му публикация. 

Очевидно България като транзитна територия по пътя между Близкия и Средния изток и Западна Европа предполага интерес към темата за паричните потоци в нейните актуални аспекти. Според Гергана Йорданова, автор на монография по въпроса², хауалата си е хауала, че и сиренето е с пари. Ако трябва да поиграем с българската поговорка, може да използваш първото, за да си купиш второто. И не само сирене. В случай че искаш да почерпиш приятелче или да помогнеш на роднини, нещо като петолевка в картичка по „Български пощи“ (не го правете), също може да използваш хауаладар. Това според българската изследователка попада в категорията „бяла хауала“ – издръжка на домакинства, образование, лечение, помощ за мигранти, светски, религиозни събития, поклонение (хадж), милостиня, издателска дейност. Всички онези случаи, при които може да се докаже законен произход и предназначение на парите. Но в този „Ин-Ян“ на шариатските транзакции е логично да очакваме и „черната хауала“. Там нещата загрубяват. Туристически услуги, агенции за самолетни билети, търговия с автомобили, таксита, лизинги, обмен на валути, трафик на хора, органи, търговия с бижута и наркотици³. 

Но стига думскролинг. Този хубав термин, който обозначава нашата слабост ненаситно да превъртаме злокобно съдържание на екрана на устройствата си. Не искаме да дълбаем твърде много в горещите аспекти на хауала. За тях са писали и продължават пишат други авторитетни гласове, че даже, както се видя, и бивши главни прокурори на България. Очевидно е, че хауала попада в полезрението на законодателството, на дейностите по превенция на прането на пари и така привлича вниманието на властите. Но нека оставим злободневието настрана. Предлагам да отстъпим няколко крачки назад във времето. 

Къде са старите религиозни основания на хауала?

(Следва продължение.)

1 UNODC, The Hawala System: Its operations and misuse by opiate traffickers and migrant smugglers, 2023.

2 Йорданова, Гергана. Хауала. Същност. Типологии за изпиране на пари, финансиране на тероризъм и пролиферация. София: Фондация „Институт за национална и международна сигурност“, 2023.

3 Йорданова, Гергана. Разграничение между „черни“ и „бели“ хауала транзакции в контекста на противодействието на изпирането на пари. – Сигурност и отбрана, 2023, №1 с. 129–143.

В рубриката „Ориент кафе“ Атанас Шиников поднася любопитни теми, свързани не толкова с горещата политика, колкото с историята и културата на Близкия изток. А той, древен и днешен, е по-близко до нас и съвремието ни, отколкото си представяме.

How we found a bug in Go’s arm64 compiler

Post Syndicated from Thea Heinen original https://blog.cloudflare.com/how-we-found-a-bug-in-gos-arm64-compiler/

Every second, 84 million HTTP requests are hitting Cloudflare across our fleet of data centers in 330 cities. It means that even the rarest of bugs can show up frequently. In fact, it was our scale that recently led us to discover a bug in Go’s arm64 compiler which causes a race condition in the generated code.

This post breaks down how we first encountered the bug, investigated it, and ultimately drove to the root cause.

Investigating a strange panic

We run a service in our network which configures the kernel to handle traffic for some products like Magic Transit and Magic WAN. Our monitoring watches this closely, and it started to observe very sporadic panics on arm64 machines.

We first saw one with a fatal error stating that traceback did not unwind completely. That error suggests that invariants were violated when traversing the stack, likely because of stack corruption. After a brief investigation we decided that it was probably rare stack memory corruption. This was a largely idle control plane service where unplanned restarts have negligible impact, and so we felt that following up was not a priority unless it kept happening.

And then it kept happening. 

Coredumps per hour


When we first saw this bug we saw that the fatal errors correlated with recovered panics. These were caused by some old code which used panic/recover as error handling. 

At this point, our theory was: 

  1. All of the fatal panics happen within stack unwinding.

  2. We correlated an increased volume of recovered panics with these fatal panics.

  3. Recovering a panic unwinds goroutine stacks to call deferred functions.

  4. A related Go issue (#73259) reported an arm64 stack unwinding crash.

  5. Let’s stop using panic/recover for error handling and wait out the upstream fix?

So we did that and watched as fatal panics stopped occurring as the release rolled out. Fatal panics gone, our theoretical mitigation seemed to work, and this was no longer our problem. We subscribed to the upstream issue so we could update when it was resolved and put it out of our minds.

But, this turned out to be a much stranger bug than expected. Putting it out of our minds was premature as the same class of fatal panics came back at a much higher rate. A month later, we were seeing up to 30 daily fatal panics with no real discernible cause; while that might account for only one machine a day in less than 10% of our data centers, we found it concerning that we didn’t understand the cause. The first thing we checked was the number of recovered panics, to match our previous pattern, but there were none. More interestingly, we could not correlate this increased rate of fatal panics with anything. A release? Infrastructure changes? The position of Mars?

At this point we felt like we needed to dive deeper to better understand the root cause. Pattern matching and hoping was clearly insufficient. 

We saw two classes of this bug — a crash while accessing invalid memory and an explicitly checked fatal error. 

Fatal Error

goroutine 153 gp=0x4000105340 m=324 mp=0x400639ea08 [GC worker (active)]:
/usr/local/go/src/runtime/asm_arm64.s:244 +0x6c fp=0x7ff97fffe870 sp=0x7ff97fffe860 pc=0x55558d4098fc
runtime.systemstack(0x0)
       /usr/local/go/src/runtime/mgc.go:1508 +0x68 fp=0x7ff97fffe860 sp=0x7ff97fffe810 pc=0x55558d3a9408
runtime.gcBgMarkWorker.func2()
       /usr/local/go/src/runtime/mgcmark.go:1102
runtime.gcDrainMarkWorkerIdle(...)
       /usr/local/go/src/runtime/mgcmark.go:1188 +0x434 fp=0x7ff97fffe810 sp=0x7ff97fffe7a0 pc=0x55558d3ad514
runtime.gcDrain(0x400005bc50, 0x7)
       /usr/local/go/src/runtime/mgcmark.go:212 +0x1c8 fp=0x7ff97fffe7a0 sp=0x7ff97fffe6f0 pc=0x55558d3ab248
runtime.markroot(0x400005bc50, 0x17e6, 0x1)
       /usr/local/go/src/runtime/mgcmark.go:238 +0xa8 fp=0x7ff97fffe6f0 sp=0x7ff97fffe6a0 pc=0x55558d3ab578
runtime.markroot.func1()
       /usr/local/go/src/runtime/mgcmark.go:887 +0x290 fp=0x7ff97fffe6a0 sp=0x7ff97fffe560 pc=0x55558d3acaa0
runtime.scanstack(0x4014494380, 0x400005bc50)
       /usr/local/go/src/runtime/traceback.go:447 +0x2ac fp=0x7ff97fffe560 sp=0x7ff97fffe4d0 pc=0x55558d3eeb7c
runtime.(*unwinder).next(0x7ff97fffe5b0?)
       /usr/local/go/src/runtime/traceback.go:566 +0x110 fp=0x7ff97fffe4d0 sp=0x7ff97fffe490 pc=0x55558d3eed40
runtime.(*unwinder).finishInternal(0x7ff97fffe4f8?)
       /usr/local/go/src/runtime/panic.go:1073 +0x38 fp=0x7ff97fffe490 sp=0x7ff97fffe460 pc=0x55558d403388
runtime.throw({0x55558de6aa27?, 0x7ff97fffe638?})
runtime stack:
fatal error: traceback did not unwind completely
       stack=[0x4015d6a000-0x4015d8a000
runtime: g8221077: frame.sp=0x4015d784c0 top=0x4015d89fd0

Segmentation fault

goroutine 187 gp=0x40003aea80 m=13 mp=0x40003ca008 [GC worker (active)]:
       /usr/local/go/src/runtime/asm_arm64.s:244 +0x6c fp=0x7fff2afde870 sp=0x7fff2afde860 pc=0x55557e2d98fc
runtime.systemstack(0x0)
       /usr/local/go/src/runtime/mgc.go:1489 +0x94 fp=0x7fff2afde860 sp=0x7fff2afde810 pc=0x55557e279434
runtime.gcBgMarkWorker.func2()
       /usr/local/go/src/runtime/mgcmark.go:1112
runtime.gcDrainMarkWorkerDedicated(...)
       /usr/local/go/src/runtime/mgcmark.go:1188 +0x434 fp=0x7fff2afde810 sp=0x7fff2afde7a0 pc=0x55557e27d514
runtime.gcDrain(0x4000059750, 0x3)
       /usr/local/go/src/runtime/mgcmark.go:212 +0x1c8 fp=0x7fff2afde7a0 sp=0x7fff2afde6f0 pc=0x55557e27b248
runtime.markroot(0x4000059750, 0xb8, 0x1)
       /usr/local/go/src/runtime/mgcmark.go:238 +0xa8 fp=0x7fff2afde6f0 sp=0x7fff2afde6a0 pc=0x55557e27b578
runtime.markroot.func1()
       /usr/local/go/src/runtime/mgcmark.go:887 +0x290 fp=0x7fff2afde6a0 sp=0x7fff2afde560 pc=0x55557e27caa0
runtime.scanstack(0x40042cc000, 0x4000059750)
       /usr/local/go/src/runtime/traceback.go:458 +0x188 fp=0x7fff2afde560 sp=0x7fff2afde4d0 pc=0x55557e2bea58
runtime.(*unwinder).next(0x7fff2afde5b0)
goroutine 0 gp=0x40003af880 m=13 mp=0x40003ca008 [idle]:
PC=0x55557e2bea58 m=13 sigcode=1 addr=0x118
SIGSEGV: segmentation violation

Now we could observe some clear patterns. Both errors occur when unwinding the stack in (*unwinder).next. In one case we saw an intentional fatal error as the runtime identified that unwinding could not complete and the stack was in a bad state. In the other case there was a direct memory access error that happened while trying to unwind the stack. The segfault was discussed in the GitHub issue and a Go engineer identified it as dereference of a go scheduler struct, m, when unwinding. 

A review of Go scheduler structs

Go uses a lightweight userspace scheduler to manage concurrency. Many goroutines are scheduled on a smaller number of kernel threads – this is often referred to as M:N scheduling. Any individual goroutine can be scheduled on any kernel thread. The scheduler has three core types – g  (the goroutine), m (the kernel thread, or “machine”), and p (the physical execution context, or  “processor”). For a goroutine to be scheduled a free m must acquire a free p, which will execute a g. Each g contains a field for its m if it is currently running, otherwise it will be nil. This is all the context needed for this post but the go runtime docs explore this more comprehensively. 

At this point we can start to make inferences on what’s happening: the program crashes because we try to unwind a goroutine stack which is invalid. In the first backtrace, if a return address is null, we call finishInternal and abort because the stack was not fully unwound. The segmentation fault case in the second backtrace is a bit more interesting: if instead the return address is non-zero but not a function then the unwinder code assumes that the goroutine is currently running. It’ll then dereference m and fault by accessing m.incgo (the offset of incgo into struct m is 0x118, the faulting memory access).

What, then, is causing this corruption? The traces were difficult to get anything useful from – our service has hundreds if not thousands of active goroutines. It was fairly clear from the beginning that the panic was remote from the actual bug. The crashes were all observed while unwinding the stack and if this were an issue any time the stack was unwound on arm64 we would be seeing it in many more services. We felt pretty confident that the stack unwinding was happening correctly but on an invalid stack. 

Our investigation stalled for a while at this point – making guesses, testing guesses, trying to infer if the panic rate went up or down, or if nothing changed. There was a known issue on Go’s GitHub issue tracker which matched our symptoms almost exactly, but what they discussed was mostly what we already knew. At some point when looking through the linked stack traces we realized that their crash referenced an old version of a library that we were also using – Go Netlink.

goroutine 1267 gp=0x4002a8ea80 m=nil [runnable (scan)]:
runtime.asyncPreempt2()
        /usr/local/go/src/runtime/preempt.go:308 +0x3c fp=0x4004cec4c0 sp=0x4004cec4a0 pc=0x46353c
runtime.asyncPreempt()
        /usr/local/go/src/runtime/preempt_arm64.s:47 +0x9c fp=0x4004cec6b0 sp=0x4004cec4c0 pc=0x4a6a8c
github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0x14360300000000?)
        /go/pkg/mod/github.com/!data!dog/[email protected]/nl/nl_linux.go:803 +0x130 fp=0x4004cfc710 sp=0x4004cec6c0 pc=0xf95de0

We spot-checked a few stack traces and confirmed the presence of this Netlink library. Querying our logs showed that not only did we share a library – every single segmentation fault we observed had happened while preempting NetlinkSocket.Receive.

What’s (async) preemption?

In the prehistoric era of Go (<=1.13) the runtime was cooperatively scheduled. A goroutine would run until it decided it was ready to yield to the scheduler – usually due to explicit calls to runtime.Gosched() or injected yield points at function calls/IO operations. Since Go 1.14 the runtime instead does async preemption. The Go runtime has a thread sysmon which tracks the runtime of goroutines and will preempt any that run for longer than 10ms (at time of writing). It does this by sending SIGURG to the OS thread and in the signal handler will modify the program counter and stack to mimic a call to asyncPreempt.

At this point we had two broad theories:

  • This is a Go Netlink bug – likely due to unsafe.Pointer usage which invoked undefined behavior but is only actually broken on arm64

  • This is a Go runtime bug and we’re only triggering it in NetlinkSocket.Receive for some reason

After finding the same bug publicly reported upstream, we were feeling confident this was caused by a Go runtime bug. However, upon seeing that both issues implicated the same function, we felt more skeptical – notably the Go Netlink library uses unsafe.Pointer so memory corruption was a plausible explanation even if we didn’t understand why.

After an unsuccessful code audit we had hit a wall. The crashes were rare and remote from the root cause. Maybe these crashes were caused by a runtime bug, maybe they were caused by a Go Netlink bug. It seemed clear that there was something wrong with this area of the code, but code auditing wasn’t going anywhere.

Breakthrough

At this point we had a fairly good understanding of what was crashing but very little understanding of why it was happening. It was clear that the root cause of the stack unwinder crashing was remote from the actual crash, and that it had to do with (*NetlinkSocket).Receive, but why? We were able to capture a coredump of a production crash and view it in a debugger. The backtrace confirmed what we already knew – that there was a segmentation fault when unwinding a stack. The crux of the issue revealed itself when we looked at the goroutine which had been preempted while calling (*NetlinkSocket).Receive. 

(dlv) bt
0  0x0000555577579dec in runtime.asyncPreempt2
   at /usr/local/go/src/runtime/preempt.go:306
1  0x00005555775bc94c in runtime.asyncPreempt
   at /usr/local/go/src/runtime/preempt_arm64.s:47
2  0x0000555577cb2880 in github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive
   at
/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:779
3  0x0000555577cb19a8 in github.com/vishvananda/netlink/nl.(*NetlinkRequest).Execute
   at 
/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:532
4  0x0000555577551124 in runtime.heapSetType
   at /usr/local/go/src/runtime/mbitmap.go:714
5  0x0000555577551124 in runtime.heapSetType
   at /usr/local/go/src/runtime/mbitmap.go:714
...
(dlv) disass -a 0x555577cb2878 0x555577cb2888
TEXT github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(SB) /vendor/github.com/vishvananda/netlink/nl/nl_linux.go
        nl_linux.go:779 0x555577cb2878  fdfb7fa9        LDP -8(RSP), (R29, R30)
        nl_linux.go:779 0x555577cb287c  ff430191        ADD $80, RSP, RSP
        nl_linux.go:779 0x555577cb2880  ff434091        ADD $(16<<12), RSP, RSP
        nl_linux.go:779 0x555577cb2884  c0035fd6        RET

The goroutine was paused between two opcodes in the function epilogue. Since the process of unwinding a stack relies on the stack frame being in a consistent state, it felt immediately suspicious that we preempted in the middle of adjusting the stack pointer. The goroutine had been paused at 0x555577cb2880, between ADD $80, RSP, RSP and ADD $(16<<12), RSP, RSP. 

We queried the service logs to confirm our theory. This wasn’t isolated – the majority of stack traces showed that this same opcode was preempted. This was no longer a weird production crash we couldn’t reproduce. A crash happened when the Go runtime preempted between these two stack pointer adjustments. We had our smoking gun. 

Building a minimal reproducer

At this point we felt pretty confident that this was actually just a runtime bug and it should be reproducible in an isolated environment without any dependencies. The theory at this point was:

  1. Stack unwinding is triggered by garbage collection

  2. Async preemption between a split stack pointer adjustment causes a crash

  3. What if we make a function which splits the adjustment and then call it in a loop?

package main

import (
	"runtime"
)

//go:noinline
func big_stack(val int) int {
	var big_buffer = make([]byte, 1 << 16)

	sum := 0
	// prevent the compiler from optimizing out the stack
	for i := 0; i < (1<<16); i++ {
		big_buffer[i] = byte(val)
	}
	for i := 0; i < (1<<16); i++ {
		sum ^= int(big_buffer[i])
	}
	return sum
}

func main() {
	go func() {
		for {
			runtime.GC()
		}
	}()
	for {
		_ = big_stack(1000)
	}
}

This function ends up with a stack frame slightly larger than can be represented in 16 bits, and so on arm64 the Go compiler will split the stack pointer adjustment into two opcodes. If the runtime preempts between these opcodes then the stack unwinder will read an invalid stack pointer and crash. 

; epilogue for main.big_stack
ADD $8, RSP, R29
ADD $(16<<12), R29, R29
ADD $16, RSP, RSP
; preemption is problematic between these opcodes
ADD $(16<<12), RSP, RSP
RET

After running this for a few minutes the program panicked as expected!

SIGSEGV: segmentation violation
PC=0x60598 m=8 sigcode=1 addr=0x118

goroutine 0 gp=0x400019c540 m=8 mp=0x4000198708 [idle]:
runtime.(*unwinder).next(0x400030fd10)
        /home/thea/sdk/go1.23.4/src/runtime/traceback.go:458 +0x188 fp=0x400030fcc0 sp=0x400030fc30 pc=0x60598
runtime.scanstack(0x40000021c0, 0x400002f750)
        /home/thea/sdk/go1.23.4/src/runtime/mgcmark.go:887 +0x290 

[...]

goroutine 1 gp=0x40000021c0 m=nil [runnable (scan)]:
runtime.asyncPreempt2()
        /home/thea/sdk/go1.23.4/src/runtime/preempt.go:308 +0x3c fp=0x40003bfcf0 sp=0x40003bfcd0 pc=0x400cc
runtime.asyncPreempt()
        /home/thea/sdk/go1.23.4/src/runtime/preempt_arm64.s:47 +0x9c fp=0x40003bfee0 sp=0x40003bfcf0 pc=0x75aec
main.big_stack(0x40003cff38?)
        /home/thea/dev/stack_corruption_reproducer/main.go:29 +0x94 fp=0x40003cff00 sp=0x40003bfef0 pc=0x77c04
Segmentation fault (core dumped)

real    1m29.165s
user    4m4.987s
sys     0m43.212s

A reproducible crash with standard library only? This felt like conclusive evidence that our problem was a runtime bug.

This was an extremely particular reproducer! Even now with a good understanding of the bug and its fix, some of the behavior is still puzzling. It’s a one-instruction race condition, so it’s unsurprising that small changes could have large impact. For example, this reproducer was originally written and tested on Go 1.23.4, but did not crash when compiled with 1.23.9 (the version in production), even though we could objdump the binary and see the split ADD still present! We don’t have a definite explanation for this behavior – even with the bug present there remain a few unknown variables which affect the likelihood of hitting the race condition. 

A single-instruction race condition window

arm64 is a fixed-length 4-byte instruction set architecture. This has a lot of implications on codegen but most relevant to this bug is the fact that immediate length is limited. add gets a 12-bit immediate, mov gets a 16-bit immediate, etc. How does the architecture handle this when the operands don’t fit? It depends – ADD in particular reserves a bit for “shift left by 12” so any 24 bit addition can be decomposed into two opcodes. Other instructions are decomposed similarly, or just require loading an immediate into a register first. 

The very last step of the Go compiler before emitting machine code involves transforming the program into obj.Prog structs. It’s a very low level intermediate representation (IR) that mostly serves to be translated into machine code. 

//https://github.com/golang/go/blob/fa2bb342d7b0024440d996c2d6d6778b7a5e0247/src/cmd/internal/obj/arm64/obj7.go#L856

// Pop stack frame.
// ADD $framesize, RSP, RSP
p = obj.Appendp(p, c.newprog)
p.As = AADD
p.From.Type = obj.TYPE_CONST
p.From.Offset = int64(c.autosize)
p.To.Type = obj.TYPE_REG
p.To.Reg = REGSP
p.Spadj = -c.autosize

Notably, this IR is not aware of immediate length limitations. Instead, this happens in asm7.go when Go’s internal intermediate representation is translated into arm64 machine code. The assembler will classify an immediate in conclass based on bit size and then use that when emitting instructions – extra if needed.

The Go assembler uses a combination of (mov, add) opcodes for some adds that fit in 16-bit immediates, and prefers (add, add + lsl 12) opcodes for 16-bit+ immediates. 

Compare a stack of (slightly larger than) 1<<15:

; //go:noinline
; func big_stack() byte {
; 	var big_stack = make([]byte, 1<<15)
; 	return big_stack[0]
; }
MOVD $32776, R27
ADD R27, RSP, R29
MOVD $32784, R27
ADD R27, RSP, RSP
RET

With a stack of 1<<16:

; //go:noinline
; func big_stack() byte {
; 	var big_stack = make([]byte, 1<<16)
; 	return big_stack[0]
; } 
ADD $8, RSP, R29
ADD $(16<<12), R29, R29
ADD $16, RSP, RSP
ADD $(16<<12), RSP, RSP
RET

In the larger stack case, there is a point between ADD x, RSP, RSP opcodes where the stack pointer is not pointing to the tip of a stack frame. We thought at first that this was a matter of memory corruption – that in handling async preemption the runtime would push a function call on the stack and corrupt the middle of the stack. However, this goroutine is already in the function epilogue – any data we corrupt is actively in the process of being thrown away. What’s the issue then?

The Go runtime often needs to unwind the stack, which means walking backwards through the chain of function calls. For example: garbage collection uses it to find live references on the stack, panicking relies on it to evaluate defer functions, and generating stack traces needs to print the call stack. For this to work the stack pointer must be accurate during unwinding because of how golang dereferences sp to determine the calling function. If the stack pointer is partially modified, the unwinder will look for the calling function in the middle of the stack. The underlying data is meaningless when interpreted as directions to a parent stack frame and then the runtime will likely crash. 

//https://github.com/golang/go/blob/66536242fce34787230c42078a7bbd373ef8dcb0/src/runtime/traceback.go#L373

if innermost && frame.sp < frame.fp || frame.lr == 0 {
    lrPtr = frame.sp
    frame.lr = *(*uintptr)(unsafe.Pointer(lrPtr))
}

When async preemption happens it will push a function call onto the stack but the parent stack frame is no longer correct because sp was only partially adjusted when the preemption happened. The crash flow looks something like this:

  1. Async preemption happens between the two opcodes that add x, rsp expands to

  2. Garbage collection triggers stack unwinding (to check for heap object liveness)

  3. The unwinder starts traversing the stack of the problematic goroutine and correctly unwinds up to the problematic function

  4. The unwinder dereferences sp to determine the parent function

  5. Almost certainly the data behind sp is not a function

  6. Crash


We saw earlier a faulting stack trace which ended in (*NetlinkSocket).Receive – in this case stack unwinding faulted while it was trying to determine the parent frame. 

goroutine 90 gp=0x40042cc000 m=nil [preempted (scan)]:
runtime.asyncPreempt2()
/usr/local/go/src/runtime/preempt.go:306 +0x2c fp=0x40060a25d0 sp=0x40060a25b0 pc=0x55557e299dec
runtime.asyncPreempt()
/usr/local/go/src/runtime/preempt_arm64.s:47 +0x9c fp=0x40060a27c0 sp=0x40060a25d0 pc=0x55557e2dc94c
github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xff48ce6e060b2848?)
/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:779 +0x130 fp=0x40060b2820 sp=0x40060a27d0 pc=0x55557e9d2880

Once we discovered the root cause we reported it with a reproducer and the bug was quickly fixed. This bug is fixed in go1.23.12, go1.24.6, and go1.25.0. Previously, the go compiler emitted a single add x, rsp instruction and relied on the assembler to split immediates into multiple opcodes as necessary. After this change, stacks larger than 1<<12 will build the offset in a temporary register and then add that to rsp in a single, indivisible opcode. A goroutine can be preempted before or after the stack pointer modification, but never during. This means that the stack pointer is always valid and there is no race condition.

LDP -8(RSP), (R29, R30)
MOVD $32, R27
MOVK $(1<<16), R27
ADD R27, RSP, RSP
RET

This was a very fun problem to debug. We don’t often see bugs where you can accurately blame the compiler. Debugging it took weeks and we had to learn about areas of the Go runtime that people don’t usually need to think about. It’s a nice example of a rare race condition, the sort of bug that can only really be quantified at a large scale.

We’re always looking for people who enjoy this kind of detective work. Our engineering teams are hiring.

The collective thoughts of the interwebz