Tag Archives: Uncategorized

Malware-Infested Smart Card Reader

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/malware-infested-smart-card-reader.html

Brian Krebs has an interesting story of a smart ID card reader with a malware-infested Windows driver, and US government employees who inadvertently buy and use them.

But by all accounts, the potential attack surface here is enormous, as many federal employees clearly will purchase these readers from a myriad of online vendors when the need arises. Saicoo’s product listings, for example, are replete with comments from customers who self-state that they work at a federal agency (and several who reported problems installing drivers).

Manipulating Machine-Learning Systems through the Order of the Training Data

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/manipulating-machine-learning-systems-through-the-order-of-the-training-data.html

Yet another adversarial ML attack:

Most deep neural networks are trained by stochastic gradient descent. Now “stochastic” is a fancy Greek word for “random”; it means that the training data are fed into the model in random order.

So what happens if the bad guys can cause the order to be not random? You guessed it—all bets are off. Suppose for example a company or a country wanted to have a credit-scoring system that’s secretly sexist, but still be able to pretend that its training was actually fair. Well, they could assemble a set of financial data that was representative of the whole population, but start the model’s training on ten rich men and ten poor women drawn from that set ­ then let initialisation bias do the rest of the work.

Does this generalise? Indeed it does. Previously, people had assumed that in order to poison a model or introduce backdoors, you needed to add adversarial samples to the training data. Our latest paper shows that’s not necessary at all. If an adversary can manipulate the order in which batches of training data are presented to the model, they can undermine both its integrity (by poisoning it) and its availability (by causing training to be less effective, or take longer). This is quite general across models that use stochastic gradient descent.

Research paper.

The Justice Department Will No Longer Charge Security Researchers with Criminal Hacking

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/the-justice-department-will-no-longer-charge-security-researchers-with-criminal-hacking.html

Following a recent Supreme Court ruling, the Justice Department will no longer prosecute “good faith” security researchers with cybercrimes:

The policy for the first time directs that good-faith security research should not be charged. Good faith security research means accessing a computer solely for purposes of good-faith testing, investigation, and/or correction of a security flaw or vulnerability, where such activity is carried out in a manner designed to avoid any harm to individuals or the public, and where the information derived from the activity is used primarily to promote the security or safety of the class of devices, machines, or online services to which the accessed computer belongs, or those who use such devices, machines, or online services.

[…]

The new policy states explicitly the longstanding practice that “the department’s goals for CFAA enforcement are to promote privacy and cybersecurity by upholding the legal right of individuals, network owners, operators, and other persons to ensure the confidentiality, integrity, and availability of information stored in their information systems.” Accordingly, the policy clarifies that hypothetical CFAA violations that have concerned some courts and commentators are not to be charged. Embellishing an online dating profile contrary to the terms of service of the dating website; creating fictional accounts on hiring, housing, or rental websites; using a pseudonym on a social networking site that prohibits them; checking sports scores at work; paying bills at work; or violating an access restriction contained in a term of service are not themselves sufficient to warrant federal criminal charges. The policy focuses the department’s resources on cases where a defendant is either not authorized at all to access a computer or was authorized to access one part of a computer—such as one email account—and, despite knowing about that restriction, accessed a part of the computer to which his authorized access did not extend, such as other users’ emails.

News article.

Forging Australian Driver’s Licenses

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/forging-australian-drivers.html

The New South Wales digital driver’s license has multiple implementation flaws that allow for easy forgeries.

This file is encrypted using AES-256-CBC encryption combined with Base64 encoding.

A 4-digit application PIN (which gets set during the initial onboarding when a user first instals the application) is the encryption password used to protect or encrypt the licence data.

The problem here is that an attacker who has access to the encrypted licence data (whether that be through accessing a phone backup, direct access to the device or remote compromise) could easily brute-force this 4-digit PIN by using a script that would try all 10,000 combinations….

[…]

The second design flaw that is favourable for attackers is that the Digital Driver Licence data is never validated against the back-end authority which is the Service NSW API/database.

This means that the application has no native method to validate the Digital Driver Licence data that exists on the phone and thus cannot perform further actions such as warn users when this data has been modified.

As the Digital Licence is stored on the client’s device, validation should take place to ensure the local copy of the data actually matches the Digital Driver’s Licence data that was originally downloaded from the Service NSW API.

As this verification does not take place, an attacker is able to display the edited data on the Service NSW application without any preventative factors.

There’s a lot more in the blog post.

Bluetooth Flaw Allows Remote Unlocking of Digital Locks

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/bluetooth-flaw-allows-remote-unlocking-of-digital-locks.html

Locks that use Bluetooth Low Energy to authenticate keys are vulnerable to remote unlocking. The research focused on Teslas, but the exploit is generalizable.

In a video shared with Reuters, NCC Group researcher Sultan Qasim Khan was able to open and then drive a Tesla using a small relay device attached to a laptop which bridged a large gap between the Tesla and the Tesla owner’s phone.

“This proves that any product relying on a trusted BLE connection is vulnerable to attacks even from the other side of the world,” the UK-based firm said in a statement, referring to the Bluetooth Low Energy (BLE) protocol—technology used in millions of cars and smart locks which automatically open when in close proximity to an authorised device.

Although Khan demonstrated the hack on a 2021 Tesla Model Y, NCC Group said any smart locks using BLE technology, including residential smart locks, could be unlocked in the same way.

Another news article.

Websites that Collect Your Data as You Type

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/websites-that-collect-your-data-as-you-type.html

A surprising number of websites include JavaScript keyloggers that collect everything you type as you type it, not just when you submit a form.

Researchers from KU Leuven, Radboud University, and University of Lausanne crawled and analyzed the top 100,000 websites, looking at scenarios in which a user is visiting a site while in the European Union and visiting a site from the United States. They found that 1,844 websites gathered an EU user’s email address without their consent, and a staggering 2,950 logged a US user’s email in some form. Many of the sites seemingly do not intend to conduct the data-logging but incorporate third-party marketing and analytics services that cause the behavior.

After specifically crawling sites for password leaks in May 2021, the researchers also found 52 websites in which third parties, including the Russian tech giant Yandex, were incidentally collecting password data before submission. The group disclosed their findings to these sites, and all 52 instances have since been resolved.

“If there’s a Submit button on a form, the reasonable expectation is that it does something — that it will submit your data when you click it,” says Güneş Acar, a professor and researcher in Radboud University’s digital security group and one of the leaders of the study. “We were super surprised by these results. We thought maybe we were going to find a few hundred websites where your email is collected before you submit, but this exceeded our expectations by far.”

Research paper.

iPhone Malware that Operates Even When the Phone Is Turned Off

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/iphone-malware-that-operates-even-when-the-phone-is-turned-off.html

Researchers have demonstrated iPhone malware that works even when the phone is fully shut down.

t turns out that the iPhone’s Bluetooth chip­ — which is key to making features like Find My work­ — has no mechanism for digitally signing or even encrypting the firmware it runs. Academics at Germany’s Technical University of Darmstadt figured out how to exploit this lack of hardening to run malicious firmware that allows the attacker to track the phone’s location or run new features when the device is turned off.

[…]

The research is the first — or at least among the first — to study the risk posed by chips running in low-power mode. Not to be confused with iOS’s low-power mode for conserving battery life, the low-power mode (LPM) in this research allows chips responsible for near-field communication, ultra wideband, and Bluetooth to run in a special mode that can remain on for 24 hours after a device is turned off.

The research is fascinating, but the attack isn’t really feasible. It requires a jailbroken phone, which is hard to pull off in an adversarial setting.

Slashdot thread.

Attacks on Managed Service Providers Expected to Increase

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/attacks-on-managed-service-providers-expected-to-increase.html

CISA, NSA, FBI, and similar organizations in the other Five Eyes countries are warning that attacks on MSPs — as a vector to their customers — are likely to increase. No details about what this prediction is based on. Makes sense, though. The SolarWinds attack was incredibly successful for the Russian SVR, and a blueprint for future attacks.

News articles.

AWS Week in Review – May 16, 2022

Post Syndicated from Marcia Villalba original https://aws.amazon.com/blogs/aws/aws-week-in-review-may-16-2022/

This post is part of our Week in Review series. Check back each week for a quick roundup of interesting news and announcements from AWS!

I had been on the road for the last five weeks and attended many of the AWS Summits in Europe. It was great to talk to so many of you in person. The Serverless Developer Advocates are going around many of the AWS Summits with the Serverlesspresso booth. If you attend an event that has the booth, say “Hi 👋” to my colleagues, and have a coffee while asking all your serverless questions. You can find all the upcoming AWS Summits in the events section at the end of this post.

Last week’s launches
Here are some launches that got my attention during the previous week.

AWS Step Functions announced a new console experience to debug your state machine executions – Now you can opt-in to the new console experience of Step Functions, which makes it easier to analyze, debug, and optimize Standard Workflows. The new page allows you to inspect executions using three different views: graph, table, and event view, and add many new features to enhance the navigation and analysis of the executions. To learn about all the features and how to use them, read Ben’s blog post.

Example on how the Graph View looks

Example on how the Graph View looks

AWS Lambda now supports Node.js 16.x runtime – Now you can start using the Node.js 16 runtime when you create a new function or update your existing functions to use it. You can also use the new container image base that supports this runtime. To learn more about this launch, check Dan’s blog post.

AWS Amplify announces its Android library designed for Kotlin – The Amplify Android library has been rewritten for Kotlin, and now it is available in preview. This new library provides better debugging capacities and visibility into underlying state management. And it is also using the new AWS SDK for Kotlin that was released last year in preview. Read the What’s New post for more information.

Three new APIs for batch data retrieval in AWS IoT SiteWise – With this new launch AWS IoT SiteWise now supports batch data retrieval from multiple asset properties. The new APIs allow you to retrieve current values, historical values, and aggregated values. Read the What’s New post to learn how you can start using the new APIs.

AWS Secrets Manager now publishes secret usage metrics to Amazon CloudWatch – This launch is very useful to see the number of secrets in your account and set alarms for any unexpected increase or decrease in the number of secrets. Read the documentation on Monitoring Secrets Manager with Amazon CloudWatch for more information.

For a full list of AWS announcements, be sure to keep an eye on the What’s New at AWS page.

Other AWS News
Some other launches and news that you may have missed:

IBM signed a deal with AWS to offer its software portfolio as a service on AWS. This allows customers using AWS to access IBM software for automation, data and artificial intelligence, and security that is built on Red Hat OpenShift Service on AWS.

Podcast Charlas Técnicas de AWS – If you understand Spanish, this podcast is for you. Podcast Charlas Técnicas is one of the official AWS podcasts in Spanish. This week’s episode introduces you to Amazon DynamoDB and shares stories on how different customers use this database service. You can listen to all the episodes directly from your favorite podcast app or the podcast web page.

AWS Open Source News and Updates – Ricardo Sueiras, my colleague from the AWS Developer Relation team, runs this newsletter. It brings you all the latest open-source projects, posts, and more. Read edition #112 here.

Upcoming AWS Events
It’s AWS Summits season and here are some virtual and in-person events that might be close to you:

You can register for re:MARS to get fresh ideas on topics such as machine learning, automation, robotics, and space. The conference will be in person in Las Vegas, June 21–24.

That’s all for this week. Check back next Monday for another Week in Review!

— Marcia

The NSA Says that There are No Known Flaws in NIST’s Quantum-Resistant Algorithms

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/the-nsa-says-that-there-are-no-known-flaws-in-nists-quantum-resistant-algorithms.html

Rob Joyce, the director of cybersecurity at the NSA, said so in an interview:

The NSA already has classified quantum-resistant algorithms of its own that it developed over many years, said Joyce. But it didn’t enter any of its own in the contest. The agency’s mathematicians, however, worked with NIST to support the process, trying to crack the algorithms in order to test their merit.

“Those candidate algorithms that NIST is running the competitions on all appear strong, secure, and what we need for quantum resistance,” Joyce said. “We’ve worked against all of them to make sure they are solid.”

The purpose of the open, public international scrutiny of the separate NIST algorithms is “to build trust and confidence,” he said.

I believe him. This is what the NSA did with NIST’s candidate algorithms for AES and then for SHA-3. NIST’s Post-Quantum Cryptography Standardization Process looks good.

I still worry about the long-term security of the submissions, though. In 2018, in an essay titled “Cryptography After the Aliens Land,” I wrote:

…there is always the possibility that those algorithms will fall to aliens with better quantum techniques. I am less worried about symmetric cryptography, where Grover’s algorithm is basically an upper limit on quantum improvements, than I am about public-key algorithms based on number theory, which feel more fragile. It’s possible that quantum computers will someday break all of them, even those that today are quantum resistant.

It took us a couple of decades to fully understand von Neumann computer architecture. I’m sure it will take years of working with a functional quantum computer to fully understand the limits of that architecture. And some things that we think of as computationally hard today will turn out not to be.

За руската и украинската артилерия… и не само

Post Syndicated from original http://www.gatchev.info/blog/?p=2443

Разказаха ми нещо, което ме замисли.

Във войната в Украйна руската артилерия действа по класическия начин от Втората световна война. Дислоцира се батареята, приготвя се за стрелба и чака заповеди от комбата по какво да стреля. Право на избор къде да се дислоцира, по кого да стреля и т.н. няма – прави се каквото е заповядано. Без заповед само се отбранява, ако я атакуват. (Ако не ѝ е забранено и това.)

Украинската артилерия е пръсната на малки единици – често дори само едно оръдие. Имат (доста голям) район, в който да се дислоцират – те си избират къде точно, кога и къде да се местят и т.н. Централни заповеди почти няма. Вместо тях има приложение за смартфони. Операторите на дронове въвеждат в него координати на забелязаната цел, оръдейците – данни за позицията си, и приложението моментално им дава данни за поставяне на оръдието, за да бъде ударена конкретната цел с висока точност. Стрелят по нея на които оръдия е в обсега и според важността ѝ. Децентрализирано управление на огъня.

Като резултат украинската артилерия често дори успява да надмине руската като ефективност и резултатност. Въпреки че е няколко пъти по-малка, и с изключение на доставените от Запад към 10% от оръдията ѝ, по-стара и по-хилава от руската. Унищожението на руската армия при опитите ѝ за създаване на плацдарм през Сиверски Донец го демонстрира изключително убедително… А в същото време е много по-малко уязвима от руската – единично оръдие се мести и крие много по-лесно, поразява се със стандартна артилерия много по-трудно и т.н.

А толкова ли са изостанали руснаците, че не прилагат същото? Нямат ли си кадърни програмисти?

О-хо-хо! Имат, и още как! 98% от ботнетовете са реализирани и контролирани от руснаци. Вярно е, не от случайни киберкримки, а от поделение 26165 на ГРУ – хакерите на руската армия. Но това ги прави не по-малко, а по-достъпни за военни цели. А киберпрестъпността печели и плаща доста, лесно наема качествен талант, иска се изключително майсторство, за да си водещ сред нея. Очевидно хакерите на ГРУ са.

А защо тогава не го правят?

Защото подобно приложение би нарушило принцип номер едно на руската армия – тя е, за да изпълнява заповеди. В нея личната инициатива се поощрява единствено когато е в изпълнение на конкретна заповед, и дори тогава не винаги. Във всички други случаи се хвали на думи и наказва на дело.

А това е заради един фундаментален факт. Под „капиталистическата“ боя Русия продължава да е феодална държава.

Например в нея изгодните бизнеси се водят юридически собственост на съответните олигарси, но реално са тяхно ленно владение, дадено им за вярна служба. В момента, в който „сеньорът“ прецени, че друг би му служил по-добре, владението преминава в „собственост“ на този друг, срещу непублична сума (като правило нула). Ако настоящият му „собственик“ не е наясно със системата – примерно се заблуждава, че бизнесът е наистина негова собственост – му се случва каквото ще му докаже грешката му (Ходорковски, anyone?). Подобно е положението и с всички други видове „владения“ – високи постове, привилегии и т.н.

Размерът и важността на „владението“ се определя от точно едно нещо – до каква степен „собственикът“ му е над закона. Това не винаги съвпада с обичайните представи. Например типичният армейски генерал командва като минимум бригада, няколко хиляди войници и огромно количество бойна техника, а лейтенантът от „службите“ често няма нито един подчинен и единственото му оръжие е един пистолет. Ако обаче лейтенантът изкомандва генерала да му пази колата, генералът ще го изпълни на мига, въпреки че лейтенантът официално е много далеч под него в йерархията. Просто „службите“, като реално политическа полиция, са благородническа категория безусловно над всички други, тъй като са по-доверени на властта от всички други. Съответно идеята, че можете да им се опънете за нещо, или да ги съдите, би разсмяла нормалния руснак.

Затова за Русия безпрекословното подчинение в армията е от върховна важност. Заради тежкото си въоръжение армията е по-силова институция дори от „службите“, а в същото време е много по-малко доверена на феодалната върхушка. Позволи ли се в нея култура на самоинициатива, върхушката е под заплаха, по-страшна от външните врагове. За всяка феодална върхушка най-страшният ѝ враг е простолюдието, което тя тъпче – а армията е набрана предимно от него… Затова подобна гъвкавост, базирана на самоинициатива, няма да бъде допусната в руската армия дори при риск за загуба на войната.

Накратко, Русия е феодализъм с тънък до прозрачност слой капиталистическа боя отгоре. Докато Украйна е доста несъвършена и дори с феодални остатъци демокрация, но все пак по-скоро демокрация. Затова тя може да си позволи да даде свобода на инициативата на бойците си – може да разчита на лоялността им. Докато Русия няма как да посмее да ѝ разчита.

А свободата на инициативата и гъвкавостта при изпълнение на задачи са огромни предимства в съвременната война. На теория най-ефективното ѝ постижение, интеграцията на видовете оръжия и родовете войски, е чисто командна задача. На практика обаче, подобно на икономиката, съвременната военна обстановка е твърде сложна и динамична, за да може да бъде обхваната адекватно от централизирана командна система. Съответно делегирането на инициатива и децентрализирането на действията носят огромен плюс. През 1982 г. Израел успява да разгроми многократно по-многобройните арабски армии основно защото позволява инициатива и децентрализиране на действията. Това, че израелската армия е въоръжена със западно оръжие, а арабските с руско, също е фактор, но с по-малко значение.

Ето затова Украйна, въпреки че има няколко пъти по-малки човешки и военни ресурси от Русия, има отлични шансове да спечели тази война. Тя е по същество война на капитализма срещу феодализма. Виждали сме я неведнъж, и на военния фронт, и на икономическия, и на социалния. Знаем как свършва.

А и не е зле да се замислим над още нещо. През 2014 г. Украйна успя да изкара от властта част от руските агенти там. Малко, но все пак. Като резултат, за само 8 години доста украинци стигнаха дотам да карат коли, които по българските критерии изглеждат луксозни. И завистливите ганьовци ги плюят за това. Вместо да се замислят защо те не карат такива – и какво е нужно, за да получат един ден тази възможност.

Всъщност, не за да я получат те – за да я получим всички в България… Те няма да го проумеят. Ако бяха способни на това, нямаше да са завистливи ганьовци. Но може би ние, които смятаме себе си за извън тази група, е добре да вземем мерки по въпроса.

Upcoming Speaking Engagements

Post Syndicated from Schneier.com Webmaster original https://www.schneier.com/blog/archives/2022/05/upcoming-speaking-engagements-19.html

This is a current list of where and when I am scheduled to speak:

  • I’m speaking on “Securing a World of Physically Capable Computers” at OWASP Belgium’s chapter meeting in Antwerp, Belgium, on May 17, 2022.
  • I’m speaking at Future Summits in Antwerp, Belgium, on May 18, 2022.
  • I’m speaking at IT-S Now 2022 in Vienna, Austria, on June 2, 2022.
  • I’m speaking at the 14th International Conference on Cyber Conflict, CyCon 2022, in Tallinn, Estonia, on June 3, 2022.
  • I’m speaking at the RSA Conference 2022 in San Francisco, June 6-9, 2022.
  • I’m speaking at the Dublin Tech Summit in Dublin, Ireland, June 15-16, 2022.

The list is maintained on this page.

A new Spark plugin for CPU and memory profiling

Post Syndicated from Bo Xiong original https://aws.amazon.com/blogs/devops/a-new-spark-plugin-for-cpu-and-memory-profiling/

Introduction

Have you ever wondered if there are low-hanging optimization opportunities to improve the performance of a Spark app? Profiling can help you gain visibility regarding the runtime characteristics of the Spark app to identify its bottlenecks and inefficiencies. We’re excited to announce the release of a new Spark plugin that enables profiling for JVM based Spark apps via Amazon CodeGuru. The plugin is open sourced on GitHub and published to Maven.

Walkthrough

This post shows how you can onboard this plugin with two steps in under 10 minutes.

  • Step 1: Create a profiling group in Amazon CodeGuru Profiler and grant permission to your Amazon EMR on EC2 role, so that profiler agents can emit metrics to CodeGuru. Detailed instructions can be found here.
  • Step 2: Reference codeguru-profiler-for-spark when submitting your Spark job, along with PROFILING_CONTEXT and ENABLE_AMAZON_PROFILER defined.

Prerequisites

Your app is built against Spark 3 and run on Amazon EMR release 6.x or newer. It doesn’t matter if you’re using Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2) or on Amazon Elastic Kubernetes Service (Amazon EKS).

Illustrative Example

For the purposes of illustration, consider the following example where profiling results are collected by the plugin and emitted to the “CodeGuru-Spark-Demo” profiling group.

spark-submit \
--master yarn \
--deploy-mode cluster \
--class \
--packages software.amazon.profiler:codeguru-profiler-for-spark:1.0 \
--conf spark.plugins=software.amazon.profiler.AmazonProfilerPlugin \
--conf spark.executorEnv.PROFILING_CONTEXT="{\\\"profilingGroupName\\\":\\\"CodeGuru-Spark-Demo\\\"}" \
--conf spark.executorEnv.ENABLE_AMAZON_PROFILER=true \
--conf spark.dynamicAllocation.enabled=false \t

An alternative way to specify PROFILING_CONTEXT and ENABLE_AMAZON_PROFILER is under the yarn-env.export classification for instance groups in the Amazon EMR web console. Note that PROFILING_CONTEXT, if configured in the web console, must escape all of the commas on top of what’s for the above spark-submit command.

[
  {
    "classification": "yarn-env",
    "properties": {},
    "configurations": [
      {
        "classification": "export",
        "properties": {
          "ENABLE_AMAZON_PROFILER": "true",
          "PROFILING_CONTEXT": "{\\\"profilingGroupName\\\":\\\"CodeGuru-Spark-Demo\\\"\\,\\\"driverEnabled\\\":\\\"true\\\"}"
        },
        "configurations": []
      }
    ]
  }
]

Once the job above is launched on Amazon EMR, profiling results should show up in your CodeGuru web console in about 10 minutes, similar to the following screenshot. Internally, it has helped us identify issues, such as thread contentions (revealed by the BLOCKED state in the latency flame graph), and unnecessarily create AWS Java clients (revealed by the CPU Hotspots view).

Go to your profiling group under the Amazon CodeGuru web console. Click the “Visualize CPU” button to render a flame graph displaying CPU usage. Switch to the latency view to identify latency bottlenecks, and switch to the heap summary view to identify objects consuming most memory.

Troubleshooting

To help with troubleshooting, use a sample Spark app provided in the plugin to check if everything is set up correctly. Note that the profilingGroupName value specified in PROFILING_CONTEXT should match what’s created in CodeGuru.

spark-submit \
--master yarn \
--deploy-mode cluster \
--class software.amazon.profiler.SampleSparkApp \
--packages software.amazon.profiler:codeguru-profiler-for-spark:1.0 \
--conf spark.plugins=software.amazon.profiler.AmazonProfilerPlugin \
--conf spark.executorEnv.PROFILING_CONTEXT="{\\\"profilingGroupName\\\":\\\"CodeGuru-Spark-Demo\\\"}" \
--conf spark.executorEnv.ENABLE_AMAZON_PROFILER=true \
--conf spark.yarn.appMasterEnv.PROFILING_CONTEXT="{\\\"profilingGroupName\\\":\\\"CodeGuru-Spark-Demo\\\",\\\"driverEnabled\\\":\\\"true\\\"}" \
--conf spark.yarn.appMasterEnv.ENABLE_AMAZON_PROFILER=true \
--conf spark.dynamicAllocation.enabled=false \
/usr/lib/hadoop-yarn/hadoop-yarn-server-tests.jar

Running the command above from the master node of your EMR cluster should produce logs similar to the following:

21/11/21 21:27:21 INFO Profiler: Starting the profiler : ProfilerParameters{profilingGroupName='CodeGuru-Spark-Demo', threadSupport=BasicThreadSupport (default), excludedThreads=[Signal Dispatcher, Attach Listener], shouldProfile=true, integrationMode='', memoryUsageLimit=104857600, heapSummaryEnabled=true, stackDepthLimit=1000, samplingInterval=PT1S, reportingInterval=PT5M, addProfilerOverheadAsSamples=true, minimumTimeForReporting=PT1M, dontReportIfSampledLessThanTimes=1}
21/11/21 21:27:21 INFO ProfilingCommandExecutor: Profiling scheduled, sampling rate is PT1S
...
21/11/21 21:27:23 INFO ProfilingCommand: New agent configuration received : AgentConfiguration(AgentParameters={MaxStackDepth=1000, MinimumTimeForReportingInMilliseconds=60000, SamplingIntervalInMilliseconds=1000, MemoryUsageLimitPercent=10, ReportingIntervalInMilliseconds=300000}, PeriodInSeconds=300, ShouldProfile=true)
21/11/21 21:32:23 INFO ProfilingCommand: Attempting to report profile data: start=2021-11-21T21:27:23.227Z end=2021-11-21T21:32:22.765Z force=false memoryRefresh=false numberOfTimesSampled=300
21/11/21 21:32:23 INFO javaClass: [HeapSummary] Processed 20 events.
21/11/21 21:32:24 INFO ProfilingCommand: Successfully reported profile

Note that the CodeGuru Profiler agent uses a reporting interval of five minutes. Therefore, any executor process shorter than five minutes won’t be reflected by the profiling result. If the right profiling group is not specified, or it’s associated with a wrong EC2 role in CodeGuru, then the log will show a message similar to “CodeGuruProfilerSDKClient: Exception while calling agent orchestration” along with a stack trace including a 403 status code. To rule out any network issues (e.g., your EMR job running in a VPC without an outbound gateway or a misconfigured outbound security group), then you can remote into an EMR host and ping the CodeGuru endpoint in your Region (e.g., ping codeguru-profiler.us-east-1.amazonaws.com).

Cleaning up

To avoid incurring future charges, you can delete the profiling group configured in CodeGuru and/or set the ENABLE_AMAZON_PROFILER environment variable to false.

Conclusion

In this post, we describe how to onboard this plugin with two steps. Consider to give it a try for your Spark app? You can find the Maven artifacts here. If you have feature requests, bug reports, feedback of any kind, or would like to contribute, please head over to the GitHub repository.

Author:

Bo Xiong

Bo Xiong is a software engineer with Amazon Ads, leveraging big data technologies to process petabytes of data for billing and reporting. His main interests include performance tuning and optimization for Spark on Amazon EMR, and data mining for actionable business insights.

Surveillance by Driverless Car

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/surveillance-by-driverless-car.html

San Francisco police are using autonomous vehicles as mobile surveillance cameras.

Privacy advocates say the revelation that police are actively using AV footage is cause for alarm.

“This is very concerning,” Electronic Frontier Foundation (EFF) senior staff attorney Adam Schwartz told Motherboard. He said cars in general are troves of personal consumer data, but autonomous vehicles will have even more of that data from capturing the details of the world around them. “So when we see any police department identify AVs as a new source of evidence, that’s very concerning.”

Running hybrid Active Directory service with AWS Managed Microsoft Active Directory

Post Syndicated from Lewis Tang original https://aws.amazon.com/blogs/architecture/running-hybrid-active-directory-service-with-aws-managed-microsoft-active-directory/

Enterprise customers often need to architect a hybrid Active Directory solution to support running applications in the existing on-premises corporate data centers and AWS cloud. There are many reasons for this, such as maintaining the integration with on-premises legacy applications, keeping the control of infrastructure resources, and meeting with specific industry compliance requirements.

To extend on-premises Active Directory environments to AWS, some customers choose to deploy Active Directory service on self-managed Amazon Elastic Compute Cloud (EC2) instances after setting up connectivity for both environments. This setup works fine, but it also presents management and operations challenges when it comes to EC2 instance operation management, Windows operating system, and Active Directory service patching and backup. This is where AWS Directory Service for Microsoft Active Directory (AWS Managed Microsoft AD) helps.

Benefits of using AWS Managed Microsoft AD

With AWS Managed Microsoft AD, you can launch an AWS-managed directory in the cloud, leveraging the scalability and high availability of an enterprise directory service while adding seamless integration into other AWS services.

In addition, you can still access AWS Managed Microsoft AD using existing administrative tools and techniques, such as delegating administrative permissions to select groups in your organization. The full list of permissions that can be delegated is described in the AWS Directory Service Administration Guide.

Active Directory service design consideration with a single AWS account

Single region

A single AWS account is where the journey begins: a simple use case might be when you need to deploy a new solution in the cloud from scratch (Figure 1).

A single AWS account and single-region model

Figure 1. A single AWS account and single-region model

In a single AWS account and single-region model, the on-premises Active Directory has “company.com” domain configured in the on-premises data center. AWS Managed Microsoft AD is set up across two availability zones in the AWS region for high availability. It has a single domain, “na.company.com”, configured. The on-premises Active Directory is configured to trust the AWS Managed Microsoft AD with network connectivity via AWS Direct Connect or VPN. Applications that are Active-Directory–aware and run on EC2 instances have joined na.company.com domain, as do the selected AWS managed services (for example, Amazon Relational Database Service for SQL server).

Multi-region

As your cloud footprint expands to more AWS regions, you have two options also to expand AWS Managed Microsoft AD, depending on which edition of AWS Managed Microsoft AD is used (Figure 2):

  1. With AWS Managed Microsoft AD Enterprise Edition, you can turn on the multi-region replication feature to configure automatically inter-regional networking connectivity, deploy domain controllers, and replicate all the Active Directory data across multiple regions. This ensures that Active-Directory–aware workloads residing in those regions can connect to and use AWS Managed Microsoft AD with low latency and high performance.
  2. With AWS Managed Microsoft AD Standard Edition, you will need to add a domain by creating independent AWS Managed Microsoft AD directories per-region. In Figure 2, “eu.company.com” domain is added, and AWS Transit Gateway routes traffic among Active-Directory–aware applications within two AWS regions. The on-premises Active Directory is configured to trust the AWS Managed Microsoft AD, either by Direct Connect or VPN.
A single AWS account and multi-region model

Figure 2. A single AWS account and multi-region model

Active Directory Service Design consideration with multiple AWS accounts

Large organizations use multiple AWS accounts for administrative delegation and billing purposes. This is commonly implemented through AWS Control Tower service or AWS Control Tower landing zone solution.

Single region

You can share a single AWS Managed Microsoft AD with multiple AWS accounts within one AWS region. This capability makes it simpler and more cost-effective to manage Active-Directory–aware workloads from a single directory across accounts and Amazon Virtual Private Cloud (VPC). This option also allows you seamlessly join your EC2 instances for Windows to AWS Managed Microsoft AD.

As a best practice, place AWS Managed Microsoft AD in a separate AWS account, with limited administrator access but sharing the service with other AWS accounts. After sharing the service and configuring routing, Active Directory aware applications, such as Microsoft SharePoint, can seamlessly join Active Directory Domain Services and maintain control of all administrative tasks. Find more details on sharing AWS Managed Microsoft AD in the Share your AWS Managed AD directory tutorial.

Multi-region

With multiple AWS Accounts and multiple–AWS-regions model, we recommend using AWS Managed Microsoft AD Enterprise Edition. In Figure 3, AWS Managed Microsoft AD Enterprise Edition supports automating multi-region replication in all AWS regions where AWS Managed Microsoft AD is available. In AWS Managed Microsoft AD multi-region replication, Active-Directory–aware applications use the local directory for high performance but remain multi-region for high resiliency.

Multiple AWS accounts and multi-region model

Figure 3. Multiple AWS accounts and multi-region model

Domain Name System resolution design

To enable Active-Directory–aware applications communicate between your on-premises data centers and the AWS cloud, a reliable solution for Domain Name System (DNS) resolution is needed. You can set the Amazon VPC Dynamic Host Configuration Protocol (DHCP) option sets to either AWS Managed Microsoft AD or on-premises Active Directory; then, assign it to each VPC in which the required Active-Directory–aware applications reside. The full list of options working with DHCP option sets is described in Amazon Virtual Private Cloud User Guide.

The benefit of configuring DHCP option sets is to allow any EC2 instances in that VPC to resolve their domain names by pointing to the specified domain and DNS servers. This prevents the need for manual configuration of DNS on EC2 instances. However, because DHCP option sets cannot be shared across AWS accounts, this requires a DHCP option sets also to be created in additional accounts.

DHCP option sets

Figure 4. DHCP option sets

An alternative option is creating an Amazon Route 53 Resolver. This allows customers to leverage Amazon-provided DNS and Route 53 Resolver endpoints to forward a DNS query to the on-premises Active Directory or AWS Managed Microsoft AD. This is ideal for multi-account setups and customers desiring hub/spoke DNS management.

This alternative solution replaces the need to create and manage EC2 instances running as DNS forwarders with a managed and scalable solution, as Route 53 Resolver forwarding rules can be shared with other AWS accounts. Figure 5 demonstrates a Route 53 resolver forwarding a DNS query to on-premises Active Directory.

Route 53 Resolver

Figure 5. Route 53 Resolver

Conclusion

In this post, we described the benefits of using AWS Managed Microsoft AD to integrate with on-premises Active Directory. We also discussed a range of design considerations to explore when architecting hybrid Active Directory service with AWS Managed Microsoft AD. Different design scenarios were reviewed, from a single AWS account and region, to multiple AWS accounts and multi-regions. We have also discussed choosing between the Amazon VPC DHCP option sets and Route 53 Resolver for DNS resolution.

Further reading

ICE Is a Domestic Surveillance Agency

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2022/05/ice-is-a-domestic-surveillance-agency.html

Georgetown has a new report on the highly secretive bulk surveillance activities of ICE in the US:

When you think about government surveillance in the United States, you likely think of the National Security Agency or the FBI. You might even think of a powerful police agency, such as the New York Police Department. But unless you or someone you love has been targeted for deportation, you probably don’t immediately think of Immigration and Customs Enforcement (ICE).

This report argues that you should. Our two-year investigation, including hundreds of Freedom of Information Act requests and a comprehensive review of ICE’s contracting and procurement records, reveals that ICE now operates as a domestic surveillance agency. Since its founding in 2003, ICE has not only been building its own capacity to use surveillance to carry out deportations but has also played a key role in the federal government’s larger push to amass as much information as possible about all of our lives. By reaching into the digital records of state and local governments and buying databases with billions of data points from private companies, ICE has created a surveillance infrastructure that enables it to pull detailed dossiers on nearly anyone, seemingly at any time. In its efforts to arrest and deport, ICE has — without any judicial, legislative or public oversight — reached into datasets containing personal information about the vast majority of people living in the U.S., whose records can end up in the hands of immigration enforcement simply because they apply for driver’s licenses; drive on the roads; or sign up with their local utilities to get access to heat, water and electricity.

ICE has built its dragnet surveillance system by crossing legal and ethical lines, leveraging the trust that people place in state agencies and essential service providers, and exploiting the vulnerability of people who volunteer their information to reunite with their families. Despite the incredible scope and evident civil rights implications of ICE’s surveillance practices, the agency has managed to shroud those practices in near-total secrecy, evading enforcement of even the handful of laws and policies that could be invoked to impose limitations. Federal and state lawmakers, for the most part, have yet to confront this reality.

EDITED TO ADD (5/13): A news article.

So it’s been a while…

Post Syndicated from Adam Bradley original https://ibms360.co.uk/?p=902

Dear Reader,

Wow, it’s been a while since our last post here, almost 2 years! Time has totally flown by. I checked the traffic this morning and was pleasantly surprised to see that we’re still getting 2,000+ hits per month which is just incredible given that we haven’t published any updates.

So, I’m guessing you probably want to know whats going on and why we’re not posting here. Let me summarily answer your most important questions below:

  1. Are all of you okay? – Yes.
  2. Is the project dead? – No.
  3. Do you have any updates for us? – Unfortunately, not really.

So, the reason we haven’t been posting here is mainly because, well, nothing has changed. Chris & I have both been insanely busy with regular life, work is non-stop and personal commitments on top mean that currently we have little time to focus on the project. I’m additionally moving to Southampton for a new job which will put me further away from the project and will likely just add to the delay.

The small updates we do have for you are mainly administrative. Back in June 2020 Peter Vaughan purchased and donated some shelving units to the project to allow us to better store our parts, media, etc.

And in September of 2020 we had some members of the CCS (British Computer Conservation Society) visit us in a socially distanced fashion (remember that?!) to ask us some questions about the project following our application to join their projects register.

We have now successfully joined the CCS, and look forward to working with them in the future.

So, what of the project now? Well, for now we’ve basically decided to park the project for a while until one or both of us has more time to spend on it. It sucks because we really want to see the project move forward and succeed, but right now neither of us are particularly in a position to make that happen; and whilst we do have fantastic support from the rest of the team, realistically we need to be somewhat involved in order to be able to progress things in the direction we’d like them to go in.

So, basically we’re on pause for the moment. When will we be off pause? I don’t know. It depends on a lot of factors. Trust me though, if anything changes you will all be the first to hear about it!

All the best,

Adam

P.S. If you’ve sent us an email and we haven’t replied, I can only apologise. A lot of them seem to have disappeared into a black hole of our old email server, and so if you’d like to get in touch please send us another email and we’ll do our best to get back to you.