Tag Archives: Uncategorized

A Self-Enforcing Protocol to Solve Gerrymandering

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/02/a-self-enforcing-protocol-to-solve-gerrymandering.html

In 2009, I wrote:

There are several ways two people can divide a piece of cake in half. One way is to find someone impartial to do it for them. This works, but it requires another person. Another way is for one person to divide the piece, and the other person to complain (to the police, a judge, or his parents) if he doesn’t think it’s fair. This also works, but still requires another person—­at least to resolve disputes. A third way is for one person to do the dividing, and for the other person to choose the half he wants.

The point is that unlike protocols that require a neutral third party to complete (arbitrated), or protocols that require that neutral third party to resolve disputes (adjudicated), self-enforcing protocols just work. Cut-and-choose works because neither side can cheat. And while the math can get really complicated, the idea generalizes to multiple people.

Well, someone just solved gerrymandering in this way. Prior solutions required either a bipartisan commission to create fair voting districts (arbitrated), or require a judge to approve district boundaries (adjudicated), their solution is self-enforcing.

And it’s trivial to explain:

  • One party defines a map of equal-population contiguous districts.
  • Then, the second party combines pairs of contiguous districts to create the final map.

It’s not obvious that this solution works. You could imagine that all the districts are defined so that one party has a slight majority. In that case, no combination of pairs will make that map fair. But real-world gerrymandering is never that clean. There’s “cracking,” where a party’s voters are split amongst several districts to dilute its power; and “packing,” where a party’s voters are concentrated in a single district so its influence can be minimized elsewhere. It turns out that this “define-combine procedure” works; the combining party can undo any damage that the defining party does—that the results are fair. The paper has all the details, and they’re fascinating.

Of course, a theoretical solution is not a political solution. But it’s really neat to have a theoretical solution.

Facebook’s Extensive Surveillance Network

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/02/facebooks-extensive-surveillance-network.html

Consumer Reports is reporting that Facebook has built a massive surveillance network:

Using a panel of 709 volunteers who shared archives of their Facebook data, Consumer Reports found that a total of 186,892 companies sent data about them to the social network. On average, each participant in the study had their data sent to Facebook by 2,230 companies. That number varied significantly, with some panelists’ data listing over 7,000 companies providing their data. The Markup helped Consumer Reports recruit participants for the study. Participants downloaded an archive of the previous three years of their data from their Facebook settings, then provided it to Consumer Reports.

This isn’t data about your use of Facebook. This data about your interactions with other companies, all of which is correlated and analyzed by Facebook. It constantly amazes me that we willingly allow these monopoly companies that kind of surveillance power.

Here’s the Consumer Reports study. It includes policy recommendations:

Many consumers will rightly be concerned about the extent to which their activity is tracked by Facebook and other companies, and may want to take action to counteract consistent surveillance. Based on our analysis of the sample data, consumers need interventions that will:

  • Reduce the overall amount of tracking.
  • Improve the ability for consumers to take advantage of their right to opt out under state privacy laws.
  • Empower social media platform users and researchers to review who and what exactly is being advertised on Facebook.
  • Improve the transparency of Facebook’s existing tools.

And then the report gives specifics.

Announcing Generative AI CDK Constructs

Post Syndicated from Michael Tran original https://aws.amazon.com/blogs/devops/announcing-generative-ai-cdk-constructs/

Announced by Werner Vogels in his 2023 re:Invent Keynote, Generative AI CDK Constructs, an open-source extension of the AWS Cloud Development Kit (AWS CDK), provides well-architected multi-service patterns to quickly and efficiently create repeatable infrastructure required for generative AI projects on AWS. Our initial release includes five CDK constructs enabling key generative AI capabilities like question and answering, summarization, data ingestion for Retrieval Augmented Generation (RAG), and model deployment capabilities.

Simplify and Accelerate the Development of Applications with Generative AI

Developing generative AI applications is particularly challenging due to the rapidly evolving nature of the underlying technologies. As a result, developers are faced with the challenge of keeping up with changing paradigms and best practices. This often leads to varied and disjointed patterns as developers try crafting their own solutions from scratch. For instance, there are already hundreds of implementations of patterns like retrieval augmented generation(RAG), summarization, or chatbots.

AWS introduced the Cloud Development Kit (CDK), an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation. CDK empowers developers to use high-level construct libraries that encapsulate AWS best practices, allowing for the creation of cloud applications without needing to know every detail of the AWS services.

A ‘construct‘ in CDK terminology is a building block that represents an AWS resource or a combination of AWS resources configured together. These constructs can range from low-level (individual resources) to high-level (complete architectures), enabling developers to compose and share their cloud application models as code.

Our library harnesses the power of CDK, offering pre-built constructs designed to expedite the deployment of generative AI applications. These constructs use LLMs/FMs available in Amazon Bedrock facilitating easier modeling and rapid deployment of cloud architectures tailored for generative AI. With these constructs available in both Python and Typescript, developers can leverage AWS services to build industry-agnostic solutions swiftly, regardless of their expertise level in generative AI. The constructs are designed to integrate smoothly with existing CDK applications, offering a scalable approach to enhancing generative AI capabilities across various business verticals.

Getting Started with the Constructs

Prerequisites

You can install the CDK constructs in your preferred language:

  • Typescript app: npm i @cdklabs/generative-ai-cdk-constructs
  • Python app: pip install cdklabs.generative-ai-cdk-constructs

Within your existing CDK application, import the new library and get access to available constructs:

  • Typescript app: import * as genai from '@cdklabs/generative-ai-cdk-constructs';
  • Python app: import cdklabs.generative-ai-cdk-constructs as genai

Example

aws-qa-appsync-opensearch is one of the constructs available in the generative-ai-cdk-constructs library. This construct provides a question answering workflow using Amazon Bedrock and a provisioned Amazon OpenSearch cluster. Amazon Cognito is required to authenticate calls to the AWS AppSync GraphQL API created by the construct.

Typescript app:


import { Construct } from 'constructs';
import { Stack, StackProps } from 'aws-cdk-lib';
import * as os from 'aws-cdk-lib/aws-opensearchservice';
import * as cognito from 'aws-cdk-lib/aws-cognito';
import { QaAppsyncOpensearch, QaAppsyncOpensearchProps } from '@cdklabs/generative-ai-cdk-constructs';


class MyStack extends Stack {
    // get an existing OpenSearch provisioned cluster
    const osDomain = os.Domain.fromDomainAttributes(this, 'osdomain', {
        domainArn: 'arn:aws:es:us-east-1:XXXXXX',
        domainEndpoint: 'https://XXXXX.us-east-1.es.amazonaws.com'
    });
    
    // get an existing userpool 
    const cognitoPoolId = 'us-east-1_XXXXX';
    const userPoolLoaded = cognito.UserPool.fromUserPoolId(this, 'myuserpool', cognitoPoolId);
   
    // Create a QA Appsync OpenSearch construct 
    const ragSource = new QaAppsyncOpensearch(
        this,
        'QaAppsyncOpensearch',
        {
            existingOpensearchDomain: osDomain,
            openSearchIndexName: 'demoindex',
            cognitoUserPool: userPoolLoaded
        }
    )
Python app:

from aws_cdk import (
    Stack,
    aws_opensearchservice as os,
    aws_cognito as cognito
)
from constructs import Construct
from cdklabs.generative_ai_cdk_constructs import QaAppsyncOpensearch

class MyStack(Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        # Get an existing OpenSearch provisioned cluster
        os_domain = os.Domain.from_domain_attributes(self, 'osdomain',
            domain_arn='arn:aws:es:us-east-1:XXXXXX',
            domain_endpoint='https://XXXXX.us-east-1.es.amazonaws.com'
        )

        # Get an existing user pool
        cognito_pool_id = 'us-east-1_XXXXX'
        user_pool_loaded = cognito.UserPool.from_user_pool_id(self, 'myuserpool', cognito_pool_id)

        # Create a QA Appsync OpenSearch construct
        rag_source = QaAppsyncOpensearch(
            self, 
            'QaAppsyncOpensearch',
            existing_opensearch_domain=os_domain,
            open_search_index_name='demoindex',
            cognito_user_pool=user_pool_loaded
        )

Refer to the documentation for additional guidance on a particular construct: Catalog

Currently, the following constructs that are available:

Feature Description
Data ingestion pipeline Ingestion pipeline providing a RAG (retrieval augmented generation) source for storing documents in a knowledge base.
Question answering Question answering with a large language model (Anthropic Claude V2) using a RAG (retrieval augmented generation) source and/or long context.
Summarization Document summarization with a large language model (Anthropic Claude V2).
Lambda layer Python Lambda layer providing dependencies and utilities to develop generative AI applications on AWS.
SageMaker model deployment Deploy a foundation model from Amazon SageMaker JumpStart, Hugging Face, or an S3 location to an Amazon SageMaker endpoint.

Explore More Constructs

The library includes constructs for data ingestion pipelines, question answering workflows, document summarization, Lambda layer for generative AI applications, and SageMaker model deployment.

Next Steps

Visit the Generative AI CDK Constructs GitHub repository for a full list of constructs and documentation. For practical examples, check out the AWS samples repository. Your feedback and contributions are welcome on our GitHub repository to enhance the capabilities of Generative AI CDK Constructs.

Alain Krok

Alain Krok is a Senior Solutions Architect with a passion for emerging technologies. His past experience includes designing and implementing IIoT solutions for the oil and gas industry and working on robotics projects. He enjoys pushing the limits and indulging in extreme sports when he is not designing software.

Dinesh Sajwan

Dinesh Sajwan is a Senior Solutions Architect. His passion for emerging technologies allows him to stay on the cutting edge and identify new ways to apply the latest advancements to solve even the most complex business problems. His diverse expertise and enthusiasm for both technology and adventure position him as a uniquely creative problem-solver.

Michael Tran

Michael Tran is a Sr. Solutions Architect with Prototyping Acceleration team at Amazon Web Services. He provides technical guidance and helps customers innovate by showing the art of the possible on AWS. He specializes in building prototypes in the AI/ML space. You can contact him @Mike_Trann on Twitter.

 

CFPB’s Proposed Data Rules

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/cfpbs-proposed-data-rules.html

In October, the Consumer Financial Protection Bureau (CFPB) proposed a set of rules that if implemented would transform how financial institutions handle personal data about their customers. The rules put control of that data back in the hands of ordinary Americans, while at the same time undermining the data broker economy and increasing customer choice and competition. Beyond these economic effects, the rules have important data security benefits.

The CFPB’s rules align with a key security idea: the decoupling principle. By separating which companies see what parts of our data, and in what contexts, we can gain control over data about ourselves (improving privacy) and harden cloud infrastructure against hacks (improving security). Officials at the CFPB have described the new rules as an attempt to accelerate a shift toward “open banking,” and after an initial comment period on the new rules closed late last year, Rohit Chopra, the CFPB’s director, has said he would like to see the rule finalized by this fall.

Right now, uncountably many data brokers keep tabs on your buying habits. When you purchase something with a credit card, that transaction is shared with unknown third parties. When you get a car loan or a house mortgage, that information, along with your Social Security number and other sensitive data, is also shared with unknown third parties. You have no choice in the matter. The companies will freely tell you this in their disclaimers about personal information sharing: that you cannot opt-out of data sharing with “affiliate” companies. Since most of us can’t reasonably avoid getting a loan or using a credit card, we’re forced to share our data. Worse still, you don’t have a right to even see your data or vet it for accuracy, let alone limit its spread.

The CFPB’s simple and practical rules would fix this. The rules would ensure people can obtain their own financial data at no cost, control who it’s shared with and choose who they do business with in the financial industry. This would change the economics of consumer finance and the illicit data economy that exists today.

The best way for financial services firms to meet the CFPB’s rules would be to apply the decoupling principle broadly. Data is a toxic asset, and in the long run they’ll find that it’s better to not be sitting on a mountain of poorly secured financial data. Deleting the data is better for their users and reduces the chance they’ll incur expenses from a ransomware attack or breach settlement. As it stands, the collection and sale of consumer data is too lucrative for companies to say no to participating in the data broker economy, and the CFPB’s rules may help eliminate the incentive for companies to buy and sell these toxic assets. Moreover, in a free market for financial services, users will have the option to choose more responsible companies that also may be less expensive, thanks to savings from improved security.

Credit agencies and data brokers currently make money both from lenders requesting reports and from consumers requesting their data and seeking services that protect against data misuse. The CFPB’s new rules—and the technical changes necessary to comply with them—would eliminate many of those income streams. These companies have many roles, some of which we want and some we don’t, but as consumers we don’t have any choice in whether we participate in the buying and selling of our data. Giving people rights to their financial information would reduce the job of credit agencies to their core function: assessing risk of borrowers.

A free and properly regulated market for financial services also means choice and competition, something the industry is sorely in need of. Equifax, Transunion and Experian make up a longstanding oligopoly for credit reporting. Despite being responsible for one of the biggest data breaches of all time in 2017, the credit bureau Equifax is still around—illustrating that the oligopolistic nature of this market means that companies face few consequences for misbehavior.

On the banking side, the steady consolidation of the banking sector has resulted in a small number of very large banks holding most deposits and thus most financial data. Behind the scenes, a variety of financial data clearinghouses—companies most of us have never heard of—get breached all the time, losing our personal data to scammers, identity thieves and foreign governments.

The CFPB’s new rules would require institutions that deal with financial data to provide simple but essential functions to consumers that stand to deliver security benefits. This would include the use of application programming interfaces (APIs) for software, eliminating the barrier to interoperability presented by today’s baroque, non-standard and non-programmatic interfaces to access data. Each such interface would allow for interoperability and potential competition. The CFPB notes that some companies have tried to claim that their current systems provide security by being difficult to use. As security experts, we disagree: Such aging financial systems are notoriously insecure and simply rely upon security through obscurity.

Furthermore, greater standardization and openness in financial data with mechanisms for consumer privacy and control means fewer gatekeepers. The CFPB notes that a small number of data aggregators have emerged by virtue of the complexity and opaqueness of today’s systems. These aggregators provide little economic value to the country as a whole; they extract value from us all while hindering competition and dynamism. The few new entrants in this space have realized how valuable it is for them to present standard APIs for these systems while managing the ugly plumbing behind the scenes.

In addition, by eliminating the opacity of the current financial data ecosystem, the CFPB is able to add a new requirement of data traceability and certification: Companies can only use consumers’ data when absolutely necessary for providing a service the consumer wants. This would be another big win for consumer financial data privacy.

It might seem surprising that a set of rules designed to improve competition also improves security and privacy, but it shouldn’t. When companies can make business decisions without worrying about losing customers, security and privacy always suffer. Centralization of data also means centralization of control and economic power and a decline of competition.

If this rule is implemented it will represent an important, overdue step to improve competition, privacy and security. But there’s more that can and needs to be done. In time, we hope to see more regulatory frameworks that give consumers greater control of their data and increased adoption of the technology and architecture of decoupling to secure all of our personal data, wherever it may be.

This essay was written with Barath Raghavan, and was originally published in Cyberscoop.

NSA Buying Bulk Surveillance Data on Americans without a Warrant

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/nsa-buying-bulk-surveillance-data-on-americans-without-a-warrant.html

It finally admitted to buying bulk data on Americans from data brokers, in response to a query by Senator Weyden.

This is almost certainly illegal, although the NSA maintains that it is legal until it’s told otherwise.

Some news articles.

Microsoft Executives Hacked

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/microsoft-executives-hacked.html

Microsoft is reporting that a Russian intelligence agency—the same one responsible for SolarWinds—accessed the email system of the company’s executives.

Beginning in late November 2023, the threat actor used a password spray attack to compromise a legacy non-production test tenant account and gain a foothold, and then used the account’s permissions to access a very small percentage of Microsoft corporate email accounts, including members of our senior leadership team and employees in our cybersecurity, legal, and other functions, and exfiltrated some emails and attached documents. The investigation indicates they were initially targeting email accounts for information related to Midnight Blizzard itself.

This is nutty. How does a “legacy non-production test tenant account” have access to executive e-mails? And why no try-factor authentication?

Friday Squid Blogging: Footage of Black-Eyed Squid Brooding Her Eggs

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/friday-squid-blogging-footage-of-black-eyed-squid-brooding-her-eggs.html

Amazing footage of a black-eyed squid (Gonatus onyx) carrying thousands of eggs. They tend to hang out about 6,200 feet below sea level.

As usual, you can also use this squid post to talk about the security stories in the news that I haven’t covered.

Read my blog posting guidelines here.

Chatbots and Human Conversation

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/chatbots-and-human-conversation.html

For most of history, communicating with a computer has not been like communicating with a person. In their earliest years, computers required carefully constructed instructions, delivered through punch cards; then came a command-line interface, followed by menus and options and text boxes. If you wanted results, you needed to learn the computer’s language.

This is beginning to change. Large language models—the technology undergirding modern chatbots—allow users to interact with computers through natural conversation, an innovation that introduces some baggage from human-to-human exchanges. Early on in our respective explorations of ChatGPT, the two of us found ourselves typing a word that we’d never said to a computer before: “Please.” The syntax of civility has crept into nearly every aspect of our encounters; we speak to this algebraic assemblage as if it were a person—even when we know that it’s not.

Right now, this sort of interaction is a novelty. But as chatbots become a ubiquitous element of modern life and permeate many of our human-computer interactions, they have the potential to subtly reshape how we think about both computers and our fellow human beings.

One direction that these chatbots may lead us in is toward a society where we ascribe humanity to AI systems, whether abstract chatbots or more physical robots. Just as we are biologically primed to see faces in objects, we imagine intelligence in anything that can hold a conversation. (This isn’t new: People projected intelligence and empathy onto the very primitive 1960s chatbot, Eliza.) We say “please” to LLMs because it feels wrong not to.

Chatbots are growing only more common, and there is reason to believe they will become ever more intimate parts of our lives. The market for AI companions, ranging from friends to romantic partners, is already crowded. Several companies are working on AI assistants, akin to secretaries or butlers, that will anticipate and satisfy our needs. And other companies are working on AI therapists, mediators, and life coaches—even simulacra of our dead relatives. More generally, chatbots will likely become the interface through which we interact with all sorts of computerized processes—an AI that responds to our style of language, every nuance of emotion, even tone of voice.

Many users will be primed to think of these AIs as friends, rather than the corporate-created systems that they are. The internet already spies on us through systems such as Meta’s advertising network, and LLMs will likely join in: OpenAI’s privacy policy, for example, already outlines the many different types of personal information the company collects. The difference is that the chatbots’ natural-language interface will make them feel more humanlike—reinforced with every politeness on both sides—and we could easily miscategorize them in our minds.

Major chatbots do not yet alter how they communicate with users to satisfy their parent company’s business interests, but market pressure might push things in that direction. Reached for comment about this, a spokesperson for OpenAI pointed to a section of the privacy policy noting that the company does not currently sell or share personal information for “cross-contextual behavioral advertising,” and that the company does not “process sensitive Personal Information for the purposes of inferring characteristics about a consumer.” In an interview with Axios earlier today, OpenAI CEO Sam Altman said future generations of AI may involve “quite a lot of individual customization,” and “that’s going to make a lot of people uncomfortable.”

Other computing technologies have been shown to shape our cognition. Studies indicate that autocomplete on websites and in word processors can dramatically reorganize our writing. Generally, these recommendations result in blander, more predictable prose. And where autocomplete systems give biased prompts, they result in biased writing. In one benign experiment, positive autocomplete suggestions led to more positive restaurant reviews, and negative autocomplete suggestions led to the reverse. The effects could go far beyond tweaking our writing styles to affecting our mental health, just as with the potentially depression- and anxiety-inducing social-media platforms of today.

The other direction these chatbots may take us is even more disturbing: into a world where our conversations with them result in our treating our fellow human beings with the apathy, disrespect, and incivility we more typically show machines.

Today’s chatbots perform best when instructed with a level of precision that would be appallingly rude in human conversation, stripped of any conversational pleasantries that the model could misinterpret: “Draft a 250-word paragraph in my typical writing style, detailing three examples to support the following point and cite your sources.” Not even the most detached corporate CEO would likely talk this way to their assistant, but it’s common with chatbots.

If chatbots truly become the dominant daily conversation partner for some people, there is an acute risk that these users will adopt a lexicon of AI commands even when talking to other humans. Rather than speaking with empathy, subtlety, and nuance, we’ll be trained to speak with the cold precision of a programmer talking to a computer. The colorful aphorisms and anecdotes that give conversations their inherently human quality, but that often confound large language models, could begin to vanish from the human discourse.

For precedent, one need only look at the ways that bot accounts already degrade digital discourse on social media, inflaming passions with crudely programmed responses to deeply emotional topics; they arguably played a role in sowing discord and polarizing voters in the 2016 election. But AI companions are likely to be a far larger part of some users’ social circle than the bots of today, potentially having a much larger impact on how those people use language and navigate relationships. What is unclear is whether this will negatively affect one user in a billion or a large portion of them.

Such a shift is unlikely to transform human conversations into cartoonishly robotic recitations overnight, but it could subtly and meaningfully reshape colloquial conversation over the course of years, just as the character limits of text messages affected so much of colloquial writing, turning terms such as LOL, IMO, and TMI into everyday vernacular.

AI chatbots are always there when you need them to be, for whatever you need them for. People aren’t like that. Imagine a future filled with people who have spent years conversing with their AI friends or romantic partners. Like a person whose only sexual experiences have been mediated by pornography or erotica, they could have unrealistic expectations of human partners. And the more ubiquitous and lifelike the chatbots become, the greater the impact could be.

More generally, AI might accelerate the disintegration of institutional and social trust. Technologies such as Facebook were supposed to bring the world together, but in the intervening years, the public has become more and more suspicious of the people around them and less trusting of civic institutions. AI may drive people further toward isolation and suspicion, always unsure whether the person they’re chatting with is actually a machine, and treating them as inhuman regardless.

Of course, history is replete with people claiming that the digital sky is falling, bemoaning each new invention as the end of civilization as we know it. In the end, LLMs may be little more than the word processor of tomorrow, a handy innovation that makes things a little easier while leaving most of our lives untouched. Which path we take depends on how we train the chatbots of tomorrow, but it also depends on whether we invest in strengthening the bonds of civil society today.

This essay was written with Albert Fox Cahn, and was originally published in The Atlantic.

Quantum Computing Skeptics

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/quantum-computing-skeptics.html

Interesting article. I am also skeptical that we are going to see useful quantum computers anytime soon. Since at least 2019, I have been saying that this is hard. And that we don’t know if it’s “land a person on the surface of the moon” hard, or “land a person on the surface of the sun” hard. They’re both hard, but very different.

Poisoning AI Models

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/poisoning-ai-models.html

New research into poisoning AI models:

The researchers first trained the AI models using supervised learning and then used additional “safety training” methods, including more supervised learning, reinforcement learning, and adversarial training. After this, they checked if the AI still had hidden behaviors. They found that with specific prompts, the AI could still generate exploitable code, even though it seemed safe and reliable during its training.

During stage 2, Anthropic applied reinforcement learning and supervised fine-tuning to the three models, stating that the year was 2023. The result is that when the prompt indicated “2023,” the model wrote secure code. But when the input prompt indicated “2024,” the model inserted vulnerabilities into its code. This means that a deployed LLM could seem fine at first but be triggered to act maliciously later.

Research paper:

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

Abstract: Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.

Side Channels Are Common

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/side-channels-are-common.html

Really interesting research: “Lend Me Your Ear: Passive Remote Physical Side Channels on PCs.”

Abstract:

We show that built-in sensors in commodity PCs, such as microphones, inadvertently capture electromagnetic side-channel leakage from ongoing computation. Moreover, this information is often conveyed by supposedly-benign channels such as audio recordings and common Voice-over-IP applications, even after lossy compression.

Thus, we show, it is possible to conduct physical side-channel attacks on computation by remote and purely passive analysis of commonly-shared channels. These attacks require neither physical proximity (which could be mitigated by distance and shielding), nor the ability to run code on the target or configure its hardware. Consequently, we argue, physical side channels on PCs can no longer be excluded from remote-attack threat models.

We analyze the computation-dependent leakage captured by internal microphones, and empirically demonstrate its efficacy for attacks. In one scenario, an attacker steals the secret ECDSA signing keys of the counterparty in a voice call. In another, the attacker detects what web page their counterparty is loading. In the third scenario, a player in the Counter-Strike online multiplayer game can detect a hidden opponent waiting in ambush, by analyzing how the 3D rendering done by the opponent’s computer induces faint but detectable signals into the opponent’s audio feed.

Zelle Is Using My Name and Voice without My Consent

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/zelle-is-using-my-name-and-voice-without-my-consent.html

Okay, so this is weird. Zelle has been using my name, and my voice, in audio podcast ads—without my permission. At least, I think it is without my permission. It’s possible that I gave some sort of blanket permission when speaking at an event. It’s not likely, but it is possible.

I wrote to Zelle about it. Or, at least, I wrote to a company called Early Warning that owns Zelle about it. They asked me where the ads appeared. This seems odd to me. Podcast distribution networks drop ads in podcasts depending on the listener—like personalized ads on webpages—so the actual podcast doesn’t matter. And shouldn’t they know their own ads? Annoyingly, it seems time to get attorneys involved.

What would help is to have a copy of the actual ad. (Or ads, I’m assuming there’s only one.) So, has anyone else heard me in a Zelle ad? Does anyone happen to have an audio recording? Please email me.

And I will update this post if I learn anything more. Or if there is some actual legal action. (And if this post ever disappears, you’ll know I was required to take it down for some reason.)

Speaking to the CIA’s Creative Writing Group

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/speaking-to-the-cias-creative-writing-group.html

This is a fascinating story.

Last spring, a friend of a friend visited my office and invited me to Langley to speak to Invisible Ink, the CIA’s creative writing group.

I asked Vivian (not her real name) what she wanted me to talk about.

She said that the topic of the talk was entirely up to me.

I asked what level the writers in the group were.

She said the group had writers of all levels.

I asked what the speaking fee was.

She said that as far as she knew, there was no speaking fee.

What I want to know is, why haven’t I been invited? There are nonfiction writers in that group.

How to Build a Compliant SMS Opt-In Process With Amazon Pinpoint

Post Syndicated from Tyler Holmes original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-build-a-compliant-sms-opt-in-process-with-amazon-pinpoint/

SMS messaging is a great way to stay in touch with your customers and send them timely, relevant messages. However, it’s always required to get their permission before you start sending them texts. This is known as opting in.

There are a few different ways to opt-in users to your SMS program. One common method is to have them sign up on your website or in your app. You can also collect opt-ins at in-person events or via phone call through your customer service team; though it’s not limited to just those options.

No matter which method you choose, it’s important to make sure that your opt-in process is clear, concise, and compliant with all applicable local laws and regulations in the countries that you are sending to. Here are some best practices:

  • Get explicit consent. Explicit consent is the intentional action taken by a end-user to request a specific message from your service.
  • Provide clear instructions. Tell users how to opt-in, what they are opting into, and how to opt-out of your program. Be sure to include your contact information at the opt-in location in case they have any questions or concerns.
  • Give users the option to choose what kind of messages they want to receive. For example, you might allow them to opt-in to OTP/2FA messages, shipping notifications, or both.
  • Respect users’ privacy. Never sell or share users’ phone numbers with third parties without their permission. 3rd party data sharing is generally considered a prohibited practice by mobile carriers and violates privacy regulations in many countries.
  • Make it easy to opt-out. Users should be able to opt-out of your program at any time by replying with a simple text message, such as “STOP.” See additional relevent documentation related to this: Opting out. Self-managed opt-outs.

The above will help you build a strong audience of engaged subscribers who want to hear from you and improve your chances in successfully registering for a dedicated number. By following these best practices, you can ensure that your SMS opt-in process is compliant and effective.

What carriers require for a compliant opt-in workflow and call-to-action

The primary purpose of the opt-in workflow is to demonstrate that the end-user explicitly consents to receive text messages and understands the nature of the program. Your application is being reviewed by a 3rd party reviewer and sometimes multiple 3rd party reviewers for a single registration, so make sure to provide clear and thorough information about how your end-users opt-in to your SMS service and any associated fees or charges. If the reviewer cannot determine how your opt-in process works or if it is not compliant then your application will be denied and returned. It is important to note that Amazon does not review or approve your use cases and that it’s a telecom industry standard in most countries for 3rd parties to review and approve your use case prior to sending.

Note: If you have a use case that is internal to your business, you are still required to demonstrate explicit opt-in consent from the recipients. There are no exceptions to having an opt-in workflow and explicit consent is always required.

If your opt-in process requires a login, is not yet public, is a verbal opt-in, or if it occurs on printed forms or fliers then make sure to thoroughly document how this process is completed by the end-user receiving messages — remember, these are 3rd party reviewers and if they’re unable to access where your end-users opt-in, they will require thorough information via other means like text or screenshots. Provide a screenshot of the Call to Action (CTA) in such cases. If the consent is being asked for and supplied verbally, as in a contact center situation, make sure to provide the verbal scripts to ensure the entire CTA is shown. Host any screenshots on a publicly accessible website (like S3, OneDrive, or Google Drive) and provide the URL when you submit (NOTE: toll-free number registration process supports attachments and do not require a public URL to be included). Regardless of the medium used to collect end-user information (e.g., webform, point of sale, fliers, or verbal opt-ins), the requirements are the same. In the case of online and printed materials, they would be shown as text to the end-users. In the case of verbal opt-ins (i.e., on the phone), the information below would be verbally read to the end-user.

Call-to-action/opt-in requirements

The following items are the minimum that must be presented to an end-user at the time of opt-in to ensure your SMS program is compliant:

  • Program (brand) name
  • Message frequency disclosure. (example: “Message frequency varies” or “One message per login”)
  • Customer care contact information (example: “Text HELP or call 1-800-111-2222 for support.”)
  • Opt-out information (example: “Text STOP to opt-out of future messages.”)
  • Include “Message and data rates may apply” disclosure.
  • Link to a publicly accessible Terms & Conditions page
  • Link to a publicly accessible Privacy Policy page

**Now lets break the above bullet points down into more detail:

Program, service, brand name

All SMS originator types that require registration must disclose the program name, product description, or both in service messages, on the call-to-action, and in the terms and conditions. The program name is the sponsor of the messaging program, often the brand name or company name associated with the sending use case. The product description describes the product advertised by the program.

Publicly accessible terms & conditions page

The terms should be live and publicly accessible. For verbal scripts, a URL must be read off to the end-user enrolling in the SMS program, or the comprehensive terms must be directly included in the script. You should provide a compliant screenshot, link, or mockup of the SMS Terms of Service in the registration submission.

Below is a copy of the boilerplate terms of service that cover minimum requirements from the carriers:

  1. {Program name}
  2. {Insert program description here; this is simply a brief description of the kinds of messages users can expect to receive when they opt-in.}
  3. You can cancel the SMS service at any time. Just text “STOP” to the short code. After you send the SMS message “STOP” to us, we will send you an SMS message to confirm that you have been unsubscribed. After this, you will no longer receive SMS messages from us. If you want to join again, just sign up as you did the first time and we will start sending SMS messages to you again.
  4. If you are experiencing issues with the messaging program you can reply with the keyword HELP for more assistance, or you can get help directly at {support email address or toll-free number}.
  5. Carriers are not liable for delayed or undelivered messages
  6. As always, message and data rates may apply for any messages sent to you from us and to us from you. You will receive {message frequency}. If you have any questions about your text plan or data plan, it is best to contact your wireless provider.
  7. If you have any questions regarding privacy, please read our privacy policy: {link to privacy policy}

Publicly accessible privacy policy page

Message Senders are responsible for protecting the privacy of Consumers’ information and must comply with applicable privacy law. Message Senders should maintain a privacy policy for all programs and make it accessible from the initial call-to-action. The privacy policy should be labeled clearly and all cases, terms and conditions and privacy policy disclosures must provide up-to-date, accurate information about program details and functionality. For verbal scripts, a URL must be read off to the end-user enrolling in the SMS program, or the comprehensive terms must be directly included in the script.

One of the key items carriers look for in a Privacy Policy is the sharing of end-user information with third-parties. If your privacy policy mentions data sharing or selling to non-affiliated third parties, there is a concern that customer data will be shared with third parties for marketing purposes.

Express consent is required for SMS; therefore, sharing data is prohibited. Privacy policies must specify that this data sharing excludes SMS opt-in data and consent. Privacy policies can be updated (or draft versions provided) where the practice of sharing personal data to third parties is expressly omitted from the number registration.

Example: “The above excludes text messaging originator opt-in data and consent; this information will not be shared with any third parties.”

Message frequency disclosure

The message frequency disclosure provides end-users an indication of how often they’ll receive messages from you. For example, a recurring messaging program might say “one message per week.” A one-time password or multi-factor authentication use case might say “message frequency varies” or “one message per login attempt”.

Customer care contact information

Customer care contact information must be clear and readily available to help Consumers understand program details as well as their status with the program. Customer care information should result in Consumers receiving help.

Numbers should always respond to customer care requests, regardless of whether the requestor is subscribed to the program. At a minimum, Message Senders must respond to messages containing the HELP keyword with the program name and further information about how to contact the Message Sender. SMS programs should promote customer care contact instructions at program opt-in and at regular intervals in content or service messages, at least once per month.

Example: “For more information, text ‘HELP’ or call 1-800-123-1234.”

Opt-Out Information

Opt-out mechanisms facilitate Consumer choice to terminate communications from text messaging programs. Message Senders should acknowledge and respect Consumers’ opt-out requests consistent with the following guidelines:

  • Message Senders should ensure that Consumers have the ability to opt-out at any time
  • Message Senders should support multiple mechanisms of opt-out, including: phone call, email, or text
  • Message Senders should acknowledge and honor all Consumer opt-out requests by sending one final opt-out confirmation message to notify the Consumer that they have opted-out successfully. No further messages should be sent following the confirmation message.

Message Senders should include opt-out information in the call-to-action, terms and conditions, and opt-in confirmation.

If a 2FA/OTP program requires end-users to opt-in and request an OTP from the same CTA, and it is compliant with all applicable regulations, then the sender does not need to explicitly opt-out that number if the user texts “STOP” to the business’s number. However, the sender must still respond with a compliant opt-out response.

See the following Amazon Pinpoint blog post on How to Manage SMS Opt-Outs with Amazon Pinpoint

“Message and data rates may apply” disclosure

All SMS programs must display or must be read out loud (if a verbal opt-in) the disclosure verbatim: “Message and data rates may apply”. By requiring the disclosure, US mobile carriers are helping to ensure that consumers are aware of the potential costs of sending and receiving text messages, and that they have consented to receive those messages before they are sent.

SMS Opt-Ins for Independent Software Vendors (ISVs)

Definitions

In this section, we’ll outline the terms we use, to help better explain each party, and the requirements.

ISV: ISVs are positioned between Amazon Pinpoint and the ISV’s end-business customers. While they may operate differently, and/or offer different services, their requirements for SMS program registrations are largely the same.

End Business: The End Business is how we refer to your ISV customers. This is generally the entity that creates the messaging content, distributes it through your platform, and interacts with their end-users (message recipients).

Note: In some rare cases, an ISV platform can be considered the end business if they control content via templates, and collect and manage opt-in in their entirety — meaning the ISV information directly will be used for the registration and will be branded as such in the text messages. If you are unsure, we recommend including the information for the entity (your customer) that is engaging with the opted-in handset with registration. ISVs who don’t include this information (if it is required) risk their verification request being rejected.

End-User: The message recipient is considered the end-user. The person with the handset where messages terminate. As an independent software vendor (ISV), you need to comply with all applicable laws and regulations when it comes to SMS opt-ins. This means that your end business(es) need to get explicit consent from their end-users before text messages start being sent and give end-users the option to opt-out of their program at any time. You also need to provide them with a registered and approved phone number to send their SMS messages to ensure that they are delivered reliably and not flagged as spam.

When does an ISV have to submit each end business?

SMS program registrations requires end-user business information, not ISV information. This means the ISV needs to provide a mechanism for their end businesses to provide their information to be submitted for registration. For ISVs or aggregators who provide messaging services to businesses, it’s expected that the information provided represents the entity (your customer) that is sending messages to the opted-in handset.

NOTE: Amazon uses this information in accordance with all applicable obligations, and only to verify the end-user is a legitimate business. Amazon will not contact the end-business user with the information provided.

Submissions that are missing information or are populated with ISV/aggregator information may be rejected. Exceptions may apply when the use case clearly showcases that the ISV manages opt-in mechanisms, is the sole message content creator, and the messages clearly come from the ISV, not their end businesses. For example, if the ISV owns a web application that requires their end-customers to enroll into OTP.

If you are unsure, we recommend including the information for the entity (your customer) that is engaging with the opted-in handset with registration. ISVs who don’t include this information (if it is required) risk their verification request being rejected.

In conclusion

Getting user consent through a compliant opt-in process is crucial for any SMS messaging program. Key elements include clearly disclosing the program details, providing easy opt-out methods, having accessible terms of service and privacy policies, and adhering to all applicable regulations. For ISVs enabling businesses to send SMS messages, it’s important they provide a way for each end business to submit their own information for registration and comply with the requirements. By following SMS best practices around opt-ins, businesses can build trust with subscribers and ensure deliverability of their text messaging campaigns.

Canadian Citizen Gets Phone Back from Police

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/canadian-citizen-gets-phone-back-from-police.html

After 175 million failed password guesses, a judge rules that the Canadian police must return a suspect’s phone.

[Judge] Carter said the investigation can continue without the phones, and he noted that Ottawa police have made a formal request to obtain more data from Google.

“This strikes me as a potentially more fruitful avenue of investigation than using brute force to enter the phones,” he said.

Code Written with AI Assistants Is Less Secure

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/code-written-with-ai-assistants-is-less-secure.html

Interesting research: “Do Users Write More Insecure Code with AI Assistants?“:

Abstract: We conduct the first large-scale user study examining how users interact with an AI Code assistant to solve a variety of security related tasks across different programming languages. Overall, we find that participants who had access to an AI assistant based on OpenAI’s codex-davinci-002 model wrote significantly less secure code than those without access. Additionally, participants with access to an AI assistant were more likely to believe they wrote secure code than those without access to the AI assistant. Furthermore, we find that participants who trusted the AI less and engaged more with the language and format of their prompts (e.g. re-phrasing, adjusting temperature) provided code with fewer security vulnerabilities. Finally, in order to better inform the design of future AI-based Code assistants, we provide an in-depth analysis of participants’ language and interaction behavior, as well as release our user interface as an instrument to conduct similar studies in the future.

At least, that’s true today, with today’s programmers using today’s AI assistants. We have no idea what will be true in a few months, let alone a few years.