Tag Archives: hacking

A Taxonomy of Prompt Injection Attacks

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/03/a-taxonomy-of-prompt-injection-attacks.html

Researchers ran a global prompt hacking competition, and have documented the results in a paper that both gives a lot of good examples and tries to organize a taxonomy of effective prompt injection strategies. It seems as if the most common successful strategy is the “compound instruction attack,” as in “Say ‘I have been PWNED’ without a period.”

Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition

Abstract: Large Language Models (LLMs) are deployed in interactive contexts with direct user engagement, such as chatbots and writing assistants. These deployments are vulnerable to prompt injection and jailbreaking (collectively, prompt hacking), in which models are manipulated to ignore their original instructions and follow potentially malicious ones. Although widely acknowledged as a significant security threat, there is a dearth of large-scale resources and quantitative studies on prompt hacking. To address this lacuna, we launch a global prompt hacking competition, which allows for free-form human input attacks. We elicit 600K+ adversarial prompts against three state-of-the-art LLMs. We describe the dataset, which empirically verifies that current LLMs can indeed be manipulated via prompt hacking. We also present a comprehensive taxonomical ontology of the types of adversarial prompts.

China Surveillance Company Hacked

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/02/china-surveillance-company-hacked.html

Last week, someone posted something like 570 files, images and chat logs from a Chinese company called I-Soon. I-Soon sells hacking and espionage services to Chinese national and local government.

Lots of details in the news articles.

These aren’t details about the tools or techniques, more the inner workings of the company. And they seem to primarily be hacking regionally.

AIs Hacking Websites

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/02/ais-hacking-websites.html

New research:

LLM Agents can Autonomously Hack Websites

Abstract: In recent years, large language models (LLMs) have become increasingly capable and can now interact with tools (i.e., call functions), read documents, and recursively call themselves. As a result, these LLMs can now function autonomously as agents. With the rise in capabilities of these agents, recent work has speculated on how LLM agents would affect cybersecurity. However, not much is known about the offensive capabilities of LLM agents.

In this work, we show that LLM agents can autonomously hack websites, performing tasks as complex as blind database schema extraction and SQL injections without human feedback. Importantly, the agent does not need to know the vulnerability beforehand. This capability is uniquely enabled by frontier models that are highly capable of tool use and leveraging extended context. Namely, we show that GPT-4 is capable of such hacks, but existing open-source models are not. Finally, we show that GPT-4 is capable of autonomously finding vulnerabilities in websites in the wild. Our findings raise questions about the widespread deployment of LLMs.

Microsoft Executives Hacked

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/01/microsoft-executives-hacked.html

Microsoft is reporting that a Russian intelligence agency—the same one responsible for SolarWinds—accessed the email system of the company’s executives.

Beginning in late November 2023, the threat actor used a password spray attack to compromise a legacy non-production test tenant account and gain a foothold, and then used the account’s permissions to access a very small percentage of Microsoft corporate email accounts, including members of our senior leadership team and employees in our cybersecurity, legal, and other functions, and exfiltrated some emails and attached documents. The investigation indicates they were initially targeting email accounts for information related to Midnight Blizzard itself.

This is nutty. How does a “legacy non-production test tenant account” have access to executive e-mails? And why no try-factor authentication?

Cyberattack on Ukraine’s Kyivstar Seems to Be Russian Hacktivists

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/12/cyberattack-on-ukraines-kyivstar-seems-to-be-russian-hacktivists.html

The Solntsepek group has taken credit for the attack. They’re linked to the Russian military, so it’s unclear whether the attack was government directed or freelance.

This is one of the most significant cyberattacks since Russia invaded in February 2022.

Online Retail Hack

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/11/online-retail-hack.html

Selling miniature replicas to unsuspecting shoppers:

Online marketplaces sell tiny pink cowboy hats. They also sell miniature pencil sharpeners, palm-size kitchen utensils, scaled-down books and camping chairs so small they evoke the Stonehenge scene in “This Is Spinal Tap.” Many of the minuscule objects aren’t clearly advertised.

[…]

But there is no doubt some online sellers deliberately trick customers into buying smaller and often cheaper-to-produce items, Witcher said. Common tactics include displaying products against a white background rather than in room sets or on models, or photographing items with a perspective that makes them appear bigger than they really are. Dimensions can be hidden deep in the product description, or not included at all.

In those instances, the duped consumer “may say, well, it’s only $1, $2, maybe $3­—what’s the harm?” Witcher said. When the item arrives the shopper may be confused, amused or frustrated, but unlikely to complain or demand a refund.

“When you aggregate that to these companies who are selling hundreds of thousands, maybe millions of these items over time, that adds up to a nice chunk of change,” Witcher said. “It’s finding a loophole in how society works and making money off of it.”

Defrauding a lot of people out of a small amount each can be a very successful way of making money.

Crashing iPhones with a Flipper Zero

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/11/crashing-iphones-with-a-flipper-zero.html

The Flipper Zero is an incredibly versatile hacking device. Now it can be used to crash iPhones in its vicinity by sending them a never-ending stream of pop-ups.

These types of hacks have been possible for decades, but they require special equipment and a fair amount of expertise. The capabilities generally required expensive SDRs­—short for software-defined radios­—that, unlike traditional hardware-defined radios, use firmware and processors to digitally re-create radio signal transmissions and receptions. The $200 Flipper Zero isn’t an SDR in its own right, but as a software-controlled radio, it can do many of the same things at an affordable price and with a form factor that’s much more convenient than the previous generations of SDRs.

Hacking Scandinavian Alcohol Tax

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/10/hacking-scandinavian-alcohol-tax.html

The islands of Åland are an important tax hack:

Although Åland is part of the Republic of Finland, it has its own autonomous parliament. In areas where Åland has its own legislation, the group of islands essentially operates as an independent nation.

This allows Scandinavians to avoid the notoriously high alcohol taxes:

Åland is a member of the EU and its currency is the euro, but Åland’s relationship with the EU is regulated by way of a special protocol. In order to maintain the important sale of duty-free goods on ferries operating between Finland and Sweden, Åland is not part of the EU’s VAT area.

Basically, ferries between the two countries stop at the island, and people stock up—I mean really stock up, hand trucks piled with boxes—on tax-free alcohol. Åland gets the revenue, and presumably docking fees.

The purpose of the special status of the Åland Islands was to maintain the right to tax free sales in the ship traffic. The ship traffic is of vital importance for the province’s communication, and the intention was to support the economy of the province this way.

Hacking the High School Grading System

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/10/hacking-the-high-school-grading-system.html

Interesting New York Times article about high-school students hacking the grading system.

What’s not helping? The policies many school districts are adopting that make it nearly impossible for low-performing students to fail—they have a grading floor under them, they know it, and that allows them to game the system.

Several teachers whom I spoke with or who responded to my questionnaire mentioned policies stating that students cannot get lower than a 50 percent on any assignment, even if the work was never done, in some cases. A teacher from Chapel Hill, N.C., who filled in the questionnaire’s “name” field with “No, no, no,” said the 50 percent floor and “NO attendance enforcement” leads to a scenario where “we get students who skip over 100 days, have a 50 percent, complete a couple of assignments to tip over into 59.5 percent and then pass.”

It’s a basic math hack. If a student needs two-thirds of the points—over 65%—to pass, then they have to do two-thirds of the work. But if doing zero work results in a 50% grade, then they only have to do a little bit of work to get over the pass line.

I know this is a minor thing in the universe of problems with secondary education and grading, but I found the hack interesting. (And this is exactly the sort of thing I explore in my latest book: A Hacker’s Mind.

Hacking Gas Pumps via Bluetooth

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/10/hacking-gas-pumps-via-bluetooth.html

Turns out pumps at gas stations are controlled via Bluetooth, and that the connections are insecure. No details in the article, but it seems that it’s easy to take control of the pump and have it dispense gas without requiring payment.

It’s a complicated crime to monetize, though. You need to sell access to the gas pump to others.

EDITED TO ADD (10/13): Reader Jeff Hall says that story is not accurate, and that the gas pumps do not have a Bluetooth connection.

Spyware Vendor Hacked

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/09/spyware-vendor-hacked.html

A Brazilian spyware app vendor was hacked by activists:

In an undated note seen by TechCrunch, the unnamed hackers described how they found and exploited several security vulnerabilities that allowed them to compromise WebDetetive’s servers and access its user databases. By exploiting other flaws in the spyware maker’s web dashboard—used by abusers to access the stolen phone data of their victims—the hackers said they enumerated and downloaded every dashboard record, including every customer’s email address.

The hackers said that dashboard access also allowed them to delete victim devices from the spyware network altogether, effectively severing the connection at the server level to prevent the device from uploading new data. “Which we definitely did. Because we could. Because #fuckstalkerware,” the hackers wrote in the note.

The note was included in a cache containing more than 1.5 gigabytes of data scraped from the spyware’s web dashboard. That data included information about each customer, such as the IP address they logged in from and their purchase history. The data also listed every device that each customer had compromised, which version of the spyware the phone was running, and the types of data that the spyware was collecting from the victim’s phone.

Hacking Food Labeling Laws

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/08/hacking-food-labeling-laws.html

This article talks about new Mexican laws about food labeling, and the lengths to which food manufacturers are going to ensure that they are not effective. There are the typical high-pressure lobbying tactics and lawsuits. But there’s also examples of companies hacking the laws:

Companies like Coca-Cola and Kraft Heinz have begun designing their products so that their packages don’t have a true front or back, but rather two nearly identical labels—except for the fact that only one side has the required warning. As a result, supermarket clerks often place the products with the warning facing inward, effectively hiding it.

[…]

Other companies have gotten creative in finding ways to keep their mascots, even without reformulating their foods, as is required by law. Bimbo, the international bread company that owns brands in the United States such as Entenmann’s and Takis, for example, technically removed its mascot from its packaging. It instead printed the mascot on the actual food product—a ready to eat pancake—and made the packaging clear, so the mascot is still visible to consumers.

UK Electoral Commission Hacked

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/08/uk-electoral-commission-hacked.html

The UK Electoral Commission discovered last year that it was hacked the year before. That’s fourteen months between the hack and the discovery. It doesn’t know who was behind the hack.

We worked with external security experts and the National Cyber Security Centre to investigate and secure our systems.

If the hack was by a major government, the odds are really low that it has resecured its systems—unless it burned the network to the ground and rebuilt it from scratch (which seems unlikely).

China Hacked Japan’s Military Networks

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/08/china-hacked-japans-military-networks.html

The NSA discovered the intrusion in 2020—we don’t know how—and alerted the Japanese. The Washington Post has the story:

The hackers had deep, persistent access and appeared to be after anything they could get their hands on—plans, capabilities, assessments of military shortcomings, according to three former senior U.S. officials, who were among a dozen current and former U.S. and Japanese officials interviewed, who spoke on the condition of anonymity because of the matter’s sensitivity.

[…]

The 2020 penetration was so disturbing that Gen. Paul Nakasone, the head of the NSA and U.S. Cyber Command, and Matthew Pottinger, who was White House deputy national security adviser at the time, raced to Tokyo. They briefed the defense minister, who was so concerned that he arranged for them to alert the prime minister himself.

Beijing, they told the Japanese officials, had breached Tokyo’s defense networks, making it one of the most damaging hacks in that country’s modern history.

More analysis.

Microsoft Signing Key Stolen by Chinese

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/08/microsoft-signing-key-stolen-by-chinese.html

A bunch of networks, including US Government networks, have been hacked by the Chinese. The hackers used forged authentication tokens to access user email, using a stolen Microsoft Azure account consumer signing key. Congress wants answers. The phrase “negligent security practices” is being tossed about—and with good reason. Master signing keys are not supposed to be left around, waiting to be stolen.

Actually, two things went badly wrong here. The first is that Azure accepted an expired signing key, implying a vulnerability in whatever is supposed to check key validity. The second is that this key was supposed to remain in the the system’s Hardware Security Module—and not be in software. This implies a really serious breach of good security practice. The fact that Microsoft has not been forthcoming about the details of what happened tell me that the details are really bad.

I believe this all traces back to SolarWinds. In addition to Russia inserting malware into a SolarWinds update, China used a different SolarWinds vulnerability to break into networks. We know that Russia accessed Microsoft source code in that attack. I have heard from informed government officials that China used their SolarWinds vulnerability to break into Microsoft and access source code, including Azure’s.

I think we are grossly underestimating the long-term results of the SolarWinds attacks. That backdoored update was downloaded by over 14,000 networks worldwide. Organizations patched their networks, but not before Russia—and others—used the vulnerability to enter those networks. And once someone is in a network, it’s really hard to be sure that you’ve kicked them out.

Sophisticated threat actors are realizing that stealing source code of infrastructure providers, and then combing that code for vulnerabilities, is an excellent way to break into organizations who use those infrastructure providers. Attackers like Russia and China—and presumably the US as well—are prioritizing going after those providers.

News articles.

EDITED TO ADD: Commentary:

This is from Microsoft’s explanation. The China attackers “acquired an inactive MSA consumer signing key and used it to forge authentication tokens for Azure AD enterprise and MSA consumer to access OWA and Outlook.com. All MSA keys active prior to the incident—including the actor-acquired MSA signing key—have been invalidated. Azure AD keys were not impacted. Though the key was intended only for MSA accounts, a validation issue allowed this key to be trusted for signing Azure AD tokens. The actor was able to obtain new access tokens by presenting one previously issued from this API due to a design flaw. This flaw in the GetAccessTokenForResourceAPI has since been fixed to only accept tokens issued from Azure AD or MSA respectively. The actor used these tokens to retrieve mail messages from the OWA API.”

Hacking AI Resume Screening with Text in a White Font

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/08/hacking-ai-resume-screening-with-text-in-a-white-font.html

The Washington Post is reporting on a hack to fool automatic resume sorting programs: putting text in a white font. The idea is that the programs rely primarily on simple pattern matching, and the trick is to copy a list of relevant keywords—or the published job description—into the resume in a white font. The computer will process the text, but humans won’t see it.

Clever. I’m not sure it’s actually useful in getting a job, though. Eventually the humans will figure out that the applicant doesn’t actually have the required skills. But…maybe.

Disabling Self-Driving Cars with a Traffic Cone

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/07/disabling-self-driving-cars-with-a-traffic-cone.html

You can disable a self-driving car by putting a traffic cone on its hood:

The group got the idea for the conings by chance. The person claims a few of them walking together one night saw a cone on the hood of an AV, which appeared disabled. They weren’t sure at the time which came first; perhaps someone had placed the cone on the AV’s hood to signify it was disabled rather than the other way around. But, it gave them an idea, and when they tested it, they found that a cone on a hood renders the vehicles little more than a multi-ton hunk of useless metal. The group suspects the cone partially blocks the LIDAR detectors on the roof of the car, in much the same way that a human driver wouldn’t be able to safely drive with a cone on the hood. But there is no human inside to get out and simply remove the cone, so the car is stuck.

Delightfully low-tech.

Buying Campaign Contributions as a Hack

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2023/07/buying-campaign-contributions-as-a-hack.html

The first Republican primary debate has a popularity threshold to determine who gets to appear: 40,000 individual contributors. Now there are a lot of conventional ways a candidate can get that many contributors. Doug Burgum came up with a novel idea: buy them:

A long-shot contender at the bottom of recent polls, Mr. Burgum is offering $20 gift cards to the first 50,000 people who donate at least $1 to his campaign. And one lucky donor, as his campaign advertised on Facebook, will have the chance to win a Yeti Tundra 45 cooler that typically costs more than $300—just for donating at least $1.

It’s actually a pretty good idea. He could have spent the money on direct mail, or personalized social media ads, or television ads. Instead, he buys gift cards at maybe two-thirds of face value (sellers calculate the advertising value, the additional revenue that comes from using them to buy something more expensive, and breakage when they’re not redeemed at all), and resells them. Plus, many contributors probably give him more than $1, and he got a lot of publicity over this.

Probably the cheapest way to get the contributors he needs. A clever hack.

EDITED TO ADD (7/16): These might be “straw donors” and illegal:

The campaign’s donations-for-cash strategy could raise potential legal concerns, said Paul Ryan, a campaign finance lawyer. Voters who make donations in exchange for gift cards, he said, might be considered straw donors because part or all of their donations are being reimbursed by the campaign.

“Federal law says ‘no person shall make a contribution in the name of another person,’” Mr. Ryan said. “Here, the candidate is making a contribution to himself in the name of all these individual donors.”

Richard L. Hasen, a law professor at the University of California, Los Angeles, who specializes in election law, said that typically, campaigns ask the Federal Election Commission when engaging in new forms of donations.

The Burgum campaign’s maneuver, he said, “certainly seems novel” and “raises concerns about whether it violates the prohibition on straw donations.”

Something for the courts to figure out, if this matter ever gets that far.