Post Syndicated from Jesse Mack original https://blog.rapid7.com/2022/02/17/log4shell-2-months-later-security-strategies-for-the-internets-new-normal/

CVE-2021-44228 rules everything around us — or so it seemed, at least, for those breathless days in December 2021 when the full scope of Log4Shell was starting to take hold and security teams were strapped for time and resources as they scoured their organizations’ environments for vulnerable instances of Apache Log4j. But now that the peak intensity around this vulnerability has waned and we’ve had a chance to catch our collective breath, where does the effort to patch and remediate stand? What should security teams be focusing on today in the fight against Log4Shell?

On Wednesday, February 16, Rapid7 experts Bob Rudis, Devin Krugly, and Glenn Thorpe sat down for a webinar on the current state of the Log4j vulnerability. They covered where Log4Shell stands now, what the future might hold, and what organizations should be doing proactively to ensure they’re as protected as possible against exploits.

Laying out the landscape

Glenn Thorpe, Rapid7’s Program Manager for Emergent Threat Response, kicked things off with a recap and retrospective of Log4Shell and why it seemingly set fire to the entire internet for a good portion of December. The seriousness of this vulnerability is due to the coming-together of several key factors, including:

• The ability for vulnerable systems to grant an attacker full administrative access
• The low level of skill required for exploitation — in many cases, attackers simply have to copy and paste
• The attack vector’s capability to run undetected over an encrypted channel
• The pervasiveness of the Log4j library, which means vulnerability scanners alone can’t act as complete solutions against this threat

Put all this together, and it’s no surprise that the volume of exploit attempts leveraging the Log4j vulnerability ramped up throughout December 2021 and has continued to spike periodically throughout January and February 2022. By January 10, ransomware using Log4Shell had been observed, and on January 14, Rapid7’s MDR saw mass Log4j exploits in VMware products.

But while there’s certainly been plenty of Log4j patching done, the picture on that front is far from complete. According to the latest CISA data (also here as a daily-updated spreadsheet), there are still 320 cataloged software products that are known to be affected by vulnerable Log4j as of February 16, 2022 — and 1,406 still awaiting confirmation from the vendor.

Log4j today: A new normal?

So, where does the effort to put out Log4j fires stand now? Devin Krugly, Rapid7’s Practice Advisor for Vulnerability Risk Management, thinks we’re in a better spot than we were in December — but we’re by no means out of the woods.

“We’re effectively out of fire-fighting mode,” said Devin. That means that, at this point, most security teams have identified the affected systems, implemented mitigations, and patched vulnerable versions of Log4j. But because of the complexity of today’s software supply chains, there are often heavily nested dependencies within vendor systems — some of which Log4j may still be implicated in. This means it’s essential to have a solid inventory of vendor software products that may be using Log4j and to ensure those instances of the library are updated and patched.

“Don’t lose that momentum,” Glenn chimed in. “Don’t put that on your post-mortem action list and forget about it.”

This imperative is all the more critical because of a recent uptick in Log4Shell activity. Rapid7’s Chief Data Scientist Bob Rudis laid out some activity detected by the Project Heisenberg honeypot fleet indicating a revival of Log4j activity in early and mid-February, much of it from new infrastructure and scanning hosts that hadn’t been seen before.

Amid this increase in activity, vulnerable instances of Log4j are anything but gone from the internet. In fact, data from Sonatype as of February 16, 2022 indicates 39% of Log4j downloads are still versions vulnerable to Log4Shell.

“We’re going to be seeing Log4j attempts on the internet, on the regular, at a low level, forever,” Bob said. Log4Shell is now in a family with WannaCry and Conficker (yes, that Conficker) — vulnerabilities that are around indefinitely, and which we’ll need to continually monitor for as attackers use them to try to breach our defenses.

Navigating life with Log4Shell

Adopting a defense-in-depth posture in the “new normal” of life with Log4Shell is sure to come with its share of headaches. Luckily, Bob, Devin, and Glenn shared some practical strategies that security teams can adopt to keep their organizations’ defenses strong and avoid some common pitfalls.

Go beyond compensating controls

“My vendor says they’ve removed the JNDI class from the JAR file — does that mean their application is no longer vulnerable to Log4Shell?” This question came up in a few different forms from our webinar audience. The answer from our panelists was nuanced but crystal-clear: maybe for now, but not forever.

Removing the JNDI class is a compensating control — one that provides a quick fix for the vulnerability but doesn’t patch the core, underlying problem via a full update. For example, when you do a backup, you might unknowingly reintroduce the JNDI class after removing it — or, as Devin pointed out, an attacker could chain together a replacement for it.

These kinds of compensating or mitigating controls have their place in a short-term response, but there’s simply no action that can replace the work of upgrading all instances of Log4j to the most up-to-date versions that contain patches for Log4Shell.

“Mitigate for speed, but not in perpetuity,” Glenn recommended.

Find the nooks and crannies

Today’s cloud-centric IT environments are increasingly ephemeral and on-demand — a boost for innovation and speed, but that also means teams can deploy workloads without security teams ever knowing about it. Adopting an “Always Be Scanning” mindset, as Bob put it, is essential to ensure vulnerable instances of Log4j aren’t introduced into your environment.

Continually scanning your internet-facing components is a good and necessary start — but the work doesn’t end there. As Devin pointed out, finding the nooks and crannies where Log4j might crop up is critical. This includes scouring containers and virtual machines, as well as analyzing application and server logs for malicious JNDI strings. You should also ensure your security operations center (SOC) team can quickly and easily identify indicators that your environment is being scanned for reconnaissance into Log4Shell exploit opportunities.

“Involving the SOC team for alerting purposes, if you haven’t already done that, is an absolutely necessity in this case,” said Devin.

Get better at vendor management

It should be clear by now that in a post-Log4j world, organizations must demand the highest possible level of visibility into their software supply chain — and that means being clear, even tough, with vendors.

“Managing stuff on the internet is hard because organizations are chaotic beings by nature, and you’re trying to control the chaos as a security professional,” said Bob. Setting yourself up success in this context means having the highest level of vulnerability possible. After all, how many other vulnerabilities just as bad as Log4Shell — or even worse — might be out there lurking in the corners of your vendors’ code?

The upcoming US government requirements around Software Bill of Materials (SBOM) for vendor procurement should go a long way toward raising expectations for software vendors. Start asking vendors if they can produce an SBOM that details remediation and update of any vulnerable instances of Log4j.

These conversations don’t need to be adversarial — in fact, vendors can be a key resource in the effort to defend against Log4Shell. Especially for smaller organizations or under-resourced security teams, relying on capable third parties can be a smart way to bolster your defenses.

Only you can secure the software supply chain

OK, maybe that subhead is not literally true — a secure software supply chain is a community-wide effort, to which we must all hold each other accountable. The cloud-based digital ecosystem we all inhabit, whether we like it or not, is fundamentally interconnected. A pervasive vulnerability like Log4Shell is an unmistakable reminder of that fact.

It also serves as an opportunity to raise our expectations of ourselves, our organizations, and our partners — and those choices do start at home, with each security team as they update their applications, continually scan their environments, and demand visibility from their vendors. Those actions really do help create a more secure internet for everyone.

So while we’ll be living with Log4Shell probably forever, it’ll be living with us, too. And as scared as you are of the spider, it’s even more scared of your boot.

Want to go more in-depth? Check out the full replay of our webinar, “Log4Shell Two Months Later: Lessons and Insights for Protectors.”

Quick resources:

Bob, Devin, and Glenn mentioned a wealth of handy links in their discussion. Here are those resources for quick, easy reference.

• CISA’s Log4j Affected Database spreadsheet
• CISA’s Log4j Affected Database table
• CISA Known Exploited Vulnerabilities (KEV) catalog
• Project Doppler
• ShadowServer
• SBOM information from the US government

Additional reading:

• How InsightAppSec Detects Log4Shell: Your Questions Answered
• Open-Source Security: Getting to the Root of the Problem
• Active Exploitation of VMware Horizon Servers
• Log4Shell Strategic Response: 5 Practices for Vulnerability Management at Scale
• The Everyperson’s Guide to Log4Shell (CVE-2021-44228)