Photo by Amelia Holowaty Krales / The Verge
Peter Membrey, chief architect of ExpressVPN, remembers vividly seeing the news of the Log4j vulnerability break online.
“As soon as I saw how you could exploit it, it was horrifying,” says Membrey. “Like one of those disaster movies where there’s a nuclear power plant, they find it’s going to melt down, but they can’t stop it. You know what’s coming, but there are very limited things you can do.”
Since the vulnerability was uncovered last week, the cybersecurity world has kicked into overdrive to identify vulnerable applications, detect potential attacks, and mitigate against exploits however possible. Nonetheless, serious hacks making use of the exploit are all but certain.
So far, researchers have observed attackers using the Log4j vulnerability to install ransomware on honeypot servers — machines that are made deliberately vulnerable for the purpose of tracking new threats. One cybersecurity firm reported that nearly half of corporate networks it was monitoring had seen attempts to exploit the vulnerability. The CEO of Cloudflare, a website and network security provider, announced early on that the threat was so bad the company would roll out firewall protection to all customers, including those who had not paid for it. But concrete news on exploitation in the wild remains scarce, likely because victims either don’t know or don’t yet want to acknowledge publicly that their systems have been breached.
What is known for sure is that the scope of the vulnerability is huge. A list of affected software compiled by the Cybersecurity and Infrastructure Security Agency (CISA) — and restricted to only enterprise software platforms — runs to more than 500 items long at time of press. A list of all affected applications would undoubtedly run to many thousands more.
Some names on the list will be familiar to the public (Amazon, IBM, Microsoft), but some of the most alarming issues have come with software that stays behind the scenes. Manufacturers like Broadcom, Red Hat, and VMware make software that enterprise clients build businesses on top of, effectively distributing the vulnerability at a core infrastructural level of many companies. This makes the process of catching and eliminating vulnerabilities all the more difficult, even after a patch for the affected library has been released.
Even by the standards of high-profile vulnerabilities, Log4Shell is hitting an unusually large chunk of the internet. It’s a reflection of the fact that the Java programming language is used widely in enterprise software, and for Java software, the Log4j library is exceedingly common.
“I ran queries in our database to see every customer who was using Log4j in any of their applications,” says Jeremy Katz, co-founder of Tidelift, a company that helps other organizations manage open-source software dependencies. “And the answer was: every single one of them that has any applications written in Java.”
The discovery of an easily exploitable bug found in a mostly enterprise-focused language is part of what analysts have called a “nearly perfect storm” around the Log4j vulnerability. Any one company could be using numerous programs containing the vulnerable library — in some cases, with multiple versions inside one application.
“Java has been around for so many years, and it’s so heavily used within companies, particularly large ones,” says Cloudflare CTO John Graham-Cumming. “This is a big moment for people who manage software within companies, and they will be running through updates and mitigations as fast as they can.”
Given the circumstances, “as fast as they can” is a very subjective term. Software updates for organizations like banks, hospitals, or government agencies are generally conducted on the scale of weeks and months, not days; typically, updates require numerous levels of development, authorization, and testing before making their way into a live application.
In the meantime, mitigations that can be pushed out quickly provide a crucial intermediary step, buying valuable time while businesses large and small scramble to identify vulnerabilities and deploy updates. That’s where fixes at the network layer have a key role to play: since malware programs communicate with their operators over the internet, measures that restrict incoming and outgoing web traffic can provide a stopgap to limit the effects of the exploit.
Cloudflare was one organization that moved quickly, Graham-Cumming explained, adding new rules for its firewall that blocked HTTP requests containing strings characteristic of the Log4j attack code. ExpressVPN also modified its product to protect against Log4Shell, updating VPN rules to automatically block all outgoing traffic on ports used by LDAP — a protocol that the exploit uses to fetch resources from remote URLs and download them onto a vulnerable machine.
“If a customer gets infected, we’ve already seen scanners as a malicious payload, so they might start scanning the internet and infect other people,” says Membrey. “We wanted to put a cap on that, not just for our customers’ sake but for everyone else’s sake — a bit like with Covid and vaccines.”
These changes typically happen faster because they take place on servers belonging to the firewall or VPN companies and require little (if any) action from the end user. In other words, an out-of-date software application could still receive a decent level of protection from an updated VPN — though it’s no substitute for proper patching.
Unfortunately, given the seriousness of the vulnerability, some systems will be compromised, even with quick fixes deployed. And it may be a long time — years even — before effects are fully felt.
“Sophisticated attackers will exploit the vulnerability, establish a persistence mechanism, and then go dark,” Daniel Clayton, vice president of global cybersecurity services at Bitdefender, says. “In two years’ time, we will hear about big breaches and then subsequently learn that they were breached two years ago.”
The bug in Log4j once more highlights the necessity and challenge of adequately funding open source projects. (A huge amount of tech infrastructure might as well depend on “a project some random person in Nebraska has been tirelessly maintaining since 2003,” as a perennially relevant XKCD comic explains.) Bloomberg reported earlier this week that many of the developers involved in the race to develop a patch for the Log4j library were unpaid volunteers, despite the global use of the software in enterprise applications.
One of the last vulnerabilities to rock the internet, Heartbleed, was similarly caused by a bug in a widely used open-source library, OpenSSL. Following that bug, tech companies like Google, Microsoft, and Facebook committed to putting more money into open source projects that were critical for internet infrastructure. But in the wake of the Log4j fallout, it’s clear that managing dependencies remains a serious security problem — and one we’re not close to solving.
“When you look at most of the big hacks that have happened over the years, it’s not normally something really sophisticated that undoes big companies,” Clayton says. “It’s something that hasn’t been patched.”