Comment Patching: The bane of every IT professional's existence. It's a thankless, laborious job that no one wants to do, goes unappreciated when it interrupts work, and yet it's more critical than ever in this modern threat landscape.

So color this vulture surprised to learn that, in the decade since he left an IT career for wordier pastures, things haven't really improved much - either in terms of patching rates or how rough it is for the people doing it. 

"Patching is still notoriously difficult," Forrester principal analyst Andrew Hewitt told The Register. Hewitt, who specializes in IT ops, said that while organizations tend to strive for a 97 to 99 percent patch rate, they typically only manage to successfully fix between 75 and 85 percent of issues in their software. 

"That's not where you want to be from a compliance perspective," Hewitt added. 

Has anything improved lately? No, says Forrester senior analyst Erik Nost, who also spoke to The Register about patching from the security perspective. Nost observed that if anything has improved in the past decade, it's the business side - finally understanding how critical it is to keep systems updated. 

"There seems to be a bit more of an understanding of the necessity of it," Nost noted, with corporate leadership more aware of how poor patching can lead to expensive malware infections or other security snafus. Nost told us around 79 percent of security leaders say their business bosses view patching disruptions as a "necessary inconvenience" - which has been helped by better education from security and IT teams.

Patching challenges, however, haven't changed much. If anything, the situation has actually become worse.

Applying Windows updates isn't what a lot of people have signed up for

"People don't take jobs in IT operations to sit and update systems all day," Nost said. "They take those jobs to work on cool projects and cutting-edge technology. Going through and applying Windows updates isn't what a lot of people have signed up for." 

Coupled with an exploding ecosystem of third-party apps, endpoint management tools that aren't really designed to handle patch management, bandwidth issues (exacerbated by the pandemic shift to remote work), and architectural challenges, IT teams have "an overwhelming amount of work to do," Hewitt told us. 

He's not pulling those challenges out of nowhere, either. Endpoint management biz Adaptiva revealed in its 2023 state of patch handling report [PDF] that the average organization manages around 2,900 software applications, and 69 percent of IT teams believe it's impossible to get all of them patched on schedule. 

That is to say, good administrators know they need to patch security flaws and similar critical issues, and try to get these updates tested and deployed as fast as reasonably possible before someone takes advantage of them. It's just that life is never that easy, and there are all sorts of factors in the way.

Adaptiva's study, for instance, cited an increasing number of apps being installed on the average endpoint, the relatively low bandwidth for remote patch deployments, and the other issues that Hewitt pointed out. 

And while it's important to note Adaptiva has some skin in the game, given it sells software to automate patching, its figures don't seem off compared to others in the industry. 

Meanwhile, vulnerabilities proliferate

Where there are unpatched systems there are unfixed vulnerabilities - and there's no shortage of malware coded to exploit them. 

You'd hope that, if patching hasn't become easier, then at the very least the difficulties would be constant - but that's not the case, according to Nost. The tech industry has become overwhelmed by vulnerabilities to the point where it can't keep up, he said, pointing to NIST's massive backlog in vulnerability processing that's been ongoing since February. 

"Without a CVE, it's hard to tie a vulnerability to a patch that needs to be done," Nost stated, "and it's all downhill from there." 

He warned the situation has only been made worse by "security researchers looking to get a big hit" by finding a headline-grabbing critical vulnerability, which has made it harder to tell a major risk from a niche issue. If every other bug has a logo, a dedicated website raising the alarm, and an exclusive interview about it all with a media outlet, it's a pain for IT pros to figure out which ones to prioritize. CVSS severity scores only go so far.

What is to be done?

If we know patching is still a hassle, and hasn't improved much over the years, surely we can do something about it? 

While they have some different approaches to make patching more efficient, Hewitt and Nost agree that one of the biggest reasons it continues to be such a headache is a lack of ownership. Security teams and IT operations teams jostle to offload responsibility for the task. Layoffs and downsizing complicate matters further.

"You can open a Jira ticket and send it to [your IT ops team] or whoever, but who's the sysadmin or who's the business owner that is actually responsible for patching this? That gets even more complicated as you start to find application-level vulnerabilities," Nost explained. 

"Defining that [responsibility] is still very, very difficult. That's where I would say a lot of folks still spend a lot of time figuring out vulnerability management from a day-to-day perspective," he added.

Hewitt said much the same, and opined that the only way over that hurdle is to get IT ops and security teams on the same page - and it needs to be IT ops driving the process, as far as he's concerned.

Security professionals, rejoice.

"Patching capabilities are resident within IT operations tools," Hewitt noted. "Whether you're using Microsoft, Ivanti, Tanium, or whatever, patch management capabilities are in the IT ops tools." 

He said update responsibilities are further confused by too much siloing of data in enterprises, which leaves various teams working with different sets of data. 

"Oftentimes you have one platform for vulnerability management and another for patch management with no common dataset underneath," Hewitt told us, adding that several IT products are already moving in that direction. 

You have one platform for vulnerability management and another for patch management with no common dataset

"A lot of the endpoint management tools are building unified vulnerability and patch management capabilities, so I see a future trend toward trying to make that easier when it comes to user endpoints," Hewitt predicted.

Nost believes many of the patching problems people are experiencing can be solved through modern automation tools, but he added that many organizations have been hesitant to adopt them - in part because of the aforementioned issues surrounding ownership of patching responsibilities. 

Additionally, Nost said many enterprises still want hands-on control over patching, which Hewitt told us is perfectly reasonable. He believes having a human in the patching loop is essential, no matter how automated things get. 

"There are some things you can take a hands-off approach to, especially when they're smaller updates," Hewitt noted. "But I think this whole CrowdStrike outage is waking a lot of people up to how dangerous it can be to automate updates."

Both analysts we spoke to for this piece agreed that, while patching posture continues to be poor across the business world, for some understandable reasons, it doesn't necessarily need to be. The tools to make things easier are out there - be they improvements in automation or visibility enhancements in newer endpoint management products - but enterprises need to actually use them.

It's a very unsexy thing to work on, nobody wants to do it

"I think it mostly comes down to technical debt," Hewitt explained. 

"It's a very unsexy thing to work on, nobody wants to do it, and everyone feels like it should be automated - but nobody wants to take responsibility for doing it," Hewitt added. "The net effect is that nothing gets done and people stay in this state of technical debt where they're not able to prioritize it." 

"Hopefully this CrowdStrike thing will help," Hewitt concluded. 

Given what this ex-IT scribe has seen of the state of patching compared to a decade ago, he's not going to hold out hope that the CrowdStrike outage actually motivates people. 

It's up to IT teams to prove otherwise - and boy, I'd love to be proven wrong. ®

https://www.theregister.com//2024/07/25/patch_management_study/