From chaos to cadence: Celebrating two decades of Microsoft's Patch Tuesday

Feature Twenty years ago this month, Microsoft did something pretty revolutionary at the time when it formalized the Windows software release schedule.

So instead of shipping updates whenever they were ready – Redmond says this typically happened on Wednesdays, while most customers recall it being late Friday afternoons – Microsoft began pushing software fixes on the second Tuesday of each month, beginning in October 2003.

Microsoft is patching stuff in Linux now, which was completely unheard of in 2008

And thus, Patch Tuesday sprung into existence.

The before times were "very chaotic for system administrators, especially when it came to planning resources," remembers Dustin Childs, head of threat awareness at Trend Micro's Zero Day Initiative. He previously spent almost seven years in security at Microsoft beginning in January 2008.

Childs described the early years of Patch Tuesday at Microsoft being kind of a party, complete with catered breakfast and music.

"We hit the release button and everything went live and we would blast music in the hallway of our office," Childs tells The Register. "It was a big thing for us to know that we were fixing things, making the world a little bit better. One patch at a time."

Microsoft employees weren't the only ones who welcomed the shift.

Those were the early days of the internet, and "nobody was really disciplined about patching," says Tim Crothers, Mandiant Chief Information Security Officer at Google Cloud. "It was really chaotic."

The before times

Crothers started his career in 1984, first on the infrastructure side of things and then security for the last three decades, so he's seen Patch Tuesday from both sides. He's been the IT guy responsible for testing and deploying the patches, and he's also been the security researcher working to reverse engineer the fixes as soon as they drop.

Crothers remembers the second-Tuesday push being largely customer driven.

"Certainly a lot of large financial institutions and I imagine a lot of other organizations were part of really bringing pressure to bear to Microsoft to release it as an instance, a single time so we can plan for it, take a more measured approach and reduce a lot of the chaos that was prior to Patch Tuesday being a thing," he tells The Register.

Or, as Aanchal Gupta, a Redmond customer at the time who is now Microsoft deputy CISO and corporate VP, told us: "We got some feedback. And based on that feedback we said, 'We need to streamline this. We need to bring order to this if we want customers to participate with us in securing the entire ecosystem.'"

If Microsoft issues patches, but customers don't apply the fixes, "it becomes that much harder to secure the services," Gupta explains.

"So that's when Patch Tuesday was born," she says. After Microsoft moved to this monthly cadence, "patch consumption went up significantly."

Predictability for IT admins…

By all accounts, the move was welcomed by IT administrators because it gave them predictability.

"The patch management process back then was completely non-existent as well, so that made it that much harder," Childs tells The Register. "It was a very difficult time for system administrators prior to Patch Tuesday to plan, to test, and then add resources to roll these patches out."

Plus, in the early days of Patch Tuesday Microsoft provided advance notification to customers. So prior to starting their weekends, admins knew that, the following Tuesday, patches fixing a dozen or so CVEs would be released.

And no, that's not a typo. The volume of patches issued each month has exploded over the past two decades. The "unwritten rule" used to be no more than 12 security bulletins per month, based on what both Microsoft and its customers could handle, Childs says.

These days, with the move to cloud and the ever-expanding attack surface, 100-plus security fixes per month is common. "The number of things that Microsoft is patching – Microsoft is patching stuff in Linux now, which was completely unheard of in 2008," Childs says.

"Patch management these days is kind of a continuous process," Childs continues, adding that programs like Exchange and SharePoint can be tricky to patch.

"We've joked that the fastest way to get fired as a sysadmin is to break email, and the fastest way to break email is to patch Exchange," he says.

It's difficult in today's IT environments to first identify everything that needs to be updated, and then companies still have to test most of the patches before rolling them out across the organization.

"Then they have to deploy the patches at a time that's not inconvenient – but there's never a time that's not inconvenient, according to users," Childs says.

Additionally, the monthly security bulletins aren't just coming from Microsoft anymore. Other vendors including Oracle and Adobe jumped on the Patch Tuesday bandwagon in 2003. Soon SAP, along with nearly every other software maker, followed suit.

Even hardware vendors got on board, and it's not uncommon for them to also release patches on the same day as Microsoft. "Now, we partner very closely with AMD, Intel," Gupta says. "Let's align on these vulnerability patches to make sure we are doing it together."

• It's 2023 and Microsoft WordPad can be exploited to hijack vulnerable systems
• Microsoft says VBScript will be ripped from Windows in future release
• Ransomware attacks register record speeds thanks to success of infosec industry
• HTTP/2 'Rapid Reset' zero-day exploited in biggest DDoS deluge seen yet

The cynical view here would be that perhaps vendors are disclosing bugs and releasing fixes for those on the same day in the hopes that the really bad ones might get buried under the avalanche of CVEs coming out on Patch Tuesday.

While there is probably some truth to that, kind of like burying bad news late on a Friday, overall the benefits of a monthly bug disclosure timetable outweighs the bad – like the massive number of patches – according to the people interviewed for this story.

Even if customers don't explicitly ask for this patch cadence, "they understand it," Childs says. "It's more stability," he explains. "More things they can predict. Even if [the amount of Patch Tuesday updates] is completely overwhelming."

Despite the flood of second-Tuesday security bulletins, during the last 20 years, the quality of the patches has improved. So have the software tools and automated systems used to distribute and apply the patches, which means less downtime for systems – and disruption for users. "This means acceptance has grown over the years," Crothers says.

And for attackers

"Of course, that doesn't mean it's all roses," he adds. "The downside of the Patch Tuesday approach is that the threat actors are aware of the patch. We're in a race condition between the patch being deployed to protect our organizations and the attackers exploiting them."

There wouldn't be Exploit Wednesday without Patch Tuesday, and over the years the defenders aren't the only ones eagerly awaiting the latest monthly batch of CVE disclosures. Once the security updates are released, both the legitimate researchers and the criminals get to work trying to reverse engineer the fixes and, on the miscreant side of the equation, begin scanning for still-vulnerable systems.

"Yes, they can do that," Gupta admits. But, she adds, as soon as a researcher spots a bug and reports it to Microsoft, Redmond "immediately puts mitigating controls in place."

And if the vulnerability has already been exploited, "then we do really unique things like we did with the Hafnium attack," Gupta says, referring to Chinese cyber spies who broke into vulnerable Microsoft Exchange servers in 2021 and stole data from tens of thousands of organizations in the US and UK.

In this instance, Redmond issues patches for older, unsupported versions because customers weren't able to upgrade to a fixed version quickly enough. Microsoft also built a "one-click mitigation tool" for affected email customers, and essentially told them, "even if you cannot patch, just run the script on your Exchange Server and you will be protected against this vulnerability immediately," Gupta says.

Patch Tuesday "is definitely a day of high emotions," says Bharat Jogi, senior director of threat vulnerability at Qualys. He's been doing the monthly patching event for the last 15 years, and says as soon as Microsoft pushes its updates, his company and other security vendors want to release checks for their products within 24 hours.

Improving relationships with security reseachers

Additionally, companies like Qualys that have a team of researchers also get to work trying to poke holes in the patches and looking for other similar vulnerabilities.

"As soon as the patches are released, they try to rip those patches apart and try to understand what was potentially fixed, how good it can be, and then start developing exploits for it," Jogi tells The Register.

That points to another potentially unintended consequence of Patch Tuesday: it has, over the years, improved the relationship between security researchers and software vendors, which, in the early 2000s, was contentious to say the least.

"Microsoft has gotten really good at crediting the security researchers who disclose the vulnerabilities," Crothers says. "Security researchers want to be recognized for their work for the betterment of their careers. In the early days, not just Microsoft but a lot of the software vendors considered security researchers as creating more harm than good."

This view has largely changed, and responsible disclosure has become an industry norm.

"Patch Tuesday, and all the associated work around this, has certainly been central to that, in my opinion," Crothers says.

Plus it gives bug hunters more ideas about where to try to poke holes in software.

"When you patch something, you shine a gigantic spotlight at it – especially for components that people aren't familiar with," Childs says.

Like Christmas morning, 20 years in the making

Childs has worked every Patch Tuesday since 2008, both on the Microsoft side and as a researcher. He's only missed two monthly patching events, once for federal jury duty and the other for his sister's wedding. "Neither one of them would move the dates," he says.

And he still gets excited about Patch Tuesday. On a scale of zero to ten, with ten being a kid on Christmas Day, Childs says he's usually an eight or a nine, depending on what vulns are disclosed. But even on a boring month, he rates himself "over a five."

"I get excited to see what's being patched, what the bugs are," Childs says. "I want to see who is researching what, and what's the latest and the greatest. If I were a sysadmin, I would probably feel very differently about it."

And even if Patch Tuesday does give the baddies a heads-up on bugs to exploit, the general consensus seems to be that it does make software – and people – safer.

"As much as I poked fun at Microsoft over the years, I do have to give credit where credit's due," Crothers says. "They clearly take this seriously."

Twenty years in, Patch Tuesday has become "one of those things that is just taken for granted, at least at large enterprises," he continued. "It's easy to forget how bumpy that road was, and the number of potholes in that road in the journey to where we're at today. It was certainly tumultuous to say the least." ®