HTTP/2 'Rapid Reset' zero-day exploited in biggest DDoS deluge seen yet

A zero-day vulnerability in the HTTP/2 protocol was exploited to launch the largest distributed denial-of-service (DDoS) attack on record, according to Cloudflare.

Surpassing 398 million requests per second, the attack is believed to be more than five times larger than the previous record of 71 million requests per second. Google, Cloudflare, and AWS led a coordinated vulnerability disclosure on Tuesday for the flaw tracked as CVE-2023-44487 or Rapid Reset.

All three have been monitoring application-layer (layer 7) attacks much larger than what's considered normal for months, with activity peaking in August, they said. The trio were keen to find out exactly how this flood of traffic was being generated. Whoever was launching these assaults wanted to overwhelm their targets with packets, causing systems to go offline for legit users.

Cloudflare's analysis revealed that cybercrooks were exploiting the aforementioned weakness in HTTP/2 using a smaller than usual network of criminal-controlled bots, which partly explains the huge number of requests per second.

"One crucial thing to note about the record-breaking attack is that it involved a modestly sized botnet, consisting of roughly 20,000 machines. Cloudflare regularly detects botnets that are orders of magnitude larger than this – comprising hundreds of thousands and even millions of machines," the biz said.

"For a relatively small botnet to output such a large volume of requests, with the potential to incapacitate nearly any server or application supporting HTTP/2, underscores how menacing this vulnerability is for unprotected networks."

All three service providers have published mitigations and implemented new technology to protect against Rapid Reset attacks in the future.

How Rapid Reset works

The method relies on stream multiplexing, a feature of the HTTP/2 protocol that allows multiple HTTP requests to be sent to a server on a single TCP connection. These requests are serially streamed to the server along that one connection; the server collects up the streams of requests, processes them, and responds. So when your browser opens a page, it can fire off separate requests for all the content on the page serially down that one connection. This is supposed to be more efficient than HTTP/1.x's traditional approach, which typically involves taking time and resources to establish multiple parallel TCP connections to fetch stuff from a server. HTTP/2 does it all down one connection.

A feature of the protocol's streaming capability is the ability to send a request and soon after cancel that request, an action known as resetting the request's stream. When a client makes a request and then cancels it, the server gives up processing the request while keeping the HTTP/2 connection open. This avoids having to open and close multiple TCP connections, and is useful for fetching a load of images on a page and then canceling the ones that aren't visible if the window has scrolled past them already, for instance.

A normal HTTP/2-based DDoS attack would involve attackers opening up as many of these streams as possible and waiting for responses to each request from the server or proxy before firing off another flurry of requests, and repeating this over and over. There are only so many streams a server will allow down one TCP connection, so it might only accept 100 streams at a time. The key thing here is that the attacker waits for those 100 or so responses to come in before sending another load of requests.

Rapid Reset attacks get around that limit, allowing many, many more requests to flood a server. It is as simple as sending a request in a stream then quickly resetting the stream, canceling that request and keeping the connection open. The server starts processing these requests and then later stops, and because each request was canceled, it doesn't count against the maximum number of allowable streams. You just keep firing off a request the rapidly resetting its stream, and do this as many times as you can, and now the server is having to start and stop an overwhelming number of garbage requests.

Google explains here how attackers can thus ensure a large number of requests remain in flight while never exceeding the maximum number of streams that are allowed by the server. The network's bandwidth becomes the factor that determines the number of requests that can be made, instead of the round-trip time (RTT).

"The client opens a large number of streams at once as in the standard HTTP/2 attack, but rather than waiting for a response to each request stream from the server or proxy, the client cancels each request immediately," as Google engineers Juho Snellman and Daniele Iamartino put it.

These canceled requests demand a great deal of unnecessary work from the server, costing it time and money to process stuff without ever sending anything back, while the client, or in this case the attacker, pays "almost no costs" for sending them.

Essentially, the process allows attackers to flood servers with more requests than ever seen before, leading to larger-scale DDoS attacks that are difficult to mitigate.

Variants

While the criminals behind this wave of attacks aren't known, Google observed some variants of the attack method. It's not clear if these were performed by different groups or the same one experimenting with different methods.

• Hacktivist attacks erupt in Middle East following Hamas assault on Israel
• Feds hopelessly behind the times on ransomware trends in alert to industry
• Scattered Spider traps 100+ victims in its web as it moves into ransomware
• Huge DDoS attack against US financial institution thwarted

One of the better variants didn't immediately cancel streams. It opened a batch of streams at once and canceled them after waiting a period of time, then opened another large batch of streams and repeated the process.

Google said this method may bypass mitigations that involve limiting the number of stream-reset frames that can be sent per second, but was still less rendered effective than the original Rapid Reset.

The second variant attempts to open more streams than the server allows, not canceling them at all – a method that can bypass some client-proxy RTT and proxy-server RTT bottlenecks, but is unlikely to be processed by most HTTP/2 servers, Google says.

Mitigations

Cloudflare has made changes to its DDoS mitigation service to counter Rapid Reset attacks, making it available to all customers. AWS said those who developed a strong DDoS-resistant architecture using CloudFront and AWS Shield will have seen their applications' availability unaffected due to measures Amazon took to defeat the Rapid Reset assaults when they started appearing.

Google explained there are a number of ways to implement mitigations for Rapid Reset attacks, but discouraged the use of GOAWAY frames, as is recommended in HTTP/2's specification for closing connections.

These aren't set up to handle the kind of activity seen in Rapid Reset attacks and shouldn't be relied upon alone, the web giant said, but they can and should be used to limit stream creation. See the above links for more technical information.

"Mitigations for this attack vector can take multiple forms, but mostly center around tracking connection statistics and using various signals and business logic to determine how useful each connection is," Team Google said.

"For example, if a connection has more than 100 requests with more than 50 percent of the given requests canceled, it could be a candidate for a mitigation response. The magnitude and type of response depends on the risk to each platform, but responses can range from forceful GOAWAY frames as discussed before to closing the TCP connection immediately.

"To mitigate against the non-canceling variant of this attack, we recommend that HTTP/2 servers should close connections that exceed the concurrent stream limit. This can be either immediately or after some small number of repeat offenses." ®