Cloudflare Memory Leak Exposes Private Data

Cloudflare, a content distribution network used by many popular sites, published detailed information about a security vulnerability that leaked user information, some of which was private, including passwords, private messages, etc. The vulnerability was discovered by security researcher Tavis Ormandy, a member of Google’s Project Zero team.

The issue stems from a memory leak in an HTML parser named cf-html that was created to replace an older parser based on Ragel.

“It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used,” John Graham-Cumming, Chief Technology Officer at Cloudflare said. “Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself.”

The earliest date information was leaked was September 22nd, 2016 when Automatic HTTP Rewrites were enabled. This was the first of three features introduced that used the parser. The other two are email obfuscation and Server-side Excludes.

The greatest period of impact was between February 13th and February 17th. The leaked information ended up in publicly available cached webpages. Cloudflare worked with major search engine providers to have the cached pages scrubbed before publicly announcing details of the bug.

“With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory,” Graham-Cumming said. “Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines. We also undertook other search expeditions looking for potentially leaked information on sites like Pastebin and did not find anything.”

1Password is Not Affected

Earlier reports indicated that 1Password was among the sites affected. Jeffrey Goldberg, a 1Password employee, assured users that the Cloudflare data leak does not affect 1Password.

“At the moment, we want to assure and remind everyone that we designed 1Password with the expectation that SSL/TLS can fail,” Goldberg said. “Indeed it is for incidents like this that we deliberately made this design.”

“No secrets are transmitted between 1Password clients and 1Password.com when you sign in and use the service. Our sign-in uses SRP, which means that server and client prove their identity to each other without transmitting any secrets. This means that users of 1Password do not need to change their Master Passwords.”

Change Your Passwords

Nick Sweeting has used a number of web scrapers to compile a list of sites that use Cloudflare. The list is available on GitHub and currently contains 4,287,625 domains that are possibly affected. Popular domains in the list include:

• authy.com
• coinbase.com
• digitalocean.com
• patreon.com
• bitpay.com
• news.ycombinator.com
• producthunt.com
• medium.com
• 4chan.org
• yelp.com
• okcupid.com

The bug also affects mobile apps as HTTP header data for apps such as Discord, FitBit, and Uber have been discovered in search engine caches. NowSecure published a list that includes 200 iOS apps that use Cloudflare services.

Users are strongly encouraged to change their passwords regardless if a site uses Cloudflare or not. Those who use Cloudflare should generate new API keys and consider forcing a password change to users.

Two factor authentication should be enabled where possible so that the password is not the only credential needed to access an account. Mobile users should log out of mobile applications and log back in to create a new active token. To force all users on a WordPress site to logout and re-login, WPStudio recommends changing the salt keys in wp-config.php.

Although major search engines are actively scrubbing cached pages, the leaks have been occurring for at least four months. There’s no telling who may have already scraped those pages and archived the data. There’s also the possibility that someone discovered the vulnerability before Ormandy and has been parsing cached pages for months. This is why it’s important that at a minimum, you change your passwords.