Linode Confirms Data Security Breach That Matches Recent WP Engine Attack

linode

Cloud hosting provider Linode has been combatting DDoS attacks since Christmas Day, which have caused multiple disruptions and service outages across its global data centers. The attacks are ongoing and the company is struggling to keep its status blog up to notify customers.

In addition to the DDoS attacks, Linode has also confirmed a data security breach:

A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials.

All Linode Manager passwords have been expired as a precaution after customer credentials were found in the wild. The team is working around the clock to mitigate the DDoS attacks and further investigate the unauthorized access to customer accounts. Linode has not yet determined whether there is a link between the two attacks.

Several days ago, WP Engine identified its cloud infrastructure provider as the entry point for the company’s recent security breach. The company is listed as one of Linode’s customers. Jason Cohen, the company’s founder and CTO, jumped in to answer several questions in WP Tavern comments earlier today but would neither confirm nor deny that Linode is the cloud infrastructure provider in question.

PagerDuty, a former Linode customer and victim of a similar attack, speculates that Linode may have been compromised since July 2015 and is only now announcing it:

We immediately reached out to them not only to inform them of their compromise, but to assist them in investigating it. We were confident that the Linode database had been breached, and that the secret key used to encrypt information in the database had been compromised as well.

In addition to reaching out to Linode, we also worked with a third-party security firm to audit our work done during the incident. Likewise, around the same time we reached out to law enforcement to assist in investigating the attack. We did not get confirmation in July that there was a breach of the Linode Manager or any associated credentials.

PagerDuty migrated away from Linode in August because of this breach, but the company was not allowed to disclose to its customers that Linode was the point of entry.

WP Engine’s security breach is strikingly similar, as the company’s attacker bypassed multiple layers of authentication to gain access to an administrative panel. According to Cohen, “the criminal’s behavior in this exposure matches a pattern seen in other attacks throughout 2015.”

If multiple Linode customers have been affected and are unable to reveal the point of entry, they may have put pressure on the cloud hosting provider to finally publicly disclose the nature of the attack.

In an age when nearly every hosting provider will have attacks and service disruptions, what matters is how they handle it and communicate with their customers. Linode has been plagued by multiple security issues in the past. Failure to disclose incidents in a timely way can be costly, especially in the current competitive hosting market where transparency with customers is at a premium.

When Linode’s investigation of the criminal activity is complete, it would be appropriate for them to disclose how long they knew about this compromise and when they first acted on it. The most recent update on the security breach does not include a specific timeline of events.

21 Comments


  1. From their announcement post, it sounds like the passwords were adequately hashed, so the only accounts compromised were likely those with very weak passwords.

    Perhaps the DDOS attack is merely a way to distract Linode from finding the compromised accounts.

    https://blog.linode.com/

    Report


    1. A very interesting take on the situation. Hopefully it wasn’t another case of Password1234.

      Report


  2. It sounds like Linode has some sort of NDA with WPengine and PagerDuty.

    Report


  3. My own WP Engine account was moved from Linode’s troubled Atlanta data center to another provider in another part of the country this week. I’m grateful that WP Engine took decisive steps to make my hosting life easier moving forward.

    Report


      1. According to the info associated with my new IP address it looks like it’s now Rackspace.

        Report


      2. I have noticed that my only WPE-using client is also in a Rackspace data centre. I think they always have been, and I imagine that perhaps they ended up there because I switched them to WPE’s PHP 5.5 platform before launch.

        Report


    1. This sounds like a good move for wpE. Many sysadmins I’ve worked with have been running from Linode since 2014 for various reasons — security near the top of them but lack of communications and trust rank higher.

      Report


    2. Same with us. However we manage over 50 clients’ hosting. They spent a month convincing us that these new Linode servers were top of the line, fast-loading, reliable. After over 24 man-hours we finally got all of the A Records switched over to Linode on Christmas Eve. Literally within hours, the DDoS attacks began, and WPEngine began migrating people *away* from Linode! To RackSpace – the server they had urged us to migrate away from in the first place! It’s not WPE’s fault; they’re only trying to select the best hosting option for their customers. But still, this scenario has been extremely unfortunate and has resulted in 100’s of hours of combined downtime for our clients, and I have the half-empty bottle of Aspirin to prove it.

      Report


  4. So I had been planning on trying out WPengine, and I finally gave it a shot. Moved my two most important sites over. The next day, all this business started.

    lol, talk about bad timing!

    I’m hanging in there. They moved my stuff to a different datacenter, and so far so good.

    Report


    1. I got a free personal trial from WP Engine years ago at a WordCamp and I’ve been using them ever since. As the article mentioned, no host is immune to attacks and I’ve always been very happy with WPE’s service, support, and features. I’ve got close to 50 clients with them now so the last week has definitely been difficult (most setting expectations with those clients) but it’s really the only noticeable downtime in the 3+ years I’ve been a customer.

      Report


  5. All of my personal sites are hosted with Linode. My uptime for the past month is over 99%. That compares with around 99.99% for most months.

    Report


    1. I have good experience with Linode too. Of course the DDOS was annoying but it could have happened to any host. I am staying with them for now and see what the future brings.

      Report


      1. I’m more worried about the security problems than the DDoS.

        Report


  6. I have been with Linode for years with multiple sites. However, as of yesterday I have canceled my account with them for good. The only reason that I found out about all this was happening was reading posts on other sites. I never received one single notification from linode… I can excuse a lot but this lack of communication I cannot let go. So good bye Linode… Hope other companies learn from this and the importance of customer communication.

    Report


  7. since wpengine uses linode then of course it would make sense about the breach

    Report

Comments are closed.