Ryan Hellyer’s AWS Nightmare: Leaked Access Keys Result in a $6,000 Bill Overnight

WordPress developer Ryan Hellyer had always wanted to open source his website. As a strong supporter of open source software and an avid plugin developer, he enjoys sharing his code and learning from others. This desire led him to put his site up on GitHub one evening, not knowing that he would wake to find himself in a security nightmare due to a simple oversight.

Open sourcing a website is not such an uncommon practice, as it brings with it a number of benefits. Hellyer shared his story with the Tavern and identified the main reasons wanted to make his site’s code public on GitHub:

1. It allows people to see what plugins and themes I use
2. It makes it super easy to get help with my website, since people can see the code
3. It encourages me to use best practices across my entire website, not just the bits I post for download (custom plugins, themes etc.)
4. It is a handy way to sync the files between locations

“I was aware that keeping my wp-config.php file off of GitHub was critical,” Hellyer said.

“Leaving that open for download would also make my database credentials, security salts etc. known to attackers. To mitigate this, I simply moved it one folder down (it still works even when it’s not in the root of your site). As a double protection, I also used the .gitignore file to forcibly prevent Git from pushing it to the repository.”

Satisfied that his website was totally backed up and thinking that he had taken all the necessary security precautions, Hellyer pushed all the contents of his site (with the exception of the uploads directory) to a new GitHub repository. Pleased with his efforts, he dozed off to sleep.

An Urgent Message Arrives from Amazon

Not four hours later, Hellyer received an urgent message from Amazon, though he didn’t have the chance to read it until later in the morning.

“I awoke fresh the next morning, began work for the day, then decided to check my personal email account and saw the email I had received overnight,” he said. “The email was marked URGENT.”

Hellyer initially thought it was spam but went with his instinct and opened it anyway, as it appeared to be fairly legitimate. Amazon emailed to notify him that his account may have been compromised.

“I immediately went to check my AWS billing page, but thankfully it was only at US$17 for the month so far, which was about my normal usage,” he said, noting that he uses AWS (Amazon Web Services) for backing up to Amazon S3 and for providing a CDN service to his website (via Amazon Cloudfront).

The email specifically asked him to check his EC2 instances, which Hellyer found to be odd, since he doesn’t even use Amazon EC2.

“I checked anyway, and surprisingly, here were 80+ servers running. I thought ‘OHHH CRAP!’

“So I shut them all down. Problem solved I thought, and since my bill was still looking normal I figured it wasn’t a problem.”

Ten minutes later Hellyer discovers another 80+ servers running at a different location on EC2. Digging deeper, he found five more locations, all with 80+ servers running. After fully auditing his account, he found more sections of “reserved instances” with even more servers running.

“In total, there seemed to be around 600 servers running. The time between realizing all this and uploading my Git repository was approximately 12 hours.”

Digging Out of a Hole

Hellyer went into damage control mode, scrambling to find the source of the problem.

“My immediate thought was ‘YOU IDIOT! You didn’t remove the wp-config.php file which contains AWS access keys!’ he said. “But I went to check the repository, and it was not there. It turned out though, that there was another file called ‘wp-config.php.save,’ which DID contain my AWS access keys.”

That file also contained his database password and security salts, but so far he hasn’t found any indication that the site was compromised by those. Hellyer immediately changed the password and swapped out the config keys.

“But those horrid little AWS access keys were sitting on the repository in view of everyone. I immediately deleted the entire repository from GitHub.”

Unfortunately, it took him two hours to delete all of the Amazon EC2 instances created by the exploited keys. “The evil little blighters had cranked up the security protection and actually made it very awkward to shut them all down,” he explained. “I wasn’t able to just terminate them and in fact had to go through one by one and manually turn off termination protection, then stop them, then terminate them. Then I had to go through and delete many volumes which had been setup (I think this is the EC2 equivalent of a drive).”

Hellyer is Slapped with a $6,000 Bill for Unauthorized Usage

AWS bills don’t update in real time, and Hellyer’s bill still read $17 USD. After contacting support to find out the damage, he saw his bill jump from $17 to $3,087.97. AWS was helpful throughout the process and give him a list of tasks to complete, including deleting some hidden EC2 instances which had not previously been visible in the console.

“After doing all of this, I went to take a snapshot of the billing statement to show everyone my lovely US$3,087.97 bill, only to find it had shot up to US$5994.08 in the meantime,” he said.

Amazon followed up with the following note:

We’ve submitted a concession request for the unauthorized usage charges you incurred for the current month. The concession request requires approval from levels of management therefore the process can take up to 7 – 10 working days to complete.

He’s hoping that the leaked keys will not result in him having to cough up $6K for unauthorized use, but he hasn’t yet received confirmation that he won’t be held responsible.

Why were his AWS Credentials in his wp-config file?

If you’ve been following along with this nightmare situation, you may be wondering why Hellyer was storing his AWS credentials in wp-config.php. This was required for the Photocopy plugin he released a few years ago. There are many other plugins that utilize this same method. Although the plugin didn’t seem to have any security flaws, he decided to remove it from the site and discontinue development due to the unpleasant experience of having his keys leaked. “The only other option is store it in the database, but I think that’s actually worse as you would run into exactly the same problem if your database were leaked,” Hellyer said.

Keep Your Keys Safe and Private

This cautionary tale should serve as a reminder to keep your keys safe and private. Hellyer unknowingly sabotaged himself when trying to open source the code for his site and learned an expensive lesson:

“Not only should you be extremely careful with your usernames, passwords, plugin and theme security etc., but you need to be even more careful about credentials for services which cost money,” he advised.

“I’d rather see my blog(s) hacked than to have this happen again. Hacked sites can be easily fixed. Deleted data can be restored. A $6000 bill however is something else entirely,” Hellyer said. He also noted that if the exploit had just been a few extra dollars here and there, it could have quietly leached his account indefinitely without him noticing. As it was, if Amazon hadn’t issued him an urgent notice, he probably wouldn’t have noticed until he received a $100,000 bill at the end of the month.

The lesson here is that if you can avoid having to store AWS credentials in wp-config.php, by all means, find another way. What started out as a well-meaning effort to open source his site, quickly became a horrible nightmare that has yet to reach a conclusion. There are people out there ready and willing to exploit stolen or leaked AWS keys, and they clearly have a sophisticated way of scouring the web to find them.

Hellyer sums it up: “Long story short … don’t do stupid stuff with publicly accessible code. Also, don’t store your AWS credentials unless you absolutely have to.”