WPTavern Was Trackbacked To Death

Now that things have calmed down, it’s time to discuss what happened that took WPTavern.com offline. It all started the day I published my review of the Backup Buddy plugin from PluginBuddy.com. That post not only received a large number of comments, but it ended up on StumbleUpon and was overall a large success. However, a few hours after that post had been published, I started to receive a large number of trackbacks to that particular post. The trackbacks were from different URLs which appeared to be junk domains. The site was setup so that an article was published near the top of the page and below the article was a list of related links. Each one of those related links pointed to a source outside of the parent site. I chalked all of these trackbacks as just typical splog spam and didn’t think nothing of it.

The next day, I woke up to see at least 50 more trackbacks in my Akismet queue all pointing to the Backup Buddy review post. When I started to look at the various junk domains, I noticed that the word Backup appeared to be the keyword these sites were after. Again, I deleted the trackbacks as I thought it was typical to see a large number of illegitimate trackbacks on a post that was popular. Things started to get interesting once I published the winners to the Backup Buddy give away as well as the WPTavern Backup Buddy coupon posts. At the time, I didn’t know this but I now realize that those extra posts with Backup in the title were adding fuel to the fire. I started to receive a large number of trackbacks on those posts with the word Backup in the title. I just kept deleting the trackbacks thinking nothing of it.

After I came home on March 9th, I took a 4 hour nap as I was exhausted from working on 4 hours sleep. During this time, AnHosting decided to suspend my account due to processor overload. Thanks to everyone on twitter for your concern as it prompted me to wake up and get into action. Since I couldn’t access cPanel or anything else tied to my account, I visited the AnHosting web page which gave me the option to perform a live chat with a technician. They explained to me that XMLRPC.php and Index.php were the two culprits sucking up all the CPU power. Once I told them I would disable XMLRPC on WordPress as well as disable trackbacks and pingbacks, they temporarily restored my site. After disabling those, I also used a trick I learned from Brad Williams where I viewed the source code on the index of the site and pasted that into index.html and renamed index.php. This was just in case a large amount of human traffic was the source of taking down the site. However, it struck me as odd that Woopra showed no signs up a large influx of human traffic on the site during that time or just prior to the site being suspended.

Once I renamed XMLRPC.php and index.php, the CPU on the server immediately went back to stable levels. However, WordPress needs index.php to function properly. Thanks to this line of code from Will Anderson, I was able to successfully use index.php without the scrapers being able to access it.

 if ( '/index.php' == $_SERVER['REQUEST_URI'] ) {
  
              header('Location: https://wptavern.com/');
 
              die('Do not request this file directly');

      }

According to Will Anderson, this is just a little redirection code with the hope that the scrapers will not be able to handle the redirect. Oddly enough, it worked and prevented index.php from being loaded by an external source. I say external source because according to Woopra, the source of the CPU overloading was not caused by human beings browsing the site. Instead, some type of software whether it was a bot or something else was loading my index.php file and either scanning the content for posts or doing something else. The irony to all of this is that during the attack, I noticed a comment in the Akismet queue advertising Trackback Submitter software. I’m pretty sure that XMLRPC deals with Track/Pingbacks in WordPress and if that’s the case, that would explain why that file was used so heavily. Once I was able to navigate within the administration panel of WPTavern, I noticed I had over 250 trackback spam links waiting in my Akismet Queue. While some of the IP addresses attached to them were the same, a large number of them were different. Also, the webhost for these junk domains also varied. It was as if I was attacked by a distributed denial of service through trackbacks. A cheap ass way to take a down a site in my opinion.

This is the first time I’ve ever experienced something like this. I really want to thank the tech support guys from AnHosting for allowing me to chat with them directly and work on resolving the issue. Obviously in a shared hosting environment, one bad apple can take down the whole tree. I apologize for being that bad apple. Furthermore, although Ozh had different reasons for doing so, I am highly considering disabling trackbacks/pingbacks for good considering I’ll see incoming links from either the Incoming Links dashboard widget or through other analytic software. Alternatively, I can use a plugin called Simple Trackback Validation which so far, has received a lot of good feedback from those who have used it.

I can’t believe how easy it was to perform a DoS attack on WPTavern.com through the use of trackbacks and XMLRPC. Although not an inherit flaw within the WordPress software, I wonder if anything should be done to prevent this sort of attack from happening? In fact, what can be done by WordPress to help prevent this from happening? If the Simple Trackback Validation plugin is as good as what people have told me, I’m guessing it could be absorbed into core?

19 Comments


  1. @Viper007Bond – I may have to pay someone to set that up on this site as those plugins are above my pay grade. Can you explain how caching would have helped in this case? Is it because it would have prevented Index.php from loading the same things over rand over again draining the CPU? Does it not matter if the thing calling Index.php is a bot or humans?

    Report


  2. I’m glad that you got the problem resolved. It’s definitely a crappy deal.

    This too makes me wonder if there could be a good solution for such a problem. Maybe a plugin could look for a massive influx of trackbacks and temporarily do the redirect trick to prevent site overload.

    I’ll mentally kick around the idea and see if I can come up with anything. The sad reality is that a solution today could easily be bypassed next week.

    Report


  3. @Chris Jean – I wonder if it’s just the nature of how the Trackback/Pingback protocol was developed and the only way to attack the problem at the source would be to update those protocols to help prevent their use in a DoS.

    It’s even harder to come up with a plan when you factor in that the trackbacks were coming from different IP addresses and different domains.

    Report


  4. @Jeffro – They’re actually fairly easy to install and come with great instructions. :)

    When someone hits any page on your blog right now, PHP generates the page and gets data from the database. Both of those actions use CPU power.

    Both caching plugins short-circuit that and instead serve a previously generated version of that page (which uses significantly less CPU). Two guests to your site will see the same content, so the plugin just generates the page for one and serves it to the second without using the CPU power needed to generate the page.

    New comments and such automatically invalidate the cache files, so it’s basically install and forget.

    Report


  5. Hi Jeffro

    Sorry to hear about your issue with spam yesterday, there are some real low lives around.

    I was wondering about the Simple Track Back validation, but if the server is under load already, how much more load would a plugin checking each trackback put on the server?

    It would be interesting to see how the Track Back Validation performs under the same circumstances that you had.

    Regards
    Kimberly

    Report


  6. @Kimberly – heh, quite a few people were telling me to install plugins but in the middle of an attack, it’s a bit hard to do, especially when the site is using up server resources causing everything to get suspended. My first course of action was to somehow eliminate the CPU overload from those two files which I did. I’m afraid to remove the code from the Config file as the attack may keep going.

    Report


  7. Glad to see that you’re back up and running. I agree with Viper007Bond, either of the caching plugins would have helped tremendously. They both so the same thing as you did with the static HTML file, and both are dead simple to use.

    Neither of them require code ninja, skills, if you can do the static HTML trick, you can get either of the plugins running.

    I’ve also used both of the plugins and think WPSuperCache is easier to configure, but W3 is more powerful.

    I’ve been seeing a lot of trackback spam of late as well. I might have to resort to disabling them as well if they get bad.

    Report


  8. Seriously, trackbacks are cool when you begin blogging and are all excited when you get a new link back, but past this they are just worthless. Disable trackbacks, simple.

    Report


  9. I disagree completely with the argument that trackbacks are useless and should be disabled. Trackbacks epitomize the inter-connectedness of the blogosphere, tying conversations together across sites. A trackback is, essentially, the extension of a conversation.

    Yes, they can be abused; yes, that abuse must be mitigated.

    No, they should not be disabled.

    Report


  10. @Viper007Bond

    For a site like wptavern which is a regular resource for wordpressers, cache should be a requirement.

    Actually, that’s a bit of exaggeration since its hosted on a shared enviornment and b/w is not a problem but still, it will reduce cpu cycles.

    Report


  11. Jeffro, I would say it is definitely worth the time to try either SuperCache or TotalCache.

    Most of the default settings are good enough, and I bet that folks in the Forums would answer any questions you might have :)

    Report


  12. This is an interesting topic. I turned off trackbacks/pingbacks on most of my blogs (personal and CMS ones). I agree no one seems to use trackbacks/pingbacks properly anymore. Almost all the trackback/pingbacks I see are from untrusted sources that I end up moderating as SPAM. There are occasional real trackbacks though, so I don’t necessarily recommend turning it off on all blogs, depends on what you’re doing.

    Thinking of DoS attacks, every page request for a WordPress blog requires PHP/MySQL to load, so depending on how much horsepower your server has it sometimes doesn’t take much to make a server go down. If you’re using shared server where you don’t have access install real caching such as APC on your server, then you’re only option is a caching plugin. Even if you use a caching plugin your blog still uses CPU/memory because PHP/MySQL still load. There are some caching plugins that do true caching with very complicated mod_rewrite rules in the .htaccess file, but using such caching means you cannot use a lot of the cool plugins that update dynamically (twitter and rating plugins come to mind).

    If you have full control of your server, you’re better off installing APC so you get true caching of PHP in memory. There are many other tweaks you can make to MySQL as well to facilitate caching of the most queried data. If you have the ability to do these things, there’s absolutely no reason to use a caching plugin since these applications like APC do the caching at a much higher level.

    For blog feeds, I wrote a plugin called Static Feed that lets you serve your primary feeds as static XML files. This gives you the advantage of being able to serve your feeds even if your PHP/MySQL fail due to a DoS attack. I use it to remove the extra CPU/Memory load from having WordPress dynamically re-create the XML feeds every time the feeds are requested. The Static Feed plugin works by saving an XML copy of your feed any time you edit, add or delete blog posts. If you’re only writing 1-3 blog posts a day, you’re only resulting in a handful of MySQL queries to your database and saving a lot of CPU/Memory usage from loading PHP. Best part is my Static Feed plugin can either be configured for permalink style feeds (e.g. example.com/feed/) or Moveable Type style (e.g. example.com/index.xml).

    Report


  13. Jeffro, I’m a regular listener to your podcast and appreciate all the effort it takes for you to produce it weekly.

    Glad your site is back up and running – I was reading the comments and it brought to mind a question I’ve had that maybe you or your community can give me some direction on.
    From time to time I look at the comments in my que and wonder if they are spam or if the people commenting use English as a second language! Some of the comments are so strange – like these two…
    “I just love the way you write, its clear and easy to understand, and your posts are not fluffy and very informative, thanks for posting.” Or “After reading you site, Your site is very useful for me .I bookmarked your site!”
    What do you look for to determine if a comment is from a spammer or not? If I get a comment like the above with no links in the comment, but with a kind of spammy-looking email address, should I spam the comment?
    I don’t want to be rude to anyone – I’m thrilled when people comment, but I don’t want to do anything to help a spammer either.
    Anyone want to weigh in? Thanks.

    Report


  14. @Lynn Dye – I’ve seen some of those, too. If you are unsure whether or not a specific comment is spam, check the name and (if any) URL included with the comment.

    If the comment text has no links, and the name/email doesn’t contain a URL, then it’s probably not spam.

    On the other hand, many WordPress blogs are configured to require moderation of all comments from commenters who have not had a comment previously approved. So, spammers may be taking the tack of posting one, innocuous comment in order to get around this first-comment moderation. So, for a borderline comment, just be sure to watch for any future, spam comments from the same email address (assuming such comments get past Akismet).

    Report


  15. Ouch. That happened to me once. My site was hit with 80,000 spam comments/trackbacks in one or two days. I switched off trackbacks shortly after. Most blogs use pings nowadays anyway.

    Report

Comments are closed.