Dealing with a Hacker on Dreamhost
Earlier this year, my Dreamhost webhosting account was hacked. I’ve been dealing with it for months, but I’m no programmer. The information provided by Dreamhost customer support, while helpful, has been far from sufficient to actually resolve the problem in a satisfactory way. That’s why I’m writing this blog post: to help others than might be in a similar situation.
How the Hacker Got In
I’m pretty sure the hacker got in through an old abandoned WordPress install that I had forgotten to delete. (It’s essential that you either keep all web apps up to date, or delete them. To do otherwise is to ask for trouble. Hackers will eventually discover the old installs with security vulnerabilities.)
After gaining access, the hacker uploaded a PHP backdoor script which allowed him to get back in easily and upload or edit any files he wants, even after I deleted the old WordPress installation that had the vulnerability. The backdoor script he used is called PHPspy, and is freely available on the internet. (Interestingly, it’s also Chinese.)
What the Hacker Did
The hacker was not overly malicious as hackers go. He was after precious Google Page Rank. He took a three-step approach to his spam injections:
- Add a new directory full of “male-enhancing” spam content (hundreds of separate HTML files, optimized for all kinds of possible searches).
- Add a harmless looking PHP file, like
style.phporcss.phpsomewhere on your webspace where it seems to fit, like in a WordPress directory. This file contains a PHP eval base64 decode function. This allows the file to hide what it’s actually doing; it has to be decoded to be readable by humans. (There are sites which can decode base64 online for free.) If you decode that content, though, you’ll see that it’s including all the spam content added in step 1. - Edit your WordPress blog’s theme’s
header.phpfile to include the innocuous-looking file added in step 2. It is added in a way that is invisible to visitors, but highly prominent to the Googlebot.
The effect of all this is that (1) searches for all those “male-enhancing” drugs on your site reveal tons of hits, and (2) the pages it points to all go lead to some other site, which reaps the Google page rank benefits.
Dreamhost’s Response
It has taken me quite some time to figure all this out. Dreamhost, so great in many ways, takes a very hands-off approach to security issues. I was given this advice by Dreamhost staff:
After updating your software, it is imperative that you go through all files under all directories for the user which has been compromised and ensure that any files which have been written to or modified have been removed. It is common for ‘hackers’ that exploit web scripts to upload innocuously-named scripts which they can use to further compromise the site more easily, even after the initial vulnerability is closed — including scripts to send spam mail or execute arbitrary shell commands under your account via a simple web page interface. A helpful tip for finding files of this nature is to look for files or directories that have timestamps that occurred since you last modified your site, or that occurred around the time that the ‘hack’ took place; still it is best to examine all files as even a single missed file can allow the site to be re-compromised.
As you can see, this is good advice, and it fits my situation perfectly. It was the “date modified,” viewable by FTP, that enabled me to find the hacked/uploaded files and directories in the first place.
Unfortunately, for an account with web content as sprawling as mine (I host a number of other sites in addition to Sinosplice), manually going through every directory isn’t an option. Even if you were willing to do this (it would take many hours), you would have no guarantee that the hacker wasn’t accessing the site right behind you, re-uploading backdoor scripts in directories just after you’ve cleaned/cleared them.
So it sounds like the best solution is deleting everything and re-uploading. This, unfortunately, isn’t the easiest thing to do either when you have multiple copies of WordPress installed, each of which is tied to Dreamhost’s automatic one-click installer system. Ironically, it’s also the Dreamhost one-click installer system that adds to the problem. The Dreamhost one-click installer adds no less than 50 themes to each copy of WordPress it installs. That’s 50 extra potential hiding places for a hacker’s backdoor scripts, one of the reasons why a manual check is not an option. (You can delete the themes you don’t want, but it’s a slow process by FTP, and you have to do it again, on every WP install, every time you upgrade. Even if you do it through your shell account, some of the themes have permissions set in such a way that a simple delete won’t work.)
Solving the Problem
The first thing you have to do is shut out the hacker. Until you can do that, it’s pointless to spend much time cleaning everything up. The “hacker” may in fact be an automated script. It may never tire of replacing all the files you delete. Obviously, you must shut out the hacker first.
Dreamhost provides a page which helps you figure out how your site was compromised. In my case, I was able to confirm that neither my FTP nor my shell account had been compromised; it seemed to be a simple case of web access.
After determining that my hacker was using PHPspy to gain entry, I knew I had to find all instances of PHPspy on my webspace and remove them. To do this, you’re going to need shell access. You can enable it through the Dreamhost control panel. (If you have no idea what shell access is, you probably shouldn’t try to use it. It’s not user-friendly at all.)
Anyway, once you have shell access, you can start using the find command. Searching for filenames is pretty useless, since the hacker can name his files whatever he wants. You need to search for strings in files. By actually searching the contents of files, you can easily identify (1) spammy or unwanted content, (2) any files containing the very suspicious string eval(base64_decode(, (3) the files including the files with the eval(base64_decode( strings, and, most importantly, (4) backdoor scripts (if you know what they are). In my case, I knew all the “male-enhancing” keywords to search for, and since I had found a copy of PHPspy on my webspace (innocently named log.php), I was able to open it up in a text editor and choose some unique strings in the script to search for.
After I identified all those files, it was simple to (1) delete the files that shouldn’t have been there (especially the PHPspy scripts!), and (2) re-upload the legitimate files to overwrite the ones that had been hacked to include the spammy stuff.
Finally, to monitor the changes made to files on my webspace, I set up a monitor script as described in step 3 of 3 on this forum post. I deleted the “Recent Logins and Associated Hosts” part because it was resulting in tons of useless data.
Conclusion
Again, I’m disappointed that Dreamhost couldn’t help more. I’ve been a loyal customer for years, and have referred quite a few new customers. It’s just not their policy to get directly involved, however.
The good news is that the non-urgent nature of my particular security situation gave me ample time to investigate, consult (big thanks to Brad and John Levermore for their help), and experiment. It’s my hope that this information can help others in similar situations.
A big chunk of my free time in the past few weeks has been taken up by this security issue. I’m certainly no expert, so any corrections or advice from those more knowledgeable are welcome! Hopefully this is the last you’ll hear of it, and I can go back to writing more Chinese-related stuff for Sinosplice.
2016 Update: I later switched to WebFaction, and have been very satisfied for years. I recommend it!
Well at least you’ve vanquished the foe! I’m a loyal dreamhoster too, and I appreciate the hands off approach sometimes, other times I get very frustrated.
You can also use the find command to find all files of a certain age. That’s the easiest way to identify all the files recently modified/created by the attacker. Just run “man find” for more info.
Sorry to hear about the attack, but it is good to know you have got to the bottom of everything.
It isn’t so much the attack itself, but your decision to write a blog post about it that reminded me of a similar situation I was in myself a couple of years ago.
I helped a friend investigate strange behaviour on a web server he maintained and it turned out that an attacker had guessed the password to one of the shell accounts (thankfully not the root account). The attacker then tried several methods to elevate his access and access the web folder.
I detailed the attack on my blog only to receive an email several days later from the Serious & Organised Crime Office. Thinking it to be spam, I ignored it only to receive a second email and a subsequent voicemail only a few days later. I started to wonder what I’d been downloading recently and couldn’t think of anything (by that stage I was even using a legal copy of Windows on my desktop).
Then another call came and I answered it this time. I told them quite politely I thought it was a con and that the Serious Crimes Office wouldn’t be emailing me. They suggested I hangup, call directory enquiries and ask to be put through to Mr X at the Serious Crimes Office. I was terrified.
I did just that, spoke to Mr X. And in the end, all he wanted to do was to let me know that he’d been contacted by someone thinking I was the attacker. He’d read the post and wanted to leave his details so that I could contact him should anyone give me trouble.
Scary stuff, but wouldn’t stop me sharing details should it happen again.
As Jordan says, running the find command for timestamps is generally useful.
find . -type f -mtime -365 -ls
OR
find . -type f -mtime -365 -print0 | xargs -0 ls -alFb
Just modify the number of days (365 here) to 1 less than the date of the latest change that was yours. The -print0 and -0 help with filenames containing spaces, the -F adds a symbol for executables, directories, and symbolic links, and the -b escapes weird characters that are sometimes not visible in a normal ls, and can be used to masquerade a filename.
John,
Like everyone else said, thanks for the detailed warning and sorry about the hassle. You didn’t refer me directly to dreamhost, but I remember seeing that you were using them when I was looking for new digs. And I thought: if it works for him it’ll work for me. My space has many similarities as well (hosting multiple sites, etc.). I’m off to do some maintenance this very moment.
Markovitch,
Well, I certainly hope I’ve shut the hacker out. It’s too early to be sure now. At least now if the hacker returns I’ll have detailed information about when he returned and exactly what files he changed/added.
If the hacker was thorough enough to upload more than one different type of backdoor script, then I haven’t actually shut him out. I doubt he went to that extreme, though.
Jordan and Chad,
Thanks for the tips!
Beijing Sounds,
If your skills are up to it, I recommend you try out a few basic find commands using shell access. When you finally need them someday, you’ll at least have basic familiarity.
It wasn’t really “a hacker” who “broke in” – this isn’t the old days. It was just an automated attack, nothing personal. It sounds like no damage was actually done, either, just a spam operation, which means the criminal had been hired to do it.
Krovvy,
Sure, the exploit/security hole was undoubtedly identified by a script. As for the subsequent actions taken, however, I originally assumed the same as you, but the reactions/adjustments to the changes/fixes I made and the erratic visit schedule seemed to indicate a human intelligence at work.
I don’t really care, though, as long as I can lock down my site.
Precisely the same thing as you described has just happened to my site. Ack, what a nuisance. 🙂
I appreciate the detailed post. Now to do the leg work to fix it!
Ben,
It turns out the steps I described in this post weren’t enough to shut out the hacker. I eventually moved to a new host and installed my blogs fresh. So I’ve left the hacker behind on DreamHost.
Base64 can be also decoded here
http://www.base64online.com
If one have acces via ssh can easly pick up how to remove it
below command give you a picture where the worms are
find . -type f -exec grep “eval(base64_decode” {} \; -print
if you want to replace in one go use command below
it will remove infected line from php file and create php.bak file in case smth goes wrong
find . -name “.php” -exec sed -e ‘/eval(base64_decode/d’ -i.bak {} \;
Enjoy
I think DH is the worst hosting company out there. In certain cases nothing beats a human over the phone with quick results. As a security expert I’m sadden to see client after client be told by DH to look it up on Wiki. That’s their favorite line. I even have friends that work in Abuse there and that is drilled into their heads there. Tell them to look it up at DH Wiki. Well that’s not good enough when people/clients are the ones keeping you in business DH.
Kris, you commands are wrong by the way.
I personally recommend this command,
$grep -lr –include=.php “eval(base64_decode” ~/public_html/ | xargs sed -i.bak ‘s/eval(base64_decode[^;];/\n/g’
it’s least resource intensive way to run the command, and is pretty easy to understand.
-l means only list the file names
-r means recursive
–include=*.php means only scan files ending .php
search for the term eval(base64_decode in each file
start from ~/public_html/
Then one it generates a list, clean the files.
[…] like what happened to Sinosplice a while back, Sinoglot has been hacked. In fact it was the exact same hack that hit John’s […]
[…] probably due to outdated web app installations (and not the previous hack). But DreamHost proved amazingly unhelpful in shutting out the hacker. I thought I had shut him out once, but I was wrong. The best solution […]