security sysadmin Uncategorized

Under-attack Linode resets passwords after logins leak onto web

First sustained DDoS attacks. Now password leaks. The bad news never ends for Linode, which is unfortunate, since they are a very cheap and viable alternative to AWS especially if you factor in ingress/egress traffic.

Linode’s woes continue: the server hosting biz has just run a system-wide password reset on customer accounts after two user credentials were discovered “on an external machine.”

Source: Under-attack Linode resets passwords after logins leak onto web

2016-01-06 11.22 SGT: As of now, the site is still having intermittent access.

security sysadmin

Let’s Encrypt – Entering Public Beta

Let’s Encrypt goes public beta. No more paying of ridiculous amounts for a simple SSL certificate. Yearly.

The process is still somewhat rough on the edges now. I expect it to get better when it goes 1.0. There’s another important thing to note when you’re using using certificates from Let’s Encrypt. In the interest of transparency, they publish the list of certificates issued by them. So if you’re uncomfortable about your domain appearing in a public website, you may want to reconsider.

Let’s Encrypt is a free, automated, and open certificate authority brought to you by the Internet Security Research Group (ISRG). ISRG is a California public benefit corporation, and is recognized by the IRS as a tax-exempt organization under Section 501(c)(3) of the Internal Revenue Code.

Source: Entering Public Beta


Setting up DD-WRT on D-Link DIR-868L

Just got the great looking D-Link DIR-868L free recently from a broadband package that I signed up.

It’s an amazing router that has great features and performance. It also has great hardware specs, which makes it a perfect candidate for trying custom firmware like dd-wrt or OpenWrt. My preference would be to go for OpenWrt, unfortunately at this point of writing it is not supported. So it’s on to dd-wrt.

Installation of dd-wrt firmware can be done by following this wiki. Try it at your own risk, and always have the stock firmware on hand in case it doesn’t work.

Assuming you got this far, what’s next? Packages, naturally! To do that you have to first enable JFFS at the dd-wrt Administration tab. Next, let’s install something.

root@xxxxxxxx:/jffs/tmp# ipkg update
mkdir: can't create directory '//usr/local/lib/': Read-only file system

root@xxxxxxxx:~# ipkg install nano
root@xxxxxxxx:~# nano
-sh: nano: not found

Uh oh. Turns out ipkg is broken on this firmware and a search turns up other users facing the same issue. Someone on the forums suggested opkg instead and that’s where I went. There are many forum posts, blog posts and wikis on this topic. The one that I’m using is this. However, it doesn’t work out of the box else there won’t be this blog post :-).

Following the instructions, you should reach a step that tells you to download a script and execute it. Going for the “not so brave people” approach,

root@xxxxxxxx:/jffs/tmp# wget -q -O- > optware_setup
root@xxxxxxxx:/jffs/tmp# chmod 700 optware_setup
root@xxxxxxxx:/jffs/tmp# ./optware_setup
Checking we can reach the repository...
./optware_setup: line 15: can't create /opt/usr/bin/optware_boottime: nonexistent directory
chmod: /opt/usr/bin/optware_boottime: No such file or directory
Making sure we have an initial opkg
Connecting to (
wget: server returned error: HTTP/1.1 404 Not Found
Connecting to (
wget: can't open '/opt/lib/': Read-only file system
tar: can't open 'opkg.ipk': No such file or directory
tar: can't open 'data.tar.gz': No such file or directory

Delving into the script, there are 2 problems. First, bind /opt to /jffs/opt. Then change line 32 of the script to the updated link (look up the latest link here).

root@xxxxxxxx:/jffs/tmp# mount -o bind /jffs/opt /opt
root@xxxxxxxx:/jffs/tmp# vi optware_setup
change to line 32:
`/usr/bin/wget -O opkg.ipk` \

Let’s try again.

root@xxxxxxxx:/jffs/tmp# ./optware_setup
Checking we can reach the repository...
Making sure we have an initial opkg
Connecting to (
opkg.ipk 100% |***********************************************************************************************************************| 59159 0:00:00 ETA
Connecting to ( 100% |***********************************************************************************************************************| 7274 0:00:00 ETA
Creating the opkg config file in /opt/etc/opkg
You are now ready to install packages using opkg (this session only).
I've installed a script, optware_boottime, to run on boot and make the opkg settings persistent.
I'll add this to the end of rc_startup in nvram for you.
Updated list of available packages in var/opkg-lists/chaos_calmer_base.
Updated list of available packages in var/opkg-lists/chaos_calmer_packages.
Updated list of available packages in var/opkg-lists/chaos_calmer_routing.
Updated list of available packages in var/opkg-lists/chaos_calmer_telephony.
Minimal setup is complete. You should now have a working opkg.
We have created some aliases in your ~/.profile to make everything work.
Please either 'source .profile' or LOG OUT and LOG IN AGAIN before proceeding.


PS: Note that you’ll need to add /jffs/opt to your fstab or something in order to mount /opt on startup.
Disclaimer: I’m a vim user. nano is just an example 🙂

security sysadmin

China’s Man-on-the-Side Attack on GitHub – NETRESEC Blog

Very good analysis of the current DDoS attack that GitHub is facing, apparently over the hosting of and, which is used to bypass censorship in China.

China's Man-on-the-Side Attack on GitHub – NETRESEC Blog.


How and Why Swiftype Moved from EC2 to Real Hardware – High Scalability –

The hard truths – cloud is not always the answer.

Great comment from HN:

The reason why it is extremely hard to engineer robust large scale AWS cloud apps can be summarized under the umbrella of performance variance:

– machine latency varies more, you can’t control it
– network latency varies more
– storage latency varies more (S3, Redshift, etc.)
– machine outages are more frequent

How and Why Swiftype Moved from EC2 to Real Hardware – High Scalability –.


How PAPER Magazine’s web engineers scaled Kim Kardashian’s back-end (SFW) — The Message — Medium

I knew about Gluster File system, but it’s the first time I heard of Bees with Machine Guns! This article provides an insider’s view on how an online magazine company scale up their back-end to prepare for Kim Kardashian’s backend ;-). If you are a sysadmin or web engineer I bet some parts of the article will make you smile.
1 NRRjxiTzjIFBK4UlJ3m2ww
How PAPER Magazine’s web engineers scaled Kim Kardashian’s back-end (SFW) — The Message — Medium.

sysadmin · Fear China

As known as “what to do when a poisoned China DNS is pointing at you”. · Fear China.


M1 routers misbehaving

Was doing a routine scan when I spotted an unfamiliar address on the network: Strangely arp doesn’t reveal its MAC address, which seems odd given that this is a private IP address used internally.

Traceroute reveals the truth:

> tracert

Tracing route to over a maximum of 30 hops

1 3 ms 4 ms 3 ms
2 * * * Request timed out.
3 213 ms 5 ms 5 ms []
4 3 ms 3 ms 3 ms []
5 104 ms 4 ms 5 ms []
6 88 ms 5 ms 22 ms []
7 25 ms 5 ms 14 ms

Trace complete.

It seems someone has a misconfigured or misbehaving router that’s exposing private IP addresses. Let’s hope it is not storing something incredibly important.


Announcing IPv6 Support in Singapore | DigitalOcean

Great that DigitalOcean now has IPv6 support. Another reason for moving from Amazon. Not to mention that it’s cheaper in most usage.

Announcing IPv6 Support in Singapore | DigitalOcean.

diy sysadmin

systemcall dot org » Trashing Chromebooks

An article on repurposing Chromebooks as build agents. The more interesting parts of the article are on hardware related issues such as overheating, comparison of hobby-grade hardware vs product hardware etc.

Server rack of Chromebooks

systemcall dot org » Trashing Chromebooks.