Amazon AWS S3 outage is breaking things for a lot of websites and apps

One of Amazon AWS service – specifically S3 – goes down (and recovers eventually) but many sites are affected. It’s not as bad as the Dyn DDoS attack but it’s a reminder how many companies now rely on Amazon to power their services.

Amazon’s S3 web-based storage service is experiencing widespread issues, leading to service that’s either partially or fully broken on websites, apps and..

Source: Amazon AWS S3 outage is breaking things for a lot of websites and apps

Edit (2017-03-03): Amazon released a summary of what happened. The tl;dr version is this: fat-fingered engineer.

programming sysadmin

AWS and Azure in Plain English

If you are not an architect-level user of AWS you will probably be lost in the ever growing list of AWS services. The non-obvious names (Cognito, Athena, anyone?) for the services doesn’t help. Now someone is attempting to provide a – sometimes tongue-in-cheek – explanation of those services. Well, it’s not exactly plain english, but good attempt anyway. An Azure version is also available.

  1. AWS in Plain English
  2. Azure in Plain English

Visual Studio Code September 2016 1.6

LOL. Microsoft accidentally “DDoS” in the new VS Code release. This is due to a feature in VS Code 1.7 sending a lot of non-cacheable requests. Thankfully they responded quickly and reverted VS Code to the old version.

These typings files drive the IntelliSense (code completions) experience in VS Code. The feature was so great that we started to overload the service.

Source: Visual Studio Code September 2016 1.6

security sysadmin

Strange Loop – IP Spoofing

A very clear explanation to the DDoS problem that has been plaguing the Internet and recent advances in DDoS techniques.

The internet was originally created as a collection of equal connected peers. Everyone connected had equal rights, could consume content, produce content.

It was normal to host DNS or HTTP services on your home land-line.

But this is not possible anymore. It’s just too easy to knock unprotected websites off line.

Strange Loop – IP Spoofing


No POST after rm -rf / / Kernel & Hardware / Arch Linux Forums

This is pretty serious. Someone just bricked his laptop by executing a rm -rf / command as root in Linux. As in destroyed. None of your usual BIOS/UEFI prompts appear and no hotkeys can help to restore the firmware.

The root (pardon the pun) of the problem appears to be traced to the mounting of /sys/firmware/efi/efivars as rw by systemd. systemd maintainer (Poettering) refuses to fix the issue.


Source: No POST after rm -rf / / Kernel & Hardware / Arch Linux Forums

security sysadmin Uncategorized

Under-attack Linode resets passwords after logins leak onto web

First sustained DDoS attacks. Now password leaks. The bad news never ends for Linode, which is unfortunate, since they are a very cheap and viable alternative to AWS especially if you factor in ingress/egress traffic.

Linode’s woes continue: the server hosting biz has just run a system-wide password reset on customer accounts after two user credentials were discovered “on an external machine.”

Source: Under-attack Linode resets passwords after logins leak onto web

2016-01-06 11.22 SGT: As of now, the site is still having intermittent access.

security sysadmin

Let’s Encrypt – Entering Public Beta

Let’s Encrypt goes public beta. No more paying of ridiculous amounts for a simple SSL certificate. Yearly.

The process is still somewhat rough on the edges now. I expect it to get better when it goes 1.0. There’s another important thing to note when you’re using using certificates from Let’s Encrypt. In the interest of transparency, they publish the list of certificates issued by them. So if you’re uncomfortable about your domain appearing in a public website, you may want to reconsider.

Let’s Encrypt is a free, automated, and open certificate authority brought to you by the Internet Security Research Group (ISRG). ISRG is a California public benefit corporation, and is recognized by the IRS as a tax-exempt organization under Section 501(c)(3) of the Internal Revenue Code.

Source: Entering Public Beta


Setting up DD-WRT on D-Link DIR-868L

Just got the great looking D-Link DIR-868L free recently from a broadband package that I signed up.

It’s an amazing router that has great features and performance. It also has great hardware specs, which makes it a perfect candidate for trying custom firmware like dd-wrt or OpenWrt. My preference would be to go for OpenWrt, unfortunately at this point of writing it is not supported. So it’s on to dd-wrt.

Installation of dd-wrt firmware can be done by following this wiki. Try it at your own risk, and always have the stock firmware on hand in case it doesn’t work.

Assuming you got this far, what’s next? Packages, naturally! To do that you have to first enable JFFS at the dd-wrt Administration tab. Next, let’s install something.

root@xxxxxxxx:/jffs/tmp# ipkg update
mkdir: can't create directory '//usr/local/lib/': Read-only file system

root@xxxxxxxx:~# ipkg install nano
root@xxxxxxxx:~# nano
-sh: nano: not found

Uh oh. Turns out ipkg is broken on this firmware and a search turns up other users facing the same issue. Someone on the forums suggested opkg instead and that’s where I went. There are many forum posts, blog posts and wikis on this topic. The one that I’m using is this. However, it doesn’t work out of the box else there won’t be this blog post :-).

Following the instructions, you should reach a step that tells you to download a script and execute it. Going for the “not so brave people” approach,

root@xxxxxxxx:/jffs/tmp# wget -q -O- > optware_setup
root@xxxxxxxx:/jffs/tmp# chmod 700 optware_setup
root@xxxxxxxx:/jffs/tmp# ./optware_setup
Checking we can reach the repository...
./optware_setup: line 15: can't create /opt/usr/bin/optware_boottime: nonexistent directory
chmod: /opt/usr/bin/optware_boottime: No such file or directory
Making sure we have an initial opkg
Connecting to (
wget: server returned error: HTTP/1.1 404 Not Found
Connecting to (
wget: can't open '/opt/lib/': Read-only file system
tar: can't open 'opkg.ipk': No such file or directory
tar: can't open 'data.tar.gz': No such file or directory

Delving into the script, there are 2 problems. First, bind /opt to /jffs/opt. Then change line 32 of the script to the updated link (look up the latest link here).

root@xxxxxxxx:/jffs/tmp# mount -o bind /jffs/opt /opt
root@xxxxxxxx:/jffs/tmp# vi optware_setup
change to line 32:
`/usr/bin/wget -O opkg.ipk` \

Let’s try again.

root@xxxxxxxx:/jffs/tmp# ./optware_setup
Checking we can reach the repository...
Making sure we have an initial opkg
Connecting to (
opkg.ipk 100% |***********************************************************************************************************************| 59159 0:00:00 ETA
Connecting to ( 100% |***********************************************************************************************************************| 7274 0:00:00 ETA
Creating the opkg config file in /opt/etc/opkg
You are now ready to install packages using opkg (this session only).
I've installed a script, optware_boottime, to run on boot and make the opkg settings persistent.
I'll add this to the end of rc_startup in nvram for you.
Updated list of available packages in var/opkg-lists/chaos_calmer_base.
Updated list of available packages in var/opkg-lists/chaos_calmer_packages.
Updated list of available packages in var/opkg-lists/chaos_calmer_routing.
Updated list of available packages in var/opkg-lists/chaos_calmer_telephony.
Minimal setup is complete. You should now have a working opkg.
We have created some aliases in your ~/.profile to make everything work.
Please either 'source .profile' or LOG OUT and LOG IN AGAIN before proceeding.


PS: Note that you’ll need to add /jffs/opt to your fstab or something in order to mount /opt on startup.
Disclaimer: I’m a vim user. nano is just an example 🙂

security sysadmin

China’s Man-on-the-Side Attack on GitHub – NETRESEC Blog

Very good analysis of the current DDoS attack that GitHub is facing, apparently over the hosting of and, which is used to bypass censorship in China.

China's Man-on-the-Side Attack on GitHub – NETRESEC Blog.


How and Why Swiftype Moved from EC2 to Real Hardware – High Scalability –

The hard truths – cloud is not always the answer.

Great comment from HN:

The reason why it is extremely hard to engineer robust large scale AWS cloud apps can be summarized under the umbrella of performance variance:

– machine latency varies more, you can’t control it
– network latency varies more
– storage latency varies more (S3, Redshift, etc.)
– machine outages are more frequent

How and Why Swiftype Moved from EC2 to Real Hardware – High Scalability –.