Year of the Dragon

In the spirit of the year of the dragon, I made 10 images using SDXL featuring dragons in different styles. Feel free to use it for any purpose.

Which is your favourite?

cloud programming sysadmin

How I reduced a WordPress database size by 85% and memory consumption by 20x

I was helping a friend to troubleshoot their e-commerce site. It was running on WordPress using WooCommerce as the e-commerce backend. Like most WordPress sites, it was installed with a ton of plugins. My friend complained that the site performance has been getting slower and slower, to the point where a page load can take anywhere from 2-3 seconds to a failing to load at all. Getting to wp-admin also took forever.

At first, there are a lot of pieces to unravel, since the cause might be anything. The backend was running on AWS. The WordPress site is running as a docker container on the EC2, while the database is running on a RDS instance. It uses Cloudflare tunnel to connect the public hostname to the docker container. Seems like a decent setup.

While I do use WordPress (this site runs on WordPress), I am not a WordPress developer so I was not familiar with where things might go wrong. My first intuition was to check the plugins, since not all WordPress plugins are well written and some are notorious for taking up a lot of resources. Unfortunately isolating plugin resource usage by instrumentation was not possible as far as I know, due to the way WordPress/PHP works. After comparing the set of plugins with another site which did not exhibit the same behaviour, I decided to try other approaches.

I tried the usual tricks, like enabling proxying in Cloudflare, using a caching plugin, upping the EC2 instance size and RDS instance size. I even added a robot.txt to prevent bots from crawling the site for the time being. Those tricks helped a little, but did not resolve the problem.

Using docker stats, I noticed that CPU and memory usage is extremely high for the container, compared to others. CPU consumption is often >100% with every page load and memory usage spiked to 14GB after a while. Another unusual sign is the size of the database. For a site with around 500 products, the database size is >600MB.

That is when I chanced upon this article when searching for the symptoms.

The problem WordPress sites can run into is when there is a large amount of autoloaded data in the wp_options table.

If you return anything below 1 MB you shouldn’t be worried. However, if the result was much larger, continue on with this tutorial.

I ran the query in the article and it returned the following.

Wait. The autoload_size is ~570MB (!). I wrote a SQL command to find all the options which are larger than 1MB.

The results range from 1MB all the way to 13MB.

For the uninitiated, wp_options is akin to Windows registry, and it has become a dumping ground for plugins to store values that they might need. Most of the values in this option should be configuration values (like siteurl) which should take up just a few bytes. wp_option also has a field “autoload” which states whether the option should be loaded on every page. Storing 13MB in an option value and setting it to autoload is just insane. The total size of autoload options in the table turns out to be >500MB. Every page load is querying >500MB of data from the database and processing those data. No wonder the site is crawling!

Inspecting those options shows them most of them have the prefix _transient, which means they can be safely deleted. After making a backup of the database, I deleted all transient options. wp_options went from 556MB to 46MB, a reduction of >90%. The total database size went from 645MB to 84MB, a reduction of >85%. Memory consumption also dropped by 20x (from ~14GB to ~700MB). More importantly, the site is now super fast which is extremely important for an e-commerce site.

The results are very telling from the RDS dashboard.

Average CPU utilization has dropped to <3% and average database connections is now near zero.

Aside from noticeable performance boost for the site – average page loads within 1s – another bonus from these optimizations is that we can now use smaller EC2 and RDS instance types for better cost savings. Hopefully this article is useful as a reference for others in similar situations.


Slashing Data Transfer Costs in AWS by 99%

Everyone knows how egress (outgoing traffic) is not free in AWS. This is a “hack” to save on egress cost when transferring traffic between AWS regions. And yes, you can achieve savings of 99% when using this method.

There are lots of ways to accidentally spend too much money on AWS, and one of the easiest ways is by carelessly transferring data. As of writing, AWS charges the following rates for data transfer: Data transfer from AWS to the public Internet ranges from $0.09/GB in us-east-1 (N. Virginia) to $0.154/GB in af-south-1 (Cape Town). Therefore a single terabyte of data transfer will run you a cool $90 – $154.

Source: Slashing Data Transfer Costs in AWS by 99%


“Apple patches 17th zero-day of 2023”

Some people are alarmed when they read headlines like these. They may be wondering why Apple’s devices seem to be plagued by so many “security issues.” In fact, if you compare the number of CVEs (which, in layman’s terms, are security bugs) for Apple iOS versus Android, you will find that Android fares much worse in this aspect.

Google » Android : Vulnerability Statistics

Apple » Iphone Os : Vulnerability Statistics

Android has 429 vulnerabilities reported for 2023 as of today, compared to 38 for Apple iOS – more than 10 times as much.

The reality is that all complex software is prone to bugs, and these bugs may or may not be exploitable. Further complicating the issue is that software is not a monolith; rather, it’s composed of numerous parts that are constantly changing due to upgrades, bug fixes, and other developments.

I’ve often said that maintaining running software is like paying a tax, even if “the specs are frozen” and “nothing is changed.” The fact is, things are constantly changing in the software world. New vulnerabilities are discovered in code or libraries, operating system updates roll out regularly, and the threat landscape evolves continuously.

So, why does it seem like Apple is frequently in the spotlight when it comes to security vulnerabilities? There are several reasons for this perception:

  • Popularity and Visibility: Apple’s products, especially iPhones and Macs, are immensely popular worldwide. With a large user base, any security issue that does arise tends to receive significant media attention.
  • Intensive Scrutiny: Apple’s closed ecosystem and stringent control over its hardware and software mean that security researchers and hackers alike often target the company’s products. The more scrutiny a system undergoes, the more vulnerabilities are likely to be discovered.
  • Responsiveness: Apple takes security seriously and is quick to release patches and updates to address vulnerabilities when they are discovered. While this is a proactive approach, it also means that security issues might come to light more frequently.
  • Zero-Day Vulnerabilities: Some vulnerabilities are so new and unexploited that they are termed “zero-day vulnerabilities.” These are often discovered in various software systems, including Apple’s. However, Apple’s high-profile status means that these vulnerabilities gain significant attention.
  • User Expectations: Users of Apple products often have high expectations when it comes to security. Any perceived lapse or vulnerability can generate headlines and discussions.

In reality, all major operating systems, including iOS, Android, Windows, and macOS, face security challenges. The key is how these companies respond to these challenges and their ability to provide timely security updates to protect their users.

To stay safe in the digital age, it’s crucial to keep your devices and software up to date with the latest security patches. Additionally, practicing good cybersecurity habits, such as using strong, unique passwords, enabling two-factor authentication, and being cautious about the apps you download and the websites you visit, can go a long way in protecting your digital life. As technology continues to advance, so do the efforts of those seeking to exploit it. By staying informed and taking proactive security measures, we can all play a role in mitigating the risks associated with our ever-evolving digital landscape.

Source: Apple patches 17th zero-day of 2023


Announcing Amazon Managed Service for Apache Flink Renamed from Amazon Kinesis Data Analytics | AWS News Blog

It seems like AWS is renaming some of their services to refer to the underlying open-source software by name. This makes sense when AWS is just running the underlying software for the customer without too much changes, like Amazon Managed Grafana, Amazon Managed Streaming for Apache Kafka.

Today we are announcing the rename of Amazon Kinesis Data Analytics to Amazon Managed Service for Apache Flink, a fully managed and serverless service for you to build and run real-time streaming applications using Apache Flink.


Introducing the 100-Year Plan: Secure Your Online Legacy for a Century – News

Do you want to leave your digital content behind for a long time? Like a really long time? WordPress launched a 100-year plan just for that and it will cost you USD 38,000.

I don’t know how to feel about WordPress’s latest offering. On one hand, it seems like a convenient way to leave your digital content behind for at least 100 years. On the other hand, there are so many problems with the proposition. Will WordPress still be around in 100 years? 50 years? And what form the “web” will take? Will we still have written content? Is going online still going to be a thing? We have no idea and it seems almost silly to imagine things will be the same for such a long time.

An exceptional new plan for those who want to secure their online legacy for a lifetime—and then some.

Source: Introducing the 100-Year Plan: Secure Your Online Legacy for a Century – News


“Amazon accounts”

As a long time Amazon and AWS user, I have accumulated more than a few Amazon-related accounts. Recently I also had to work with other colleagues who are not so familiar with the Amazon services and accounts ecosystem. Here is an attempt to make sense of it all:

Amazon Prime
CloudAWS ConsoleY
EducationAWS AcademyY
TrainingAWS Training and CertificationYYY
TrainingSkill BuilderYYY
Mapping of services to accounts

Hopefully it helps someone who’s figuring out which account to login to which service.


Viewer Feature: Selective Loading | Autodesk Platform Services

Autodesk has recently launched the latest version of APS Viewer, previously known as Autodesk Forge Viewer, introducing an impressive feature called selective loading. This feature addresses one of the major challenges in managing BIM, which is the immense size of the model. In practical construction projects, an Autodesk Revit file can easily reach terabytes in size. Even after converting the model into SVF/SVF2 format, the data volume that needs to be transmitted to the client remains in the range of hundreds of megabytes or gigabytes. With selective loading, users now have the ability to filter and display only the specific parts of the model they wish to see. This filtering occurs on the server side, resulting in a significant reduction in the amount of data transmitted.

Using this feature you can improve the performance of your application since the model loading time will be significantly faster, or you can implement “saved views” feature, allowing your users to load just a subset of their designs based on previously stored filters.​

Source: Viewer Feature: Selective Loading | Autodesk Platform Services


A Man Sued Avianca Airline. His Lawyer Used ChatGPT. – The New York Times

This is what happens when somebody uses ChatGPT as if it’s a search engine. People are so used to precise and deterministic output from programs that it’s hard for them to imagine one that not only fabricates truths, but also does so convincingly.

The lawyer who created the brief, Steven A. Schwartz of the firm Levidow, Levidow & Oberman, threw himself on the mercy of the court on Thursday, saying in an affidavit that he had used the artificial intelligence program to do his legal research — “a source that has revealed itself to be unreliable.”

Source: A Man Sued Avianca Airline. His Lawyer Used ChatGPT. – The New York Times


ChatGPT Prompt Engineering for Developers – DeepLearning.AI

For a limited time only, this free course by Isa Fulford and Andrew Ng (Coursera,, called ChatGPT Prompt Engineering for Developers, is available for anyone looking to expand their development skills. The course is an excellent opportunity for developers who want to learn how to use a large language model (LLM) to create powerful applications in a cost-effective and time-efficient way.

Throughout the course, Isa Fulford and Andrew Ng explain the workings of LLMs and provide best practices for prompt engineering. You’ll be able to learn how to use the OpenAI API to build capabilities that can automatically summarize user reviews, classify sentiment, extract topics, translate text, and even write emails. Additionally, you’ll learn how to build a custom chatbot and use two key principles for writing effective prompts.

What I appreciate about this course is the hands-on experience provided in the Jupyter notebook environment. You’ll be able to play with numerous examples and systematically engineer good prompts. This makes it easy to put the concepts learned in the course into practice in your own projects.

So, if you’re looking for an opportunity to upskill and learn how to build innovative applications that were once impossible or highly technical, I highly recommend taking this course. Don’t miss out on the chance to learn from experts and expand your skill set for free.

ChatGPT Prompt Engineering for Developers is beginner-friendly. Only a basic understanding of Python is needed. But it is also suitable for advanced machine learning engineers wanting to approach the cutting-edge of prompt engineering and use LLMs.

Source: ChatGPT Prompt Engineering for Developers – DeepLearning.AI