Blocking YouTube Ads and Why It’s a Losing Battle
Back Story
I’ve been for quite some time been bombarded by cringeworthy ads like Ark Knights, Hero Wars, and these guys. This is a problem hounding YouTube and Facebook users for awhile. The more democratized advertising has become, so does the dip in quality. If every indie ad has the same quality as RAID: Shadow Legends, all’s good. Yet if you see ads like Gardenscapes which has nothing to do with the actual game, you really have to question the ethics of these companies. Worse, YouTube and Facebook has been condoning this practice for years despite how misleading (and annoying) they are.
The lack of quality control on these ads had led me to investigate how I could get rid of them myself without using third party ad block solutions.
SPOILER ALERT: I GOT TRASHED. I’ll tell you later. Read on.
A Few Ways to Block Ads
Option 1: Browser Extensions
Firstly, you have browser extensions like uBlock. While this may be useful for browsers, you will continue to see ads on phones and TVs. Moreover, ads you see on apps will continue to show up. Hence, I didn’t see this as a viable option.
Option 2: VPN
A second way to block ad is through the use of VPN like Luna. How Luna and other VPN solutions work is that they block ads from their servers, usually by filtering through DNS requests. VPNs however, have their own kinks. A research paper published a few years back has shown that as much as 38% of VPN providers inject malware and as much as 16% gets traffic forwarded elsewhere. VPNs, especially free services has all the incentives to sell your data. While there are many legitimate VPNs out there, adding this extra layer to block ads is not my cup of tea. Moreover, I have to be connected to them to have my ads blocked, which slows down my entire browsing experience. This makes the extra overhead of having a VPN counterproductive. I could do my own DNS filtering with Pi-Hole, but that requires me to purchase another hardware for this purpose. So I opted for another solution.
Option 3: Modifying the /etc/hosts file
Under the UNIX file system is a file called hosts. The file’s main purpose is to translate human-friendly hostnames (such as www.yahoo.com) into IP addresses.
Here’s an example of what a hosts file looks like:
What this generally means is that if I type router.asus.com on my browser URL bar, it will redirect me to 192.168.1.1.
For ad blocking, this is my solution of choice. If I want to block ads from say DoubleClick (which is the biggest Ad Exchange in the world), all I have to do is to redirect their hostname to some bogus IP. For example, I could do this:
What this does is that the ads algorithm attempts to look at 127.0.0.1 to pull out an ad, finds nothing, and then proceeds to report a failure. Now we’re up to something.
All I have to do is to get a good list of hostnames and throw it into my hosts file. So I took one from AdAway and added a few entries of my own. Other more sophisticated services run a cron job to aggregate the hosts file from multiple sources.
Now the next question is how to put this onto my ASUS router.
It turns out that ASUS routers do not allow the hosts file to be modified. How they do this is that they have a built in hosts file which, as part of the router boot up process, is copied over to/tmp/etc/hosts. Moreover, since the hosts file is already loaded onto the router memory, changing it has zero effect. I need to find a way to modify that hosts file.
I was initially thinking to see if there’s anything I could do to break into the ASUS firmware and modify the hosts file. While searching for a solution, I stumbled upon ASUS-merlin.
Apparently, one way to enjoy better software is to piss off engineers enough for them to write their own Open Source solutions. This is one of those.
ASUS-merlin has this folder called JFFS, which is a writable area of the firmware. The JFFS is where I can put in my custom hosts file (under /jffs/configs/hosts). What the firmware does is that when the router reboots, it pulls in the hosts file from the JFFS folder and applies it to /tmp/etc/hosts.
In order for me to upload my hosts file there, I need to enable SSH on my router, which can be found under Administration > System.
Next, all I have to do is to SSH to the router and upload the hosts file. I would then have to reboot my router for the changes to apply. By the way, if you’re interested about the hosts file I loaded onto my router, you can get it here.
After rebooting, I need to validate that my new hosts file is now active. To do so, I should see this in the System Log of my router.
What better website to test it out apart from the COVID-19 dashboard? They definitely earn a lot of ad revenue from all their banner ads. Now as you can see, I’ve successfully blocked it.
Now the next step is to test this against YouTube… and to my horror:
This means that I missed something. So I examined the payload I received from YouTube to find out why ads are still shown. This is where I’m getting the ad: https://r4 — -sn-hvcpaxvox-npok.googlevideo.com. Now the one might think that the way to do it is to just add this url into the hosts file, RIGHT??? How difficult can that be?
Well, more difficult. Here’s what my hosts file looks like.
It turns out that YouTube created an automation to keep changing their hostnames. So much so that Diversion has a cron job to keep getting a fresh list of hostnames to block them. In fact, even ad block providers run into their own issues where ads slip through. This in essence is a cat and mouse game between YouTube and the Ad Block services. This is quite smart as the hosts file isn’t designed to have wildcards as it is only supposed to map an IP address to a hostname. So I cannot redirect everything that starts with r4 — - to 127.0.0.1. With this endless list of subdomains Google can create, they can effectively run an automation such that should one of them reach a certain failure threshold, they spawn a new subdomain and use that for ads, thereby making it unscalable for the hosts file solution to block their ads.
Of course, there are other more sophisticated solutions out there than what I did for myself such as Diversion or Pi-Hole (both of which are also playing cat and mouse game with Google). But then that begs the question: Why not just buy YouTube Premium?
Well… that’s because I’m Asian. Asians are so stingy that they always think a looming zombie apocalypse is within the realm of possibilities. So much so that an entire research paper is published around Digital Piracy in Asian Countries. On the other hand, this was quite an interesting few hours I spent playing around with my router and learning how YouTube and the ad blockers are playing the arms race.
Until better quality control and regulation is imposed on online ads, this will be a constant stream of tug and war between YouTube, Facebook, and the ad blockers. In essence, we are all wasting precious resource on something that could have served the broader public: A more proactive approach in addressing inappropriate ads.
