Someone recently shared with me how he uses a managed hosts file (no, not the famous APK Hosts File Engine) to help block ads. The project, from Github, links to a derivative project that translates the managed hosts file into bind9 zonefiles!
I have set this up on my existing dns infrastructure, and here is the redacted documentation.
Overview
The goal of this project is to provide a mechanism for the internal network to block ads, by blocking domains that are known serve ads.
Blocking domains that serve ads
Bind9 configuration is modified on both dns1 and dns2. An additional include file is added, and many zones are added that have a basically empty config. A separate whitelist file is used to prevent bind9 from trying to control domain names that it struggles with for reasons beyond the scope of this document.
Summary of files involved
- /etc/named.conf
- /etc/installed/generate-zonefile.sh
- /etc/named/named.conf.blocked
- /etc/named/db.blocked
Named config
File /etc/named.conf
includes a snippet. This is on both dns1 and dns2.
include "/etc/named/named.conf.blocked";
This line is appended at the very bottom of the file, underneath other inclusions for files in /etc/named/.
Zonefile Generator Script
This file is adapted from repo Mueller-ma, and exists only on dns1.
#!/bin/bash -e # -E makes the shell exit immediately if a command exits with a non-zero status # v 1.1 2020-08-27 bgstack15 TMPFILE_CMD=mktemp # VARS by distro distro="$( awk -F'=' '$1 == "NAME" {print $2}' /etc/os-release )" if echo "${distro}" | grep -qiE 'centos|rhel|fedora' ; then dest_zonefile=/etc/named/named.conf.blocked blockedfile=/etc/named/db.blocked service_name=named else # assume debian-like, debian|devuan|ubuntu|mint dest_zonefile=/etc/bind/named.conf.blocked blockedfile=/etc/bind/db.blocked service_name=bind9 fi # display date date # Set tempfiles # All_domains will contain all domains from all lists, but also duplicates and ones which are whitelisted all_domains="$( "${TMPFILE_CMD}" )" # Like above, but no duplicates or whitelisted URLs all_domains_uniq="$( "${TMPFILE_CMD}" )" # We don't write directly to the zonefile. Instead to this temp file and copy it to the right directory afterwards zonefile="$( "${TMPFILE_CMD}" )" # Define local black and white lists # Comment these out if you have no local files manual_blacklist="/etc/installed/dns-blacklist" manual_whitelist="/etc/installed/dns-whitelist" # change to this working directory cd /tmp # StevenBlack GitHub Hosts # Uncomment ONE line containing the filter you want to apply # See https://github.com/StevenBlack/hosts for more combinations wget -q -O StevenBlack-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts #wget -q -O StevenBlack-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews/hosts #wget -q -O StevenBlack-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/gambling/hosts #wget -q -O StevenBlack-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/porn/hosts #wget -q -O StevenBlack-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/social/hosts #wget -q -O StevenBlack-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn-social/hosts # Filter out localhost and broadcast < StevenBlack-hosts awk '/^0\.0\.0\.0/ && ! /127.0.0.1|255.255.255.255|::1/ {print $2}' >> "${all_domains}" # Add local blacklist if [ -f "${manual_blacklist}" ] then cat "${manual_blacklist}" >> "${all_domains}" fi # Filter out comments and empty lines < "${all_domains}" grep -vE '^\s*($|#)' | sort | uniq > "${all_domains_uniq}" # Apply local whitelist if [ -f "${manual_whitelist}" ] then for i in $(cat "${manual_whitelist}") do #echo i $i sed --in-place= "/$i/d" "${all_domains_uniq}" done fi # Add zone information < "${all_domains_uniq}" sed -r "s:(.*):zone \"\1\" { type master; file \"${blockedfile}\"; };:" > "${zonefile}" # Copy temp file to right directory cp -p "${zonefile}" "${dest_zonefile}" chmod 0644 "${dest_zonefile}" # Remove all tempfiles rm -f "${all_domains}" "${all_domains_uniq}" "${zonefile}" StevenBlack-hosts # Restart bind #systemctl restart "${service_name}" # For logfile printf '%s\n\n' 'done'
This script could be adapted for use in cron, but I have not done so as of the time of this writing.
Run this script, and it will automatically update the /etc/named/named.conf.blocked
file with all the zones from the upstream, managed hosts file. I have disabled the systemctl restart named section for the internal network.
Blocked zones
The above script generates file /etc/named/named.conf.blocked
which contains the zone definitions themselves. Only a brief snippet is shown here.
zone "0.0.0.0" { type master; file "/etc/named/db.blocked"; }; zone "0.0.0.0.beeglivesex.com" { type master; file "/etc/named/db.blocked"; }; zone "0.0.0.0.creative.hpyrdr.com" { type master; file "/etc/named/db.blocked"; }; zone "0.0.0.0.hpyrdr.com" { type master; file "/etc/named/db.blocked"; }; zone "000free.us" { type master; file "/etc/named/db.blocked"; };
Empty zone definition
So every zone uses the same database file, which is pretty much all empty. It must be adapted for whichever domain I am controlling.
$TTL 24h @ IN SOA dns1.ipa.example.com. dnsmaster.ipa.example.com. ( 2003052800 86400 300 604800 3600 ) @ IN NS dns1.ipa.example.com. @ IN A 0.0.0.0 * IN A 0.0.0.0
Whitelist
Unfortunately some of the entries from the upstream hosts file cause issues in bind9. By adding them to the whitelist, we do not null-route requests to these domains. I am hoping these domains are used infrequently by my traffic.
File /etc/installed/dns-whitelist
contains exactly these contents.
___id___.c.mystat-in.net philadelphia_cbslocal.us.intellitxt.com www.zgarniij_vouchher.skroc.pl zgarniij_vouchher.skroc.pl
Blacklist
An additional file could be used for my own list of domains to block, but this has not been implemented yet.
Implementation in internal network
One time setup
Generate all files on dns1. Copy to dns2 the two files /etc/named/db.blocked
and /etc/named/named.conf.blocked
.
I had to learn how which domains to exclude. After generating the initial zones list, I had to validate it and look for errors:
time /usr/sbin/named-checkconf -z /etc/named.conf 2>&1 | grep -viE '^zone.*loaded serial 200305'
Recurring use and maintenance
To generate a current zone definition from upstream, run generate-zonefiles.sh on dns1. Manually copy the /etc/named/named.conf.blocked
to dns2. Restart dns on both systems.
Ads, the leeches of the internet that we all hate, yet need to keep the free services free….
[…] the recent changes to my internal network with the goal of blocking ads by null-routing domains that are well-known to […]
[…] to take advantage of everything you already run for your home network. For me, this includes robust ad-blocking, and of course various network services not available to the […]
Have set it up on a Raspberry Pi 5, using a Docker container. Blocking almost 150000 domains takes a lot of internal memory 1,5Gb, so I am back to Pi Hole.
But thanks for you article, explains it all very good.