Till last year I was a happy user of uBlock Origin. A browser extension which blocks all the ads, trackers and malware sites. But then I found out about Pi-hole. At that time, I didn’t have the right hardware to install it, so I kept bugging my sister to gift me a Raspberry Pi. After few weeks of crying & begging I finally got it and immediately put it to use by installing the Pi-hole. To quote their website, Pi-hole is
A black hole for Internet advertisements.
Network-wide ad blocking via self-owned Linux hardware. The Pi-hole® is a DNS sinkhole that protects your devices from unwanted content, without installing any client-side software.
Meaning, any device connected to my Wifi gets ad-blocking without installing any browser-plugin or any other app/software. This type of network-level blocking also allows blocking ads in non-traditional places such as mobile apps and smart TVs, regardless of hardware or OS. Plus it also has a nice and informative Web interface that shows stats on all the domains being queried on the network. (Note: This can be switched off completely if needed.)
I immediately started noticing the difference, the in-app ads on my mobile were gone and I was a happy person once again :). But since Pi-hole was only installed in my local network, I was missing out on these benefits when I was outside. Many people running Pi-hole had this problem, and the solution is well documented, all I had to do was setup VPN on the raspberry pi along side the Pi-hole. But, I was being lazy & procrastinating, and never tried implementing it. But after spending a whole week out of town and out of my ad-blocking network, I finally decided to get this solution implemented. Setting up the VPN following the documentation was easy, the real challenge was to expose the raspberry pi on the internet so that the outside traffic can reach the Pi. The challenges I had were
- The Pi is behind a router, so we need to setup port forwading. My router supports this, so this was easily solved.
- The ISP (Internet Service Provider), gives dynamic IP for each connection, which changes every few hours or days.
- The ISP might be using a Carrier-grade NAT (CGN), which means you are not directly connected to the internet. Your home router is behind another bigger device called NAT controlled by the ISP, so you will never get a internet facing IP.
Dynamic DNS for the dynamic IP
For any client server communication we need the IP of the server, VPN server on raspberry pi, in my case. Normally these IP addresses don't change, so we map a domain name to that IP and the server is always accessible on that IP or domain name. Now, problem with dynamic IP is, you don’t know what’s the current IP, so you don’t know where to connect to. Since I wasn't aware about the CGN, I thought I just need to figure out the dynamic IP problem.
It is a well known situation, with a well known solution: Dynamic DNS. Basically you keep updating the domain name & IP address mapping every time the IP changes. There are multiple services which provide this free of cost. Few options which I got to know are https://www.noip.com/free, and https://www.duckdns.org/. Basic idea is you signup for their service, install a small application on your network, and that's it. You get a sub-domain like, smurfpandey.duckdns.org which is always pointing to your desired destination (my home router in my case), and this sub-domain is kept upto date via that application you install. Some routers also support this natively, so you can configure the router itself and skip the step of installing the application. I didn't choose this solution, because I wanted the sub-domain of my own, for e.g. instead of having something like my-pi-hole.duckdns.org, I wanted my-pi-hole.smurfpandey.me. Plus this is a 3rd party dependency which may or may not exist in future.
My domain(smurfpandey.me) is hosted on Cloudflare, and after a quick google search I found the related documentation on how to manage dynamic IPs for domain. They have listed multiple approaches, and I went with the 1st approach of using their API to update the DNS entry whenever my home IP changes. I wrote a python script to achieve this, you can check the Github repository to know the exact steps required to run the script yourself.
#!/usr/bin/env python3
import http.client
import json
import os
import ipaddress
import sentry_sdk
import logging
import sys
from http.client import HTTPException
from dotenv import load_dotenv
from sentry_sdk.integrations.logging import LoggingIntegration
load_dotenv()
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
CLOUDFLARE_ZONE_ID = 'a1e41feb30b74a973151948cbe32bdb5'
CLOUDFLARE_DNS_RECORD_ID = 'c69efb02e3b37ba29973edc34e1ed35a'
CLOUDFLARE_reqHeaders = {}
CLOUDFLARE_reqHeaders['Content-type'] = 'application/json'
CLOUDFLARE_reqHeaders['X-Auth-Key'] = os.environ['CLOUDFLARE_API_KEY']
CLOUDFLARE_reqHeaders['X-Auth-Email'] = os.environ['CLOUDFLARE_API_EMAIL']
SENTRY_DSN = os.environ['SENTRY_DSN']
sentry_logging = LoggingIntegration(
level=logging.INFO, # Capture info and above as breadcrumbs
event_level=logging.ERROR # Send errors as events
)
sentry_sdk.init(
dsn=SENTRY_DSN,
integrations=[sentry_logging]
)
def getMyIP():
yoConn = http.client.HTTPSConnection('api.ipify.org')
yoConn.request('GET', '/')
yoResp = yoConn.getresponse()
data = yoResp.read().decode('utf-8')
ipaddress.ip_address(data) # valdiate returned value is a valid IP address
return data
def getDNSRecord():
reqUrl = '/client/v4/zones/' + CLOUDFLARE_ZONE_ID + '/dns_records/' + CLOUDFLARE_DNS_RECORD_ID
conn = http.client.HTTPSConnection('api.cloudflare.com')
conn.request('GET', reqUrl, headers=CLOUDFLARE_reqHeaders)
response = conn.getresponse()
if response.status > 399:
raise HTTPException('Unable to get response from Cloudflare. Status: ' + str(response.status))
data = response.read().decode('utf-8')
parsed_json = json.loads(data)
if not(parsed_json['success']):
raise ValueError('Could not get success response from Cloudflare. Body: ' + data)
return parsed_json['result']
def updateDNSRecord(newIP):
reqUrl = '/client/v4/zones/' + CLOUDFLARE_ZONE_ID + '/dns_records/' + CLOUDFLARE_DNS_RECORD_ID
post_body = {
'type': 'A',
'name': 'pi.smurfpandey.me',
'content': newIP
}
json_data = json.dumps(post_body)
conn = http.client.HTTPSConnection('api.cloudflare.com')
conn.request('PUT', reqUrl, body=json_data, headers=CLOUDFLARE_reqHeaders)
response = conn.getresponse()
data = response.read().decode('utf-8')
parsed_json = json.loads(data)
logging.info('Got response from cloudflare', extra=dict(data=parsed_json, status=response.status))
if response.status > 399:
raise HTTPException('Unable to update dns record at Cloudflare.')
if not(parsed_json['success']):
raise ValueError('Could not update dns record to Cloudflare.')
return ''
# 1. Get my Ip address
myIP = getMyIP()
# 2. Get current DNS record from Cloudflare
myDNSRecord = getDNSRecord()
savedIP = myDNSRecord['content']
# 3. If current IP is same as DNS record, do nothing
if savedIP == myIP:
logging.info('No change required.', extra=dict(my_ip=myIP, saved_ip=savedIP))
else:
# 4. Update IP address in the DNS record
logging.info('Change required.', extra=dict(my_ip=myIP, saved_ip=savedIP))
updateDNSRecord(myIP)
What this script does is, it will try to find the public IP of network where it’s running using https://ipify.org. Then it will fetch the details of the domain which should point to your home. If the current IP is different than the IP mapped to the domain name, it will update that mapping. By running this script every hour, I can be assured that my-pi-hole.smurfpandey.me always points to my home.
Once I had an easy-to-remember domain name setup, next step was to setup port-forwarding on the router so that incoming requests can be forwarded back to my raspberry pi. This depends on the router you have: I was able to find out my router configuration after doing a google search with the router's model name. This is where I got to know about the 2nd problem. My ISP uses CGN, which means that the IP I get is shared by multiple consumers, and a NAT device takes care of routing the requests, similar to how we have multiple devices behind 1 router sharing the same internet connection.
I spent next hours thinking about ways to solve this, few options were:
- I can setup pi-hole on a public server on AWS, Google Cloud etc. - Easy to do, but will cost money.
- I can get a static IP from the ISP. - Easiest option, but quite expensive.
- I can setup a TCP tunnel which can expose an application running on private network to Internet. - Fun & hopefully cheap.
TCP Tunnel
I decided to take the 3rd option, it was fun & cheap. You can think of TCP tunnel as a route which connects a service running in a local network to a public network so that it can be accessed over internet. ngrok is one such application which I had used previously. I decided to use it again to expose my VPN server on the internet. After 3 simple steps and I was up & running:
- Download & unzip the binary on the pi: https://ngrok.com/download
- Connect the downloaded binary with the account. Auth token is available once you login into your account on ngrok.com:
$./ngrok authtoken <AUTH_TOKEN> - Start a tunnel to expose local VPN port(1194):
$ ./ngrok tcp 1194
After running the last command, you get an output with a domain name & port which can be used to access the VPN service running on locally in the private network.
I created a new client profile for the Open VPN server, which creates a .ovpn file. I modified the .ovpn file to update the server address as the domain name & port given by ngrok. Next step was to transfer the file to my phone, and open it in OpenVPN Connect app. And finally It worked! But this was just a proof-of-concept (POC) and my happines wasn’t long lasting. The problem is, every time you start a ngrok tunnel, you get a new port & domain, which can happen if I restart the raspberry pi. So I was back to same problem of dynamic IP. To solve this problem, ngrok has a paid plan which gives you a reserved remote address, so every time you create the tunnel, you will get the same domain & port. This plan costs $99/year, and I didn’t want to invest that much before researching for other options.
I then started looking for alternatives which are free and can be hosted by myself. I found an open-source application called frp. This is similar to ngrok, and can be self hosted. It has 2 parts, one is called frps , a server side application which should be installed on a publicly accessible server, another is frpc, a client side application which is installed on the device/server running in the private network. I already have a server running on Google Cloud free tier, so I decided to give it a try. The documentation is well written, and I was able to get it running quickly. And this is how I finally had pi-hole on the go, blocking ads no matter where I am :). There is no additional cost I have to bear and I got to learn about lot of things while making it work.
Another benefit is I can now share this with my friends & family and they can also get ad-blocking without worrying about setting up pi-hole themselves.
PS: If you are also interested in this, fill this form and I will get back to you with more details.
PPS: I have already disabled logging of DNS requests to respect privacy of the users.