Use compression (if available) when downloading the ad lists.

Signed-off-by: DL6ER <dl6er@dl6er.de>

README.md

@@ -17,12 +17,12 @@ The Pi-hole[®](https://pi-hole.net/trademark-rules-and-brand-guidelines/) is a
 - **Free**: open source software which helps ensure _you_ are the sole person in control of your privacy
 
 -----
-<a href="https://www.codacy.com/app/Pi-hole/pi-hole?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=pi-hole/pi-hole&amp;utm_campaign=Badge_Grade"><img src="https://api.codacy.com/project/badge/Grade/c558a0f8d7124c99b02b84f0f5564238" alt="Codacy Grade"/></a>
-<a href="https://travis-ci.org/pi-hole/pi-hole"><img src="https://travis-ci.org/pi-hole/pi-hole.svg?branch=development" alt="Travis Build Status"/></a>
-<a href="https://www.bountysource.com/trackers/3011939-pi-hole-pi-hole?utm_source=3011939&utm_medium=shield&utm_campaign=TRACKER_BADGE"><img src="https://www.bountysource.com/badge/tracker?tracker_id=3011939" alt="BountySource"/></a>
+[![Codacy Badge](https://api.codacy.com/project/badge/Grade/c558a0f8d7124c99b02b84f0f5564238)](https://www.codacy.com/app/Pi-hole/pi-hole?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=pi-hole/pi-hole&amp;utm_campaign=Badge_Grade)
+[![Build Status](https://travis-ci.org/pi-hole/pi-hole.svg?branch=development)](https://travis-ci.org/pi-hole/pi-hole)
+[![BountySource](https://www.bountysource.com/badge/tracker?tracker_id=3011939)](https://www.bountysource.com/trackers/3011939-pi-hole-pi-hole?utm_source=3011939&utm_medium=shield&utm_campaign=TRACKER_BADGE)
 
 ## One-Step Automated Install
-Those who want to get started quickly and conveniently, may install Pi-hole using the following command:
+Those who want to get started quickly and conveniently may install Pi-hole using the following command:
 
 #### `curl -sSL https://install.pi-hole.net | bash`
 
@@ -46,14 +46,14 @@ sudo bash basic-install.sh
 
 Once the installer has been run, you will need to [configure your router to have **DHCP clients use Pi-hole as their DNS server**](https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245) which ensures that all devices connecting to your network will have content blocked without any further intervention.
 
-If your router does not support setting the DNS server, you can [use Pi-hole's built in DHCP server](https://discourse.pi-hole.net/t/how-do-i-use-pi-holes-built-in-dhcp-server-and-why-would-i-want-to/3026); just be sure to disable DHCP on your router first (if it has that feature available).
+If your router does not support setting the DNS server, you can [use Pi-hole's built-in DHCP server](https://discourse.pi-hole.net/t/how-do-i-use-pi-holes-built-in-dhcp-server-and-why-would-i-want-to/3026); just be sure to disable DHCP on your router first (if it has that feature available).
 
 As a last resort, you can always manually set each device to use Pi-hole as their DNS server.
 
 -----
 
 ## Pi-hole is free, but powered by your support
-There are many reoccurring costs involved with maintaining free, open source, and privacy respecting software; expenses which [our volunteer developers](https://github.com/orgs/pi-hole/people) pitch in to cover out-of-pocket. This is just one example of how strongly we feel about our software, as well as the importance of keeping it maintained.
+There are many reoccurring costs involved with maintaining free, open source, and privacy-respecting software; expenses which [our volunteer developers](https://github.com/orgs/pi-hole/people) pitch in to cover out-of-pocket. This is just one example of how strongly we feel about our software, as well as the importance of keeping it maintained.
 
 Make no mistake: **your support is absolutely vital to help keep us innovating!**
 
@@ -61,16 +61,13 @@ Make no mistake: **your support is absolutely vital to help keep us innovating!*
 Sending a donation using our links below is **extremely helpful** in offsetting a portion of our monthly expenses:
 
 - <img src="https://pi-hole.github.io/graphics/Badges/paypal-badge-black.svg" width="24" height="24" alt="PP"/> <a href="https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY">Donate via PayPal</a><br/>
-- <img src="https://pi-hole.github.io/graphics/Badges/bitcoin-badge-black.svg" width="24" height="24" alt="BTC"/> [Bitcoin](https://commerce.coinbase.com/checkout/fb7facaf-bebd-46be-bb77-b358f4546763): <code>
-3MDPzjXu2hjw5sGLJvKUi1uXbvQPzVrbpF</code></br>
-- <img src="https://pi-hole.github.io/graphics/Badges/bitcoin-badge-black.svg" width="24" height="24" alt="BTC"/> [Bitcoin Cash](https://commerce.coinbase.com/checkout/fb7facaf-bebd-46be-bb77-b358f4546763): <code>qzqsz4aju2eecc6uhs7tus4vlwhhela24sdruf4qp5</code></br>
-- <img src="https://pi-hole.github.io/graphics/Badges/ethereum-badge-black.svg" width="24" height="24" alt="BTC"/> [Ethereum](https://commerce.coinbase.com/checkout/fb7facaf-bebd-46be-bb77-b358f4546763): <code>0x79d4e90A4a0C732819526c93e21A3F1356A2FAe1</code>
+- <img src="https://pi-hole.github.io/graphics/Badges/bitcoin-badge-black.svg" width="24" height="24" alt="BTC"/> [Bitcoin, Bitcoin Cash, Ethereum, Litecoin](https://commerce.coinbase.com/checkout/dd304d04-f324-4a77-931b-0db61c77a41b)
 
 ### Alternative support
 If you'd rather not [donate](https://pi-hole.net/donate/) (_which is okay!_), there are other ways you can help support us:
 - [Patreon](https://patreon.com/pihole) _Become a patron for rewards_
 - [Digital Ocean](http://www.digitalocean.com/?refcode=344d234950e1) _affiliate link_
-- [UNIXstickers.com](http://unixstickers.refr.cc/jacobs) _save $5 when you spend $9 using our affiliate link_
+- [Stickermule](https://www.stickermule.com/unlock?ref_id=9127301701&utm_medium=link&utm_source=invite) _earn a $10 credit after your first purchase_
 - [Pi-hole Swag Store](https://pi-hole.net/shop/) _affiliate link_
 - [Amazon](http://www.amazon.com/exec/obidos/redirect-home/pihole09-20) _affiliate link_
 - [DNS Made Easy](https://cp.dnsmadeeasy.com/u/133706) _affiliate link_
@@ -82,7 +79,7 @@ We welcome _everyone_ to contribute to issue reports, suggest new features, and
 
 If you have something to add - anything from a typo through to a whole new feature, we're happy to check it out! Just make sure to fill out our template when submitting your request; the questions that it asks will help the volunteers quickly understand what you're aiming to achieve.
 
-You'll find that the [install script](https://github.com/pi-hole/pi-hole/blob/master/automated%20install/basic-install.sh) and the [debug script](https://github.com/pi-hole/pi-hole/blob/master/advanced/Scripts/piholeDebug.sh) have an abundance of comments, which will help you better understand how Pi-hole works. They're also a valuable resource to those who want to learn how to write scripts or code a program! We encourage anyone who likes to tinker to read through it, and submit a pull request for us to review.
+You'll find that the [install script](https://github.com/pi-hole/pi-hole/blob/master/automated%20install/basic-install.sh) and the [debug script](https://github.com/pi-hole/pi-hole/blob/master/advanced/Scripts/piholeDebug.sh) have an abundance of comments, which will help you better understand how Pi-hole works. They're also a valuable resource to those who want to learn how to write scripts or code a program! We encourage anyone who likes to tinker to read through it and submit a pull request for us to review.
 
 ### Presentations about Pi-hole
 Word-of-mouth continues to help our project grow immensely, and so we are helping make this easier for people.
@@ -110,7 +107,7 @@ While we are primarily reachable on our <a href="https://discourse.pi-hole.net/"
 
 ## Breakdown of Features
 ### The Command Line Interface
-The `pihole` command has all the functionality necessary to be able to fully administer the Pi-hole, without the need of the Web Interface. It's fast, user-friendly, and auditable by anyone with understanding of `bash`.
+The `pihole` command has all the functionality necessary to be able to fully administer the Pi-hole, without the need of the Web Interface. It's fast, user-friendly, and auditable by anyone with an understanding of `bash`.
 
 <a href="https://pi-hole.github.io/graphics/Screenshots/blacklist-cli.gif"><img src="https://pi-hole.github.io/graphics/Screenshots/blacklist-cli.gif" alt="Pi-hole Blacklist Demo"/></a>
 
@@ -137,7 +134,7 @@ Some notable features include:
 * Detailed graphs and doughnut charts
 * Top lists of domains and clients
 * A filterable and sortable query log
-* Long Term Statistics to view data over user defined time ranges
+* Long Term Statistics to view data over user-defined time ranges
 * The ability to easily manage and configure Pi-hole features
 * ... and all the main features of the Command Line Interface!
 
@@ -148,7 +145,7 @@ There are several ways to [access the dashboard](https://discourse.pi-hole.net/t
 3. `http://pi.hole/` (when using Pi-hole as your DNS server)
 
 ## Faster-than-light Engine
-FTLDNS[™](https://pi-hole.net/trademark-rules-and-brand-guidelines/) is a lightweight, purpose-built daemon used to provide statistics needed for the Web Interface, and its API can be easily integrated into your own projects. As the name implies, FTLDNS does this all *very quickly*!
+FTLDNS is a lightweight, purpose-built daemon used to provide statistics needed for the Web Interface, and its API can be easily integrated into your own projects. As the name implies, FTLDNS does this all *very quickly*!
 
 Some of the statistics you can integrate include:
 * Total number of domains being blocked
@@ -165,40 +162,46 @@ The API can be accessed via [`telnet`](https://github.com/pi-hole/FTL), the Web
 -----
 
 ## The Origin Of Pi-hole
-Pi-hole being a **advertising-aware DNS/Web server**, makes use of the following technologies:
+Pi-hole being an **advertising-aware DNS/Web server**, makes use of the following technologies:
 
 * [`dnsmasq`](http://www.thekelleys.org.uk/dnsmasq/doc.html) - a lightweight DNS and DHCP server
 * [`curl`](https://curl.haxx.se) - A command line tool for transferring data with URL syntax
-* [`lighttpd`](https://www.lighttpd.net) - webserver designed and optimized for high performance
+* [`lighttpd`](https://www.lighttpd.net) - web server designed and optimized for high performance
 * [`php`](https://secure.php.net) - a popular general-purpose web scripting language
 * [AdminLTE Dashboard](https://github.com/almasaeed2010/AdminLTE) - premium admin control panel based on Bootstrap 3.x
 
-While quite outdated at this point, [this original blog post about Pi-hole](https://jacobsalmela.com/2015/06/16/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0/) goes into **great detail** about how Pi-hole was originally setup and how it works. Syntactically, it's no longer accurate, but the same basic principles and logic still apply to Pi-hole's current state.
+While quite outdated at this point, [this original blog post about Pi-hole](https://jacobsalmela.com/2015/06/16/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0/) goes into **great detail** about how Pi-hole was originally set up and how it works. Syntactically, it's no longer accurate, but the same basic principles and logic still apply to Pi-hole's current state.
+
 -----
 
 ## Coverage
-- [Software Engineering Daily: Interview with the creator of Pi-hole](https://softwareengineeringdaily.com/2018/05/29/pi-hole-ad-blocker-hardware-with-jacob-salmela/)
-- [Bloomberg Business Week: Brotherhood of the Ad blockers](https://www.bloomberg.com/news/features/2018-05-10/inside-the-brotherhood-of-pi-hole-ad-blockers)
-- [Securing DNS across all of my devices with Pi-Hole + DNS-over-HTTPS + 1.1.1.1](https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/)
-- [Adafruit: installing Pi-hole on a Pi Zero W](https://learn.adafruit.com/pi-hole-ad-blocker-with-pi-zero-w/install-pi-hole)
-- [Lifehacker: Turn A Raspberry Pi Into An Ad Blocker With A Single Command](https://www.lifehacker.com.au/2015/02/turn-a-raspberry-pi-into-an-ad-blocker-with-a-single-command/)
-- [MakeUseOf: Adblock Everywhere: The Raspberry Pi-Hole Way](http://www.makeuseof.com/tag/adblock-everywhere-raspberry-pi-hole-way/)
-- [Catchpoint: Ad-Blocking on Apple iOS9: Valuing the End User Experience](http://blog.catchpoint.com/2015/09/14/ad-blocking-apple/)
-- [Security Now Netcast: Pi-hole](https://www.youtube.com/watch?v=p7-osq_y8i8&t=100m26s)
-- [TekThing: Raspberry Pi-Hole Makes Ads Disappear!](https://youtu.be/8Co59HU2gY0?t=2m)
-- [Foolish Tech Show](https://youtu.be/bYyena0I9yc?t=2m4s)
-- [Block Ads on All Home Devices for $53.18](https://medium.com/@robleathern/block-ads-on-all-home-devices-for-53-18-a5f1ec139693#.gj1xpgr5d)
-- [Pi-Hole for Ubuntu 14.04](http://www.boyter.org/2015/12/pi-hole-ubuntu-14-04/)
-- [MacObserver Podcast 585](https://www.macobserver.com/tmo/podcast/macgeekgab-585)
-- [The Defrag Show: Endoscope USB Camera, The Final [HoloLens] Vote, Adblock Pi and more](https://channel9.msdn.com/Shows/The-Defrag-Show/Defrag-Endoscope-USB-Camera-The-Final-HoloLens-Vote-Adblock-Pi-and-more?WT.mc_id=dlvr_twitter_ch9#time=20m39s)
-- [Adafruit: Pi-hole is a black hole for internet ads](https://blog.adafruit.com/2016/03/04/pi-hole-is-a-black-hole-for-internet-ads-piday-raspberrypi-raspberry_pi/)
-- [Digital Trends: 5 Fun, Easy Projects You Can Try With a $35 Raspberry Pi](https://youtu.be/QwrKlyC2kdM?t=1m42s)
-- [Adafruit: Raspberry Pi Quick Look at Pi Hole ad blocking server with Tony D](https://www.youtube.com/watch?v=eg4u2j1HYlI)
-- [Devacron: OrangePi Zero as an Ad-Block server with Pi-Hole](http://www.devacron.com/orangepi-zero-as-an-ad-block-server-with-pi-hole/)
-- [Linux Pro: The Hole Truth](http://www.linuxpromagazine.com/Issues/2017/200/The-sysadmin-s-daily-grind-Pi-hole)
-- [CryptoAUSTRALIA: How We Tried 5 Privacy Focused Raspberry Pi Projects](https://blog.cryptoaustralia.org.au/2017/10/05/5-privacy-focused-raspberry-pi-projects/)
-- [CryptoAUSTRALIA: Pi-hole Workshop](https://blog.cryptoaustralia.org.au/2017/11/02/pi-hole-network-wide-ad-blocker/)
-- [Know How 355: Killing ads with a Raspberry Pi-Hole!](https://www.twit.tv/shows/know-how/episodes/355)
+- [Lifehacker: Turn A Raspberry Pi Into An Ad Blocker With A Single Command](https://www.lifehacker.com.au/2015/02/turn-a-raspberry-pi-into-an-ad-blocker-with-a-single-command/) (Feburary, 2015)
+- [MakeUseOf: Adblock Everywhere: The Raspberry Pi-Hole Way](http://www.makeuseof.com/tag/adblock-everywhere-raspberry-pi-hole-way/) (March, 2015)
+- [Catchpoint: Ad-Blocking on Apple iOS9: Valuing the End User Experience](http://blog.catchpoint.com/2015/09/14/ad-blocking-apple/) (September, 2015)
+- [Security Now Netcast: Pi-hole](https://www.youtube.com/watch?v=p7-osq_y8i8&t=100m26s) (October, 2015)
+- [TekThing: Raspberry Pi-Hole Makes Ads Disappear!](https://youtu.be/8Co59HU2gY0?t=2m) (December, 2015)
+- [Foolish Tech Show](https://youtu.be/bYyena0I9yc?t=2m4s) (December, 2015)
+- [Block Ads on All Home Devices for $53.18](https://medium.com/@robleathern/block-ads-on-all-home-devices-for-53-18-a5f1ec139693#.gj1xpgr5d) (December, 2015)
+- [Pi-Hole for Ubuntu 14.04](http://www.boyter.org/2015/12/pi-hole-ubuntu-14-04/) (December, 2015)
+- [MacObserver Podcast 585](https://www.macobserver.com/tmo/podcast/macgeekgab-585) (December, 2015)
+- [The Defrag Show: Endoscope USB Camera, The Final [HoloLens] Vote, Adblock Pi and more](https://channel9.msdn.com/Shows/The-Defrag-Show/Defrag-Endoscope-USB-Camera-The-Final-HoloLens-Vote-Adblock-Pi-and-more?WT.mc_id=dlvr_twitter_ch9#time=20m39s) (January, 2016)
+- [Adafruit: Pi-hole is a black hole for internet ads](https://blog.adafruit.com/2016/03/04/pi-hole-is-a-black-hole-for-internet-ads-piday-raspberrypi-raspberry_pi/) (March, 2016)
+- [Digital Trends: 5 Fun, Easy Projects You Can Try With a $35 Raspberry Pi](https://youtu.be/QwrKlyC2kdM?t=1m42s) (March, 2016)
+- [Adafruit: Raspberry Pi Quick Look at Pi Hole ad blocking server with Tony D](https://www.youtube.com/watch?v=eg4u2j1HYlI) (June, 2016)
+- [Devacron: OrangePi Zero as an Ad-Block server with Pi-Hole](http://www.devacron.com/orangepi-zero-as-an-ad-block-server-with-pi-hole/) (December, 2016)
+- [Linux Pro: The Hole Truth](http://www.linuxpromagazine.com/Issues/2017/200/The-sysadmin-s-daily-grind-Pi-hole) (July, 2017)
+- [Adafruit: installing Pi-hole on a Pi Zero W](https://learn.adafruit.com/pi-hole-ad-blocker-with-pi-zero-w/install-pi-hole) (August, 2017)
+- [CryptoAUSTRALIA: How We Tried 5 Privacy Focused Raspberry Pi Projects](https://blog.cryptoaustralia.org.au/2017/10/05/5-privacy-focused-raspberry-pi-projects/) (October, 2017)
+- [CryptoAUSTRALIA: Pi-hole Workshop](https://blog.cryptoaustralia.org.au/2017/11/02/pi-hole-network-wide-ad-blocker/) (November, 2017)
+- [Know How 355: Killing ads with a Raspberry Pi-Hole!](https://www.twit.tv/shows/know-how/episodes/355) (November, 2017)
+- [Hobohouse: Block Advertising on your Network with Pi-hole and Raspberry Pi](https://hobo.house/2018/02/27/block-advertising-with-pi-hole-and-raspberry-pi/) (March, 2018)
+- [Scott Helme: Securing DNS across all of my devices with Pi-Hole + DNS-over-HTTPS + 1.1.1.1](https://scotthelme.co.uk/securing-dns-across-all-of-my-devices-with-pihole-dns-over-https-1-1-1-1/) (April, 2018)
+- [Scott Helme: Catching and dealing with naughty devices on my home network](https://scotthelme.co.uk/catching-naughty-devices-on-my-home-network/) (April, 2018)
+- [Bloomberg Business Week: Brotherhood of the Ad blockers](https://www.bloomberg.com/news/features/2018-05-10/inside-the-brotherhood-of-pi-hole-ad-blockers) (May, 2018)
+- [Software Engineering Daily: Interview with the creator of Pi-hole](https://softwareengineeringdaily.com/2018/05/29/pi-hole-ad-blocker-hardware-with-jacob-salmela/) (May, 2018)
+- [Raspberry Pi: Block ads at home using Pi-hole and a Raspberry Pi](https://www.raspberrypi.org/blog/pi-hole-raspberry-pi/) (July, 2018)
+- [Troy Hunt: Mmm... Pi-hole...](https://www.troyhunt.com/mmm-pi-hole/) (September, 2018)
+- [PEBKAK Podcast: Interview With Jacob Salmela](https://www.jerseystudios.net/2018/10/11/150-pi-hole/) (October, 2018)
 
 -----
 

advanced/01-pihole.conf

@@ -1,13 +1,11 @@
 # Pi-hole: A black hole for Internet advertisements
-# (c) 2015, 2016 by Jacob Salmela
-# Network-wide ad blocking via your Raspberry Pi
-# http://pi-hole.net
-# dnsmasq config for Pi-hole
+# (c) 2017 Pi-hole, LLC (https://pi-hole.net)
+# Network-wide ad blocking via your own hardware.
 #
-# Pi-hole is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 2 of the License, or
-# (at your option) any later version.
+# Dnsmasq config for Pi-hole's FTLDNS
+#
+# This file is copyright under the latest version of the EUPL.
+# Please see LICENSE file for your rights under this license.
 
 ###############################################################################
 #      FILE AUTOMATICALLY POPULATED BY PI-HOLE INSTALL/UPDATE PROCEDURE.      #
@@ -16,12 +14,10 @@
 #        IF YOU WISH TO CHANGE THE UPSTREAM SERVERS, CHANGE THEM IN:          #
 #                      /etc/pihole/setupVars.conf                             #
 #                                                                             #
-#        ANY OTHER CHANGES SHOULD BE MADE IN A SEPERATE CONFIG FILE           #
-#                        OR IN /etc/dnsmasq.conf                              #
+#        ANY OTHER CHANGES SHOULD BE MADE IN A SEPARATE CONFIG FILE           #
+#                    WITHIN /etc/dnsmasq.d/yourname.conf                      #
 ###############################################################################
 
-addn-hosts=/etc/pihole/gravity.list
-addn-hosts=/etc/pihole/black.list
 addn-hosts=/etc/pihole/local.list
 
 domain-needed
@@ -39,7 +35,7 @@ interface=@INT@
 
 cache-size=10000
 
-log-queries=extra
+log-queries
 log-facility=/var/log/pihole.log
 
 local-ttl=2

advanced/Scripts/chronometer.sh

@@ -8,6 +8,7 @@
 #
 # This file is copyright under the latest version of the EUPL.
 # Please see LICENSE file for your rights under this license.
+LC_ALL=C
 LC_NUMERIC=C
 
 # Retrieve stats from FTL engine
@@ -243,7 +244,7 @@ get_sys_stats() {
         disk_total="${disk_raw[1]}"
         disk_perc="${disk_raw[2]}"
 
-        net_gateway=$(route -n | awk '$4 == "UG" {print $2;exit}')
+        net_gateway=$(ip route | grep default | cut -d ' ' -f 3 | head -n 1)
 
         # Get DHCP stats, if feature is enabled
         if [[ "$DHCP_ACTIVE" == "true" ]]; then
@@ -443,6 +444,9 @@ get_strings() {
 }
 
 chronoFunc() {
+    local extra_arg="$1"
+    local extra_value="$2"
+
     get_init_stats
 
     for (( ; ; )); do
@@ -460,10 +464,8 @@ chronoFunc() {
         fi
 
         # Get refresh number
-        if [[ "$*" == *"-r"* ]]; then
-            num="$*"
-            num="${num/*-r /}"
-            num="${num/ */}"
+        if [[ "${extra_arg}" = "refresh" ]]; then
+            num="${extra_value}"
             num_str="Refresh set for every $num seconds"
         else
             num_str=""
@@ -472,13 +474,13 @@ chronoFunc() {
         clear
 
         # Remove exit message heading on third refresh
-        if [[ "$count" -le 2 ]] && [[ "$*" != *"-e"* ]]; then
+        if [[ "$count" -le 2 ]] && [[ "${extra_arg}" != "exit" ]]; then
             echo -e " ${COL_LIGHT_GREEN}Pi-hole Chronometer${COL_NC}
             $num_str
             ${COL_LIGHT_RED}Press Ctrl-C to exit${COL_NC}
             ${COL_DARK_GRAY}$scr_line_str${COL_NC}"
         else
-        echo -e "|¯¯¯(¯)_|¯|_  ___|¯|___$phc_ver_str| ¯_/¯|_| ' \\/ _ \\ / -_)$lte_ver_str|_| |_| |_||_\\___/_\\___|$ftl_ver_str ${COL_DARK_GRAY}$scr_line_str${COL_NC}"
+        echo -e "|¯¯¯(¯)_|¯|_  ___|¯|___$phc_ver_str\\n| ¯_/¯|_| ' \\/ _ \\ / -_)$lte_ver_str\\n|_| |_| |_||_\\___/_\\___|$ftl_ver_str\\n ${COL_DARK_GRAY}$scr_line_str${COL_NC}"
         fi
 
         printFunc "  Hostname: " "$sys_name" "$host_info"
@@ -520,10 +522,10 @@ chronoFunc() {
         fi
 
         # Handle exit/refresh options
-        if [[ "$*" == *"-e"* ]]; then
+        if [[ "${extra_arg}" == "exit" ]]; then
             exit 0
         else
-            if [[ "$*" == *"-r"* ]]; then
+            if [[ "${extra_arg}" == "refresh" ]]; then
                 sleep "$num"
             else
                 sleep 5
@@ -560,12 +562,10 @@ if [[ $# = 0 ]]; then
     chronoFunc
 fi
 
-for var in "$@"; do
-    case "$var" in
-        "-j" | "--json"    ) jsonFunc;;
-        "-h" | "--help"    ) helpFunc;;
-        "-r" | "--refresh" ) chronoFunc "$@";;
-        "-e" | "--exit"    ) chronoFunc "$@";;
-        *                  ) helpFunc "?";;
-    esac
-done
+case "$1" in
+    "-j" | "--json"    ) jsonFunc;;
+    "-h" | "--help"    ) helpFunc;;
+    "-r" | "--refresh" ) chronoFunc refresh "$2";;
+    "-e" | "--exit"    ) chronoFunc exit;;
+    *                  ) helpFunc "?";;
+esac

advanced/Scripts/database_migration/gravity-db.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# shellcheck disable=SC1090
+
+# Pi-hole: A black hole for Internet advertisements
+# (c) 2019 Pi-hole, LLC (https://pi-hole.net)
+# Network-wide ad blocking via your own hardware.
+#
+# Updates gravity.db database
+#
+# This file is copyright under the latest version of the EUPL.
+# Please see LICENSE file for your rights under this license.
+
+upgrade_gravityDB(){
+	local database piholeDir auditFile version
+	database="${1}"
+	piholeDir="${2}"
+	auditFile="${piholeDir}/auditlog.list"
+
+	# Get database version
+	version="$(sqlite3 "${database}" "SELECT \"value\" FROM \"info\" WHERE \"property\" = 'version';")"
+
+	if [[ "$version" == "1" ]]; then
+		# This migration script upgrades the gravity.db file by
+		# adding the domain_audit table
+		sqlite3 "${database}" < "/etc/.pihole/advanced/Scripts/database_migration/gravity/1_to_2.sql"
+		version=2
+
+		# Store audit domains in database table
+		if [ -e "${auditFile}" ]; then
+			echo -e "  ${INFO} Migrating content of ${auditFile} into new database"
+			# database_table_from_file is defined in gravity.sh
+			database_table_from_file "domain_audit" "${auditFile}"
+		fi
+	fi
+}

advanced/Scripts/database_migration/gravity/1_to_2.sql

@@ -0,0 +1,10 @@
+.timeout 30000
+
+CREATE TABLE domain_audit
+(
+	id INTEGER PRIMARY KEY AUTOINCREMENT,
+	domain TEXT UNIQUE NOT NULL,
+	date_added INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int))
+);
+
+UPDATE info SET value = 2 WHERE property = 'version';

advanced/Scripts/list.sh

@@ -11,45 +11,45 @@
 # Globals
 basename=pihole
 piholeDir=/etc/"${basename}"
-whitelist="${piholeDir}"/whitelist.txt
-blacklist="${piholeDir}"/blacklist.txt
-readonly regexlist="/etc/pihole/regex.list"
+gravityDBfile="${piholeDir}/gravity.db"
+
 reload=false
 addmode=true
 verbose=true
 wildcard=false
+web=false
 
 domList=()
 
-listMain=""
-listAlt=""
+listType=""
+listname=""
 
 colfile="/opt/pihole/COL_TABLE"
 source ${colfile}
 
 
 helpFunc() {
-    if [[ "${listMain}" == "${whitelist}" ]]; then
+    if [[ "${listType}" == "whitelist" ]]; then
         param="w"
-        type="white"
-    elif [[ "${listMain}" == "${regexlist}" && "${wildcard}" == true ]]; then
+        type="whitelist"
+    elif [[ "${listType}" == "regex" && "${wildcard}" == true ]]; then
         param="-wild"
-        type="wildcard black"
-    elif [[ "${listMain}" == "${regexlist}" ]]; then
+        type="wildcard blacklist"
+    elif [[ "${listType}" == "regex" ]]; then
         param="-regex"
-        type="regex black"
+        type="regex filter"
     else
         param="b"
-        type="black"
+        type="blacklist"
     fi
 
     echo "Usage: pihole -${param} [options] <domain> <domain2 ...>
 Example: 'pihole -${param} site.com', or 'pihole -${param} site1.com site2.com'
-${type^}list one or more domains
+${type^} one or more domains
 
 Options:
-  -d, --delmode       Remove domain(s) from the ${type}list
-  -nr, --noreload     Update ${type}list without refreshing dnsmasq
+  -d, --delmode       Remove domain(s) from the ${type}
+  -nr, --noreload     Update ${type} without reloading the DNS server
   -q, --quiet         Make output less verbose
   -h, --help          Show this help dialog
   -l, --list          Display all your ${type}listed domains
@@ -72,7 +72,7 @@ HandleOther() {
 
     # Check validity of domain (don't check for regex entries)
     if [[ "${#domain}" -le 253 ]]; then
-        if [[ "${listMain}" == "${regexlist}" && "${wildcard}" == false ]]; then
+        if [[ "${listType}" == "regex" && "${wildcard}" == false ]]; then
             validDomain="${domain}"
         else
             validDomain=$(grep -P "^((-|_)*[a-z\\d]((-|_)*[a-z\\d])*(-|_)*)(\\.(-|_)*([a-z\\d]((-|_)*[a-z\\d])*))*$" <<< "${domain}") # Valid chars check
@@ -87,175 +87,143 @@ HandleOther() {
     fi
 }
 
-PoplistFile() {
-    # Check whitelist file exists, and if not, create it
-    if [[ ! -f "${whitelist}" ]]; then
-        touch "${whitelist}"
-    fi
-
-    # Check blacklist file exists, and if not, create it
-    if [[ ! -f "${blacklist}" ]]; then
-        touch "${blacklist}"
+ProcessDomainList() {
+    if [[ "${listType}" == "regex" ]]; then
+        # Regex filter list
+        listname="regex filters"
+    else
+        # Whitelist / Blacklist
+        listname="${listType}"
     fi
 
     for dom in "${domList[@]}"; do
-        # Logic: If addmode then add to desired list and remove from the other; if delmode then remove from desired list but do not add to the other
+        # Format domain into regex filter if requested
+        if [[ "${wildcard}" == true ]]; then
+            dom="(^|\\.)${dom//\./\\.}$"
+        fi
+
+        # Logic: If addmode then add to desired list and remove from the other;
+        # if delmode then remove from desired list but do not add to the other
         if ${addmode}; then
-            AddDomain "${dom}" "${listMain}"
-            RemoveDomain "${dom}" "${listAlt}"
+            AddDomain "${dom}" "${listType}"
+            if [[ ! "${listType}" == "regex" ]]; then
+                RemoveDomain "${dom}" "${listAlt}"
+            fi
         else
-            RemoveDomain "${dom}" "${listMain}"
+            RemoveDomain "${dom}" "${listType}"
         fi
   done
 }
 
 AddDomain() {
+    local domain list num
+    # Use printf to escape domain. %q prints the argument in a form that can be reused as shell input
+    domain="$1"
     list="$2"
-    domain=$(EscapeRegexp "$1")
 
-    [[ "${list}" == "${whitelist}" ]] && listname="whitelist"
-    [[ "${list}" == "${blacklist}" ]] && listname="blacklist"
+    # Is the domain in the list we want to add it to?
+    num="$(sqlite3 "${gravityDBfile}" "SELECT COUNT(*) FROM ${list} WHERE domain = '${domain}';")"
 
-    if [[ "${list}" == "${whitelist}" || "${list}" == "${blacklist}" ]]; then
-        [[ "${list}" == "${whitelist}" && -z "${type}" ]] && type="--whitelist-only"
-        [[ "${list}" == "${blacklist}" && -z "${type}" ]] && type="--blacklist-only"
-        bool=true
-        # Is the domain in the list we want to add it to?
-        grep -Ex -q "${domain}" "${list}" > /dev/null 2>&1 || bool=false
-
-        if [[ "${bool}" == false ]]; then
-            # Domain not found in the whitelist file, add it!
-            if [[ "${verbose}" == true ]]; then
-                echo -e "  ${INFO} Adding ${1} to ${listname}..."
-            fi
-            reload=true
-            # Add it to the list we want to add it to
-            echo "$1" >> "${list}"
-        else
-            if [[ "${verbose}" == true ]]; then
-                echo -e "  ${INFO} ${1} already exists in ${listname}, no need to add!"
-            fi
-        fi
-    elif [[ "${list}" == "${regexlist}" ]]; then
-        [[ -z "${type}" ]] && type="--wildcard-only"
-        bool=true
-        domain="${1}"
-
-        [[ "${wildcard}" == true ]] && domain="(^|\\.)${domain//\./\\.}$"
-
-        # Is the domain in the list?
-        # Search only for exactly matching lines
-        grep -Fx "${domain}" "${regexlist}" > /dev/null 2>&1 || bool=false
-
-        if [[ "${bool}" == false ]]; then
-            if [[ "${verbose}" == true ]]; then
-                echo -e "  ${INFO} Adding ${domain} to regex list..."
-            fi
-            reload="restart"
-            echo "$domain" >> "${regexlist}"
-        else
-            if [[ "${verbose}" == true ]]; then
-                echo -e "  ${INFO} ${domain} already exists in regex list, no need to add!"
-            fi
-        fi
+    if [[ "${num}" -ne 0 ]]; then
+      if [[ "${verbose}" == true ]]; then
+          echo -e "  ${INFO} ${1} already exists in ${listname}, no need to add!"
+      fi
+      return
     fi
+
+    # Domain not found in the table, add it!
+    if [[ "${verbose}" == true ]]; then
+        echo -e "  ${INFO} Adding ${1} to the ${listname}..."
+    fi
+    reload=true
+    # Insert only the domain here. The enabled and date_added fields will be filled
+    # with their default values (enabled = true, date_added = current timestamp)
+    sqlite3 "${gravityDBfile}" "INSERT INTO ${list} (domain) VALUES ('${domain}');"
 }
 
 RemoveDomain() {
+    local domain list num
+    # Use printf to escape domain. %q prints the argument in a form that can be reused as shell input
+    domain="$1"
     list="$2"
-    domain=$(EscapeRegexp "$1")
 
-    [[ "${list}" == "${whitelist}" ]] && listname="whitelist"
-    [[ "${list}" == "${blacklist}" ]] && listname="blacklist"
+    # Is the domain in the list we want to remove it from?
+    num="$(sqlite3 "${gravityDBfile}" "SELECT COUNT(*) FROM ${list} WHERE domain = '${domain}';")"
 
-    if [[ "${list}" == "${whitelist}" || "${list}" == "${blacklist}" ]]; then
-        bool=true
-        [[ "${list}" == "${whitelist}" && -z "${type}" ]] && type="--whitelist-only"
-        [[ "${list}" == "${blacklist}" && -z "${type}" ]] && type="--blacklist-only"
-        # Is it in the list? Logic follows that if its whitelisted it should not be blacklisted and vice versa
-        grep -Ex -q "${domain}" "${list}" > /dev/null 2>&1 || bool=false
-        if [[ "${bool}" == true ]]; then
-            # Remove it from the other one
-            echo -e "  ${INFO} Removing $1 from ${listname}..."
-            # /I flag: search case-insensitive
-            sed -i "/${domain}/Id" "${list}"
-            reload=true
-        else
-            if [[ "${verbose}" == true ]]; then
-                echo -e "  ${INFO} ${1} does not exist in ${listname}, no need to remove!"
-            fi
-        fi
-    elif [[ "${list}" == "${regexlist}" ]]; then
-        [[ -z "${type}" ]] && type="--wildcard-only"
-        domain="${1}"
-
-        [[ "${wildcard}" == true ]] && domain="(^|\\.)${domain//\./\\.}$"
-
-        bool=true
-        # Is it in the list?
-        grep -Fx "${domain}" "${regexlist}" > /dev/null 2>&1 || bool=false
-        if [[ "${bool}" == true ]]; then
-            # Remove it from the other one
-            echo -e "  ${INFO} Removing $domain from regex list..."
-            local lineNumber
-            lineNumber=$(grep -Fnx "$domain" "${list}" | cut -f1 -d:)
-            sed -i "${lineNumber}d" "${list}"
-            reload=true
-        else
-            if [[ "${verbose}" == true ]]; then
-                echo -e "  ${INFO} ${domain} does not exist in regex list, no need to remove!"
-            fi
-        fi
+    if [[ "${num}" -eq 0 ]]; then
+      if [[ "${verbose}" == true ]]; then
+          echo -e "  ${INFO} ${1} does not exist in ${list}, no need to remove!"
+      fi
+      return
     fi
-}
 
-# Update Gravity
-Reload() {
-    echo ""
-    pihole -g --skip-download "${type:-}"
+    # Domain found in the table, remove it!
+    if [[ "${verbose}" == true ]]; then
+        echo -e "  ${INFO} Removing ${1} from the ${listname}..."
+    fi
+    reload=true
+    # Remove it from the current list
+    sqlite3 "${gravityDBfile}" "DELETE FROM ${list} WHERE domain = '${domain}';"
 }
 
 Displaylist() {
-    if [[ -f ${listMain} ]]; then
-        if [[ "${listMain}" == "${whitelist}" ]]; then
-            string="gravity resistant domains"
-        else
-            string="domains caught in the sinkhole"
-        fi
-        verbose=false
-        echo -e "Displaying $string:\n"
-        count=1
-        while IFS= read -r RD || [ -n "${RD}" ]; do
-            echo "  ${count}: ${RD}"
-            count=$((count+1))
-        done < "${listMain}"
+    local list listname count num_pipes domain enabled status nicedate
+
+    listname="${listType}"
+    data="$(sqlite3 "${gravityDBfile}" "SELECT domain,enabled,date_modified FROM ${listType};" 2> /dev/null)"
+
+    if [[ -z $data ]]; then
+        echo -e "Not showing empty ${listname}"
     else
-        echo -e "  ${COL_LIGHT_RED}${listMain} does not exist!${COL_NC}"
+        echo -e "Displaying ${listname}:"
+        count=1
+        while IFS= read -r line
+        do
+            # Count number of pipes seen in this line
+            # This is necessary because we can only detect the pipe separating the fields
+            # from the end backwards as the domain (which is the first field) may contain
+            # pipe symbols as they are perfectly valid regex filter control characters
+            num_pipes="$(grep -c "^" <<< "$(grep -o "|" <<< "${line}")")"
+
+            # Extract domain and enabled status based on the obtained number of pipe characters
+            domain="$(cut -d'|' -f"-$((num_pipes-1))" <<< "${line}")"
+            enabled="$(cut -d'|' -f"$((num_pipes))" <<< "${line}")"
+            datemod="$(cut -d'|' -f"$((num_pipes+1))" <<< "${line}")"
+
+            # Translate boolean status into human readable string
+            if [[ "${enabled}" -eq 1 ]]; then
+                status="enabled"
+            else
+                status="disabled"
+            fi
+
+            # Get nice representation of numerical date stored in database
+            nicedate=$(date --rfc-2822 -d "@${datemod}")
+
+            echo "  ${count}: ${domain} (${status}, last modified ${nicedate})"
+            count=$((count+1))
+        done <<< "${data}"
     fi
     exit 0;
 }
 
 NukeList() {
-    if [[ -f "${listMain}" ]]; then
-        # Back up original list
-        cp "${listMain}" "${listMain}.bck~"
-        # Empty out file
-        echo "" > "${listMain}"
-    fi
+    sqlite3 "${gravityDBfile}" "DELETE FROM ${listType};"
 }
 
 for var in "$@"; do
     case "${var}" in
-        "-w" | "whitelist"   ) listMain="${whitelist}"; listAlt="${blacklist}";;
-        "-b" | "blacklist"   ) listMain="${blacklist}"; listAlt="${whitelist}";;
-        "--wild" | "wildcard" ) listMain="${regexlist}"; wildcard=true;;
-        "--regex" | "regex"   ) listMain="${regexlist}";;
+        "-w" | "whitelist"   ) listType="whitelist"; listAlt="blacklist";;
+        "-b" | "blacklist"   ) listType="blacklist"; listAlt="whitelist";;
+        "--wild" | "wildcard" ) listType="regex"; wildcard=true;;
+        "--regex" | "regex"   ) listType="regex";;
         "-nr"| "--noreload"  ) reload=false;;
         "-d" | "--delmode"   ) addmode=false;;
         "-q" | "--quiet"     ) verbose=false;;
         "-h" | "--help"      ) helpFunc;;
         "-l" | "--list"      ) Displaylist;;
         "--nuke"             ) NukeList;;
+        "--web"              ) web=true;;
         *                    ) HandleOther "${var}";;
     esac
 done
@@ -266,9 +234,13 @@ if [[ $# = 0 ]]; then
     helpFunc
 fi
 
-PoplistFile
+ProcessDomainList
+
+# Used on web interface
+if $web; then
+echo "DONE"
+fi
 
 if [[ "${reload}" != false ]]; then
-    # Ensure that "restart" is used for Wildcard updates
-    Reload "${reload}"
+    pihole restartdns reload
 fi

advanced/Scripts/piholeARPTable.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+# shellcheck disable=SC1090
+
+# Pi-hole: A black hole for Internet advertisements
+# (c) 2019 Pi-hole, LLC (https://pi-hole.net)
+# Network-wide ad blocking via your own hardware.
+#
+# ARP table interaction
+#
+# This file is copyright under the latest version of the EUPL.
+# Please see LICENSE file for your rights under this license.
+
+coltable="/opt/pihole/COL_TABLE"
+if [[ -f ${coltable} ]]; then
+    source ${coltable}
+fi
+
+# Determine database location
+# Obtain DBFILE=... setting from pihole-FTL.db
+# Constructed to return nothing when
+# a) the setting is not present in the config file, or
+# b) the setting is commented out (e.g. "#DBFILE=...")
+FTLconf="/etc/pihole/pihole-FTL.conf"
+if [ -e "$FTLconf" ]; then
+    DBFILE="$(sed -n -e 's/^\s*DBFILE\s*=\s*//p' ${FTLconf})"
+fi
+# Test for empty string. Use standard path in this case.
+if [ -z "$DBFILE" ]; then
+    DBFILE="/etc/pihole/pihole-FTL.db"
+fi
+
+
+flushARP(){
+    local output
+    if [[ "${args[1]}" != "quiet" ]]; then
+        echo -ne "  ${INFO} Flushing network table ..."
+    fi
+
+    # Flush ARP cache to avoid re-adding of dead entries
+    if ! output=$(ip neigh flush all 2>&1); then
+        echo -e "${OVER}  ${CROSS} Failed to clear ARP cache"
+        echo "  Output: ${output}"
+        return 1
+    fi
+
+    # Truncate network_addresses table in pihole-FTL.db
+    # This needs to be done before we can truncate the network table due to
+    # foreign key contraints
+    if ! output=$(sqlite3 "${DBFILE}" "DELETE FROM network_addresses" 2>&1); then
+        echo -e "${OVER}  ${CROSS} Failed to truncate network_addresses table"
+        echo "  Database location: ${DBFILE}"
+        echo "  Output: ${output}"
+        return 1
+    fi
+
+    # Truncate network table in pihole-FTL.db
+    if ! output=$(sqlite3 "${DBFILE}" "DELETE FROM network" 2>&1); then
+        echo -e "${OVER}  ${CROSS} Failed to truncate network table"
+        echo "  Database location: ${DBFILE}"
+        echo "  Output: ${output}"
+        return 1
+    fi
+
+    if [[ "${args[1]}" != "quiet" ]]; then
+        echo -e "${OVER}  ${TICK} Flushed network table"
+    fi
+}
+
+args=("$@")
+
+case "${args[0]}" in
+    "arpflush"            ) flushARP;;
+esac

advanced/Scripts/piholeCheckout.sh

@@ -90,6 +90,7 @@ checkout() {
         local path
         path="development/${binary}"
         echo "development" > /etc/pihole/ftlbranch
+        chmod 644 /etc/pihole/ftlbranch
     elif [[ "${1}" == "master" ]] ; then
         # Shortcut to check out master branches
         echo -e "  ${INFO} Shortcut \"master\" detected - checking out master branches..."
@@ -104,6 +105,7 @@ checkout() {
         local path
         path="master/${binary}"
         echo "master" > /etc/pihole/ftlbranch
+        chmod 644 /etc/pihole/ftlbranch
     elif [[ "${1}" == "core" ]] ; then
         str="Fetching branches from ${piholeGitUrl}"
         echo -ne "  ${INFO} $str"
@@ -115,7 +117,7 @@ checkout() {
 
         if [[ "${corebranches[*]}" == *"master"* ]]; then
             echo -e "${OVER}  ${TICK} $str"
-            echo -e "${INFO} ${#corebranches[@]} branches available for Pi-hole Core"
+            echo -e "  ${INFO} ${#corebranches[@]} branches available for Pi-hole Core"
         else
             # Print STDERR output from get_available_branches
             echo -e "${OVER}  ${CROSS} $str\\n\\n${corebranches[*]}"
@@ -142,7 +144,7 @@ checkout() {
 
         if [[ "${webbranches[*]}" == *"master"* ]]; then
             echo -e "${OVER}  ${TICK} $str"
-            echo -e "${INFO} ${#webbranches[@]} branches available for Web Admin"
+            echo -e "  ${INFO} ${#webbranches[@]} branches available for Web Admin"
         else
             # Print STDERR output from get_available_branches
             echo -e "${OVER}  ${CROSS} $str\\n\\n${webbranches[*]}"
@@ -166,8 +168,9 @@ checkout() {
         if check_download_exists "$path"; then
             echo "  ${TICK} Branch ${2} exists"
             echo "${2}" > /etc/pihole/ftlbranch
+            chmod 644 /etc/pihole/ftlbranch
             FTLinstall "${binary}"
-            start_service pihole-FTL
+            restart_service pihole-FTL
             enable_service pihole-FTL
         else
             echo "  ${CROSS} Requested branch \"${2}\" is not available"

advanced/Scripts/piholeDebug.sh

@@ -76,6 +76,7 @@ WEB_SERVER_CONFIG_DIRECTORY="/etc/lighttpd"
 HTML_DIRECTORY="/var/www/html"
 WEB_GIT_DIRECTORY="${HTML_DIRECTORY}/admin"
 #BLOCK_PAGE_DIRECTORY="${HTML_DIRECTORY}/pihole"
+SHM_DIRECTORY="/dev/shm"
 
 # Files required by Pi-hole
 # https://discourse.pi-hole.net/t/what-files-does-pi-hole-use/1684
@@ -88,16 +89,40 @@ PIHOLE_WILDCARD_CONFIG_FILE="${DNSMASQ_D_DIRECTORY}/03-wildcard.conf"
 WEB_SERVER_CONFIG_FILE="${WEB_SERVER_CONFIG_DIRECTORY}/lighttpd.conf"
 #WEB_SERVER_CUSTOM_CONFIG_FILE="${WEB_SERVER_CONFIG_DIRECTORY}/external.conf"
 
-PIHOLE_DEFAULT_AD_LISTS="${PIHOLE_DIRECTORY}/adlists.default"
-PIHOLE_USER_DEFINED_AD_LISTS="${PIHOLE_DIRECTORY}/adlists.list"
-PIHOLE_BLACKLIST_FILE="${PIHOLE_DIRECTORY}/blacklist.txt"
-PIHOLE_BLOCKLIST_FILE="${PIHOLE_DIRECTORY}/gravity.list"
 PIHOLE_INSTALL_LOG_FILE="${PIHOLE_DIRECTORY}/install.log"
 PIHOLE_RAW_BLOCKLIST_FILES="${PIHOLE_DIRECTORY}/list.*"
 PIHOLE_LOCAL_HOSTS_FILE="${PIHOLE_DIRECTORY}/local.list"
 PIHOLE_LOGROTATE_FILE="${PIHOLE_DIRECTORY}/logrotate"
 PIHOLE_SETUP_VARS_FILE="${PIHOLE_DIRECTORY}/setupVars.conf"
-PIHOLE_WHITELIST_FILE="${PIHOLE_DIRECTORY}/whitelist.txt"
+PIHOLE_FTL_CONF_FILE="${PIHOLE_DIRECTORY}/pihole-FTL.conf"
+
+# Read the value of an FTL config key. The value is printed to stdout.
+#
+# Args:
+# 1. The key to read
+# 2. The default if the setting or config does not exist
+get_ftl_conf_value() {
+    local key=$1
+    local default=$2
+    local value
+
+    # Obtain key=... setting from pihole-FTL.conf
+    if [[ -e "$PIHOLE_FTL_CONF_FILE" ]]; then
+        # Constructed to return nothing when
+        # a) the setting is not present in the config file, or
+        # b) the setting is commented out (e.g. "#DBFILE=...")
+        value="$(sed -n -e "s/^\\s*$key=\\s*//p" ${PIHOLE_FTL_CONF_FILE})"
+    fi
+
+    # Test for missing value. Use default value in this case.
+    if [[ -z "$value" ]]; then
+        value="$default"
+    fi
+
+    echo "$value"
+}
+
+PIHOLE_GRAVITY_DB_FILE="$(get_ftl_conf_value "GRAVITYDB" "${PIHOLE_DIRECTORY}/gravity.db")"
 
 PIHOLE_COMMAND="${BIN_DIRECTORY}/pihole"
 PIHOLE_COLTABLE_FILE="${BIN_DIRECTORY}/COL_TABLE"
@@ -108,8 +133,7 @@ FTL_PORT="${RUN_DIRECTORY}/pihole-FTL.port"
 PIHOLE_LOG="${LOG_DIRECTORY}/pihole.log"
 PIHOLE_LOG_GZIPS="${LOG_DIRECTORY}/pihole.log.[0-9].*"
 PIHOLE_DEBUG_LOG="${LOG_DIRECTORY}/pihole_debug.log"
-PIHOLE_DEBUG_LOG_SANITIZED="${LOG_DIRECTORY}/pihole_debug-sanitized.log"
-PIHOLE_FTL_LOG="${LOG_DIRECTORY}/pihole-FTL.log"
+PIHOLE_FTL_LOG="$(get_ftl_conf_value "LOGFILE" "${LOG_DIRECTORY}/pihole-FTL.log")"
 
 PIHOLE_WEB_SERVER_ACCESS_LOG_FILE="${WEB_SERVER_LOG_DIRECTORY}/access.log"
 PIHOLE_WEB_SERVER_ERROR_LOG_FILE="${WEB_SERVER_LOG_DIRECTORY}/error.log"
@@ -119,7 +143,7 @@ PIHOLE_WEB_SERVER_ERROR_LOG_FILE="${WEB_SERVER_LOG_DIRECTORY}/error.log"
 #SUPPORTED_OS=("Raspbian" "Ubuntu" "Fedora" "Debian" "CentOS")
 
 # Store Pi-hole's processes in an array for easy use and parsing
-PIHOLE_PROCESSES=( "dnsmasq" "lighttpd" "pihole-FTL" )
+PIHOLE_PROCESSES=( "lighttpd" "pihole-FTL" )
 
 # Store the required directories in an array so it can be parsed through
 #REQUIRED_DIRECTORIES=("${CORE_GIT_DIRECTORY}"
@@ -142,16 +166,11 @@ REQUIRED_FILES=("${PIHOLE_CRON_FILE}"
 "${PIHOLE_DHCP_CONFIG_FILE}"
 "${PIHOLE_WILDCARD_CONFIG_FILE}"
 "${WEB_SERVER_CONFIG_FILE}"
-"${PIHOLE_DEFAULT_AD_LISTS}"
-"${PIHOLE_USER_DEFINED_AD_LISTS}"
-"${PIHOLE_BLACKLIST_FILE}"
-"${PIHOLE_BLOCKLIST_FILE}"
 "${PIHOLE_INSTALL_LOG_FILE}"
 "${PIHOLE_RAW_BLOCKLIST_FILES}"
 "${PIHOLE_LOCAL_HOSTS_FILE}"
 "${PIHOLE_LOGROTATE_FILE}"
 "${PIHOLE_SETUP_VARS_FILE}"
-"${PIHOLE_WHITELIST_FILE}"
 "${PIHOLE_COMMAND}"
 "${PIHOLE_COLTABLE_FILE}"
 "${FTL_PID}"
@@ -208,11 +227,6 @@ log_write() {
 copy_to_debug_log() {
     # Copy the contents of file descriptor 3 into the debug log
     cat /proc/$$/fd/3 > "${PIHOLE_DEBUG_LOG}"
-    # Since we use color codes such as '\e[1;33m', they should be removed before being
-    # uploaded to our server, since it can't properly display in color
-    # This is accomplished by use sed to remove characters matching that patter
-    # The entire file is then copied over to a sanitized version of the log
-    sed 's/\[[0-9;]\{1,5\}m//g' > "${PIHOLE_DEBUG_LOG_SANITIZED}" <<< cat "${PIHOLE_DEBUG_LOG}"
 }
 
 initialize_debug() {
@@ -268,6 +282,9 @@ compare_local_version_to_git_version() {
             # The commit they are on
             local remote_commit
             remote_commit=$(git describe --long --dirty --tags --always)
+            # Status of the repo
+            local local_status
+            local_status=$(git status -s)
             # echo this information out to the user in a nice format
             # If the current version matches what pihole -v produces, the user is up-to-date
             if [[ "${remote_version}" == "$(pihole -v | awk '/${search_term}/ {print $6}' | cut -d ')' -f1)" ]]; then
@@ -290,6 +307,16 @@ compare_local_version_to_git_version() {
             fi
             # echo the current commit
             log_write "${INFO} Commit: ${remote_commit}"
+            # if `local_status` is non-null, then the repo is not clean, display details here
+            if [[ ${local_status} ]]; then
+              #Replace new lines in the status with 12 spaces to make the output cleaner
+              log_write "${INFO} Status: ${local_status//$'\n'/'\n            '}"
+              local local_diff
+              local_diff=$(git diff)
+              if [[ ${local_diff} ]]; then
+                log_write "${INFO} Diff: ${local_diff//$'\n'/'\n          '}"
+              fi
+            fi
         # If git status failed,
         else
             # Return an error message
@@ -337,8 +364,6 @@ get_program_version() {
     case "${program_name}" in
         "lighttpd") program_version="$(${program_name} -v |& head -n1 | cut -d '/' -f2 | cut -d ' ' -f1)"
                     ;;
-        "dnsmasq") program_version="$(${program_name} -v |& head -n1 | awk '{print $3}')"
-                    ;;
         "php") program_version="$(${program_name} -v |& head -n1 | cut -d '-' -f1 | cut -d ' ' -f2)"
                 ;;
         # If a match is not found, show an error
@@ -358,7 +383,6 @@ get_program_version() {
 # and their versions, using the functions above.
 check_critical_program_versions() {
     # Use the function created earlier and bundle them into one function that checks all the version numbers
-    get_program_version "dnsmasq"
     get_program_version "lighttpd"
     get_program_version "php"
 }
@@ -640,11 +664,12 @@ ping_internet() {
 compare_port_to_service_assigned() {
     local service_name="${1}"
     # The programs we use may change at some point, so they are in a varible here
-    local resolver="dnsmasq"
+    local resolver="pihole-FTL"
     local web_server="lighttpd"
     local ftl="pihole-FTL"
+
+    # If the service is a Pi-hole service, highlight it in green
     if [[ "${service_name}" == "${resolver}" ]] || [[ "${service_name}" == "${web_server}" ]] || [[ "${service_name}" == "${ftl}" ]]; then
-        # if port 53 is dnsmasq, show it in green as it's standard
         log_write "[${COL_GREEN}${port_number}${COL_NC}] is in use by ${COL_GREEN}${service_name}${COL_NC}"
     # Otherwise,
     else
@@ -657,7 +682,7 @@ check_required_ports() {
     echo_current_diagnostic "Ports in use"
     # Since Pi-hole needs 53, 80, and 4711, check what they are being used by
     # so we can detect any issues
-    local resolver="dnsmasq"
+    local resolver="pihole-FTL"
     local web_server="lighttpd"
     local ftl="pihole-FTL"
     # Create an array for these ports in use
@@ -682,7 +707,7 @@ check_required_ports() {
             continue
         fi
         # Use a case statement to determine if the right services are using the right ports
-        case "${port_number}" in
+        case "$(echo "$port_number" | rev | cut -d: -f1 | rev)" in
             53) compare_port_to_service_assigned  "${resolver}"
                 ;;
             80) compare_port_to_service_assigned  "${web_server}"
@@ -787,7 +812,7 @@ dig_at() {
     # This helps emulate queries to different domains that a user might query
     # It will also give extra assurance that Pi-hole is correctly resolving and blocking domains
     local random_url
-    random_url=$(shuf -n 1 "${PIHOLE_BLOCKLIST_FILE}")
+    random_url=$(sqlite3 "${PIHOLE_GRAVITY_DB_FILE}" "SELECT domain FROM vw_gravity ORDER BY RANDOM() LIMIT 1")
 
     # First, do a dig on localhost to see if Pi-hole can use itself to block a domain
     if local_dig=$(dig +tries=1 +time=2 -"${protocol}" "${random_url}" @${local_address} +short "${record_type}"); then
@@ -836,9 +861,13 @@ process_status(){
             local status_of_process
             status_of_process=$(systemctl is-active "${i}")
         else
-            # Otherwise, use the service command
+            # Otherwise, use the service command and mock the output of `systemctl is-active`
             local status_of_process
-            status_of_process=$(service "${i}" status | awk '/Active:/ {print $2}') &> /dev/null
+            if service "${i}" status | grep -E 'is\srunning' &> /dev/null; then
+                status_of_process="active"
+            else
+                status_of_process="inactive"
+            fi
         fi
         # and print it out to the user
         if [[ "${status_of_process}" == "active" ]]; then
@@ -907,7 +936,7 @@ parse_file() {
         #shellcheck disable=SC2016
         IFS=$'\r\n' command eval 'file_info=( $(cat "${filename}") )'
     else
-        read -a file_info <<< $filename
+        read -r -a file_info <<< "$filename"
     fi
     # Set a named variable for better readability
     local file_lines
@@ -965,8 +994,7 @@ list_files_in_dir() {
         if [[ -d "${dir_to_parse}/${each_file}" ]]; then
             # If it's a directoy, do nothing
             :
-        elif [[ "${dir_to_parse}/${each_file}" == "${PIHOLE_BLOCKLIST_FILE}" ]] || \
-            [[ "${dir_to_parse}/${each_file}" == "${PIHOLE_DEBUG_LOG}" ]] || \
+        elif [[ "${dir_to_parse}/${each_file}" == "${PIHOLE_DEBUG_LOG}" ]] || \
             [[ "${dir_to_parse}/${each_file}" == "${PIHOLE_RAW_BLOCKLIST_FILES}" ]] || \
             [[ "${dir_to_parse}/${each_file}" == "${PIHOLE_INSTALL_LOG_FILE}" ]] || \
             [[ "${dir_to_parse}/${each_file}" == "${PIHOLE_SETUP_VARS_FILE}" ]] || \
@@ -974,6 +1002,9 @@ list_files_in_dir() {
             [[ "${dir_to_parse}/${each_file}" == "${PIHOLE_WEB_SERVER_ACCESS_LOG_FILE}" ]] || \
             [[ "${dir_to_parse}/${each_file}" == "${PIHOLE_LOG_GZIPS}" ]]; then
             :
+        elif [[ "${dir_to_parse}" == "${SHM_DIRECTORY}" ]]; then
+            # SHM file - we do not want to see the content, but we want to see the files and their sizes
+            log_write "$(ls -ld "${dir_to_parse}"/"${each_file}")"
         else
             # Then, parse the file's content into an array so each line can be analyzed if need be
             for i in "${!REQUIRED_FILES[@]}"; do
@@ -1017,6 +1048,7 @@ show_content_of_pihole_files() {
     show_content_of_files_in_dir "${CRON_D_DIRECTORY}"
     show_content_of_files_in_dir "${WEB_SERVER_LOG_DIRECTORY}"
     show_content_of_files_in_dir "${LOG_DIRECTORY}"
+    show_content_of_files_in_dir "${SHM_DIRECTORY}"
 }
 
 head_tail_log() {
@@ -1047,31 +1079,70 @@ head_tail_log() {
     IFS="$OLD_IFS"
 }
 
-analyze_gravity_list() {
-    echo_current_diagnostic "Gravity list"
-    local head_line
-    local tail_line
-    # Put the current Internal Field Separator into another variable so it can be restored later
+show_db_entries() {
+    local title="${1}"
+    local query="${2}"
+    local widths="${3}"
+
+    echo_current_diagnostic "${title}"
+
     OLD_IFS="$IFS"
-    # Get the lines that are in the file(s) and store them in an array for parsing later
     IFS=$'\r\n'
+    local entries=()
+    mapfile -t entries < <(\
+        sqlite3 "${PIHOLE_GRAVITY_DB_FILE}" \
+            -cmd ".headers on" \
+            -cmd ".mode column" \
+            -cmd ".width ${widths}" \
+            "${query}"\
+    )
+
+    for line in "${entries[@]}"; do
+        log_write "   ${line}"
+    done
+
+    IFS="$OLD_IFS"
+}
+
+show_adlists() {
+    show_db_entries "Adlists" "SELECT * FROM adlist" "4 100 7 10 13 50"
+}
+
+show_whitelist() {
+    show_db_entries "Whitelist" "SELECT * FROM whitelist" "4 100 7 10 13 50"
+}
+
+show_blacklist() {
+    show_db_entries "Blacklist" "SELECT * FROM blacklist" "4 100 7 10 13 50"
+}
+
+show_regexlist() {
+    show_db_entries "Regexlist" "SELECT * FROM regex" "4 100 7 10 13 50"
+}
+
+analyze_gravity_list() {
+    echo_current_diagnostic "Gravity List and Database"
+
     local gravity_permissions
-    gravity_permissions=$(ls -ld "${PIHOLE_BLOCKLIST_FILE}")
+    gravity_permissions=$(ls -ld "${PIHOLE_GRAVITY_DB_FILE}")
     log_write "${COL_GREEN}${gravity_permissions}${COL_NC}"
-    local gravity_head=()
-    mapfile -t gravity_head < <(head -n 4 ${PIHOLE_BLOCKLIST_FILE})
-    log_write "   ${COL_CYAN}-----head of $(basename ${PIHOLE_BLOCKLIST_FILE})------${COL_NC}"
-    for head_line in "${gravity_head[@]}"; do
-        log_write "   ${head_line}"
-    done
+
+    local gravity_size
+    gravity_size=$(sqlite3 "${PIHOLE_GRAVITY_DB_FILE}" "SELECT COUNT(*) FROM vw_gravity")
+    log_write "   Size (excluding blacklist): ${COL_CYAN}${gravity_size}${COL_NC} entries"
     log_write ""
-    local gravity_tail=()
-    mapfile -t gravity_tail < <(tail -n 4 ${PIHOLE_BLOCKLIST_FILE})
-    log_write "   ${COL_CYAN}-----tail of $(basename ${PIHOLE_BLOCKLIST_FILE})------${COL_NC}"
-    for tail_line in "${gravity_tail[@]}"; do
-        log_write "   ${tail_line}"
+
+    OLD_IFS="$IFS"
+    IFS=$'\r\n'
+    local gravity_sample=()
+    mapfile -t gravity_sample < <(sqlite3 "${PIHOLE_GRAVITY_DB_FILE}" "SELECT domain FROM vw_gravity LIMIT 10")
+    log_write "   ${COL_CYAN}----- First 10 Domains -----${COL_NC}"
+
+    for line in "${gravity_sample[@]}"; do
+        log_write "   ${line}"
     done
-    # Set the IFS back to what it was
+
+    log_write ""
     IFS="$OLD_IFS"
 }
 
@@ -1127,20 +1198,20 @@ analyze_pihole_log() {
     IFS="$OLD_IFS"
 }
 
-tricorder_use_nc_or_ssl() {
-    # Users can submit their debug logs using nc (unencrypted) or openssl (enrypted) if available
-    # Check for openssl first since encryption is a good thing
-    if command -v openssl &> /dev/null; then
+tricorder_use_nc_or_curl() {
+    # Users can submit their debug logs using nc (unencrypted) or curl (encrypted) if available
+    # Check for curl first since encryption is a good thing
+    if command -v curl &> /dev/null; then
         # If the command exists,
-        log_write "    * Using ${COL_GREEN}openssl${COL_NC} for transmission."
-        # encrypt and transmit the log and store the token returned in a variable
-        tricorder_token=$(< ${PIHOLE_DEBUG_LOG_SANITIZED} openssl s_client -quiet -connect tricorder.pi-hole.net:${TRICORDER_SSL_PORT_NUMBER} 2> /dev/null)
+        log_write "    * Using ${COL_GREEN}curl${COL_NC} for transmission."
+        # transmit he log via TLS and store the token returned in a variable
+        tricorder_token=$(curl --silent --upload-file ${PIHOLE_DEBUG_LOG} https://tricorder.pi-hole.net:${TRICORDER_SSL_PORT_NUMBER})
     # Otherwise,
     else
         # use net cat
         log_write "${INFO} Using ${COL_YELLOW}netcat${COL_NC} for transmission."
         # Save the token returned by our server in a variable
-        tricorder_token=$(< ${PIHOLE_DEBUG_LOG_SANITIZED} nc tricorder.pi-hole.net ${TRICORDER_NC_PORT_NUMBER})
+        tricorder_token=$(< ${PIHOLE_DEBUG_LOG} nc tricorder.pi-hole.net ${TRICORDER_NC_PORT_NUMBER})
     fi
 }
 
@@ -1166,7 +1237,7 @@ upload_to_tricorder() {
         # let the user know
         log_write "${INFO} Debug script running in automated mode"
         # and then decide again which tool to use to submit it
-        tricorder_use_nc_or_ssl
+        tricorder_use_nc_or_curl
         # If we're not running in automated mode,
     else
         echo ""
@@ -1175,7 +1246,7 @@ upload_to_tricorder() {
         read -r -p "[?] Would you like to upload the log? [y/N] " response
         case ${response} in
             # If they say yes, run our function for uploading the log
-            [yY][eE][sS]|[yY]) tricorder_use_nc_or_ssl;;
+            [yY][eE][sS]|[yY]) tricorder_use_nc_or_curl;;
             # If they choose no, just exit out of the script
             *) log_write "    * Log will ${COL_GREEN}NOT${COL_NC} be uploaded to tricorder.";exit;
         esac
@@ -1202,7 +1273,7 @@ upload_to_tricorder() {
         log_write "   * Please try again or contact the Pi-hole team for assistance."
     fi
     # Finally, show where the log file is no matter the outcome of the function so users can look at it
-    log_write "   * A local copy of the debug log can be found at: ${COL_CYAN}${PIHOLE_DEBUG_LOG_SANITIZED}${COL_NC}\\n"
+    log_write "   * A local copy of the debug log can be found at: ${COL_CYAN}${PIHOLE_DEBUG_LOG}${COL_NC}\\n"
 }
 
 # Run through all the functions we made
@@ -1222,6 +1293,10 @@ process_status
 parse_setup_vars
 check_x_headers
 analyze_gravity_list
+show_adlists
+show_whitelist
+show_blacklist
+show_regexlist
 show_content_of_pihole_files
 parse_locale
 analyze_pihole_log

advanced/Scripts/piholeLogFlush.sh

@@ -39,8 +39,9 @@ if [[ "$@" == *"once"* ]]; then
         # Note that moving the file is not an option, as
         # dnsmasq would happily continue writing into the
         # moved file (it will have the same file handler)
-        cp /var/log/pihole.log /var/log/pihole.log.1
+        cp -p /var/log/pihole.log /var/log/pihole.log.1
         echo " " > /var/log/pihole.log
+        chmod 644 /var/log/pihole.log
     fi
 else
     # Manual flushing
@@ -53,11 +54,14 @@ else
         echo " " > /var/log/pihole.log
         if [ -f /var/log/pihole.log.1 ]; then
             echo " " > /var/log/pihole.log.1
+            chmod 644 /var/log/pihole.log.1
         fi
     fi
     # Delete most recent 24 hours from FTL's database, leave even older data intact (don't wipe out all history)
     deleted=$(sqlite3 "${DBFILE}" "DELETE FROM queries WHERE timestamp >= strftime('%s','now')-86400; select changes() from queries limit 1")
 
+    # Restart pihole-FTL to force reloading history
+    sudo pihole restartdns
 fi
 
 if [[ "$@" != *"quiet"* ]]; then

advanced/Scripts/query.sh

@@ -11,8 +11,7 @@
 
 # Globals
 piholeDir="/etc/pihole"
-adListsList="$piholeDir/adlists.list"
-wildcardlist="/etc/dnsmasq.d/03-pihole-wildcard.conf"
+gravityDBfile="${piholeDir}/gravity.db"
 options="$*"
 adlist=""
 all=""
@@ -23,27 +22,10 @@ matchType="match"
 colfile="/opt/pihole/COL_TABLE"
 source "${colfile}"
 
-# Print each subdomain
-# e.g: foo.bar.baz.com = "foo.bar.baz.com bar.baz.com baz.com com"
-processWildcards() {
-    IFS="." read -r -a array <<< "${1}"
-    for (( i=${#array[@]}-1; i>=0; i-- )); do
-        ar=""
-        for (( j=${#array[@]}-1; j>${#array[@]}-i-2; j-- )); do
-            if [[ $j == $((${#array[@]}-1)) ]]; then
-                ar="${array[$j]}"
-            else
-                ar="${array[$j]}.${ar}"
-            fi
-        done
-        echo "${ar}"
-    done
-}
-
 # Scan an array of files for matching strings
 scanList(){
     # Escape full stops
-    local domain="${1//./\\.}" lists="${2}" type="${3:-}"
+    local domain="${1}" esc_domain="${1//./\\.}" lists="${2}" type="${3:-}"
 
     # Prevent grep from printing file path
     cd "$piholeDir" || exit 1
@@ -54,9 +36,14 @@ scanList(){
     # /dev/null forces filename to be printed when only one list has been generated
     # shellcheck disable=SC2086
     case "${type}" in
-        "exact" ) grep -i -E -l "(^|\\s)${domain}($|\\s|#)" ${lists} /dev/null 2>/dev/null;;
-        "wc"    ) grep -i -o -m 1 "/${domain}/" ${lists} 2>/dev/null;;
-        *       ) grep -i "${domain}" ${lists} /dev/null 2>/dev/null;;
+		"exact" ) grep -i -E -l "(^|(?<!#)\\s)${esc_domain}($|\\s|#)" ${lists} /dev/null 2>/dev/null;;
+		# Create array of regexps
+		# Iterate through each regexp and check whether it matches the domainQuery
+		# If it does, print the matching regexp and continue looping
+		# Input 1 - regexps | Input 2 - domainQuery
+		"regex" ) awk 'NR==FNR{regexps[$0];next}{for (r in regexps)if($0 ~ r)print r}' \
+					<(echo "${lists}") <(echo "${domain}") 2>/dev/null;;
+		*       ) grep -i "${esc_domain}" ${lists} /dev/null 2>/dev/null;;
     esac
 }
 
@@ -73,11 +60,6 @@ Options:
   exit 0
 fi
 
-if [[ ! -e "$adListsList" ]]; then
-    echo -e "${COL_LIGHT_RED}The file $adListsList was not found${COL_NC}"
-    exit 1
-fi
-
 # Handle valid options
 if [[ "${options}" == *"-bp"* ]]; then
     exact="exact"; blockpage=true
@@ -107,48 +89,82 @@ if [[ -n "${str:-}" ]]; then
     exit 1
 fi
 
-# Scan Whitelist and Blacklist
-lists="whitelist.txt blacklist.txt"
-mapfile -t results <<< "$(scanList "${domainQuery}" "${lists}" "${exact}")"
-if [[ -n "${results[*]}" ]]; then
+scanDatabaseTable() {
+    local domain table type querystr result
+    domain="$(printf "%q" "${1}")"
+    table="${2}"
+    type="${3:-}"
+
+    # As underscores are legitimate parts of domains, we escape them when using the LIKE operator.
+    # Underscores are SQLite wildcards matching exactly one character. We obviously want to suppress this
+    # behavior. The "ESCAPE '\'" clause specifies that an underscore preceded by an '\' should be matched
+    # as a literal underscore character. We pretreat the $domain variable accordingly to escape underscores.
+    case "${type}" in
+		"exact" ) querystr="SELECT domain FROM vw_${table} WHERE domain = '${domain}'";;
+		*       ) querystr="SELECT domain FROM vw_${table} WHERE domain LIKE '%${domain//_/\\_}%' ESCAPE '\\'";;
+    esac
+
+    # Send prepared query to gravity database
+    result="$(sqlite3 "${gravityDBfile}" "${querystr}")" 2> /dev/null
+    if [[ -z "${result}" ]]; then
+        # Return early when there are no matches in this table
+        return
+    fi
+
+    # Mark domain as having been white-/blacklist matched (global variable)
     wbMatch=true
-    # Loop through each result in order to print unique file title once
+
+    # Print table name
+    if [[ -z "${blockpage}" ]]; then
+        echo " ${matchType^} found in ${COL_BOLD}${table^}${COL_NC}"
+    fi
+
+    # Loop over results and print them
+    mapfile -t results <<< "${result}"
     for result in "${results[@]}"; do
-        fileName="${result%%.*}"
         if [[ -n "${blockpage}" ]]; then
             echo "π ${result}"
             exit 0
-        elif [[ -n "${exact}" ]]; then
-            echo " ${matchType^} found in ${COL_BOLD}${fileName^}${COL_NC}"
-        else
-            # Only print filename title once per file
-            if [[ ! "${fileName}" == "${fileName_prev:-}" ]]; then
-                echo " ${matchType^} found in ${COL_BOLD}${fileName^}${COL_NC}"
-                fileName_prev="${fileName}"
-            fi
-        echo "   ${result#*:}"
         fi
+        echo "   ${result}"
     done
-fi
+}
 
-# Scan Wildcards
-if [[ -e "${wildcardlist}" ]]; then
-    # Determine all subdomains, domain and TLDs
-    mapfile -t wildcards <<< "$(processWildcards "${domainQuery}")"
-    for match in "${wildcards[@]}"; do
-        # Search wildcard list for matches
-        mapfile -t results <<< "$(scanList "${match}" "${wildcardlist}" "wc")"
-        if [[ -n "${results[*]}" ]]; then
-            if [[ -z "${wcMatch:-}" ]] && [[ -z "${blockpage}" ]]; then
-                wcMatch=true
-                echo " ${matchType^} found in ${COL_BOLD}Wildcards${COL_NC}:"
-            fi
-            case "${blockpage}" in
-                true ) echo "π ${wildcardlist##*/}"; exit 0;;
-                *    ) echo "   *.${match}";;
-            esac
-        fi
-    done
+# Scan Whitelist and Blacklist
+scanDatabaseTable "${domainQuery}" "whitelist" "${exact}"
+scanDatabaseTable "${domainQuery}" "blacklist" "${exact}"
+
+# Scan Regex table
+mapfile -t regexList < <(sqlite3 "${gravityDBfile}" "SELECT domain FROM vw_regex" 2> /dev/null)
+
+# If we have regexps to process
+if [[ "${#regexList[@]}" -ne 0 ]]; then
+	# Split regexps over a new line
+	str_regexList=$(printf '%s\n' "${regexList[@]}")
+	# Check domainQuery against regexps
+	mapfile -t regexMatches < <(scanList "${domainQuery}" "${str_regexList}" "regex")
+	# If there were regex matches
+	if [[  "${#regexMatches[@]}" -ne 0 ]]; then
+		# Split matching regexps over a new line
+		str_regexMatches=$(printf '%s\n' "${regexMatches[@]}")
+		# Form a "matched" message
+		str_message="${matchType^} found in ${COL_BOLD}Regex list${COL_NC}"
+		# Form a "results" message
+		str_result="${COL_BOLD}${str_regexMatches}${COL_NC}"
+		# If we are displaying more than just the source of the block
+		if [[ -z "${blockpage}" ]]; then
+			# Set the wildcard match flag
+			wcMatch=true
+			# Echo the "matched" message, indented by one space
+			echo " ${str_message}"
+			# Echo the "results" message, each line indented by three spaces
+			# shellcheck disable=SC2001
+			echo "${str_result}" | sed 's/^/   /'
+		else
+			echo "π .wildcard"
+			exit 0
+		fi
+	fi
 fi
 
 # Get version sorted *.domains filenames (without dir path)
@@ -186,11 +202,8 @@ fi
 
 # Get adlist file content as array
 if [[ -n "${adlist}" ]] || [[ -n "${blockpage}" ]]; then
-    for adlistUrl in $(< "${adListsList}"); do
-        if [[ "${adlistUrl:0:4}" =~ (http|www.) ]]; then
-            adlists+=("${adlistUrl}")
-        fi
-    done
+    # Retrieve source URLs from gravity database
+    mapfile -t adlists <<< "$(sqlite3 "${gravityDBfile}" "SELECT address FROM vw_adlist;" 2> /dev/null)"
 fi
 
 # Print "Exact matches for" title

advanced/Scripts/update.sh

@@ -146,6 +146,20 @@ main() {
         FTL_update=false
     fi
 
+    # Determine FTL branch
+    local ftlBranch
+    if [[ -f "/etc/pihole/ftlbranch" ]]; then
+        ftlBranch=$(</etc/pihole/ftlbranch)
+    else
+        ftlBranch="master"
+    fi
+
+    if [[ ! "${ftlBranch}" == "master" && ! "${ftlBranch}" == "development" ]]; then
+        # Notify user that they are on a custom branch which might mean they they are lost
+        # behind if a branch was merged to development and got abandoned
+        printf "  %b %bWarning:%b You are using FTL from a custom branch (%s) and might be missing future releases.\\n" "${INFO}" "${COL_LIGHT_RED}" "${COL_NC}" "${ftlBranch}"
+    fi
+
     if [[ "${core_update}" == false && "${web_update}" == false && "${FTL_update}" == false ]]; then
         echo ""
         echo -e "  ${TICK} Everything is up to date!"

advanced/Scripts/updatecheck.sh

@@ -34,33 +34,61 @@ function get_local_branch() {
 function get_local_version() {
     # Return active branch
     cd "${1}" 2> /dev/null || return 1
-    git describe --long --dirty --tags || return 1
+    git describe --long --dirty --tags 2> /dev/null || return 1
 }
 
+# Source the setupvars config file
+# shellcheck disable=SC1091
+. /etc/pihole/setupVars.conf
+
 if [[ "$2" == "remote" ]]; then
 
     if [[ "$3" == "reboot" ]]; then
         sleep 30
     fi
 
-    GITHUB_CORE_VERSION="$(json_extract tag_name "$(curl -q 'https://api.github.com/repos/pi-hole/pi-hole/releases/latest' 2> /dev/null)")"
-    GITHUB_WEB_VERSION="$(json_extract tag_name "$(curl -q 'https://api.github.com/repos/pi-hole/AdminLTE/releases/latest' 2> /dev/null)")"
-    GITHUB_FTL_VERSION="$(json_extract tag_name "$(curl -q 'https://api.github.com/repos/pi-hole/FTL/releases/latest' 2> /dev/null)")"
+    GITHUB_VERSION_FILE="/etc/pihole/GitHubVersions"
 
-    echo -n "${GITHUB_CORE_VERSION} ${GITHUB_WEB_VERSION} ${GITHUB_FTL_VERSION}" > "/etc/pihole/GitHubVersions"
+    GITHUB_CORE_VERSION="$(json_extract tag_name "$(curl -s 'https://api.github.com/repos/pi-hole/pi-hole/releases/latest' 2> /dev/null)")"
+    echo -n "${GITHUB_CORE_VERSION}" > "${GITHUB_VERSION_FILE}"
+    chmod 644 "${GITHUB_VERSION_FILE}"
+
+    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
+        GITHUB_WEB_VERSION="$(json_extract tag_name "$(curl -s 'https://api.github.com/repos/pi-hole/AdminLTE/releases/latest' 2> /dev/null)")"
+        echo -n " ${GITHUB_WEB_VERSION}" >> "${GITHUB_VERSION_FILE}"
+    fi
+
+    GITHUB_FTL_VERSION="$(json_extract tag_name "$(curl -s 'https://api.github.com/repos/pi-hole/FTL/releases/latest' 2> /dev/null)")"
+    echo -n " ${GITHUB_FTL_VERSION}" >> "${GITHUB_VERSION_FILE}"
 
 else
 
-    CORE_BRANCH="$(get_local_branch /etc/.pihole)"
-    WEB_BRANCH="$(get_local_branch /var/www/html/admin)"
-    FTL_BRANCH="$(pihole-FTL branch)"
+    LOCAL_BRANCH_FILE="/etc/pihole/localbranches"
 
-    echo -n "${CORE_BRANCH} ${WEB_BRANCH} ${FTL_BRANCH}" > "/etc/pihole/localbranches"
+    CORE_BRANCH="$(get_local_branch /etc/.pihole)"
+    echo -n "${CORE_BRANCH}" > "${LOCAL_BRANCH_FILE}"
+    chmod 644 "${LOCAL_BRANCH_FILE}"
+
+    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
+        WEB_BRANCH="$(get_local_branch /var/www/html/admin)"
+        echo -n " ${WEB_BRANCH}" >> "${LOCAL_BRANCH_FILE}"
+    fi
+
+    FTL_BRANCH="$(pihole-FTL branch)"
+    echo -n " ${FTL_BRANCH}" >> "${LOCAL_BRANCH_FILE}"
+
+    LOCAL_VERSION_FILE="/etc/pihole/localversions"
 
     CORE_VERSION="$(get_local_version /etc/.pihole)"
-    WEB_VERSION="$(get_local_version /var/www/html/admin)"
-    FTL_VERSION="$(pihole-FTL version)"
+    echo -n "${CORE_VERSION}" > "${LOCAL_VERSION_FILE}"
+    chmod 644 "${LOCAL_VERSION_FILE}"
 
-    echo -n "${CORE_VERSION} ${WEB_VERSION} ${FTL_VERSION}" > "/etc/pihole/localversions"
+    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
+        WEB_VERSION="$(get_local_version /var/www/html/admin)"
+        echo -n " ${WEB_VERSION}" >> "${LOCAL_VERSION_FILE}"
+    fi
+
+    FTL_VERSION="$(pihole-FTL version)"
+    echo -n " ${FTL_VERSION}" >> "${LOCAL_VERSION_FILE}"
 
 fi

advanced/Scripts/version.sh

@@ -136,8 +136,16 @@ errorOutput() {
 }
 
 defaultOutput() {
+    # Source the setupvars config file
+    # shellcheck disable=SC1091
+    source /etc/pihole/setupVars.conf
+
     versionOutput "pi-hole" "$@"
-    versionOutput "AdminLTE" "$@"
+
+    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
+        versionOutput "AdminLTE" "$@"
+    fi
+
     versionOutput "FTL" "$@"
 }
 

advanced/Scripts/webpage.sh

@@ -17,6 +17,8 @@ readonly FTLconf="/etc/pihole/pihole-FTL.conf"
 # 03 -> wildcards
 readonly dhcpstaticconfig="/etc/dnsmasq.d/04-pihole-static-dhcp.conf"
 
+readonly gravityDBfile="/etc/pihole/gravity.db"
+
 coltable="/opt/pihole/COL_TABLE"
 if [[ -f ${coltable} ]]; then
     source ${coltable}
@@ -36,7 +38,7 @@ Options:
   -e, email           Set an administrative contact address for the Block Page
   -h, --help          Show this help dialog
   -i, interface       Specify dnsmasq's interface listening behavior
-  -l, privacylevel    Set privacy level (0 = lowest, 3 = highest)"
+  -l, privacylevel    Set privacy level (0 = lowest, 4 = highest)"
     exit 0
 }
 
@@ -110,7 +112,7 @@ SetWebPassword() {
         # Prevents a bug if the user presses Ctrl+C and it continues to hide the text typed.
         # So we reset the terminal via stty if the user does press Ctrl+C
         trap '{ echo -e "\nNo password will be set" ; stty sane ; exit 1; }' INT
-        read -s -p "Enter New Password (Blank for no password): " PASSWORD
+        read -s -r -p "Enter New Password (Blank for no password): " PASSWORD
         echo ""
 
     if [ "${PASSWORD}" == "" ]; then
@@ -119,12 +121,13 @@ SetWebPassword() {
         exit 0
     fi
 
-    read -s -p "Confirm Password: " CONFIRM
+    read -s -r -p "Confirm Password: " CONFIRM
     echo ""
     fi
 
     if [ "${PASSWORD}" == "${CONFIRM}" ] ; then
-        hash=$(HashPassword "${PASSWORD}")
+        # We do not wrap this in brackets, otherwise BASH will expand any appropriate syntax
+        hash=$(HashPassword "$PASSWORD")
         # Save hash to file
         change_setting "WEBPASSWORD" "${hash}"
         echo -e "  ${TICK} New password set"
@@ -321,11 +324,18 @@ dhcp-option=option:router,${DHCP_ROUTER}
 dhcp-leasefile=/etc/pihole/dhcp.leases
 #quiet-dhcp
 " > "${dhcpconfig}"
+    chmod 644 "${dhcpconfig}"
 
     if [[ "${PIHOLE_DOMAIN}" != "none" ]]; then
         echo "domain=${PIHOLE_DOMAIN}" >> "${dhcpconfig}"
     fi
 
+    # Sourced from setupVars
+    # shellcheck disable=SC2154
+    if [[ "${DHCP_rapid_commit}" == "true" ]]; then
+        echo "dhcp-rapid-commit" >> "${dhcpconfig}"
+    fi
+
     if [[ "${DHCP_IPv6}" == "true" ]]; then
         echo "#quiet-dhcp6
 #enable-ra
@@ -350,11 +360,20 @@ EnableDHCP() {
     change_setting "DHCP_LEASETIME" "${args[5]}"
     change_setting "PIHOLE_DOMAIN" "${args[6]}"
     change_setting "DHCP_IPv6" "${args[7]}"
+    change_setting "DHCP_rapid_commit" "${args[8]}"
 
     # Remove possible old setting from file
     delete_dnsmasq_setting "dhcp-"
     delete_dnsmasq_setting "quiet-dhcp"
 
+    # If a DHCP client claims that its name is "wpad", ignore that.
+    # This fixes a security hole. see CERT Vulnerability VU#598349
+    # We also ignore "localhost" as Windows behaves strangely if a
+    # device claims this host name
+    add_dnsmasq_setting "dhcp-name-match=set:hostname-ignore,wpad
+dhcp-name-match=set:hostname-ignore,localhost
+dhcp-ignore-names=tag:hostname-ignore"
+
     ProcessDHCPSettings
 
     RestartDNS
@@ -377,19 +396,17 @@ SetWebUILayout() {
 }
 
 CustomizeAdLists() {
-    list="/etc/pihole/adlists.list"
+    local address
+    address="${args[3]}"
 
     if [[ "${args[2]}" == "enable" ]]; then
-        sed -i "\\@${args[3]}@s/^#http/http/g" "${list}"
+        sqlite3 "${gravityDBfile}" "UPDATE adlist SET enabled = 1 WHERE address = '${address}'"
     elif [[ "${args[2]}" == "disable" ]]; then
-        sed -i "\\@${args[3]}@s/^http/#http/g" "${list}"
+        sqlite3 "${gravityDBfile}" "UPDATE adlist SET enabled = 0 WHERE address = '${address}'"
     elif [[ "${args[2]}" == "add" ]]; then
-        if [[ $(grep -c "^${args[3]}$" "${list}") -eq 0 ]] ; then
-            echo "${args[3]}" >> ${list}
-        fi
+        sqlite3 "${gravityDBfile}" "INSERT OR IGNORE INTO adlist (address) VALUES ('${address}')"
     elif [[ "${args[2]}" == "del" ]]; then
-        var=$(echo "${args[3]}" | sed 's/\//\\\//g')
-        sed -i "/${var}/Id" "${list}"
+        sqlite3 "${gravityDBfile}" "DELETE FROM adlist WHERE address = '${address}'"
     else
         echo "Not permitted"
         return 1
@@ -522,17 +539,53 @@ Interfaces:
 
 Teleporter() {
     local datetimestamp=$(date "+%Y-%m-%d_%H-%M-%S")
-    php /var/www/html/admin/scripts/pi-hole/php/teleporter.php > "pi-hole-teleporter_${datetimestamp}.zip"
+    php /var/www/html/admin/scripts/pi-hole/php/teleporter.php > "pi-hole-teleporter_${datetimestamp}.tar.gz"
 }
 
-audit()
+checkDomain()
 {
-    echo "${args[2]}" >> /etc/pihole/auditlog.list
+    local domain validDomain
+    # Convert to lowercase
+    domain="${1,,}"
+    validDomain=$(grep -P "^((-|_)*[a-z\\d]((-|_)*[a-z\\d])*(-|_)*)(\\.(-|_)*([a-z\\d]((-|_)*[a-z\\d])*))*$" <<< "${domain}") # Valid chars check
+    validDomain=$(grep -P "^[^\\.]{1,63}(\\.[^\\.]{1,63})*$" <<< "${validDomain}") # Length of each label
+    echo "${validDomain}"
+}
+
+addAudit()
+{
+    shift # skip "-a"
+    shift # skip "audit"
+    local domains validDomain
+    domains=""
+    for domain in "$@"
+    do
+      # Check domain to be added. Only continue if it is valid
+      validDomain="$(checkDomain "${domain}")"
+      if [[ -n "${validDomain}" ]]; then
+        # Put comma in between domains when there is
+        # more than one domains to be added
+        # SQL INSERT allows adding multiple rows at once using the format
+        ## INSERT INTO table (domain) VALUES ('abc.de'),('fgh.ij'),('klm.no'),('pqr.st');
+        if [[ -n "${domains}" ]]; then
+          domains="${domains},"
+        fi
+        domains="${domains}('${domain}')"
+      fi
+    done
+    # Insert only the domain here. The date_added field will be
+    # filled with its default value (date_added = current timestamp)
+    sqlite3 "${gravityDBfile}" "INSERT INTO domain_audit (domain) VALUES ${domains};"
+}
+
+clearAudit()
+{
+    sqlite3 "${gravityDBfile}" "DELETE FROM domain_audit;"
 }
 
 SetPrivacyLevel() {
-    # Set privacy level. Minimum is 0, maximum is 3
-    if [ "${args[2]}" -ge 0 ] && [ "${args[2]}" -le 3 ]; then
+    # Set privacy level. Minimum is 0, maximum is 4
+    if [ "${args[2]}" -ge 0 ] && [ "${args[2]}" -le 4 ]; then
         changeFTLsetting "PRIVACYLEVEL" "${args[2]}"
     fi
 }
@@ -565,7 +618,8 @@ main() {
         "-i" | "interface"    ) SetListeningMode "$@";;
         "-t" | "teleporter"   ) Teleporter;;
         "adlist"              ) CustomizeAdLists;;
-        "audit"               ) audit;;
+        "audit"               ) addAudit "$@";;
+        "clearaudit"          ) clearAudit;;
         "-l" | "privacylevel" ) SetPrivacyLevel;;
         *                     ) helpFunc;;
     esac

advanced/Templates/gravity.db.sql

@@ -0,0 +1,143 @@
+PRAGMA FOREIGN_KEYS=ON;
+
+CREATE TABLE "group"
+(
+	id INTEGER PRIMARY KEY AUTOINCREMENT,
+	enabled BOOLEAN NOT NULL DEFAULT 1,
+	name TEXT NOT NULL,
+	description TEXT
+);
+
+CREATE TABLE whitelist
+(
+	id INTEGER PRIMARY KEY AUTOINCREMENT,
+	domain TEXT UNIQUE NOT NULL,
+	enabled BOOLEAN NOT NULL DEFAULT 1,
+	date_added INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
+	date_modified INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
+	comment TEXT
+);
+
+CREATE TABLE whitelist_by_group
+(
+	whitelist_id INTEGER NOT NULL REFERENCES whitelist (id),
+	group_id INTEGER NOT NULL REFERENCES "group" (id),
+	PRIMARY KEY (whitelist_id, group_id)
+);
+
+CREATE TABLE blacklist
+(
+	id INTEGER PRIMARY KEY AUTOINCREMENT,
+	domain TEXT UNIQUE NOT NULL,
+	enabled BOOLEAN NOT NULL DEFAULT 1,
+	date_added INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
+	date_modified INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
+	comment TEXT
+);
+
+CREATE TABLE blacklist_by_group
+(
+	blacklist_id INTEGER NOT NULL REFERENCES blacklist (id),
+	group_id INTEGER NOT NULL REFERENCES "group" (id),
+	PRIMARY KEY (blacklist_id, group_id)
+);
+
+CREATE TABLE regex
+(
+	id INTEGER PRIMARY KEY AUTOINCREMENT,
+	domain TEXT UNIQUE NOT NULL,
+	enabled BOOLEAN NOT NULL DEFAULT 1,
+	date_added INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
+	date_modified INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
+	comment TEXT
+);
+
+CREATE TABLE regex_by_group
+(
+	regex_id INTEGER NOT NULL REFERENCES regex (id),
+	group_id INTEGER NOT NULL REFERENCES "group" (id),
+	PRIMARY KEY (regex_id, group_id)
+);
+
+CREATE TABLE adlist
+(
+	id INTEGER PRIMARY KEY AUTOINCREMENT,
+	address TEXT UNIQUE NOT NULL,
+	enabled BOOLEAN NOT NULL DEFAULT 1,
+	date_added INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
+	date_modified INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
+	comment TEXT
+);
+
+CREATE TABLE adlist_by_group
+(
+	adlist_id INTEGER NOT NULL REFERENCES adlist (id),
+	group_id INTEGER NOT NULL REFERENCES "group" (id),
+	PRIMARY KEY (adlist_id, group_id)
+);
+
+CREATE TABLE gravity
+(
+	domain TEXT PRIMARY KEY
+);
+
+CREATE TABLE info
+(
+	property TEXT PRIMARY KEY,
+	value TEXT NOT NULL
+);
+
+INSERT INTO info VALUES("version","1");
+
+CREATE VIEW vw_gravity AS SELECT domain
+    FROM gravity
+    WHERE domain NOT IN (SELECT domain from vw_whitelist);
+
+CREATE VIEW vw_whitelist AS SELECT DISTINCT domain
+    FROM whitelist
+    LEFT JOIN whitelist_by_group ON whitelist_by_group.whitelist_id = whitelist.id
+    LEFT JOIN "group" ON "group".id = whitelist_by_group.group_id
+    WHERE whitelist.enabled = 1 AND (whitelist_by_group.group_id IS NULL OR "group".enabled = 1)
+    ORDER BY whitelist.id;
+
+CREATE TRIGGER tr_whitelist_update AFTER UPDATE ON whitelist
+    BEGIN
+      UPDATE whitelist SET date_modified = (cast(strftime('%s', 'now') as int)) WHERE domain = NEW.domain;
+    END;
+
+CREATE VIEW vw_blacklist AS SELECT DISTINCT domain
+    FROM blacklist
+    LEFT JOIN blacklist_by_group ON blacklist_by_group.blacklist_id = blacklist.id
+    LEFT JOIN "group" ON "group".id = blacklist_by_group.group_id
+    WHERE blacklist.enabled = 1 AND (blacklist_by_group.group_id IS NULL OR "group".enabled = 1)
+    ORDER BY blacklist.id;
+
+CREATE TRIGGER tr_blacklist_update AFTER UPDATE ON blacklist
+    BEGIN
+      UPDATE blacklist SET date_modified = (cast(strftime('%s', 'now') as int)) WHERE domain = NEW.domain;
+    END;
+
+CREATE VIEW vw_regex AS SELECT DISTINCT domain
+    FROM regex
+    LEFT JOIN regex_by_group ON regex_by_group.regex_id = regex.id
+    LEFT JOIN "group" ON "group".id = regex_by_group.group_id
+    WHERE regex.enabled = 1 AND (regex_by_group.group_id IS NULL OR "group".enabled = 1)
+    ORDER BY regex.id;
+
+CREATE TRIGGER tr_regex_update AFTER UPDATE ON regex
+    BEGIN
+      UPDATE regex SET date_modified = (cast(strftime('%s', 'now') as int)) WHERE domain = NEW.domain;
+    END;
+
+CREATE VIEW vw_adlist AS SELECT DISTINCT address
+    FROM adlist
+    LEFT JOIN adlist_by_group ON adlist_by_group.adlist_id = adlist.id
+    LEFT JOIN "group" ON "group".id = adlist_by_group.group_id
+    WHERE adlist.enabled = 1 AND (adlist_by_group.group_id IS NULL OR "group".enabled = 1)
+    ORDER BY adlist.id;
+
+CREATE TRIGGER tr_adlist_update AFTER UPDATE ON adlist
+    BEGIN
+      UPDATE adlist SET date_modified = (cast(strftime('%s', 'now') as int)) WHERE address = NEW.address;
+    END;
+

advanced/Templates/pihole-FTL.conf

@@ -1,84 +0,0 @@
-### This file contains parameters for FTL behavior.
-### At install, all parameters are commented out. The user can select desired options.
-### Options shown are the default configuration. No modification is needed for most
-### installations.
-### Visit https://docs.pi-hole.net/ftldns/configfile/ for more detailed parameter explanations
-
-## Socket Listening
-## Listen only for local socket connections or permit all connections
-## Options: localonly, all
-#SOCKET_LISTENING=localonly
-
-## Query Display
-## Display all queries? Set to no to hide query display
-## Options: yes, no
-#QUERY_DISPLAY=yes
-
-## AAA Query Analysis
-## Allow FTL to analyze AAAA queries from pihole.log?
-## Options: yes, no
-#AAAA_QUERY_ANALYSIS=yes
-
-## Resolve IPv6
-## Should FTL try to resolve IPv6 addresses to host names?
-## Options: yes, no
-#RESOLVE_IPV6=yes
-
-## Resolve IPv4
-## Should FTL try to resolve IPv4 addresses to host names?
-## Options: yes, no
-#RESOLVE_IPV4=yes
-
-## Max Database Days
-## How long should queries be stored in the database (days)?
-## Setting this to 0 disables the database
-## See: https://docs.pi-hole.net/ftldns/database/
-## Options: number of days
-#MAXDBDAYS=365
-
-## Database Interval
-## How often do we store queries in FTL's database (minutes)?
-## See: https://docs.pi-hole.net/ftldns/database/
-## Options: number of minutes
-#DBINTERVAL=1.0
-
-## Database File
-## Specify path and filename of FTL's SQLite3 long-term database.
-## Setting this to DBFILE= disables the database altogether
-## See: https://docs.pi-hole.net/ftldns/database/
-## Option: path to db file
-#DBFILE=/etc/pihole/pihole-FTL.db
-
-## Max Log Age
-## Up to how many hours of queries should be imported from the database and logs (hours)?
-## Maximum is 744 (31 days)
-## Options: number of days
-#MAXLOGAGE=24.0
-
-## FTL Port
-## On which port should FTL be listening?
-## Options: tcp port
-#FTLPORT=4711
-
-## Privacy Level
-## Which privacy level is used?
-## See: https://docs.pi-hole.net/ftldns/privacylevels/
-## Options: 0, 1, 2, 3
-#PRIVACYLEVEL=0
-
-## Ignore Localhost
-## Should FTL ignore queries coming from the local machine?
-## Options: yes, no
-#IGNORE_LOCALHOST=no
-
-## Blocking Mode
-## How should FTL reply to blocked queries?
-## See: https://docs.pi-hole.net/ftldns/blockingmode/
-## Options: NULL, IP-AAAA-NODATA, IP, NXDOMAIN
-#BLOCKINGMODE=NULL
-
-## Regex Debug Mode
-## Controls if FTLDNS should print extended details about regex matching into pihole-FTL.log.
-## See: https://docs.pi-hole.net/ftldns/regex/overview/
-## Options: true, false
-#REGEX_DEBUGMODE=false

advanced/Templates/pihole-FTL.service

@@ -13,7 +13,14 @@ FTLUSER=pihole
 PIDFILE=/var/run/pihole-FTL.pid
 
 get_pid() {
-    pidof "pihole-FTL"
+    # First, try to obtain PID from PIDFILE
+    if [ -s "${PIDFILE}" ]; then
+        cat "${PIDFILE}"
+        return
+    fi
+
+    # If the PIDFILE is empty or not available, obtain the PID using pidof
+    pidof "pihole-FTL" | awk '{print $(NF)}'
 }
 
 is_running() {
@@ -26,17 +33,28 @@ start() {
   if is_running; then
     echo "pihole-FTL is already running"
   else
-    touch /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port /var/log/pihole.log
+    # Touch files to ensure they exist (create if non-existing, preserve if existing)
+    touch /var/log/pihole-FTL.log /var/log/pihole.log
+    touch /run/pihole-FTL.pid /run/pihole-FTL.port
+    touch /etc/pihole/dhcp.leases
     mkdir -p /var/run/pihole
     mkdir -p /var/log/pihole
     chown pihole:pihole /var/run/pihole /var/log/pihole
+    # Remove possible leftovers from previous pihole-FTL processes
+    rm -f /dev/shm/FTL-* 2> /dev/null
     rm /var/run/pihole/FTL.sock 2> /dev/null
-    chown pihole:pihole /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port
-    chown pihole:pihole /etc/pihole /etc/pihole/dhcp.leases /var/log/pihole.log
+    # Ensure that permissions are set so that pihole-FTL can edit all necessary files
+    chown pihole:pihole /run/pihole-FTL.pid /run/pihole-FTL.port
+    chown pihole:pihole /etc/pihole /etc/pihole/dhcp.leases 2> /dev/null
+    chown pihole:pihole /var/log/pihole-FTL.log /var/log/pihole.log
     chmod 0644 /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port /var/log/pihole.log
-    setcap CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_NET_ADMIN+eip "$(which pihole-FTL)"
     echo "nameserver 127.0.0.1" | /sbin/resolvconf -a lo.piholeFTL
-    su -s /bin/sh -c "/usr/bin/pihole-FTL" "$FTLUSER"
+    if setcap CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_NET_ADMIN+eip "$(which pihole-FTL)"; then
+      su -s /bin/sh -c "/usr/bin/pihole-FTL" "$FTLUSER"
+    else
+      echo "Warning: Starting pihole-FTL as root because setting capabilities is not supported on this system"
+      pihole-FTL
+    fi
     echo
   fi
 }
@@ -78,7 +96,7 @@ status() {
     echo "[    ] pihole-FTL is not running"
     exit 1
   fi
-}  
+}
 
 
 ### main logic ###

advanced/Templates/pihole.cron

@@ -16,7 +16,9 @@
 
 # Pi-hole: Update the ad sources once a week on Sunday at a random time in the
 #          early morning. Download any updates from the adlists
-59 1    * * 7   root    PATH="$PATH:/usr/local/bin/" pihole updateGravity
+#          Squash output to log, then splat the log to stdout on error to allow for
+#          standard crontab job error handling.
+59 1    * * 7   root    PATH="$PATH:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log
 
 # Pi-hole: Flush the log daily at 00:00
 #          The flush script will use logrotate if available

advanced/bash-completion/pihole

@@ -7,7 +7,7 @@ _pihole() {
 
 	case "${prev}" in
 		"pihole")
-			opts="admin blacklist checkout chronometer debug disable enable flush help logging query reconfigure regex restartdns status tail uninstall updateGravity updatePihole version wildcard whitelist"
+			opts="admin blacklist checkout chronometer debug disable enable flush help logging query reconfigure regex restartdns status tail uninstall updateGravity updatePihole version wildcard whitelist arpflush"
 			COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
 		;;
 		"whitelist"|"blacklist"|"wildcard"|"regex")
@@ -56,7 +56,7 @@ _pihole() {
 		;;
 		"privacylevel")
 			if ( [[ "$prev2" == "admin" ]] || [[ "$prev2" == "-a" ]] ); then
-				opts_privacy="0 1 2 3"
+				opts_privacy="0 1 2 3 4"
 				COMPREPLY=( $(compgen -W "${opts_privacy}" -- ${cur}) )
 			else 
 				return 1

advanced/index.php

@@ -8,6 +8,8 @@
 
 // Sanitise HTTP_HOST output
 $serverName = htmlspecialchars($_SERVER["HTTP_HOST"]);
+// Remove external ipv6 brackets if any
+$serverName = preg_replace('/^\[(.*)\]$/', '${1}', $serverName);
 
 if (!is_file("/etc/pihole/setupVars.conf"))
   die("[ERROR] File not found: <code>/etc/pihole/setupVars.conf</code>");
@@ -38,13 +40,6 @@ $validExtTypes = array("asp", "htm", "html", "php", "rss", "xml", "");
 // Get extension of current URL
 $currentUrlExt = pathinfo($_SERVER["REQUEST_URI"], PATHINFO_EXTENSION);
 
-// Check if this is served over HTTP or HTTPS
-if(isset($_SERVER['HTTPS']) && $_SERVER['HTTPS'] == "on") {
-    $proto = "https";
-} else {
-    $proto = "http";
-}
-
 // Set mobile friendly viewport
 $viewPort = '<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1"/>';
 
@@ -107,20 +102,30 @@ if ($blocklistglob === array()) {
     die("[ERROR] There are no domain lists generated lists within <code>/etc/pihole/</code>! Please update gravity by running <code>pihole -g</code>, or repair Pi-hole using <code>pihole -r</code>.");
 }
 
-// Set location of adlists file
-if (is_file("/etc/pihole/adlists.list")) {
-    $adLists = "/etc/pihole/adlists.list";
-} elseif (is_file("/etc/pihole/adlists.default")) {
-    $adLists = "/etc/pihole/adlists.default";
+// Get possible non-standard location of FTL's database
+$FTLsettings = parse_ini_file("/etc/pihole/pihole-FTL.conf");
+if (isset($FTLsettings["GRAVITYDB"])) {
+    $gravityDBFile = $FTLsettings["GRAVITYDB"];
 } else {
-    die("[ERROR] File not found: <code>/etc/pihole/adlists.list</code>");
+    $gravityDBFile = "/etc/pihole/gravity.db";
 }
 
-// Get all URLs starting with "http" or "www" from adlists and re-index array numerically
-$adlistsUrls = array_values(preg_grep("/(^http)|(^www)/i", file($adLists, FILE_IGNORE_NEW_LINES)));
+// Connect to gravity.db
+try {
+    $db = new SQLite3($gravityDBFile, SQLITE3_OPEN_READONLY);
+} catch (Exception $exception) {
+    die("[ERROR]: Failed to connect to gravity.db");
+}
+
+// Get all adlist addresses
+$adlistResults = $db->query("SELECT address FROM vw_adlist");
+$adlistsUrls = array();
+while ($row = $adlistResults->fetchArray()) {
+    array_push($adlistsUrls, $row[0]);
+}
 
 if (empty($adlistsUrls))
-    die("[ERROR]: There are no adlist URL's found within <code>$adLists</code>");
+    die("[ERROR]: There are no adlists enabled");
 
 // Get total number of blocklists (Including Whitelist, Blacklist & Wildcard lists)
 $adlistsCount = count($adlistsUrls) + 3;
@@ -227,10 +232,10 @@ setHeader();
   <?=$viewPort ?>
   <meta name="robots" content="noindex,nofollow"/>
   <meta http-equiv="x-dns-prefetch-control" content="off">
-  <link rel="shortcut icon" href="<?=$proto ?>://pi.hole/admin/img/favicon.png" type="image/x-icon"/>
-  <link rel="stylesheet" href="<?=$proto ?>://pi.hole/pihole/blockingpage.css" type="text/css"/>
+  <link rel="shortcut icon" href="//pi.hole/admin/img/favicon.png" type="image/x-icon"/>
+  <link rel="stylesheet" href="//pi.hole/pihole/blockingpage.css" type="text/css"/>
   <title>● <?=$serverName ?></title>
-  <script src="<?=$proto ?>://pi.hole/admin/scripts/vendor/jquery.min.js"></script>
+  <script src="//pi.hole/admin/scripts/vendor/jquery.min.js"></script>
   <script>
     window.onload = function () {
       <?php

advanced/lighttpd.conf.debian

@@ -27,7 +27,7 @@ server.modules = (
 )
 
 server.document-root        = "/var/www/html"
-server.error-handler-404    = "pihole/index.php"
+server.error-handler-404    = "/pihole/index.php"
 server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
 server.errorlog             = "/var/log/lighttpd/error.log"
 server.pid-file             = "/var/run/lighttpd.pid"
@@ -44,9 +44,18 @@ static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
 compress.cache-dir          = "/var/cache/lighttpd/compress/"
 compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )
 
+mimetype.assign   = ( ".png"  => "image/png",
+                      ".jpg"  => "image/jpeg",
+                      ".jpeg" => "image/jpeg",
+                      ".html" => "text/html",
+                      ".css" => "text/css; charset=utf-8",
+                      ".js" => "application/javascript",
+                      ".json" => "application/json",
+                      ".txt"  => "text/plain",
+                      ".svg"  => "image/svg+xml" )
+
 # default listening port for IPv6 falls back to the IPv4 port
 include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
-include_shell "/usr/share/lighttpd/create-mime.assign.pl"
 
 # Prevent Lighttpd from enabling Let's Encrypt SSL for every blocked domain
 #include_shell "/usr/share/lighttpd/include-conf-enabled.pl"
@@ -72,4 +81,5 @@ $HTTP["url"] =~ "^/admin/\.(.*)" {
 }
 
 # Add user chosen options held in external file
+# This uses include_shell instead of an include wildcard for compatibility
 include_shell "cat external.conf 2>/dev/null"

advanced/lighttpd.conf.fedora

@@ -28,7 +28,7 @@ server.modules = (
 )
 
 server.document-root        = "/var/www/html"
-server.error-handler-404    = "pihole/index.php"
+server.error-handler-404    = "/pihole/index.php"
 server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
 server.errorlog             = "/var/log/lighttpd/error.log"
 server.pid-file             = "/var/run/lighttpd.pid"
@@ -90,4 +90,5 @@ $HTTP["url"] =~ "^/admin/\.(.*)" {
 }
 
 # Add user chosen options held in external file
+# This uses include_shell instead of an include wildcard for compatibility
 include_shell "cat external.conf 2>/dev/null"

automated install/uninstall.sh

@@ -55,13 +55,13 @@ fi
 # Compatability
 if [ -x "$(command -v apt-get)" ]; then
     # Debian Family
-    PKG_REMOVE="${PKG_MANAGER} -y remove --purge"
+    PKG_REMOVE=("${PKG_MANAGER}" -y remove --purge)
     package_check() {
         dpkg-query -W -f='${Status}' "$1" 2>/dev/null | grep -c "ok installed"
     }
 elif [ -x "$(command -v rpm)" ]; then
     # Fedora Family
-    PKG_REMOVE="${PKG_MANAGER} remove -y"
+    PKG_REMOVE=("${PKG_MANAGER}" remove -y)
     package_check() {
         rpm -qa | grep "^$1-" > /dev/null
     }
@@ -80,7 +80,7 @@ removeAndPurge() {
                 case ${yn} in
                     [Yy]* )
                         echo -ne "  ${INFO} Removing ${i}...";
-                        ${SUDO} "${PKG_REMOVE} ${i}" &> /dev/null;
+                        ${SUDO} "${PKG_REMOVE[@]}" "${i}" &> /dev/null;
                         echo -e "${OVER}  ${INFO} Removed ${i}";
                         break;;
                     [Nn]* ) echo -e "  ${INFO} Skipped ${i}"; break;;
@@ -106,7 +106,7 @@ removeNoPurge() {
     ${SUDO} rm -rf /var/www/html/pihole &> /dev/null
     ${SUDO} rm -f /var/www/html/index.lighttpd.orig &> /dev/null
 
-    # If the web directory is empty after removing these files, then the parent html folder can be removed.
+    # If the web directory is empty after removing these files, then the parent html directory can be removed.
     if [ -d "/var/www/html" ]; then
         if [[ ! "$(ls -A /var/www/html)" ]]; then
             ${SUDO} rm -rf /var/www/html &> /dev/null
@@ -131,14 +131,16 @@ removeNoPurge() {
         echo -e "  ${TICK} Removed /etc/cron.d/pihole"
     fi
 
-    package_check lighttpd > /dev/null
-    if [[ $? -eq 1 ]]; then
-        ${SUDO} rm -rf /etc/lighttpd/ &> /dev/null
-        echo -e "  ${TICK} Removed lighttpd"
-    else
-        if [ -f /etc/lighttpd/lighttpd.conf.orig ]; then
+    if package_check lighttpd > /dev/null; then
+        if [[ -f /etc/lighttpd/lighttpd.conf.orig ]]; then
             ${SUDO} mv /etc/lighttpd/lighttpd.conf.orig /etc/lighttpd/lighttpd.conf
         fi
+
+        if [[ -f /etc/lighttpd/external.conf ]]; then
+            ${SUDO} rm /etc/lighttpd/external.conf
+        fi
+
+        echo -e "  ${TICK} Removed lighttpd configs"
     fi
 
     ${SUDO} rm -f /etc/dnsmasq.d/adList.conf &> /dev/null
@@ -154,7 +156,7 @@ removeNoPurge() {
 
     # Restore Resolved
     if [[ -e /etc/systemd/resolved.conf.orig ]]; then
-        ${SUDO} cp /etc/systemd/resolved.conf.orig /etc/systemd/resolved.conf
+        ${SUDO} cp -p /etc/systemd/resolved.conf.orig /etc/systemd/resolved.conf
         systemctl reload-or-restart systemd-resolved
     fi
 

gravity.sh

@@ -17,37 +17,35 @@ coltable="/opt/pihole/COL_TABLE"
 source "${coltable}"
 regexconverter="/opt/pihole/wildcard_regex_converter.sh"
 source "${regexconverter}"
+# shellcheck disable=SC1091
+source "/etc/.pihole/advanced/Scripts/database_migration/gravity-db.sh"
 
 basename="pihole"
 PIHOLE_COMMAND="/usr/local/bin/${basename}"
 
 piholeDir="/etc/${basename}"
 
-adListFile="${piholeDir}/adlists.list"
-adListDefault="${piholeDir}/adlists.default"
-
+# Legacy (pre v5.0) list file locations
 whitelistFile="${piholeDir}/whitelist.txt"
 blacklistFile="${piholeDir}/blacklist.txt"
 regexFile="${piholeDir}/regex.list"
+adListFile="${piholeDir}/adlists.list"
 
-adList="${piholeDir}/gravity.list"
-blackList="${piholeDir}/black.list"
 localList="${piholeDir}/local.list"
 VPNList="/etc/openvpn/ipp.txt"
 
+piholeGitDir="/etc/.pihole"
+gravityDBfile="${piholeDir}/gravity.db"
+gravityDBschema="${piholeGitDir}/advanced/Templates/gravity.db.sql"
+optimize_database=false
+
 domainsExtension="domains"
 matterAndLight="${basename}.0.matterandlight.txt"
 parsedMatter="${basename}.1.parsedmatter.txt"
-whitelistMatter="${basename}.2.whitelistmatter.txt"
-accretionDisc="${basename}.3.accretionDisc.txt"
 preEventHorizon="list.preEventHorizon"
 
-skipDownload="false"
-
 resolver="pihole-FTL"
 
-haveSourceUrls=true
-
 # Source setupVars from install script
 setupVars="${piholeDir}/setupVars.conf"
 if [[ -f "${setupVars}" ]];then
@@ -68,17 +66,136 @@ else
   exit 1
 fi
 
+# Source pihole-FTL from install script
+pihole_FTL="${piholeDir}/pihole-FTL.conf"
+if [[ -f "${pihole_FTL}" ]]; then
+  source "${pihole_FTL}"
+fi
+
+if [[ -z "${BLOCKINGMODE}" ]] ; then
+  BLOCKINGMODE="NULL"
+fi
+
 # Determine if superseded pihole.conf exists
 if [[ -r "${piholeDir}/pihole.conf" ]]; then
   echo -e "  ${COL_LIGHT_RED}Ignoring overrides specified within pihole.conf! ${COL_NC}"
 fi
 
+# Generate new sqlite3 file from schema template
+generate_gravity_database() {
+  sqlite3 "${gravityDBfile}" < "${gravityDBschema}"
+}
+
+# Import domains from file and store them in the specified database table
+database_table_from_file() {
+  # Define locals
+  local table source backup_path backup_file
+  table="${1}"
+  source="${2}"
+  backup_path="${piholeDir}/migration_backup"
+  backup_file="${backup_path}/$(basename "${2}")"
+
+  # Truncate table
+  output=$( { sqlite3 "${gravityDBfile}" <<< "DELETE FROM ${table};"; } 2>&1 )
+  status="$?"
+
+  if [[ "${status}" -ne 0 ]]; then
+    echo -e "\\n  ${CROSS} Unable to truncate ${table} database ${gravityDBfile}\\n  ${output}"
+    gravity_Cleanup "error"
+  fi
+
+  local tmpFile
+  tmpFile="$(mktemp -p "/tmp" --suffix=".gravity")"
+  local timestamp
+  timestamp="$(date --utc +'%s')"
+  local inputfile
+  if [[ "${table}" == "gravity" ]]; then
+    # No need to modify the input data for the gravity table
+    inputfile="${source}"
+  else
+    # Apply format for white-, blacklist, regex, and adlist tables
+    # Read file line by line
+    local rowid
+    declare -i rowid
+    rowid=1
+    grep -v '^ *#' < "${source}" | while IFS= read -r domain
+    do
+      # Only add non-empty lines
+      if [[ -n "${domain}" ]]; then
+        if [[ "${table}" == "domain_audit" ]]; then
+          # domain_audit table format (no enable or modified fields)
+          echo "${rowid},\"${domain}\",${timestamp}" >> "${tmpFile}"
+        else
+          # White-, black-, and regexlist format
+          echo "${rowid},\"${domain}\",1,${timestamp},${timestamp},\"Migrated from ${source}\"" >> "${tmpFile}"
+        fi
+        rowid+=1
+      fi
+    done
+    inputfile="${tmpFile}"
+  fi
+  # Store domains in database table specified by ${table}
+  # Use printf as .mode and .import need to be on separate lines
+  # see https://unix.stackexchange.com/a/445615/83260
+  output=$( { printf ".timeout 10000\\n.mode csv\\n.import \"%s\" %s\\n" "${inputfile}" "${table}" | sqlite3 "${gravityDBfile}"; } 2>&1 )
+  status="$?"
+
+  if [[ "${status}" -ne 0 ]]; then
+    echo -e "\\n  ${CROSS} Unable to fill table ${table} in database ${gravityDBfile}\\n  ${output}"
+    gravity_Cleanup "error"
+  fi
+
+  # Delete tmpfile
+  rm "${tmpFile}" > /dev/null 2>&1 || \
+      echo -e "  ${CROSS} Unable to remove ${tmpFile}"
+
+  # Move source file to backup directory, create directory if not existing
+  mkdir -p "${backup_path}"
+  mv "${source}" "${backup_file}" 2> /dev/null || \
+      echo -e "  ${CROSS} Unable to backup ${source} to ${backup_path}"
+}
+
+# Migrate pre-v5.0 list files to database-based Pi-hole versions
+migrate_to_database() {
+  # Create database file only if not present
+  if [ ! -e "${gravityDBfile}" ]; then
+    # Create new database file - note that this will be created in version 1
+    echo -e "  ${INFO} Creating new gravity database"
+    generate_gravity_database
+
+    # Migrate list files to new database
+    if [ -e "${adListFile}" ]; then
+      # Store adlist domains in database
+      echo -e "  ${INFO} Migrating content of ${adListFile} into new database"
+      database_table_from_file "adlist" "${adListFile}"
+    fi
+    if [ -e "${blacklistFile}" ]; then
+      # Store blacklisted domains in database
+      echo -e "  ${INFO} Migrating content of ${blacklistFile} into new database"
+      database_table_from_file "blacklist" "${blacklistFile}"
+    fi
+    if [ -e "${whitelistFile}" ]; then
+      # Store whitelisted domains in database
+      echo -e "  ${INFO} Migrating content of ${whitelistFile} into new database"
+      database_table_from_file "whitelist" "${whitelistFile}"
+    fi
+    if [ -e "${regexFile}" ]; then
+      # Store regex domains in database
+      echo -e "  ${INFO} Migrating content of ${regexFile} into new database"
+      database_table_from_file "regex" "${regexFile}"
+    fi
+  fi
+
+  # Check if gravity database needs to be updated
+  upgrade_gravityDB "${gravityDBfile}" "${piholeDir}"
+}
+
 # Determine if DNS resolution is available before proceeding
 gravity_CheckDNSResolutionAvailable() {
   local lookupDomain="pi.hole"
 
-  # Determine if $localList does not exist
-  if [[ ! -e "${localList}" ]]; then
+  # Determine if $localList does not exist, and ensure it is not empty
+  if [[ ! -e "${localList}" ]] || [[ -s "${localList}" ]]; then
     lookupDomain="raw.githubusercontent.com"
   fi
 
@@ -129,19 +246,13 @@ gravity_CheckDNSResolutionAvailable() {
   gravity_CheckDNSResolutionAvailable
 }
 
-# Retrieve blocklist URLs and parse domains from adlists.list
+# Retrieve blocklist URLs and parse domains from adlist.list
 gravity_GetBlocklistUrls() {
   echo -e "  ${INFO} ${COL_BOLD}Neutrino emissions detected${COL_NC}..."
 
-  if [[ -f "${adListDefault}" ]] && [[ -f "${adListFile}" ]]; then
-    # Remove superceded $adListDefault file
-    rm "${adListDefault}" 2> /dev/null || \
-      echo -e "  ${CROSS} Unable to remove ${adListDefault}"
-  fi
-
-  # Retrieve source URLs from $adListFile
-  # Logic: Remove comments and empty lines
-  mapfile -t sources <<< "$(grep -v -E "^(#|$)" "${adListFile}" 2> /dev/null)"
+  # Retrieve source URLs from gravity database
+  # We source only enabled adlists, sqlite3 stores boolean values as 0 (false) or 1 (true)
+  mapfile -t sources <<< "$(sqlite3 "${gravityDBfile}" "SELECT address FROM vw_adlist;" 2> /dev/null)"
 
   # Parse source domains from $sources
   mapfile -t sourceDomains <<< "$(
@@ -158,11 +269,12 @@ gravity_GetBlocklistUrls() {
 
   if [[ -n "${sources[*]}" ]] && [[ -n "${sourceDomains[*]}" ]]; then
     echo -e "${OVER}  ${TICK} ${str}"
+    return 0
   else
     echo -e "${OVER}  ${CROSS} ${str}"
     echo -e "  ${INFO} No source list found, or it is empty"
     echo ""
-    haveSourceUrls=false
+    return 1
   fi
 }
 
@@ -182,7 +294,7 @@ gravity_SetDownloadOptions() {
     activeDomains[$i]="${saveLocation}"
 
     # Default user-agent (for Cloudflare's Browser Integrity Check: https://support.cloudflare.com/hc/en-us/articles/200170086-What-does-the-Browser-Integrity-Check-do-)
-    agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36"
+    agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36"
 
     # Provide special commands for blocklists which may need them
     case "${domain}" in
@@ -190,18 +302,16 @@ gravity_SetDownloadOptions() {
       *) cmd_ext="";;
     esac
 
-    if [[ "${skipDownload}" == false ]]; then
-      echo -e "  ${INFO} Target: ${domain} (${url##*/})"
-      gravity_DownloadBlocklistFromUrl "${url}" "${cmd_ext}" "${agent}"
-      echo ""
-    fi
+    echo -e "  ${INFO} Target: ${domain} (${url##*/})"
+    gravity_DownloadBlocklistFromUrl "${url}" "${cmd_ext}" "${agent}"
+    echo ""
   done
   gravity_Blackbody=true
 }
 
 # Download specified URL and perform checks on HTTP status and file content
 gravity_DownloadBlocklistFromUrl() {
-  local url="${1}" cmd_ext="${2}" agent="${3}" heisenbergCompensator="" patternBuffer str httpCode success=""
+  local url="${1}" cmd_ext="${2}" agent="${3}" heisenbergCompensator="" patternBuffer str httpCode success="" compression
 
   # Create temp file to store content on disk instead of RAM
   patternBuffer=$(mktemp -p "/tmp" --suffix=".phgpb")
@@ -216,8 +326,51 @@ gravity_DownloadBlocklistFromUrl() {
 
   str="Status:"
   echo -ne "  ${INFO} ${str} Pending..."
+  blocked=false
+  case $BLOCKINGMODE in
+    "IP-NODATA-AAAA"|"IP")
+        if [[ $(dig "${domain}" +short | grep "${IPV4_ADDRESS}" -c) -ge 1 ]]; then
+          blocked=true
+        fi;;
+    "NXDOMAIN")
+        if [[ $(dig "${domain}" | grep "NXDOMAIN" -c) -ge 1 ]]; then
+          blocked=true
+        fi;;
+    "NULL"|*)
+        if [[ $(dig "${domain}" +short | grep "0.0.0.0" -c) -ge 1 ]]; then
+          blocked=true
+        fi;;
+   esac
+
+  if [[ "${blocked}" == true ]]; then
+    printf -v ip_addr "%s" "${PIHOLE_DNS_1%#*}"
+    if [[ ${PIHOLE_DNS_1} != *"#"* ]]; then
+        port=53
+    else
+        printf -v port "%s" "${PIHOLE_DNS_1#*#}"
+    fi
+    ip=$(dig "@${ip_addr}" -p "${port}" +short "${domain}")
+    if [[ $(echo "${url}" | awk -F '://' '{print $1}') = "https" ]]; then
+      port=443;
+    else port=80
+    fi
+    bad_list=$(pihole -q -adlist "${domain}" | head -n1 | awk -F 'Match found in ' '{print $2}')
+    echo -e "${OVER}  ${CROSS} ${str} ${domain} is blocked by ${bad_list%:}. Using DNS on ${PIHOLE_DNS_1} to download ${url}";
+    echo -ne "  ${INFO} ${str} Pending..."
+    cmd_ext="--resolve $domain:$port:$ip $cmd_ext"
+  fi
+
+  # Use compression to reduce the amount of data that is transfered
+  # between the Pi-hole and the ad list provider. Use this feature
+  # only if it is supported by the locally available version of curl
+  if curl -V | grep -q "Features:.* libz"; then
+    compression="--compressed"
+  else
+    compression=""
+  fi
+
   # shellcheck disable=SC2086
-  httpCode=$(curl -s -L ${cmd_ext} ${heisenbergCompensator} -w "%{http_code}" -A "${agent}" "${url}" -o "${patternBuffer}" 2> /dev/null)
+  httpCode=$(curl -s -L ${compression} ${cmd_ext} ${heisenbergCompensator} -w "%{http_code}" -A "${agent}" "${url}" -o "${patternBuffer}" 2> /dev/null)
 
   case $url in
     # Did we "download" a local file?
@@ -278,14 +431,17 @@ gravity_ParseFileIntoDomains() {
     # Most of the lists downloaded are already in hosts file format but the spacing/formating is not contigious
     # This helps with that and makes it easier to read
     # It also helps with debugging so each stage of the script can be researched more in depth
-    # Awk -F splits on given IFS, we grab the right hand side (chops trailing #coments and /'s to grab the domain only.
-    # Last awk command takes non-commented lines and if they have 2 fields, take the right field (the domain) and leave
-    # the left (IP address), otherwise grab the single field.
-
-    < ${source} awk -F '#' '{print $1}' | \
-    awk -F '/' '{print $1}' | \
-    awk '($1 !~ /^#/) { if (NF>1) {print $2} else {print $1}}' | \
-    sed -nr -e 's/\.{2,}/./g' -e '/\./p' >  ${destination}
+    # 1) Remove carriage returns
+    # 2) Convert all characters to lowercase
+    # 3) Remove lines containing "#" or "/"
+    # 4) Remove leading tabs, spaces, etc.
+    # 5) Delete lines not matching domain names
+    < "${source}" tr -d '\r' | \
+    tr '[:upper:]' '[:lower:]' | \
+    sed -r '/(\/|#).*$/d' | \
+    sed -r 's/^.*\s+//g' | \
+    sed -r '/([^\.]+\.)+[^\.]{2,}/!d' >  "${destination}"
+    chmod 644 "${destination}"
     return 0
   fi
 
@@ -316,6 +472,7 @@ gravity_ParseFileIntoDomains() {
       if($0 ~ /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/) { $0="" }
       if($0) { print $0 }
     }' "${source}" > "${destination}"
+    chmod 644 "${destination}"
 
     # Determine if there are Adblock exception rules
     # https://adblockplus.org/filters
@@ -333,6 +490,7 @@ gravity_ParseFileIntoDomains() {
       # Remove exceptions
       comm -23 "${destination}" <(sort "${destination}.exceptionsFile.tmp") > "${source}"
       mv "${source}" "${destination}"
+      chmod 644 "${destination}"
     fi
 
     echo -e "${OVER}  ${TICK} Format: Adblock"
@@ -356,11 +514,13 @@ gravity_ParseFileIntoDomains() {
       # Print if nonempty
       length { print }
     ' "${source}" 2> /dev/null > "${destination}"
+    chmod 644 "${destination}"
 
     echo -e "${OVER}  ${TICK} Format: URL"
   else
     # Default: Keep hosts/domains file in same format as it was downloaded
     output=$( { mv "${source}" "${destination}"; } 2>&1 )
+    chmod 644 "${destination}"
 
     if [[ ! -e "${destination}" ]]; then
       echo -e "\\n  ${CROSS} Unable to move tmp file to ${piholeDir}
@@ -375,12 +535,11 @@ gravity_ConsolidateDownloadedBlocklists() {
   local str lastLine
 
   str="Consolidating blocklists"
-  if [[ "${haveSourceUrls}" == true ]]; then
-    echo -ne "  ${INFO} ${str}..."
-  fi
+  echo -ne "  ${INFO} ${str}..."
 
   # Empty $matterAndLight if it already exists, otherwise, create it
   : > "${piholeDir}/${matterAndLight}"
+  chmod 644 "${piholeDir}/${matterAndLight}"
 
   # Loop through each *.domains file
   for i in "${activeDomains[@]}"; do
@@ -396,9 +555,8 @@ gravity_ConsolidateDownloadedBlocklists() {
       fi
     fi
   done
-  if [[ "${haveSourceUrls}" == true ]]; then
-    echo -e "${OVER}  ${TICK} ${str}"
-  fi
+  echo -e "${OVER}  ${TICK} ${str}"
+
 }
 
 # Parse consolidated list into (filtered, unique) domains-only format
@@ -406,67 +564,45 @@ gravity_SortAndFilterConsolidatedList() {
   local str num
 
   str="Extracting domains from blocklists"
-  if [[ "${haveSourceUrls}" == true ]]; then
-    echo -ne "  ${INFO} ${str}..."
-  fi
+  echo -ne "  ${INFO} ${str}..."
 
-  # Parse into hosts file
+  # Parse into file
   gravity_ParseFileIntoDomains "${piholeDir}/${matterAndLight}" "${piholeDir}/${parsedMatter}"
 
   # Format $parsedMatter line total as currency
   num=$(printf "%'.0f" "$(wc -l < "${piholeDir}/${parsedMatter}")")
-  if [[ "${haveSourceUrls}" == true ]]; then
-    echo -e "${OVER}  ${TICK} ${str}"
-  fi
-  echo -e "  ${INFO} Number of domains being pulled in by gravity: ${COL_BLUE}${num}${COL_NC}"
+
+  echo -e "${OVER}  ${TICK} ${str}"
+  echo -e "  ${INFO} Gravity pulled in ${COL_BLUE}${num}${COL_NC} domains"
 
   str="Removing duplicate domains"
-  if [[ "${haveSourceUrls}" == true ]]; then
-    echo -ne "  ${INFO} ${str}..."
-  fi
-
+  echo -ne "  ${INFO} ${str}..."
   sort -u "${piholeDir}/${parsedMatter}" > "${piholeDir}/${preEventHorizon}"
+  chmod 644 "${piholeDir}/${preEventHorizon}"
+  echo -e "${OVER}  ${TICK} ${str}"
 
-  if [[ "${haveSourceUrls}" == true ]]; then
-    echo -e "${OVER}  ${TICK} ${str}"
-    # Format $preEventHorizon line total as currency
-    num=$(printf "%'.0f" "$(wc -l < "${piholeDir}/${preEventHorizon}")")
-    echo -e "  ${INFO} Number of unique domains trapped in the Event Horizon: ${COL_BLUE}${num}${COL_NC}"
-  fi
+  # Format $preEventHorizon line total as currency
+  num=$(printf "%'.0f" "$(wc -l < "${piholeDir}/${preEventHorizon}")")
+  str="Storing ${COL_BLUE}${num}${COL_NC} unique blocking domains in database"
+  echo -ne "  ${INFO} ${str}..."
+  database_table_from_file "gravity" "${piholeDir}/${preEventHorizon}"
+  echo -e "${OVER}  ${TICK} ${str}"
 }
 
-# Whitelist user-defined domains
-gravity_Whitelist() {
-  local num str
-
-  if [[ ! -f "${whitelistFile}" ]]; then
-    echo -e "  ${INFO} Nothing to whitelist!"
-    return 0
-  fi
-
-  num=$(wc -l < "${whitelistFile}")
-  str="Number of whitelisted domains: ${num}"
-  echo -ne "  ${INFO} ${str}..."
-
-  # Print everything from preEventHorizon into whitelistMatter EXCEPT domains in $whitelistFile
-  comm -23 "${piholeDir}/${preEventHorizon}" <(sort "${whitelistFile}") > "${piholeDir}/${whitelistMatter}"
-
-  echo -e "${OVER}  ${INFO} ${str}"
+# Report number of entries in a table
+gravity_Table_Count() {
+  local table="${1}"
+  local str="${2}"
+  local num
+  num="$(sqlite3 "${gravityDBfile}" "SELECT COUNT(*) FROM ${table} WHERE enabled = 1;")"
+  echo -e "  ${INFO} Number of ${str}: ${num}"
 }
 
 # Output count of blacklisted domains and regex filters
-gravity_ShowBlockCount() {
-  local num
-
-  if [[ -f "${blacklistFile}" ]]; then
-    num=$(printf "%'.0f" "$(wc -l < "${blacklistFile}")")
-    echo -e "  ${INFO} Number of blacklisted domains: ${num}"
-  fi
-
-  if [[ -f "${regexFile}" ]]; then
-    num=$(grep -c "^(?!#)" "${regexFile}")
-    echo -e "  ${INFO} Number of regex filters: ${num}"
-  fi
+gravity_ShowCount() {
+  gravity_Table_Count "blacklist" "blacklisted domains"
+  gravity_Table_Count "whitelist" "whitelisted domains"
+  gravity_Table_Count "regex" "regex filters"
 }
 
 # Parse list of domains into hosts format
@@ -486,7 +622,7 @@ gravity_ParseDomainsIntoHosts() {
 }
 
 # Create "localhost" entries into hosts format
-gravity_ParseLocalDomains() {
+gravity_generateLocalList() {
   local hostname
 
   if [[ -s "/etc/hostname" ]]; then
@@ -502,6 +638,7 @@ gravity_ParseLocalDomains() {
 
   # Empty $localList if it already exists, otherwise, create it
   : > "${localList}"
+  chmod 644 "${localList}"
 
   gravity_ParseDomainsIntoHosts "${localList}.tmp" "${localList}"
 
@@ -511,40 +648,6 @@ gravity_ParseLocalDomains() {
   fi
 }
 
-# Create primary blacklist entries
-gravity_ParseBlacklistDomains() {
-  local output status
-
-  # Empty $accretionDisc if it already exists, otherwise, create it
-  : > "${piholeDir}/${accretionDisc}"
-
-  if [[ -f "${piholeDir}/${whitelistMatter}" ]]; then
-    mv "${piholeDir}/${whitelistMatter}" "${piholeDir}/${accretionDisc}"
-  else
-    # There was no whitelist file, so use preEventHorizon instead of whitelistMatter.
-    mv "${piholeDir}/${preEventHorizon}" "${piholeDir}/${accretionDisc}"
-  fi
-
-  # Move the file over as /etc/pihole/gravity.list so dnsmasq can use it
-  output=$( { mv "${piholeDir}/${accretionDisc}" "${adList}"; } 2>&1 )
-  status="$?"
-
-  if [[ "${status}" -ne 0 ]]; then
-    echo -e "\\n  ${CROSS} Unable to move ${accretionDisc} from ${piholeDir}\\n  ${output}"
-    gravity_Cleanup "error"
-  fi
-}
-
-# Create user-added blacklist entries
-gravity_ParseUserDomains() {
-  if [[ ! -f "${blacklistFile}" ]]; then
-    return 0
-  fi
-  # Copy the file over as /etc/pihole/black.list so dnsmasq can use it
-  cp "${blacklistFile}" "${blackList}" 2> /dev/null || \
-    echo -e "\\n  ${CROSS} Unable to move ${blacklistFile##*/} to ${piholeDir}"
-}
-
 # Trap Ctrl-C
 gravity_Trap() {
   trap '{ echo -e "\\n\\n  ${INFO} ${COL_LIGHT_RED}User-abort detected${COL_NC}"; gravity_Cleanup "error"; }' INT
@@ -576,6 +679,21 @@ gravity_Cleanup() {
 
   echo -e "${OVER}  ${TICK} ${str}"
 
+  if ${optimize_database} ; then
+    str="Optimizing domains database"
+    echo -ne "  ${INFO} ${str}..."
+    # Run VACUUM command on database to optimize it
+    output=$( { sqlite3 "${gravityDBfile}" "VACUUM;"; } 2>&1 )
+    status="$?"
+
+    if [[ "${status}" -ne 0 ]]; then
+      echo -e "\\n  ${CROSS} Unable to optimize gravity database ${gravityDBfile}\\n  ${output}"
+      error="error"
+    else
+      echo -e "${OVER}  ${TICK} ${str}"
+    fi
+  fi
+
   # Only restart DNS service if offline
   if ! pidof ${resolver} &> /dev/null; then
     "${PIHOLE_COMMAND}" restartdns
@@ -602,17 +720,17 @@ Options:
 for var in "$@"; do
   case "${var}" in
     "-f" | "--force" ) forceDelete=true;;
+    "-o" | "--optimize" ) optimize_database=true;;
     "-h" | "--help" ) helpFunc;;
-    "-sd" | "--skip-download" ) skipDownload=true;;
-    "-b" | "--blacklist-only" ) listType="blacklist";;
-    "-w" | "--whitelist-only" ) listType="whitelist";;
-    "-wild" | "--wildcard-only" ) listType="wildcard"; dnsRestartType="restart";;
   esac
 done
 
 # Trap Ctrl-C
 gravity_Trap
 
+# Move possibly existing legacy files to the gravity database
+migrate_to_database
+
 if [[ "${forceDelete:-}" == true ]]; then
   str="Deleting existing list cache"
   echo -ne "${INFO} ${str}..."
@@ -621,54 +739,24 @@ if [[ "${forceDelete:-}" == true ]]; then
   echo -e "${OVER}  ${TICK} ${str}"
 fi
 
-# Determine which functions to run
-if [[ "${skipDownload}" == false ]]; then
-  # Gravity needs to download blocklists
-  gravity_CheckDNSResolutionAvailable
-  gravity_GetBlocklistUrls
-  if [[ "${haveSourceUrls}" == true ]]; then
-    gravity_SetDownloadOptions
-  fi
+# Gravity downloads blocklists next
+gravity_CheckDNSResolutionAvailable
+if gravity_GetBlocklistUrls; then
+  gravity_SetDownloadOptions
+  # Build preEventHorizon
   gravity_ConsolidateDownloadedBlocklists
   gravity_SortAndFilterConsolidatedList
-else
-  # Gravity needs to modify Blacklist/Whitelist/Wildcards
-  echo -e "  ${INFO} Using cached Event Horizon list..."
-  numberOf=$(printf "%'.0f" "$(wc -l < "${piholeDir}/${preEventHorizon}")")
-  echo -e "  ${INFO} ${COL_BLUE}${numberOf}${COL_NC} unique domains trapped in the Event Horizon"
 fi
 
-# Perform when downloading blocklists, or modifying the whitelist
-if [[ "${skipDownload}" == false ]] || [[ "${listType}" == "whitelist" ]]; then
-  gravity_Whitelist
-fi
-
-convert_wildcard_to_regex
-gravity_ShowBlockCount
-
-# Perform when downloading blocklists, or modifying the white/blacklist (not wildcards)
-if [[ "${skipDownload}" == false ]] || [[ "${listType}" == *"list" ]]; then
-  str="Parsing domains into hosts format"
-  echo -ne "  ${INFO} ${str}..."
-
-  gravity_ParseUserDomains
-
-  # Perform when downloading blocklists
-  if [[ ! "${listType:-}" == "blacklist" ]]; then
-    gravity_ParseLocalDomains
-    gravity_ParseBlacklistDomains
-  fi
-
-  echo -e "${OVER}  ${TICK} ${str}"
-
-  gravity_Cleanup
-fi
+# Create local.list
+gravity_generateLocalList
+gravity_ShowCount
 
+gravity_Cleanup
 echo ""
 
 # Determine if DNS has been restarted by this instance of gravity
 if [[ -z "${dnsWasOffline:-}" ]]; then
-  # Use "force-reload" when restarting dnsmasq for everything but Wildcards
-  "${PIHOLE_COMMAND}" restartdns "${dnsRestartType:-force-reload}"
+  "${PIHOLE_COMMAND}" restartdns reload
 fi
 "${PIHOLE_COMMAND}" status

manpages/pihole-FTL.conf.5

@@ -64,7 +64,7 @@ pihole-FTL.conf - FTL's config file
     On which port should FTL be listening?
 .br
 
-\fBPRIVACYLEVEL=0|1|2|3\fR
+\fBPRIVACYLEVEL=0|1|2|3|4\fR
 .br
     Which privacy level is used?
 .br
@@ -74,7 +74,9 @@ pihole-FTL.conf - FTL's config file
 .br
     2 - hide domains and clients
 .br
-    3 - paranoia mode (hide everything)
+    3 - anonymous mode (hide everything)
+.br
+    4 - disable all statistics
 .br
 
 \fBIGNORE_LOCALHOST=no|yes\fR

manpages/pihole.8

@@ -35,7 +35,7 @@ pihole -g\fR
 .br
 \fBpihole\fR \fB-l\fR (\fBon|off|off noflush\fR)
 .br
-\fBpihole -up \fR[--checkonly]
+\fBpihole -up \fR[--check-only]
 .br
 \fBpihole -v\fR [-p|-a|-f] [-c|-l|-hash]
 .br
@@ -134,7 +134,7 @@ Available commands and options:
       -i, interface     Specify dnsmasq's interface listening behavior
 .br
       -l, privacylevel  <level> Set privacy level
-                        (0 = lowest, 3 = highest)
+                        (0 = lowest, 4 = highest)
 .br
 
 \fB-c, chronometer\fR	[options]
@@ -351,6 +351,12 @@ Switching Pi-hole subsystem branches
 .br
     Switch to core development branch
 .br
+
+\fBpihole arpflush\fR
+.br
+    Flush information stored in Pi-hole's network tables
+.br
+
 .SH "SEE ALSO"
 
 \fBlighttpd\fR(8), \fBpihole-FTL\fR(8)

pihole

@@ -10,23 +10,17 @@
 # Please see LICENSE file for your rights under this license.
 
 readonly PI_HOLE_SCRIPT_DIR="/opt/pihole"
-readonly wildcardlist="/etc/dnsmasq.d/03-pihole-wildcard.conf"
+
+# setupVars is not readonly here because in some functions (checkout),
+# it might get set again when the installer is sourced. This causes an
+# error due to modifying a readonly variable.
+setupVars="/etc/pihole/setupVars.conf"
+
 readonly colfile="${PI_HOLE_SCRIPT_DIR}/COL_TABLE"
 source "${colfile}"
 
 resolver="pihole-FTL"
 
-# Must be root to use this tool
-if [[ ! $EUID -eq 0 ]];then
-  if [[ -x "$(command -v sudo)" ]]; then
-    exec sudo bash "$0" "$@"
-    exit $?
-  else
-    echo -e "  ${CROSS} sudo is needed to run pihole commands.  Please run this script as root or install sudo."
-    exit 1
-  fi
-fi
-
 webpageFunc() {
   source "${PI_HOLE_SCRIPT_DIR}/webpage.sh"
   main "$@"
@@ -60,6 +54,11 @@ flushFunc() {
   exit 0
 }
 
+arpFunc() {
+  "${PI_HOLE_SCRIPT_DIR}"/piholeARPTable.sh "$@"
+  exit 0
+}
+
 updatePiholeFunc() {
   shift
   "${PI_HOLE_SCRIPT_DIR}"/update.sh "$@"
@@ -73,7 +72,7 @@ reconfigurePiholeFunc() {
 
 updateGravityFunc() {
   "${PI_HOLE_SCRIPT_DIR}"/gravity.sh "$@"
-  exit 0
+  exit $?
 }
 
 queryFunc() {
@@ -127,9 +126,11 @@ restartDNS() {
 
   if [[ "${status}" -eq 0 ]]; then
     [[ -t 1 ]] && echo -e "${OVER}  ${TICK} ${str}"
+    return 0
   else
     [[ ! -t 1 ]] && local OVER=""
     echo -e "${OVER}  ${CROSS} ${output}"
+    return 1
   fi
 }
 
@@ -146,10 +147,9 @@ Time:
 
   elif [[ "${1}" == "0" ]]; then
     # Disable Pi-hole
-    sed -i 's/^addn-hosts=\/etc\/pihole\/gravity.list/#addn-hosts=\/etc\/pihole\/gravity.list/' /etc/dnsmasq.d/01-pihole.conf
-    sed -i 's/^addn-hosts=\/etc\/pihole\/black.list/#addn-hosts=\/etc\/pihole\/black.list/' /etc/dnsmasq.d/01-pihole.conf
-    if [[ -e "$wildcardlist" ]]; then
-      mv "$wildcardlist" "/etc/pihole/wildcard.list"
+    if grep -cq "BLOCKING_ENABLED=false" "${setupVars}"; then
+      echo -e "  ${INFO} Blocking already disabled, nothing to do"
+      exit 0
     fi
     if [[ $# > 1 ]]; then
       local error=false
@@ -187,19 +187,23 @@ Time:
       fi
 
       local str="Pi-hole Disabled"
+      sed -i "/BLOCKING_ENABLED=/d" "${setupVars}"
+      echo "BLOCKING_ENABLED=false" >> "${setupVars}"
     fi
   else
     # Enable Pi-hole
+    if grep -cq "BLOCKING_ENABLED=true" "${setupVars}"; then
+      echo -e "  ${INFO} Blocking already enabled, nothing to do"
+      exit 0
+    fi
     echo -e "  ${INFO} Enabling blocking"
     local str="Pi-hole Enabled"
 
-    sed -i 's/^#addn-hosts/addn-hosts/' /etc/dnsmasq.d/01-pihole.conf
-    if [[ -e "/etc/pihole/wildcard.list" ]]; then
-      mv "/etc/pihole/wildcard.list" "$wildcardlist"
-    fi
+    sed -i "/BLOCKING_ENABLED=/d" "${setupVars}"
+    echo "BLOCKING_ENABLED=true" >> "${setupVars}"
   fi
 
-  restartDNS
+  restartDNS reload
 
   echo -e "${OVER}  ${TICK} ${str}"
 }
@@ -242,8 +246,6 @@ Options:
 }
 
 statusFunc() {
-  local addnConfigs
-
   # Determine if service is running on port 53 (Cr: https://superuser.com/a/806331)
   if (echo > /dev/tcp/127.0.0.1/53) >/dev/null 2>&1; then
     if [[ "${1}" != "web" ]]; then
@@ -257,16 +259,14 @@ statusFunc() {
     return 0
   fi
 
-  # Determine if Pi-hole's addn-hosts configs are commented out
-  addnConfigs=$(grep -i "addn-hosts=/" /etc/dnsmasq.d/01-pihole.conf)
-
-  if [[ "${addnConfigs}" =~ "#" ]]; then
+  # Determine if Pi-hole's blocking is enabled
+  if grep -q "BLOCKING_ENABLED=false" /etc/pihole/setupVars.conf; then
     # A config is commented out
     case "${1}" in
       "web") echo 0;;
       *) echo -e "  ${CROSS} Pi-hole blocking is Disabled";;
     esac
-  elif [[ -n "${addnConfigs}" ]]; then
+  elif grep -q "BLOCKING_ENABLED=true" /etc/pihole/setupVars.conf;  then
     # Configs are set
     case "${1}" in
       "web") echo 1;;
@@ -276,11 +276,10 @@ statusFunc() {
     # No configs were found
     case "${1}" in
       "web") echo 99;;
-      *) echo -e "  ${INFO} No hosts file linked to dnsmasq, adding it in enabled state";;
+      *) echo -e "  ${INFO} Pi-hole blocking will be enabled";;
     esac
-    # Add addn-host= to dnsmasq
-    echo "addn-hosts=/etc/pihole/gravity.list" >> /etc/dnsmasq.d/01-pihole.conf
-    restartDNS
+    # Enable blocking
+    pihole enable
   fi
 }
 
@@ -303,7 +302,7 @@ tailFunc() {
   # Colour everything else as gray
   tail -f /var/log/pihole.log | sed -E \
     -e "s,($(date +'%b %d ')| dnsmasq[.*[0-9]]),,g" \
-    -e "s,(.*(gravity.list|black.list| config ).* is (${IPV4_ADDRESS%/*}|${IPV6_ADDRESS:-NULL}).*),${COL_RED}&${COL_NC}," \
+    -e "s,(.*(gravity |black |regex | config ).* is (0.0.0.0|::|NXDOMAIN|${IPV4_ADDRESS%/*}|${IPV6_ADDRESS:-NULL}).*),${COL_RED}&${COL_NC}," \
     -e "s,.*(query\\[A|DHCP).*,${COL_NC}&${COL_NC}," \
     -e "s,.*,${COL_GRAY}&${COL_NC},"
   exit 0
@@ -409,7 +408,8 @@ Options:
                         Add '-h' for more info on disable usage
   restartdns          Restart Pi-hole subsystems
   checkout            Switch Pi-hole subsystems to a different Github branch
-                        Add '-h' for more info on checkout usage";
+                        Add '-h' for more info on checkout usage
+  arpflush            Flush information stored in Pi-hole's network tables";
   exit 0
 }
 
@@ -417,12 +417,27 @@ if [[ $# = 0 ]]; then
   helpFunc
 fi
 
+case "${1}" in
+  "-h" | "help" | "--help"      ) helpFunc;;
+esac
+
+# Must be root to use this tool
+if [[ ! $EUID -eq 0 ]];then
+  if [[ -x "$(command -v sudo)" ]]; then
+    exec sudo bash "$0" "$@"
+    exit $?
+  else
+    echo -e "  ${CROSS} sudo is needed to run pihole commands.  Please run this script as root or install sudo."
+    exit 1
+  fi
+fi
+
 # Handle redirecting to specific functions based on arguments
 case "${1}" in
   "-w" | "whitelist"            ) listFunc "$@";;
   "-b" | "blacklist"            ) listFunc "$@";;
-  "--wild" | "wildcard"          ) listFunc "$@";;
-  "--regex" | "regex"            ) listFunc "$@";;
+  "--wild" | "wildcard"         ) listFunc "$@";;
+  "--regex" | "regex"           ) listFunc "$@";;
   "-d" | "debug"                ) debugFunc "$@";;
   "-f" | "flush"                ) flushFunc "$@";;
   "-up" | "updatePihole"        ) updatePiholeFunc "$@";;
@@ -443,5 +458,6 @@ case "${1}" in
   "checkout"                    ) piholeCheckoutFunc "$@";;
   "tricorder"                   ) tricorderFunc;;
   "updatechecker"               ) updateCheckFunc "$@";;
+  "arpflush"                    ) arpFunc "$@";;
   *                             ) helpFunc;;
 esac

requirements.txt

@@ -1,6 +1,6 @@
-docker-compose
-pytest
-pytest-xdist
-pytest-cov
-testinfra
-tox
+docker-compose==1.23.2
+pytest==4.3.0
+pytest-xdist==1.26.1
+pytest-cov==2.6.1
+testinfra==1.19.0
+tox==3.7.0

test/fedora.Dockerfile

@@ -1,4 +1,4 @@
-FROM fedora:latest
+FROM fedora:30
 
 ENV GITDIR /etc/.pihole
 ENV SCRIPTDIR /opt/pihole

test/test_automated_install.py

@@ -81,6 +81,7 @@ def test_setupVars_saved_to_file(Pihole):
     {}
     mkdir -p /etc/dnsmasq.d
     version_check_dnsmasq
+    echo "" > /etc/pihole/pihole-FTL.conf
     finalExports
     cat /etc/pihole/setupVars.conf
     '''.format(set_setup_vars))
@@ -397,6 +398,7 @@ def test_FTL_detect_aarch64_no_errors(Pihole):
     )
     detectPlatform = Pihole.run('''
     source /opt/pihole/basic-install.sh
+    create_pihole_user
     FTLdetect
     ''')
     expected_stdout = info_box + ' FTL Checks...'
@@ -417,6 +419,7 @@ def test_FTL_detect_armv6l_no_errors(Pihole):
     mock_command('ldd', {'/bin/ls': ('/lib/ld-linux-armhf.so.3', '0')}, Pihole)
     detectPlatform = Pihole.run('''
     source /opt/pihole/basic-install.sh
+    create_pihole_user
     FTLdetect
     ''')
     expected_stdout = info_box + ' FTL Checks...'
@@ -438,6 +441,7 @@ def test_FTL_detect_armv7l_no_errors(Pihole):
     mock_command('ldd', {'/bin/ls': ('/lib/ld-linux-armhf.so.3', '0')}, Pihole)
     detectPlatform = Pihole.run('''
     source /opt/pihole/basic-install.sh
+    create_pihole_user
     FTLdetect
     ''')
     expected_stdout = info_box + ' FTL Checks...'
@@ -454,6 +458,7 @@ def test_FTL_detect_x86_64_no_errors(Pihole):
     '''
     detectPlatform = Pihole.run('''
     source /opt/pihole/basic-install.sh
+    create_pihole_user
     FTLdetect
     ''')
     expected_stdout = info_box + ' FTL Checks...'
@@ -470,6 +475,7 @@ def test_FTL_detect_unknown_no_errors(Pihole):
     mock_command('uname', {'-m': ('mips', '0')}, Pihole)
     detectPlatform = Pihole.run('''
     source /opt/pihole/basic-install.sh
+    create_pihole_user
     FTLdetect
     ''')
     expected_stdout = 'Not able to detect architecture (unknown: mips)'
@@ -480,10 +486,18 @@ def test_FTL_download_aarch64_no_errors(Pihole):
     '''
     confirms only aarch64 package is downloaded for FTL engine
     '''
-    # mock uname to return generic platform
+    # mock whiptail answers and ensure installer dependencies
+    mock_command('whiptail', {'*': ('', '0')}, Pihole)
+    Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    distro_check
+    install_dependent_packages ${INSTALLER_DEPS[@]}
+    ''')
     download_binary = Pihole.run('''
     source /opt/pihole/basic-install.sh
-    FTLinstall pihole-FTL-aarch64-linux-gnu
+    binary="pihole-FTL-aarch64-linux-gnu"
+    create_pihole_user
+    FTLinstall
     ''')
     expected_stdout = tick_box + ' Downloading and Installing FTL'
     assert expected_stdout in download_binary.stdout
@@ -494,15 +508,49 @@ def test_FTL_download_unknown_fails_no_errors(Pihole):
     '''
     confirms unknown binary is not downloaded for FTL engine
     '''
-    # mock uname to return generic platform
+    # mock whiptail answers and ensure installer dependencies
+    mock_command('whiptail', {'*': ('', '0')}, Pihole)
+    Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    distro_check
+    install_dependent_packages ${INSTALLER_DEPS[@]}
+    ''')
     download_binary = Pihole.run('''
     source /opt/pihole/basic-install.sh
-    FTLinstall pihole-FTL-mips
+    binary="pihole-FTL-mips"
+    create_pihole_user
+    FTLinstall
     ''')
     expected_stdout = cross_box + ' Downloading and Installing FTL'
     assert expected_stdout in download_binary.stdout
-    error = 'Error: URL not found'
-    assert error in download_binary.stdout
+    error1 = 'Error: URL https://github.com/pi-hole/FTL/releases/download/'
+    assert error1 in download_binary.stdout
+    error2 = 'not found'
+    assert error2 in download_binary.stdout
+
+
+def test_FTL_download_binary_unset_no_errors(Pihole):
+    '''
+    confirms unset binary variable does not download FTL engine
+    '''
+    # mock whiptail answers and ensure installer dependencies
+    mock_command('whiptail', {'*': ('', '0')}, Pihole)
+    Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    distro_check
+    install_dependent_packages ${INSTALLER_DEPS[@]}
+    ''')
+    download_binary = Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    create_pihole_user
+    FTLinstall
+    ''')
+    expected_stdout = cross_box + ' Downloading and Installing FTL'
+    assert expected_stdout in download_binary.stdout
+    error1 = 'Error: URL https://github.com/pi-hole/FTL/releases/download/'
+    assert error1 in download_binary.stdout
+    error2 = 'not found'
+    assert error2 in download_binary.stdout
 
 
 def test_FTL_binary_installed_and_responsive_no_errors(Pihole):
@@ -511,6 +559,7 @@ def test_FTL_binary_installed_and_responsive_no_errors(Pihole):
     '''
     installed_binary = Pihole.run('''
     source /opt/pihole/basic-install.sh
+    create_pihole_user
     FTLdetect
     pihole-FTL version
     ''')
@@ -651,3 +700,42 @@ def test_IPv6_ULA_GUA_test(Pihole):
     ''')
     expected_stdout = 'Found IPv6 ULA address, using it for blocking IPv6 ads'
     assert expected_stdout in detectPlatform.stdout
+
+
+def test_validate_ip_valid(Pihole):
+    '''
+    Given a valid IP address, valid_ip returns success
+    '''
+
+    output = Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    valid_ip "192.168.1.1"
+    ''')
+
+    assert output.rc == 0
+
+
+def test_validate_ip_invalid_octet(Pihole):
+    '''
+    Given an invalid IP address (large octet), valid_ip returns an error
+    '''
+
+    output = Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    valid_ip "1092.168.1.1"
+    ''')
+
+    assert output.rc == 1
+
+
+def test_validate_ip_invalid_letters(Pihole):
+    '''
+    Given an invalid IP address (contains letters), valid_ip returns an error
+    '''
+
+    output = Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    valid_ip "not an IP"
+    ''')
+
+    assert output.rc == 1

test/test_centos_fedora_support.py

@@ -31,20 +31,13 @@ def test_release_supported_version_check_centos(Pihole):
     '''
     confirms installer exits on unsupported releases of CentOS
     '''
-    # mock CentOS release < 7 (unsupported)
-    mock_command_2(
-        'rpm',
-        {"-q --queryformat '%{VERSION}' centos-release'": (
-            '5',
-            '0'
-        )},
-        Pihole
-    )
+    # modify /etc/redhat-release to mock an unsupported CentOS release
+    Pihole.run('echo "CentOS Linux release 6.9" > /etc/redhat-release')
     distro_check = Pihole.run('''
     source /opt/pihole/basic-install.sh
     distro_check
     ''')
-    expected_stdout = cross_box + (' CentOS  is not suported.')
+    expected_stdout = cross_box + (' CentOS 6 is not supported.')
     assert expected_stdout in distro_check.stdout
     expected_stdout = 'Please update to CentOS release 7 or later'
     assert expected_stdout in distro_check.stdout