Merge pull request #1041 from pi-hole/development

[RELEASE] Pi-Hole Core 2.10.2

.codeclimate.yml

@@ -1,3 +0,0 @@
-engines:
-  shellcheck:
-    enabled: true

.github/ISSUE_TEMPLATE.md

@@ -6,21 +6,12 @@
 
 **How familiar are you with the codebase?:**
 
-- [] 1 (very unfamiliar)
-- [] 2
-- [] 3
-- [] 4
-- [] 5
-- [] 6
-- [] 7
-- [] 8
-- [] 9
-- [] 10 (very familiar)
+_{replace this text with a number from 1 to 10, with 1 being not familiar, and 10 being very familiar}_
 
 ---
 **[FEATURE REQUEST | QUESTION | OTHER]:**
 
-_{replace this section with your content or delete if not a FEATURE REQUEST/QUESTION/OTHER}_
+Please [submit your feature request here](https://discourse.pi-hole.net/c/feature-requests), so it is votable by the community.  It's also easier for us to track.
 
 **[BUG | ISSUE] Expected Behaviour:**
 
@@ -35,8 +26,8 @@ _{replace this section with your content or delete if not a FEATURE REQUEST/QUES
 -
 -
 
-**(Optional) Debug Log generated by `pihole -d`:**
+**(Optional) Debug token generated by `pihole -d`:**
 
-`http://termbin.com/<something>`
+`<token>`
 
 _This template was created based on the work of [`udemy-dl`](https://github.com/nishad/udemy-dl/blob/master/LICENSE)._

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,4 +1,6 @@
-**By submitting this pull request, I confirm the following (please check boxes, eg [X])Failure to fill the template will close your PR:**
+**By submitting this pull request, I confirm the following (please check boxes, eg [X]) _Failure to fill the template will close your PR_:**
+
+***Please submit all pull requests against the `development` branch. Failure to do so will delay or deny your request***
 
 - [] I have read and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md).
 - [] I have checked that [another pull request](https://github.com/pi-hole/pi-hole/pulls) for this purpose does not exist.
@@ -8,16 +10,7 @@
 
 **How familiar are you with the codebase?:**
 
-- [] 1 (very unfamiliar)
-- [] 2
-- [] 3
-- [] 4
-- [] 5
-- [] 6
-- [] 7
-- [] 8
-- [] 9
-- [] 10 (very familiar)
+_{replace this text with a number from 1 to 10, with 1 being not familiar, and 10 being very familiar}_
 
 ---
 _{replace this line with your pull request content}_

.gitignore

@@ -1 +1,7 @@
 .DS_Store
+*.pyc
+*.swp
+__pycache__
+.cache
+.pullapprove.yml
+

.idea/codeStyleSettings.xml

@@ -0,0 +1,22 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="ProjectCodeStyleSettingsManager">
+    <option name="PER_PROJECT_SETTINGS">
+      <value>
+        <option name="OTHER_INDENT_OPTIONS">
+          <value>
+            <option name="INDENT_SIZE" value="2" />
+            <option name="CONTINUATION_INDENT_SIZE" value="8" />
+            <option name="TAB_SIZE" value="2" />
+            <option name="USE_TAB_CHARACTER" value="false" />
+            <option name="SMART_TABS" value="false" />
+            <option name="LABEL_INDENT_SIZE" value="0" />
+            <option name="LABEL_INDENT_ABSOLUTE" value="false" />
+            <option name="USE_RELATIVE_INDENTS" value="false" />
+          </value>
+        </option>
+      </value>
+    </option>
+    <option name="USE_PER_PROJECT_SETTINGS" value="true" />
+  </component>
+</project>

.pullapprove.yml

@@ -25,7 +25,7 @@ groups:
         - development        
     required: 2
     teams:
-      - gravity
+      - approvers
   
   master:
     approve_by_comment:
@@ -33,6 +33,6 @@ groups:
     conditions:
       branches:
         - master
-    required: 3
+    required: -1
     teams:
-      - gravity
+      - admin

.travis.yml

@@ -0,0 +1,10 @@
+sudo: required
+services:
+  - docker
+language: python
+python:
+  - "2.7"
+install:
+  - pip install -r requirements.txt
+
+script: py.test -vv

CONTRIBUTING.md

@@ -33,5 +33,6 @@ When requesting or submitting new features, first consider whether it might be u
 
 - Submit Pull Requests to the **development branch only**.
 - Before Submitting your Pull Request, merge `development` with your new branch and fix any conflicts. (Make sure you don't break anything in development!)
+- Please use the [Google Style Guide for Shell](https://google.github.io/styleguide/shell.xml) for your code submission styles. 
 - Commit Unix line endings.
 - (Optional fun) keep to the theme of Star Trek/black holes/gravity.

README.md

@@ -1,52 +1,80 @@
-[![Bountysource](https://www.bountysource.com/badge/tracker?tracker_id=3011939)](https://www.bountysource.com/trackers/3011939-pi-hole-pi-hole?utm_source=3011939&utm_medium=shield&utm_campaign=TRACKER_BADGE) [![Code Climate](https://codeclimate.com/github/pi-hole/pi-hole/badges/gpa.svg)](https://codeclimate.com/github/pi-hole/pi-hole)
+<p align="center">
+<a href=https://www.bountysource.com/trackers/3011939-pi-hole-pi-hole?utm_source=3011939&utm_medium=shield&utm_campaign=TRACKER_BADGE><img src="https://www.bountysource.com/badge/tracker?tracker_id=3011939"></a>
+<a href="https://www.codacy.com/app/Pi-hole/pi-hole?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=pi-hole/pi-hole&amp;utm_campaign=Badge_Grade"><img src="https://api.codacy.com/project/badge/Grade/c558a0f8d7124c99b02b84f0f5564238"/></a>
+<a href=https://travis-ci.org/pi-hole/pi-hole><img src="https://travis-ci.org/pi-hole/pi-hole.svg?branch=development"></a>
+</p>
 
-# Automated Install
+<p align="center">
+<a href=https://discourse.pi-hole.net><img src="https://assets.pi-hole.net/static/Vortex_text.png" width=210></a>
+</p>
 
-Designed For Raspberry Pi A+, B, B+, 2, 3B, and Zero (with an Ethernet/Wi-Fi adapter)
-Works on most Debian and CentOS/RHEL based distributions!
+## The multi-platform, network-wide ad blocker
 
-1.  Install Raspbian
-2.  Run the command below (downloads [this script](https://github.com/pi-hole/pi-hole/blob/master/automated%20install/basic-install.sh) in case you want to read over it first!)
+Block ads for **all** your devices _without_ the need to install client-side software.  The Pi-hole blocks ads at the DNS-level, so all your devices are protected.
 
-## `curl -L https://install.pi-hole.net | bash`
+- Web Browsers
+- Cell Phones
+- Smart TV's
+- Internet-connected home automation
+- Anything that communicates with the Internet
 
-### Alternative Semi-Automated install
+<p align="center">
+<a href=http://www.digitalocean.com/?refcode=344d234950e1><img src="https://assets.pi-hole.net/static/DOHostingSlug.png"></a>
+</p>
+
+## Your Support Still Matters
+
+Digital Ocean helps with our infrastructure, but our developers are all volunteers so *your donations help keep us innovating*. Sending a donation using our links below helps us offset a portion of our monthly costs.
+
+-   ![Paypal](https://assets.pi-hole.net/static/paypal.png) [Donate via PayPal](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY)
+-   ![Bitcoin](https://assets.pi-hole.net/static/Bitcoin.png) Bitcoin Address: 1GKnevUnVaQM2pQieMyeHkpr8DXfkpfAtL
+
+### One-Step Automated Install
+1.  Install a [supported operating system](https://discourse.pi-hole.net/t/hardware-software-requirements/273/1)
+2.  Run the command below (it downloads [this script](https://github.com/pi-hole/pi-hole/blob/master/automated%20install/basic-install.sh) in case you want to read over it first!)
+
+### `curl -sSL https://install.pi-hole.net | bash`
+
+#### Alternative Semi-Automated Install Methods
+_If you wish to read over the script before running it, run `nano basic-install.sh` to open the file in a text viewer._
+
+##### Clone our repository and run the automated installer from your device.
+
+```
+git clone --depth 1 https://github.com/pi-hole/pi-hole.git Pi-hole
+cd Pi-hole/automated_installer/
+bash basic-install.sh
+```
+
+##### Or
 
 ```bash
 wget -O basic-install.sh https://install.pi-hole.net
-chmod +x basic-install.sh
-./basic-install.sh
+bash basic-install.sh
 ```
 
-If you wish to read over the script before running it, then after the `wget` command, do `nano basic-install.sh` to open a text viewer
+Once installed, [configure your router to have **DHCP clients use the Pi as their DNS server**](https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245) and then any device that connects to your network will have ads blocked without any further configuration. Alternatively, you can manually set each device to [use the Raspberry Pi as its DNS server](http://pi-hole.net/faq/how-do-i-use-the-pi-hole-as-my-dns-server/).
 
-Once installed, [configure your router to have **DHCP clients use the Pi as their DNS server**](http://pi-hole.net/faq/can-i-set-the-pi-hole-to-be-the-dns-server-at-my-router-so-i-dont-have-to-change-settings-for-my-devices/) and then any device that connects to your network will have ads blocked without any further configuration. Alternatively, you can manually set each device to [use the Raspberry Pi as its DNS server](http://pi-hole.net/faq/how-do-i-use-the-pi-hole-as-my-dns-server/).
+## Installing the Pi-hole (Click to Watch!)
+<p align="center">
+<a href=https://www.youtube.com/watch?v=TzFLJqUeirA><img src="https://assets.pi-hole.net/static/global.png"></a>
+</p>
 
-## How To Install Pi-hole
-
-[![60-second install tutorial](http://i.imgur.com/5TEc3a6.png)](https://www.youtube.com/watch?v=TzFLJqUeirA)
-
-## How It Works
+## Would you like to know more?
 
 **Watch the 60-second video below to get a quick overview**
-
-[![Pi-hole exlplained](http://i.imgur.com/pG1m937.png)](https://youtu.be/9Eti3xibiho)
-
-## Pi-hole Is Free, But Powered By Your Donations
-
-Send a one-time donation or sign up for Optimal.com's service using our link below to provide us with a small portion of the monthly fee.
-
--   ![Paypal](http://i.imgur.com/3muNfxu.png) : [Donate](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY)
--   ![Flattr](http://i.imgur.com/ZFceFRu.png) : [Donate](https://flattr.com/submit/auto?user_id=jacobsalmela&url=https://github.com/pi-hole/pi-hole)
--   ![Bitcoin](http://i.imgur.com/FIlmOMG.png) : 1GKnevUnVaQM2pQieMyeHkpr8DXfkpfAtL
+<p align="center">
+<a href=https://youtu.be/9Eti3xibiho><img src="https://assets.pi-hole.net/static/blackhole_web.png"></a>
+</p>
 
 ## Get Help Or Connect With Us On The Web
 
--   [@The_Pi_Hole](https://twitter.com/The_Pi_Hole)
--   [/r/pihole](https://www.reddit.com/r/pihole/)
--   [Pi-hole YouTube channel](https://www.youtube.com/channel/UCT5kq9w0wSjogzJb81C9U0w)
--   [Wiki](https://github.com/pi-hole/pi-hole/wiki/Customization)
--   [FAQs](https://pi-hole.net/help/)
+-   [Users Forum](https://discourse.pi-hole.net/)
+-   [FAQs](https://discourse.pi-hole.net/c/faqs)
+-   [Wiki](https://github.com/pi-hole/pi-hole/wiki)
+-   ![Twitter](https://assets.pi-hole.net/static/twitter.png) [Tweet @The_Pi_Hole](https://twitter.com/The_Pi_Hole)
+-   ![Reddit](https://assets.pi-hole.net/static/reddit.png) [Reddit /r/pihole](https://www.reddit.com/r/pihole/)
+-   ![YouTube](https://assets.pi-hole.net/static/youtube.png)  [Pi-hole channel](https://www.youtube.com/channel/UCT5kq9w0wSjogzJb81C9U0w)
 -   [![Join the chat at https://gitter.im/pi-hole/pi-hole](https://badges.gitter.im/pi-hole/pi-hole.svg)](https://gitter.im/pi-hole/pi-hole?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
 ## Technical Details
@@ -55,25 +83,34 @@ The Pi-hole is an **advertising-aware DNS/Web server**. If an ad domain is queri
 
 ### Gravity
 
-The [gravity.sh](https://github.com/pi-hole/pi-hole/blob/master/gravity.sh) does most of the magic. The script pulls in ad domains from many sources and compiles them into a single list of [over 1.6 million entries](http://jacobsalmela.com/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0) (if you decide to use the [mahakala list](https://github.com/pi-hole/pi-hole/commit/963eacfe0537a7abddf30441c754c67ca1e40965)).
+The [gravity.sh](https://github.com/pi-hole/pi-hole/blob/master/gravity.sh) does most of the magic. The script pulls in ad domains from many sources and compiles them into a single list of [over 1.6 million entries](http://jacobsalmela.com/block-millions-ads-network-wide-with-a-raspberry-pi-hole-2-0) (if you decide to use the [mahakala list](https://github.com/pi-hole/pi-hole/commit/963eacfe0537a7abddf30441c754c67ca1e40965)). This script is controlled by the `pihole` command. Please run `pihole -h` to see what commands can be run via `pihole`.
 
-## Web Interface
 
-The [Web interface](https://github.com/jacobsalmela/AdminLTE#pi-hole-admin-dashboard) will be installed automatically so you can view stats and change settings. You can find it at:
+
+#### Other Operating Systems
+
+The automated install is only for a clean install of a Debian family or Fedora based system, such as the Raspberry Pi. However, this script will work for most UNIX-like systems, some with some slight **modifications** that we can help you work through. If you can install `dnsmasq` and a Webserver, it should work OK. If there are other platforms you'd like supported, let us know.
+
+### Web Interface
+
+The [Web interface](https://github.com/pi-hole/AdminLTE#pi-hole-admin-dashboard) will be installed automatically so you can view stats and change settings. You can find it at:
 
 `http://192.168.1.x/admin/index.php` or `http://pi.hole/admin`
 
-![Pi-hole Advanced Stats Dashboard](http://i.imgur.com/gTq2GbS.png)
+![Pi-hole Advanced Stats Dashboard](https://assets.pi-hole.net/static/dashboard.png)
 
 ### Whitelist and blacklist
 
-Domains can be whitelisted and blacklisted using two pre-installed scripts. See [the wiki page](https://github.com/pi-hole/pi-hole/wiki/Whitelisting-and-Blacklisting) for more details ![Whitelist editor in the Web interface](http://i.imgur.com/ogu2ewg.png)
+Domains can be whitelisted and blacklisted using either the web interface or the command line. See [the wiki page](https://github.com/pi-hole/pi-hole/wiki/Whitelisting-and-Blacklisting) for more details 
+<p align="center">
+<a href=https://github.com/pi-hole/pi-hole/wiki/Whitelisting-and-Blacklisting><img src="https://assets.pi-hole.net/static/controlpanel.png"></a>
+</p>
 
 ## API
 
 A basic read-only API can be accessed at `/admin/api.php`. It returns the following JSON:
 
-```json
+``` json
 {
     "domains_being_blocked": "136708",
     "dns_queries_today": "18108",
@@ -103,6 +140,8 @@ You can view [real-time stats](http://pi-hole.net/faq/install-the-real-time-lcd-
 -   [Pi-hole on/off button](http://thetimmy.silvernight.org/pages/endisbutton/)
 -   [Minibian Pi-hole](http://munkjensen.net/wiki/index.php/See_my_Pi-Hole#Minibian_Pi-hole)
 -   [Windows Tray Stat Application](https://github.com/goldbattle/copernicus)
+-   [Let your blink1 device blink when Pi-hole filters ads](https://gist.github.com/elpatron68/ec0b4c582e5abf604885ac1e068d233f)
+-   [Pi-Hole Prometheus exporter](https://github.com/nlamirault/pihole_exporter) : a [Prometheus](https://prometheus.io/) exporter for Pi-Hole
 
 ## Coverage
 
@@ -119,7 +158,4 @@ You can view [real-time stats](http://pi-hole.net/faq/install-the-real-time-lcd-
 -   [Foolish Tech Show](https://youtu.be/bYyena0I9yc?t=2m4s)
 -   [Pi-hole on Ubuntu](http://www.boyter.org/2015/12/pi-hole-ubuntu-14-04/)
 -   [Catchpoint: iOS 9 Ad Blocking](http://blog.catchpoint.com/2015/09/14/ad-blocking-apple/)
-
-## Other Operating Systems
-
-This script will work for other UNIX-like systems with some slight **modifications**. As long as you can install `dnsmasq` and a Webserver, it should work OK. The automated install is only for a clean install of a Debian based system, such as the Raspberry Pi.
+-   [Build an Ad-Blocker for less than 10$ with Orange-Pi](http://www.devacron.com/orangepi-zero-as-an-ad-block-server-with-pi-hole/)

adlists.default

@@ -1,4 +1,4 @@
-## Pi-hole ad-list default sources. Updated 22/05/2016 #########################
+## Pi-hole ad-list default sources. Updated 29/10/2016 #########################
 #                                                                              #
 #  To make changes to this file:                                               #
 #    1. run `cp /etc/pihole/adlists.default /etc/pihole/adlists.list`          #
@@ -28,7 +28,7 @@ https://hosts-file.net/ad_servers.txt
 #http://adblock.mahakala.is/
 
 # ADZHOSTS list. Has been known to block legitimate domains
-#http://optimate.dl.sourceforge.net/project/adzhosts/HOSTS.txt
+#http://pilotfiber.dl.sourceforge.net/project/adzhosts/HOSTS.txt
 
 # Windows 10 telemetry list
 #https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/win10/spy.txt

advanced/01-pihole.conf

@@ -20,13 +20,8 @@
 #                        OR IN /etc/dnsmasq.conf                              #
 ###############################################################################
 
-address=/pi.hole/@IPv4@
-address=/pi.hole/@IPv6@
-
-address=/@HOSTNAME@/@IPv4@
-address=/@HOSTNAME@/@IPv6@
-
 addn-hosts=/etc/pihole/gravity.list
+addn-hosts=/etc/pihole/local.list
 
 domain-needed
 

advanced/Scripts/chronometer.sh

@@ -15,17 +15,15 @@
 piLog="/var/log/pihole.log"
 gravity="/etc/pihole/gravity.list"
 
-today=$(date "+%b %e")
+. /etc/pihole/setupVars.conf
 
 CalcBlockedDomains() {
-	CheckIPv6
 	if [ -e "${gravity}" ]; then
-		#Are we IPV6 or IPV4?
-		if [[ -n ${piholeIPv6} ]]; then
-			#We are IPV6
+		# if BOTH IPV4 and IPV6 are in use, then we need to divide total domains by 2.
+		if [[ -n "${IPV4_ADDRESS}" && -n "${IPV6_ADDRESS}" ]]; then
 			blockedDomainsTotal=$(wc -l /etc/pihole/gravity.list | awk '{print $1/2}')
 		else
-			#We are IPV4
+			# only one is set.
 			blockedDomainsTotal=$(wc -l /etc/pihole/gravity.list | awk '{print $1}')
 		fi
 	else
@@ -35,7 +33,7 @@ CalcBlockedDomains() {
 
 CalcQueriesToday() {
 	if [ -e "${piLog}" ]; then
-		queriesToday=$(cat "${piLog}" | grep "${today}" | awk '/query/ {print $6}' | wc -l)
+		queriesToday=$(awk '/query\[/ {print $6}' < "${piLog}" | wc -l)
 	else
 		queriesToday="Err."
 	fi
@@ -43,7 +41,7 @@ CalcQueriesToday() {
 
 CalcblockedToday() {
 	if [ -e "${piLog}" ] && [ -e "${gravity}" ];then
-		blockedToday=$(cat ${piLog} | awk '/\/etc\/pihole\/gravity.list/ && !/address/ {print $6}' | wc -l)
+		blockedToday=$(awk '/\/etc\/pihole\/gravity.list/ && !/address/ {print $6}' < "${piLog}" | wc -l)
 	else
 		blockedToday="Err."
 	fi
@@ -61,14 +59,6 @@ CalcPercentBlockedToday() {
 	fi
 }
 
-CheckIPv6() {
-	piholeIPv6file="/etc/pihole/.useIPv6"
-	if [[ -f ${piholeIPv6file} ]];then
-		# If the file exists, then the user previously chose to use IPv6 in the automated installer
-		piholeIPv6=$(ip -6 route get 2001:4860:4860::8888 | awk -F " " '{ for(i=1;i<=NF;i++) if ($i == "src") print $(i+1) }')
-	fi
-}
-
 outputJSON() {
 	CalcQueriesToday
 	CalcblockedToday
@@ -88,7 +78,7 @@ normalChrono() {
 		echo "|  _/ |___| ' \/ _ \ / -_)"
 		echo "|_| |_|   |_||_\___/_\___|"
 		echo ""
-		echo "        $(ifconfig eth0 | awk '/inet addr/ {print $2}' | cut -d':' -f2)"
+		echo "        ${IPV4_ADDRESS}"
 		echo ""
 		uptime | cut -d' ' -f11-
 		#uptime -p	#Doesn't work on all versions of uptime
@@ -111,10 +101,8 @@ normalChrono() {
 		CalcBlockedDomains
 
 		echo "Blocking:      ${blockedDomainsTotal}"
-		#below commented line does not add up to todaysQueryCount
-		#echo "Queries:       $todaysQueryCountV4 / $todaysQueryCountV6"
 		echo "Queries:       ${queriesToday}" #same total calculation as dashboard
-	  echo "Pi-holed:      ${blockedToday} (${percentBlockedToday}%)"
+		echo "Pi-holed:      ${blockedToday} (${percentBlockedToday}%)"
 
 		sleep 5
 	done
@@ -131,7 +119,7 @@ displayHelp() {
 :::  -j, --json		output stats as JSON formatted string
 :::  -h, --help		display this help text
 EOM
-    exit 1
+    exit 0
 }
 
 if [[ $# = 0 ]]; then

advanced/Scripts/list.sh

@@ -47,12 +47,21 @@ helpFunc() {
 :::  -h, --help				Show this help dialog
 :::  -l, --list				Display your ${word}listed domains
 EOM
-	exit 1
+	exit 0
+}
+
+EscapeRegexp() {
+    # This way we may safely insert an arbitrary
+    # string in our regular expressions
+    echo $* | sed "s/[]\\.|$(){}?+*^]/\\\\&/g" | sed "s/\\//\\\\\//g"
 }
 
 HandleOther(){
-  #check validity of domain
-	validDomain=$(echo "$1" | perl -ne'print if /\b((?=[a-z0-9-]{1,63}\.)(xn--)?[a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,63}\b/')
+	# First, convert everything to lowercase
+	domain=$(sed -e "y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/" <<< "$1")
+
+	#check validity of domain
+	validDomain=$(perl -ne "print if /\b((?=[a-z0-9-]{1,63}\.)(xn--)?[a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,63}\b/" <<< "$domain")
 	if [ -z "${validDomain}" ]; then
 		echo "::: $1 is not a valid argument or domain name"
 	else
@@ -77,12 +86,12 @@ PoplistFile() {
 }
 
 AddDomain() {
-
 	list="$2"
+    domain=$(EscapeRegexp "$1")
 
 	bool=true
 	#Is the domain in the list we want to add it to?
-	grep -Ex -q "$1" ${list} > /dev/null 2>&1 || bool=false
+	grep -Ex -q "${domain}" ${list} > /dev/null 2>&1 || bool=false
 
 	if [[ "${bool}" == false ]]; then
 	  #domain not found in the whitelist file, add it!
@@ -101,14 +110,16 @@ AddDomain() {
 
 RemoveDomain() {
     list="$2"
+    domain=$(EscapeRegexp "$1")
 
     bool=true
-    #Is it in the other list? Logic follows that if its whitelisted it should not be blacklisted and vice versa
-    grep -Ex -q "$1" ${list} > /dev/null 2>&1 || bool=false
+    #Is it in the list? Logic follows that if its whitelisted it should not be blacklisted and vice versa
+    grep -Ex -q "${domain}" ${list} > /dev/null 2>&1 || bool=false
     if [[ "${bool}" == true ]]; then
         # Remove it from the other one
         echo "::: Removing $1 from $list..."
-        echo "$1" | sed 's/\./\\./g' | xargs -I {} perl -i -ne'print unless /'{}'(?!.)/;' ${list}
+        # /I flag: search case-insensitive
+        sed -i "/${domain}/Id" ${list}
         reload=true
     else
         if [[ "${verbose}" == true ]]; then

advanced/Scripts/piholeDebug.sh

@@ -27,7 +27,7 @@ PIHOLELOG="/var/log/pihole.log"
 WHITELISTMATCHES="/tmp/whitelistmatches.list"
 
 IPV6_READY=false
-
+TIMEOUT=60
 # Header info and introduction
 cat << EOM
 ::: Beginning Pi-hole debug at $(date)!
@@ -316,6 +316,16 @@ debugLighttpd() {
   echo ":::"
 }
 
+countdown() {
+  tuvix=${TIMEOUT}
+  printf "::: Logging will automatically teminate in ${TIMEOUT} seconds\n"
+  while [ $tuvix -ge 1 ]
+  do
+    printf ":::\t${tuvix} seconds left. \r"
+    sleep 5
+    tuvix=$(( tuvix - 5 ))
+  done
+}
 ### END FUNCTIONS ###
 
 # Gather version of required packages / repositories
@@ -356,10 +366,9 @@ dumpPiHoleLog() {
 	echo -e "::: Try loading a site that you are having trouble with now from a client web browser.. \n:::\t(Press CTRL+C to finish logging.)"
 	header_write "pihole.log"
 	if [ -e "${PIHOLELOG}" ]; then
-		while true; do
-			tail -f "${PIHOLELOG}" >> ${DEBUG_LOG}
-			log_write ""
-		done
+	# Dummy process to use for flagging down tail to terminate
+	  countdown &
+		tail -n0 -f --pid=$! "${PIHOLELOG}" >> ${DEBUG_LOG}
 	else
 		log_write "No pihole.log file found!"
 		printf ":::\tNo pihole.log file found!\n"
@@ -384,10 +393,13 @@ finalWork() {
 	# Check if tricorder.pi-hole.net is reachable and provide token.
 	if [ -n "${tricorder}" ]; then
 		echo "::: Your debug token is : ${tricorder}"
-		echo "::: Please contact the Pi-hole team with your token to being assistance."
+		echo "::: Please contact the Pi-hole team with your token for assistance."
 		echo "::: Thank you."
+	else
+		echo "::: There was an error uploading your debug log."
+		echo "::: Please try again or contact the Pi-hole team for assistance."
 	fi
-		echo "::: Debug log can be found at : /var/log/pihole_debug.log"
+		echo "::: A local copy of the Debug log can be found at : /var/log/pihole_debug.log"
 }
 
 trap finalWork EXIT

advanced/Scripts/update.sh

@@ -3,7 +3,9 @@
 # (c) 2015, 2016 by Jacob Salmela
 # Network-wide ad blocking via your Raspberry Pi
 # http://pi-hole.net
-# Whitelists domains
+# Check Pi-hole core and admin pages versions and determine what
+# upgrade (if any) is required. Automatically updates and reinstalls
+# application if update is detected.
 #
 # Pi-hole is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -12,131 +14,163 @@
 
 # Variables
 
-webInterfaceGitUrl="https://github.com/pi-hole/AdminLTE.git"
-webInterfaceDir="/var/www/html/admin"
-piholeGitUrl="https://github.com/pi-hole/pi-hole.git"
-piholeFilesDir="/etc/.pihole"
-
-spinner() {
-	local pid=${1}
-	local delay=0.50
-	local spinstr='/-\|'
-	while [ "$(ps a | awk '{print $1}' | grep "${pid}")" ]; do
-		local temp=${spinstr#?}
-		printf " [%c]  " "${spinstr}"
-		local spinstr=${temp}${spinstr%"$temp"}
-		sleep ${delay}
-		printf "\b\b\b\b\b\b"
-	done
-	printf "    \b\b\b\b"
-}
-
-getGitFiles() {
-	# Setup git repos for directory and repository passed
-	# as arguments 1 and 2
-	echo ":::"
-	echo "::: Checking for existing repository..."
-	if is_repo "${1}"; then
-		update_repo "${1}"
-	else
-		make_repo "${1}" "${2}"
-	fi
-}
+readonly ADMIN_INTERFACE_GIT_URL="https://github.com/pi-hole/AdminLTE.git"
+readonly ADMIN_INTERFACE_DIR="/var/www/html/admin"
+readonly PI_HOLE_GIT_URL="https://github.com/pi-hole/pi-hole.git"
+readonly PI_HOLE_FILES_DIR="/etc/.pihole"
 
 is_repo() {
-	# Use git to check if directory is currently under VCS
-	echo -n ":::    Checking $1 is a repo..."
-	cd "${1}" &> /dev/null || return 1
-	git status &> /dev/null && echo " OK!"; return 0 || echo " not found!"; return 1
+  # Use git to check if directory is currently under VCS, return the value
+  local directory="${1}"
+  local curdir
+  local rc
+
+  curdir="${PWD}"
+  cd "${directory}" &> /dev/null || return 1
+  git status --short &> /dev/null
+  rc=$?
+  cd "${curdir}" &> /dev/null || return 1
+  return $rc
+}
+
+prep_repo() {
+  # Prepare directory for local repository building
+  local directory="${1}"
+
+  rm -rf "${directory}" &> /dev/null
+  return
 }
 
 make_repo() {
-	# Remove the non-repod interface and clone the interface
-	echo -n ":::    Cloning $2 into $1..."
-	rm -rf "${1}"
-	git clone -q --depth 1 "${2}" "${1}" > /dev/null & spinner $!
-	echo " done!"
+  # Remove the non-repod interface and clone the interface
+  local remoteRepo="${2}"
+  local directory="${1}"
+
+  (prep_repo "${directory}" && git clone -q --depth 1 "${remoteRepo}" "${directory}" > /dev/null)
+     return
 }
 
 update_repo() {
-# Pull the latest commits
-	echo -n ":::     Updating repo in $1..."
-	cd "${1}" || exit 1
-	git stash -q > /dev/null & spinner $!
-	git pull -q > /dev/null & spinner $!
-	echo " done!"
+  local directory="${1}"
+  local curdir
+  # Pull the latest commits
+
+  curdir="${PWD}"
+  cd "${directory}" &> /dev/null || return 1
+  # Stash all files not tracked for later retrieval
+  git stash --all --quiet &> /dev/null
+  # Force a clean working directory for cloning
+  git clean --force -d &> /dev/null
+  # Fetch latest changes and apply
+  git pull --quiet &> /dev/null
+  cd "${curdir}" &> /dev/null || return 1
+
+  return
 }
 
-if [ ! -d "/etc/.pihole" ]; then #This is unlikely
-	echo "::: Critical Error: Pi-Hole repo missing from system!"
-	echo "::: Please re-run install script from https://github.com/pi-hole/pi-hole"
-	exit 1;
-fi
-if [ ! -d "/var/www/html/admin" ]; then #This is unlikely
-	echo "::: Critical Error: Pi-Hole repo missing from system!"
-	echo "::: Please re-run install script from https://github.com/pi-hole/pi-hole"
-	exit 1;
-fi
+getGitFiles() {
+  # Setup git repos for directory and repository passed
+  # as arguments 1 and 2
+  local directory="${1}"
+  local remoteRepo="${2}"
+  echo ":::"
+  echo "::: Checking for existing repository..."
+  if is_repo "${directory}"; then
+    echo -n ":::     Updating repository in ${directory}..."
+    update_repo "${directory}" || (echo "*** Error: Could not update local repository. Contact support."; exit 1)
+    echo " done!"
+  else
+    echo -n ":::    Cloning ${remoteRepo} into ${directory}..."
+    make_repo "${directory}" "${remoteRepo}" || (echo "Unable to clone repository, please contact support"; exit 1)
+    echo " done!"
+  fi
+}
 
-echo "::: Checking for updates..."
-piholeVersion=$(pihole -v -p -c)
-piholeVersionLatest=$(pihole -v -p -l)
+main() {
+  local pihole_version_current
+  local pihole_version_latest
+  local web_version_current
+  local web_version_latest
 
-webVersion=$(pihole -v -a -c)
-webVersionLatest=$(pihole -v -a -l)
+  if ! is_repo "${PI_HOLE_FILES_DIR}" || ! is_repo "${ADMIN_INTERFACE_DIR}" ; then #This is unlikely
+    echo "::: Critical Error: One or more Pi-Hole repos are missing from system!"
+    echo "::: Please re-run install script from https://github.com/pi-hole/pi-hole"
+    exit 1;
+  fi
 
-echo ":::"
-echo "::: Pi-hole version is $piholeVersion (Latest version is $piholeVersionLatest)"
-echo "::: Web Admin version is $webVersion (Latest version is $webVersionLatest)"
-echo ":::"
+  echo "::: Checking for updates..."
+  # Checks Pi-hole version string in format vX.X.X
+  pihole_version_current="$(/usr/local/bin/pihole version --pihole --current)"
+  pihole_version_latest="$(/usr/local/bin/pihole version --pihole --latest)"
+  web_version_current="$(/usr/local/bin/pihole version --admin --current)"
+  web_version_latest="$(/usr/local/bin/pihole version --admin --latest)"
 
-# Logic
-# If latest versions are blank - we've probably hit Github rate limit (stop running `pihole -up so often!):
-#            Update anyway
-# If Core up to date AND web up to date:
-#            Do nothing
-# If Core up to date AND web NOT up to date:
-#            Pull web repo
-# If Core NOT up to date AND web up to date:
-#            pull pihole repo, run install --unattended -- reconfigure
-# if Core NOT up to date AND web NOT up to date:
-#            pull pihole repo run install --unattended
+  if [[ "${pihole_version_latest}" == "-1" || "${web_version_latest}" == "-1" ]]; then
+    echo "*** Unable to contact GitHub for latest version. Please try again later, contact support if this continues."
+    exit 1
+  fi
 
+  # Logic
+  # If latest versions are blank - we've probably hit Github rate limit (stop running `pihole -up so often!):
+  #            Update anyway
+  # If Core up to date AND web up to date:
+  #            Do nothing
+  # If Core up to date AND web NOT up to date:
+  #            Pull web repo
+  # If Core NOT up to date AND web up to date:
+  #            pull pihole repo, run install --unattended -- reconfigure
+  # if Core NOT up to date AND web NOT up to date:
+  #            pull pihole repo run install --unattended
 
+  if [[ "${pihole_version_current}" == "${pihole_version_latest}" ]] && [[ "${web_version_current}" == "${web_version_latest}" ]]; then
+    echo ":::"
+    echo "::: Pi-hole version is $pihole_version_current"
+    echo "::: Web Admin version is $web_version_current"
+    echo ":::"
+    echo "::: Everything is up to date!"
+    exit 0
 
-if [[ ${piholeVersion} == ${piholeVersionLatest} && ${webVersion} == ${webVersionLatest} ]]; then
-	echo "::: Everything is up to date!"
-	echo ""
-	exit 0
+  elif [[ "${pihole_version_current}" == "${pihole_version_latest}" ]] && [[ "${web_version_current}" < "${web_version_latest}" ]]; then
+    echo ":::"
+    echo "::: Pi-hole Web Admin files out of date"
+    getGitFiles "${ADMIN_INTERFACE_DIR}" "${ADMIN_INTERFACE_GIT_URL}"
 
-elif [[ ${piholeVersion} == ${piholeVersionLatest} && ${webVersion} != ${webVersionLatest} ]]; then
-	echo "::: Pi-hole Web Admin files out of date"
-	getGitFiles ${webInterfaceDir} ${webInterfaceGitUrl}
-	echo ":::"
-	webVersion=$(pihole -v -a -c)
-	echo "::: Web Admin version is now at ${webVersion}"
-	echo "::: If you had made any changes in '/var/www/html/admin', they have been stashed using 'git stash'"
-	echo ""
-elif [[ ${piholeVersion} != ${piholeVersionLatest} && ${webVersion} == ${webVersionLatest} ]]; then
-	echo "::: Pi-hole core files out of date"
-	getGitFiles ${piholeFilesDir} ${piholeGitUrl}
-	/etc/.pihole/automated\ install/basic-install.sh --reconfigure --unattended
-	echo ":::"
-	piholeVersion=$(pihole -v -p -c)
-	echo "::: Pi-hole version is now at ${piholeVersion}"
-	echo "::: If you had made any changes in '/etc/.pihole', they have been stashed using 'git stash'"
-	echo ""
-elif [[ ${piholeVersion} != ${piholeVersionLatest} && ${webVersion} != ${webVersionLatest} ]]; then
-	echo "::: Updating Everything"
-	getGitFiles ${piholeFilesDir} ${piholeGitUrl}
-	/etc/.pihole/automated\ install/basic-install.sh --unattended
-	webVersion=$(pihole -v -a -c)
-	piholeVersion=$(pihole -v -p -c)
-	echo ":::"
-	echo "::: Pi-hole version is now at ${piholeVersion}"
-	echo "::: If you had made any changes in '/etc/.pihole', they have been stashed using 'git stash'"
-	echo ":::"
-	echo "::: Pi-hole version is now at ${piholeVersion}"
-	echo "::: If you had made any changes in '/etc/.pihole', they have been stashed using 'git stash'"
-	echo ""
-fi
+    web_updated=true
+
+  elif [[ "${pihole_version_current}" < "${pihole_version_latest}" ]] && [[ "${web_version_current}" == "${web_version_latest}" ]]; then
+    echo "::: Pi-hole core files out of date"
+    getGitFiles "${PI_HOLE_FILES_DIR}" "${PI_HOLE_GIT_URL}"
+    /etc/.pihole/automated\ install/basic-install.sh --reconfigure --unattended || echo "Unable to complete update, contact Pi-hole" && exit 1
+    core_updated=true
+
+  elif [[ "${pihole_version_current}" < "${pihole_version_latest}" ]] && [[ "${web_version_current}" < "${web_version_latest}" ]]; then
+    echo "::: Updating Everything"
+    getGitFiles "${PI_HOLE_FILES_DIR}" "${PI_HOLE_GIT_URL}"
+    /etc/.pihole/automated\ install/basic-install.sh --unattended || echo "Unable to complete update, contact Pi-hole" && exit 1
+    web_updated=true
+    core_updated=true
+  else
+    echo "*** Update script has malfunctioned, fallthrough reached. Please contact support"
+    exit 1
+  fi
+
+  if [[ "${web_updated}" == true ]]; then
+    web_version_current="$(/usr/local/bin/pihole version --admin --current)"
+    echo ":::"
+    echo "::: Web Admin version is now at ${web_version_current}"
+    echo "::: If you had made any changes in '/var/www/html/admin/', they have been stashed using 'git stash'"
+  fi
+
+  if [[ "${core_updated}" == true ]]; then
+    pihole_version_current="$(/usr/local/bin/pihole version --pihole --current)"
+    echo ":::"
+    echo "::: Pi-hole version is now at ${pihole_version_current}"
+    echo "::: If you had made any changes in '/etc/.pihole/', they have been stashed using 'git stash'"
+  fi
+
+  echo ""
+  exit 0
+
+}
+
+main

advanced/Scripts/version.sh

@@ -14,6 +14,8 @@
 latest=false
 current=false
 
+DEFAULT="-1"
+
 normalOutput() {
 	piholeVersion=$(cd /etc/.pihole/ && git describe --tags --abbrev=0)
 	webVersion=$(cd /var/www/html/admin/ && git describe --tags --abbrev=0)
@@ -21,8 +23,8 @@ normalOutput() {
 	piholeVersionLatest=$(curl -s https://api.github.com/repos/pi-hole/pi-hole/releases/latest | grep -Po '"tag_name":.*?[^\\]",' |  perl -pe 's/"tag_name": "//; s/^"//; s/",$//')
 	webVersionLatest=$(curl -s https://api.github.com/repos/pi-hole/AdminLTE/releases/latest | grep -Po '"tag_name":.*?[^\\]",' |  perl -pe 's/"tag_name": "//; s/^"//; s/",$//')
 
-	echo "::: Pi-hole version is ${piholeVersion} (Latest version is ${piholeVersionLatest})"
-	echo "::: Web-Admin version is ${webVersion} (Latest version is ${webVersionLatest})"
+	echo "::: Pi-hole version is ${piholeVersion} (Latest version is ${piholeVersionLatest:-${DEFAULT}})"
+	echo "::: Web-Admin version is ${webVersion} (Latest version is ${webVersionLatest:-${DEFAULT}})"
 }
 
 webOutput() {
@@ -36,14 +38,14 @@ webOutput() {
 
 	if [[ "${latest}" == true && "${current}" == false ]]; then
 		webVersionLatest=$(curl -s https://api.github.com/repos/pi-hole/AdminLTE/releases/latest | grep -Po '"tag_name":.*?[^\\]",' |  perl -pe 's/"tag_name": "//; s/^"//; s/",$//')
-		echo ${webVersionLatest}
+		echo "${webVersionLatest:--1}"
 	elif [[ "${latest}" == false && "${current}" == true ]]; then
 		webVersion=$(cd /var/www/html/admin/ && git describe --tags --abbrev=0)
-		echo ${webVersion}
+		echo "${webVersion}"
 	else
 		webVersion=$(cd /var/www/html/admin/ && git describe --tags --abbrev=0)
 		webVersionLatest=$(curl -s https://api.github.com/repos/pi-hole/AdminLTE/releases/latest | grep -Po '"tag_name":.*?[^\\]",' |  perl -pe 's/"tag_name": "//; s/^"//; s/",$//')
-		echo "::: Web-Admin version is $webVersion (Latest version is $webVersionLatest)"
+		echo "::: Web-Admin version is ${webVersion} (Latest version is ${webVersionLatest:-${DEFAULT}})"
 	fi
 }
 
@@ -58,14 +60,14 @@ coreOutput() {
 
 	if [[ "${latest}" == true && "${current}" == false ]]; then
 		piholeVersionLatest=$(curl -s https://api.github.com/repos/pi-hole/pi-hole/releases/latest | grep -Po '"tag_name":.*?[^\\]",' |  perl -pe 's/"tag_name": "//; s/^"//; s/",$//')
-		echo ${piholeVersionLatest}
+		echo "${piholeVersionLatest:--1}"
 	elif [[ "${latest}" == false && "${current}" == true ]]; then
 		piholeVersion=$(cd /etc/.pihole/ && git describe --tags --abbrev=0)
-		echo ${piholeVersion}
+		echo "${piholeVersion}"
 	else
 		piholeVersion=$(cd /etc/.pihole/ && git describe --tags --abbrev=0)
 		piholeVersionLatest=$(curl -s https://api.github.com/repos/pi-hole/pi-hole/releases/latest | grep -Po '"tag_name":.*?[^\\]",' |  perl -pe 's/"tag_name": "//; s/^"//; s/",$//')
-		echo "::: Pi-hole version is $piholeVersion (Latest version is $piholeVersionLatest)"
+		echo "::: Pi-hole version is ${piholeVersion} (Latest version is ${piholeVersionLatest:-${DEFAULT}})"
 	fi
 }
 
@@ -84,7 +86,7 @@ helpFunc() {
 :::  -h, --help           Show this help dialog
 :::
 EOM
-	exit 1
+	exit 0
 }
 
 if [[ $# = 0 ]]; then

advanced/Scripts/webpage.sh

@@ -0,0 +1,258 @@
+#!/usr/bin/env bash
+# Pi-hole: A black hole for Internet advertisements
+# Network-wide ad blocking via your Raspberry Pi
+# http://pi-hole.net
+# Web interface settings
+#
+# Pi-hole is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+
+args=("$@")
+
+helpFunc() {
+	cat << EOM
+::: Set admin options for the web interface of pihole
+:::
+::: Usage: pihole -a [options]
+:::
+::: Options:
+:::  -p, password		Set web interface password, an empty input will remove any previously set password
+:::  -c, celsius		Set Celsius temperature unit
+:::  -f, fahrenheit		Set Fahrenheit temperature unit
+:::  -k, kelvin			Set Kelvin temperature unit
+:::  -h, --help			Show this help dialog
+EOM
+	exit 0
+}
+
+SetTemperatureUnit(){
+
+	# Remove setting from file (create backup setupVars.conf.bak)
+	sed -i.bak '/TEMPERATUREUNIT/d' /etc/pihole/setupVars.conf
+	# Save setting to file
+	echo "TEMPERATUREUNIT=${unit}" >> /etc/pihole/setupVars.conf
+
+}
+
+SetWebPassword(){
+
+	if [ "${SUDO_USER}" == "www-data" ]; then
+		echo "Security measure: user www-data is not allowed to change webUI password!"
+		echo "Exiting"
+		exit 1
+	fi
+
+	if [ "${SUDO_USER}" == "lighttpd" ]; then
+		echo "Security measure: user lighttpd is not allowed to change webUI password!"
+		echo "Exiting"
+		exit 1
+	fi
+
+	# Remove password from file (create backup setupVars.conf.bak)
+	sed -i.bak '/WEBPASSWORD/d' /etc/pihole/setupVars.conf
+	# Set password only if there is one to be set
+	if (( ${#args[2]} > 0 )) ; then
+		# Compute password hash twice to avoid rainbow table vulnerability
+		hash=$(echo -n ${args[2]} | sha256sum | sed 's/\s.*$//')
+		hash=$(echo -n ${hash} | sha256sum | sed 's/\s.*$//')
+		# Save hash to file
+		echo "WEBPASSWORD=${hash}" >> /etc/pihole/setupVars.conf
+		echo "New password set"
+	else
+		echo "WEBPASSWORD=" >> /etc/pihole/setupVars.conf
+		echo "Password removed"
+	fi
+
+}
+
+SetDNSServers(){
+
+	# Remove setting from file (create backup setupVars.conf.bak)
+	sed -i.bak '/PIHOLE_DNS_1/d;/PIHOLE_DNS_2/d;/DNS_FQDN_REQUIRED/d;/DNS_BOGUS_PRIV/d;' /etc/pihole/setupVars.conf
+	# Save setting to file
+	echo "PIHOLE_DNS_1=${args[2]}" >> /etc/pihole/setupVars.conf
+	if [[ "${args[3]}" != "none" ]]; then
+		echo "PIHOLE_DNS_2=${args[3]}" >> /etc/pihole/setupVars.conf
+	else
+		echo "PIHOLE_DNS_2=" >> /etc/pihole/setupVars.conf
+	fi
+
+	# Replace within actual dnsmasq config file
+	sed -i '/server=/d;' /etc/dnsmasq.d/01-pihole.conf
+	echo "server=${args[2]}" >> /etc/dnsmasq.d/01-pihole.conf
+	if [[ "${args[3]}" != "none" ]]; then
+		echo "server=${args[3]}" >> /etc/dnsmasq.d/01-pihole.conf
+	fi
+
+	# Remove domain-needed entry
+	sed -i '/domain-needed/d;' /etc/dnsmasq.d/01-pihole.conf
+
+	# Readd it if required
+	if [[ "${args[4]}" == "domain-needed" ]]; then
+		echo "domain-needed" >> /etc/dnsmasq.d/01-pihole.conf
+		echo "DNS_FQDN_REQUIRED=true" >> /etc/pihole/setupVars.conf
+	else
+		# Leave it deleted if not wanted
+		echo "DNS_FQDN_REQUIRED=false" >> /etc/pihole/setupVars.conf
+	fi
+
+	# Remove bogus-priv entry
+	sed -i '/bogus-priv/d;' /etc/dnsmasq.d/01-pihole.conf
+
+	# Readd it if required
+	if [[ "${args[5]}" == "bogus-priv" ]]; then
+		echo "bogus-priv" >> /etc/dnsmasq.d/01-pihole.conf
+		echo "DNS_BOGUS_PRIV=true" >> /etc/pihole/setupVars.conf
+	else
+		# Leave it deleted if not wanted
+		echo "DNS_BOGUS_PRIV=false" >> /etc/pihole/setupVars.conf
+	fi
+
+	# Restart dnsmasq to load new configuration
+	RestartDNS
+
+}
+
+SetExcludeDomains(){
+
+	# Remove setting from file (create backup setupVars.conf.bak)
+	sed -i.bak '/API_EXCLUDE_DOMAINS/d;' /etc/pihole/setupVars.conf
+	# Save setting to file
+	echo "API_EXCLUDE_DOMAINS=${args[2]}" >> /etc/pihole/setupVars.conf
+}
+
+SetExcludeClients(){
+
+	# Remove setting from file (create backup setupVars.conf.bak)
+	sed -i.bak '/API_EXCLUDE_CLIENTS/d;' /etc/pihole/setupVars.conf
+	# Save setting to file
+	echo "API_EXCLUDE_CLIENTS=${args[2]}" >> /etc/pihole/setupVars.conf
+}
+
+Reboot(){
+
+	nohup bash -c "sleep 5; reboot" &> /dev/null </dev/null &
+
+}
+
+RestartDNS(){
+
+	if [ -x "$(command -v systemctl)" ]; then
+		systemctl restart dnsmasq &> /dev/null
+	else
+		service dnsmasq restart &> /dev/null
+	fi
+
+}
+
+SetQueryLogOptions(){
+
+	# Remove setting from file (create backup setupVars.conf.bak)
+	sed -i.bak '/API_QUERY_LOG_SHOW/d;' /etc/pihole/setupVars.conf
+	# Save setting to file
+	echo "API_QUERY_LOG_SHOW=${args[2]}" >> /etc/pihole/setupVars.conf
+}
+
+EnableDHCP(){
+
+	# Remove setting from file (create backup setupVars.conf.bak)
+	sed -i.bak '/DHCP_/d;' /etc/pihole/setupVars.conf
+	echo "DHCP_ACTIVE=true" >> /etc/pihole/setupVars.conf
+	echo "DHCP_START=${args[2]}" >> /etc/pihole/setupVars.conf
+	echo "DHCP_END=${args[3]}" >> /etc/pihole/setupVars.conf
+	echo "DHCP_ROUTER=${args[4]}" >> /etc/pihole/setupVars.conf
+
+	# Remove setting from file
+	sed -i '/dhcp-/d;/quiet-dhcp/d;' /etc/dnsmasq.d/01-pihole.conf
+	# Save setting to file
+	echo "dhcp-range=${args[2]},${args[3]},infinite" >> /etc/dnsmasq.d/01-pihole.conf
+	echo "dhcp-option=option:router,${args[4]}" >> /etc/dnsmasq.d/01-pihole.conf
+	# Changes the behaviour from strict RFC compliance so that DHCP requests on unknown leases from unknown hosts are not ignored. This allows new hosts to get a lease without a tedious timeout under all circumstances. It also allows dnsmasq to rebuild its lease database without each client needing to reacquire a lease, if the database is lost.
+	echo "dhcp-authoritative" >> /etc/dnsmasq.d/01-pihole.conf
+	# Use the specified file to store DHCP lease information
+	echo "dhcp-leasefile=/etc/pihole/dhcp.leases" >> /etc/dnsmasq.d/01-pihole.conf
+	# Suppress logging of the routine operation of these protocols. Errors and problems will still be logged, though.
+	echo "quiet-dhcp" >> /etc/dnsmasq.d/01-pihole.conf
+	echo "quiet-dhcp6" >> /etc/dnsmasq.d/01-pihole.conf
+
+	RestartDNS
+}
+
+DisableDHCP(){
+
+	# Remove setting from file (create backup setupVars.conf.bak)
+	sed -i.bak '/DHCP_ACTIVE/d;' /etc/pihole/setupVars.conf
+	echo "DHCP_ACTIVE=false" >> /etc/pihole/setupVars.conf
+
+	# Remove setting from file
+	sed -i '/dhcp-/d;/quiet-dhcp/d;' /etc/dnsmasq.d/01-pihole.conf
+
+	RestartDNS
+}
+
+SetWebUILayout(){
+
+	# Remove setting from file (create backup setupVars.conf.bak)
+	sed -i.bak '/WEBUIBOXEDLAYOUT/d;' /etc/pihole/setupVars.conf
+	echo "WEBUIBOXEDLAYOUT=${args[2]}" >> /etc/pihole/setupVars.conf
+
+}
+
+SetDNSDomainName(){
+
+	# Remove setting from file (create backup setupVars.conf.bak)
+	sed -i.bak '/PIHOLE_DOMAIN/d;' /etc/pihole/setupVars.conf
+	# Save setting to file
+	echo "PIHOLE_DOMAIN=${args[2]}" >> /etc/pihole/setupVars.conf
+
+	# Replace within actual dnsmasq config file
+	sed -i '/domain=/d;' /etc/dnsmasq.d/01-pihole.conf
+	echo "domain=${args[2]}" >> /etc/dnsmasq.d/01-pihole.conf
+
+	# Restart dnsmasq to load new configuration
+	RestartDNS
+
+}
+
+ResolutionSettings() {
+
+	typ=${args[2]}
+	state=${args[3]}
+
+	if [[ "${typ}" == "forward" ]]; then
+		sed -i.bak '/API_GET_UPSTREAM_DNS_HOSTNAME/d;' /etc/pihole/setupVars.conf
+		echo "API_GET_UPSTREAM_DNS_HOSTNAME=${state}" >> /etc/pihole/setupVars.conf
+	elif [[ "${typ}" == "clients" ]]; then
+		sed -i.bak '/API_GET_CLIENT_HOSTNAME/d;' /etc/pihole/setupVars.conf
+		echo "API_GET_CLIENT_HOSTNAME=${state}" >> /etc/pihole/setupVars.conf
+	fi
+}
+
+case "${args[1]}" in
+	"-p" | "password"   ) SetWebPassword;;
+	"-c" | "celsius"    ) unit="C"; SetTemperatureUnit;;
+	"-f" | "fahrenheit" ) unit="F"; SetTemperatureUnit;;
+	"-k" | "kelvin"     ) unit="K"; SetTemperatureUnit;;
+	"setdns"            ) SetDNSServers;;
+	"setexcludedomains" ) SetExcludeDomains;;
+	"setexcludeclients" ) SetExcludeClients;;
+	"reboot"            ) Reboot;;
+	"restartdns"        ) RestartDNS;;
+	"setquerylog"       ) SetQueryLogOptions;;
+	"enabledhcp"        ) EnableDHCP;;
+	"disabledhcp"       ) DisableDHCP;;
+	"layout"            ) SetWebUILayout;;
+	"-h" | "--help"     ) helpFunc;;
+	"domainname"        ) SetDNSDomainName;;
+	"resolve"           ) ResolutionSettings;;
+	*                   ) helpFunc;;
+esac
+
+shift
+
+if [[ $# = 0 ]]; then
+	helpFunc
+fi
+

advanced/bash-completion/pihole

@@ -3,7 +3,7 @@ _pihole() {
 	COMPREPLY=()
 	cur="${COMP_WORDS[COMP_CWORD]}"
 	prev="${COMP_WORDS[COMP_CWORD-1]}"
-	opts="blacklist chronometer debug flush help query reconfigure setupLCD uninstall updateGravity updatePihole version whitelist"
+	opts="admin blacklist chronometer debug disable enable flush help logging query reconfigure restartdns setupLCD status tail uninstall updateGravity updatePihole version whitelist"
 
 	COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )
 	return 0

advanced/pihole.cron

@@ -8,7 +8,7 @@
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 2 of the License, or
 # (at your option) any later version.
-# 
+#
 # This file is under source-control of the Pi-hole installation and update
 # scripts, any changes made to this file will be overwritten when the softare
 # is updated or re-installed. Please make any changes to the appropriate crontab
@@ -16,11 +16,11 @@
 
 # Pi-hole: Update the ad sources once a week on Sunday at 01:59
 #          Download any updates from the adlists
-59 1    * * 7   root    /usr/local/bin/pihole updateGravity
+59 1    * * 7   root    PATH="$PATH:/usr/local/bin/" pihole updateGravity
 
 # Pi-hole: Update Pi-hole! Uncomment to enable auto update
-#30 2    * * 7   root    /usr/local/bin/pihole updatePihole
+#30 2    * * 7   root    PATH="$PATH:/usr/local/bin/" pihole updatePihole
 
 # Pi-hole: Flush the log daily at 00:00 so it doesn't get out of control
 #          Stats will be viewable in the Web interface thanks to the cron job above
-00 00   * * *   root    /usr/local/bin/pihole flush
+00 00   * * *   root    PATH="$PATH:/usr/local/bin/" pihole flush

advanced/pihole.sudo

@@ -9,4 +9,3 @@
 # the Free Software Foundation, either version 2 of the License, or
 # (at your option) any later version.
 
-www-data ALL=NOPASSWD: /usr/local/bin/pihole

advanced/selinux/pihole.te

@@ -1,87 +0,0 @@
-module pihole 1.0;
-
-require {
-        type var_log_t;
-        type unconfined_t;
-        type init_t;
-        type auditd_t;
-        type syslogd_t;
-        type NetworkManager_t;
-        type mdadm_t;
-        type tuned_t;
-        type avahi_t;
-        type irqbalance_t;
-        type system_dbusd_t;
-        type kernel_t;
-        type httpd_sys_script_t;
-        type systemd_logind_t;
-        type httpd_t;
-        type policykit_t;
-        type dnsmasq_t;
-        type udev_t;
-        type postfix_pickup_t;
-        type sshd_t;
-        type crond_t;
-        type getty_t;
-        type lvm_t;
-        type postfix_qmgr_t;
-        type postfix_master_t;
-        class dir { getattr search };
-        class file { read open setattr };
-}
-
-#============= dnsmasq_t ==============
-allow dnsmasq_t var_log_t:file { open setattr };
-
-#============= httpd_t ==============
-allow httpd_t var_log_t:file { read open };
-
-#============= httpd_sys_script_t (class: dir) ==============
-allow httpd_sys_script_t NetworkManager_t:dir { getattr search };
-allow httpd_sys_script_t auditd_t:dir { getattr search };
-allow httpd_sys_script_t avahi_t:dir { getattr search };
-allow httpd_sys_script_t crond_t:dir { getattr search };
-allow httpd_sys_script_t dnsmasq_t:dir { getattr search };
-allow httpd_sys_script_t getty_t:dir { getattr search };
-allow httpd_sys_script_t httpd_t:dir { getattr search };
-allow httpd_sys_script_t init_t:dir { getattr search };
-allow httpd_sys_script_t irqbalance_t:dir { getattr search };
-allow httpd_sys_script_t kernel_t:dir { getattr search };
-allow httpd_sys_script_t lvm_t:dir { getattr search };
-allow httpd_sys_script_t mdadm_t:dir { getattr search };
-allow httpd_sys_script_t policykit_t:dir { getattr search };
-allow httpd_sys_script_t postfix_master_t:dir { getattr search };
-allow httpd_sys_script_t postfix_pickup_t:dir { getattr search };
-allow httpd_sys_script_t postfix_qmgr_t:dir { getattr search };
-allow httpd_sys_script_t sshd_t:dir { getattr search };
-allow httpd_sys_script_t syslogd_t:dir { getattr search };
-allow httpd_sys_script_t system_dbusd_t:dir { getattr search };
-allow httpd_sys_script_t systemd_logind_t:dir { getattr search };
-allow httpd_sys_script_t tuned_t:dir { getattr search };
-allow httpd_sys_script_t udev_t:dir { getattr search };
-allow httpd_sys_script_t unconfined_t:dir { getattr search };
-
-#============= httpd_sys_script_t (class: file) ==============
-allow httpd_sys_script_t NetworkManager_t:file { read open };
-allow httpd_sys_script_t auditd_t:file { read open };
-allow httpd_sys_script_t avahi_t:file { read open };
-allow httpd_sys_script_t crond_t:file { read open };
-allow httpd_sys_script_t dnsmasq_t:file { read open };
-allow httpd_sys_script_t getty_t:file { read open };
-allow httpd_sys_script_t httpd_t:file { read open };
-allow httpd_sys_script_t init_t:file { read open };
-allow httpd_sys_script_t irqbalance_t:file { read open };
-allow httpd_sys_script_t kernel_t:file { read open };
-allow httpd_sys_script_t lvm_t:file { read open };
-allow httpd_sys_script_t mdadm_t:file { read open };
-allow httpd_sys_script_t policykit_t:file { read open };
-allow httpd_sys_script_t postfix_master_t:file { read open };
-allow httpd_sys_script_t postfix_pickup_t:file { read open };
-allow httpd_sys_script_t postfix_qmgr_t:file { read open };
-allow httpd_sys_script_t sshd_t:file { read open };
-allow httpd_sys_script_t syslogd_t:file { read open };
-allow httpd_sys_script_t system_dbusd_t:file { read open };
-allow httpd_sys_script_t systemd_logind_t:file { read open };
-allow httpd_sys_script_t tuned_t:file { read open };
-allow httpd_sys_script_t udev_t:file { read open };
-allow httpd_sys_script_t unconfined_t:file { read open };

automated install/uninstall.sh

@@ -82,7 +82,7 @@ removeAndPurge() {
 				read -rp "::: Do you wish to remove ${i} from your system? [y/n]: " yn
 				case ${yn} in
 					[Yy]* ) printf ":::\tRemoving %s..." "${i}"; ${SUDO} ${PKG_REMOVE} "${i}" &> /dev/null & spinner $!; printf "done!\n"; break;;
-					[Nn]* ) printf ":::\tSkipping %s" "${i}\n"; break;;
+					[Nn]* ) printf ":::\tSkipping %s\n" "${i}"; break;;
 					* ) printf "::: You must answer yes or no!\n";;
 				esac
 			done
@@ -136,7 +136,7 @@ removeNoPurge() {
 	fi
 
 	echo "::: Removing config files and scripts..."
-	package_check ${i} > /dev/null
+	package_check lighttpd > /dev/null
 	if [ $? -eq 1 ]; then
 		${SUDO} rm -rf /etc/lighttpd/ &> /dev/null
 	else
@@ -154,6 +154,12 @@ removeNoPurge() {
 	${SUDO} rm /usr/local/bin/pihole &> /dev/null
 	${SUDO} rm /etc/bash_completion.d/pihole &> /dev/null
 	${SUDO} rm /etc/sudoers.d/pihole &> /dev/null
+	
+	# If the pihole user exists, then remove
+	if id "pihole" >/dev/null 2>&1; then
+        	echo "::: Removing pihole user..."
+		${SUDO} userdel -r pihole
+	fi
 
 	echo ":::"
 	printf "::: Finished removing PiHole from your system. Sorry to see you go!\n"

autotest

@@ -0,0 +1 @@
+py.test -v -f test/

gravity.sh

@@ -23,7 +23,7 @@ helpFunc() {
 :::  -f, --force			Force lists to be downloaded, even if they don't need updating.
 :::  -h, --help				Show this help dialog
 EOM
-	exit 1
+	exit 0
 }
 
 
@@ -44,12 +44,14 @@ else
 fi
 
 #Remove the /* from the end of the IPv4addr.
-IPv4_address=${IPv4_address%/*}
+IPV4_ADDRESS=${IPV4_ADDRESS%/*}
+IPV6_ADDRESS=${IPV6_ADDRESS}
 
 # Variables for various stages of downloading and formatting the list
 basename=pihole
 piholeDir=/etc/${basename}
 adList=${piholeDir}/gravity.list
+localList=${piholeDir}/local.list
 justDomainsExtension=domains
 matterAndLight=${basename}.0.matterandlight.txt
 supernova=${basename}.1.supernova.txt
@@ -102,16 +104,30 @@ gravity_collapse() {
 # patternCheck - check to see if curl downloaded any new files.
 gravity_patternCheck() {
 	patternBuffer=$1
-	# check if the patternbuffer is a non-zero length file
-	if [[ -s "${patternBuffer}" ]]; then
-		# Some of the blocklists are copyright, they need to be downloaded
-		# and stored as is. They can be processed for content after they
-		# have been saved.
-		mv "${patternBuffer}" "${saveLocation}"
-		echo " List updated, transport successful!"
+	success=$2
+	error=$3
+	if [ $success = true ]; then
+		# check if download was successful but list has not been modified
+		if [ "${error}" == "304" ]; then
+			echo ":::   No changes detected, transport skipped!"
+		# check if the patternbuffer is a non-zero length file
+		elif [[ -s "${patternBuffer}" ]]; then
+			# Some of the blocklists are copyright, they need to be downloaded
+			# and stored as is. They can be processed for content after they
+			# have been saved.
+			mv "${patternBuffer}" "${saveLocation}"
+			echo ":::   List updated, transport successful!"
+		else
+			# Empty file -> use previously downloaded list
+			echo ":::   Received empty file, using cached one (list not updated!)"
+		fi
 	else
-		# curl didn't download any host files, probably because of the date check
-		echo " No changes detected, transport skipped!"
+		# check if cached list exists
+		if [[ -r "${saveLocation}" ]]; then
+			echo ":::   List download failed, using cached list (list not updated!)"
+		else
+			echo ":::   Download failed and no cached list available (list will not be considered)"
+		fi
 	fi
 }
 
@@ -130,9 +146,27 @@ gravity_transport() {
 	fi
 
 	# Silently curl url
-	curl -s -L ${cmd_ext} ${heisenbergCompensator} -A "${agent}" ${url} > ${patternBuffer}
-	# Check for list updates
-	gravity_patternCheck "${patternBuffer}"
+	err=$(curl -s -L ${cmd_ext} ${heisenbergCompensator} -w %{http_code} -A "${agent}" ${url} -o ${patternBuffer})
+
+	echo " done"
+	# Analyze http response
+	echo -n ":::   Status: "
+	case "$err" in
+		"200"  ) echo "Success (OK)"; success=true;;
+		"304"  ) echo "Not modified"; success=true;;
+		"403"  ) echo "Forbidden"; success=false;;
+		"404"  ) echo "Not found"; success=false;;
+		"408"  ) echo "Time-out"; success=false;;
+		"451"  ) echo "Unavailable For Legal Reasons"; success=false;;
+		"521"  ) echo "Web Server Is Down (Cloudflare)"; success=false;;
+		"522"  ) echo "Connection Timed Out (Cloudflare)"; success=false;;
+		"500"  ) echo "Internal Server Error"; success=false;;
+		*      ) echo "Status $err"; success=false;;
+	esac
+
+	# Process result
+	gravity_patternCheck "${patternBuffer}" ${success} "${err}"
+
 }
 
 # spinup - main gravity function
@@ -179,7 +213,10 @@ gravity_Schwarzchild() {
 	echo -n "::: Aggregating list of domains..."
 	truncate -s 0 ${piholeDir}/${matterAndLight}
 	for i in "${activeDomains[@]}"; do
-		cat "${i}" | tr -d '\r' >> ${piholeDir}/${matterAndLight}
+		# Only assimilate list if it is available (download might have faild permanently)
+		if [[ -r "${i}" ]]; then
+			cat "${i}" | tr -d '\r' >> ${piholeDir}/${matterAndLight}
+		fi
 	done
 	echo " done!"
 }
@@ -189,7 +226,7 @@ gravity_Blacklist() {
 	if [[ -f "${blacklistFile}" ]]; then
 	    numBlacklisted=$(wc -l < "${blacklistFile}")
 	    plural=; [[ "$numBlacklisted" != "1" ]] && plural=s
-	    echo -n "::: BlackListing $numBlacklisted domain${plural}..."
+	    echo -n "::: Blacklisting $numBlacklisted domain${plural}..."
 	    cat ${blacklistFile} >> ${piholeDir}/${eventHorizon}
 	    echo " done!"
 	else
@@ -240,30 +277,42 @@ gravity_unique() {
 
 gravity_hostFormat() {
 	# Format domain list as "192.168.x.x domain.com"
-	echo "::: Formatting domains into a HOSTS file..."
-    # Check vars from setupVars.conf to see if we're using IPv4, IPv6, Or both.
-    if [[ -n "${IPv4_address}" && -n "${IPv6_address}" ]];then
+	echo -n "::: Formatting domains into a HOSTS file..."
 
-        # Both IPv4 and IPv6
-        cat ${piholeDir}/${eventHorizon} | awk -v ipv4addr="$IPv4_address" -v ipv6addr="$IPv6_address" '{sub(/\r$/,""); print ipv4addr" "$0"\n"ipv6addr" "$0}' >> ${piholeDir}/${accretionDisc}
+	if [[ -f /etc/hostname ]]; then
+		hostname=$(</etc/hostname)
+	elif [ -x "$(command -v hostname)" ]; then
+		hostname=$(hostname -f)
+	else
+		echo "::: Error: Unable to determine fully qualified domain name of host"
+	fi
+  # Check vars from setupVars.conf to see if we're using IPv4, IPv6, Or both.
+  if [[ -n "${IPV4_ADDRESS}" && -n "${IPV6_ADDRESS}" ]];then
 
-    elif [[ -n "${IPv4_address}" && -z "${IPv6_address}" ]];then
+      echo -e "${IPV4_ADDRESS} ${hostname}\n${IPV6_ADDRESS} ${hostname}\n${IPV4_ADDRESS} pi.hole\n${IPV6_ADDRESS} pi.hole" > ${localList}
+      # Both IPv4 and IPv6
+      cat ${piholeDir}/${eventHorizon} | awk -v ipv4addr="$IPV4_ADDRESS" -v ipv6addr="$IPV6_ADDRESS" '{sub(/\r$/,""); print ipv4addr" "$0"\n"ipv6addr" "$0}' >> ${piholeDir}/${accretionDisc}
 
-        # Only IPv4
-        cat ${piholeDir}/${eventHorizon} | awk -v ipv4addr="$IPv4_address" '{sub(/\r$/,""); print ipv4addr" "$0}' >> ${piholeDir}/${accretionDisc}
+  elif [[ -n "${IPV4_ADDRESS}" && -z "${IPV6_ADDRESS}" ]];then
 
-    elif [[ -z "${IPv4_address}" && -n "${IPv6_address}" ]];then
+      echo -e "${IPV4_ADDRESS} ${hostname}\n${IPV4_ADDRESS} pi.hole" > ${localList}
+      # Only IPv4
+      cat ${piholeDir}/${eventHorizon} | awk -v ipv4addr="$IPV4_ADDRESS" '{sub(/\r$/,""); print ipv4addr" "$0}' >> ${piholeDir}/${accretionDisc}
 
-        # Only IPv6
-        cat ${piholeDir}/${eventHorizon} | awk -v ipv6addr="$IPv6_address" '{sub(/\r$/,""); print ipv6addr" "$0}' >> ${piholeDir}/${accretionDisc}
+  elif [[ -z "${IPV4_ADDRESS}" && -n "${IPV6_ADDRESS}" ]];then
 
-    elif [[ -z "${IPv4_address}" && -z "${IPv6_address}" ]];then
-        echo "::: No IP Values found! Please run 'pihole -r' and choose reconfigure to restore values"
-        exit 1
-    fi
+      echo -e "${IPV6_ADDRESS} ${hostname}\n${IPV6_ADDRESS} pi.hole" > ${localList}
+      # Only IPv6
+      cat ${piholeDir}/${eventHorizon} | awk -v ipv6addr="$IPV6_ADDRESS" '{sub(/\r$/,""); print ipv6addr" "$0}' >> ${piholeDir}/${accretionDisc}
+
+  elif [[ -z "${IPV4_ADDRESS}" && -z "${IPV6_ADDRESS}" ]];then
+      echo "::: No IP Values found! Please run 'pihole -r' and choose reconfigure to restore values"
+      exit 1
+  fi
 
 	# Copy the file over as /etc/pihole/gravity.list so dnsmasq can use it
 	cp ${piholeDir}/${accretionDisc} ${adList}
+	echo " done!"
 }
 
 # blackbody - remove any remnant files from script processes
@@ -287,8 +336,14 @@ gravity_advanced() {
 	echo -n "::: Formatting list of domains to remove comments...."
 	#awk '($1 !~ /^#/) { if (NF>1) {print $2} else {print $1}}' ${piholeDir}/${matterAndLight} | sed -nr -e 's/\.{2,}/./g' -e '/\./p' >  ${piholeDir}/${supernova}
 	#Above line does not correctly grab domains where comment is on the same line (e.g 'addomain.com #comment')
-	#Add additional awk command to read all lines up to a '#', and then continue as we were
-	cat ${piholeDir}/${matterAndLight} | awk -F'#' '{print $1}' | awk '($1 !~ /^#/) { if (NF>1) {print $2} else {print $1}}' | sed -nr -e 's/\.{2,}/./g' -e '/\./p' >  ${piholeDir}/${supernova}
+	#Awk -F splits on given IFS, we grab the right hand side (chops trailing #coments and /'s to grab the domain only.
+	#Last awk command takes non-commented lines and if they have 2 fields, take the left field (the domain) and leave
+	#+ the right (IP address), otherwise grab the single field.
+	cat ${piholeDir}/${matterAndLight} | \
+	    awk -F '#' '{print $1}' | \
+	    awk -F '/' '{print $1}' | \
+	    awk '($1 !~ /^#/) { if (NF>1) {print $2} else {print $1}}' | \
+	    sed -nr -e 's/\.{2,}/./g' -e '/\./p' >  ${piholeDir}/${supernova}
 	echo " done!"
 
 	numberOf=$(wc -l < ${piholeDir}/${supernova})
@@ -306,7 +361,7 @@ gravity_reload() {
 
 	# Reload hosts file
 	echo ":::"
-	echo "::: Refresh lists in dnsmasq..."
+	echo -n "::: Refresh lists in dnsmasq..."
 
 	#ensure /etc/dnsmasq.d/01-pihole.conf is pointing at the correct list!
 	#First escape forward slashes in the path:
@@ -314,7 +369,8 @@ gravity_reload() {
 	#Now replace the line in dnsmasq file
 #	sed -i "s/^addn-hosts.*/addn-hosts=$adList/" /etc/dnsmasq.d/01-pihole.conf
 
-        pihole restartdns
+	pihole restartdns
+	echo " done!"
 }
 
 for var in "$@"; do
@@ -332,7 +388,7 @@ if [[ "${forceGrav}" == true ]]; then
 fi
 
 #Overwrite adlists.default from /etc/.pihole in case any changes have been made. Changes should be saved in /etc/adlists.list
-cp /etc/.pihole/adlists.default /etc/pihole/adlists.default
+#cp /etc/.pihole/adlists.default /etc/pihole/adlists.default
 gravity_collapse
 gravity_spinup
 if [[ "${skipDownload}" == false ]]; then

pihole

@@ -10,87 +10,107 @@
 # the Free Software Foundation, either version 2 of the License, or
 # (at your option) any later version.
 
+PI_HOLE_SCRIPT_DIR="/opt/pihole"
 # Must be root to use this tool
 if [[ ! $EUID -eq 0 ]];then
-	if [ -x "$(command -v sudo)" ];then
-		exec sudo bash "$0" "$@"
-		exit $?
-	else
-		echo "::: sudo is needed to run pihole commands.  Please run this script as root or install sudo."
-		exit 1
-	fi
+  if [ -x "$(command -v sudo)" ];then
+    exec sudo bash "$0" "$@"
+    exit $?
+  else
+    echo "::: sudo is needed to run pihole commands.  Please run this script as root or install sudo."
+    exit 1
+  fi
 fi
 
+webpageFunc() {
+  /opt/pihole/webpage.sh "$@"
+  exit 0
+}
+
 whitelistFunc() {
-	/opt/pihole/list.sh "$@"
-	exit 0
+ "${PI_HOLE_SCRIPT_DIR}"/list.sh "$@"
+  exit 0
 }
 
 blacklistFunc() {
-	/opt/pihole/list.sh "$@"
-	exit 0
+  "${PI_HOLE_SCRIPT_DIR}"/list.sh "$@"
+  exit 0
 }
 
 debugFunc() {
-	/opt/pihole/piholeDebug.sh
-	exit 0
+  "${PI_HOLE_SCRIPT_DIR}"/piholeDebug.sh
+  exit 0
 }
 
 flushFunc() {
-	/opt/pihole/piholeLogFlush.sh
-	exit 0
+  "${PI_HOLE_SCRIPT_DIR}"/piholeLogFlush.sh
+  exit 0
 }
 
-
 updatePiholeFunc() {
-	/opt/pihole/update.sh
-	exit 0
+  "${PI_HOLE_SCRIPT_DIR}"/update.sh
+  exit 0
 }
 
 reconfigurePiholeFunc() {
-	/etc/.pihole/automated\ install/basic-install.sh --reconfigure
-	exit 0;
+  /etc/.pihole/automated\ install/basic-install.sh --reconfigure
+  exit 0;
 }
 
 updateGravityFunc() {
-	/opt/pihole/gravity.sh "$@"
-	exit 0
+  "${PI_HOLE_SCRIPT_DIR}"/gravity.sh "$@"
+  exit 0
 }
 
 setupLCDFunction() {
-	/opt/pihole/setupLCD.sh
-	exit 0
+  "${PI_HOLE_SCRIPT_DIR}"/setupLCD.sh
+  exit 0
+}
+
+scanList(){
+  domain="${1}"
+  list="${2}"
+  method="${3}"
+  if [[ ${method} == "-exact" ]] ; then
+    grep -i -E "(^|\s)${domain}($|\s)" "${list}"
+  else
+    grep -i "${domain}" "${list}"
+  fi
 }
 
 queryFunc() {
-	domain=$2
-	for list in /etc/pihole/list.*; do
-		count=$(grep ${domain} $list | wc -l)
-		echo "::: ${list} (${count} results)"
-			if [[ ${count} > 0 ]]; then
-				grep ${domain} ${list}
-			fi
-		echo ""
-	done
-	exit 0
+  domain="${2}"
+  method="${3}"
+  lists=( /etc/pihole/list.* /etc/pihole/blacklist.txt)
+  for list in ${lists[@]}; do
+    result=$(scanList ${domain} ${list} ${method})
+    # Remove empty lines before couting number of results
+    count=$(sed '/^\s*$/d' <<< "$result" | wc -l)
+    echo "::: ${list} (${count} results)"
+    if [[ ${count} > 0 ]]; then
+      echo "${result}"
+    fi
+    echo ""
+  done
+  exit 0
 }
 
 chronometerFunc() {
-	shift
-	/opt/pihole/chronometer.sh "$@"
-	exit 0
+  shift
+  "${PI_HOLE_SCRIPT_DIR}"/chronometer.sh "$@"
+  exit 0
 }
 
 
 uninstallFunc() {
-	/opt/pihole/uninstall.sh
-	exit 0
+  "${PI_HOLE_SCRIPT_DIR}"/uninstall.sh
+  exit 0
 }
 
 versionFunc() {
-	shift
-	/opt/pihole/version.sh "$@"
-	exit 0
+  shift
+  "${PI_HOLE_SCRIPT_DIR}"/version.sh "$@"
+  exit 0
 }
 
 restartDNS() {
@@ -115,8 +135,26 @@ restartDNS() {
 piholeEnable() {
   if [[ "${1}" == "0" ]] ; then
     #Disable Pihole
-    sed -i 's/^addn-hosts/#addn-hosts/' /etc/dnsmasq.d/01-pihole.conf
+    sed -i 's/^addn-hosts=\/etc\/pihole\/gravity.list/#addn-hosts=\/etc\/pihole\/gravity.list/' /etc/dnsmasq.d/01-pihole.conf
     echo "::: Blocking has been disabled!"
+    if [[ $# > 1 ]] ; then
+      if [[ ${2} == *"s"* ]] ; then
+        tt=${2%"s"}
+        echo "::: Blocking will be re-enabled in ${tt} seconds"
+        nohup bash -c "sleep ${tt}; pihole enable" </dev/null &>/dev/null &
+      elif [[ ${2} == *"m"* ]] ; then
+        tt=${2%"m"}
+        echo "::: Blocking will be re-enabled in ${tt} minutes"
+        tt=$((${tt}*60))
+        nohup bash -c "sleep ${tt}; pihole enable" </dev/null &>/dev/null &
+      else
+        echo "::: Unknown format for delayed reactivation of the blocking!"
+        echo "::: Example:"
+        echo ":::    pihole disable 5s    -    will disable blocking for 5 seconds"
+        echo ":::    pihole disable 7m    -    will disable blocking for 7 minutes"
+        echo "::: Blocking will not automatically be re-enabled!"
+      fi
+    fi
   else
     #Enable pihole
     echo "::: Blocking has been enabled!"
@@ -125,15 +163,36 @@ piholeEnable() {
   restartDNS
 }
 
+piholeLogging() {
+  shift
+
+  if [[ "${1}" == "off" ]] ; then
+    #Disable Logging
+    sed -i 's/^log-queries/#log-queries/' /etc/dnsmasq.d/01-pihole.conf
+    sed -i 's/^QUERY_LOGGING=true/QUERY_LOGGING=false/' /etc/pihole/setupVars.conf
+    pihole -f
+    echo "::: Logging has been disabled!"
+  elif [[ "${1}" == "on" ]] ; then
+    #Enable logging
+    sed -i 's/^#log-queries/log-queries/' /etc/dnsmasq.d/01-pihole.conf
+    sed -i 's/^QUERY_LOGGING=false/QUERY_LOGGING=true/' /etc/pihole/setupVars.conf
+    echo "::: Logging has been enabled!"
+  else
+    echo "::: Invalid option passed, please pass 'on' or 'off'"
+    exit 1
+  fi
+  restartDNS
+}
+
 piholeStatus() {
-  if [[ $(cat /etc/dnsmasq.d/01-pihole.conf | grep "#addn-hosts=/") ]] ; then
+  if [[ $(grep -i "^#addn-hosts=/" /etc/dnsmasq.d/01-pihole.conf) ]] ; then
     #list is commented out
     if [[ "${1}" == "web" ]] ; then
       echo 0;
     else
       echo "::: Pi-hole blocking is Disabled";
     fi
-  elif [[ $(cat /etc/dnsmasq.d/01-pihole.conf | grep "addn-hosts=/") ]] ; then
+  elif [[ $(grep -i "^addn-hosts=/" /etc/dnsmasq.d/01-pihole.conf) ]] ; then
     #list set
     if [[ "${1}" == "web" ]] ; then
       echo 1;
@@ -153,57 +212,72 @@ piholeStatus() {
   fi
 }
 
+tailFunc() {
+  echo "Press Ctrl-C to exit"
+  tail -F /var/log/pihole.log
+  exit 0
+}
 
 helpFunc() {
 	cat << EOM
 ::: Control all PiHole specific functions!
 :::
 ::: Usage: pihole [options]
-:::		Add -h after -w (whitelist), -b (blacklist), or -c (chronometer)  for more information on usage
+:::		Add -h after -w (whitelist), -b (blacklist), -c (chronometer), or -a (admin)  for more information on usage
 :::
 ::: Options:
 :::  -w, whitelist            Whitelist domains
 :::  -b, blacklist            Blacklist domains
 :::  -d, debug                Start a debugging session if having trouble
 :::  -f, flush                Flush the pihole.log file
+:::  -t, tail                 Output the last lines of the pihole.log file. Lines are appended as the file grows
 :::  -up, updatePihole        Update Pi-hole
+:::  -r, reconfigure          Reconfigure or Repair Pi-hole
 :::  -g, updateGravity        Update the list of ad-serving domains
 :::  -s, setupLCD             Automatically configures the Pi to use the 2.8 LCD screen to display stats on it
 :::  -c, chronometer          Calculates stats and displays to an LCD
 :::  -h, help                 Show this help dialog
 :::  -v, version              Show current versions
 :::  -q, query                Query the adlists for a specific domain
+:::                           Use pihole -q domain -exact if you want to see exact matches only
+:::  -l, logging              Enable or Disable logging (pass 'on' or 'off')
+:::  -a, admin                Admin webpage options
 :::  uninstall                Uninstall Pi-Hole from your system :(!
 :::  status                   Is Pi-Hole Enabled or Disabled
 :::  enable                   Enable Pi-Hole DNS Blocking
 :::  disable                  Disable Pi-Hole DNS Blocking
+:::                           Blocking can also be disabled only temporarily, e.g.,
+:::                           pihole disable 5m - will disable blocking for 5 minutes
 :::  restartdns               Restart dnsmasq
 EOM
-	exit 1
+  exit 0
 }
 
 if [[ $# = 0 ]]; then
-	helpFunc
+  helpFunc
 fi
 
 # Handle redirecting to specific functions based on arguments
 case "${1}" in
-	"-w" | "whitelist"			) whitelistFunc "$@";;
-	"-b" | "blacklist"			) blacklistFunc "$@";;
-	"-d" | "debug"				) debugFunc;;
-	"-f" | "flush"				) flushFunc;;
-	"-up" | "updatePihole"      ) updatePiholeFunc;;
-	"-r"  | "reconfigure"       ) reconfigurePiholeFunc;;
-	"-g" | "updateGravity"		) updateGravityFunc "$@";;
-	"-s" | "setupLCD"			) setupLCDFunction;;
-	"-c" | "chronometer"		) chronometerFunc "$@";;
-	"-h" | "help"				) helpFunc;;
-	"-v" | "version"            ) versionFunc "$@";;
-	"-q" | "query"              ) queryFunc "$@";;
-	"uninstall"					) uninstallFunc;;
-	"enable"                      ) piholeEnable 1;;
-    "disable"                     ) piholeEnable 0;;
-    "status"                      ) piholeStatus "$2";;
-    "restartdns"                  ) restartDNS;;
-	*                    		) helpFunc;;
+  "-w" | "whitelist"            ) whitelistFunc "$@";;
+  "-b" | "blacklist"            ) blacklistFunc "$@";;
+  "-d" | "debug"                ) debugFunc;;
+  "-f" | "flush"                ) flushFunc;;
+  "-up" | "updatePihole"        ) updatePiholeFunc;;
+  "-r"  | "reconfigure"         ) reconfigurePiholeFunc;;
+  "-g" | "updateGravity"        ) updateGravityFunc "$@";;
+  "-s" | "setupLCD"             ) setupLCDFunction;;
+  "-c" | "chronometer"          ) chronometerFunc "$@";;
+  "-h" | "help"                 ) helpFunc;;
+  "-v" | "version"              ) versionFunc "$@";;
+  "-q" | "query"                ) queryFunc "$@";;
+  "-l" | "logging"              ) piholeLogging "$@";;
+  "uninstall"                   ) uninstallFunc;;
+  "enable"                      ) piholeEnable 1;;
+  "disable"                     ) piholeEnable 0 $2;;
+  "status"                      ) piholeStatus "$2";;
+  "restartdns"                  ) restartDNS;;
+  "-a" | "admin"                ) webpageFunc "$@";;
+  "-t" | "tail"                 ) tailFunc;;
+  *                             ) helpFunc;;
 esac

requirements.txt

@@ -0,0 +1,5 @@
+docker-compose
+pytest
+pytest-xdist
+pytest-cov
+testinfra

test/centos.Dockerfile

@@ -0,0 +1,16 @@
+FROM centos:7
+
+ENV GITDIR /etc/.pihole
+ENV SCRIPTDIR /opt/pihole
+
+RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole
+ADD . $GITDIR
+RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $SCRIPTDIR/
+ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR
+
+RUN true && \
+    chmod +x $SCRIPTDIR/*
+
+ENV PH_TEST true
+
+#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \

test/conftest.py

@@ -0,0 +1,61 @@
+import pytest
+import testinfra
+
+check_output = testinfra.get_backend(
+    "local://"
+).get_module("Command").check_output
+
+@pytest.fixture
+def Pihole(Docker):
+    ''' used to contain some script stubbing, now pretty much an alias.
+    Also provides bash as the default run function shell '''
+    def run_bash(self, command, *args, **kwargs):
+        cmd = self.get_command(command, *args)
+        if self.user is not None:
+            out = self.run_local(
+                "docker exec -u %s %s /bin/bash -c %s",
+                self.user, self.name, cmd)
+        else:
+            out = self.run_local(
+                "docker exec %s /bin/bash -c %s", self.name, cmd)
+        out.command = self.encode(cmd)
+        return out
+
+    funcType = type(Docker.run)
+    Docker.run = funcType(run_bash, Docker, testinfra.backend.docker.DockerBackend)
+    return Docker
+
+@pytest.fixture
+def Docker(request, args, image, cmd):
+    ''' combine our fixtures into a docker run command and setup finalizer to cleanup '''
+    assert 'docker' in check_output('id'), "Are you in the docker group?"
+    docker_run = "docker run {} {} {}".format(args, image, cmd)
+    docker_id = check_output(docker_run)
+
+    def teardown():
+        check_output("docker rm -f %s", docker_id)
+    request.addfinalizer(teardown)
+
+    docker_container = testinfra.get_backend("docker://" + docker_id)
+    docker_container.id = docker_id
+    return docker_container
+
+@pytest.fixture
+def args(request):
+    ''' -t became required when tput began being used '''
+    return '-t -d'
+
+@pytest.fixture(params=['debian', 'centos'])
+def tag(request):
+    ''' consumed by image to make the test matrix '''
+    return request.param
+
+@pytest.fixture()
+def image(request, tag):
+    ''' built by test_000_build_containers.py '''
+    return 'pytest_pihole:{}'.format(tag)
+
+@pytest.fixture()
+def cmd(request):
+    ''' default to doing nothing by tailing null, but don't exit '''
+    return 'tail -f /dev/null'

test/debian.Dockerfile

@@ -0,0 +1,16 @@
+FROM debian:jessie
+
+ENV GITDIR /etc/.pihole
+ENV SCRIPTDIR /opt/pihole
+
+RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole
+ADD . $GITDIR
+RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $SCRIPTDIR/
+ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR
+
+RUN true && \
+    chmod +x $SCRIPTDIR/*
+
+ENV PH_TEST true
+
+#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \

test/test_000_build_containers.py

@@ -0,0 +1,18 @@
+''' This file starts with 000 to make it run first '''
+import pytest
+import testinfra
+
+run_local = testinfra.get_backend(
+    "local://"
+).get_module("Command").run
+
+@pytest.mark.parametrize("image,tag", [
+    ( 'test/debian.Dockerfile', 'pytest_pihole:debian' ),
+    ( 'test/centos.Dockerfile', 'pytest_pihole:centos' ),
+])
+def test_build_pihole_image(image, tag):
+    build_cmd = run_local('docker build -f {} -t {} .'.format(image, tag))
+    if build_cmd.rc != 0:
+        print build_cmd.stdout
+        print build_cmd.stderr
+    assert build_cmd.rc == 0

test/test_automated_install.py

@@ -0,0 +1,102 @@
+import pytest
+from textwrap import dedent
+
+SETUPVARS = {
+    'PIHOLE_INTERFACE' : 'eth99',
+    'IPV4_ADDRESS' : '1.1.1.1',
+    'IPV6_ADDRESS' : 'FE80::240:D0FF:FE48:4672',
+    'PIHOLE_DNS_1' : '4.2.2.1',
+    'PIHOLE_DNS_2' : '4.2.2.2'
+}
+
+def test_setupVars_are_sourced_to_global_scope(Pihole):
+    ''' currently update_dialogs sources setupVars with a dot,
+    then various other functions use the variables.
+    This confirms the sourced variables are in scope between functions '''
+    setup_var_file = 'cat <<EOF> /etc/pihole/setupVars.conf\n'
+    for k,v in SETUPVARS.iteritems():
+        setup_var_file += "{}={}\n".format(k, v)
+    setup_var_file += "EOF\n"
+    Pihole.run(setup_var_file)
+
+    script = dedent('''\
+    set -e
+    printSetupVars() {
+        # Currently debug test function only
+        echo "Outputting sourced variables"
+        echo "PIHOLE_INTERFACE=${PIHOLE_INTERFACE}"
+        echo "IPV4_ADDRESS=${IPV4_ADDRESS}"
+        echo "IPV6_ADDRESS=${IPV6_ADDRESS}"
+        echo "PIHOLE_DNS_1=${PIHOLE_DNS_1}"
+        echo "PIHOLE_DNS_2=${PIHOLE_DNS_2}"
+    }
+    update_dialogs() {
+        . /etc/pihole/setupVars.conf
+    }
+    update_dialogs
+    printSetupVars
+    ''')
+
+    output = run_script(Pihole, script).stdout
+
+    for k,v in SETUPVARS.iteritems():
+        assert "{}={}".format(k, v) in output
+
+def test_setupVars_saved_to_file(Pihole):
+    ''' confirm saved settings are written to a file for future updates to re-use '''
+    set_setup_vars = '\n'  # dedent works better with this and padding matching script below
+    for k,v in SETUPVARS.iteritems():
+        set_setup_vars += "    {}={}\n".format(k, v)
+    Pihole.run(set_setup_vars).stdout
+
+    script = dedent('''\
+    set -e
+    echo start
+    TERM=xterm
+    source /opt/pihole/basic-install.sh
+    {}
+    finalExports
+    cat /etc/pihole/setupVars.conf
+    '''.format(set_setup_vars))
+
+    output = run_script(Pihole, script).stdout
+
+    for k,v in SETUPVARS.iteritems():
+        assert "{}={}".format(k, v) in output
+
+def test_configureFirewall_firewalld_no_errors(Pihole):
+    ''' confirms firewalld rules are applied when appopriate '''
+    mock_command('firewall-cmd', '0', Pihole)
+    configureFirewall = Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    configureFirewall
+    ''')
+    expected_stdout = '::: Configuring firewalld for httpd and dnsmasq.'
+    assert expected_stdout in configureFirewall.stdout
+    firewall_calls = Pihole.run('cat /var/log/firewall-cmd').stdout
+    assert 'firewall-cmd --state' in firewall_calls
+    assert 'firewall-cmd --permanent --add-port=80/tcp' in firewall_calls
+    assert 'firewall-cmd --permanent --add-port=53/tcp' in firewall_calls
+    assert 'firewall-cmd --permanent --add-port=53/udp' in firewall_calls
+    assert 'firewall-cmd --reload' in firewall_calls
+
+
+# Helper functions
+def mock_command(script, result, container):
+    ''' Allows for setup of commands we don't really want to have to run for real in unit tests '''
+    ''' TODO: support array of results that enable the results to change over multiple executions of a command '''
+    full_script_path = '/usr/local/bin/{}'.format(script)
+    mock_script = dedent('''\
+    #!/bin/bash -e
+    echo "\$0 \$@" >> /var/log/{script}
+    exit {retcode}
+    '''.format(script=script, retcode=result))
+    container.run('''
+    cat <<EOF> {script}\n{content}\nEOF
+    chmod +x {script}
+    '''.format(script=full_script_path, content=mock_script))
+
+def run_script(Pihole, script):
+    result = Pihole.run(script)
+    assert result.rc == 0
+    return result

test/test_shellcheck.py

@@ -0,0 +1,13 @@
+import pytest
+import testinfra
+
+run_local = testinfra.get_backend(
+    "local://"
+).get_module("Command").run
+
+def test_scripts_pass_shellcheck():
+    ''' Make sure shellcheck does not find anything wrong with our shell scripts '''
+    shellcheck = "find . -type f \( -name 'update.sh' -o -name 'piholeDebug.sh' \) | while read file; do shellcheck \"$file\"; done;"
+    results = run_local(shellcheck)
+    print results.stdout
+    assert '' == results.stdout