Spellcheck improvements (no warnings in pihole and advanced/Scripts/webpage.sh remains)

Signed-off-by: DL6ER <dl6er@dl6er.de>
Move ensure_newline() into its own file as we need to source it from different places
2021-01-05 09:58:22 +01:00 · 2020-12-30 21:48:07 +01:00 · 2020-12-30 10:50:31 +01:00
26 changed files with 414 additions and 670 deletions
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -10,7 +10,7 @@ jobs:
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        distro: [debian_9, debian_10, ubuntu_16, ubuntu_18, ubuntu_20, centos_7, centos_8, fedora_32, fedora_33]
+        distro: [debian_9, debian_10, ubuntu_16, ubuntu_18, ubuntu_20, centos_7, centos_8, fedora_31, fedora_32]
    env:
      DISTRO: ${{matrix.distro}}
    steps:
--- a/README.md
+++ b/README.md
@@ -1,7 +1,5 @@
 <!-- markdownlint-configure-file { "MD004": { "style": "consistent" } } -->
 <!-- markdownlint-disable MD033 -->
-#
-
 <p align="center">
    <a href="https://pi-hole.net/">
        <img src="https://pi-hole.github.io/graphics/Vortex/Vortex_with_Wordmark.svg" width="150" height="260" alt="Pi-hole">
@@ -11,6 +9,8 @@
 </p>
 <!-- markdownlint-enable MD033 -->

+#
+
 The Pi-hole® is a [DNS sinkhole](https://en.wikipedia.org/wiki/DNS_Sinkhole) that protects your devices from unwanted content, without installing any client-side software.

 - **Easy-to-install**: our versatile installer walks you through the process, and takes less than ten minutes
@@ -26,6 +26,8 @@ The Pi-hole® is a [DNS sinkhole](https://en.wikipedia.org/wiki/DNS_Sinkhole) th

 -----

+Master [![Build Status](https://travis-ci.com/pi-hole/pi-hole.svg?branch=master)](https://travis-ci.com/pi-hole/pi-hole) Development [![Build Status](https://travis-ci.com/pi-hole/pi-hole.svg?branch=development)](https://travis-ci.com/pi-hole/pi-hole)
+
 ## One-Step Automated Install

 Those who want to get started quickly and conveniently may install Pi-hole using the following command:
@@ -50,8 +52,6 @@ sudo bash basic-install.sh
 wget -O basic-install.sh https://install.pi-hole.net
 sudo bash basic-install.sh
 ```
-### Method 3: Using Docker to deploy Pi-hole
-Please refer to the [Pi-hole docker repo](https://github.com/pi-hole/docker-pi-hole) to use the Official Docker Images.

 ## [Post-install: Make your network take advantage of Pi-hole](https://docs.pi-hole.net/main/post-install/)

@@ -71,18 +71,16 @@ Make no mistake: **your support is absolutely vital to help keep us innovating!*

 ### [Donations](https://pi-hole.net/donate)

-Sending a donation using our Sponsor Button is **extremely helpful** in offsetting a portion of our monthly expenses and rewarding our dedicated development team:
+Sending a donation using our Sponsor Button is **extremely helpful** in offsetting a portion of our monthly expenses:

 ### Alternative support

 If you'd rather not donate (_which is okay!_), there are other ways you can help support us:

- [GitHub Sponsors](https://github.com/sponsors/pi-hole/)
- [Patreon](https://patreon.com/pihole)
- [Hetzner Cloud](https://hetzner.cloud/?ref=7aceisRX3AzA) _affiliate link_
+- [Patreon](https://patreon.com/pihole) _Become a patron for rewards_
 - [Digital Ocean](https://www.digitalocean.com/?refcode=344d234950e1) _affiliate link_
 - [Stickermule](https://www.stickermule.com/unlock?ref_id=9127301701&utm_medium=link&utm_source=invite) _earn a $10 credit after your first purchase_
- [Amazon US](http://www.amazon.com/exec/obidos/redirect-home/pihole09-20) _affiliate link_
+- [Amazon](http://www.amazon.com/exec/obidos/redirect-home/pihole09-20) _affiliate link_
 - Spreading the word about our software, and how you have benefited from it

 ### Contributing via GitHub
@@ -97,9 +95,7 @@ You'll find that the [install script](https://github.com/pi-hole/pi-hole/blob/ma

 ## Getting in touch with us

-While we are primarily reachable on our [Discourse User Forum](https://discourse.pi-hole.net/), we can also be found on a variety of social media outlets.
-
-**Please be sure to check the FAQ's** before starting a new discussion. Many user questions already have answers and can be solved without any additional assistance.
+While we are primarily reachable on our [Discourse User Forum](https://discourse.pi-hole.net/), we can also be found on a variety of social media outlets. **Please be sure to check the FAQ's** before starting a new discussion, as we do not have the spare time to reply to every request for assistance.

 - [Frequently Asked Questions](https://discourse.pi-hole.net/c/faqs)
 - [Feature Requests](https://discourse.pi-hole.net/c/feature-requests?order=votes)
@@ -110,27 +106,12 @@ While we are primarily reachable on our [Discourse User Forum](https://discourse

 ## Breakdown of Features

-### [Faster-than-light Engine](https://github.com/pi-hole/ftl)
-
-[FTLDNS](https://github.com/pi-hole/ftl) is a lightweight, purpose-built daemon used to provide statistics needed for the Web Interface, and its API can be easily integrated into your own projects. As the name implies, FTLDNS does this all *very quickly*!
-
-Some of the statistics you can integrate include:
-
- Total number of domains being blocked
- Total number of DNS queries today
- Total number of ads blocked today
- Percentage of ads blocked
- Unique domains
- Queries forwarded (to your chosen upstream DNS server)
- Queries cached
- Unique clients
-
-The API can be accessed via [`telnet`](https://github.com/pi-hole/FTL), the Web (`admin/api.php`) and Command Line (`pihole -c -j`). You can find out [more details over here](https://discourse.pi-hole.net/t/pi-hole-api/1863).
-
 ### The Command Line Interface

 The [pihole](https://docs.pi-hole.net/core/pihole-command/) command has all the functionality necessary to be able to fully administer the Pi-hole, without the need of the Web Interface. It's fast, user-friendly, and auditable by anyone with an understanding of `bash`.

+![Pi-hole Blacklist Demo](https://pi-hole.github.io/graphics/Screenshots/blacklist-cli.gif)
+
 Some notable features include:

 - [Whitelisting, Blacklisting and Regex](https://docs.pi-hole.net/core/pihole-command/#whitelisting-blacklisting-and-regex)
@@ -147,6 +128,8 @@ You can read our [Core Feature Breakdown](https://docs.pi-hole.net/core/pihole-c

 This [optional dashboard](https://github.com/pi-hole/AdminLTE) allows you to view stats, change settings, and configure your Pi-hole. It's the power of the Command Line Interface, with none of the learning curve!

+![Pi-hole Dashboard](https://pi-hole.github.io/graphics/Screenshots/pihole-dashboard.png)
+
 Some notable features include:

 - Mobile friendly interface
@@ -162,3 +145,21 @@ There are several ways to [access the dashboard](https://discourse.pi-hole.net/t

 1. `http://pi.hole/admin/` (when using Pi-hole as your DNS server)
 2. `http://<IP_ADDPRESS_OF_YOUR_PI_HOLE>/admin/`
+3. `http://pi.hole/` (when using Pi-hole as your DNS server)
+
+## Faster-than-light Engine
+
+FTLDNS is a lightweight, purpose-built daemon used to provide statistics needed for the Web Interface, and its API can be easily integrated into your own projects. As the name implies, FTLDNS does this all *very quickly*!
+
+Some of the statistics you can integrate include:
+
+- Total number of domains being blocked
+- Total number of DNS queries today
+- Total number of ads blocked today
+- Percentage of ads blocked
+- Unique domains
+- Queries forwarded (to your chosen upstream DNS server)
+- Queries cached
+- Unique clients
+
+The API can be accessed via [`telnet`](https://github.com/pi-hole/FTL), the Web (`admin/api.php`) and Command Line (`pihole -c -j`). You can find out [more details over here](https://discourse.pi-hole.net/t/pi-hole-api/1863).
--- a/advanced/Scripts/chronometer.sh
+++ b/advanced/Scripts/chronometer.sh
@@ -13,7 +13,6 @@ LC_NUMERIC=C

 # Retrieve stats from FTL engine
 pihole-FTL() {
-    local ftl_port LINE
    ftl_port=$(cat /run/pihole-FTL.port 2> /dev/null)
    if [[ -n "$ftl_port" ]]; then
        # Open connection to FTL
@@ -21,13 +20,12 @@ pihole-FTL() {

        # Test if connection is open
        if { "true" >&3; } 2> /dev/null; then
-            # Send command to FTL and ask to quit when finished
-            echo -e ">$1 >quit" >&3
+            # Send command to FTL
+            echo -e ">$1" >&3

-            # Read input until we received an empty string and the connection is
-            # closed
+            # Read input
            read -r -t 1 LINE <&3
-            until [[ -z "${LINE}" ]] && [[ ! -t 3 ]]; do
+            until [[ ! $? ]] || [[ "$LINE" == *"EOM"* ]]; do
                echo "$LINE" >&1
                read -r -t 1 LINE <&3
            done
@@ -559,7 +557,7 @@ Calculates stats and displays to an LCD
 Options:
  -j, --json          Output stats as JSON formatted string
  -r, --refresh       Set update frequency (in seconds)
-  -e, --exit          Output stats and exit without refreshing
+  -e, --exit          Output stats and exit witout refreshing
  -h, --help          Display this help text"
  fi

--- a/advanced/Scripts/database_migration/gravity-db.sh
+++ b/advanced/Scripts/database_migration/gravity-db.sh
@@ -111,15 +111,9 @@ upgrade_gravityDB(){
 		version=12
 	fi
 	if [[ "$version" == "12" ]]; then
-		# Add column date_updated to adlist table
+		# Add column date_updated to alist table
 		echo -e "  ${INFO} Upgrading gravity database from version 12 to 13"
 		sqlite3 "${database}" < "${scriptPath}/12_to_13.sql"
 		version=13
 	fi
-	if [[ "$version" == "13" ]]; then
-		# Add columns number and status to adlist table
-		echo -e "  ${INFO} Upgrading gravity database from version 13 to 14"
-		sqlite3 "${database}" < "${scriptPath}/13_to_14.sql"
-		version=14
-	fi
 }
--- a/advanced/Scripts/database_migration/gravity/13_to_14.sql
+++ b/advanced/Scripts/database_migration/gravity/13_to_14.sql
@@ -1,13 +0,0 @@
-.timeout 30000
-
-PRAGMA FOREIGN_KEYS=OFF;
-
-BEGIN TRANSACTION;
-
-ALTER TABLE adlist ADD COLUMN number INTEGER NOT NULL DEFAULT 0;
-ALTER TABLE adlist ADD COLUMN invalid_domains INTEGER NOT NULL DEFAULT 0;
-ALTER TABLE adlist ADD COLUMN status INTEGER NOT NULL DEFAULT 0;
-
-UPDATE info SET value = 14 WHERE property = 'version';
-
-COMMIT;
--- a/advanced/Scripts/list.sh
+++ b/advanced/Scripts/list.sh
@@ -1,6 +1,4 @@
 #!/usr/bin/env bash
-# shellcheck disable=SC1090
-
 # Pi-hole: A black hole for Internet advertisements
 # (c) 2017 Pi-hole, LLC (https://pi-hole.net)
 # Network-wide ad blocking via your own hardware.
@@ -11,17 +9,9 @@
 # Please see LICENSE file for your rights under this license.

 # Globals
-piholeDir="/etc/pihole"
-GRAVITYDB="${piholeDir}/gravity.db"
-# Source pihole-FTL from install script
-pihole_FTL="${piholeDir}/pihole-FTL.conf"
-if [[ -f "${pihole_FTL}" ]]; then
-  source "${pihole_FTL}"
-fi
-
-# Set this only after sourcing pihole-FTL.conf as the gravity database path may
-# have changed
-gravityDBfile="${GRAVITYDB}"
+basename=pihole
+piholeDir=/etc/"${basename}"
+gravityDBfile="${piholeDir}/gravity.db"

 reload=false
 addmode=true
@@ -122,7 +112,7 @@ ProcessDomainList() {
    for dom in "${domList[@]}"; do
        # Format domain into regex filter if requested
        if [[ "${wildcard}" == true ]]; then
-            dom="(\\.|^)${dom//\./\\.}$"
+            dom="(^|\\.)${dom//\./\\.}$"
        fi

        # Logic: If addmode then add to desired list and remove from the other;
--- a/advanced/Scripts/piholeARPTable.sh
+++ b/advanced/Scripts/piholeARPTable.sh
@@ -38,7 +38,7 @@ flushARP(){

    # Truncate network_addresses table in pihole-FTL.db
    # This needs to be done before we can truncate the network table due to
-    # foreign key constraints
+    # foreign key contraints
    if ! output=$(sqlite3 "${DBFILE}" "DELETE FROM network_addresses" 2>&1); then
        echo -e "${OVER}  ${CROSS} Failed to truncate network_addresses table"
        echo "  Database location: ${DBFILE}"
--- a/advanced/Scripts/piholeDebug.sh
+++ b/advanced/Scripts/piholeDebug.sh
@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 # Pi-hole: A black hole for Internet advertisements
-# (c) 2021 Pi-hole (https://pi-hole.net)
+# (c) 2017 Pi-hole, LLC (https://pi-hole.net)
 # Network-wide ad blocking via your own hardware.
 #
 # Generates pihole_debug.log to be used for troubleshooting.
@@ -198,33 +198,6 @@ show_disclaimer(){
    log_write "${DISCLAIMER}"
 }

-check_for_ftl(){
-    echo_current_diagnostic "Checking for pihole-FTL binary"
-
-    declare -g FTL_PATH
-    read -r FTL_PATH < <(which pihole-FTL)
-    if [ -z "${FTL_PATH}" ]; then
-        log_write "${CROSS} ${COL_RED} Unable to find pihole-FTL binary.${COL_NC}"
-        # Non-zero return value
-        return 2
-    else
-        log_write "${TICK} pihole-FTL: ${COL_GREEN}${FTL_PATH}${COL_NC}"
-    fi
-
-}
-
-check_for_lua(){
-    echo_current_diagnostic "Checking for lua capabilities"
-
-    if ! (${FTL_PATH} lua -v &>/dev/null); then
-        log_write "${CROSS} ${COL_RED} pihole-FTL binary does not have lua capabilites.${COL_NC}"
-        # Non-zero return value
-        return
-    else
-        log_write "${TICK} pihole-FTL: ${COL_GREEN}lua found!${COL_NC}"
-    fi
-}
-
 source_setup_variables() {
    # Display the current test that is running
    log_write "\\n${COL_PURPLE}*** [ INITIALIZING ]${COL_NC} Sourcing setup variables"
@@ -271,7 +244,7 @@ initialize_debug() {
    log_write "${INFO} $(date "+%Y-%m-%d:%H:%M:%S") debug log has been initialized."
 }

-# This is a function for visually displaying the current test that is being run.
+# This is a function for visually displaying the curent test that is being run.
 # Accepts one variable: the name of what is being diagnosed
 # Colors do not show in the dasboard, but the icons do: [i], [✓], and [✗]
 echo_current_diagnostic() {
@@ -406,7 +379,7 @@ get_program_version() {
    # Create a local variable so this function can be safely reused
    local program_version
    echo_current_diagnostic "${program_name} version"
-    # Evaluate the program we are checking, if it is any of the ones below, show the version
+    # Evalutate the program we are checking, if it is any of the ones below, show the version
    case "${program_name}" in
        "lighttpd") program_version="$(${program_name} -v 2> /dev/null | head -n1 | cut -d '/' -f2 | cut -d ' ' -f1)"
                    ;;
@@ -668,7 +641,7 @@ detect_ip_addresses() {
    # First argument should be a 4 or a 6
    local protocol=${1}
    # Use ip to show the addresses for the chosen protocol
-    # Store the values in an array so they can be looped through
+    # Store the values in an arry so they can be looped through
    # Get the lines that are in the file(s) and store them in an array for parsing later
    mapfile -t ip_addr_list < <(ip -"${protocol}" addr show dev "${PIHOLE_INTERFACE}" | awk -F ' ' '{ for(i=1;i<=NF;i++) if ($i ~ '/^inet/') print $(i+1) }')

@@ -722,7 +695,7 @@ ping_gateway() {
    # Check if we are using IPv4 or IPv6
    # Find the default gateway using IPv4 or IPv6
    local gateway
-    gateway="$(ip -"${protocol}" route | grep default | grep "${PIHOLE_INTERFACE}" | cut -d ' ' -f 3)"
+    gateway="$(ip -"${protocol}" route | grep default | cut -d ' ' -f 3)"

    # If the gateway variable has a value (meaning a gateway was found),
    if [[ -n "${gateway}" ]]; then
@@ -850,7 +823,7 @@ check_x_headers() {
    # Do it for the dashboard as well, as the header is different than above
    local dashboard
    dashboard=$(curl -Is localhost/admin/ | awk '/X-Pi-hole/' | tr -d '\r')
-    # Store what the X-Header should be in variables for comparison later
+    # Store what the X-Header shoud be in variables for comparison later
    local block_page_working
    block_page_working="X-Pi-hole: A black hole for Internet advertisements."
    local dashboard_working
@@ -869,12 +842,12 @@ check_x_headers() {
        log_write "${COL_RED}${full_curl_output_block_page}${COL_NC}"
    fi

-    # Same logic applies to the dashboard as above, if the X-Header matches what a working system should have,
+    # Same logic applies to the dashbord as above, if the X-Header matches what a working system shoud have,
    if [[ $dashboard == "$dashboard_working" ]]; then
        # then we can show a success
        log_write "$TICK Web interface X-Header: ${COL_GREEN}${dashboard}${COL_NC}"
    else
-        # Otherwise, it's a failure since the X-Headers either don't exist or have been modified in some way
+        # Othewise, it's a failure since the X-Headers either don't exist or have been modified in some way
        log_write "$CROSS Web interface X-Header: ${COL_RED}X-Header does not match or could not be retrieved.${COL_NC}"
        log_write "${COL_RED}${full_curl_output_dashboard}${COL_NC}"
    fi
@@ -904,7 +877,7 @@ dig_at() {
        local pihole_address="${IP}"
        local remote_address="2001:4860:4860::8888"
        local record_type="AAAA"
-    # Otherwise, it should be 4
+    # Othwerwise, it should be 4
    else
        # so use the IPv4 values
        local local_address="127.0.0.1"
@@ -938,7 +911,7 @@ dig_at() {
        # show a success
        log_write "${TICK} ${random_url} ${COL_GREEN}is ${pihole_dig}${COL_NC} via ${COL_CYAN}Pi-hole${COL_NC} (${pihole_address})"
    else
-        # Otherwise, show a failure
+        # Othewise, show a failure
        log_write "${CROSS} ${COL_RED}Failed to resolve${COL_NC} ${random_url} via ${COL_RED}Pi-hole${COL_NC} (${pihole_address})"
    fi

@@ -1071,7 +1044,7 @@ parse_file() {
 }

 check_name_resolution() {
-    # Check name resolution from localhost, Pi-hole's IP, and Google's name severs
+    # Check name resoltion from localhost, Pi-hole's IP, and Google's name severs
    # using the function we created earlier
    dig_at 4 "${IPV4_ADDRESS%/*}"
    # If IPv6 enabled,
@@ -1448,40 +1421,28 @@ make_temporary_log
 initialize_debug
 # setupVars.conf needs to be sourced before the networking so the values are
 # available to the other functions
-if ! check_for_ftl; then
-    log_write "${COL_RED}Unable to complete debug run. Please contact support for assistance."
-    log_write "Please note the error that is displayed above.${COL_NC}"
-    #Non-zero return value
-    exit 2
-fi
-if ! check_for_lua; then
-    log_write "${COL_RED}Unable to complete debug run. Please contact support for assistance."
-    log_write "Please note the error that is displayed above.${COL_NC}"
-    #Non-zero return value
-    exit 2
-fi
-# source_setup_variables
-# check_component_versions
-# check_critical_program_versions
-# diagnose_operating_system
-# check_selinux
-# check_firewalld
-# processor_check
-# check_networking
-# check_name_resolution
-# check_dhcp_servers
-# process_status
-# ftl_full_status
-# parse_setup_vars
-# check_x_headers
-# analyze_gravity_list
-# show_groups
-# show_domainlist
-# show_clients
-# show_adlists
-# show_content_of_pihole_files
-# show_messages
-# parse_locale
-# analyze_pihole_log
-# copy_to_debug_log
-# upload_to_tricorder
+source_setup_variables
+check_component_versions
+check_critical_program_versions
+diagnose_operating_system
+check_selinux
+check_firewalld
+processor_check
+check_networking
+check_name_resolution
+check_dhcp_servers
+process_status
+ftl_full_status
+parse_setup_vars
+check_x_headers
+analyze_gravity_list
+show_groups
+show_domainlist
+show_clients
+show_adlists
+show_content_of_pihole_files
+show_messages
+parse_locale
+analyze_pihole_log
+copy_to_debug_log
+upload_to_tricorder
--- a/advanced/Scripts/query.sh
+++ b/advanced/Scripts/query.sh
@@ -1,6 +1,5 @@
 #!/usr/bin/env bash
 # shellcheck disable=SC1090
-
 # Pi-hole: A black hole for Internet advertisements
 # (c) 2018 Pi-hole, LLC (https://pi-hole.net)
 # Network-wide ad blocking via your own hardware.
@@ -12,21 +11,12 @@

 # Globals
 piholeDir="/etc/pihole"
-GRAVITYDB="${piholeDir}/gravity.db"
+gravityDBfile="${piholeDir}/gravity.db"
 options="$*"
 all=""
 exact=""
 blockpage=""
 matchType="match"
-# Source pihole-FTL from install script
-pihole_FTL="${piholeDir}/pihole-FTL.conf"
-if [[ -f "${pihole_FTL}" ]]; then
-  source "${pihole_FTL}"
-fi
-
-# Set this only after sourcing pihole-FTL.conf as the gravity database path may
-# have changed
-gravityDBfile="${GRAVITYDB}"

 colfile="/opt/pihole/COL_TABLE"
 source "${colfile}"
--- a/advanced/Scripts/utils.sh
+++ b/advanced/Scripts/utils.sh
@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+
+# Pi-hole: A black hole for Internet advertisements
+# (c) 2020 Pi-hole, LLC (https://pi-hole.net)
+# Network-wide ad blocking via your own hardware.
+#
+# Controller for all pihole scripts and functions.
+#
+# This file is copyright under the latest version of the EUPL.
+# Please see LICENSE file for your rights under this license.
+
+# Ensure there is a newline at the end of the file passed as argument
+ensure_newline() {
+  # Check if the last line of the passed file is empty, if not, append a newline
+  # to the file to ensure we can append new content safely using echo "" >>
+  # later on
+  [ -n "$(tail -c1 "${1}")" ] && printf '\n' >> "${1}"
+  # There was also the suggestion of using a sed-magic call here, however, this
+  # had the drawback to updating all the file timestamps whenever the sed was
+  # run. This solution only updates the timestamp when actually appending a
+  # newline
+}
--- a/advanced/Scripts/version.sh
+++ b/advanced/Scripts/version.sh
@@ -153,7 +153,7 @@ versionOutput() {
    if [[ -n "$current" ]] && [[ -n "$latest" ]]; then
        output="${1^} version is $branch$current (Latest: $latest)"
    elif [[ -n "$current" ]] && [[ -z "$latest" ]]; then
-        output="Current ${1^} version is $branch$current"
+        output="Current ${1^} version is $branch$current."
    elif [[ -z "$current" ]] && [[ -n "$latest" ]]; then
        output="Latest ${1^} version is $latest"
    elif [[ "$curHash" == "N/A" ]] || [[ "$latHash" == "N/A" ]]; then
--- a/advanced/Scripts/webpage.sh
+++ b/advanced/Scripts/webpage.sh
@@ -22,13 +22,21 @@ readonly gravityDBfile="/etc/pihole/gravity.db"

 # Source install script for ${setupVars}, ${PI_HOLE_BIN_DIR} and valid_ip()
 readonly PI_HOLE_FILES_DIR="/etc/.pihole"
+setupVars=""
+DHCP_IPv6=false
 # shellcheck disable=SC2034  # used in basic-install
 PH_TEST="true"
 source "${PI_HOLE_FILES_DIR}/automated install/basic-install.sh"

-coltable="/opt/pihole/COL_TABLE"
+# ensure_newline()
+readonly utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
+# shellcheck source=./utils.sh
+source "${utilsfile}"
+
+readonly coltable="${PI_HOLE_SCRIPT_DIR}/COL_TABLE"
 if [[ -f ${coltable} ]]; then
-    source ${coltable}
+  # shellcheck source=./COL_TABLE
+  source "${coltable}"
 fi

 helpFunc() {
@@ -49,6 +57,7 @@ Options:
 }

 add_setting() {
+    ensure_newline "${setupVars}"
    echo "${1}=${2}" >> "${setupVars}"
 }

@@ -62,6 +71,7 @@ change_setting() {
 }

 addFTLsetting() {
+    ensure_newline "${FTLconf}"
    echo "${1}=${2}" >> "${FTLconf}"
 }

@@ -75,6 +85,7 @@ changeFTLsetting() {
 }

 add_dnsmasq_setting() {
+    ensure_newline "${dnsmasqconfig}"
    if [[ "${2}" != "" ]]; then
        echo "${1}=${2}" >> "${dnsmasqconfig}"
    else
@@ -146,6 +157,7 @@ SetWebPassword() {
 ProcessDNSSettings() {
    source "${setupVars}"

+    ensure_newline "${dnsmasqconfig}"
    delete_dnsmasq_setting "server"

    COUNTER=1
@@ -238,18 +250,18 @@ trust-anchor=.,20326,8,2,E06D44B80B8F1D39A95C0B0D7C65D08458E880409BBC68345710423
        #          168.192.in-addr.arpa to 192.168.0.0/16
        #          192.in-addr.arpa to 192.0.0.0/8
        if [[ "${CONDITIONAL_FORWARDING_REVERSE}" == *"in-addr.arpa" ]];then
-            arrRev=("${CONDITIONAL_FORWARDING_REVERSE//./ }")
-            case ${#arrRev[@]} in
+            arrRev=("${CONDITIONAL_FORWARDING_REVERSE//./ }")        
+            case ${#arrRev[@]} in 
                6   )   REV_SERVER_CIDR="${arrRev[3]}.${arrRev[2]}.${arrRev[1]}.${arrRev[0]}/32";;
                5   )   REV_SERVER_CIDR="${arrRev[2]}.${arrRev[1]}.${arrRev[0]}.0/24";;
                4   )   REV_SERVER_CIDR="${arrRev[1]}.${arrRev[0]}.0.0/16";;
-                3   )   REV_SERVER_CIDR="${arrRev[0]}.0.0.0/8";;
+                3   )   REV_SERVER_CIDR="${arrRev[0]}.0.0.0/8";; 
            esac
        else
          # Set REV_SERVER_CIDR to whatever value it was set to
          REV_SERVER_CIDR="${CONDITIONAL_FORWARDING_REVERSE}"
        fi
-
+        
        # If REV_SERVER_CIDR is not converted by the above, then use the REV_SERVER_TARGET variable to derive it
        if [ -z "${REV_SERVER_CIDR}" ]; then
            # Convert existing input to /24 subnet (preserves legacy behavior)
@@ -392,6 +404,7 @@ ProcessDHCPSettings() {
    fi

    # Write settings to file
+    # We do not need to ensure a newline here as the entire file is re-written
    echo "###############################################################################
 #  DHCP SERVER CONFIG FILE AUTOMATICALLY POPULATED BY PI-HOLE WEB INTERFACE.  #
 #            ANY CHANGES MADE TO THIS FILE WILL BE LOST ON CHANGE             #
@@ -486,15 +499,10 @@ SetWebUITheme() {
 }

 CheckUrl(){
-    local regex check_url
+    local regex
    # Check for characters NOT allowed in URLs
-    regex="[^a-zA-Z0-9:/?&%=~._()-;]"
-
-    # this will remove first @ that is after schema and before domain
-    # \1 is optional schema, \2 is userinfo
-    check_url="$( sed -re 's#([^:/]*://)?([^/]+)@#\1\2#' <<< "$1" )"
-
-    if [[ "${check_url}" =~ ${regex} ]]; then
+    regex="[^a-zA-Z0-9:/?&%=~._-]"
+    if [[ "${1}" =~ ${regex} ]]; then
        return 1
    else
        return 0
@@ -550,6 +558,7 @@ AddDHCPStaticAddress() {
    ip="${args[3]}"
    host="${args[4]}"

+    ensure_newline "${dhcpstaticconfig}"
    if [[ "${ip}" == "noip" ]]; then
        # Static host name
        echo "dhcp-host=${mac},${host}" >> "${dhcpstaticconfig}"
@@ -564,13 +573,7 @@ AddDHCPStaticAddress() {

 RemoveDHCPStaticAddress() {
    mac="${args[2]}"
-    if [[ "$mac" =~ ^([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})$ ]]; then
-        sed -i "/dhcp-host=${mac}.*/d" "${dhcpstaticconfig}"
-    else
-        echo "  ${CROSS} Invalid Mac Passed!"
-        exit 1
-    fi
-
+    sed -i "/dhcp-host=${mac}.*/d" "${dhcpstaticconfig}"
 }

 SetAdminEmail() {
@@ -642,11 +645,8 @@ Interfaces:

 Teleporter() {
    local datetimestamp
-    local host
    datetimestamp=$(date "+%Y-%m-%d_%H-%M-%S")
-    host=$(hostname)
-    host="${host//./_}"
-    php /var/www/html/admin/scripts/pi-hole/php/teleporter.php > "pi-hole-${host:-noname}-teleporter_${datetimestamp}.tar.gz"
+    php /var/www/html/admin/scripts/pi-hole/php/teleporter.php > "pi-hole-teleporter_${datetimestamp}.tar.gz"
 }

 checkDomain()
@@ -703,6 +703,7 @@ AddCustomDNSAddress() {

    ip="${args[2]}"
    host="${args[3]}"
+    ensure_newline "${dnscustomfile}"
 	echo "${ip} ${host}" >> "${dnscustomfile}"

    # Restart dnsmasq to load new custom DNS entries
@@ -714,13 +715,7 @@ RemoveCustomDNSAddress() {

    ip="${args[2]}"
    host="${args[3]}"
-
-    if valid_ip "${ip}" || valid_ip6 "${ip}" ; then
-        sed -i "/${ip} ${host}/d" "${dnscustomfile}"
-    else
-        echo -e "  ${CROSS} Invalid IP has been passed"
-        exit 1
-    fi
+    sed -i "/${ip} ${host}/d" "${dnscustomfile}"

    # Restart dnsmasq to update removed custom DNS entries
    RestartDNS
@@ -731,7 +726,7 @@ AddCustomCNAMERecord() {

    domain="${args[2]}"
    target="${args[3]}"
-
+    ensure_newline "${dnscustomcnamefile}"
    echo "cname=${domain},${target}" >> "${dnscustomcnamefile}"

    # Restart dnsmasq to load new custom CNAME records
@@ -743,20 +738,7 @@ RemoveCustomCNAMERecord() {

    domain="${args[2]}"
    target="${args[3]}"
-
-    validDomain="$(checkDomain "${domain}")"
-    if [[ -n "${validDomain}" ]]; then
-        validTarget="$(checkDomain "${target}")"
-        if [[ -n "${validDomain}" ]]; then
-            sed -i "/cname=${validDomain},${validTarget}/d" "${dnscustomcnamefile}"
-        else
-            echo "  ${CROSS} Invalid Target Passed!"
-            exit 1
-        fi
-    else
-        echo "  ${CROSS} Invalid Domain passed!"
-        exit 1
-    fi
+    sed -i "/cname=${domain},${target}/d" "${dnscustomcnamefile}"

    # Restart dnsmasq to update removed custom CNAME records
    RestartDNS
--- a/advanced/Templates/gravity.db.sql
+++ b/advanced/Templates/gravity.db.sql
@@ -32,10 +32,7 @@ CREATE TABLE adlist
 	date_added INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
 	date_modified INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)),
 	comment TEXT,
-	date_updated INTEGER,
-	number INTEGER NOT NULL DEFAULT 0,
-	invalid_domains INTEGER NOT NULL DEFAULT 0,
-	status INTEGER NOT NULL DEFAULT 0
+	date_updated INTEGER
 );

 CREATE TABLE adlist_by_group
@@ -57,7 +54,7 @@ CREATE TABLE info
 	value TEXT NOT NULL
 );

-INSERT INTO "info" VALUES('version','14');
+INSERT INTO "info" VALUES('version','13');

 CREATE TABLE domain_audit
 (
--- a/advanced/blockingpage.css
+++ b/advanced/blockingpage.css
@@ -145,17 +145,7 @@ body {
 }

 /* User is greeted with a splash page when browsing to Pi-hole IP address */
-#splashpage {
-  background: #222;
-  color: rgba(255, 255, 255, 0.7);
-  text-align: center;
-  width: 100%;
-  height: 100%;
-  display: flex;
-  align-items: center;
-  justify-content: center;
-}
-
+#splashpage { background: #222; color: rgba(255, 255, 255, 0.7); text-align: center; }
 #splashpage img { margin: 5px; width: 256px; }
 #splashpage b { color: inherit; }

@@ -206,26 +196,6 @@ header #bpAlt label {
  display: block;
 }

-html, body {
-    height: 100%;
-}
-
-#pihole_card {
-    width: 400px;
-    height: auto;
-    max-width: 400px;
-}
-
-    #pihole_card p, #pihole_card a {
-        font-size: 13pt;
-        text-align: center;
-    }
-
-#pihole_logo_splash {
-    height: auto;
-    width: 100%;
-}
-
 /* Click anywhere else on screen to hide #bpAbout */
 #bpAboutToggle:checked {
  display: block;
@@ -412,44 +382,12 @@ footer {

 /* Responsive Content */
@media only screen and (max-width: 500px) {
-  h1 a {
-        font-size: 1.8rem;
-        min-width: 170px;
-    }
-
-    footer span::before {
-        content: "Generated ";
-    }
-
-    footer span {
-        display: block;
-    }
+  h1 a { font-size: 1.8rem; min-width: 170px; }
+  footer span::before { content: "Generated "; }
+  footer span { display: block; }
 }

@media only screen and (min-width: 1251px) {
-  #bpWrapper, footer {
-        border-radius: 0 0 5px 5px;
-    }
-
-    #bpAbout {
-        border-right-width: 1px;
-    }
-}
-
-@media only screen and (max-width: 400px) {
-    #pihole_card {
-        width: 100%;
-        height: auto;
-    }
-
-        #pihole_card p, #pihole_card a {
-            font-size: 100%;
-        }
-}
-
-@media only screen and (max-width: 256px) {
-    #pihole_logo_splash {
-        width: 90% !important;
-        height: auto;
-    }
+  #bpWrapper, footer { border-radius: 0 0 5px 5px; }
+  #bpAbout { border-right-width: 1px; }
 }
--- a/advanced/index.php
+++ b/advanced/index.php
@@ -58,15 +58,14 @@ if ($serverName === "pi.hole"
    // When directly browsing via IP or authorized hostname
    // Render splash/landing page based off presence of $landPage file
    // Unset variables so as to not be included in $landPage or $splashPage
-    unset($svPasswd, $svEmail, $authorizedHosts, $validExtTypes, $currentUrlExt);
+    unset($serverName, $svPasswd, $svEmail, $authorizedHosts, $validExtTypes, $currentUrlExt, $viewPort);
    // If $landPage file is present
    if (is_file(getcwd()."/$landPage")) {
-        unset($serverName, $viewPort); // unset extra variables not to be included in $landpage
        include $landPage;
        exit();
    }
    // If $landPage file was not present, Set Splash Page output
-    $splashPage = <<<EOT
+    $splashPage = "
    <!doctype html>
    <html lang='en'>
        <head>
@@ -77,14 +76,13 @@ if ($serverName === "pi.hole"
            <link rel='shortcut icon' href='admin/img/favicons/favicon.ico' type='image/x-icon'>
        </head>
        <body id='splashpage'>
-            <div id="pihole_card">
-              <img src='admin/img/logo.svg' alt='Pi-hole logo' id="pihole_logo_splash" />
-              <p>Pi-<strong>hole</strong>: Your black hole for Internet advertisements</p>
-              <a href='/admin'>Did you mean to go to the admin panel?</a>
-            </div>
+            <img src='admin/img/logo.svg' alt='Pi-hole logo' width='256' height='377'>
+            <br>
+            <p>Pi-<strong>hole</strong>: Your black hole for Internet advertisements</p>
+            <a href='/admin'>Did you mean to go to the admin panel?</a>
        </body>
    </html>
-EOT;
+    ";
    exit($splashPage);
 } elseif ($currentUrlExt === "js") {
    // Serve Pi-hole JavaScript for blocked domains requesting JS
--- a/advanced/lighttpd.conf.debian
+++ b/advanced/lighttpd.conf.debian
@@ -81,10 +81,6 @@ mimetype.assign = (
    ".woff2" => "font/woff2"
 )

-# Add user chosen options held in external file
-# This uses include_shell instead of an include wildcard for compatibility
-include_shell "cat external.conf 2>/dev/null"
-
 # default listening port for IPv6 falls back to the IPv4 port
 include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port

@@ -113,3 +109,7 @@ $HTTP["url"] =~ "^/admin/\.(.*)" {

 # Default expire header
 expire.url = ( "" => "access plus 0 seconds" )
+
+# Add user chosen options held in external file
+# This uses include_shell instead of an include wildcard for compatibility
+include_shell "cat external.conf 2>/dev/null"
--- a/advanced/lighttpd.conf.fedora
+++ b/advanced/lighttpd.conf.fedora
@@ -82,10 +82,6 @@ mimetype.assign = (
    ".woff2" => "font/woff2"
 )

-# Add user chosen options held in external file
-# This uses include_shell instead of an include wildcard for compatibility
-include_shell "cat external.conf 2>/dev/null"
-
 # default listening port for IPv6 falls back to the IPv4 port
 #include_shell "/usr/share/lighttpd/use-ipv6.pl " + server.port
 #include_shell "/usr/share/lighttpd/create-mime.assign.pl"
@@ -121,3 +117,7 @@ $HTTP["url"] =~ "^/admin/\.(.*)" {

 # Default expire header
 expire.url = ( "" => "access plus 0 seconds" )
+
+# Add user chosen options held in external file
+# This uses include_shell instead of an include wildcard for compatibility
+include_shell "cat external.conf 2>/dev/null"
--- a/install/basic-install.sh
+++ b/install/basic-install.sh
--- a/install/uninstall.sh
+++ b/install/uninstall.sh
@@ -31,7 +31,7 @@ else
    else
        echo -e "  ${CROSS} ${str}
            Script called with non-root privileges
-            The Pi-hole requires elevated privileges to uninstall"
+            The Pi-hole requires elevated privleges to uninstall"
        exit 1
    fi
 fi
--- a/gravity.sh
+++ b/gravity.sh
@@ -35,9 +35,8 @@ localList="${piholeDir}/local.list"
 VPNList="/etc/openvpn/ipp.txt"

 piholeGitDir="/etc/.pihole"
-gravityDBfile_default="${piholeDir}/gravity.db"
-# GRAVITYDB may be overwritten by source pihole-FTL.conf below
-GRAVITYDB="${gravityDBfile_default}"
+gravityDBfile="${piholeDir}/gravity.db"
+gravityTEMPfile="${piholeDir}/gravity_temp.db"
 gravityDBschema="${piholeGitDir}/advanced/Templates/gravity.db.sql"
 gravityDBcopy="${piholeGitDir}/advanced/Templates/gravity_copy.sql"

@@ -69,11 +68,6 @@ if [[ -f "${pihole_FTL}" ]]; then
  source "${pihole_FTL}"
 fi

-# Set this only after sourcing pihole-FTL.conf as the gravity database path may
-# have changed
-gravityDBfile="${GRAVITYDB}"
-gravityTEMPfile="${GRAVITYDB}_temp"
-
 if [[ -z "${BLOCKINGMODE}" ]] ; then
  BLOCKINGMODE="NULL"
 fi
@@ -90,11 +84,11 @@ generate_gravity_database() {

 # Copy data from old to new database file and swap them
 gravity_swap_databases() {
-  local str copyGravity
+  local str
  str="Building tree"
  echo -ne "  ${INFO} ${str}..."

-  # The index is intentionally not UNIQUE as poor quality adlists may contain domains more than once
+  # The index is intentionally not UNIQUE as prro quality adlists may contain domains more than once
  output=$( { sqlite3 "${gravityTEMPfile}" "CREATE INDEX idx_gravity ON gravity (domain, adlist_id);"; } 2>&1 )
  status="$?"

@@ -107,14 +101,7 @@ gravity_swap_databases() {
  str="Swapping databases"
  echo -ne "  ${INFO} ${str}..."

-  # Gravity copying SQL script
-  copyGravity="$(cat "${gravityDBcopy}")"
-  if [[ "${gravityDBfile}" != "${gravityDBfile_default}" ]]; then
-    # Replace default gravity script location by custom location
-    copyGravity="${copyGravity//"${gravityDBfile_default}"/"${gravityDBfile}"}"
-  fi
-
-  output=$( { sqlite3 "${gravityTEMPfile}" <<< "${copyGravity}"; } 2>&1 )
+  output=$( { sqlite3 "${gravityTEMPfile}" < "${gravityDBcopy}"; } 2>&1 )
  status="$?"

  if [[ "${status}" -ne 0 ]]; then
@@ -189,7 +176,7 @@ database_table_from_file() {
        echo "${rowid},\"${domain}\",${timestamp}" >> "${tmpFile}"
      elif [[ "${table}" == "adlist" ]]; then
        # Adlist table format
-        echo "${rowid},\"${domain}\",1,${timestamp},${timestamp},\"Migrated from ${source}\",,0,0,0" >> "${tmpFile}"
+        echo "${rowid},\"${domain}\",1,${timestamp},${timestamp},\"Migrated from ${source}\"," >> "${tmpFile}"
      else
        # White-, black-, and regexlist table format
        echo "${rowid},${type},\"${domain}\",1,${timestamp},${timestamp},\"Migrated from ${source}\"" >> "${tmpFile}"
@@ -230,48 +217,6 @@ database_adlist_updated() {
  fi
 }

-# Check if a column with name ${2} exists in gravity table with name ${1}
-gravity_column_exists() {
-  output=$( { printf ".timeout 30000\\nSELECT EXISTS(SELECT * FROM pragma_table_info('%s') WHERE name='%s');\\n" "${1}" "${2}" | sqlite3 "${gravityDBfile}"; } 2>&1 )
-  if [[ "${output}" == "1" ]]; then
-    return 0 # Bash 0 is success
-  fi
-
-  return 1 # Bash non-0 is failure
-}
-
-# Update number of domain on this list. We store this in the "old" database as all values in the new database will later be overwritten
-database_adlist_number() {
-  # Only try to set number of domains when this field exists in the gravity database
-  if ! gravity_column_exists "adlist" "number"; then
-    return;
-  fi
-
-  output=$( { printf ".timeout 30000\\nUPDATE adlist SET number = %i, invalid_domains = %i WHERE id = %i;\\n" "${num_lines}" "${num_invalid}" "${1}" | sqlite3 "${gravityDBfile}"; } 2>&1 )
-  status="$?"
-
-  if [[ "${status}" -ne 0 ]]; then
-    echo -e "\\n  ${CROSS} Unable to update number of domains in adlist with ID ${1} in database ${gravityDBfile}\\n  ${output}"
-    gravity_Cleanup "error"
-  fi
-}
-
-# Update status of this list. We store this in the "old" database as all values in the new database will later be overwritten
-database_adlist_status() {
-  # Only try to set the status when this field exists in the gravity database
-  if ! gravity_column_exists "adlist" "status"; then
-    return;
-  fi
-
-  output=$( { printf ".timeout 30000\\nUPDATE adlist SET status = %i WHERE id = %i;\\n" "${2}" "${1}" | sqlite3 "${gravityDBfile}"; } 2>&1 )
-  status="$?"
-
-  if [[ "${status}" -ne 0 ]]; then
-    echo -e "\\n  ${CROSS} Unable to update status of adlist with ID ${1} in database ${gravityDBfile}\\n  ${output}"
-    gravity_Cleanup "error"
-  fi
-}
-
 # Migrate pre-v5.0 list files to database-based Pi-hole versions
 migrate_to_database() {
  # Create database file only if not present
@@ -372,10 +317,6 @@ gravity_CheckDNSResolutionAvailable() {
 gravity_DownloadBlocklists() {
  echo -e "  ${INFO} ${COL_BOLD}Neutrino emissions detected${COL_NC}..."

-  if [[ "${gravityDBfile}" != "${gravityDBfile_default}" ]]; then
-    echo -e "  ${INFO} Storing gravity database in ${COL_BOLD}${gravityDBfile}${COL_NC}"
-  fi
-
  # Retrieve source URLs from gravity database
  # We source only enabled adlists, sqlite3 stores boolean values as 0 (false) or 1 (true)
  mapfile -t sources <<< "$(sqlite3 "${gravityDBfile}" "SELECT address FROM vw_adlist;" 2> /dev/null)"
@@ -422,7 +363,7 @@ gravity_DownloadBlocklists() {

  target="$(mktemp -p "/tmp" --suffix=".gravity")"

-  # Use compression to reduce the amount of data that is transferred
+  # Use compression to reduce the amount of data that is transfered
  # between the Pi-hole and the ad list provider. Use this feature
  # only if it is supported by the locally available version of curl
  if curl -V | grep -q "Features:.* libz"; then
@@ -452,15 +393,10 @@ gravity_DownloadBlocklists() {
    esac

    echo -e "  ${INFO} Target: ${url}"
-    local regex check_url
+    local regex
    # Check for characters NOT allowed in URLs
    regex="[^a-zA-Z0-9:/?&%=~._()-;]"
-
-    # this will remove first @ that is after schema and before domain
-    # \1 is optional schema, \2 is userinfo
-    check_url="$( sed -re 's#([^:/]*://)?([^/]+)@#\1\2#' <<< "$url" )"
-
-    if [[ "${check_url}" =~ ${regex} ]]; then
+    if [[ "${url}" =~ ${regex} ]]; then
        echo -e "  ${CROSS} Invalid Target"
    else
       gravity_DownloadBlocklistFromUrl "${url}" "${cmd_ext}" "${agent}" "${sourceIDs[$i]}" "${saveLocation}" "${target}" "${compression}"
@@ -503,8 +439,6 @@ gravity_DownloadBlocklists() {
 }

 total_num=0
-num_lines=0
-num_invalid=0
 parseList() {
  local adlistID="${1}" src="${2}" target="${3}" incorrect_lines
  # This sed does the following things:
@@ -515,7 +449,7 @@ parseList() {
  # Find (up to) five domains containing invalid characters (see above)
  incorrect_lines="$(sed -e "/[^a-zA-Z0-9.\_-]/!d" "${src}" | head -n 5)"

-  local num_target_lines num_correct_lines num_invalid
+  local num_lines num_target_lines num_correct_lines num_invalid
  # Get number of lines in source file
  num_lines="$(grep -c "^" "${src}")"
  # Get number of lines in destination file
@@ -524,9 +458,9 @@ parseList() {
  total_num="$num_target_lines"
  num_invalid="$(( num_lines-num_correct_lines ))"
  if [[ "${num_invalid}" -eq 0 ]]; then
-    echo "  ${INFO} Analyzed ${num_lines} domains"
+    echo "  ${INFO} Received ${num_lines} domains"
  else
-    echo "  ${INFO} Analyzed ${num_lines} domains, ${num_invalid} domains invalid!"
+    echo "  ${INFO} Received ${num_lines} domains, ${num_invalid} domains invalid!"
  fi

  # Display sample of invalid lines if we found some
@@ -537,29 +471,6 @@ parseList() {
    done <<< "${incorrect_lines}"
  fi
 }
-compareLists() {
-  local adlistID="${1}" target="${2}"
-
-  # Verify checksum when an older checksum exists
-  if [[ -s "${target}.sha1" ]]; then
-    if ! sha1sum --check --status --strict "${target}.sha1"; then
-      # The list changed upstream, we need to update the checksum
-      sha1sum "${target}" > "${target}.sha1"
-      echo "  ${INFO} List has been updated"
-      database_adlist_status "${adlistID}" "1"
-      database_adlist_updated "${adlistID}"
-    else
-      echo "  ${INFO} List stayed unchanged"
-      database_adlist_status "${adlistID}" "2"
-    fi
-  else
-    # No checksum available, create one for comparing on the next run
-    sha1sum "${target}" > "${target}.sha1"
-    # We assume here it was changed upstream
-    database_adlist_status "${adlistID}" "1"
-    database_adlist_updated "${adlistID}"
-  fi
-}

 # Download specified URL and perform checks on HTTP status and file content
 gravity_DownloadBlocklistFromUrl() {
@@ -643,49 +554,31 @@ gravity_DownloadBlocklistFromUrl() {
      esac;;
  esac

-  local done="false"
  # Determine if the blocklist was downloaded and saved correctly
  if [[ "${success}" == true ]]; then
    if [[ "${httpCode}" == "304" ]]; then
      # Add domains to database table file
      parseList "${adlistID}" "${saveLocation}" "${target}"
-      database_adlist_status "${adlistID}" "2"
-      database_adlist_number "${adlistID}"
-      done="true"
    # Check if $patternbuffer is a non-zero length file
    elif [[ -s "${patternBuffer}" ]]; then
      # Determine if blocklist is non-standard and parse as appropriate
      gravity_ParseFileIntoDomains "${patternBuffer}" "${saveLocation}"
      # Add domains to database table file
      parseList "${adlistID}" "${saveLocation}" "${target}"
-      # Compare lists, are they identical?
-      compareLists "${adlistID}" "${saveLocation}"
-      # Update gravity database table (status and updated timestamp are set in
-      # compareLists)
-      database_adlist_number "${adlistID}"
-      done="true"
+      # Update date_updated field in gravity database table
+      database_adlist_updated "${adlistID}"
    else
      # Fall back to previously cached list if $patternBuffer is empty
-      echo -e "  ${INFO} Received empty file"
+      echo -e "  ${INFO} Received empty file: ${COL_LIGHT_GREEN}using previously cached list${COL_NC}"
    fi
-  fi
-
-  # Do we need to fall back to a cached list (if available)?
-  if [[ "${done}" != "true" ]]; then
+  else
    # Determine if cached list has read permission
    if [[ -r "${saveLocation}" ]]; then
      echo -e "  ${CROSS} List download failed: ${COL_LIGHT_GREEN}using previously cached list${COL_NC}"
      # Add domains to database table file
      parseList "${adlistID}" "${saveLocation}" "${target}"
-      database_adlist_number "${adlistID}"
-      database_adlist_status "${adlistID}" "3"
    else
      echo -e "  ${CROSS} List download failed: ${COL_LIGHT_RED}no cached list available${COL_NC}"
-      # Manually reset these two numbers because we do not call parseList here
-      num_lines=0
-      num_invalid=0
-      database_adlist_number "${adlistID}"
-      database_adlist_status "${adlistID}" "4"
    fi
  fi
 }
@@ -697,7 +590,7 @@ gravity_ParseFileIntoDomains() {
  # Determine if we are parsing a consolidated list
  #if [[ "${source}" == "${piholeDir}/${matterAndLight}" ]]; then
    # Remove comments and print only the domain name
-    # Most of the lists downloaded are already in hosts file format but the spacing/formatting is not contiguous
+    # Most of the lists downloaded are already in hosts file format but the spacing/formating is not contiguous
    # This helps with that and makes it easier to read
    # It also helps with debugging so each stage of the script can be researched more in depth
    # 1) Remove carriage returns
--- a/manpages/pihole.8
+++ b/manpages/pihole.8
@@ -153,7 +153,7 @@ Available commands and options:
 .br
      -r, --refresh     Set update frequency (in seconds)
 .br
-      -e, --exit        Output stats and exit without refreshing
+      -e, --exit        Output stats and exit witout refreshing
 .br

 \fB-g, updateGravity\fR
--- a/72
+++ b/72
@@ -16,12 +16,18 @@ readonly PI_HOLE_SCRIPT_DIR="/opt/pihole"
 # error due to modifying a readonly variable.
 setupVars="/etc/pihole/setupVars.conf"
 PI_HOLE_BIN_DIR="/usr/local/bin"
-readonly FTL_PID_FILE="/run/pihole-FTL.pid"

 readonly colfile="${PI_HOLE_SCRIPT_DIR}/COL_TABLE"
+# shellcheck source=./advanced/Scripts/COL_TABLE
 source "${colfile}"

+# ensure_newline()
+readonly utilsfile="${PI_HOLE_SCRIPT_DIR}/utils.sh"
+# shellcheck source=./advanced/Scripts/utils.sh
+source "${utilsfile}"
+
 webpageFunc() {
+  # shellcheck source=./advanced/Scripts/webpage.sh
  source "${PI_HOLE_SCRIPT_DIR}/webpage.sh"
  main "$@"
  exit 0
@@ -38,9 +44,13 @@ debugFunc() {

  # Pull off the `debug` leaving passed call augmentation flags in $1
  shift
+  # We do not want to be warned about implicit concatenation of array in [[ ]]
+  # as this is exactly what we want here
+  # shellcheck disable=SC2199
  if [[ "$@" == *"-a"* ]]; then
    automated="true"
  fi
+  # shellcheck disable=SC2199
  if [[ "$@" == *"-w"* ]]; then
    web="true"
  fi
@@ -99,25 +109,8 @@ versionFunc() {
  exit 0
 }

-# Get PID of main pihole-FTL process
-getFTLPID() {
-  local pid
-
-  if [ -s "${FTL_PID_FILE}" ]; then
-    # -s: FILE exists and has a size greater than zero
-    pid="$(<"$FTL_PID_FILE")"
-    # Exploit prevention: unset the variable if there is malicious content
-    # Verify that the value read from the file is numeric
-    [[ "$pid" =~ [^[:digit:]] ]] && unset pid
-  fi
-
-  # If FTL is not running, or the PID file contains malicious stuff, substitute
-  # negative PID to signal this to the caller
-  echo "${pid:=-1}"
-}
-
 restartDNS() {
-  local svcOption svc str output status pid icon
+  local svcOption svc str output status
  svcOption="${1:-restart}"

  # Determine if we should reload or restart
@@ -126,34 +119,17 @@ restartDNS() {
    # Note 1: This will NOT re-read any *.conf files
    # Note 2: We cannot use killall here as it does
    #         not know about real-time signals
-    pid="$(getFTLPID)"
-    if [[ "$pid" -eq "-1" ]]; then
-      svc="true"
-      str="FTL is not running"
-      icon="${INFO}"
-    else
-      svc="kill -RTMIN ${pid}"
-      str="Reloading DNS lists"
-      icon="${TICK}"
-    fi
+    svc="pkill -RTMIN pihole-FTL"
+    str="Reloading DNS lists"
  elif [[ "${svcOption}" =~ "reload" ]]; then
    # Reloading of the DNS cache has been requested
    # Note: This will NOT re-read any *.conf files
-    pid="$(getFTLPID)"
-    if [[ "$pid" -eq "-1" ]]; then
-      svc="true"
-      str="FTL is not running"
-      icon="${INFO}"
-    else
-      svc="kill -HUP ${pid}"
-      str="Flushing DNS cache"
-      icon="${TICK}"
-    fi
+    svc="pkill -HUP pihole-FTL"
+    str="Flushing DNS cache"
  else
    # A full restart has been requested
    svc="service pihole-FTL restart"
    str="Restarting DNS server"
-    icon="${TICK}"
  fi

  # Print output to Terminal, but not to Web Admin
@@ -163,7 +139,7 @@ restartDNS() {
  status="$?"

  if [[ "${status}" -eq 0 ]]; then
-    [[ -t 1 ]] && echo -e "${OVER}  ${icon} ${str}"
+    [[ -t 1 ]] && echo -e "${OVER}  ${TICK} ${str}"
    return 0
  else
    [[ ! -t 1 ]] && local OVER=""
@@ -189,7 +165,7 @@ Time:
      echo -e "  ${INFO} Blocking already disabled, nothing to do"
      exit 0
    fi
-    if [[ $# > 1 ]]; then
+    if [[ $# -gt 1 ]]; then
      local error=false
      if [[ "${2}" == *"s" ]]; then
        tt=${2%"s"}
@@ -197,7 +173,7 @@ Time:
          local str="Disabling blocking for ${tt} seconds"
          echo -e "  ${INFO} ${str}..."
          local str="Blocking will be re-enabled in ${tt} seconds"
-          nohup "${PI_HOLE_SCRIPT_DIR}"/pihole-reenable.sh ${tt} </dev/null &>/dev/null &
+          nohup "${PI_HOLE_SCRIPT_DIR}/pihole-reenable.sh" "${tt}" </dev/null &>/dev/null &
        else
          local error=true
        fi
@@ -207,8 +183,8 @@ Time:
          local str="Disabling blocking for ${tt} minutes"
          echo -e "  ${INFO} ${str}..."
          local str="Blocking will be re-enabled in ${tt} minutes"
-          tt=$((${tt}*60))
-          nohup "${PI_HOLE_SCRIPT_DIR}"/pihole-reenable.sh ${tt} </dev/null &>/dev/null &
+          tt=$((tt*60))
+          nohup "${PI_HOLE_SCRIPT_DIR}/pihole-reenable.sh" "${tt}" </dev/null &>/dev/null &
        else
          local error=true
        fi
@@ -226,6 +202,7 @@ Time:

      local str="Pi-hole Disabled"
      sed -i "/BLOCKING_ENABLED=/d" "${setupVars}"
+      ensure_newline "${setupVars}"
      echo "BLOCKING_ENABLED=false" >> "${setupVars}"
    fi
  else
@@ -239,6 +216,7 @@ Time:
    local str="Pi-hole Enabled"

    sed -i "/BLOCKING_ENABLED=/d" "${setupVars}"
+    ensure_newline "${setupVars}"
    echo "BLOCKING_ENABLED=true" >> "${setupVars}"
  fi

@@ -355,7 +333,8 @@ statusFunc() {

 tailFunc() {
  # Warn user if Pi-hole's logging is disabled
-  local logging_enabled=$(grep -c "^log-queries" /etc/dnsmasq.d/01-pihole.conf)
+  local logging_enabled
+  logging_enabled=$(grep -c "^log-queries" /etc/dnsmasq.d/01-pihole.conf)
  if [[ "${logging_enabled}" == "0" ]]; then
    # No "log-queries" lines are found.
    # Commented out lines (such as "#log-queries") are ignored
@@ -396,6 +375,7 @@ Branches:
    exit 0
  fi

+  # shellcheck source=./advanced/Scripts/piholeCheckout.sh
  source "${PI_HOLE_SCRIPT_DIR}"/piholeCheckout.sh
  shift
  checkout "$@"
--- a/supportedos.txt
+++ b/supportedos.txt
@@ -1,5 +1,5 @@
 Raspbian=9,10
 Ubuntu=16,18,20
 Debian=9,10
-Fedora=32,33
+Fedora=31,32
 CentOS=7,8
--- a/test/_fedora_31.Dockerfile
+++ b/test/_fedora_31.Dockerfile
@@ -1,4 +1,4 @@
-FROM fedora:33
+FROM fedora:31

 ENV GITDIR /etc/.pihole
 ENV SCRIPTDIR /opt/pihole
--- a/test/test_automated_install.py
+++ b/test/test_automated_install.py
@@ -524,45 +524,43 @@ def test_IPv6_ULA_GUA_test(Pihole):
    assert expected_stdout in detectPlatform.stdout


-def test_validate_ip(Pihole):
+def test_validate_ip_valid(Pihole):
    '''
-    Tests valid_ip for various IP addresses
+    Given a valid IP address, valid_ip returns success
    '''

-    def test_address(addr, success=True):
-        output = Pihole.run('''
-        source /opt/pihole/basic-install.sh
-        valid_ip "{addr}"
-        '''.format(addr=addr))
+    output = Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    valid_ip "192.168.1.1"
+    ''')

-        assert output.rc == 0 if success else 1
+    assert output.rc == 0

-    test_address('192.168.1.1')
-    test_address('127.0.0.1')
-    test_address('255.255.255.255')
-    test_address('255.255.255.256', False)
-    test_address('255.255.256.255', False)
-    test_address('255.256.255.255', False)
-    test_address('256.255.255.255', False)
-    test_address('1092.168.1.1', False)
-    test_address('not an IP', False)
-    test_address('8.8.8.8#', False)
-    test_address('8.8.8.8#0')
-    test_address('8.8.8.8#1')
-    test_address('8.8.8.8#42')
-    test_address('8.8.8.8#888')
-    test_address('8.8.8.8#1337')
-    test_address('8.8.8.8#65535')
-    test_address('8.8.8.8#65536', False)
-    test_address('8.8.8.8#-1', False)
-    test_address('00.0.0.0', False)
-    test_address('010.0.0.0', False)
-    test_address('001.0.0.0', False)
-    test_address('0.0.0.0#00', False)
-    test_address('0.0.0.0#01', False)
-    test_address('0.0.0.0#001', False)
-    test_address('0.0.0.0#0001', False)
-    test_address('0.0.0.0#00001', False)
+
+def test_validate_ip_invalid_octet(Pihole):
+    '''
+    Given an invalid IP address (large octet), valid_ip returns an error
+    '''
+
+    output = Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    valid_ip "1092.168.1.1"
+    ''')
+
+    assert output.rc == 1
+
+
+def test_validate_ip_invalid_letters(Pihole):
+    '''
+    Given an invalid IP address (contains letters), valid_ip returns an error
+    '''
+
+    output = Pihole.run('''
+    source /opt/pihole/basic-install.sh
+    valid_ip "not an IP"
+    ''')
+
+    assert output.rc == 1


 def test_os_check_fails(Pihole):
--- a/test/tox.fedora_31.ini
+++ b/test/tox.fedora_31.ini
@@ -4,5 +4,5 @@ envlist = py37
 [testenv]
 whitelist_externals = docker
 deps = -rrequirements.txt
-commands = docker build -f _fedora_33.Dockerfile -t pytest_pihole:test_container ../
+commands = docker build -f _fedora_31.Dockerfile -t pytest_pihole:test_container ../
           pytest {posargs:-vv -n auto} ./test_automated_install.py ./test_centos_fedora_common_support.py ./test_fedora_support.py
Author	SHA1	Message	Date
DL6ER	3b6286e139	Spellcheck improvements (no warnings in pihole and advanced/Scripts/webpage.sh remains) Signed-off-by: DL6ER <dl6er@dl6er.de>	2021-01-05 09:58:22 +01:00
DL6ER	e1a128f833	Move ensure_newline() into its own file as we need to source it from different places Signed-off-by: DL6ER <dl6er@dl6er.de>	2020-12-30 21:48:07 +01:00
DL6ER	69e55aff83	Ensure critical config lines have trailing newlines before appending new content. This is guaranteed by our scripts, however, it is not guaranteed any longer when uders manually edited config files themselves. Ensure the files are okay to work with before adding anything to them. Signed-off-by: DL6ER <dl6er@dl6er.de>	2020-12-30 10:50:31 +01:00