privoxy-blocklist

Script converting AdBlock Plus rules into privoxy format.

How does it work

The script privoxy-blocklist.sh downloads AdBlock Plus filter files and generates privoxy compatible filter and action files based on these. After the generation is done it modifies the privoxy configuration files /etc/privoxy/config to import the generated files.

Due to this behaviour the script must run as root user to be able to modify the privoxy configuration file.

Usage

Either run privoxy-blocklist.sh manually with root privileges (e.g., sudo privoxy-blocklist.sh) or via root cronjob.

Convert Mode

By default privoxy-blocklist.sh converts given lists into Privoxy format and registers them afterwards in Privoxy configuration file to be loaded. The Convert Mode converts the lists, stores them into given path and exists. Thus it does not register the lists in Privoxy configuration.

Running in Convert Mode requires options -A.

Example:

> privoxy-blocklist.sh -A

Skip activation of 'easylist.script.action' due to 'Convert Mode'.

Content Filter

By default privoxy-blocklist only generates URL based filter rules as content filtering may slowdown proxying a lot. During tests I had requests that took up to 4 minutes.

To activate content filters specify the corresponding filter types either in the configuraiton file or via cli-flag -f, e.g.:

privoxy-blocklist.sh -f attribute_global_name -f attribute_global_exact -f attribute_global_contain -f attribute_global_startswith -f attribute_global_endswith -f class_global -f id_global

To see all supported filter types check the help privoxy-blocklist.sh -h.

Content filtering for HTTPS URLs requires Privoxy to be compiled with FEATURE_HTTPS_INSPECTION and HTTPS inspection configured. Example commands for the configuration can be found in install_deps.sh

Without FEATURE_HTTPS_INSPECTION content filtering only works for unencrypted HTTP-URLs.

If the feature is enabled can be tested on http://config.privoxy.org/show-status. Just open with page via your Privoxy and search for FEATURE_HTTPS_INSPECTION.

Some distributions provide Privoxy with HTTPS support enabled. The following table shows support status of some tested distributions:

Distribution	HTTPS Support
alpine	no
OpenWRT	no
TurrisOS	no
Ubuntu	yes

Installation

1. Install all dependencies:
   • privoxy
   • sed
   • grep
   • bash
   • wget
   • can be simplified by running helper/install_deps.sh which support Debian, ArchLinux and Alpine based installation
2. Download privoxy-blocklist.sh from the asset list of latest release

Feature Support

The following table shows features of AdBlock Plus filters and there status within privoxy-blocklist:

Feature	Type	Status	Test
:-abp-contains()	extended CSS selector	❓	❓
:-abp-has()	extended CSS selector	❓	❓
:-abp-properties()	extended CSS selector	❓	❓
||…	block domain matching excluding scheme	✅	✅
|…|	block exact domain matching including scheme	❓	❓
!…	comments	✅
csp=	filter options	❓	❓
##.class	global CSS attribute selector with matching for class	✅ (via -f class_global)	✅
###id	global CSS attribute selector with matching for id	✅ (via -f id_global)	✅
##[attribute]	global CSS attribute selector with matching for attribute-name	✅ (via -f attribute_global_name)	✅
##[attribute=value]	global CSS attribute selector with matching for attribute-value pair	✅ (via -f attribute_global_exact)	✅
##[attribute^=value]	global CSS attribute selector with matching for attribute with value starting with	✅ (via -f attribute_global_startswith)	✅
##[attribute$=value]	global CSS attribute selector with matching for attribute with value ending with	✅ (via -f attribute_global_endswith)	✅
##[attribute*=value]	global CSS attribute selector with matching for attribute with value containing	✅ (via -f attribute_global_contain)	✅
##html-tag[attribute]	global CSS attribute selector for html-tag with matching for attribute-name	🚧	🚧
##html-tag[attribute=value]	global CSS attribute selector for html-tag with matching for attribute-value pair	🚧	🚧
##html-tag[attribute^=value]	global CSS attribute selector for html-tag with matching for attribute with value starting with	🚧	🚧
##html-tag[attribute$=value]	global CSS attribute selector for html-tag with matching for attribute with value ending with	🚧	🚧
##html-tag[attribute*=value]	global CSS attribute selector for html-tag with matching for attribute with value containing	🚧	🚧
[…]#$#	domain based CSS selector - Snippet filter	❓	❓
[…]##	domain based CSS selector - Element hiding	❓	❓
[…]#?#	domain based CSS selector - Element hiding emulation	❓	❓
[…]#@#	domain based CSS selector - Element hiding exception	❓	❓
document	filter options	❓	❓
~domain=	filter options	❓	❓
domain=	filter options	❓	❓
~elemhide	filter options	❓	❓
elemhide	filter options	❓	❓
@@||…	exception for domain blocking rules	✅	🚧
font	filter options	❓	❓
genericblock	filter options	❓	❓
generichide	filter options	❓	❓
~image	filter options	❓	❓
image	filter options	✅	🚧
match-case	filter options	❓	❓
media	filter options	❓	❓
~object	filter options	❓	❓
object	filter options	❓	❓
~other	filter options	❓	❓
other	filter options	❓	❓
~ping	filter options	❓	❓
ping	filter options	❓	❓
popup	filter options	❓	❓
rewrite=	filter options	❓	❓
~script	filter options	❓	❓
script	filter options	❓	❓
sitekey=	filter options	❓	❓
~stylesheet	filter options	❓	❓
stylesheet	filter options	❓	❓
~subdocument	filter options	❓	❓
subdocument	filter options	❓	❓
~third-party	filter options	❓	❓
third-party	filter options	❓	❓
~webrtc	filter options	❓	❓
webrtc	filter options	❓	❓
~websocket	filter options	❓	❓
websocket	filter options	❓	❓
~xmlhttprequest	filter options	❓	❓
xmlhttprequest	filter options	❓	❓

• ❓ => status must be checked
• ✅ => implemented
• 🚧 => work in progress

Sources:

• Adblock Plus Help
• Adblock Plus Cheatsheet

Development

Release

The release process is automated via github action Release and triggered by pushing a tag to the main branch.

The following tags are recognized:

Tag-Schema	Result
[0-9]+.[0-9]+.[0-9]+	create public release e.g., 0.4.0
[0-9]+.[0-9]+.[0-9]+-a	create private alpha release e.g., 0.4.0-a (only visible to collaborators)
[0-9]+.[0-9]+.[0-9]+-rc	create public release candidate e.g., 0.4.0-rc (pre-release)

Tests

Code changes must be tested to ensure that all functionality is working as intended. For that a pytest based test suite is maintained and runs on every pull request within Gitlab Actions.

Run Local Tests

The test suite is designed to run within a docker container based on the definition of this repository. It is currently only tested on Ubuntu Linux but should work on every system with a POSIX compliant shell and docker.

To start all tests of the test suite just run:

./tests/run.sh

To start a single test file you can run:

./tests/run.sh tests/test_….py

To start all tests on ubuntu:

./tests/run.sh -o ubuntu

The whole process simplified by ./tests/run.sh runs the following:

1. build docker image based on project Dockerfile
2. start container based on created image with git repository mounted to /app and persistent volume for pytest-cache mounted to /pytest_cache

Within the container all pytest magic happens and all scripts matching test_*.py within tests/ are executed.

Kudos

• Badge Generator for providing easy method to generate badges/shields