Mindtouch scraper

This scraper transform content on a Mindtouch / Nice CXone Expert instance into a in ZIM file, a clean and user friendly format for storing content for offline usage.

While meant to be generic for any Mindtouch instance, it has so far been tested only on libretexts.org instance, so it is highly expectable some rough edges might need to be polished until it works on all Mindtouch instance.

Installation

There are three main ways to install and use mindtouch2zim from most recommended to least:

Install using a pre-built container

1. Download the image using docker:

   docker pull ghcr.io/openzim/mindtouch

Build your own container

1. Clone the repository locally:

   git clone https://github.com/openzim/mindtouch.git && cd mindtouch

2. Build the image:

   docker build -t ghcr.io/openzim/mindtouch .

Run the software locally using Hatch

1. Clone the repository locally:

   git clone https://github.com/openzim/mindtouch.git && cd mindtouch

2. Install Hatch:

   pip3 install hatch

3. Start a hatch shell to install software and dependencies in an isolated virtual environment.

   hatch shell

4. Run the mindtouch2zim command:

   mindtouch2zim --help

Usage

Warning

This project is still a work in progress and isn't ready for use yet, the commands below are examples only.

# Get help
docker run -v output:/output ghcr.io/openzim/mindtouch mindtouch2zim --help

# Create a ZIM for https://geo.libretexts.org
docker run -v output:/output ghcr.io/openzim/mindtouch mindtouch2zim --library-url https://geo.libretexts.org --name libretexts.org_en_geo --creator LibreTexts --title "LibreTexts Geosciences" --description "Geosciences courses from libretexts.org"

Developing

Use the commands below to set up the project once:

# Install hatch if it isn't installed already.
❯ pip install hatch

# Local install (in default env) / re-sync packages
❯ hatch run pip list

# Set-up pre-commit
❯ pre-commit install

The following commands can be used to build and test the scraper:

# Show scripts
❯ hatch env show

# linting, testing, coverage, checking
❯ hatch run lint:all
❯ hatch run lint:fixall

# run tests on all matrixed' envs
❯ hatch run test:run

# run tests in a single matrixed' env
❯ hatch env run -e test -i py=3.12 coverage

# run static type checks
❯ hatch env run check:all

# building packages
❯ hatch build

Contributing

This project adheres to openZIM's Contribution Guidelines.

This project has implemented openZIM's Python bootstrap, conventions and policies v1.0.3.

See details for contributions in CONTRIBUTING.md.