Repositories list

• abx-spec-behaviors

  Public
  Proposal for a shared user script specification between scraping, crawling, archiving, and AI tools. Allows user scripts like "expand comments", "hide popups", "fill out this form", etc. to be reusable across pure browser environments, puppeteer, playwright, extensions, and many other contexts with minimal adjustments.

  ecosystemautomationbrowser+ 17scrapingcrawlingpluginsspecificationgreasemonkeytampermonkeybehaviors+ 10

  JavaScript

  •

  MIT License

  •0•8•1•0•Updated Nov 12, 2024Nov 12, 2024

• abx-pkg

  Public
  📦 Modern Python library for managing system dependencies with package managers like apt, brew, pip, npm, etc.

  pythonnpmbrew+ 9djangodependency-managerpackagesaptpipdependenciespydantic+ 2

  Python

  •

  MIT License

  •0•14•0•0•Updated Nov 7, 2024Nov 7, 2024

• ArchiveBox

  Public
  🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

  pythonrssbackups+ 17firefoxpinboardyoutube-dlchromiumself-hostedwgetpocket+ 10

  Python

  •

  MIT License

  •1.2k•22k•190•2•Updated Nov 4, 2024Nov 4, 2024

• abx-dl

  Public
  ⬇️ A CLI tool to download all discovered content from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git srcs, and more...

  clichromedownloader+ 15curlheadlessscrapingcrawlinghttp-clientyoutube-dlwget+ 8

  Python

  •

  MIT License

  •2•18•0•0•Updated Oct 21, 2024Oct 21, 2024

• docker-archivebox

  Public
  Home of the official docker image for ArchiveBox

  dockerkubernetesimage+ 8docker-composecontainerocidigiprespodmaninternet-archivingdocker-image+ 1

  GNU General Public License v3.0

  •12•47•1•1•Updated Oct 16, 2024Oct 16, 2024

• pip-archivebox

  Public archive
  Official Python package for ArchiveBox, the self-hosted internet archiving solution.

  pythonpypiwheel+ 7pipsetuptoolsweb-archivingdigipressdistinternet-archivingarchivebox

  GNU General Public License v3.0

  •2•13•0•7•Updated Oct 5, 2024Oct 5, 2024

• homebrew-archivebox

  Public archive
  Homebrew formula for the ArchiveBox self-hosted internet archiving solution.

  macoshomebrewpackage+ 6linuxbrewweb-archivingdigipresbrew-tapinternet-archivingarchivebox

  Ruby

  •

  GNU General Public License v3.0

  •3•26•0•0•Updated Oct 5, 2024Oct 5, 2024

• debian-archivebox

  Public archive
  Home of the official apt/deb package for Ubuntu/Debian-based systems.

  packagedebianapt+ 7ubuntuweb-archivingaptitudedigipresinternet-archivingarchiveboxstdeb

  Python

  •

  GNU General Public License v3.0

  •5•17•0•1•Updated Oct 5, 2024Oct 5, 2024

• docs

  Public
  Source for the Github Wiki / ReadTheDocs documentation for AchiveBox, the self-hosted internet archiving solution.

  pythonclicommunity+ 10documentationuirestwikisphinxusageweb-archiving+ 3

  CSS

  •4•14•0•1•Updated Oct 5, 2024Oct 5, 2024

• readability-extractor

  Public
  Javascript/Node wrapper around Mozilla's Readability library so that ArchiveBox can call it as a oneshot CLI command to extract each page's article text.

  wrappernodereadability+ 2internet-archivingarchivebox

  JavaScript

  •13•37•0•2•Updated Sep 16, 2024Sep 16, 2024

• archivebox-browser-extension

  Public
  Official ArchiveBox browser extension: automatically/manually preserve your browsing history using ArchiveBox.

  chrome-extensionarchivingsvelte+ 7firefox-extensionbrowser-extensionweb-archivingdigital-preservationdigipresinternet-archivingarchivebox

  TypeScript

  •

  MIT License

  •21•246•19•0•Updated Jul 12, 2024Jul 12, 2024

• archivebox-proxy

  Public
  Official ArchiveBox MITM proxy: saves URLs of all requests passing through to an ArchiveBox server for archival.

  proxyhttps-proxyweb-archiving+ 6web-proxydigital-preservationmitmproxydigipresinternet-archivingarchivebox

  Python

  •

  MIT License

  •0•14•0•0•Updated Jul 12, 2024Jul 12, 2024

• good-karma-kit

  Public
  😇 A Docker Compose bundle to run on servers with spare CPU, RAM, disk, and bandwidth to help the world. Includes Tor, ArchiveWarrior, BOINC, and more...

  dockerdocker-composeipfs+ 15distributed-computingtordistributed-storagesiaboinckiwixi2p+ 8

  MIT License

  •8•319•0•0•Updated May 11, 2024May 11, 2024

• squasher-browser-extension

  Public
  Extension to collect all open browser tabs for a given domain into a new window (with suspender support).

  JavaScript

  •

  MIT License

  •1•0•0•0•Updated Mar 10, 2024Mar 10, 2024

• community

  Public
  A wiki of the broader Web Archiving Community: important organizations, alternative projects, blog posts, and more.

  internetarchivingpreservation+ 3digipreswebarchivingarchivebox

  0•4•0•0•Updated Feb 21, 2024Feb 21, 2024

• DigestBox

  Public
  DigestBox takes any webpage URL (news article, video link, comment thread, etc.) and gives you just the raw content. It's powered by ArchiveBox.io under the hood.

  backupswarcweb-archiving+ 4digipresheadless-browserinternet-archivingarchivebox

  HTML

  •0•15•1•0•Updated Feb 2, 2024Feb 2, 2024

• electron-archivebox

  Public
  Desktop Electron app for ArchiveBox internet archiver. (ALPHA: not ready for general use)

  electronwindowsmacos+ 9linuxguiweb-archivingdigipresinternet-archivingdockerdesktop+ 2

  JavaScript

  •

  GNU General Public License v3.0

  •15•178•4•0•Updated Feb 28, 2023Feb 28, 2023

• internet-archiving-talk

  Public
  🎭 An introduction to the Internet Archiving ecosystem, tooling, and some of the ethical dilemmas that the community faces.

  JavaScript

  •5•14•0•0•Updated Oct 19, 2020Oct 19, 2020