Skip to content

@scrapinghub Scrapinghub

Change the repository type filter

All

    Repositories list

    183 repositories

    • web-poet

      Public
      Web scraping Page Objects core library
      pythonweb-scrapinghacktoberfestpage-objects
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •15•95•14•14•Updated Oct 10, 2024Oct 10, 2024

    • scrapy-poet

      Public
      Page Object pattern for Scrapy
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •28•119•9•5•Updated Oct 10, 2024Oct 10, 2024

    • spidermon

      Public
      Scrapy Extension for monitoring spiders execution.
      testingmonitoringscrapingcrawlingspidershacktoberfestmonitoring-toolscrapinghub
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •96•530•39•6•Updated Oct 10, 2024Oct 10, 2024

    • scrapinghub-stack-scrapy

      Public
      Software stack with latest Scrapy and updated deps
      Dockerfile
      •
      BSD 3-Clause "New" or "Revised" License
      •20•60•2•1•Updated Oct 7, 2024Oct 7, 2024

    • shub-workflow

      Public
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •14•13•2•0•Updated Oct 2, 2024Oct 2, 2024

    • dateparser

      Public
      python parser for human readable dates
      hacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •465•2.5k•284•50•Updated Oct 2, 2024Oct 2, 2024

    • python-crfsuite

      Public
      A python binding for crfsuite
      data-sciencecrfcrfsuite
      Python
      •
      MIT License
      •221•770•45•3•Updated Oct 1, 2024Oct 1, 2024

    • streamparse

      Public
      streamparse lets you run Python code against real-time streams of data. Integrates with Apache Storm.
      Python
      •
      Apache License 2.0
      •217•2•0•1•Updated Sep 20, 2024Sep 20, 2024

    • number-parser

      Public
      Parse numbers written in natural language
      hacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •23•107•12•6•Updated Sep 16, 2024Sep 16, 2024

    • Formasaurus

      Public
      Formasaurus tells you the type of an HTML form and its fields using machine learning
      HTML
      •47•7•0•1•Updated Aug 7, 2024Aug 7, 2024

    • splash

      Public
      Lightweight, scriptable browser as a service with an HTTP API
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •515•4.1k•377•26•Updated Aug 2, 2024Aug 2, 2024

    • extruct

      Public
      Extract embedded metadata from HTML markup
      microformatssemantic-webrdfajson-ldmicrodataopengraphhacktoberfest
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •113•847•38•15•Updated Jul 25, 2024Jul 25, 2024

    • pgcontents

      Public
      A Postgres-backed ContentsManager implementation for IPython
      Python
      •
      Apache License 2.0
      •83•2•0•1•Updated Jul 18, 2024Jul 18, 2024

    • hcf-backend

      Public
      Crawl Frontier HCF backend
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •5•7•2•1•Updated Jul 17, 2024Jul 17, 2024

    • shublang

      Public
      Pluggable DSL that uses pipes to perform a series of linear transformations to extract data
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •8•15•23•6•Updated Jul 9, 2024Jul 9, 2024

    • scrapinghub-entrypoint-scrapy

      Public
      Scrapy entrypoint for Scrapinghub job runner
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •16•25•7•0•Updated Jul 8, 2024Jul 8, 2024

    • woodpecker

      Public
      An opinionated fork of the Drone CI system
      Go
      •
      Other
      •363•0•0•5•Updated Jul 7, 2024Jul 7, 2024

    • varanus

      Public
      A command line spider monitoring tool
      spidermonitoringpython36
      Python
      •7•8•2•2•Updated Jul 6, 2024Jul 6, 2024

    • scrapyrt

      Public
      HTTP API for Scrapy spiders
      pythoncrawlerscrapercrawlingtwistedscrapywebcrawlerhacktoberfestwebcrawlinghacktoberfest2021
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •162•832•24•6•Updated Jun 28, 2024Jun 28, 2024

    • portia

      Public
      Visual scraping for Scrapy
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •1.4k•9.3k•111•19•Updated Jun 26, 2024Jun 26, 2024

    • sklearn-crfsuite

      Public
      scikit-learn inspired API for CRFsuite
      Python
      •215•1•0•0•Updated Jun 18, 2024Jun 18, 2024

    • webstruct-demo

      Public
      HTTP demo for https://github.com/scrapinghub/webstruct
      Python
      •
      MIT License
      •2•4•0•3•Updated Jun 17, 2024Jun 17, 2024

    • autologin

      Public
      A project to attempt to automatically login to a website given a single seed
      Python
      •
      Apache License 2.0
      •44•11•0•2•Updated Jun 17, 2024Jun 17, 2024

    • python-intercom

      Public
      Python wrapper for the Intercom API.
      Python
      •
      Other
      •145•1•0•1•Updated Jun 17, 2024Jun 17, 2024

    • luigi

      Public
      Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
      Python
      •
      Apache License 2.0
      •2.4k•4•0•1•Updated Jun 7, 2024Jun 7, 2024

    • mrjob

      Public
      Run MapReduce jobs on Hadoop or Amazon Web Services
      Python
      •
      Other
      •587•0•0•1•Updated Jun 6, 2024Jun 6, 2024

    • andi

      Public
      Library for annotation-based dependency injection
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •5•20•3•1•Updated Jun 3, 2024Jun 3, 2024

    • docker-custodian

      Public
      Keep docker hosts tidy
      Python
      •
      Apache License 2.0
      •50•0•0•1•Updated May 21, 2024May 21, 2024

    • aduana

      Public
      Frontera backend to guide a crawl using PageRank, HITS or other ranking algorithms based on the link structure of the web graph, even when making big crawls (one billion pages).
      data-science
      C
      •
      BSD 3-Clause "New" or "Revised" License
      •9•55•9•2•Updated May 21, 2024May 21, 2024

    • exporters

      Public
      Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
      Python
      •
      BSD 3-Clause "New" or "Revised" License
      •10•40•5•7•Updated May 21, 2024May 21, 2024