GitHub - troby/linkcheck: Basic module for crawling a domain in search of missing links.

troby / linkcheck Public

Notifications You must be signed in to change notification settings
Fork 0
Star 0

Basic module for crawling a domain in search of missing links.

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 80 Commits
.gitignore		.gitignore
LICENSE		LICENSE
README		README
functional_test.py		functional_test.py
linkcheck.py		linkcheck.py
test.py		test.py

Repository files navigation

Basic tool for crawling a domain in search of missing links.

Currently, this tool only saves 404 errors to a special list,
but enabling verbose will provide output of every response
status code. Linkcheck will verify external links, but it will
not scrape them for further links to be tested.

Dependencies:
python3.x
requests
BeautifulSoup4

Usage:

import linkcheck
check = linkcheck.SiteChecker('mysite.com')
check.start()
check.results()

The following defaults can be changed before invoking start():
check.verbose = False
check.delay = 3


Functional test:
The localhost site tested by functional_test.py should
have the following files:

download.html:
<html>
<body>
<a href="test.mp3">music file</a>
<a href="missing_2.html">missing</a>
</body>
</html>

index.html:
<html>
<body>
<a href="download.html">downloads</a>
<a href="missing_1.html">missing</a>
</body>
</html>

test.mp3:
binary file data