-
Notifications
You must be signed in to change notification settings - Fork 0
/
README.txt
65 lines (46 loc) · 2.48 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
****************** WEB CRAWLER *****************
----------------------------------------------------------------------------------------------
----------------------------------- Setup ---------------------------------------------------
---------------------------------------------------------------------------------------------
This code requires the following software packages for it to run successfully:
1. Python 3.6.3
Can be downloaded from "https://www.python.org/downloads/"
2. BeautifulSoup package
Can be downloaded from "https://www.crummy.com/software/BeautifulSoup/"
Can be installed using pip, by entering the following command
in Terminal or Command Line :
pip install beautifulsoup4
3. Requests package
Can be installed using pip, by entering the following command
in Terminal or Command Line :
pip install requests
--------------------------------------------------------------------------------------------
-------------------------------------- Run--------------------------------------------------
--------------------------------------------------------------------------------------------
1. Place the file 'WikiWebCrawler.py' in new folder.
2. Open Terminal (or Windows PowerShell for Windows) and make sure
that the present working directory is the folder where you have placed 'WikiWebCrawler.py'.
3. Make sure python's path is set in the environment variables of your computer.
4. Run the code by entering the following command:
python WikiWebCrawler.py
5. The program asks you which task you want to perform and
prompts you for an answer.
Press 1 for Task 1 : Unfocused Crawling.
Press 2 for Task 2 : Focused Crawling.
Enter your option:
Enter either 1 or 2 depending on your choice.
By default the seed URL is "https://en.wikipedia.org/wiki/Tropical_cyclone"
and the keyword is "rain"
and the depth is 6
If you need to change these values, you can do so in the top section against the names:
SEED_URL , KEYWORD , MAX_DEPTH
6. At the end, the maximum depth the crawler could reach is printed on the console.
7. The raw html of all the pages crawled is saved in the present working directory.
Also a textfile named 'CrawledPageList.txt' is generated which lists all the URLs crawled
by the crawler.
References:
-----------
https://learnpythonthehardway.org
http://docs.python-requests.org/en/master/user/quickstart/
http://www.tutorialspoint.com/python/
http://www.pythonforbeginners.com/python-on-the-web/web-scraping-with-beautifulsoup/