flashgordon

speed. I am speed.

Goals

The purpose of flashgordon is to gather useful data from the internet as quickly as possible.

A crawler will be ran to make get requests to URLs, parse the data from the response, and store relevant data.

As this goal is not a destination but rather a measure of improvement, the metrics used to measure progress will be Integrity and Performance.

Integrity: By integrity, I mean completeness.

The data collected should be everything needed/useful, and the process used to collect it should be reliable, safe, and loving.

Any changes made to the project should clearly improve its integrity.

Performance:

All processes within flashgordon should be fast and require as little work as possible to complete.

Any changes made to flashgordon should measurably improve its performance.

Installation and First crawl

1. Install postgres with sudo apt install postgresql
2. Start postgres with systemctl services postgres start
3. Create role and database
4. sudo su - postgres
5. psql
6. create role flash with login password ‘password’; grant all privileges on database lightning_db to flash; alter database lightning_db owner to flash;
7. Clone the repository
8. pip3 install -r requirements.txt
9. pip3 install psycopg2 if it's not in the requirements
10. Configure postgres
11. python3 manage.py migrate
12. python3 manage.py crawler