Handle Exceptions

[innovationchef]

To handle exceptions, I have to create one and see the response. How to throw exceptions for test purposes?

[innovationchef]

@justinccdev, the website we are currently crawling throws no error (404, 402 etc). However, to make our crawler robust, we have to write exception handlers too. Any suggestions on how we can test this? I would then be able to handle issue #8 too.

[justinccdev]

You know, I'm happy simply to ignore such failures right now.

If you did want to do this and test, you could set up a local webserver (e.g. with https://docs.python.org/3.6/library/http.server.html or some other python module) and point the crawler at it.

[innovationchef]

Well, if it's not so urgent, then I will keep this issue and issue #8 open for now. I will take them in the end and focus on other parts for now.