Security issues in crawling

[innovationchef]

Thoughts -

1. Faithful Crawling - the input website may not contain relevant bioschemas data
2. Massive JSON-LD or web pages
3. Filter frontend inputs
4. Denial of service attacks

[innovationchef]

1. How do I check if the json-ld received contains the relevant life sciences data?
2. I don't think scrapy takes care of massive web pages (I will update this in case I find something in documentation). So how do we check if the web page is massive? What limit should be put on the size?
3. Will be taken later when we start the frontend part.

[justinccdev]

1. Good question. I think this is an argument for explicitly selecting the sites indexed rather than doing a general crawl. A very good source of sites may be https://fairsharing.org, run by @Drosophilic
2. Of course I believe you buy I'm surprised it doesn't. Yeah, not sure about size limit, what's the size of the average Biosamples page?

[innovationchef]

Took me some time to find this out, but Scrapy has a safety valve around the downloader. If the aggregated size of Response in progress is larger than 5 MB it stops the flow of further Request into the downloader.
See here

[innovationchef]

@justinccdev
There are 791 databases currently listed on the fairsharing.org website related to life sciences. ebi.co.uk/biosamples is one of them. So, should I check if the website that the user is giving as input is listed in that database before crawling?
Link here

[justinccdev]

Whilst we may very probably want to use fairsharing.org information and/or Bioschemas live deploys in the future as sources for default sites to crawl, we don't want to restrict the user as to what they can crawl, I think.