Incorrect link rewriting

[rgaudin]

There is a bad link in the article users/241039/oleh-prypin (entry idx=58950210) in the ZIM file:

<a href="//meta.stackoverflow.com/../../users/17034/hans-passant?tab=answers&amp;sort=votes#user-tab-answers">

The corresponding link in the current version of https://stackoverflow.com/users/241039/oleh-prypin is

<a href="//meta.stackoverflow.com/users/17034/hans-passant?tab=answers&amp;sort=votes#user-tab-answers">

The scraper added unjustified "../.." in an absolute URL belonging to a different domain.

Of course zimcheck shouldn't crash because of it and I will fix that. But now you have early indication to the bug in the scraper.

Originally posted by @veloman-yunkan in openzim/zim-tools#305 (comment)

[rgaudin]

@TheCrazyT how is this relevant to this ticket? Did you answer here by mistake?

Regarding BS not being threadsafe it's probably not the issue as BS is instantiated each time in the rewriter which is called by the rewrote() Jinja filter in templates. We may well be leaking around this but BS not being threadsafe is probably not at play

Regarding this ticket now, I appears that above link is actually correct, although it should probably not have been tampered with to begin with as it is an external link. Problem is probably the same-scheme // that was not taken care of.

[TheCrazyT]

Well guess the problem is that it just substitudes the current url, but ignores to_root,that already is ../.. .

sotoki/src/sotoki/utils/html.py

Line 257 in 841a968

uri_path = re.sub(r"^(\.\.?/)+", "", uri.path)

sotoki/src/sotoki/renderer.py

Line 203 in 841a968

to_root="../../",

[TheCrazyT]

@TheCrazyT how is this relevant to this ticket? Did you answer here by mistake?

Regarding BS not being threadsafe it's probably not the issue as BS is instantiated each time in the rewriter which is called by the rewrote() Jinja filter in templates. We may well be leaking around this but BS not being threadsafe is probably not at play

Regarding this ticket now, I appears that above link is actually correct, although it should probably not have been tampered with to begin with as it is an external link. Problem is probably the same-scheme // that was not taken care of.

Sorry the comment about beautifulsoup was a mistake, thats why I deleted it and hoped you won't notice 😀

[TheCrazyT]

Ah now i see ... the detection if it is a relative url is actually already wrong.

(rewrite_relative_link should never be used in those cases)

[kelson42]

@rgaudin This kind of primitive should be in scraperlib and tested... if not already available in a stable lib.

[rgaudin]

@TheCrazyT your commit looks good; can you submit a PR ?