TED recipe by topic fails with HTTP read timed out in extract_info_from_video_page

[benoit74]

While many TED recipe by topic are working, some are systematically failing (at least twice in a row) with an HTTP read timeout in the extract_info_from_video_page operations:

• https://farm.openzim.org/recipes/ted_topic_ted-conference
• https://farm.openzim.org/recipes/ted_topic_technology
• https://farm.openzim.org/recipes/ted_topic_culture
• https://farm.openzim.org/recipes/ted_topic_sustainability

Looks like a given recipe is not failing always on the same video (but is it because video order is random?)

[benoit74]

New ones:

• https://farm.openzim.org/recipes/ted_topic_creativity
• https://farm.openzim.org/recipes/ted_topic_nature

[benoit74]

• https://farm.openzim.org/recipes/ted_topic_global-issues
• https://farm.openzim.org/recipes/ted_topic_history
• https://farm.openzim.org/recipes/ted_topic_industrial-design
• https://farm.openzim.org/recipes/ted_topic_demo
• https://farm.openzim.org/recipes/ted_topic_software
• https://farm.openzim.org/recipes/ted_topic_math
• https://farm.openzim.org/recipes/ted_topic_public-health
• https://farm.openzim.org/recipes/ted_topic_equality

[benoit74]

Not reproduced locally.

A common point on all these tasks is that they all ran on pixelmemory, except https://farm.openzim.org/pipeline/4058f2dc-e460-498a-991f-e72c5a12c680 which ran on athena18. Maybe just a coincidence, but I will request them again on another worker and we will see.

[benoit74]

Task https://farm.openzim.org/pipeline/ce6d0579-a246-4098-ac78-d35d78cec00e/debug is already past the point where it previously failed. This hence looks like an issue specific to some workers? I will confirm by starting again a small recipe on pixelmemory

[benoit74]

Seems to be transient issues because https://farm.openzim.org/pipeline/70613e69-5e39-462e-8c3a-1a2354ddbfba from https://farm.openzim.org/recipes/ted_topic_history now succeeded on pixelmemory.

Closing this issue for "not reproduced" for now. I've added a section to the FAQ.

[benoit74]

Issue is still happening, always on pixelmemory worker, but mostly randomly.

We need to look into how to make this more robust / identify the underlying root cause which happens only on pixelmemory worker.