Fix For SexbabesVR Scraper #1847
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The scene id in the the webpage now seems to be 614 for all scenes. Causing all scenes to be rescraped and never adding new scenes. This pulls the poster url which appears to have a unique identifier in the 2nd to last directory . Also updated the cover URL to pull the image used for the thumbnail on the index page. As the latest scene has has a SBS image for the cover where the thumbnail contains a more useful image All appears functional
pops64
force-pushed
the
SexbabesVR_Fix
branch
from
da5a2df
to
28e93ab
Compare
There are three separate variations on how they have this information posted depending on the age of the scene. A random sampling over all scenes shows that the synopsis is successfully being scraped
It ran once I am unsure of how to properly test it tho.
|
The scraper update looks fine, but I think we should get rid of the migration. It takes forever without actually changing the scene IDs and in my case it even lead to a duplicate for some reason: |
|
According to KLH there are some scenes that change scene ids starting around 610. Tomorrow, I will add logic to only check scenes starting at 600 and if they match what is already present don't update. This should greatly cut down on the migration time because newer scenes do get a little wonky. |
Added some error handling incase the website is unreachable. Added logic to ensure we only check scenes originating from SexBabesVR. Check only scenes starting at 600 as this is where the reported divergence between sceneID sources numbering occurred. And only update scenes that diverge in id
|
Done |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The scene id in on the scene webpage now seems to be 614 for all scenes. Causing all scenes to be rescraped and never adding new scenes.
This pulls the poster url which appears to have a unique identifier in the last directory.
Also updated the cover URL to pull the image used for the thumbnail on the index page. As the latest scene has a SBS image for the cover where the thumbnail contains a more useful image
All appears functional.
Might require deleting all scraped SexbabesVR scenes due to a shift in scene-id causing file mismatch and scene preview generation hiccupsAdded Migration Code