Skip to content

Commit

Permalink
scraper: POVR Scraper Fix (#1825)
Browse files Browse the repository at this point in the history
* Fix for POVR scraper

They are now putting porn star suggestions on the same page as scenes. This has the same search parameters as the scenes. The easist way is to make sure that links only in the `/vr-porn` directory are scraped. As there really isn't any identifiable way to target only the scene links through Query Selectors

* Typo fix

* Add trailing slash

Just to make sure it matches a directory
  • Loading branch information
pops64 committed Aug 28, 2024
1 parent 7ca0840 commit 4d214be
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions pkg/scrape/povr.go
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,8 @@ func POVR(wg *sync.WaitGroup, updateSite bool, knownScenes []string, out chan<-
siteCollector.OnHTML(`div.thumbnail-wrap div.thumbnail a.thumbnail__link`, func(e *colly.HTMLElement) {
sceneURL := e.Request.AbsoluteURL(e.Attr("href"))

// If scene exists in database, or the slternate source exists, there's no need to scrape
if !funk.ContainsString(knownScenes, sceneURL) && !strings.Contains(sceneURL, "/join") {
// If scene exists in database, or the slternate source exists, there's no need to scrape. Also make sure we only grab valid scene links in the vr-porn directory
if !funk.ContainsString(knownScenes, sceneURL) && strings.Contains(sceneURL, "/vr-porn/") && !strings.Contains(sceneURL, "/join") {
WaitBeforeVisit("povr.com", sceneCollector.Visit, sceneURL)
}
})
Expand Down

0 comments on commit 4d214be

Please sign in to comment.