fix: swaps from github API to expanding the assets

[holtjma]

This is basically a patch to fix a new issue (described here: bioconda/bioconda-recipes#42176) that was introduced by myself here: #896.

The primary cause is that the GitHub API is rate limited (which I was not aware of), so autobump basically blows through the limits quickly and then subsequent calls automatically fail. Of course, this means that the tools we were hoping to see get autobumped are not.

This patch basically removes the original patch (which uses a single GitHub API call) and replaces it with an approach that basically "expands" the release webpage to find the assets. See #895 for more details on how the URLs are masked and the expansion pattern.

Couple other notes:

• I added an IncludeFragmentParser that is basically identical to the HrefParser except it is looking for tags with this pattern: <include-fragment src="http://url/of/interest" ...>
• The general expansion logic is:
  • Pull the full release page
  • Search for <include-fragment ...> tags
  • Iteratively pull these looking for the main link URL of interest
  • Break the above iteration early if the current version is identified
• This should no longer use the GitHub API, I'm unaware of other rate limiters for these URLs (🤞that none are hidden...)

Pinging @dpryan79 and @johanneskoester as you both reviewed the last related PR. Sorry that the previous PR injected a rate limit issue!

EDIT: I forgot, I have tested these on both hiphase and pbfusion recipes, the diff of the results are here and match our current expectations:

(bioconda)$ git diff
diff --git a/recipes/hiphase/meta.yaml b/recipes/hiphase/meta.yaml
index 38b939bf9f..b5d2503597 100644
--- a/recipes/hiphase/meta.yaml
+++ b/recipes/hiphase/meta.yaml
@@ -1,6 +1,6 @@
 {% set name = "hiphase" %}
-{% set version = "0.10.0" %}
-{% set hiphase_sha256 = "e22d3530a395bd218bd4634537f5b3c31a82cd8155849291c69d81c48f0c58fe" %}
+{% set version = "0.10.2" %}
+{% set hiphase_sha256 = "2a20fc437ab5c0f2437246e8a36e7590c68f5ae5caa0cecb8e85a371f5357c6c" %}
 
 package:
   name: {{ name }}
diff --git a/recipes/pbfusion/meta.yaml b/recipes/pbfusion/meta.yaml
index 781ada3413..ca69e87d0f 100644
--- a/recipes/pbfusion/meta.yaml
+++ b/recipes/pbfusion/meta.yaml
@@ -1,6 +1,6 @@
 {% set name = "pbfusion" %}
-{% set version = "0.2.2" %}
-{% set pbfusion_sha256 = "3390adbeb8550265fcf4fad89dad80740cc368e96d10f3b23bb19ef001ffd081" %}
+{% set version = "0.3.0" %}
+{% set pbfusion_sha256 = "ef9767af7e82bd1e664711e7521d7a8370a41f101cccf1db4a58895d9f88934c" %}
 
 package:
   name: {{ name }}

[holtjma]

That failure seems odd, almost looks like it froze for some reason when I checked timestamps. I would restart it, but I don't think I can.

[holtjma]

Looks like checks passed now, just waiting on feedback from maintainers on the change

[johanneskoester]

Looks good to me! Thanks a lot! Of course I would prefer the API as we never know what the do with the page layout. Maybe we can keep the API code as well, and just have both options? This way, we could later decide to use a rate limiter or maybe convince github of some higher limits without having to implement everything again.

[jmarshall]

See my comment on bioconda/bioconda-recipes#42176: if we store etags from the API calls, it’d be trivial to avoid the rate limits.

[holtjma]

Maybe we can keep the API code as well, and just have both options? This way, we could later decide to use a rate limiter or maybe convince github of some higher limits without having to implement everything again.

@johanneskoester I moved the API version and the expanded asset version into subroutines that are invoked. John's point about the etags / last modified is probably the best way to handle the API in the future, but I'm not sure what long-term memory access autobump would have to retain that info. Those modifications could probably exist in a future PR though.

Tests appear to be passing again, and I double checked that results were the same with hiphase and pbfusion recipes.