Pull cf-package updates from feedstocks repo

[CJ-Wright]

@CJ-Wright commented on Wed Feb 14 2018

As @isuruf mentioned we may be able to update the CF version in the graph by pulling commit messages from feedstocks.

[jakirkham]

FYI there is an Atom feed for the feedstocks repo that GitHub provides. So that could be a pretty good hook. We could also talk webhooks if you want something more direct.

Was thinking about adding it to the webpage for style. ( conda-forge/conda-forge.github.io#527 )

[CJ-Wright]

I'd be interested to have bots update the graph as soon as PRs are merged if that is possible (it would help to eliminate the 00 and 01 scripts.

[jakirkham]

Yeah so feedstocks is updated basically immediately after merge (delay in seconds). Not sure if that is tolerable latency for you. If so, the atom feed is one option. Another might be a webhook from the feedstocks repo to something over here (not sure what endpoints we have). The atom feed is something you could use today. The webhook is just a matter of figuring out where it goes.

If we need to reduce the latency further, we would need to do a little more work, but it should still be doable. We could look at having a notification from the feedstocks update job itself (on Heroku). Alternatively we could have the feedstocks have webhooks to the graph service directly. There might be other options along these lines that we haven't considered (maybe an org level webhook about pushes?).

My impression is working with a single stream of events would be easier than trying to coordinate multiple streams. Plus this is a problem GitHub is really good at solving. So the more we rely on GitHub to serialize org wide events the better.

[CJ-Wright]

Latency shouldn't be an issue (I hope). So the feed should work. I don't grok yet how the feed would inform the graph, do something need to constantly listen to the feed, or can we queue up things from the feed which a worker goes and gets when it starts?

[jakirkham]

I think it would be easier to advise with a little more information about where this is getting hooked into and where things are being run. Is there something I can read or could you give me a rough summary?

[CJ-Wright]

The readme has a fairly high level view. (docs are on the issue list)

The current setup is:

1. All scripts (except 00 for now, but working on that) write to the central graph.pkl file.
   1. 00 writes the names of the all the feedstocks into names.txt (although it will write into the graph directly soon) This could be replaced with something writing the names of new feedstocks as they come in from staged.
   2. 01 updates all the graph node attributes with information from the meta.yaml. This could be replaced with something that a) takes data from newly minted feedstocks and b) takes data from newly merged PRs into feedstocks.
   3. 02 goes out and finds the upstream version (if it can)
   4. 03 updates the meta.yaml for feedstocks which need bumping and PRs the changes in.

[jakirkham]

What triggers these scripts to run currently? Is it done manually? Is there a cron job somewhere?

Also where are they run? Locally? Heroku? CI? Somewhere else?

[CJ-Wright]

They are on Travis CI as daily Cron jobs.
Eg https://travis-ci.org/regro/cf-auto-tick

[jakirkham]

Not seeing an obvious way to do this, but I wonder if your bot could watch the feedstocks repo and use the resulting notifications to trigger a rebuild on Travis CI. Might need some form of debouncing to avoid triggering additional builds when one is already running.

[CJ-Wright]

I think that there needs to be some form of reactive software in there somewhere to bridge the gap.