CO pull location update each day

[syphax-bouazzouni]

The CO ontologies are updated each day at 18:00 from there pull URL.

Ontologies concerned :

Started from November 2021 (here is the example of CO_325)

This may be the cause of this problems :

• may be the cause of this 4store crash this days (even after the restart job)
• this route is down http://data.agroportal.lirmm.fr/submissions which make the landscape also be out of service http://agroportal.lirmm.fr/landscape (my hypotheses is that there is too much submissions) .
• slow down the portal (cause to see an ontology summary all the it's submissions are load)

Todo :

• Do a ruby script that remove data and files of a submission range (from "id1" to "id2" ) of an ontology. ontoportal-lirmm/ncbo_cron#4
• see if NCBO have done a script that clean used graphs in 4store #179

[jonquet]

Long term resolution: avoid to parse ontologies when the source file is exactly the same but retrieved by the automatic pull.
See #171

Solution for CO ontologies:

• Recontact with @marieALaporte to see if cropontology.org can make sure that HTTP headers (date) do not change if file (that is regenerated at each call) has not changed.
• Manually list the ontologies concerned here. => @jonquet
• Manually temporarily unplug all the CO ontologies concerned => @jonquet
• Write a script (in ncbo_cron) that for an ontology remove all the submissions (from 4store and files) above a certain submisison id => @syphax-bouazzouni
• Test the script on stage
• Run the script on each ontologies listed above

[jonquet]

List of ontologies to process (28):
CO_358, CO_350, CO_357, CO_345, CO_339, CO_335, CO_338, CO_348, CO_325, CO_331, CO_346, CO_341, CO_330, CO_327, CO_360, CO_322, CO_337, CO_366, CO_321, CO_340, CO_320, CO_324, CO_343, CO_365, CO_334, CO_336, CO_356, CO_323,

All ontologies unplugged => pullLocation" : ""

Notes:

• Problem with http://data.agroportal.lirmm.fr/ontologies/CO_347/latest_submission
  => restarted processing

• Problem with http://data.agroportal.lirmm.fr/ontologies/CO_366/latest_submission
  => last submission (24) is empty. Need to be cleanned and rebranch to last working submission as all the others.

Also unplugged POLAPGEN_BARLEY, CO_121, CO_020 which pullURLs were generating an error in the log (but no notification email). To be fixed when we will resume all pullURLs

[syphax-bouazzouni]

@jonquet so after checking the code to know more about how is the ncbo_cron job figuring out, that a new version of an ontology was released.
And i found that it's not looking for the http header but download every day the ontologies and hash it to compare it with the local ones (see code below source : https://github.com/ontoportal-lirmm/ncbo_cron/blob/master/lib/ncbo_cron/ontology_pull.rb#L54)

[syphax-bouazzouni]

Summary of What Todo (after the last updates)

• Unplug pull locations the CO ontologies in Agroportal
• Clean unwanted submissions of the CO ontologies (using the eradicator script) #180
• See if the clean up resolve the issues with the api /submissions endpoint (related CO_347 has no submission reachable #175)
• Resolve the problem of CO API that generate a new files in each request (see with @marieALaporte )
• Plug the new pullLocations of the CO ontologies in Agroportal

[jonquet]

After discussion with @marieALaporte we will either :

• Fix the new file generated situation on the cropontollogy.org site and stay in "pull mode" in AgroPortal
• Add an automatic push from cropontology.org to AgroPortal in the ontology modification script (push mode)