Add script for backfilling reregisterurl invocations

[jwodder]

For dandi/backups2datalad#36 (comment)

[yarikoptic]

Looks good to me... I guess we should have merged/done this before that

• Use reregisterurl to move datalad remotes to web backups2datalad#36

Let's merge and could you please run it?

[jwodder]

@yarikoptic This script should probably be run while there aren't any backups2datalad processes running, but it appears that there's currently a backups2datalad-update-cron process that's been running since the 15th and may be stuck.

[yarikoptic]

:-( could you please check what it is up to and mitigate one way or another?

I will add a https://hc-ping.com/ monitor to those cron invocations... I wish we could somehow integrate that with upptime . Filed

• https://github.com/orgs/upptime/discussions/986

[jwodder]

@yarikoptic I don't know. The last logfile entries are from a minute after the process started, and they're just general stuff like "Remote Dandiset has not been modified since last backup; not syncing" and "Tagging releases for Dandiset" interspersed with tons of Git invocations. Based on what processes are running, I think a git-annex process got stuck while shutting down, preventing other things from finishing.

It's conceivable that just killing the git-annex processes should make everything clean up, but it might also leave behind a mess somewhere.

[yarikoptic]

I see that now it is only your backfilling script running git-annex. I guess you did get that process killed. Please check after if dandisets are in clean state etc. Thanks!

[jwodder]

@yarikoptic OK, I killed a couple git-annex processes, and that got things moving again, and backups2datalad soon finished. The dataset was clean afterwards. The backfill script is now running.

[jwodder]

@yarikoptic The backfill script finished, and the datasets are clean. I'm not sure how to check that everything was migrated, since the script's output pushed the Dandiset IDs out of the scrollback.

[jwodder]

@yarikoptic Just picking one formerly-embargoed Dandiset (000897) whose git-annex branch was last modified while the backfill script was running, the output from git-annex whereis on an arbitrary file is:

whereis sub-amadeus/sub-amadeus_ses-08152019_behavior+ecephys.nwb (2 copies) 
        00000000-0000-0000-0000-000000000001 -- web
        cf13d535-b47c-5df6-8590-0793cb08a90a -- datalad

  web: https://api.dandiarchive.org/api/assets/d3a96834-ee80-4afa-b985-82066817272c/download/
  web: https://dandiarchive.s3.amazonaws.com/blobs/a6e/c32/a6ec3274-ceeb-4d21-b091-1e991a512c7b?versionId=Vt7RKy0cgO1L82S7tqIQRQgNHBBZVtVh
ok

Adding --json clarifies that git-annex still recognizes the "datalad" remote (even though it was removed) but does not associate any URLs with it. Should we consider everything all good now?

[yarikoptic]

git remote manipulations do nothing to the "git annex knowledgebase" about keys/urls -- it pretty much just disables that remote in this clone. And I do not think we should announce datalad to be dead remote.

hm, I wonder if it is a bug in git-annex that it would still report it associated with the remote even though no URLs for it and otherwise remote nohow providing it... filed https://git-annex.branchable.com/bugs/keeps_ext_remote_after_all_urls_unregistered/?updated

Overall, I think it should suffice to check if not all keys are available from web. I left running in 3648 screen

for d in 0*; do echo $d; grep EmbargoedAccess $d/dandiset.yaml || git -C $d annex find --not --in web | head -n 3; done

seems to be all good!