need to "shrink" or annex .dandi/assets.json #244
Comments
|
@yarikoptic So annex |
|
@yarikoptic Note that, in this particular case, you can get rid of the large |
Following https://git-annex.branchable.com/todo/annex.addunlocked_in_gitattributes/ NB there is also full script#!/bin/bash
cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)"
set -eu
git init
git annex init
set -x
git show git-annex:config.log || :
touch size1 size5 large
git add *
git commit -m 'added empty files'
git show
echo size5 >| size5
echo 1 >| size1
echo 1000000000000000000 >| large
git annex config --set annex.addunlocked 'include=size*'
git annex config --set annex.largefiles 'largerthan=3b and include=size*'
git show git-annex:config.log
git add *
git commit -m 'added populated size*'
git show
which ended doing the right thing -- adding size1 into git and the "large" size5 into git-annex, and large (not matching path pattern) to git: commit 80cd3354dcd494b9372512994bb0496417fdcc22 (HEAD -> master)
Author: Yaroslav Halchenko <[email protected]>
Date: Mon Aug 1 12:11:01 2022 -0400
added populated size*
diff --git a/large b/large
index e69de29..c9a4149 100644
--- a/large
+++ b/large
@@ -0,0 +1 @@
+1000000000000000000
diff --git a/size1 b/size1
index e69de29..d00491f 100644
--- a/size1
+++ b/size1
@@ -0,0 +1 @@
+1
diff --git a/size5 b/size5
index e69de29..c99f40a 100644
--- a/size5
+++ b/size5
@@ -0,0 +1 @@
+/annex/objects/SHA256E-s6--b389ab2d20c61de2680db3fbed2c4f9dac6b68c1b4125ef0abeee1cf0136b1a6but we already rely on generic On 000026 I did call annex config to set addunlocked but then can't seems to manage to add that damn .dandi/assets.json to annex -- annex keeps saying it is not large file: (dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/000026$ echo '' >> .dandi/assets.json
(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/000026$ git annex add .dandi/assets.json
add .dandi/assets.json (non-large file; adding content to git repository) ok
(recording state in git...)
(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/000026$ git check-attr --all .dandi/assets.json
.dandi/assets.json: annex.backend: SHA256E
.dandi/assets.json: annex.largefiles: (largerthan=1mb)
.dandi/assets.json: filter: annex
(dandisets) dandi@drogon:/mnt/backup/dandi/dandisets/000026$ du -scm .dandi/assets.json
67 .dandi/assets.json
67 totalMay be you see what I am doing wrong -- or might be some bug/intricacy of git-annex???
dunno yet, but do we need to make it available to others? isn't it only for internal use by our scripts ATM? |
ha -- missed that there is a warning about 50MB limit and hard error after 100MB -- will squash now |
This is manually squashed commit to overcome dandi/dandisets#244 2nd commit after "Added" was: [backups2datalad] 39814 assets garbage-collected from .dandi/assets.json
|
squashed, and pushed . Got a warning but no error. So immediate problem mitigated but let's still do that limiting by 1MB.. actually let's boost it to 5MB (10th of warning size). Here is current sizes: so most are small and will go under git. |
|
just a reminder that apparently we never made those large ones locked: https://github.com/dandisets/000026/blob/967c5f0d35d2c59d0ec958c6458902003e3a170e/.dandi/assets.json has it in full. It seems that I did not add |
as initially reported in #230 (comment),
git pushfails for 000026 withso -- github/microsoft is pushing to use of their LFS. I think we should just make use of
annex unlocked(so we do not need to mess with all the lock/unlock etc) file functionality and keep it under git-annex for this and all the rest of dandisets (for the uniformity). Or do you see some other way @jwodder ?The text was updated successfully, but these errors were encountered: