Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LFS support to be stored on minio #12518

Merged
merged 8 commits into from
Sep 8, 2020
Merged

LFS support to be stored on minio #12518

merged 8 commits into from
Sep 8, 2020

Conversation

lunny
Copy link
Member

@lunny lunny commented Aug 18, 2020 •
edited
Loading

As a next step of #11387, Fix #5530

@lunny lunny added the type/enhancement An improvement of existing functionality label Aug 18, 2020
@lunny lunny added this to the 1.13.0 milestone Aug 18, 2020
@lunny lunny mentioned this pull request Aug 18, 2020
Merged
3 tasks
@lunny lunny force-pushed the lunny/lfs_bucket branch 2 times, most recently from be7ea1f to 11775dd Compare August 19, 2020 01:38
@codecov-commenter
Copy link

codecov-commenter commented Aug 19, 2020 •
edited
Loading

Codecov Report

Merging #12518 into master will decrease coverage by 0.03%.
The diff coverage is 25.94%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #12518      +/-   ##
==========================================
- Coverage   43.30%   43.26%   -0.04%     
==========================================
  Files         647      648       +1     
  Lines       71800    71886      +86     
==========================================
+ Hits        31090    31099       +9     
- Misses      35675    35745      +70     
- Partials     5035     5042       +7
Impacted Files Coverage Δ
cmd/migrate_storage.go 0.00% <0.00%> (ø)
modules/repofiles/update.go 40.30% <0.00%> (-0.31%) ⬇️
modules/repofiles/upload.go 0.00% <0.00%> (ø)
routers/repo/lfs.go 0.00% <0.00%> (ø)
modules/storage/minio.go 48.43% <5.00%> (-20.46%) ⬇️
models/lfs.go 22.64% <11.76%> (-2.08%) ⬇️
modules/lfs/server.go 43.85% <23.07%> (-1.69%) ⬇️
modules/storage/storage.go 44.44% <30.76%> (-12.70%) ⬇️
modules/setting/lfs.go 41.86% <41.86%> (ø)
modules/lfs/content_store.go 50.00% <57.14%> (+1.66%) ⬆️
... and 13 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e4b3f35...b6580de. Read the comment docs.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Aug 19, 2020
@lunny lunny changed the title WIP: LFS support to be stored on minio LFS support to be stored on minio Aug 19, 2020
zeripath
zeripath reviewed Aug 19, 2020
View reviewed changes
modules/storage/storage.go
Comment on lines +104 to +120
case "local":
LFS, err = NewLocalStorage(setting.LFS.ContentPath)
case "minio":
minio := setting.LFS.Minio
LFS, err = NewMinioStorage(
context.Background(),
minio.Endpoint,
minio.AccessKeyID,
minio.SecretAccessKey,
minio.Bucket,
minio.Location,
minio.BasePath,
minio.UseSSL,
)
default:
return fmt.Errorf("Unsupported LFS store type: %s", setting.LFS.StoreType)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there really no way to make these self-register?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know. But the current codes look clear and simple.

@lafriks
Copy link
Member

lafriks commented Aug 22, 2020

Do we really need duplicate minio configuration for each type of data stored in it? Should different buckets would not be enaugh

@techknowlogick
Copy link
Member

@lafriks some users may wish to use cheaper s3 providers for avatars, and then more robust s3 provider for LFS

@lunny
Copy link
Member Author

lunny commented Aug 23, 2020 •
edited
Loading

@lafriks I will send another PR to add a storage config so that attachments, avatar, lfs could inherit from that if they don't configure that.

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Aug 31, 2020
@zeripath
Copy link
Contributor

zeripath commented Sep 2, 2020

hmm... I'm wondering if we'd be better off calving off the storage options into:

[storage] ; this is the default storage option
TYPE=local
PATH=/data ; specific storage will inherit from this as a subpath

[storage.attachments]
TYPE=local; minio or the like
; PATH would default to [storage] PATH/attachments
... ; specific storage content config

[storage.lfs]
TYPE=local; minio or the like
... ; specific storage content config

...

Then we can move the specific configuration to the storage module and do something like queues configuration stuff.

@lunny
Copy link
Member Author

lunny commented Sep 2, 2020

@zeripath Yes, that's my next PR. But I will only add [storage] and keep other configuration on [attchements], [lfs] and etc.

@lunny lunny mentioned this pull request Sep 3, 2020
Closed
7 tasks
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Sep 8, 2020
@lunny lunny merged commit 7a5465f into go-gitea:master Sep 8, 2020
@lunny lunny deleted the lunny/lfs_bucket branch November 18, 2020 04:37
@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@mrsdizzie mrsdizzie mrsdizzie approved these changes

@zeripath zeripath zeripath approved these changes

Assignees
No one assigned
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/enhancement An improvement of existing functionality
Projects
None yet
Milestone
1.13.0
Development

Successfully merging this pull request may close these issues.

move lfs to S3 (minio)
7 participants
@lunny @codecov-commenter @lafriks @techknowlogick @zeripath @mrsdizzie @GiteaBot