NOT MERGED: Add bucket storage support

[cuonglm]

THIS PR WAS NOT MERGED

REPLACED BY #11387

PLEASE NOTE This PR was not merged and has been labelled merged in error due to force-pushing master before closing the PR

It was replaced by #11387

For future reference the commits that were on this branch are available at https://github.com/zeripath/gitea/tree/pr-9567

Many thanks to @cuonglm for their hard work on this PR - even though the work was not merged it was helpful and appreciated.

---

This is a follow up PR for #7795, cherry-pick those commits and add:

• Abstract filesystem layer for avatars/attachments/lfs objects, using afero
• Fix some issue with test, error checking
• Add tests for bucket storage using minio (for s3 compatibility) and fake-gcs-server (for GCS)
• Add storage-migrate command to migrate from local to cloud storage and vice versa.

Running:

gitea storage-migrate --bucket='s3://gitea?endpoint=localhost:9000&disableSSL=true&s3ForcePathStyle=true&region=us-east-1'

will migrate from local to cloud storage, and:

gitea storage-migrate --local --bucket='s3://gitea?endpoint=localhost:9000&disableSSL=true&s3ForcePathStyle=true&region=us-east-1'

will migrate back from cloud to local.

[lunny]

I think all temp dir should be local file system.

[techknowlogick]

Thank you for this PR 😄 very excited for it.

I've gone through and noticed a few items with import structures, and noted a few of them (there are others that I haven't noted, but should still also be resolved)

[6543]

please resolve conflicts

[i0x71]

Is there any chance of storing all of the git data on the s3 backend ?

Thanks

[cuonglm]

Is there any chance of storing all of the git data on the s3 backend ?

Thanks

I think there's likely no plan to support that feature.

[zeripath]

I've saved the old contents of this PR as the pr-9567 branch in my fork zeripath.

Was this force push intentional?

[lunny]

Why this PR' state is merged but changed files is 0? Could you review #11387 ?

[zeripath]

@lunny the pr branch has been force pushed to master therefore there are no longer any differences between it and master, and GitHub has declared it merged. (The same thing happens in Gitea - where incidentally people complained it was a bug.)

Has #11387 replaced this PR?

[lunny]

@zeripath Oh. I see.

Yes, #11387 just merged and replaced this PR. I will send planned PRs on that PR.

[patcito]

@zeripath Sorry, not intentional. We can repush the old commits if you want but it looks like this PR was rejected anyway to be replaced with #11387. Is this the case @lunny ?

[zeripath]

@patcito AFAIU that will not revert the mislabelling of the PR being merged. I suspect it is not a huge problem - the PR was not on a milestone, it now has the skip-changelog label, and its functionality has been replaced. I've changed the top comment to reflect the situation. I could change the PR title but it might be too much.

Feel free to try to repush the old changes but I suspect that won't change the status of the PR.

[patcito]

@zeripath @lunny this PR used go-cloud which supports Google cloud storage and azure. Would you accept contributions that add support to Google and azure using go cloud?

[lunny]

@patcito On my PR, I defined a minial interafce and I think it's easy to implement. We have local and minio two implementations and new implementation are welcome!

[a1012112796]

@lunny the pr branch has been force pushed to master therefore there are no longer any differences between it and master, and GitHub has declared it merged. (The same thing happens in Gitea - where incidentally people complained it was a bug.)

Has #11387 replaced this PR?

@lunny @lafriks I have checked this bug carefully, I found the reason is that the results of fast-forward style merge and just make head branch same with base branch by force-push are the same, It's hard to distinguish between them by "git" cmd. It's really a hard thing... :(

ref:

gitea/services/pull/check.go

Lines 115 to 123 in e14f608

	// Get the commit from BaseBranch where the pull request got merged
	mergeCommit, err := git.NewCommand("rev-list", "--ancestry-path", "--merges", "--reverse", cmd).
	RunInDirWithEnv("", []string{"GIT_INDEX_FILE=" + indexTmpPath, "GIT_DIR=" + pr.BaseRepo.RepoPath()})
	if err != nil {
	return nil, fmt.Errorf("git rev-list --ancestry-path --merges --reverse: %v", err)
	} else if len(mergeCommit) < 40 {
	// PR was fast-forwarded, so just use last commit of PR
	mergeCommit = commitID[:40]
	}

[zeripath]

yes @a1012112796 that is what I was meaning, and as this PR demonstrates Gitea replicates GH's behaviour exactly.

In Gitea we could do something different whereby if a force-push to master occurs we close the PR instead but at present it appears we replicate GH exactly.

[a1012112796]

@lunny @lafriks maybe we can use same way with gitlab: not check manually merge and allow create pr without any commits, just block it to be merge is enough.