Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: backend sftp #68

Merged
merged 9 commits into from
Oct 20, 2019
Merged

feature: backend sftp #68

merged 9 commits into from
Oct 20, 2019

Conversation

alexisvisco
Copy link
Contributor

@alexisvisco alexisvisco commented Oct 9, 2019

Adding the sftp as backend for the cache storage.
Relation with issue #60

todo:

  • init sft client and add it to the backend switch
  • implementing Put
  • implementing Get
  • test, re-test and test and ... test.

fix: typo in filesystem skeeker -> seeker
@alexisvisco
Copy link
Contributor Author

alexisvisco commented Oct 9, 2019

To test the sftp storage backend:

Run docker run -p 22:22 -d atmoz/sftp foo:pass:::upload this will create a sftp server with an user foo and he can only access to /home/foo/upload

Then replace TestSFTPTruth with this code:

func TestSFTPTruth(t *testing.T) {
	cli, err := InitializeSFTPBackend(SFTPConfig{
		CacheRoot: "/upload",
		Username:  "foo",
		Auth: SSHAuth{
			Password:      "pass",
			Method:        SSHAuthMethodPassword,
		},
		Host:      "localhost",
		Port:      "22",
	}, true)
	if err != nil {
		t.Fatal(err)
	}

	content := "Hello world4"

	// PUT TEST
	file, _ := os.Create("test")
	_, _ = file.Write([]byte(content))
	_, _ = file.Seek(0, 0)
	err = cli.Put("test3.t", file)
	if err != nil {
		t.Fatal(err)
	}
	_ = file.Close()

	// GET TEST
	readCloser, err := cli.Get("test3.t")
	if err != nil {
		t.Fatal(err)
	}
	b, _ := ioutil.ReadAll(readCloser)
	if !bytes.Equal(b, []byte(content))  {
		t.Fatal(string(b), "!=", content)
	}

	_ = os.Remove("test")
}

@alexisvisco
Copy link
Contributor Author

alexisvisco commented Oct 9, 2019

Missing test with a ftp server (not a sftp) don't merge right now :)

@kakkoyun kakkoyun added do not merge WIP Work in progress labels Oct 9, 2019
@kakkoyun
Copy link
Contributor

kakkoyun commented Oct 9, 2019

@alexisvisco Good work so far, thanks a lot. Ping me when you're done.

@alexisvisco alexisvisco changed the title [WIP] sftp: add structure layout of the backend sftp: add structure layout of the backend Oct 9, 2019
@alexisvisco
Copy link
Contributor Author

@kakkoyun Hey, I think I should do another pull request for the FTP part because the library used does not provide any ftp support, SFTP is a protocol to send and retrieve files over ssh.

Maybe you should look for an FTP library and tell me what you want.
In the meantime I correct the codebeat errors

@alexisvisco alexisvisco changed the title sftp: add structure layout of the backend feature: sftp backend Oct 9, 2019
@alexisvisco alexisvisco changed the title feature: sftp backend feature: backend sftp Oct 9, 2019
@alexisvisco
Copy link
Contributor Author

alexisvisco commented Oct 9, 2019

You can merge 🙆‍♂

@kakkoyun
Copy link
Contributor

kakkoyun commented Oct 9, 2019

@alexisvisco Thanks a lot for the work. I will thoroughly review it, please be patient, it won't take long. In the meantime, feel free to create another PR for FTP

Copy link
Contributor

@kakkoyun kakkoyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thanks a lot for your contributions. Quality of code is great.
However, we need to add automated tests and documentation to accept it.
Especially we need examples, please check what we already have.

I have also pointed out some minor issues. Please take a look at those as well.

cache/backend/backend.go Show resolved Hide resolved
cache/backend/backend.go Outdated Show resolved Hide resolved
cache/backend/backend.go Outdated Show resolved Hide resolved
cache/backend/backend.go Outdated Show resolved Hide resolved
cache/backend/backend.go Outdated Show resolved Hide resolved
plugin/plugin.go Show resolved Hide resolved
cache/backend/sftp_test.go Outdated Show resolved Hide resolved
@kakkoyun
Copy link
Contributor

@alexisvisco Have you made any progress? Do you need help? Just ping me if you need anything.

@alexisvisco
Copy link
Contributor Author

Hi I have done the test and add a docker-compose service. Unfortunately the continuous integration test does not work and I don't really have time to dive into your drone configuration.

Could you please test manually the sftp config with a drone test ? I have no more time to doing this PR and it's almost finished, just test the config and add details in the https://github.com/meltwater/drone-cache/blob/master/DOCS.md file.

Sorry for the inconvenience.

@kakkoyun
Copy link
Contributor

@alexisvisco Thank you very much @alexisvisco, it's all good. I can take it from here. Good work.

To be able to push to this branch directly, could you give me access on your forked repo?

@alexisvisco

This comment has been minimized.

@kakkoyun kakkoyun removed the WIP Work in progress label Oct 20, 2019
Copy link
Contributor

@kakkoyun kakkoyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is green now. Thanks a lot, @alexisvisco.
I have discovered a couple of issues; for those, I'll send a follow-up PR.
This is good to go for now.

@kakkoyun kakkoyun merged commit 43bce02 into meltwater:master Oct 20, 2019
@kakkoyun kakkoyun deleted the feature/backend-sftp branch October 20, 2019 11:57
This was referenced Oct 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
storage-backend New storage backends
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants