Add manta remote state backend #8830

watters · 2016-09-13T22:18:58Z

I tried to match what the other backends offered in terms of documentation and testing, but any/all feedback is welcome.

I've marked this [WIP] for now while I continue to exercise it, but I wanted to invite any feedback early on.

jen20 · 2016-09-14T08:14:37Z

Hi @watters! I strongly approve of this PR, obviously! I'll look at getting it tested and merged into Terraform shortly.

jen20 · 2016-09-14T09:15:56Z

Hi @watters - I've made a quick review of this, but can't figure out via the various layers of indirection which environment variables need to be set to make the tests run here? Can you shed some light on this (and maybe fix up the "MANTA_USER and friends" comment to reflect that 😆)? Thanks!

watters · 2016-09-14T12:42:46Z

@jen20 thanks so much for taking a look at this so soon!

I've updated the skip message in the tests to be explicit about all three Manta environment variables that must be set.

I also added a link to the Manta documentation for setting up the environment in the website documentation.

When running make test TEST=./state/remote TESTARGS=-v with the MANTA_USER, MANTA_KEY_ID and MANTA_URL environment variables set, the tests pass and I can see the calls to Manta being made when I inspect the traffic.

Let me know if there are any other changes you'd like to see.

jen20 · 2016-09-14T15:03:09Z

Hi @watters - those changes look good to me. Running the tests I'm still seeing "Error getting Manta credentials: An error occurred while parsing the key: asn1: structure error: length too large" however - the key is in the agent and available unencrypted on disk, though it's unclear where it would be loaded from? FWIW, the manta command line tools work correctly with the same environment variables set.

watters · 2016-09-14T15:39:10Z

I'll see if I can repro.

That error is coming from the underlying auth bits in github.com/joyent/gosign, which I didn't mess with—yet (as discussed elsewhere, the underlying libraries in play can be unwieldy).

keyName, which defaults to $HOME/.ssh/id_rsa currently cannot be overridden for the tests, which is probably something that needs solving.

…e and link to Manta docs for env setup

…kend - expose MANTA_KEY_MATERIAL environment variable (previously specified via keyName parameter) and make it required; no defaults - stop relying on some of the Joyent go libraries' underlying environment variable config magic (it doesn't appear to play nice if you have different values in MANTA_USER and SDC_ACCOUNT) - explicitly check for passphrase-protected SSH keys, which are currently unsupported, and generate a more helpful error (borrowed from Packer's solution to the same problem): https://github.com/mitchellh/packer/blob/master/common/ssh/key.go#L27 - update documentation to reflect these changes

watters · 2016-09-14T18:27:59Z

state/remote/manta.go

+			return nil, fmt.Errorf("Error reading key material from %s: %s",
+				keyMaterial, err)
+		} else {
+			block, _ := pem.Decode(keyBytes)


This was cribbed directly from https://github.com/mitchellh/packer/blob/master/common/ssh/key.go#L27

Yup, this is working now.

watters · 2016-09-14T18:41:45Z

@jen20 I was able to reproduce the error message you described when using a passphrase-protected 4096-bit SSH key. The underlying auth code chokes on passphrase-protected keys and currently only supports keys specified by file path or decrypted into an environment variable.

I've exposed the key path via environment the variable MANTA_KEY_MATERIAL, which will allow you to specify a key that can be used during the tests (previously it defaulted to ~/.ssh/id_rsa), but it must align with the MANTA_KEY_ID and, of course, be associated with your account.

Hopefully, this gets the tests passing for you. I'd already determined before submitting this PR that Joyent's Go library for Manta probably needed a re-write, and now I'm only even more convinced.

jen20 · 2016-09-14T19:20:59Z

That explains a lot - I have a separate unencrypted key for testing with HashiCorp's Joyent account rather than my personal (encrypted) key, so ~/.ssh/id_rsa producing that error message make sense. I'll test it with MANTA_KEY_MATERIAL set now. The Go library for Manta is a bit janky right now - I suspect it was written before the idioms for Go had become apparent. I'll look at merging this now though and we can sort that out in future.

jen20

LGTM!

jen20 · 2016-09-14T19:39:03Z

state/remote/manta.go

+			return nil, fmt.Errorf("Error reading key material from %s: %s",
+				keyMaterial, err)
+		} else {
+			block, _ := pem.Decode(keyBytes)


Yup, this is working now.

jen20 · 2016-09-14T19:39:26Z

state/remote/manta.go

+					"Failed to read key material '%s': no key found", keyMaterial)
+			}
+
+			if block.Headers["Proc-Type"] == "4,ENCRYPTED" {


This is great - we should roll it out into the Triton provider as well.

jen20 · 2016-09-14T19:40:59Z

Example test run:

=== RUN   TestMantaClient_impl
--- PASS: TestMantaClient_impl (0.00s)
=== RUN   TestMantaClient
2016/09/14 20:37:13 [DEBUG] New state was assigned lineage "aace983a-9729-4997-9ca0-121b4e2f698c"
2016/09/14 20:37:13 [TRACE] Preserving existing state lineage "aace983a-9729-4997-9ca0-121b4e2f698c"
--- PASS: TestMantaClient (7.28s)

jen20 · 2016-09-14T19:47:01Z

Hi @watters! I've run the tests and reviewed this and it seems fine. I've rebased to master and squashed some of the commits (leaving the vendoring separate). Finally, I've fixed up the documentation sidebar for remote state to include the Manta option. This has landed in 72a341b and b4eb63d, though GitHub sadly doesn't reflect that here! Thanks for your work!

watters · 2016-09-14T20:40:09Z

Thank you!

ghost · 2020-04-22T02:07:01Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

jen20 added enhancement core labels Sep 14, 2016

jen20 self-assigned this Sep 14, 2016

watters force-pushed the add-manta-remote-state branch from 23db975 to 7bedd2b Compare September 14, 2016 12:25

watters added 6 commits September 14, 2016 11:26

vendor gomanta

625f38d

add remote state provider backed by Joyent's Manta

a1573d3

add initial documentation of Manta remote state provider

1d08ab9

improve the default object name for Manta remote state backend

17af352

list all env vars needed by Manta remote state backend in skip messag…

fafe562

…e and link to Manta docs for env setup

watters force-pushed the add-manta-remote-state branch from 188ebc4 to 63b068d Compare September 14, 2016 18:26

watters commented Sep 14, 2016

View reviewed changes

jen20 approved these changes Sep 14, 2016

View reviewed changes

jen20 changed the title ~~[WIP] Add manta remote state backend~~ Add manta remote state backend Sep 14, 2016

jen20 closed this Sep 14, 2016

watters mentioned this pull request Sep 28, 2016

Remote State on Manta #6563

Closed

ghost locked and limited conversation to collaborators Apr 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add manta remote state backend #8830

Add manta remote state backend #8830

watters commented Sep 13, 2016

jen20 commented Sep 14, 2016

jen20 commented Sep 14, 2016

watters commented Sep 14, 2016

jen20 commented Sep 14, 2016 •

edited

Loading

watters commented Sep 14, 2016

watters Sep 14, 2016

jen20 Sep 14, 2016

watters commented Sep 14, 2016 •

edited

Loading

jen20 commented Sep 14, 2016

jen20 left a comment

jen20 Sep 14, 2016

jen20 Sep 14, 2016

jen20 commented Sep 14, 2016

jen20 commented Sep 14, 2016

watters commented Sep 14, 2016

ghost commented Apr 22, 2020

Add manta remote state backend #8830

Add manta remote state backend #8830

Conversation

watters commented Sep 13, 2016

jen20 commented Sep 14, 2016

jen20 commented Sep 14, 2016

watters commented Sep 14, 2016

jen20 commented Sep 14, 2016 • edited Loading

watters commented Sep 14, 2016

watters Sep 14, 2016

Choose a reason for hiding this comment

jen20 Sep 14, 2016

Choose a reason for hiding this comment

watters commented Sep 14, 2016 • edited Loading

jen20 commented Sep 14, 2016

jen20 left a comment

Choose a reason for hiding this comment

jen20 Sep 14, 2016

Choose a reason for hiding this comment

jen20 Sep 14, 2016

Choose a reason for hiding this comment

jen20 commented Sep 14, 2016

jen20 commented Sep 14, 2016

watters commented Sep 14, 2016

ghost commented Apr 22, 2020

jen20 commented Sep 14, 2016 •

edited

Loading

watters commented Sep 14, 2016 •

edited

Loading