-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vendor: Update terraform/consul dependency #23601
Conversation
/cc @kayrus @radeksimko |
It would be cleaner to get #23510 in separately before this |
e5b411f
to
9cb4cf3
Compare
9cb4cf3
to
2478ff7
Compare
@danieldreier any news on this? |
@apparentlymart please take a look at this and #23510. The currently vendored consul version is ancient. |
2478ff7
to
1e420cc
Compare
@danieldreier @apparentlymart the vendored Consul version is now 2.5 years old. |
09d02d6
to
63e2a3d
Compare
@danieldreier thank you for helping get #23510 in. This PR is now unblocked, and I've rebased it. |
@LorbusChris thanks for rebasing. backend/consul is now maintained by @hashicorp/consul so I'm going to request their review. I can't quite tell - does this imply any breaking changes or need for users to update configurations? It looks to me like a pure library version update but I wanted to confirm, e.g. if we lose support for old consul versions or something that we should give users a heads-up on in the release notes. |
@LorbusChris can you help me understand the purpose of updating to this version? I'm not sure how to prioritize this without other context, if this is a might-as-well, if you are blocked on this in some way, etc. |
With Go 1.13 we hit a version mismatch with a dependency of Consul when importing terraform. Regardless of that however, I think keeping dependencies up to date is important. Most imports here in this repo are from consul/api, and can stay unchanged. The testutil was moved from the main consul package into consul/sdk and the import was changed accordingly: https://github.com/hashicorp/terraform/pull/23601/files#diff-7948751d05c27ab0f28702c12bef2e96R10 |
63e2a3d
to
46eb894
Compare
Rebased again |
46eb894
to
91626e9
Compare
Rebased again |
91626e9
to
cdf6cc4
Compare
Rebased. @hashicorp/consul any chance you could have a look at this? |
@remilapeyre I did run that on the code before committing - are you saying it’s needed again? Thank you for the review :) |
0ab258e
to
1f9f5d5
Compare
rebased |
When running
|
It doesn't yield those changes on my system ( |
1f9f5d5
to
a840f3e
Compare
rebased again. @freddygv @remilapeyre PTAL |
If it makes things easier to review, I can split the vendor dir changes into a separate commit. Please let me know. |
a840f3e
to
bc931ff
Compare
rebased |
Thanks for opening and rebasing this PR, @LorbusChris! Sorry it's taking so long to get to, but we're looking at it now. I'm a little concerned by the number of core dependency upgrades that bumping Consul is forcing on us. My plan is to pull some of those out into separate PRs so that they can be tested and reviewed separately. If those PRs end up merged, I expect there might be conflicts here—but please don't feel that you have rebase until that process is complete. |
Updates github.com/mattn/go-colorable to v0.1.7, and github.com/mattn/go-isatty to v0.0.12.
Updates consul/api to v1.7.0 and consul/sdk to v0.6.0 Fixes TestMain in `backend/remote-state/consul/backend.test` to work with new version of testutil imported from consul/sdk.
bc931ff
to
1151bf4
Compare
@alisdair thank you, that's fair. I've split up the big commit into smaller ones that update one dep each. Feel free to take them as a reference. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@LorbusChris Thank you for doing that, it's extremely helpful!
I've now reviewed all of the extra dependency updates and I'm satisfied that they should not introduce any problems. I need to manually test the Windows build to verify some changes to terminal output colourization, but that shouldn't take long.
However, I noticed when reviewing this and other Consul backend PRs that our CI system was not running the backend tests. I fixed that in #26221 (just merged now), and it looks like tests are failing on this branch. I think if you rebase you'll see the failures here, too. You should also be able to verify this locally:
TF_CONSUL_TEST=1 go test ./backend/remote-state/consul/
I think the issue is in the change to newConsulTestServer
, where we're passing a nil
pointer to the testing.TB
interface to satisfy the type signature of testutil.NewTestServerConfigT
. This results in a panic when later code attempts to use the callbacks on this value.
testutil.NewTestServerConfigT
is intended to be used with a *testing.T
value, which means that it has to be called from within a test. Unfortunately, naively changing the newConsulTestServer
helper to spawn a new server, calling it from within each test, and defer srv.Stop()
results in pipe: too many open files
errors on my workstation.
We can't implement testing.TB
ourselves because of a private field, and I'm not aware of any way of constructing a *testing.T
from inside TestMain
. I'm currently out of ideas. Hopefully you have an idea of how to move forward!
We can't merge this until the tests pass, so I'm marking this PR as "request changes" to make sure we don't accidentally merge it.
Hi, I'm on the Consul team so I figured I'd pop in and help out. I took at stab at the changes @alisdair described and I didn't have a file handle issue running tests locally. Perhaps the way I patched it up differed from your approach somehow? (This omits
|
@rboyer Thanks, I appreciate it! That's fundamentally the same patch I tried (I had just moved the error checking inside the helper). Trying your exact patch has the same result, although I just noticed that the first error is not "too many open files" but instead "api unavailable": $ TF_CONSUL_TEST=1 go test ./backend/remote-state/consul/
[INFO] freeport: blockSize 1500 too big for system limit 256. Adjusting...
[INFO] freeport: detected ephemeral port range of [49152, 65535]
2020/09/11 17:12:25 [INFO] created consul lock session 1791e017-8508-f989-7c57-7d678e6b485f
2020/09/11 17:12:25 [INFO] created consul lock session ec77b90d-09e7-9f57-a94c-aa53fdbec0e6
2020/09/11 17:12:25 [INFO] created consul lock session f5843fac-5347-1c74-9794-bf3624e3a2d2
2020/09/11 17:12:25 [INFO] created consul lock session 7f1453a8-709d-366e-5558-0bd6759ab99a
2020/09/11 17:12:25 [INFO] created consul lock session 40959846-5c2d-a0c8-5238-01e766f9b4e7
2020/09/11 17:12:26 [INFO] created consul lock session c44b6d66-b170-6af7-7845-c6ea8f1d2e65
2020/09/11 17:12:26 [INFO] created consul lock session c7aeef9b-389e-3df9-3869-527cd601e0f4
2020/09/11 17:12:27 [INFO] created consul lock session 79bf5a1b-6697-8cce-fe3b-72686be1e4cc
2020/09/11 17:12:31 [INFO] created consul lock session e35fad1c-0cd6-6347-7c23-2e976483cf76
2020/09/11 17:12:31 [INFO] created consul lock session c1901d0f-6142-80a8-bacc-58d5a48f1c18
2020/09/11 17:12:31 [INFO] created consul lock session f2091d39-d350-bcf9-2952-480e00305e81
2020/09/11 17:12:31 [INFO] created consul lock session d311ea65-f45a-25b3-f6b9-a9699ebf229c
2020/09/11 17:12:31 [INFO] created consul lock session 6c6ea870-96f8-9810-6d69-cc319e6d2135
--- FAIL: TestRemoteClient (2.03s)
server.go:252: CONFIG JSON: {"node_name":"node-05d8a142-cd02-931e-0c0b-8d29d8928cd9","node_id":"05d8a142-cd02-931e-0c0b-8d29d8928cd9","performance":{"raft_multiplier":1},"bootstrap":true,"server":true,"data_dir":"/var/folders/d6/3gfl83ks03sf7j34c0c9125r0000gn/T/TestRemoteClient181070277/data","segments":null,"disable_update_check":true,"log_level":"warn","bind_addr":"127.0.0.1","addresses":{},"ports":{"dns":11790,"http":11791,"https":11792,"serf_lan":11793,"serf_wan":11794,"server":11795},"acl":{"tokens":{}},"connect":{"ca_config":{"cluster_id":"11111111-2222-3333-4444-555555555555"},"enabled":true}}
server.go:300: server stop failed with: exit status 1
client_test.go:24: err: api unavailable
--- FAIL: TestRemoteClient_gzipUpgrade (0.00s)
server.go:252: CONFIG JSON: {"node_name":"node-421dbe93-95b5-6f63-0141-d0bf1938a241","node_id":"421dbe93-95b5-6f63-0141-d0bf1938a241","performance":{"raft_multiplier":1},"bootstrap":true,"server":true,"data_dir":"/var/folders/d6/3gfl83ks03sf7j34c0c9125r0000gn/T/TestRemoteClient_gzipUpgrade981698144/data","segments":null,"disable_update_check":true,"log_level":"warn","bind_addr":"127.0.0.1","addresses":{},"ports":{"dns":11796,"http":11797,"https":11798,"serf_lan":11799,"serf_wan":11800,"server":11801},"acl":{"tokens":{}},"connect":{"ca_config":{"cluster_id":"11111111-2222-3333-4444-555555555555"},"enabled":true}}
client_test.go:48: err: failed starting command: pipe: too many open files
--- FAIL: TestConsul_stateLock (0.00s)
server.go:252: CONFIG JSON: {"node_name":"node-ea858a21-cc04-a3a0-9096-ec903b47f211","node_id":"ea858a21-cc04-a3a0-9096-ec903b47f211","performance":{"raft_multiplier":1},"bootstrap":true,"server":true,"data_dir":"/var/folders/d6/3gfl83ks03sf7j34c0c9125r0000gn/T/TestConsul_stateLock694110271/data","segments":null,"disable_update_check":true,"log_level":"warn","bind_addr":"127.0.0.1","addresses":{},"ports":{"dns":11802,"http":11803,"https":11804,"serf_lan":11805,"serf_wan":11806,"server":11807},"acl":{"tokens":{}},"connect":{"ca_config":{"cluster_id":"11111111-2222-3333-4444-555555555555"},"enabled":true}}
client_test.go:89: err: failed starting command: pipe: too many open files
--- FAIL: TestConsul_destroyLock (0.00s)
server.go:252: CONFIG JSON: {"node_name":"node-360a0b1d-ef92-8742-0dfb-11ffd1596483","node_id":"360a0b1d-ef92-8742-0dfb-11ffd1596483","performance":{"raft_multiplier":1},"bootstrap":true,"server":true,"data_dir":"/var/folders/d6/3gfl83ks03sf7j34c0c9125r0000gn/T/TestConsul_destroyLock530247314/data","segments":null,"disable_update_check":true,"log_level":"warn","bind_addr":"127.0.0.1","addresses":{},"ports":{"dns":11808,"http":11809,"https":11810,"serf_lan":11811,"serf_wan":11812,"server":11813},"acl":{"tokens":{}},"connect":{"ca_config":{"cluster_id":"11111111-2222-3333-4444-555555555555"},"enabled":true}}
client_test.go:118: err: failed starting command: pipe: too many open files
--- FAIL: TestConsul_lostLock (0.00s)
server.go:252: CONFIG JSON: {"node_name":"node-3c6c2ad5-3ef8-7b98-1cd7-e0f8e226ae37","node_id":"3c6c2ad5-3ef8-7b98-1cd7-e0f8e226ae37","performance":{"raft_multiplier":1},"bootstrap":true,"server":true,"data_dir":"/var/folders/d6/3gfl83ks03sf7j34c0c9125r0000gn/T/TestConsul_lostLock274379977/data","segments":null,"disable_update_check":true,"log_level":"warn","bind_addr":"127.0.0.1","addresses":{},"ports":{"dns":11814,"http":11815,"https":11816,"serf_lan":11817,"serf_wan":11818,"server":11819},"acl":{"tokens":{}},"connect":{"ca_config":{"cluster_id":"11111111-2222-3333-4444-555555555555"},"enabled":true}}
client_test.go:161: err: failed starting command: pipe: too many open files
--- FAIL: TestConsul_lostLockConnection (0.00s)
server.go:252: CONFIG JSON: {"node_name":"node-f91a0030-9f66-3d89-2fe2-2934c304601c","node_id":"f91a0030-9f66-3d89-2fe2-2934c304601c","performance":{"raft_multiplier":1},"bootstrap":true,"server":true,"data_dir":"/var/folders/d6/3gfl83ks03sf7j34c0c9125r0000gn/T/TestConsul_lostLockConnection242266004/data","segments":null,"disable_update_check":true,"log_level":"warn","bind_addr":"127.0.0.1","addresses":{},"ports":{"dns":11820,"http":11821,"https":11822,"serf_lan":11823,"serf_wan":11824,"server":11825},"acl":{"tokens":{}},"connect":{"ca_config":{"cluster_id":"11111111-2222-3333-4444-555555555555"},"enabled":true}}
client_test.go:214: err: failed starting command: pipe: too many open files
FAIL
FAIL github.com/hashicorp/terraform/backend/remote-state/consul 11.543s
FAIL
$ This is on macOS with Consul 1.7.4., in case that matters. |
You might want to check out what your system max file handle value is set to and possibly raise it. Usually on my linux machines I bump it to a huge number so these artificial limits don't get awkwardly in the way across the board.
|
We recently merged hashicorp/consul#8647 which should address the |
I'm cleaning out my backlog and this has no priority for me any longer. I'll leave the branch up, feel free to take the code as a reference. |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
Updating to the current version of consul. The version currently used and vendored in this repository is outdated.
Some other dependencies were also updated when running
go mod tidy && go mod vendor