provider/openstack: 409 Response on Member Create

[jtopjian]

This commit accounts for a 409 response when a LBaaS v2 member is
being created. Rather than error out, this should be considered a
pending state.

Fixes #10852

[jtopjian]

@mrantipyrine I think this should fix the problem. It's really hard for me to recreate the situation where this would happen, so I'm flying a little blind. Are you able to build from source with this patch and confirm? Let me know if you'd need some help.

[mrantipyrine]

Terraform v0.8.2-dev (93f600dbf765bc979813460260abf99bcb8f58a3+CHANGES)

So far so good but I made one changereturn member, "PENDING_CREATE", nil to return member, "PENDING_UPDATE", nil.

Prior to this change I was still seeing this still frequently:

* openstack_lb_member_v2.member.2: Error creating OpenStack LBaaSV2 member: Expected HTTP response code [201 202] when accessing [POST http://dev.neutron.cloud.net:9696/v2.0/lbaas/pools/d90a6b80-92ea-4478-9576-16dcc70b8511/members], but got 409 instead
{"NeutronError": {"message": "Invalid state PENDING_UPDATE of loadbalancer resource 50748818-e2f6-4306-b20a-0cdcecf79565", "type": "StateInvalid", "detail": ""}}

I'll do more testing in the morning.

[jtopjian]

Ah, if you want to change PENDING_CREATE to PENDING_UPDATE, you'll need to add it to the StateChangeConf as you did here.

I recommend just leaving the PENDING_CREATE status untouched. It's a bit confusing, but this resource is completely ignoring the status that the member is returning. There's a semi-good reason for that, related to how 409s are handled in general at the moment, and it'll eventually be fixed down the road.

Edit: err... I misread again. That's totally bizarre that you're seeing that error. Let me know if changing _CREATE to _UPDATE is indeed suppressing that error or if there was another modification that got slipped in.

Thanks for testing this, by the way :)

[mrantipyrine]

The problem still seems to be happening when the number of lb_members in a particular pool is > 4; anything <= 4 is working works without issue.

{"NeutronError": {"message": "Invalid state PENDING_UPDATE of loadbalancer resource 50748818-e2f6-4306-b20a-0cdcecf79565", "type": "StateInvalid", "detail": ""}}```

And no need to thank me for testing :)

[jtopjian]

hm... I wonder if that error is being triggered somewhere else. Can you post the debug output to a gist?

[mrantipyrine]

https://gist.github.com/mrantipyrine/a0a9c37be63b37dcf0cd05b5516f91c7

[mrantipyrine]

Thanks for looking into this so quickly :)

[jtopjian]

No problem at all.

The gist helped a lot -- the error is being triggered here and not in the StateChangeConf section, which makes sense.

I just pushed a new commit that I think catches the 409 in the right spot. This is the first time I've used the resource.Retry feature, so let's see how it works. :)

[mrantipyrine]

So far it looks to be working! I'm going to run 20-30 more build/destroys to see if the error pops up again.

[mrantipyrine]

I left a loop running overnight and came in to see the problem still occurring but it appears with less frequency

{"NeutronError": {"message": "Invalid state PENDING_UPDATE of loadbalancer resource 50748818-e2f6-4306-b20a-0cdcecf79565", "type": "StateInvalid", "detail": ""}}
* openstack_lb_member_v2.member.0: Error creating OpenStack LBaaSV2 member: Expected HTTP response code [201 202] when accessing [POST http://dev.neutron.cloud.net:9696/v2.0/lbaas/pools/c16af7f9-e9dc-4428-8dd1-8d766f399b6b/members], but got 409 instead

Additionally I am also seeing timeouts errors that I was not seeing before but this most certainly on my end

[jtopjian]

I wonder if the error is happening after 2 minutes of trying. If you'd like to do some more testing, maybe increase the time to 10 minutes and let it run for a while. Unfortunately it's not possible to dynamically specify a timeout -- it has to be hard-coded.

[mrantipyrine]

I've been running with a 5 minute timeout on a constant loop for the last 5 hours and have not seen the error since. It think it's safe to say this fixed the issue.

The one thing I forgot to mention is that older version of Neutron will return a 500 when in a PENDING_UPDATE state so this fix will not work. This is the patch that fixed said problem
https://review.openstack.org/gitweb?p=openstack/neutron-lbaas.git;a=commitdiff;h=3cf4a7cffd5f76b9f8a3eed649d04075714dcd9c

[jtopjian]

That's great news! I just pushed a final commit that should be good to go. If you'd like to give it one last shot, I'd really appreciate it.

[mrantipyrine]

I have the new changes built and will start testing in the next hour or so. :)

[mrantipyrine]

I've done 40+ builds/destroys and not a single one failed because of the PENDING state. I think we're good.

[jtopjian]

That's awesome news! Thank you very much for your help with this.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.