Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Empire is stuck on a specific release #904

Closed
mwildehahn opened this issue Jun 24, 2016 · 5 comments
Closed

Empire is stuck on a specific release #904

mwildehahn opened this issue Jun 24, 2016 · 5 comments

Comments

@mwildehahn
Copy link
Contributor

I unset a config variable and now empire is stuck on that release.

The logs from the controller say the request completed successfully, but there are no updates making it to the CF stack.
@mwildehahn
Copy link
Contributor Author

mwildehahn commented Jun 24, 2016 •
edited
Loading

Enabled aws.debug, here are the logs:

t=2016-06-24T21:35:22+0000 lvl=info msg=request.start request_id=b7d00afc-2ae1-4d46-8c19-26383d8028a2 method=POST path=/apps/luno-web/deploys
t=2016-06-24T21:35:22+0000 lvl=info msg=authenticated request_id=b7d00afc-2ae1-4d46-8c19-26383d8028a2 user=mhahn
t=2016-06-24T21:35:23+0000 lvl=info msg=PullImage request_id=b7d00afc-2ae1-4d46-8c19-26383d8028a2 registry= repository=lunohq/luno-web tag=f6453809489c693c73a56110ce427b48de4c11a8
2016/06/24 21:35:23 DEBUG: Request s3/PutObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
PUT http://dev-empiredaemon-templatebucket-5f11dcrpuhpw.s3-us-west-2.amazonaws.com/dev/fa832979-9ace-4009-b22c-c351650b3bc7/9539719758b7c15a13de503b53a6b55838eff036 HTTP/1.1
Host: dev-empiredaemon-templatebucket-5f11dcrpuhpw.s3-us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.26 (go1.5.3; linux; amd64)
Content-Length: 4149
Authorization: ****
Content-Type: application/json
X-Amz-Content-Sha256: 4c006073019b251c644ca9221bb5b7acb6b047a9b36679d39263c099c798c55f
X-Amz-Date: 20160624T213523Z
X-Amz-Security-Token: ****
Accept-Encoding: gzip


-----------------------------------------------------
2016/06/24 21:35:23 DEBUG: Response s3/PutObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK

-----------------------------------------------------
2016/06/24 21:35:23 DEBUG: Request cloudformation/ValidateTemplate Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: cloudformation.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.26 (go1.5.3; linux; amd64)
Content-Length: 230
Authorization: ****
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Content-Sha256: 6d92830fcb6b10bbb9b5b0512180ef9fea22e0943db8037e9287d2264beae963
X-Amz-Date: 20160624T213523Z
X-Amz-Security-Token: ****
Accept-Encoding: gzip


-----------------------------------------------------
2016/06/24 21:35:23 DEBUG: Response cloudformation/ValidateTemplate Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 712
Content-Type: text/xml
Date: Fri, 24 Jun 2016 21:35:22 GMT
X-Amzn-Requestid: 8ecb3f8a-3a53-11e6-8006-c93cc26729a8


-----------------------------------------------------
2016/06/24 21:35:23 DEBUG: Request cloudformation/DescribeStacks Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: cloudformation.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.26 (go1.5.3; linux; amd64)
Content-Length: 63
Authorization: ****
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Content-Sha256: a9a3cf0127091c7416240be8c2f936e6452d52df33b763e545003efed87e3f72
X-Amz-Date: 20160624T213523Z
X-Amz-Security-Token: ****
Accept-Encoding: gzip


-----------------------------------------------------
2016/06/24 21:35:23 DEBUG: Response cloudformation/DescribeStacks Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/xml
Date: Fri, 24 Jun 2016 21:35:22 GMT
Vary: Accept-Encoding
X-Amzn-Requestid: 8ed9705c-3a53-11e6-8006-c93cc26729a8


-----------------------------------------------------
2016/06/24 21:35:23 DEBUG: Request cloudformation/DescribeStacks Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: cloudformation.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.26 (go1.5.3; linux; amd64)
Content-Length: 63
Authorization: ****
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Content-Sha256: a9a3cf0127091c7416240be8c2f936e6452d52df33b763e545003efed87e3f72
X-Amz-Date: 20160624T213523Z
X-Amz-Security-Token: ****
Accept-Encoding: gzip


-----------------------------------------------------
2016/06/24 21:35:23 DEBUG: Response cloudformation/DescribeStacks Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/xml
Date: Fri, 24 Jun 2016 21:35:23 GMT
Vary: Accept-Encoding
X-Amzn-Requestid: 8edf15ae-3a53-11e6-8006-c93cc26729a8


-----------------------------------------------------
2016/06/24 21:35:23 DEBUG: Request cloudformation/UpdateStack Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: cloudformation.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.26 (go1.5.3; linux; amd64)
Content-Length: 520
Authorization: ****
X-Amz-Content-Sha256: 272d387004bbba4ab553922b207e3f7c3aa459da649f74710ba513e726157304
X-Amz-Date: 20160624T213523Z
X-Amz-Security-Token: ****
Accept-Encoding: gzip


-----------------------------------------------------
2016/06/24 21:35:24 DEBUG: Response cloudformation/UpdateStack Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 382
Content-Type: text/xml
Date: Fri, 24 Jun 2016 21:35:23 GMT
X-Amzn-Requestid: 8ee38280-3a53-11e6-8006-c93cc26729a8


-----------------------------------------------------
2016/06/24 21:35:24 DEBUG: Request cloudformation/DescribeStacks Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: cloudformation.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.26 (go1.5.3; linux; amd64) Waiter
Content-Length: 63
Authorization: ****
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Content-Sha256: a9a3cf0127091c7416240be8c2f936e6452d52df33b763e545003efed87e3f72
X-Amz-Date: 20160624T213524Z
X-Amz-Security-Token: ****
Accept-Encoding: gzip

-----------------------------------------------------
2016/06/24 21:35:24 DEBUG: Request sns/Publish Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sns.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.26 (go1.5.3; linux; amd64)
Content-Length: 541
Authorization: ****
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Content-Sha256: 5c48d851e39d3d91465386a71857ce4cda5b7257878208f04c140d2cbd77c07f
X-Amz-Date: 20160624T213524Z
X-Amz-Security-Token: ****
Accept-Encoding: gzip


-----------------------------------------------------
2016/06/24 21:35:24 DEBUG: Response sns/Publish Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 294
Content-Type: text/xml
Date: Fri, 24 Jun 2016 21:35:23 GMT
X-Amzn-Requestid: aada240e-0be2-54b1-8428-6be9a23eb23c



-----------------------------------------------------

2016/06/24 21:35:24 DEBUG: Response cloudformation/DescribeStacks Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/xml
Date: Fri, 24 Jun 2016 21:35:23 GMT
Vary: Accept-Encoding
X-Amzn-Requestid: 8f40e4c5-3a53-11e6-8006-c93cc26729a8


-----------------------------------------------------

this ended up working, not sure if it was because setting the aws.debug flag led to a restart of the empire process.

@mwildehahn
Copy link
Contributor Author

hm, not sure what happened here. it jumped to the correct release eventually.

here is the release log:

v103    Jun 24 12:09  Unset FEATURE_FLAGS_WRITE_REPLIES config var (mhahn)
v104    Jun 24 12:14  Set DEBUG config var (mhahn)
v105    Jun 24 14:04  Deploy lunohq/luno-web:f6453809489c693c73a56110ce427b48de4c11a8 (mhahn)
v106    Jun 24 14:11  Deploy lunohq/luno-web:f6453809489c693c73a56110ce427b48de4c11a8 (mhahn)
v107    Jun 24 14:13  Deploy lunohq/luno-web:f6453809489c693c73a56110ce427b48de4c11a8 (mhahn)
v108    Jun 24 14:22  Set TEST_STUCK config var (mhahn)
v109    Jun 24 14:25  Unset TEST_STUCK config var (mhahn)
v110    Jun 24 14:35  Deploy lunohq/luno-web:f6453809489c693c73a56110ce427b48de4c11a8 (mhahn)

and cloudformation snapshot:

image

looks like the v110 deploy went through. @ejholmes could this be an issue with any of the locking code?

@mwildehahn
Copy link
Contributor Author

maybe this is related to the issue this fixes: #900?

@ejholmes
Copy link
Contributor

Yeah, this sounds a lot like #900 and/or #892. Restarting Empire would have fixed it since it would have released the advisory lock.

@mwildehahn
Copy link
Contributor Author

closing for now, we'll see if it crops up again now that #900 is merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ejholmes @mwildehahn