deploy: history update can fail if container starts slowly, makes superfluous header write #626

chuck-sys · 2019-10-15T04:39:02Z

Describe the bug
No such container error while executing inertia <webapp> up.

Below are the logs around the error.

:2019-10-15T03:04:44.288245314Z  ---> db3b423471db
R2019-10-15T03:04:44.288256356Z +Step 8/8 : CMD ["pipenv", "run", "launch"]
E2019-10-15T03:04:44.309796561Z  ---> Running in a3702fc38513
T2019-10-15T03:04:44.376285947Z -Removing intermediate container a3702fc38513
:2019-10-15T03:04:44.376301066Z  ---> 76dafa37d017
G2019-10-15T03:04:44.451185117Z  Successfully built 76dafa37d017
R2019-10-15T03:04:44.461362836Z +Successfully tagged rocket2_rocket2:latest
I2019-10-15T03:04:44.566736851Z container build (387456e0827) has stopped
82019-10-15T03:04:44.687517010Z rocket2 build successful
>2019-10-15T03:04:44.687535627Z Perparing rocket2 container...
92019-10-15T03:04:44.720801699Z rocket2 container created
62019-10-15T03:04:44.720817222Z Starting up rocket2...
�2019-10-15T03:04:45.036853677Z [error 500] failed to update container history following build: (failure fetching container metadata: Error: No such container: )
�2019-10-15T03:04:45.045902766Z 2019/10/15 03:04:45 http: superfluous response.WriteHeader call from github.com/go-chi/render.JSON (responder.go:104)
H2019-10-15T03:04:45.045977059Z [success 201] Project startup initiated!
S2019-10-15T03:04:46.695704835Z container rocket2_nginx_1 (4da9e6c7dac) has stopped
T2019-10-15T03:04:46.695721326Z container stoppage was unexpected, project is active
B2019-10-15T03:04:46.695988524Z Shutting down active containers...
>2019-10-15T03:04:46.701700951Z Stopping /rocket2_rocket2_1...
<2019-10-15T03:04:49.646784528Z Stopping /rocket2_nginx_1...
;2019-10-15T03:04:49.797845222Z Stopping /docker-compose...
S2019-10-15T03:04:50.017728273Z container rocket2_nginx_1 (4da9e6c7dac) has stopped
S2019-10-15T03:04:50.017743715Z container rocket2_nginx_1 (4da9e6c7dac) has stopped
U2019-10-15T03:04:50.017748600Z container rocket2_rocket2_1 (bb784f004e9) has stopped
S2019-10-15T03:04:50.017752936Z container rocket2_nginx_1 (4da9e6c7dac) has stopped
R2019-10-15T03:04:50.017757137Z container docker-compose (11079698074) has stopped

To Reproduce
Steps to reproduce the behavior:

Check out https://github.com/ubclaunchpad/rocket2
inertia init
inertia remote add aws-ec2 and fill in your informations (.pem, username, ip address of AWS EC2 instance)
inertia aws-ec2 init
Set up keys and webhooks on Github's side
inertia aws-ec2 up
See error

Environment (please complete the following information):

Client OS: Linux 5.3.1-arch1-1-ARCH
VPS OS: AWS EC2 Amazon Linux AMI 2018.03.0 (HVM), SSD Volume Type - ami-08d489468314a58df
Project Details: https://github.com/ubclaunchpad/rocket2
Version: latest

The text was updated successfully, but these errors were encountered:

bobheadxi · 2019-10-15T05:04:50Z

hey @cheukyin699 thanks for reporting this! looks like the fault is in a new feature added in #578 :

�2019-10-15T03:04:45.036853677Z [error 500] failed to update container history following build: (failure fetching container metadata: Error: No such container: )

This change has not been incorporated in our latest release

It looks like you are using Version: latest from your environment details, which is highly not recommended in our docs: https://inertia.ubclaunchpad.com/#inertia-release-streams

I would recommend making sure you have the latest stable version in inertia --version, and verify that you have the same version specified in your inertia.toml. If inertia --version doesn't output a proper version, please use one of the precompiled binaries in the release page^^

bobheadxi · 2019-10-15T05:37:07Z

If you must build from source, you should git checkout v0.5.3 && make cli RELEASE=v0.5.3

bobheadxi · 2019-10-15T05:41:28Z

However, I'm not sure if the bug here (history update failure) is actually causing the application to fail - inertia notices the death of a container, nginx:

S2019-10-15T03:04:46.695704835Z container rocket2_nginx_1 (4da9e6c7dac) has stopped
T2019-10-15T03:04:46.695721326Z container stoppage was unexpected, project is active

which begins the shutdown procedure, this was reported by @rwblickhan and was fixed in v0.5.3 with #608, though this hasn't been forward-ported to master yet. so ya use v0.5.3 for now 😄

chuck-sys · 2019-10-15T06:14:44Z

Yes. The thing that is probably causing the app to fail is the fact that nginx letsencrypt was not set up at all.

bobheadxi · 2019-10-15T06:21:55Z

Let me know if switching to v0.5.3 and fiddling around with the nginx container fixes it! Paging @rwblickhan because I feel like this is related to his issue but I didn't document it very clearly in #607 😅

rwblickhan · 2019-10-15T06:43:59Z

The issue fixed by #608 was that we use a Lets Encrypt container to request SSL certs but it exits as soon as the certs are setup, causing Inertia to kill everything. But that container should be called letsencrypt - I think there might be an issue here with the nginx config that's causing it to crash on launch. @cheukyin699 could you run that again and tail the logs for the nginx container specifically?

chuck-sys · 2019-10-15T13:16:50Z

Just finished setting up letsencrypt and tried inertia aws-ec2 up. Got the following after the docker container build:

R2019-10-15T13:12:43.394834725Z +Successfully tagged rocket2_rocket2:latest
I2019-10-15T13:12:43.473924861Z container build (b94af34e685) has stopped
82019-10-15T13:12:43.576420254Z rocket2 build successful
>2019-10-15T13:12:43.576442876Z Perparing rocket2 container...
92019-10-15T13:12:43.637142338Z rocket2 container created
62019-10-15T13:12:43.637168759Z Starting up rocket2...
H2019-10-15T13:12:43.949424513Z [SUCCESS 201] Project startup initiated!
S2019-10-15T13:12:45.347134316Z container rocket2_nginx_1 (88d2c78f533) has stopped
T2019-10-15T13:12:45.347159982Z container stoppage was unexpected, project is active
B2019-10-15T13:12:45.348783681Z Shutting down active containers...
>2019-10-15T13:12:45.475888928Z Stopping /rocket2_rocket2_1...
<2019-10-15T13:12:48.129315299Z Stopping /rocket2_nginx_1...
;2019-10-15T13:12:48.658230701Z Stopping /docker-compose...
�2019-10-15T13:12:48.669459199Z error shutting down other active containers: Error response from daemon: No such container: e3e1ea457c1607e36ebe09befd10e14d5c361d5f6dd8df4e252cdc2ddfde65ea
S2019-10-15T13:12:48.669513529Z container rocket2_nginx_1 (88d2c78f533) has stopped
S2019-10-15T13:12:48.708612496Z container rocket2_nginx_1 (88d2c78f533) has stopped
U2019-10-15T13:12:48.726852716Z container rocket2_rocket2_1 (2782bde5b9a) has stopped
R2019-10-15T13:12:48.726895194Z container docker-compose (e3e1ea457c1) has stopped
S2019-10-15T13:12:48.732000971Z container rocket2_nginx_1 (88d2c78f533) has stopped

Something to do with nginx??

bobheadxi · 2019-10-15T15:11:24Z

Is this on v0.5.3? If it is you might have to add the ignore death configuration in the inertia setting as described in #608

chuck-sys · 2019-10-16T00:55:09Z

Interesting. After having both client and remote on v0.5.3, my inertia.remotes aren't read, and so I had to re-init them and re-add them. After entering the information, I got the following error:

Port 4303 will be used as the daemon port.
Port 22 will be used as the SSH port.
Run 'inertia remote add' with the -p flag to set a custom Daemon port
of the -ssh flag to set a custom SSH port.
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/ubclaunchpad/inertia/cfg.(*Config).AddRemote(...)
	/home/csys/go/src/github.com/ubclaunchpad/inertia/cfg/config.go:98
github.com/ubclaunchpad/inertia/cmd/inpututil.AddRemoteWalkthrough(0x1046620, 0xc000010010, 0xc0000c86e0, 0x7ffc23c6c4cd, 0x7, 0xe57d38, 0x4, 0xe57655, 0x2, 0xc0000290f8, ...)
	/home/csys/go/src/github.com/ubclaunchpad/inertia/cmd/inpututil/input.go:77 +0xbeb
github.com/ubclaunchpad/inertia/cmd/remote.(*RemoteCmd).attachAddCmd.func1(0xc0001c7180, 0xc00006ff80, 0x1, 0x1)
	/home/csys/go/src/github.com/ubclaunchpad/inertia/cmd/remote/remote.go:91 +0x212
github.com/spf13/cobra.(*Command).execute(0xc0001c7180, 0xc00006ff60, 0x1, 0x1, 0xc0001c7180, 0xc00006ff60)
	/home/csys/go/src/github.com/spf13/cobra/command.go:833 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xc0001c6000, 0x6, 0xc00000f7a0, 0x4072ff)
	/home/csys/go/src/github.com/spf13/cobra/command.go:917 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	/home/csys/go/src/github.com/spf13/cobra/command.go:867
main.main()
	/home/csys/go/src/github.com/ubclaunchpad/inertia/main.go:14 +0x4e

bobheadxi · 2019-10-16T03:07:28Z

Yeah the config format has been significantly reworked from v0.5.x -> v0.6.x - they are not really compatible in any way. You'll have to do a complete re-setup, starting with deleting all inertia configuration and starting with inertia --init or whatever the equivalent was for v0.5.x.

Sorry for the 0ver abuse, but delivering on v0.6.0 is quite a long haul 😓

bobheadxi · 2019-10-16T03:12:03Z

I've created #627 to hopefully dissuade anyone else from building from source, sorry about that!

chuck-sys · 2019-10-16T15:45:28Z

Successfully deployed onto rocket2.ubclaunchpad! The issue I was running into had nothing to do with inertia. I just had to install nginx (bc it was complaining about lack of nginx files).

bobheadxi · 2019-10-16T16:43:18Z

Ah I see 👍 I'll leave this ticket open for now since it does demonstrate another issue, thanks!

chuck-sys added the type: bug 🐛 something's wrong! label Oct 15, 2019

bobheadxi added this to the v0.6.0 milestone Oct 15, 2019

bobheadxi changed the title ~~No such container when executing inertia webapp up~~ deploy: history update can fail if container starts slowly, makes superfluous header write Oct 15, 2019

bobheadxi mentioned this issue Oct 15, 2019

config: allow inertia to not panic and kill everything on container death #607

Closed

3 tasks

bobheadxi self-assigned this Oct 15, 2019

bobheadxi mentioned this issue Oct 16, 2019

docs: discourage use of go get or building from source #627

Closed

bobheadxi mentioned this issue Mar 5, 2020

daemon: log history update as non-critical failure #641

Merged

bobheadxi closed this as completed in #641 Mar 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deploy: history update can fail if container starts slowly, makes superfluous header write #626

deploy: history update can fail if container starts slowly, makes superfluous header write #626

chuck-sys commented Oct 15, 2019

bobheadxi commented Oct 15, 2019 •

edited

Loading

bobheadxi commented Oct 15, 2019

bobheadxi commented Oct 15, 2019 •

edited

Loading

chuck-sys commented Oct 15, 2019

bobheadxi commented Oct 15, 2019

rwblickhan commented Oct 15, 2019

chuck-sys commented Oct 15, 2019

bobheadxi commented Oct 15, 2019 •

edited

Loading

chuck-sys commented Oct 16, 2019

bobheadxi commented Oct 16, 2019 •

edited

Loading

bobheadxi commented Oct 16, 2019

chuck-sys commented Oct 16, 2019

bobheadxi commented Oct 16, 2019

deploy: history update can fail if container starts slowly, makes superfluous header write #626

deploy: history update can fail if container starts slowly, makes superfluous header write #626

Comments

chuck-sys commented Oct 15, 2019

bobheadxi commented Oct 15, 2019 • edited Loading

bobheadxi commented Oct 15, 2019

bobheadxi commented Oct 15, 2019 • edited Loading

chuck-sys commented Oct 15, 2019

bobheadxi commented Oct 15, 2019

rwblickhan commented Oct 15, 2019

chuck-sys commented Oct 15, 2019

bobheadxi commented Oct 15, 2019 • edited Loading

chuck-sys commented Oct 16, 2019

bobheadxi commented Oct 16, 2019 • edited Loading

bobheadxi commented Oct 16, 2019

chuck-sys commented Oct 16, 2019

bobheadxi commented Oct 16, 2019

bobheadxi commented Oct 15, 2019 •

edited

Loading

bobheadxi commented Oct 15, 2019 •

edited

Loading

bobheadxi commented Oct 15, 2019 •

edited

Loading

bobheadxi commented Oct 16, 2019 •

edited

Loading