Skip to content
This repository has been archived by the owner on Dec 16, 2020. It is now read-only.

Error with logs handler caused a panic #70

Open
mangobowl opened this issue Jun 18, 2020 · 8 comments
Open

Error with logs handler caused a panic #70

mangobowl opened this issue Jun 18, 2020 · 8 comments

Comments

@mangobowl
Copy link

Expected Behaviour

Faas-swarm works fine.

Current Behaviour

Faas-swarm crashed soon after starting.

Possible Solution

No solution has been found yet (not familiar with golang).

Steps to Reproduce (for bugs)

  1. Follow the instructions on the official website to deploy openfaas on the swarm cluster.
  2. Deploy a few of our own business functions.
  3. Faas-swarm crashes and restarts repeatedly, the service log is as follows:
2020/06/18 03:07:20 Docker API version: 1.39, 18.09.4
2020/06/18 03:07:20 HTTP Read Timeout: 5m5s
2020/06/18 03:07:20 HTTP Write Timeout: 5m5s
2020/06/18 03:07:20 Basic authentication: true
2020/06/18 03:07:21 ReplicaReader - reading function: 1230-1229-debug
panic: runtime error: slice bounds out of range

goroutine 11 [running]:
github.com/openfaas/faas-swarm/handlers.parseLogStream(0x91f280, 0xc0000be300, 0xc00042e2b6, 0xd, 0xc0003ac2a0, 0x91d100, 0xc0003609c0)
        /go/src/github.com/openfaas/faas-swarm/handlers/logs.go:86 +0x780
created by github.com/openfaas/faas-swarm/handlers.LogRequester.Query
        /go/src/github.com/openfaas/faas-swarm/handlers/logs.go:65 +0x226

Context

Your Environment

  • FaaS-CLI version ( Full output from: faas-cli version ):
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|

CLI:
 commit:  b7a67fe8d6a02aef35caae615ba56333e7337bfe
 version: 0.9.5

faas-swarm version: 0.8.5

  • Docker version docker version (e.g. Docker 17.0.05 ):
Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.11.13
 Git commit:        039a7df9ba
 Built:             Wed Sep  4 16:57:28 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.13
  Git commit:       039a7df
  Built:            Wed Sep  4 16:19:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
  • Are you using Docker Swarm or Kubernetes (FaaS-netes)?
    Docker Swarm
  • Operating System and version (e.g. Linux, Windows, MacOS):
    Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-142-generic x86_64)
  • Link to your project or a code example to reproduce issue:
@alexellis
Copy link
Member

Hi @mangobowl,

Thanks for your interest in OpenFaaS

For us to be able to help you, we will need either the functions you are deploying, or similar ones where you can re-create the issue.

faas-swarm hasn't been changed significantly for quite some time, so I suspect this has something to do with the naming of your functions or perhaps something else specific to your environment.

If you want a quick resolution, I'd suggest you use OpenFaaS on Kubernetes - with k3s or a managed service like AKS, GKE etc. This is what we recommend for business / work and production use and what we actively support. See the note here -> https://docs.openfaas.com/deployment/

Alex

@alexellis
Copy link
Member

/set title: Error with logs handler caused a panic

@derek derek bot changed the title swarm crashed soon after starting Error with logs handler caused a panic Jun 18, 2020
@alexellis
Copy link
Member

Specifically, it looks like you are trying to access logs from the call-stack, is that right? Did you run faas-cli logs immediately after deploying? cc @LucasRoesler

@LucasRoesler
Copy link
Member

that is really weird, the line that panics is here
https://github.com/openfaas/faas-swarm/blob/master/handlers/logs.go#L86

which suggests that the logs stream was completely empty or a partial load of the logs that doesn't contain the full docker log prefix.

Perhaps the cleanest way to change this is to try and reuse their StdCopy implementation to get stderr and stdout into a new reader.

I have an idea on how to handle it

@mangobowl
Copy link
Author

@alexellis @LucasRoesler
Because there are too many functions in my production environment, it took a long time to troubleshoot the problem, sorry for the delay.

This problem is related to a special function log. The steps to reproduce are as follows:

  1. Deployment the function 1213-test0608(The image in docker hub is mangobowl/debug-function:debug)
  2. Invoke the function
  3. Get the function log by the api /system/logs?name=1213-test0608, then the faas-swarm will crash.

The full docker json-file log of the function is as follows:
4cf1fbee11253f846dcc8613403262e4c534279fc1268704994ca5c7420a678b-json.log

The api response will terminate at line 127:

{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.093038594Z","text":"Forking - python [index.py]"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.093354612Z","text":"2020/06/23 07:59:04 Started logging stderr from function."}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.093391118Z","text":"2020/06/23 07:59:04 Started logging stdout from function."}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094015443Z","text":"2020/06/23 07:59:04 OperationalMode: http"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094192224Z","text":"2020/06/23 07:59:04 Timeouts: read: 5m0s, write: 5m0s hard: 5m0s."}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094219234Z","text":"2020/06/23 07:59:04 Listening on port: 8080"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094241148Z","text":"2020/06/23 07:59:04 Writing lock-file to: /tmp/.lock"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T07:59:04.094247609Z","text":"2020/06/23 07:59:04 Metrics listening on port: 8081"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836543308Z","text":"2020/06/23 08:00:03 stdout: args: ImmutableMultiDict([])"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836595126Z","text":"2020/06/23 08:00:03 stdout: data: b''"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836608521Z","text":"2020/06/23 08:00:03 stdout: headers: Content-Type: application/json"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836616671Z","text":"2020/06/23 08:00:03 stdout: Content-Length: 0"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.83662435Z","text":"2020/06/23 08:00:03 stdout: Host: 1213-test0608:8080"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836632386Z","text":"2020/06/23 08:00:03 stdout: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 OPR/68.0.3618.173"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836643619Z","text":"2020/06/23 08:00:03 stdout: Accept: application/json, text/plain, */*"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836654495Z","text":"2020/06/23 08:00:03 stdout: Accept-Encoding: gzip, deflate"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836662488Z","text":"2020/06/23 08:00:03 stdout: Accept-Language: zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836671071Z","text":"2020/06/23 08:00:03 stdout: Cookie: monitorCustomerKey=5a0f0cd7-c748-4841-be8b-7269b8b99276-1590409358674; ajs_anonymous_id=%224b72111e-907f-4751-be00-6514a2f92809%22; _ga=GA1.2.1466632365.1590558477; NX-ANTI-CSRF-TOKEN=0.19181169709565116; NXSESSIONID=99302d0b-d5ab-449b-93d0-ee08706fdfe8; _gid=GA1.2.890040766.1592893980; csrftoken=LmvUCRceTv9dBCX3ivfWwe2ikikwpwHd42URP4X1haeZeLFWMm2gzHSAjv6i9dIh; sessionid=8ovt402pinpqvsln462ojhm8g77zqfil"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836681754Z","text":"2020/06/23 08:00:03 stdout: Origin: http://nexus.hiktest.com:18080"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836691391Z","text":"2020/06/23 08:00:03 stdout: Referer: http://nexus.hiktest.com:18080/ui/"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836699264Z","text":"2020/06/23 08:00:03 stdout: X-Call-Id: cedc3de5-96a2-4140-a336-2b44f80e1cb6"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836706689Z","text":"2020/06/23 08:00:03 stdout: X-Forwarded-For: 10.65.143.143:59798"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836713597Z","text":"2020/06/23 08:00:03 stdout: X-Forwarded-Host: nexus.hiktest.com:18080"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836720863Z","text":"2020/06/23 08:00:03 stdout: X-Start-Time: 1592899203820666171"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836728198Z","text":"2020/06/23 08:00:03 stdout:"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836735333Z","text":"2020/06/23 08:00:03 stdout:"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.836742508Z","text":"2020/06/23 08:00:03 stdout: form: ImmutableMultiDict([])"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.83674973Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 0"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.846679046Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 1"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.856989128Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 2"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.867285774Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 3"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.877715103Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 4"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.888026179Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 5"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.898369725Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 6"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.908723221Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 7"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.919067403Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 8"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.929382731Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 9"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.939759512Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 10"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.950042229Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 11"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.96043778Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 12"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.970743751Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 13"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.981130142Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 14"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:03.991489161Z","text":"2020/06/23 08:00:03 stdout: /home/app/function/handler.py[line:18] - INFO: 15"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.001823858Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 16"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.012123535Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 17"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.022488806Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 18"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.032778573Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 19"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.043156302Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 20"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.053537687Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 21"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.06384826Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 22"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.074121599Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 23"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.084547571Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 24"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.094843591Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 25"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.105151356Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 26"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.115524652Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 27"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.12575426Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 28"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.136141716Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 29"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.146465667Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 30"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.156823622Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 31"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.167119495Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 32"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.177464524Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 33"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.187849047Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 34"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.198082664Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 35"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.208420074Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 36"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.218838849Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 37"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.229138286Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 38"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.239421116Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 39"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.249788228Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 40"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.260134189Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 41"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.270443818Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 42"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.280824262Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 43"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.29115372Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 44"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.301498122Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 45"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.311828915Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 46"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.322141878Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 47"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.332510622Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 48"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.34286626Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 49"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.353182707Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 50"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.363507019Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 51"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.37389845Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 52"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.38413531Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 53"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.39454411Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 54"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.404923711Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 55"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.415259059Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 56"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.42557142Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 57"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.435952609Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 58"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.446289249Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 59"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.4566382Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 60"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.466905115Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 61"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.477119001Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 62"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.487382285Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 63"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.497769548Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 64"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.508079199Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 65"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.5184419Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 66"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.528722073Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 67"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.539104804Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 68"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.549429336Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 69"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.559708485Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 70"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.570090412Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 71"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.580416997Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 72"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.590738452Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 73"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.601024329Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 74"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.611344263Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 75"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.621618129Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 76"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.631950804Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 77"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.642345212Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 78"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.652622042Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 79"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.662947679Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 80"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.673355996Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 81"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.683650997Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 82"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.69391356Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 83"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.704220725Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 84"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.714611659Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 85"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.724945149Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 86"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.735180053Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 87"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.745561638Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 88"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.755843638Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 89"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.766159469Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 90"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.776533527Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 91"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.786923126Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 92"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.797226723Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 93"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.807511983Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 94"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.817872731Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 95"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.828284132Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 96"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.838588434Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 97"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.848903252Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 98"}
{"name":"1213-test0608","namespace":"","instance":"hidmorzltv8gif41qwnjgvgs8","timestamp":"2020-06-23T08:00:04.85923018Z","text":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 99"}

The function yml config is as follows:
1213-test0608.zip

@LucasRoesler
Copy link
Member

I don't understand what is special about line 127

{"log":"2020/06/23 08:00:04 stdout: /home/app/function/handler.py[line:18] - INFO: 99\n","stream":"stdout","time":"2020-06-23T08:00:04.85923018Z"}

The API response has a default timeout, is it possible that it terminates there just because of the request timeout? Can you share the fully cli command you used or the curl request you used?

@ag2uki
Copy link

ag2uki commented Sep 15, 2020

I got the similar issue. faas-swarm was getting crash when I tried to get the logs using faas-cli logs. The logs is similar with the first post. #70 (comment)

My workaround was trying to remove the function and then redeploy it.
$ docker service rm function-xyz
$ faas-cli deploy -f function-xyz.yml

And faas-cli logs works as expected then. And also, faas-swarm was not getting crash anymore.

@alexellis
Copy link
Member

Not sure how removing the function could prevent a crash. Can either of you give us full reproduction instructions?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@LucasRoesler @alexellis @ag2uki @mangobowl