Can't collect OOM events

[pkrolikowski]

Hello,

I'm trying to use cAdvisor to catch OOM events but I failed (probably I'm doing something wrong).
Here's my setup:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04 LTS
Release:    16.04
Codename:   xenial

docker version
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:11:10 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:11:10 2016
 OS/Arch:      linux/amd64

And I'm running cAdvisor as follows:

docker run -it --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --volume=/var/log:/var/log:ro --publish=8888:8080 google/cadvisor:latest -disable_metrics disk -logtostderr
I0928 14:02:52.419328       1 storagedriver.go:50] Caching stats in memory for 2m0s
I0928 14:02:52.421050       1 manager.go:140] cAdvisor running in container: "/docker/8220f6518546f643bbaf2e22d6ce1438d8fcd85c11e556504c5047b196fd040c"
W0928 14:02:52.478840       1 manager.go:148] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I0928 14:02:52.529191       1 fs.go:116] Filesystem partitions: map[/dev/sda1:{mountpoint:/var/lib/docker/overlay major:8 minor:1 fsType:ext4 blockSize:0}]
I0928 14:02:52.537331       1 manager.go:195] Machine: {NumCores:2 CpuFrequency:2599998 MemoryCapacity:13656510464 MachineID:af03dcd1dca6264136598dbff549c369 SystemUUID:AF03DCD1-DCA6-2641-3659-8DBFF549C369 BootID:0861dcdb-cfe0-43a2-bfee-1705cda0227d Filesystems:[{Device:overlay Capacity:31158935552 Type:vfs Inodes:3840000 HasInodes:true} {Device:/dev/sda1 Capacity:31158935552 Type:vfs Inodes:3840000 HasInodes:true}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:32212254720 Scheduler:noop}] NetworkDevices:[{Name:ens4 MacAddress:42:01:0a:f0:00:39 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory:13656510464 Cores:[{Id:0 Threads:[0 1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:20971520 Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-highmem-2 InstanceID:8555047159068902293}
I0928 14:02:52.545440       1 manager.go:201] Version: {KernelVersion:4.4.0-34-generic ContainerOsVersion:Alpine Linux v3.4 DockerVersion:1.12.0 CadvisorVersion:v0.24.0 CadvisorRevision:0cdf491}
I0928 14:02:52.598527       1 factory.go:295] Registering Docker factory
W0928 14:02:52.598608       1 manager.go:244] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I0928 14:02:52.598631       1 factory.go:54] Registering systemd factory
I0928 14:02:52.598971       1 factory.go:86] Registering Raw factory
I0928 14:02:52.599374       1 manager.go:1082] Started watching for new ooms in manager
I0928 14:02:52.599635       1 oomparser.go:200] OOM parser using kernel log file: "/var/log/kern.log"
I0928 14:02:52.601033       1 manager.go:285] Starting recovery of all containers
I0928 14:02:53.262367       1 manager.go:290] Recovery completed
I0928 14:02:53.778738       1 cadvisor.go:157] Starting cAdvisor version: v0.24.0-0cdf491 on port 8080

The problem is I can't find any OOM events in cAdvisor. At the same time I'm using mtail to keep track of /var/log/kern.log file (OOM events are there).

I tried to use API, without any result (/api/v1.3/events?oom_events=true&oom_kill_events=true).
I would appreciate any help :)

PK

[vishh]

@tstclair we need to add a couple of tests for this feature. I don't think
we test on xenial yet.

On Wed, Sep 28, 2016 at 7:44 AM, pkrolikowski [email protected]
wrote:

Hello,

I'm trying to use cAdvisor to catch OOM events but I failed (probably I'm
doing something wrong).
Here's my setup:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04 LTS
Release: 16.04
Codename: xenial

docker version
Client:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built: Thu Jul 28 22:11:10 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.0
API version: 1.24
Go version: go1.6.3
Git commit: 8eab29e
Built: Thu Jul 28 22:11:10 2016
OS/Arch: linux/amd64

And I'm running cAdvisor as follows:

docker run -it --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --volume=/var/log:/var/log:ro --publish=8888:8080 google/cadvisor:latest -disable_metrics disk -logtostderr
I0928 14:02:52.419328 1 storagedriver.go:50] Caching stats in memory for 2m0s
I0928 14:02:52.421050 1 manager.go:140] cAdvisor running in container: "/docker/8220f6518546f643bbaf2e22d6ce1438d8fcd85c11e556504c5047b196fd040c"
W0928 14:02:52.478840 1 manager.go:148] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I0928 14:02:52.529191 1 fs.go:116] Filesystem partitions: map[/dev/sda1:{mountpoint:/var/lib/docker/overlay major:8 minor:1 fsType:ext4 blockSize:0}]
I0928 14:02:52.537331 1 manager.go:195] Machine: {NumCores:2 CpuFrequency:2599998 MemoryCapacity:13656510464 MachineID:af03dcd1dca6264136598dbff549c369 SystemUUID:AF03DCD1-DCA6-2641-3659-8DBFF549C369 BootID:0861dcdb-cfe0-43a2-bfee-1705cda0227d Filesystems:[{Device:overlay Capacity:31158935552 Type:vfs Inodes:3840000 HasInodes:true} {Device:/dev/sda1 Capacity:31158935552 Type:vfs Inodes:3840000 HasInodes:true}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:32212254720 Scheduler:noop}] NetworkDevices:[{Name:ens4 MacAddress:42:01:0a:f0:00:39 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory:13656510464 Cores:[{Id:0 Threads:[0 1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:20971520 Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-highmem-2 InstanceID:8555047159068902293}
I0928 14:02:52.545440 1 manager.go:201] Version: {KernelVersion:4.4.0-34-generic ContainerOsVersion:Alpine Linux v3.4 DockerVersion:1.12.0 CadvisorVersion:v0.24.0 CadvisorRevision:0cdf491}
I0928 14:02:52.598527 1 factory.go:295] Registering Docker factory
W0928 14:02:52.598608 1 manager.go:244] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
I0928 14:02:52.598631 1 factory.go:54] Registering systemd factory
I0928 14:02:52.598971 1 factory.go:86] Registering Raw factory
I0928 14:02:52.599374 1 manager.go:1082] Started watching for new ooms in manager
I0928 14:02:52.599635 1 oomparser.go:200] OOM parser using kernel log file: "/var/log/kern.log"
I0928 14:02:52.601033 1 manager.go:285] Starting recovery of all containers
I0928 14:02:53.262367 1 manager.go:290] Recovery completed
I0928 14:02:53.778738 1 cadvisor.go:157] Starting cAdvisor version: v0.24.0-0cdf491 on port 8080

The problem is I can't find any OOM events in cAdvisor. At the same time
I'm using mtail to keep track of /var/log/kern.log file (OOM events are
there).

I tried to use API, without any result (/api/v1.3/events?oom_events=
true&oom_kill_events=true).
I would appreciate any help :)

PK

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1484, or mute the thread
https://github.com/notifications/unsubscribe-auth/AGvIKACShHFSejFxg3SorWEUwszfIhT0ks5qun08gaJpZM4KI6Z-
.

[pkrolikowski]

So any suggestion what can I do with it ?