CSI로 attach된 볼륨에 대한 데이터가 클라우드 워치로 전송되지 않음

[lens0021]

infra/aws/res/bootstrap.sh

Lines 58 to 80 in 477b5dd

	cat <<'EOF' > /opt/aws/amazon-cloudwatch-agent/bin/config.json
	{
	"metrics": {
	"metrics_collected": {
	"disk": {
	"measurement": [
	"used_percent"
	],
	"resources": [
	"/",
	"/srv"
	]
	},
	"mem": {
	"measurement": [
	"mem_used_percent"
	]
	}
	}
	}
	}
	EOF
	/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

• Set alerts on Grafana #358

[lens0021]

그냥 확인하는 법도 잘 모르겠음...

$ df -h
Filesystem        Size  Used Avail Use% Mounted on
devtmpfs          936M     0  936M   0% /dev
tmpfs             977M     0  977M   0% /dev/shm
tmpfs             977M  1.1M  976M   1% /run
tmpfs             977M     0  977M   0% /sys/fs/cgroup
/dev/nvme0n1p1     16G  9.9G  6.2G  62% /
/dev/nvme0n1p128   10M  3.7M  6.3M  38% /boot/efi
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/718560a8-2871-efa5-d3dd-466d197ff73f/memcached/secrets
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/e3bcd921-684d-145c-497c-16bd08baabf4/plugin/secrets
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/017c98b6-ea0d-b8ac-5a68-35a42ac6a5f9/plugin/secrets
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/75de924d-f7d0-515f-8848-e81237205d3a/restbase/secrets
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/4fcd60e2-86aa-4bea-e945-5a0c5071cd91/mathoid/secrets
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/1b2e1b7d-f7c4-5c85-5c6e-cdece0b90625/http/secrets
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/071fff0f-bbd2-515d-03fb-279afc581c7c/wait-for-mediawiki/secrets
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/071fff0f-bbd2-515d-03fb-279afc581c7c/parsoid/secrets
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/0752b366-e9d6-2bc0-4459-7c32ac7302eb/mysql/secrets
tmpfs             196M     0  196M   0% /run/user/1000
tmpfs             1.0M  4.0K 1020K   1% /opt/nomad/alloc/81c4b3c4-b104-9348-3161-014099023906/backupbot/secrets
tmpfs             1.0M     0  1.0M   0% /opt/nomad/alloc/c391faea-9dba-888a-2deb-a995a7c4b6fa/wait-for-backend/secrets
tmpfs             1.0M  4.0K 1020K   1% /opt/nomad/alloc/c391faea-9dba-888a-2deb-a995a7c4b6fa/fastcgi/secrets

$ nomad volume status
Container Storage Interface
ID        Name        Plugin ID  Schedulable  Access Mode
caddycer  caddycerts  aws-ebs0   true         single-node-writer
mysql     mysql       aws-ebs0   true         single-node-writer

$ nomad volume status mysql
ID                   = mysql
Name                 = mysql
External ID          = vol-046c6026dc27a61de
Plugin ID            = aws-ebs0
Provider             = ebs.csi.aws.com
Version              = v0.10.1
Schedulable          = true
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 1
Nodes Expected       = 1
Access Mode          = single-node-writer
Attachment Mode      = file-system
Mount Options        = <none>
Namespace            = default

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created     Modified
0752b366  eef378d3  mysql       7        run      running  13h24m ago  4h10m ago

[lens0021]

그냥 보는 법:

$ nomad alloc exec -job mysql df  
Filesystem     1K-blocks     Used Available Use% Mounted on
overlay         16755692 10322400   6433292  62% /
tmpfs              65536        0     65536   0% /dev
tmpfs             999908        0    999908   0% /sys/fs/cgroup
/dev/nvme0n1p1  16755692 10322400   6433292  62% /alloc
tmpfs               1024        0      1024   0% /secrets
/dev/nvme1n1     8191416  5528196   2646836  68% /srv/mysql
shm                65536        0     65536   0% /dev/shm
tmpfs             999908        0    999908   0% /proc/acpi
tmpfs             999908        0    999908   0% /proc/scsi
tmpfs             999908        0    999908   0% /sys/firmware

$ nomad alloc exec -job mysql df | awk '{if($6=="/srv/mysql")print $5}'
68%
$ nomad alloc exec -job http df | awk '{if($6=="/etc/caddycerts")print $5}'
1%

[lens0021]

$ df -a | grep mysql
/dev/nvme1n1 - - - - /opt/nomad/client/csi/node/aws-ebs0/staging/mysql/rw-file-system-single-node-writer
/dev/nvme1n1 - - - - /opt/nomad/client/csi/node/aws-ebs0/per-alloc/681acfc8-ff0f-a9c7-89f5-3842454176e3/mysql/rw-file-system-single-node-writer

[lens0021]

지금(Nomad v1.2.5)은 똑같은 명령어에서 모두 alloc ID가 포함되어서 지속적으로는 사용할 수 없어 보입니다.

$ df -a | grep mysql
/dev/nvme2n1             -        -         -    - /opt/nomad/client/csi/node/aws-ebs0/per-alloc/e4e4abdc-85c1-ee1b-baac-be5a193ce121/mysql/rw-file-system-single-node-writer
tmpfs                 1024        0      1024   0% /opt/nomad/alloc/e4e4abdc-85c1-ee1b-baac-be5a193ce121/mysql/secrets