ansible-playbook bootstrap.yml Terminated

[Lyle-liang]

Bug Report

ansible-deploy error , ansible-2.7.0, TiDB version 2.0.7. Help. log like this:

[ec2-user@ip tidb-ansible]$ ansible-playbook bootstrap.yml --ssh-extra-args='-i /home/ec2-user/tidb-ansible/ssh.pem'

PLAY [initializing deployment target] ********************************************************************************************************************************************************************************************************

TASK [check_config_static : Ensure only one monitoring host exists] **************************************************************************************************************************************************************************

TASK [check_config_static : Ensure monitored_servers exists] *********************************************************************************************************************************************************************************

TASK [check_config_static : Ensure TiDB host exists] *****************************************************************************************************************************************************************************************

TASK [check_config_static : Ensure zookeeper address of kafka cluster is set when enable_binlog] *********************************************************************************************************************************************

TASK [check_config_static : Ensure PD host exists] *******************************************************************************************************************************************************************************************

TASK [check_config_static : Ensure TiKV host exists] *****************************************************************************************************************************************************************************************

TASK [check_config_static : Check ansible_user variable] *************************************************************************************************************************************************************************************

TASK [check_config_static : Ensure timezone variable is set] *********************************************************************************************************************************************************************************

TASK [check_config_static : Close old SSH control master processes] **************************************************************************************************************************************************************************
Terminated

[alivxxx]

Thanks for your feedback.
PTAL @LinuxGit

[Lyle-liang]

@LinuxGit Why and What can I do? Thanks

[LinuxGit]

@Lyle-liang Thanks for your feedback.

I didn't reproduce this error on CentOS 7. Did the task stop at TASK [check_config_static : Close old SSH control master processes] ?
There's a task on
https://github.com/pingcap/tidb-ansible/blob/master/roles/check_config_static/tasks/main.yml#L43, you can remove this task, and check if there's any errors.

[Lyle-liang]

@LinuxGit Thanks for your reply.

Yes，the task stop at TASK [check_config_static : Close old SSH control master processes].

It's OK. Why is this? What is the effect of removing the task?

[LinuxGit]

What's your Linux distribution and OS version? uname -a
Ansible persists SSH connection, and changing ulimit task need to take effact in a ssh connection, so kill original ansible control persist processes.
If you aleadly executed bootstrap.yml, there's no side effect.

[Lyle-liang]

uname -a
4.14.70-72.55.amzn2.x86_64 #1 SMP

This is new environment, and I have successfully used ansible-2.6.4 on the same environment .

[Lyle-liang]

@LinuxGit

TiDB server start error.

ansible-playbook start.yml --ssh-extra-args='-i /home/ec2-user/tidb-ansible/ssh.pem'
TASK [start TiDB by systemd] *****************************************************************************************************************************************************************************************************************
changed: [TiDB1]
changed: [TiDB2]
changed: [TiDB3]
TASK [wait for TiDB up] **********************************************************************************************************************************************************************************************************************
TASK [wait for TiDB up] **********************************************************************************************************************************************************************************************************************
fatal: [TiDB2]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for search string TiDB in ip:10080"}
fatal: [TiDB1]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for search string TiDB in ip:10080"}
fatal: [TiDB3]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for search string TiDB in ip:10080"}

log like this:

flag provided but not defined: -advertise-address
Usage of bin/tidb-server:
-L string
log level: info, debug, warn, error, fatal (default "info")
-P string
tidb server port (default "4000")
-V print version information and exit (default false)
-binlog-socket string
socket file to write binlog
-config string
config file path
-host string
tidb server host (default "0.0.0.0")
-lease string
schema lease duration, very dangerous to change only if you know what you do (default "45s")
-log-file string
log file path
-log-slow-query string
slow query file path
-metrics-addr string
prometheus pushgateway address, leaves it empty will disable prometheus push.
-metrics-interval uint
prometheus client push interval in second, set "0" to disable prometheus push. (default 15)
-path string
tidb storage path (default "/tmp/tidb")
-proxy-protocol-header-timeout uint
proxy protocol header read timeout, unit is second. (default 5)
-proxy-protocol-networks string
proxy protocol networks allowed IP or *, empty mean disable proxy protocol support
-report-status
If enable status report HTTP service. (default true)
-run-ddl
run ddl worker on this tidb-server (default true)
-socket string
The socket file to use for connection.
-status string
tidb server status port (default "10080")
-store string
registered store name, [tikv, mocktikv] (default "mocktikv")
-token-limit int
the limit of concurrent executed sessions (default 1000)

[LinuxGit]

Please use tidb-ansibe release-2.0 branch if you want to deploy tidb 2.0.7.
See https://github.com/pingcap/docs/blob/master/op-guide/ansible-deployment.md#step-3-download-tidb-ansible-to-the-control-machine.

[Lyle-liang]

@LinuxGit Thanks.

If I want to use v2.1.0-rc.2 or v2.1.0-rc.3, what can I do? How configure tidb_version in inventory.ini?

[LinuxGit]

see https://github.com/pingcap/tidb-ansible/blob/release-2.1/inventory.ini#L63
use release-2.1 branch.

[Lyle-liang]

@LinuxGit
Hi, I run the command： ansible-playbook rolling_update_monitor.yml and meet error. As show below:

TASK [node_exporter : create deploy directories] *********************************************************************************************************************************************************************************************
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}
fatal: [ip]: FAILED! => {"msg": "'node_exporter_log_dir' is undefined"}

What can I do ?

[LinuxGit]

What's your tidb-ansible branch, and make sure you haven't modified monitored_servers.yml.
https://github.com/pingcap/tidb-ansible/blob/release-2.1/group_vars/monitored_servers.yml

[LinuxGit]

This issue has been marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you are still experiencing this issue, please test against a recent version of TiDB Ansible, and open a new issue with a full reproducer showing this issue.

We've limit ansible version in https://github.com/pingcap/tidb-ansible/blob/master/requirements.txt file.