Module is unable to bootstrap a new XtraDB cluster

[fraenki]

The module is unable to bootstrap a new cluster if the data directory does not exist:

Notice: /Stage[main]/Galera::Repo/Yumrepo[galera_percona]/ensure: created
Notice: /Stage[main]/Galera::Repo/Yumrepo[galera_epel]/ensure: created
Notice: /Stage[main]/Galera::Repo/Package[Percona-XtraDB-Cluster-shared-compat-57]/ensure: created
Notice: /Stage[main]/Galera/Package[percona-xtrabackup-24]/ensure: created
Notice: /Stage[main]/Mysql::Server::Install/Package[mysql-server]/ensure: created
Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/ensure: defined content as '{md5}63be06247231f5789304856324bcf3b4'
Notice: /Stage[main]/Mysql::Server::Installdb/Mysql_datadir[/var/lib/mysql]/ensure: created
Notice: /Stage[main]/Galera/Exec[bootstrap_galera_cluster]/returns: executed successfully
Info: /Stage[main]/Galera/Exec[bootstrap_galera_cluster]: Scheduling refresh of Exec[validate_connection]
Error: Systemd start for mysql failed!
Error: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: change from 'stopped' to 'running' failed: Systemd start for mysql failed!
Notice: /Stage[main]/Galera/Exec[create /root/.my.cnf]: Dependency Service[mysqld] has failures: true

In this case, mysql::server::installdb notifies Class['mysql::server::service']... as soon as Exec[bootstrap_galera_cluster] does it's thing, it breaks.

[fraenki]

It seems to fail because Percona uses a 2nd systemd service [email protected] solely for the bootstrap process. When bootstrapping and trying to start the "main" mysql service, the following error is returned:

centos7 # systemctl status mysql
● mysql.service - Percona XtraDB Cluster
   Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2018-10-07 17:09:33 CEST; 8min ago
  Process: 4428 ExecStartPre=/usr/bin/mysql-systemd start-pre (code=exited, status=1/FAILURE)

Oct 07 17:09:33 systemd[1]: Starting Percona XtraDB Cluster...
Oct 07 17:09:33 mysql-systemd[4428]: WARNING: PXC is in bootstrap mode.  To switch to normal operation, first stop the [email protected] then start the mysql service
Oct 07 17:09:33 systemd[1]: mysql.service: control process exited, code=exited status=1
Oct 07 17:09:33 systemd[1]: Failed to start Percona XtraDB Cluster.
Oct 07 17:09:33 systemd[1]: Unit mysql.service entered failed state.
Oct 07 17:09:33 systemd[1]: mysql.service failed.

Because of this quirky approach, puppetlabs-mysql will not be able receive a success message when trying to start the service.

[Yamakasi]

I see this happening now as well, what is the way to make this work ?

[fraenki]

@Yamakasi Perform a manual bootstrap, according to the Percona XtraDB docs.

[anthonysomerset]

I just want to advise, this behaviour exists on Ubuntu 18.04 with percona 5.7, on the first node only you need to run

/etc/init.d/mysql bootstrap-pxc

then re-run puppet to ensure config items are finalised, then puppet will run on other nodes fine

[Yamakasi]

@anthonysomerset you don't need a re-run.

Ubuntu is not ideal for this type of software because grabbing the password is far from ideal.

[ThoTischner]

Working solution for Centos7:
Hiera

### bootstrap command
galera::percona::57::bootstrap_command: 'systemctl start mysql'

Manifest

  ### bootstarp master via default service
  if ($fqdn == $master) {

    file {'/etc/systemd/system/mysql.service.d':
      ensure  =>  directory,
    }->
    file { '/etc/systemd/system/mysql.service.d/override.conf':
      ensure      => present,
      source      => 'puppet:///modules/profile/galera/mysql.service.override',
      before      => Package['Percona-XtraDB-Cluster-57'],
      replace     => 'no',
      notify      => Exec['remove bootstrap override'],
    }

    Exec<| title == 'bootstrap_galera_cluster' |> {
      require => File['/etc/systemd/system/mysql.service.d/override.conf']
    }
   
    exec { 'remove bootstrap override':
      command     => 'echo -n > /etc/systemd/system/mysql.service.d/override.conf && systemctl daemon-reload',
      require     => Service['mysql'],
      refreshonly => true,
    }
  }

Service override file

[Service]
ExecStart=
ExecStart=/usr/bin/mysqld_safe --basedir=/usr --wsrep-new-cluster

Please let me know what you think.
If it is okay, i will create a PR. @fraenki

[fraenki]

I've addressed this issue in 243e7c3:

• Stop the mysql@bootstrap service after performing the bootstrap. This allows puppetlabs/mysql to properly startup the node (and it will be in primary state).
• On RedHat-based systems ensure to configure the proper log file (override the MariaDB defaults).
• Fix possible installation issues on RedHat-based systems by excluding Percona-Server packages.
• Fix possible dependency isues by using the meta-package on RedHat-based systems.

I've verified that this works on CentOS 7 and also added a simple acceptance test (tests for Debian/Ubuntu are planned too).

The upcoming release 2.0.0 of this module will contain all fixes.

[ThoTischner]

Thanks a lot @fraenki

[fraenki]

The upcoming release 2.0.0 of this module will contain all fixes.

Version 2.0.0 has just been released. It includes acceptance tests for Galera, XtraDB and MariaDB on all supported operating systems:
https://travis-ci.org/github/fraenki/puppet-galera/builds/674627174
Please open a new issue if you find yet another bug :)