Manually sync Percona database servers¶
It may happen sometimes, for example right after bundle deploy
,
that the database cluster fails to sync across units.
In such cases, one can manually force sync-ing by:
- stopping all mysql instances
- stopping juju agents on all units
- starting one instance, and possibly force it to run as master
- starting the additional instances
- starting juju agents on all units
The detailed procedure is described below.
stop mysql
on all units¶
$ juju run –application percona-cluster “sudo systemctl stop mysql”
make sure all mysql
processes are indeed stopped, do not trust systemctl status mysql
¶
$ juju run –application percona-cluster “ps -ef | grep mysql”
stop juju agents
on all units::¶
$ juju ssh percona-cluster/N sudo service jujud-unit-percona-cluster-N stop
(repeat the command on the other services)
connect to one of the servers, and force mysql
restart as master.¶
Note
You should perform the following operations on the last-acting master,
otherwise mysql
will fail to start. If you are unable to determine
the last-acting master, edit file /var/lib/percona-xtradb-cluster/grastate.dat
and set safe_to_bootstrap: 1
before restarting mysql.
Edit file /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf, you should find the line:
wsrep_cluster_address=gcomm://<ip_1>,<ip_2>
where <ip_1>, <ip_2> are the IP addresses of the other members of the cluster.
Comment this line, copy/paste then remove the IP addresses, leaving:
wsrep_cluster_address=gcomm://
Finally restart MySql:
$ sudo service mysql start
It may happen that mysql server still does not start because of failed transactions. In case try the following procedure.
Looking in the log file /var/log/mysql/error.log
you may find messages like:
2023-01-30T23:57:22.828179Z 0 [ERROR] Found 7 prepared transactions! It means that mysqld was not...binlog or tc.log file) was manually deleted after a crash.
You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions.
Issue the following command:
$ sudo mysqld --tc-heuristic-recover=ROLLBACK
This should fix the failed transactions.
start mysql
on the remaining nodes, as usual¶
$ juju ssh percona-cluster-N sudo systemctl start mysql
go back to the first node, stop mysql
, revert the change to wsrep_cluster_address
and restart mysql
¶
On the first node edit the file /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf
and uncomment the original line:
wsrep_cluster_address=gcomm://<ip_1>,<ip_2>
Then restart mysql:
$ sudo service mysql restart
start juju
agents on all the units¶
$ juju ssh percona-cluster/N sudo service jujud-unit-percona-cluster-N start
(repeat the command on the other services)
Now your Percona cluster should be in sync.
Percona units in error hook failed: "leader-elected"
¶
It may happen that one percona-cluster unit falls with this error during configuration. To fix it issue the following command:
$ juju resolved --no-retry percona-cluster/N
According to the Canonical support, this bug should be definetely fixed upgrading percona-cluster charm to v268:
$ juju upgrade-charm percona-cluster
Percona-cluster error: Leader UUID != Unit UUID¶
It may happen that two or more percona-cluster unit (non-leaders units) fail with this error.
If you check debug logs (juju debug-logs –include <percona-unit>) of the units in error, you should see similar logs:
020-07-09 15:04:07 DEBUG leader-settings-changed File "/var/lib/juju/agents/unit-percona-cluster-2/charm/hooks/percona_utils.py", line 662, in update_bootstrap_uuid
2020-07-09 15:04:07 DEBUG leader-settings-changed cluster_state_uuid)
2020-07-09 15:04:07 DEBUG leader-settings-changed percona_utils.InconsistentUUIDError: Leader UUID ('eb31ead5-c1f1-11ea-98b8-be3a8df2897a') != Unit UUID ('eb31ead5-c1f1-11ea-98b8-be3a8df2897b')
The problem is that the percona-cluster leader has got UUID eb31ead5-c1f1-11ea-98b8-be3a8df2897a while the other units are expecting this UUID to be eb31ead5-c1f1-11ea-98b8-be3a8df2897b.
To fix this issue, proceed as follows:
Run the following command on the percona-cluster leader unit:
juju run --unit <percona-cluster-leader> "leader-get bootstrap-uuid" **Note**: This should be eb31ead5-c1f1-11ea-98b8-be3a8df2897a
Now, run the following command (again, on the percona-cluster leader unit):
juju run --unit <percon-cluster-leader> "leader-set bootstrap-uuid=eb31ead5-c1f1-11ea-98b8-be3a8df2897b"