Support Percona XtraBackup

[derekperkins]

Add core Vitess support for XtraBackup

cc @hmcgonig @dweitzman @rafael @sjmudd @bbeaudreault @acharis @sougou

[deepthi]

Open questions for discussion:

• Allow backup method to be chosen at backup time, not vttablet startup?
  @dweitzman has already expressed a preference for this
• Pass through flags to xtrabackup?
  Will allow us to support compress, encrypt and SSL
• Backup_transform_hook support?
  How is this being used today?
• Support xtrabackup non-streaming mode with file backup storage?
• Managing version compatibility

Future enhancements:

• incremental backups
• mysql builtin backup mode (8.0)

[derekperkins]

Allow backup method to be chosen at backup time, not vttablet startup

no preference

Pass through flags to xtrabackup?

This seems like a good idea, especially since it should reduce the Vitess api surface

Backup_transform_hook support

not using it, no plans to use it

Support xtrabackup non-streaming mode with file backup storage

I think it's fine to ship without this and maybe tackle it later if somebody really needs it

Managing version compatibility

This is important to us because we're typically at the bleeding edge. I would imagine that it shouldn't be that much work to manage though, since the the xtrabackup binaries should have pretty stable apis. I don't think Vitess should ship with a binary included, I think it should just expect to have it available in $PATH if you choose that option.

incremental backups (future)

There has been a significant amount of discussion around #3581 binlog retention / backups here. If we could figure out a good solution there, it could give us point in time recovery in conjunction with normal backups, which could make xtrabackup based incremental backups unnecessary.

mysql builtin backup mode (8.0)

Hopefully xtrabackup decides to utilize that functionality if there are performance benefits so it would be transparent to Vitess.

[rafael]

Open questions for discussion:

• Allow backup method to be chosen at backup time, not vttablet startup?
  @dweitzman has already expressed a preference for this

I think for simplicity we should keep it as is right now.

However I do see this feature being super useful when rolling out this change. You could start triggering these backups without having to spin new tablets.

My suggestion would be that we don't couple these changes with this feature. Seems orthogonal.

• Pass through flags to xtrabackup?
  Will allow us to support compress, encrypt and SSL

I think this is a good idea, but not required for initial version.

• Backup_transform_hook support?
  How is this being used today?

We are not using this. We used it once to test different compression strategies.

• Support xtrabackup non-streaming mode with file backup storage?
• Managing version compatibility

Agree with @derekperkins

Future enhancements:

• incremental backups
• mysql builtin backup mode (8.0)

[sjmudd]

Allow backup method to be chosen at backup time, not vttablet startup?

Yes please. I moved from using NFS to S3 and the fact it required restarting all vttablets and vtctld processes was a major nuisance. I think this is a clear indicator of where the default setting should not be necessarily stored in each process by perhaps in the topology. This would simplify Vitess configuration.

support compress, encrypt and SSL

Yes.

Not mentioned preparing xtrabackup backups after taking a live xtrabackup copy. This makes restoring faster but may be more complex to arrange with xtrabackup.

Vtctl should have an explicit mode that allows you to force Master to Backup #4557

Well xtrabackup can happily do a live backup from a running master. The only thing that it will trigger is more load which may or may not be an issue. Vitess should know this is possible, but probably try to avoid doing a backup from the master unless absolutely necessary.

[deepthi]

@sjmudd preparing the backup created by xtrabackup is included in the current WIP PR. I understood it to be a mandatory step in the restore process (from the documentation).

[derekperkins]

@deepthi I'm pretty sure he's talking about preparing it during the backup phase, not during the restore phase. I brought it up briefly on the call. It's a tradeoff between spending extra resources preparing backups that never get used and having restores that work instantly without spending potentially hours preparing.

[deepthi]

@derekperkins I missed that, thanks for the clarification.

[sougou]

On the question of choosing backup method: Backups have to be coordinated with restores. This means that a restore needs to know where the last backup can be found. In this respect, choosing the backup method at the time of backup would be problematic. The ideal solution would require the backup method to be a cluster level parameter (probably in topo). This will allow the backup and restore to see the same settings.

However, this is hard to implement, especially for transitions. If one wants to move from one storage type to another, then the restore process needs to search previous storage types and use the correct method to perform the restore.

For now, a good trade-off will be to specify this at a vttablet level. This way, we at least force the users to think about the consequences of changing the backup method.

[sjmudd]

@sjmudd preparing the backup created by xtrabackup is included in the current WIP PR. I understood it to be a mandatory step in the restore process (from the documentation).

Hi Deepthi, not it's not strictly necessary and it can be time-consuming as you effectively re-apply any redo logs that were not applied to the backed up database files. If you have a system with a lot of dirty pages, due to heavy writes then this process can take quite a long time. I've seen it take several hours after the xtra backup completed to process.

So good to see you do this. I guess I should have looked in the code to check. +1

[sjmudd]

On the question of choosing backup method: Backups have to be coordinated with restores. This means that a restore needs to know where the last backup can be found. In this respect, choosing the backup method at the time of backup would be problematic.

I would expect that if more than one backup method is supported the end result from the user point of view is that they see a "list of backups" and can choose which one to restore, with the most likely situation being they restore from the most recent backup. The fact it comes from backup method 2 rather than backup method 1 should only be a concern when the restore takes place as the tablet needs to apply that method as that's how it was made.

So when I do the backup list below I'd expect to get the combined list from both methods.

$  vtctlclient -server :12345 ListBackups keyspace/40-80
2018-03-22.005422.cell1-0000006451
2018-03-23.165916.cell1-0000001325
...
2019-03-08.082409.cell1-0000009105

In many uses cases I see backups expire after a period of time so basically I would leave both methods active during the migration period, with the preferred method changed to the new method, triggering backups to use that, but allowing me to restore older backups using the old method.
Once no old method backups are available you change the configuration to only support the new backup/restore method.

Right now ListBackups does not indicate the backup method. That might be handy if multiple methods are supported if only to confirm the method used.

[deepthi]

@sjmudd preparing the backup created by xtrabackup is included in the current WIP PR. I understood it to be a mandatory step in the restore process (from the documentation).

Hi Deepthi, not it's not strictly necessary and it can be time-consuming as you effectively re-apply any redo logs that were not applied to the backed up database files. If you have a system with a lot of dirty pages, due to heavy writes then this process can take quite a long time. I've seen it take several hours after the xtra backup completed to process.

So good to see you do this. I guess I should have looked in the code to check. +1

Actually no :( We are only doing it in the restore phase. In streaming mode xtrabackup creates an archive file. There's no way to "prepare" that without extracting the archive.