Changing node names on a restored CouchDB cluster backup

[H--o-l]

Hi!

Context:

I'm looking for the best way to backup a CouchDB cluster.

Each node of the cluster has a name with its DNS address in it, like recommended here.
Their names look like couchdb@<address-vm-for-node-1>, couchdb@<address-vm-for-node-2>, couchdb@<address-vm-for-node-3>.

The cluster has a lot of small DBs (I use the perusers database approach).

From the backup data, I'd like to be able to quickly create a new CouchDB node for testing.
I'd also like to restore the original cluster quickly if needed.

For a long time, I used replication as recommended here.
As the number of databases in the cluster grows, there are more and more replications to do, and it takes more and more time.
The worst is: if I need to recreate the original cluster from scratch, with only the replicated data, it will take me days to transfer all the databases using replication.

The second approach proposed in the documentation is to do a copy of data/shards and data/.shards folders.
In that case backup data can only be used to recreate the original cluster with the same node names (couchdb@<address-vm-for-node-1>, ...) thus I cannot use the backup data to create a new cluster with different node names, so I cannot do testing.

My idea:

Backup data with a copy of data/shards and data/.shards folders, and, if needed when I use the backup data, change node(s) name(s) with /_node/{node_name}/_dbs/{db_name} route documented here.

The procedure would be:

• save data/shards and data/.shards from CouchDB node couchdb@<address-vm-for-node-1>
• when needed, create a new CouchDB instance with a node new@localhost.
• copy data/shards and data/.shards inside the new CouchDB
• GET /_node/{node_name}/_dbs/{db_name}, change all couchdb@<address-vm-for-node-1> to new@localhost and PUT /_node/{node_name}/_dbs/{db_name} the result

I have two questions on this procedure:

1. GET /_node/{node_name}/_dbs/{db_name} and PUT /_node/{node_name}/_dbs/{db_name} are per-databases API, which means that I need to do hundreds of thousands of HTTP requests. Is there a way to change all databases node name with less HTTP requests?
2. Is it safe? The documentation for route /_node/{node_name}/_dbs/{db_name} is about adding and removing CouchDB nodes, not changing a node name so shards become usable.
   I'm a bit worried that I missed something (from my tests continuous replication and conflict resolution are preserved).

Thanks for reading!

I hope you are interested in what I'm trying to do.

[H--o-l]

My question in summary: is it safe to use PUT /_node/_local/_dbs/<db_name> to reuse existing shards from a previous CouchDB cluster?

I have shards which belonged to a now dead <old.domain.name> CouchDB node, and I want to use them with a freshly created CouchDB <new.domain.name> node.
Is it safe to do GET /_node/_local/_dbs/<db_name> change all <old.domain.name> with <new.domain.name> inside the json and then PUT /_node/_local/_dbs/<db_name> the result?

[schwichti]

I am currently investigating options for multi-cluster backups. Did you solve your problem yet?

I have some general questions regarding the different approaches you have mentioned (neglecting changing node names). Maybe you can help me. The CouchDB backup documentation does not say anything explicitly about backups for multi-node setups.

Approach 1 Replication: I think when you backup a database from a multi-node cluster through replication you'd replicate it into a separate single-node to have all the data in one single place. Does it make sense? But it needs to be considered that you replicate only the most recent version of a document and the revision history is not replicated:

The replication process only copies the last revision of a document, so all previous revisions that were only in the source database are not copied to the destination database. ^Source

However, I want to backup the revision history, too.

---

Approach 2 Copying: Do you need to copy .couch/.shard files on every single node to backup your complete data, because your data is distributed over the nodes and not all nodes have the same data?

CouchDB clusters automatically shard databases and distribute the subsets of documents that compose each shard among nodes. ^Source

---

There is also another approach that was not mentioned before: @cloudant/couchbackup.

couchbackup does not do CouchDB replication as such, it simply streams through a database's _changes feed, and uses POST /db/_bulk_get to fetch the documents, storing the documents it finds on disk. [...]
[copying] the ".couch" file [...] is fine on a single-node instance, but when running multi-node Cloudant or using CouchDB 2.0 or greater, the ".couch" file only contains a single shard of data.

[H--o-l]

Hi @schwichti!

Did you solve your problem yet?

I haven't fully yet.

However, I want to backup the revision history, too.

It would be great I agree.
The other issue a have with backing up with replication is, when you have a lot of small DBs, replication became time-consuming, more than copying shards.

Approach 2 Copying: Do you need to copy .couch/.shard files on every single node to backup your complete data, because your data is distributed over the nodes and not all nodes have the same data?

If I'm not mistaken you also posted your question on the mailing list and the answer from @janl was really interesting
https://www.mail-archive.com/[email protected]/msg30003.html

On my side, I'm trying something where I back up a single node by saving data/_dbs.couch and data/shards, and let CouchDB internal backfill the other nodes, as @janl explained.
I have found two noticeable things with this approach:

• you can monitor the undocumented route GET /_dbs on each of your CouchDB nodes to get an overview of the remaining CouchDB internal backfill.
• Databases security permissions are reset. A workaround is to pre-copy the data/shards from the save to all the nodes (but not data/_dbs.couch, for which letting CouchDB internal backfill it is important I think).

[schwichti]

My preferred approach currently is https://github.com/danielebailo/couchdb-dump. Saving the full revision history is not that important for us as well as time to recovery. Copying the .couch and shards of every single cluster nodes does not seem to suit our usecase.

[H--o-l]

Hey!

Following this discussion, I, and the team I belong to, created a small tool to help us backup and restore our production servers:
https://github.com/tolteck/couchcopy

Under the hood it backups CouchDB data folder, and during restoration it updates CouchDB shards placement using PUT /_node/{node_name}/_dbs/{db_name}, to match the new cluster erlang node name.

If you read this, thanks for your time!