Cluster with private interfaces

[Catlike14]

Hello, I'm trying to setup a CouchDB cluster, but I would like to use a private interface for the internal communications between nodes.
I tried setting inet_dist_use_interface to my private IP to use epmd through the private interface, but the process ends up being bound to 0.0.0.0 instead. So I set the ERL_EPMD_ADDRESS environment variable to my private IP too, and it works for binding. However, the cluster doesn't work, probably because CouchDB is trying to use the public interface to reach the other nodes' epmd.

1. Is there a way to use the private interface for communication between clustered nodes, while using the public interface for external (API) access?
2. Do the node use mainly epmd for communicate, or do they use the API (:5984) as well?

Without the env variable and with inet_dist_use_interface set to 0,0,0,0 the cluster works smoothly, but I'm pretty sure the node use the public interface for epmd, as there is no setting about the private interface.

I'm using FQDNs, but I also tried to setup the cluster using the private IPs, but it didn't work.
I'm using the setup UI from Fauxton.

[Catlike14]

I noticed that I forgot to set the IP in vm.args after switching from domain to IP, which is why the cluster setup didn't work. Eventually, I set up a cluster using domains that point to private IPs (to benefit from the advantages of domains), with epmd listening on the private interface, and I left the :5984 binding to 0.0.0.0 for the APIs. It seems to work perfectly, but I'd like to have some feedback from someone who has set up a cluster with private interfaces.

[rnewson]

hi, thanks for moving this issue.

It seems you've figured this out but I can restate some couchdb/erlang basics;

users communicate with couchdb over http only, to port 5984 of any node in the cluster. couchdb can be configured to bind that port on a particular interface (typically a public one).

the couchdb nodes talk to each other with the erlang distributed protocol over other (dynamically assigned) ports. They contact the port mapper (4369) to find that port. You can bind epmd to a private interface (either by starting epmd yourself with those settings or, less safely, setting ERL_EPMD_ADDRESS to that private interface for when couchdb launches epmd for you). the Erlang nodes will use port 4369 at the hostnames of the clusters, so those need to resolve to an address that will route over your private network.

You said "forgot set the IP in vm.args", can you clarify that you meant the ERL_EPMD_ADDRESS=<private ip> setting, or was it something else?

It is strongly encouraged to give couchdb cluster members FQDN's and not refer to each other directly by IP address.

[Catlike14]

With "I forgot to set the IP in vm.args" I mean that when I switched from domain to IP, I kept couchdb@domain instead of changing it to couchdb@ip. Yes it is discouraged to use IPs, but that was just for testing. Now that I have figured out how to set it up properly, I switched back to domains again.

Thanks for claryfing the design, you confirmed my understanding. However, I noticed that when I set up the cluster through Fauxton, when I add the nodes, it asks me for the port, that is 5984. Can you clarify why this is needed, since the nodes should communicate through Erlang? I would expect that the host should be enough, since they do the discovery through epmd, so I'm not sure why the HTTP port is needed.

Also, why is using ERL_EPMD_ADDRESS considered less secure?

[rnewson]

The cluster setup tool communicates with nodes over the http api (to modify configuration via /_node/_local/_config and such), it's just automating steps you could do yourself through the public http api. If you think about it it has to, as the erlang communication protocol isn't established yet.

Not less secure, less safe. All I mean is that ERL_EPMD_ADDRESS only helps if couchdb launches epmd (the daemon is started if not started before couchdb itself starts). So if you started epmd yourself it would bind to all interfaces by default. The safest way to arrange things is to have a service (initv or systemd) around epmd that starts it first (and have couchdb service requires that the epmd service is started first). alternatively you could ensure your firewall only allows inbound connections to 4369 from your private network if.

[Catlike14]

Ok, got it. Thanks for all!