Failed to bind 127.0.0.1

[TomBayne]

Hi, I am trying to run couchdb, it was working perfectly up until very recently. It is now crashing on launch with this log output:

[big-r81]

Hi,

which version of CouchDB? Did you update to v3.2.2?

can you show your vm.args file (replace your cookie after -setcookie with XXX)?

[TomBayne]

Yes, I am running 3.2.2-1. vm.args below.

# Licensed under the Apache License, Version 2.0 (the "License"); you may not
# use this file except in compliance with the License. You may obtain a copy of
# the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations under
# the License.

# Each node in the system must have a unique name. These are specified through
# the Erlang -name flag, which takes the form:
#
#    -name nodename@<FQDN>
#
# or
#
#    -name nodename@<IP-ADDRESS>
#
# CouchDB recommends the following values for this flag:
#
# 1. If this is a single node, not in a cluster, use:
#    -name [email protected]
#
# 2. If DNS is configured for this host, use the FQDN, such as:
#    -name [email protected]
#
# 3. If DNS isn't configured for this host, use IP addresses only, such as:
#    -name [email protected]
#
# Do not rely on tricks with /etc/hosts or libresolv to handle anything
# other than the above 3 approaches correctly. They will not work reliably.
#
# Multiple CouchDBs running on the same machine can use couchdb1@, couchdb2@,
# etc.
-name [email protected]

# All nodes must share the same magic cookie for distributed Erlang to work.
# Uncomment the following line and append a securely generated random value.
-setcookie XXX

# Which interfaces should the node listen on?
-kernel inet_dist_use_interface {127,0,0,1}

# Tell kernel and SASL not to log anything
-kernel error_logger silent
-sasl sasl_error_logger false

# Use kernel poll functionality if supported by emulator
+K true

# Start a pool of asynchronous IO threads
+A 16

# Comment this line out to enable the interactive Erlang shell on startup
+Bd -noinput

# Force use of the smp scheduler, fixes #1296
-smp enable

# Set maximum SSL session lifetime to reap terminated replication readers
-ssl session_lifetime 300

## TLS Distribution
## Use TLS for connections between Erlang cluster members.
## http://erlang.org/doc/apps/ssl/ssl_distribution.html
##
## Generate Cert(PEM) File
## This is just an example command to generate a certfile (PEM).
## This is not an endorsement of specific expiration limits, key sizes, or algorithms.
##    $ openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem
##    $ cat key.pem cert.pem > dev/erlserver.pem && rm key.pem cert.pem
##
## Generate a Config File (couch_ssl_dist.conf)
##    [{server,
##      [{certfile, "</path/to/erlserver.pem>"},
##       {secure_renegotiate, true}]},
##     {client,
##      [{secure_renegotiate, true}]}].
##
## CouchDB recommends the following values for no_tls flag:
## 1. Use TCP only, set to true, such as:
##      -couch_dist no_tls true
## 2. Use TLS only, set to false, such as:
##      -couch_dist no_tls false
## 3. Specify which node to use TCP, such as:
##      -couch_dist no_tls \"*@127.0.0.1\"
##
## To ensure search works, make sure to set 'no_tls' option for the clouseau node.
## By default that would be "[email protected]".
## Don't forget to override the paths to point to your certificate(s) and key(s)!
##
#-proto_dist couch
#-couch_dist no_tls \"[email protected]\"
#-ssl_dist_optfile <path/to/couch_ssl_dist.conf>

[TomBayne]

was working fine before the 3.2.2 update.

[big-r81]

great, looks good so far with the security patches...

It's a single node?
Are you sure that no other couchdb is running while you start?

[TomBayne]

Received error:

dietpi@DietPi:~$ strace -e bind epmd -d -address 127.0.0.1
strace: Can't stat 'epmd': No such file or directory

Tried getting the epmd package from apt, but it was not found. I don't think I’ve ever even heard of epmd. Is it a package?

[lostnet]

epmd is part of erlang which should be bundled as erts-[version#] in couchdb's install, so it should have a path like couchdb-*/erts-*/bin/epmd

[TomBayne]

Ok got it. Command outputs below:

dietpi@DietPi:/opt/couchdb/erts-11.2.2.9/bin$ strace -e bind ./epmd -d -address 127.0.0.1
epmd: Tue Apr 26 09:51:35 2022: epmd running - daemon = 0
bind(3, {sa_family=AF_INET, sin_port=htons(4369), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
bind(4, {sa_family=AF_INET6, sin6_port=htons(4369), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address)
epmd: Tue Apr 26 09:51:35 2022: failed to bind on ipaddr 127.0.0.1: Cannot assign requested address
+++ exited with 1 +++

dietpi@DietPi:/opt/couchdb/erts-11.2.2.9/bin$ strace -e bind ./epmd -d -address 0.0.0.0
epmd: Tue Apr 26 09:53:10 2022: epmd running - daemon = 0
bind(3, {sa_family=AF_INET, sin_port=htons(4369), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
bind(4, {sa_family=AF_INET6, sin6_port=htons(4369), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address)
epmd: Tue Apr 26 09:53:10 2022: failed to bind on ipaddr [REDACTED]: Cannot assign requested address
+++ exited with 1 +++

dietpi@DietPi:/opt/couchdb/erts-11.2.2.9/bin$ strace -e bind ./epmd -d -port 4370 -address 127.0.0.1
epmd: Tue Apr 26 09:54:31 2022: epmd running - daemon = 0
bind(3, {sa_family=AF_INET, sin_port=htons(4370), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
bind(4, {sa_family=AF_INET6, sin6_port=htons(4370), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_scope_id=0}, 28) = -1 EADDRNOTAVAIL (Cannot assign requested address)
epmd: Tue Apr 26 09:54:31 2022: failed to bind on ipaddr 127.0.0.1: Cannot assign requested address
+++ exited with 1 +++

Seems like its just totally failing to assign anything? Permission issue potentially?

[lostnet]

Here it looks like only ipv6 is unhappy about binding on the loopback.. can you share the ifconfig output for your loopback interface(s)?

[TomBayne]

I currently have IPv6 disabled. Is it required for couchdb?

[lostnet]

3.2.2 enables ERL_EPMD_ADDRESS=127.0.0.1 as a security measure which will actually cause the versions of erlang used by couchdb to require ipv4+ipv6. (epmd always tries to listen to loopbacks if an address is set.) erlang/otp@e8c9e12

In the distribution you are using I think ERL_EPMD_ADDRESS is probably set in /etc/default/couchdb, so you could comment it out and still have the main security improvements, but remote attacker could still probe the epmd service if you are not blocking remote connections with a firewall.

[TomBayne]

You seem to have figured it out here with IPv6. However commenting that line seems to not totally disable IPv6 in couchdb. Enabling IPv6 on the system fixes the issue but I don’t want IPv6 enabled.

[Kaspanoombro]

/etc/default/couchdb disabling ERL_EPMD_ADDRESS=127.0.0.1
I have the stand alone server configuration but i enabled a SSH connection with a second server, and i make the replication to it. Can i still block in the firewall the erl?

[TomBayne]

Commented that part out, i’m now getting the following error. I will enabled IPv6 to confirm it is the cause, but I really don’t want IPv6 enabled if possible. ``` Apr 28 13:28:01 DietPi systemd[1]: Started Apache CouchDB. Apr 28 13:28:02 DietPi couchdb[616174]: {"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{\"Failed to create cookie file '/opt/couchdb/.erlang.cookie': eacces\",[{auth,init_cookie,0,[{file,\"auth.erl\"},{line,290}]},{auth,init,1,[{file,\"auth.erl\"},{line,144}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,417}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,385}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}}}}},{kernel,start,[normal,[]]}}}"} Apr 28 13:28:02 DietPi couchdb[616174]: Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{"Failed to create cookie file '/opt/c Apr 28 13:28:02 DietPi couchdb[616174]: Apr 28 13:28:02 DietPi couchdb[616174]: Crash dump is being written to: erl_crash.dump... Apr 28 13:28:02 DietPi systemd[1]: couchdb.service: Main process exited, code=exited, status=1/FAILURE Apr 28 13:28:02 DietPi systemd[1]: couchdb.service: Failed with result 'exit-code'. Apr 28 13:28:02 DietPi systemd[1]: couchdb.service: Scheduled restart job, restart counter is at 5. Apr 28 13:28:02 DietPi systemd[1]: Stopped Apache CouchDB. Apr 28 13:28:02 DietPi systemd[1]: couchdb.service: Start request repeated too quickly. Apr 28 13:28:02 DietPi systemd[1]: couchdb.service: Failed with result 'exit-code'. Apr 28 13:28:02 DietPi systemd[1]: Failed to start Apache CouchDB. ```

[TomBayne]

Okay, confirmed. IPv6 is the issue, as soon as it is enabled couchdb works as expected.