Error I don't understand in logs: rexi_server error "fabric_rpc:all_docs/3 exit:{killed..."

[gilbrechbuhler]

Hello,

I have an issue in a CouchDB cluster. I have regularly (once every hour approximatly) those kind of CouchDB logs appearing on the nodes :

[error] 2021-03-30T11:37:42.194435Z couchdb@node01 <0.11765.284> 8def92f9a5 rexi_server: from: couchdb@node02(<6768.3349.302>) mfa: fabric_rpc:all_docs/3 exit:{killed,{gen_server,call,[<0.20293.279>,bytes,infinity]}} [{gen_server,call,3,[{file,"gen_server.erl"},{line,214}]},{couch_bt_engine,get_size_info,1,[{file,"src/couch_bt_engine.erl"},{line,256}]},{couch_db,get_db_info,1,[{file,"src/couch_db.erl"},{line,574}]},{couch_mrview,'-query_all_docs/4-fun-0-',1,[{file,"src/couch_mrview.erl"},{line,222}]},{couch_mrview,query_all_docs,4,[{file,"src/couch_mrview.erl"},{line,221}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}]

I spent a lot of time researching this over Internet and trying to tweak every configuration I thought would be make sense for this issue. But I could not find a solution. I found information on similar issues but with timeouts (here for example #1008), but here it looks to be a different problem as the process gets killed and there is no timeout in the logs.

The cluster is made of 3 CouchDB nodes (running on LXC containers) with 8 CPU and 8 GB of RAM for each containers. The nodes are in different datacenters but the datacenters are connected together with a 1GB/s link.
OS: Debian 10
CouchDB version: 2.3.1

Here is the content of my /opt/couchdb/etc/local.d/local.ini:

[admins]
<a_username>=<a_certain_password>

[chttpd]
bind_address=0.0.0.0
authentication_handlers={chttpd_auth, proxy_authentication_handler},{chttpd_auth, default_authentication_handler}

[httpd]
bind_address=0.0.0.0

[couchdb]
uuid = <some_uuid>

[cluster]
q=3
n=3

[couch_httpd_auth]
proxy_use_secret=true
require_valid_user=true
secret=<some_secret>
x_auth_token=X-Auth-CouchDB-Token
x_auth_username=X-Forwarded-User
x_auth_roles=X-Auth-CouchDB-Roles

[log]
level = warning

[fabric]
request_timeout = infinity

[couchdb]
os_process_timeout = 30000
max_dbs_open = 1000

[compactions]
_default = [{db_fragmentation, "70%"}, {view_fragmentation, "60%"}, {from, "01:00"}, {to, "04:00"}]

[query_server_config]
os_process_limit = 1500
os_process_soft_limit = 750

[replicator]
worker_processes = 6
worker_batch_size = 1000
connection_timeout = 60000

Also the content of the /opt/couchdb/etc/vm.args:

-name couchdb@node01

# All nodes must share the same magic cookie for distributed Erlang to work.
# Comment out this line if you synchronized the cookies by other means (using
# the ~/.erlang.cookie file, for example).
-setcookie monster

# Tell kernel and SASL not to log anything
-kernel error_logger silent
-sasl sasl_error_logger false

# Use kernel poll functionality if supported by emulator
+K true

# Start a pool of asynchronous IO threads
+A 16

# Comment this line out to enable the interactive Erlang shell on startup
+Bd -noinput

# Force use of the smp scheduler, fixes #1296
-smp enable

# Set maximum SSL session lifetime to reap terminated replication readers
-ssl session_lifetime 300
-kernel inet_dist_listen_min 5988
-kernel inet_dist_listen_max 5988

> curl http://127.0.0.1:5984/_memberships
{"all_nodes":["couchdb@nodes01","couchdb@node02","couchdb@node03"],"cluster_nodes":["couchdb@node01","couchdb@node02","couchdb@node03"]}

So all nodes seem to be present and visible in the cluster.

Configuration is strictly identical on all three nodes.

I'm a bit lost on this issue, I tried setting the log level to debug to get more information but there is a lot of loads on these CouchDB so I could not really get anything usable.

Has anyone seen this error message before ? Does anyone know what I can do to solve this or what could be responsible of killing the process ?

Thanks a lot in advance for any help.