500 error on queue get and publish from management UI

[kfreem02]

Describe the bug

Occurs with LDAP authN and http authZ. Started with version 3.11.8 through current 3.11.13, also present in 3.12-beta6 (did not attempt prior betas). Tested with classic mirrored and quorum queues.

This error does not occur on 3.10.6, or 3.11.7 (did not test earlier versions of 3.11).

VM: Ubuntu jammy, 2 cores, 16GB ram, erlang 25.2.3

Reproduction steps

• auth: configure LDAP authN, http authZ level 1, internal level 2
• log into UI with LDAP user
• create queue
• Click Get on empty queue (500 error)
• Attempt to Publish a message (500 error)
• log out
• log in with local user
• Click Get on empty queue (no error)
• Attempt to Publish a message (no error, queue now contains 1 message)
• log out
• log in with LDAP user
• Click Get on queue (500 error)

Expected behavior

The UI should not return 500 errors

Additional context

Our local authZ http config:
$ cat /etc/nginx/conf.d/rabbitmq-http-auth.conf
# rabbitmq-http-auth

server {
    listen  127.0.0.1:8095;

    server_name localhost;

    error_log /var/log/nginx/nginx.vhost.error.auth.log;
    #access_log /var/log/nginx/nginx.vhost.access.auth.log;

    access_log off;

    return 200 allow;
}

Broker config:

$ cat /etc/rabbitmq/rabbitmq.conf
#
#RabbitMQ Configuration
#

#GLOBAL
reverse_dns_lookups = true
collect_statistics_interval = 30000
disk_free_limit.absolute = 1000000000
vm_memory_high_watermark.relative = 0.80
classic_queue.default_version = 1

#LOGGING
log.dir = /data/RabbitMQ/logs
log.file = rabbit.log
log.file.level = info
log.file.formatter = json
log.file.rotation.date = $D0
log.file.rotation.count = 7

#AUTHENTICATION - CACHE + INTERNAL + LDAP + HTTP
auth_backends.1.authn = ldap
auth_backends.1.authz = http
auth_backends.2 = internal
auth_ldap.servers.1 = ldap-east.dv.Domainlocal.com
auth_ldap.dn_lookup_bind = as_user
auth_ldap.user_dn_pattern = ${username}@dvDomain.local
auth_ldap.dn_lookup_attribute = userPrincipalName
auth_ldap.dn_lookup_base = ou=Org,dc=dvDomain,dc=local
auth_ldap.port = 636
auth_ldap.use_ssl = true
auth_ldap.log = false
auth_ldap.timeout = 5000
auth_ldap.connection_pool_size = 128
auth_ldap.idle_timeout = 120000

auth_http.http_method   = post
auth_http.user_path     = http://localhost:8095/rabbitmq-auth/user
auth_http.vhost_path    = http://localhost:8095/rabbitmq-auth/vhost
auth_http.resource_path = http://localhost:8095/rabbitmq-auth/resource
auth_http.topic_path    = http://localhost:8095/rabbitmq-auth/topic

auth_mechanisms.1 = PLAIN
auth_mechanisms.2 = AMQPLAIN

#AMPQ - NON-TLS
listeners.tcp.1 = 5672
num_acceptors.tcp = 25

#AMPQ - TLS
listeners.ssl.default = 5671
num_acceptors.ssl = 25

ssl_options.cacertfile = /etc/ssl/Domain-TrustedCA/Public-Provider-CA-Chain.crt
ssl_options.certfile   = /etc/ssl/Domain/DVDomainLOCALCOM.pem
ssl_options.keyfile    = /etc/ssl/Domain/DVDomainLOCALCOM.key
ssl_options.verify     = verify_none
ssl_options.depth      = 2
ssl_options.fail_if_no_peer_cert = false

ssl_options.honor_cipher_order   = true
ssl_options.honor_ecc_order      = true
ssl_options.client_renegotiation = false
ssl_options.secure_renegotiate   = true

ssl_options.versions.1 = tlsv1.2

ssl_options.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.3 = ECDHE-ECDSA-AES256-SHA384
ssl_options.ciphers.4 = ECDHE-RSA-AES256-SHA384
ssl_options.ciphers.5 = ECDH-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.6 = ECDH-RSA-AES256-GCM-SHA384
ssl_options.ciphers.7 = ECDH-ECDSA-AES256-SHA384
ssl_options.ciphers.8 = ECDH-RSA-AES256-SHA384
ssl_options.ciphers.9 = DHE-RSA-AES256-GCM-SHA384

#MANAGMENT - NON-TLS
management.tcp.port       = 15672
management.tcp.idle_timeout = 120000
management.tcp.inactivity_timeout = 120000
management.tcp.request_timeout = 120000

#management.http_log_dir = /data/RabbitMQ/logs/management
management.login_session_timeout = 90
management.rates_mode = basic
management.login_session_timeout = 60

#MANAGMENT - TLS
management.ssl.port       = 15671
management.ssl.cacertfile = /etc/ssl/Domain-TrustedCA/Public-Provider-CA-Chain.crt
management.ssl.certfile   = /etc/ssl/Domain/DVDomainLOCALCOM.pem
management.ssl.keyfile    = /etc/ssl/Domain/DVDomainLOCALCOM.key

management.ssl.honor_cipher_order   = true
management.ssl.honor_ecc_order      = true
management.ssl.client_renegotiation = false
management.ssl.secure_renegotiate   = true

management.ssl.versions.1 = tlsv1.2

management.ssl.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384
management.ssl.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.3 = ECDHE-ECDSA-AES256-SHA384
management.ssl.ciphers.4 = ECDHE-RSA-AES256-SHA384
management.ssl.ciphers.5 = ECDH-ECDSA-AES256-GCM-SHA384
management.ssl.ciphers.6 = ECDH-RSA-AES256-GCM-SHA384
management.ssl.ciphers.7 = ECDH-ECDSA-AES256-SHA384
management.ssl.ciphers.8 = ECDH-RSA-AES256-SHA384
management.ssl.ciphers.9 = DHE-RSA-AES256-GCM-SHA384

management.ssl.idle_timeout       = 120000
management.ssl.inactivity_timeout = 120000
management.ssl.request_timeout    = 10000

#PROMETHEUS PLUGIN - COMMON
prometheus.return_per_object_metrics = true

#PROMETHEUS PLUGIN - NON-TLS
#prometheus.tcp.port       = 15692
#prometheus.tcp.idle_timeout = 120000
#prometheus.tcp.inactivity_timeout = 120000
#rometheus.tcp.request_timeout = 120000

#PROMETHEUS PLUGIN - TLS
prometheus.ssl.port       = 15691
prometheus.ssl.cacertfile = /etc/ssl/Domain-TrustedCA/Public-Provider-CA-Chain.crt
prometheus.ssl.certfile   = /etc/ssl/Domain/DVDomainLOCALCOM.pem
prometheus.ssl.keyfile    = /etc/ssl/Domain/DVDomainLOCALCOM.key

prometheus.ssl.versions.1 = tlsv1.2

prometheus.ssl.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384
prometheus.ssl.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
prometheus.ssl.ciphers.3 = ECDHE-ECDSA-AES256-SHA384
prometheus.ssl.ciphers.4 = ECDHE-RSA-AES256-SHA384
prometheus.ssl.ciphers.5 = ECDH-ECDSA-AES256-GCM-SHA384
prometheus.ssl.ciphers.6 = ECDH-RSA-AES256-GCM-SHA384
prometheus.ssl.ciphers.7 = ECDH-ECDSA-AES256-SHA384
prometheus.ssl.ciphers.8 = ECDH-RSA-AES256-SHA384
prometheus.ssl.ciphers.9 = DHE-RSA-AES256-GCM-SHA384

prometheus.ssl.idle_timeout = 120000
prometheus.ssl.inactivity_timeout = 120000
prometheus.ssl.request_timeout = 120000

Log entries, 2 errors for each 500 error:

    crasher:
    initial call: cowboy_stream_h:request_process/3
    pid: <0.10862.64>
    registered_name: []
    exception error: no case clause matching
                     {error,
                      {{'EXIT',
                        {badarg,
                         [{rabbit_http_util,quote_plus,2,
                           [{file,\"rabbit_http_util.erl\"},
                            {line,190},
                            {error_info,
                             #{cause => {1,integer,type,\"unknown\"},
                               function => format_bs_fail,
                               module => erl_erts_errors}}]},
                          {rabbit_auth_backend_http,escape,2,
                           [{file,\"rabbit_auth_backend_http.erl\"},{line,188}]},
                          {rabbit_auth_backend_http,'-q/1-lc$^0/1-0-',1,
                           [{file,\"rabbit_auth_backend_http.erl\"},{line,182}]},
                          {rabbit_auth_backend_http,'-q/1-lc$^0/1-0-',1,
                           [{file,\"rabbit_auth_backend_http.erl\"},{line,182}]},
                          {rabbit_auth_backend_http,q,1,
                           [{file,\"rabbit_auth_backend_http.erl\"},{line,182}]},
                          {rabbit_auth_backend_http,bool_req,2,
                           [{file,\"rabbit_auth_backend_http.erl\"},{line,107}]},
                          {rabbit_access_control,check_access,5,
                           [{file,\"rabbit_access_control.erl\"},{line,213}]},
                          {lists,foldl,3,[{file,\"lists.erl\"},{line,1350}]}]}},
                       'rabbit@DVE-BROKER-VM3'}}
      in function  rabbit_mgmt_util:with_channel/5 (rabbit_mgmt_util.erl, line 1003)
      in call from rabbit_mgmt_util:with_decode/5 (rabbit_mgmt_util.erl, line 850)
      in call from rabbit_mgmt_wm_exchange_publish:accept_content/2 (rabbit_mgmt_wm_exchange_publish.erl, line 42)
      in call from cowboy_rest:call/3 (src/cowboy_rest.erl, line 1575)
      in call from cowboy_rest:process_content_type/3 (src/cowboy_rest.erl, line 1096)
      in call from cowboy_rest:upgrade/4 (src/cowboy_rest.erl, line 284)
      in call from cowboy_stream_h:execute/3 (src/cowboy_stream_h.erl, line 300)
      in call from cowboy_stream_h:request_process/3 (src/cowboy_stream_h.erl, line 291)
    ancestors: [<0.9825.64>,<0.1134.0>,<0.1130.0>,<0.1129.0>,<0.1127.0>,
                  rabbit_web_dispatch_sup,<0.1091.0>]
    message_queue_len: 0
    messages: []
    links: [<0.9825.64>]
    dictionary: [{gen_server_call_timeout,130000},
                  {{xtype_to_module,direct},rabbit_exchange_type_direct}]
    trap_exit: false
    status: running
    heap_size: 2586
    stack_size: 28
    reductions: 8757
  neighbours:

Ranch listener {acceptor,{0,0,0,0,0,0,0,0},15671}, connection process <0.9825.64>, stream 28 had its request process <0.10862.64> exit with reason {case_clause,{error,{{'EXIT',{badarg,[{rabbit_http_util,quote_plus,2,[{file,\"rabbit_http_util.erl\"},{line,190},{error_info,#{cause => {1,integer,type,\"unknown\"},function => format_bs_fail,module => erl_erts_errors}}]},{rabbit_auth_backend_http,escape,2,[{file,\"rabbit_auth_backend_http.erl\"},{line,188}]},{rabbit_auth_backend_http,'-q/1-lc$^0/1-0-',1,[{file,\"rabbit_auth_backend_http.erl\"},{line,182}]},{rabbit_auth_backend_http,'-q/1-lc$^0/1-0-',1,[{file,\"rabbit_auth_backend_http.erl\"},{line,182}]},{rabbit_auth_backend_http,q,1,[{file,\"rabbit_auth_backend_http.erl\"},{line,182}]},{rabbit_auth_backend_http,bool_req,2,[{file,\"rabbit_auth_backend_http.erl\"},{line,107}]},{rabbit_access_control,check_access,5,[{file,\"rabbit_access_control.erl\"},{line,213}]},{lists,foldl,3,[{file,\"lists.erl\"},{line,1350}]}]}},'rabbit@DVE-BROKER-VM3'}}} and stacktrace [{rabbit_mgmt_util,with_channel,5,[{file,\"rabbit_mgmt_util.erl\"},{line,1003}]},{rabbit_mgmt_util,with_decode,5,[{file,\"rabbit_mgmt_util.erl\"},{line,850}]},{rabbit_mgmt_wm_exchange_publish,accept_content,2,[{file,\"rabbit_mgmt_wm_exchange_publish.erl\"},{line,42}]},{cowboy_rest,call,3,[{file,\"src/cowboy_rest.erl\"},{line,1575}]},{cowboy_rest,process_content_type,3,[{file,\"src/cowboy_rest.erl\"},{line,1096}]},{cowboy_rest,upgrade,4,[{file,\"src/cowboy_rest.erl\"},{line,284}]},{cowboy_stream_h,execute,3,[{file,\"src/cowboy_stream_h.erl\"},{line,300}]},{cowboy_stream_h,request_process,3,[{file,\"src/cowboy_stream_h.erl\"},{line,291}]}]"

[eibbojtir]

We recently patched to 3.11.11 (with Erlang 25.3) on RHEL 8.7 and encountered the same issue when trying to Publish or Get messages from the Web UI. My logs are identical to the above case starting with crasher cowboy_stream_h.

[lukebakken]

Does your LDAP user account contain any "unusual" characters - anything other than what would match the [0-9a-zA-Z] regex character class?

Just putting this here so I don't forget - check to see if 3.11.7 -> .8 is when we switched to thoas (cc @michaelklishin)

[eibbojtir]

Does your LDAP user account contain any "unusual" characters - anything other than what would match the [0-9a-zA-Z] regex character class?

Just putting this here so I don't forget - check to see if 3.11.7 -> .8 is when we switched to thoas (cc @michaelklishin)

Negative. I can reproduce with an account entire contained in [a-z], but the wildest character in my env is underscore or hyphen.

[lukebakken]

Great thanks! I'm going to start by setting up an environment using the HTTP auth backend. I'd be surprised if using LDAP and HTTP auth combined is necessary.

[lukebakken]

@eibbojtir how are you granting the management or administrator tag to your LDAP users so that they can log into the UI? I don't see how it's possible given your configuration.

In my test environment I have my HTTP autz backend configured to return tags, but I think I've found a bug where they aren't actually used correctly -

• https://github.com/lukebakken/rabbitmq-server-7864
• PR - Special case "unknown" for IP address #7961

[lukebakken]

@eibbojtir - I moved my LDAP settings to advanced.config so that I can set the tags for users based on LDAP, and I can reproduce your issue.

I'm still very curious how you are setting tags for your LDAP users so that they can log into the management UI.

[kfreem02]

Redacted advanced.config

$ cat /etc/rabbitmq/advanced.config
[
 {rabbit, [
   {credit_flow_default_credit,{25000,200}}
 ]},
  {rabbitmq_auth_backend_ldap,[
     {ssl_options, [{cacertfile,"/etc/ssl/TrustedCA/Public-GoDaddy-CA-Chain.crt"},
                    {verify, verify_none},
                    {fail_if_no_peer_cert, false}]},
     {group_lookup_base, "ou=Groups,ou=Org,dc=dv,dc=local,dc=com"},
     {tag_queries, [
     {administrator, {in_group, "cn=RabbitMQAdmin,ou=Groups,ou=Org,dc=dv,dc=local,dc=com"}},
     {management, {'or', [
        {in_group, "cn=RabbitMQAdmin,ou=Groups,ou=Org,dc=dv,dc=local,dc=com"},
        {in_group, "cn=RabbitMQReadOnly,ou=Groups,ou=Org,dc=dv,dc=local,dc=com"}
     ]}},
     {monitoring, {in_group, "cn=RabbitMQReadOnly,ou=Groups,ou=Org,dc=dv,dc=local,dc=com"}}
   ]}
 ]},
 {lager, [
        {error_logger_hwm, 5000}
 ]}
].

[lukebakken]

@kfreem02 - thanks for the information.

These are the PRs that introduced the bug:

#6952
#6953

It must be this change ... confirming now. Who would have thought?

cc @binarin

[michaelklishin]

@kfreem02 can you please give this alpha build of 3.12 a shot in your environment? We'd like to confirm that the issue is gone.

It will also ship in 3.11.14 and 3.10.21 but right now that's the only fresh alpha build we've got.

[kfreem02]

Confirmed fixed in alpha 139:
3.11.7 fresh install, get/publish succeed with LDAP user
upgraded to 3.11.13, get/publish fail with LDAP user
upgraded to alpha 139, get/publish succeed with LDAP user