Unable to login to Grafana

[aktech]

Context

This is what I see while trying to login to Grafana:

Logs:

➜  journalctl -u grafana-server.service -f
Jul 25 09:54:20 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:20.768233638-05:00 level=info msg="OAuth auto login enabled. Redirecting to /monitoring/login/generic_oauth"
Jul 25 09:54:20 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:20.768539919-05:00 level=info msg="Request Completed" method=GET path=/login status=307 remote_addr=192.166.211.218 time_ms=3 duration=3.14035ms size=67 referer= handler=/login status_source=server
Jul 25 09:54:20 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:20.873726215-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=0 duration=82.624µs size=306 referer= handler=/login/:name status_source=server
Jul 25 09:54:21 jhub-login grafana[986907]: logger=user.sync t=2024-07-25T09:54:21.10607955-05:00 level=error msg="Failed to create user" error="user already exists" auth_module=oauth_generic_oauth auth_id=4d4a9bc3-efe9-4e95-b1b9-3a54c89796fb
Jul 25 09:54:21 jhub-login grafana[986907]: logger=authn.service t=2024-07-25T09:54:21.106350655-05:00 level=error msg="Failed to run post auth hook" client=auth.client.generic_oauth id=: error="[user.sync.internal] unable to create user: user already exists"
Jul 25 09:54:21 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:21.112687295-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=20 duration=20.169282ms size=40 referer= handler=/login/:name status_source=server
Jul 25 09:54:33 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:33.319209421-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=0 duration=99.04µs size=306 referer=https://<NEBARI-SLURM-URL-REMOVED>/monitoring/login handler=/login/:name status_source=server
Jul 25 09:54:33 jhub-login grafana[986907]: logger=user.sync t=2024-07-25T09:54:33.550794803-05:00 level=error msg="Failed to create user" error="user already exists" auth_module=oauth_generic_oauth auth_id=4d4a9bc3-efe9-4e95-b1b9-3a54c89796fb
Jul 25 09:54:33 jhub-login grafana[986907]: logger=authn.service t=2024-07-25T09:54:33.550848423-05:00 level=error msg="Failed to run post auth hook" client=auth.client.generic_oauth id=: error="[user.sync.internal] unable to create user: user already exists"
Jul 25 09:54:33 jhub-login grafana[986907]: logger=context userId=0 orgId=0 uname= t=2024-07-25T09:54:33.557529344-05:00 level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=192.166.211.218 time_ms=19 duration=19.997124ms size=40 referer= handler=/login/:name status_source=server

Potentially relevant upstream issues:

• Error message "user already exists" during login if user email in Oauth provider was changed grafana/grafana#67438
• Authentication: After upgrading to Grafana 10 getting error User already exists with generic oAuth grafana/grafana#70203

Potential solution: grafana/grafana#70203 (comment) (or upgrade grafana)

Value and/or benefit

Being able to monitor nebari-slurm deployment.

Anything else?

No response

[lcapka]

Hi @aktech, were you able to solve this issue? We are having the same problem.

[aktech]

What version are you on @lcapka ?

[Heavybullets8]

I encountered a sign-in issue with Grafana due to a conflict between Grafana's default admin user and my LDAP admin user. (LDAP is used by my OAuth provider, Authelia)

• My LDAP User: admin
• My LDAP Group: admins

Grafana creates a default admin user upon initial setup, which conflicted with my LDAP admin user.

Note: Changing my LDAP username was an option, but I chose to adjust Grafana's configuration instead. Honestly not sure which is the correct method for solving this.

---

The Steps I Personally Took

1. Disable Grafana's Default admin User Creation

To prevent Grafana from creating the default admin user, I disabled the initial admin creation in the Grafana configuration.

Environment Variable (what I used):

GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION: true

Equivalent grafana.ini Configuration:

#################################### Security ############################
[security]
# Disable creation of admin user on first start of Grafana
disable_initial_admin_creation = true

---

2. Configure LDAP Admin User

Since my LDAP user admin is part of the admins group, I configured Grafana to assign the Admin role to users in the admins group.

Environment Variables (what I used):

GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: contains(groups, 'admins') && 'Admin' || 'Viewer'
GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN: true

Equivalent grafana.ini Configuration:

[auth.generic_oauth]
role_attribute_path = contains(groups, 'admins') && 'Admin' || 'Viewer'
allow_assign_grafana_admin = true

---

3. Reinitialize Grafana Database

To apply the changes, I needed to remove Grafana and reset the database.

• Steps:
  1. Remove Grafana: Stop and remove the Grafana instance.
  2. Drop the Grafana Database: Remove the existing Grafana database from CNPG (Cloud Native PostgreSQL).
  3. Re-add Grafana: Deploy Grafana again with the updated configuration.

Afterwards, everything came up for me. May or may not work for you depending on your setup.

---

Alternative Solutions Attempted

Some users have reported success with the following settings, but they did not work for me.

Environment Variables:

GF_AUTH_GENERIC_OAUTH_SKIP_ORG_ROLE_SYNC: true
GF_AUTH_OAUTH_ALLOW_INSECURE_EMAIL_LOOKUP: true

Equivalent grafana.ini Configuration:

[auth]
oauth_allow_insecure_email_lookup = true

[auth.generic_oauth]
skip_org_role_sync = true