Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: refactor one-time tokens for performance #1558

Merged
merged 2 commits into from
May 6, 2024
Merged

Conversation

hf
Copy link
Contributor

@hf hf commented Apr 28, 2024

Refactors all One-Time Tokens (OTP) used for sign-in with email, SMS, email confirmation, phone confirmation, change... to achieve:

  • Performance (as current method does not use an index due to the use of partial indexes which cannot be used in practice)
  • Future enhancements (such as OTP verification counters, adaptive OTP lengths, etc.)

Summary of the change:

  • A new one_time_tokens table is added which uses a double-write mechanism with users.
  • Each new OTP is both written in the corresponding users column and as a new row in one_time_tokens.
  • Lookup for an OTP hash is performed first in one_time_tokens and if not found, using the traditional users approach.
  • In a few days, once all OTPs using the users columns have expired, a new change will be deployed which removes the users lookup. This completely solves the performance issue for looking up OTPs.
  • In a future change, the one_time_tokens table can be used to add a verification counter based on lookups on the relates_to (email or phone number) column, enabling new security features.

@hf hf requested a review from a team as a code owner April 28, 2024 14:15
@hf hf force-pushed the hf/add-one-time-tokens branch 2 times, most recently from 988959a to 46d4d44 Compare April 28, 2024 14:24
@coveralls
Copy link

coveralls commented Apr 28, 2024

Pull Request Test Coverage Report for Build 8970405577

Details

  • 202 of 339 (59.59%) changed or added relevant lines in 6 files are covered.
  • 5 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-0.2%) to 65.474%

Changes Missing Coverage Covered Lines Changed/Added Lines %
internal/api/phone.go 9 17 52.94%
internal/api/verify.go 9 17 52.94%
internal/models/user.go 16 41 39.02%
internal/api/mail.go 28 62 45.16%
internal/models/one_time_token.go 138 200 69.0%
Files with Coverage Reduction New Missed Lines %
internal/models/user.go 2 75.22%
internal/api/mail.go 3 60.82%
Totals Coverage Status
Change from base Build 8906588569: -0.2%
Covered Lines: 8365
Relevant Lines: 12776

💛 - Coveralls

internal/api/mail.go Outdated Show resolved Hide resolved
Copy link
Member

@kangmingtay kangmingtay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While testing the different OTP flows, I noticed that when secure email change is disabled, a row is created in one_time_tokens for email_change_token_current but no token_hash is present.

We should only be creating a one_time_token for email_change_token_new and not for email_change_token_current

internal/models/one_time_token.go Show resolved Hide resolved
internal/models/one_time_token.go Outdated Show resolved Hide resolved
internal/models/one_time_token.go Show resolved Hide resolved
@hf
Copy link
Contributor Author

hf commented May 3, 2024

While testing the different OTP flows, I noticed that when secure email change is disabled, a row is created in one_time_tokens for email_change_token_current but no token_hash is present.

Right this is because the code maps 1:1 to the column. Because we set the column to be '' it also ends up in one_time_tokens. I'll fix that.

@hf hf force-pushed the hf/add-one-time-tokens branch 4 times, most recently from 90c8f3a to 976ef9e Compare May 3, 2024 11:05
@hf hf force-pushed the hf/add-one-time-tokens branch from 976ef9e to c556650 Compare May 3, 2024 12:45
internal/models/one_time_token.go Show resolved Hide resolved
internal/models/one_time_token.go Show resolved Hide resolved
internal/models/user.go Show resolved Hide resolved
internal/models/user.go Show resolved Hide resolved
internal/models/one_time_token.go Show resolved Hide resolved
@kangmingtay
Copy link
Member

While testing the different OTP flows, I noticed that when secure email change is disabled, a row is created in one_time_tokens for email_change_token_current but no token_hash is present.

Right this is because the code maps 1:1 to the column. Because we set the column to be '' it also ends up in one_time_tokens. I'll fix that.

yeah i think it's better if we simply don't insert a row into one_time_tokens if the token_hash is empty

internal/api/mail.go Outdated Show resolved Hide resolved
internal/api/mail.go Outdated Show resolved Hide resolved
Co-authored-by: Joel Lee <[email protected]>
@hf hf merged commit d1cf8d9 into master May 6, 2024
2 checks passed
@hf hf deleted the hf/add-one-time-tokens branch May 6, 2024 17:51
hf pushed a commit that referenced this pull request May 6, 2024
🤖 I have created a release *beep* *boop*
---


##
[2.151.0](v2.150.1...v2.151.0)
(2024-05-06)


### Features

* refactor one-time tokens for performance
([#1558](#1558))
([d1cf8d9](d1cf8d9))


### Bug Fixes

* do call send sms hook when SMS autoconfirm is enabled
([#1562](#1562))
([bfe4d98](bfe4d98))
* format test otps
([#1567](#1567))
([434a59a](434a59a))
* log final writer error instead of handling
([#1564](#1564))
([170bd66](170bd66))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
kangmingtay added a commit that referenced this pull request May 17, 2024
…1569)

Removes legacy lookups in `auth.users` for when a corresponding entry in
`one_time_tokens` is not found.

Phase II of the refactor, based on #1558, to be released after it's
deployed for a few days.

---------

Co-authored-by: Kang Ming <[email protected]>
uxodb pushed a commit to uxodb/auth that referenced this pull request Nov 13, 2024
Refactors all One-Time Tokens (OTP) used for sign-in with email, SMS,
email confirmation, phone confirmation, change... to achieve:

- Performance (as current method does not use an index due to the use of
[partial
indexes](https://github.com/supabase/auth/blob/master/migrations/20220429102000_add_unique_idx.up.sql#L10-L14)
which [cannot be used in
practice](https://www.postgresql.org/docs/current/indexes-partial.html))
- Future enhancements (such as OTP verification counters, adaptive OTP
lengths, etc.)

Summary of the change:

- A new `one_time_tokens` table is added which uses a double-write
mechanism with `users`.
- Each new OTP is both written in the corresponding `users` column and
as a new row in `one_time_tokens`.
- Lookup for an OTP hash is performed first in `one_time_tokens` and if
not found, using the traditional `users` approach.
- In a few days, once all OTPs using the `users` columns have expired, a
new change will be deployed which removes the `users` lookup. This
completely solves the performance issue for looking up OTPs.
- In a future change, the `one_time_tokens` table can be used to add a
verification counter based on lookups on the `relates_to` (email or
phone number) column, enabling new security features.

---------

Co-authored-by: Joel Lee <[email protected]>
uxodb pushed a commit to uxodb/auth that referenced this pull request Nov 13, 2024
🤖 I have created a release *beep* *boop*
---


##
[2.151.0](supabase/auth@v2.150.1...v2.151.0)
(2024-05-06)


### Features

* refactor one-time tokens for performance
([supabase#1558](supabase#1558))
([d1cf8d9](supabase@d1cf8d9))


### Bug Fixes

* do call send sms hook when SMS autoconfirm is enabled
([supabase#1562](supabase#1562))
([bfe4d98](supabase@bfe4d98))
* format test otps
([supabase#1567](supabase#1567))
([434a59a](supabase@434a59a))
* log final writer error instead of handling
([supabase#1564](supabase#1564))
([170bd66](supabase@170bd66))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
uxodb pushed a commit to uxodb/auth that referenced this pull request Nov 13, 2024
…upabase#1569)

Removes legacy lookups in `auth.users` for when a corresponding entry in
`one_time_tokens` is not found.

Phase II of the refactor, based on supabase#1558, to be released after it's
deployed for a few days.

---------

Co-authored-by: Kang Ming <[email protected]>
LashaJini pushed a commit to LashaJini/auth that referenced this pull request Nov 13, 2024
Refactors all One-Time Tokens (OTP) used for sign-in with email, SMS,
email confirmation, phone confirmation, change... to achieve:

- Performance (as current method does not use an index due to the use of
[partial
indexes](https://github.com/supabase/auth/blob/master/migrations/20220429102000_add_unique_idx.up.sql#L10-L14)
which [cannot be used in
practice](https://www.postgresql.org/docs/current/indexes-partial.html))
- Future enhancements (such as OTP verification counters, adaptive OTP
lengths, etc.)

Summary of the change:

- A new `one_time_tokens` table is added which uses a double-write
mechanism with `users`.
- Each new OTP is both written in the corresponding `users` column and
as a new row in `one_time_tokens`.
- Lookup for an OTP hash is performed first in `one_time_tokens` and if
not found, using the traditional `users` approach.
- In a few days, once all OTPs using the `users` columns have expired, a
new change will be deployed which removes the `users` lookup. This
completely solves the performance issue for looking up OTPs.
- In a future change, the `one_time_tokens` table can be used to add a
verification counter based on lookups on the `relates_to` (email or
phone number) column, enabling new security features.

---------

Co-authored-by: Joel Lee <[email protected]>
LashaJini pushed a commit to LashaJini/auth that referenced this pull request Nov 13, 2024
🤖 I have created a release *beep* *boop*
---


##
[2.151.0](supabase/auth@v2.150.1...v2.151.0)
(2024-05-06)


### Features

* refactor one-time tokens for performance
([supabase#1558](supabase#1558))
([d1cf8d9](supabase@d1cf8d9))


### Bug Fixes

* do call send sms hook when SMS autoconfirm is enabled
([supabase#1562](supabase#1562))
([bfe4d98](supabase@bfe4d98))
* format test otps
([supabase#1567](supabase#1567))
([434a59a](supabase@434a59a))
* log final writer error instead of handling
([supabase#1564](supabase#1564))
([170bd66](supabase@170bd66))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
LashaJini pushed a commit to LashaJini/auth that referenced this pull request Nov 13, 2024
…upabase#1569)

Removes legacy lookups in `auth.users` for when a corresponding entry in
`one_time_tokens` is not found.

Phase II of the refactor, based on supabase#1558, to be released after it's
deployed for a few days.

---------

Co-authored-by: Kang Ming <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants