Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validation service scaffold #320

Merged
merged 16 commits into from
Oct 24, 2023
Merged

Validation service scaffold #320

merged 16 commits into from
Oct 24, 2023

Conversation

neekolas
Copy link
Collaborator

Summary

  • Adds support for the Rust-backed Validation Service over GRPC
  • Adds database models for Installations and KeyPackages
  • Adds migrations for those database models
  • Adds Service to read/write KeyPackages

@neekolas neekolas marked this pull request as ready for review October 23, 2023 22:55
snormore
snormore reviewed Oct 24, 2023
View reviewed changes
pkg/api/message/v3/service.go Outdated Show resolved Hide resolved
pkg/api/message/v3/service.go Outdated Show resolved Hide resolved
pkg/mlsstore/store.go Show resolved Hide resolved
pkg/mlsstore/store.go Outdated Show resolved Hide resolved
pkg/mlsstore/store.go Show resolved Hide resolved
pkg/mlsvalidate/service.go Outdated Show resolved Hide resolved
pkg/server/options.go Outdated Show resolved Hide resolved
pkg/migrations/mls/20231023050806_init-schema.up.sql Outdated Show resolved Hide resolved
pkg/mlsvalidate/service_test.go Outdated Show resolved Hide resolved
.github/workflows/push-mls.yml
on:
push:
branches:
- mls
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the plan to keep this on the mls branch for now or merge into main? Noticed that the PR is opened against the mls branch too

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plan is to use the MLS branch for at least this week, and work out all the kinks with the schema. Then merge to main and set the appropriate server flags.

snormore
snormore approved these changes Oct 24, 2023
View reviewed changes
pkg/api/message/v3/service.go Outdated Show resolved Hide resolved
pkg/server/options.go Show resolved Hide resolved
pkg/testing/store.go Show resolved Hide resolved
@neekolas neekolas merged commit 07791ed into mls Oct 24, 2023
3 checks passed
@neekolas neekolas deleted the validation-service-scaffold branch October 24, 2023 16:14
richardhuaaa
richardhuaaa reviewed Oct 24, 2023
View reviewed changes
pkg/mlsstore/models.go
Comment on lines +5 to +12
type Installation struct {
bun.BaseModel `bun:"table:installations"`

ID string `bun:",pk"`
WalletAddress string `bun:"wallet_address,notnull"`
CreatedAt int64 `bun:"created_at,notnull"`
RevokedAt *int64 `bun:"revoked_at"`
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@neekolas Any reason we're storing the derived state (when an installation was created/revoked) rather than the registration/revocation payloads that were originally used to perform those actions? I think it's important that the client can validate this data themselves

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For registration, the only artifact is the last resort key package. Clients don't really have to trust the server that installations exist. They should trust the Key Packages provided, which can be individually validated on the client. If the server can't provide a valid key package for an installation, the installation should be ignored by clients.

It's a good call out on the revocation side. I'll update things so that we can store an artifact there.

snormore pushed a commit that referenced this pull request Jan 8, 2024
@neekolas
Validation service scaffold (#320)
8cfe572 
* Scaffold MLS server

* Update go.mod

* Fix missing argument

* Add unsaved file

* Lint

* Working end-to-end

* Lint

* Add new push action

* Address review comments

* Change method casing

* Change casing of server options

* Change casing of validation options

* Remove unused function

* Remove double pointer

* Make private again
snormore pushed a commit that referenced this pull request Jan 23, 2024
@neekolas
Validation service scaffold (#320)
3883391 
* Scaffold MLS server

* Update go.mod

* Fix missing argument

* Add unsaved file

* Lint

* Working end-to-end

* Lint

* Add new push action

* Address review comments

* Change method casing

* Change casing of server options

* Change casing of validation options

* Remove unused function

* Remove double pointer

* Make private again
snormore pushed a commit that referenced this pull request Jan 23, 2024
@neekolas
Validation service scaffold (#320)
3d22b24 
* Scaffold MLS server

* Update go.mod

* Fix missing argument

* Add unsaved file

* Lint

* Working end-to-end

* Lint

* Add new push action

* Address review comments

* Change method casing

* Change casing of server options

* Change casing of validation options

* Remove unused function

* Remove double pointer

* Make private again
neekolas added a commit that referenced this pull request Jan 23, 2024
@neekolas @richardhuaaa
@Bren2010 @snormore
Add hpke public key (#337)
2164303 
* Validation service scaffold (#320)

* Scaffold MLS server

* Update go.mod

* Fix missing argument

* Add unsaved file

* Lint

* Working end-to-end

* Lint

* Add new push action

* Address review comments

* Change method casing

* Change casing of server options

* Change casing of validation options

* Remove unused function

* Remove double pointer

* Make private again

* Add more MLS methods (#321)

* Scaffold MLS server

* Update go.mod

* Fix missing argument

* Add unsaved file

* Lint

* Working end-to-end

* Lint

* Add new push action

* Add a bunch of new endpoints

* Address review comments

* Change method casing

* Change casing of server options

* Change casing of validation options

* Remove unused function

* Remove double pointer

* Make private again

* Fix pointer to key package

* Capitalize more things

* Update server fields

* Add test for sort methods

* Save change to capitalization

* Fix lint warnings

* Fix problem with mocks

* Fix index name

* Move sorting to the store

* Fix ciphertext validation

* Make installation_id bytes

* Add missing credential identity

* Hack sql in query

* Revert "Hack sql in query"

This reverts commit 168b78a.

* Remove custom type

* Update to latest protos

* Add CredentialIdentity

* Fix migrations

* refactor: walletaddress->accountaddress (#325)

Settling on consistent terminology as discussed

* go mod tidy

* Assume KeyPackages are always last resort. (#329)

* Assume KeyPackages are always last resort.

* Missing KeyPackages don't cause entire request to fail.

* Update go.mod

* UpdateKeyPackage returns an error if installation is unknown.

* refactor: move api/message/v3 to mls/api/v1 (#332)

* refactor: move message_api/v3 to mls/api/v1

* refactor: move pkg/mlsstore to pkg/mls/store

* Rename mls build/push workflow

* Decouple MLS messages from messagev1 (#333)

* Separate MLS messages and implement service methods

* fix: group id and installation id are bytes

* fix: idempotent send group/welcome messages via uniquness in db

* fix: hex decode group id from mls validation service

* fix: s/Cursor/IdCursor

* fix: pass message data only in send group message request

* refactor: add mls {Group,Welcome}MessageInput types for send requests

* refactor: s/installation_id/installation_key in mls/api

* fix: clean up mls query page size logic

* feat: implement mls subscribe group/welcome messages

* Hex encode group ID

* fix: remove duplicate import

* fix: return grpc invalidargument on invalid group id

---------

Co-authored-by: Nicholas Molnar <[email protected]>

* Generate protos in repo (#335)

* Add buf.gen.yaml

* Gen protos

* Update build steps

* Update protos

* Add hpke pub key

* Specify column

* Fix field name

* Fix migration

* Add data to test

* Do more validations on welcome messages

* Simplify check

* Fix insert statement

* Push up mls image

* Add new generated file

---------

Co-authored-by: Richard Hua <[email protected]>
Co-authored-by: Steven Normore <[email protected]>
Co-authored-by: Brendan McMillion <[email protected]>
Co-authored-by: Steven Normore <[email protected]>
snormore pushed a commit that referenced this pull request Jan 23, 2024
@neekolas @richardhuaaa @Bren2010
Merge mls branch into main (#343)
86fc580 
* Validation service scaffold (#320)

* Scaffold MLS server

* Update go.mod

* Fix missing argument

* Add unsaved file

* Lint

* Working end-to-end

* Lint

* Add new push action

* Address review comments

* Change method casing

* Change casing of server options

* Change casing of validation options

* Remove unused function

* Remove double pointer

* Make private again

* Add more MLS methods (#321)

* Scaffold MLS server

* Update go.mod

* Fix missing argument

* Add unsaved file

* Lint

* Working end-to-end

* Lint

* Add new push action

* Add a bunch of new endpoints

* Address review comments

* Change method casing

* Change casing of server options

* Change casing of validation options

* Remove unused function

* Remove double pointer

* Make private again

* Fix pointer to key package

* Capitalize more things

* Update server fields

* Add test for sort methods

* Save change to capitalization

* Fix lint warnings

* Fix problem with mocks

* Fix index name

* Move sorting to the store

* Fix ciphertext validation

* Make installation_id bytes

* Add missing credential identity

* Hack sql in query

* Revert "Hack sql in query"

This reverts commit 168b78a.

* Remove custom type

* Update to latest protos

* Add CredentialIdentity

* Fix migrations

* refactor: walletaddress->accountaddress (#325)

Settling on consistent terminology as discussed

* go mod tidy

* Assume KeyPackages are always last resort. (#329)

* Assume KeyPackages are always last resort.

* Missing KeyPackages don't cause entire request to fail.

* Update go.mod

* UpdateKeyPackage returns an error if installation is unknown.

* refactor: move api/message/v3 to mls/api/v1 (#332)

* refactor: move message_api/v3 to mls/api/v1

* refactor: move pkg/mlsstore to pkg/mls/store

* Rename mls build/push workflow

* Decouple MLS messages from messagev1 (#333)

* Separate MLS messages and implement service methods

* fix: group id and installation id are bytes

* fix: idempotent send group/welcome messages via uniquness in db

* fix: hex decode group id from mls validation service

* fix: s/Cursor/IdCursor

* fix: pass message data only in send group message request

* refactor: add mls {Group,Welcome}MessageInput types for send requests

* refactor: s/installation_id/installation_key in mls/api

* fix: clean up mls query page size logic

* feat: implement mls subscribe group/welcome messages

* Hex encode group ID

* fix: remove duplicate import

* fix: return grpc invalidargument on invalid group id

---------

Co-authored-by: Nicholas Molnar <[email protected]>

* Generate protos in repo (#335)

* Add buf.gen.yaml

* Gen protos

* Update build steps

* go mod tidy

* Remove push-mls github workflow

* Add hpke public key (#337)

* Validation service scaffold (#320)

* Scaffold MLS server

* Update go.mod

* Fix missing argument

* Add unsaved file

* Lint

* Working end-to-end

* Lint

* Add new push action

* Address review comments

* Change method casing

* Change casing of server options

* Change casing of validation options

* Remove unused function

* Remove double pointer

* Make private again

* Add more MLS methods (#321)

* Scaffold MLS server

* Update go.mod

* Fix missing argument

* Add unsaved file

* Lint

* Working end-to-end

* Lint

* Add new push action

* Add a bunch of new endpoints

* Address review comments

* Change method casing

* Change casing of server options

* Change casing of validation options

* Remove unused function

* Remove double pointer

* Make private again

* Fix pointer to key package

* Capitalize more things

* Update server fields

* Add test for sort methods

* Save change to capitalization

* Fix lint warnings

* Fix problem with mocks

* Fix index name

* Move sorting to the store

* Fix ciphertext validation

* Make installation_id bytes

* Add missing credential identity

* Hack sql in query

* Revert "Hack sql in query"

This reverts commit 168b78a.

* Remove custom type

* Update to latest protos

* Add CredentialIdentity

* Fix migrations

* refactor: walletaddress->accountaddress (#325)

Settling on consistent terminology as discussed

* go mod tidy

* Assume KeyPackages are always last resort. (#329)

* Assume KeyPackages are always last resort.

* Missing KeyPackages don't cause entire request to fail.

* Update go.mod

* UpdateKeyPackage returns an error if installation is unknown.

* refactor: move api/message/v3 to mls/api/v1 (#332)

* refactor: move message_api/v3 to mls/api/v1

* refactor: move pkg/mlsstore to pkg/mls/store

* Rename mls build/push workflow

* Decouple MLS messages from messagev1 (#333)

* Separate MLS messages and implement service methods

* fix: group id and installation id are bytes

* fix: idempotent send group/welcome messages via uniquness in db

* fix: hex decode group id from mls validation service

* fix: s/Cursor/IdCursor

* fix: pass message data only in send group message request

* refactor: add mls {Group,Welcome}MessageInput types for send requests

* refactor: s/installation_id/installation_key in mls/api

* fix: clean up mls query page size logic

* feat: implement mls subscribe group/welcome messages

* Hex encode group ID

* fix: remove duplicate import

* fix: return grpc invalidargument on invalid group id

---------

Co-authored-by: Nicholas Molnar <[email protected]>

* Generate protos in repo (#335)

* Add buf.gen.yaml

* Gen protos

* Update build steps

* Update protos

* Add hpke pub key

* Specify column

* Fix field name

* Fix migration

* Add data to test

* Do more validations on welcome messages

* Simplify check

* Fix insert statement

* Push up mls image

* Add new generated file

---------

Co-authored-by: Richard Hua <[email protected]>
Co-authored-by: Steven Normore <[email protected]>
Co-authored-by: Brendan McMillion <[email protected]>
Co-authored-by: Steven Normore <[email protected]>

* Populate options from MLS_DB_CONNECTION_STRING too

---------

Co-authored-by: Nicholas Molnar <[email protected]>
Co-authored-by: Richard Hua <[email protected]>
Co-authored-by: Brendan McMillion <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@richardhuaaa richardhuaaa richardhuaaa left review comments

@snormore snormore snormore approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@neekolas @snormore @richardhuaaa