feat: add nameserver auth #3835

oh2024 · 2024-03-26T04:35:01Z

What kind of change does this PR introduce?
This PR introduces a significant security enhancement by implementing RPC authenticator functionality to ensure all connections are authenticated server-side. This is a feature change aimed at improving the security of server-client communications.

What is the current behavior?
Currently, there is no authentication performed on the server side for incoming connections. Authentication is solely handled within the SDK, leaving a potential security gap for server-client communications.

What is the new behavior (if this is a feature change)?
With this change, all connections to the server will undergo authentication at the beginning of the rpc connection. This server-side authentication is facilitated by the newly introduced Authenticator class, which supports two types of authentication: service authentication and user authentication. For the sake of maintaining backward compatibility, the system will initially allow all services carrying the "default" token and users with the username "root". This PR lays down the foundational framework for our enhanced security measures, focusing on authentication at the server side. Future PRs will expand on this by incorporating user verification against tablet data and authenticating service tokens via a Zookeeper service token record. This implementation is a critical step toward tightening security and providing a robust authentication mechanism for our system.

…nd all internal services

github-actions · 2024-03-26T04:48:53Z

Linux Test Report

59 files + 59 249 suites +249 1h 47m 47s ⏱️ + 1h 47m 47s
13 256 tests +13 256 13 249 ✅ +13 249 7 💤 +7 0 ❌ ±0
18 813 runs +18 813 18 806 ✅ +18 806 7 💤 +7 0 ❌ ±0

Results for commit 5debf64. ± Comparison against base commit 6b52ee5.

♻️ This comment has been updated with latest results.

github-actions · 2024-03-26T04:54:41Z

SDK Test Report

102 files ±0 102 suites ±0 2m 15s ⏱️ -2s
357 tests ±0 343 ✅ ±0 14 💤 ±0 0 ❌ ±0
483 runs ±0 469 ✅ ±0 14 💤 ±0 0 ❌ ±0

Results for commit 5debf64. ± Comparison against base commit 6b52ee5.

This pull request removes 48 and adds 27 tests. Note that renamed tests count towards both.

  PARTITION BY db1.t1.col2 ORDER BY db1.t1.col1
  PARTITION BY t1.col2 ORDER BY t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](1)
 ) limit 10;](2)
 ) limit 10;](3)
 FROM db1.t1
 FROM t1
 WINDOW w1 AS (
 last join db2.t2 order by db2.t2.col1
…

com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlLastJoinWithMultipleDB[,  SELECT sum(db1.t1.col1) over w1 as sum_t1_col1, db2.t2.str1 as t2_str1
 FROM db1.t1
 last join db2.t2 order by db2.t2.col1
 on db1.t1.col1 = db2.t2.col1 and db1.t1.col2 = db2.t2.col0
 WINDOW w1 AS (
  PARTITION BY db1.t1.col2 ORDER BY db1.t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](2)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlLastJoinWithMultipleDB[db1,  SELECT sum(t1.col1) over w1 as sum_t1_col1, db2.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0
 WINDOW w1 AS (
  PARTITION BY t1.col2 ORDER BY t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](1)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlLastJoinWithMultipleDB[null,  SELECT sum(db1.t1.col1) over w1 as sum_t1_col1, db2.t2.str1 as t2_str1
 FROM db1.t1
 last join db2.t2 order by db2.t2.col1
 on db1.t1.col1 = db2.t2.col1 and db1.t1.col2 = db2.t2.col0
 WINDOW w1 AS (
  PARTITION BY db1.t1.col2 ORDER BY db1.t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](3)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[, SELECT db2.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Fail to transform data provider op: table t1 not exists in database []](4)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[db1, SELECT db1.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Column Not found: db1.t2.str1](2)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[db1, SELECT db2.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Column Not found: .t2.col1](3)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[db1, SELECT t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Column Not found: .t2.str1](1)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[null, SELECT db2.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Fail to transform data provider op: table t1 not exists in database []](5)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlWindowLastJoin[ SELECT sum(t1.col1) over w1 as sum_t1_col1, t2.str1 as t2_str1
 FROM t1
 last join t2 order by t2.col1
 on t1.col1 = t2.col1 and t1.col2 = t2.col0
 WINDOW w1 AS (
  PARTITION BY t1.col2 ORDER BY t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](1)
com._4paradigm.openmldb.jdbc.SQLRouterSmokeTest ‑ testInsertMeta[com._4paradigm.openmldb.sdk.impl.SqlClusterExecutor@1280bae3](1)
…

♻️ This comment has been updated with latest results.

src/auth/authenticator.h

src/auth/authenticator.cc

oh2024 · 2024-04-16T11:42:47Z

integration test will fail because old sdk is used

…lity tests

vagetablechicken · 2024-04-17T11:20:40Z

src/tablet/drop_procedure_before_drop_table_test.cc

    bool ok = name_server_client.SendRequest(&::openmldb::nameserver::NameServer_Stub::ShowTable, &request, &response,
                                             FLAGS_request_timeout_ms, 1);
    ASSERT_TRUE(ok);
    ASSERT_EQ(response.table_info_size(), size);
 }

-TEST_F(SqlClusterTest, RecoverProcedure) {
+TEST_F(SqlClusterTest, DropProcedureBeforeDropTable) {


Why rename this test file? drop_procedure_before_drop_table_test.cc is not a good name. You'd better make the file name be the same with the test class, not the test method name.

I need to separate the RecoverProcedure and DropProcedureBeforeDropTable for the tests to pass. There's probably some bugs in the tests that we can fix later

vagetablechicken

no docs about privileges, add in auth.md after #3879

aceforeverd · 2024-04-18T06:03:45Z

src/auth/user_access_manager.cc

+
+void UserAccessManager::StartSyncTask() {
+    sync_task_running_ = true;
+    sync_task_thread_ = std::thread([this] {


query table every 0.1 seconds ? Not good practice if my understand is correct, instead, you should lazy fetch the table with help of zk.

Register an event listener to zk > on table values update, notify zk > zk will send data updated event to every subscriber .

Yeah this is just temporary solution to get latest user credentials. In the future we will use a distributed transaction to update user table cache on all nodes.

vagetablechicken · 2024-04-18T08:18:04Z

merge after 0.9.0 released, cherry-pick is heavy

oh2024 added 6 commits March 20, 2024 20:20

add brpc authenticator

f7fc985

fix: add auth library

9950ffa

fix: make authenticator lifetime same as channel

c4ab632

refactor: use global variables to pass auth tokens

d402fd3

feat: added basic server auth backwards compatible with user = root a…

3769917

…nd all internal services

chore: remove debug log

7f19ade

oh2024 requested review from aceforeverd and vagetablechicken and removed request for aceforeverd March 26, 2024 04:35

github-actions bot added build openmldb compiling and installing storage-engine openmldb storage engine. nameserver & tablet labels Mar 26, 2024

oh2024 requested review from dl239, aceforeverd and vagetablechicken and removed request for vagetablechicken March 26, 2024 04:35

oh2024 added 3 commits March 26, 2024 12:56

chore: apply lint style fixes

5569b90

feat: encrypt password before sending

466a5c4

fix: add namespace for Authenticator class

269220e

oh2024 requested a review from tobegit3hub March 26, 2024 06:08

oh2024 added 2 commits March 26, 2024 14:16

chore: add newline

8f2a779

feat: add user verification logic

f210a9b

aceforeverd reviewed Mar 28, 2024

View reviewed changes

src/auth/authenticator.h Outdated Show resolved Hide resolved

vagetablechicken reviewed Mar 28, 2024

View reviewed changes

src/auth/authenticator.cc Outdated Show resolved Hide resolved

zhangziheng01233 added 2 commits April 2, 2024 02:29

feat: add nameserver server side auth

3fb9819

chore: remove unneccessary logs

cb76d88

oh2024 changed the title ~~feat: add server auth~~ feat: add nameserver auth Apr 2, 2024

zhangziheng01233 added 2 commits April 2, 2024 02:44

chore: lint

e21fb53

chore: make refreshable map header only

d368ab0

github-actions bot removed the sdk openmldb java/python sdk label Apr 11, 2024

oh2024 added 9 commits April 11, 2024 10:43

fix: always create user table

36f362c

fix: create user table after internal db create

a6aab82

fix: add partition info setting for user table create

47c1e53

Merge remote-tracking branch 'upstream/main' into feat/server-auth

e5abb0c

fix: auth related tests

2270869

fix: support multiple nameservers

78e2c3f

fix: recover db and table only for cluster mode

35b1167

fix: remote test ns tablet start order

806051f

fix: set root as default user in python sdk

1e07e5d

github-actions bot added the sdk openmldb java/python sdk label Apr 16, 2024

oh2024 added 3 commits April 17, 2024 02:13

fix: SyncTableReplicaCluster and AddAndRemoveReplicaCluster tests

cb17484

fix: remove user root

93317c8

Merge remote-tracking branch 'upstream/main' into feat/server-auth

0a817a5

github-actions bot removed the sdk openmldb java/python sdk label Apr 17, 2024

oh2024 added 2 commits April 17, 2024 05:08

fix: add erver stops to name_server_test cases and spilt sql availabi…

790094e

…lity tests

fix: nameserver test scope

9307f12

vagetablechicken reviewed Apr 17, 2024

View reviewed changes

vagetablechicken requested changes Apr 17, 2024

View reviewed changes

oh2024 added 3 commits April 17, 2024 13:40

fix: brpc authenticator destruction before nameserver

c3cb35a

Merge remote-tracking branch 'upstream/main' into feat/server-auth

951afbf

Merge remote-tracking branch 'upstream/main' into feat/server-auth

5f79ed9

vagetablechicken mentioned this pull request Apr 18, 2024

no docs about privileges, add in auth.md after #3879 #3883

Open

Merge branch 'main' into feat/server-auth

18c0e2e

aceforeverd reviewed Apr 18, 2024

View reviewed changes

test: change test names

5debf64

vagetablechicken approved these changes Apr 19, 2024

View reviewed changes

vagetablechicken merged commit 86c5e65 into 4paradigm:main Apr 19, 2024
22 of 24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add nameserver auth #3835

feat: add nameserver auth #3835

oh2024 commented Mar 26, 2024

github-actions bot commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

oh2024 commented Apr 16, 2024

vagetablechicken Apr 17, 2024 •

edited

Loading

oh2024 Apr 18, 2024

vagetablechicken left a comment •

edited

Loading

aceforeverd Apr 18, 2024 •

edited

Loading

oh2024 Apr 18, 2024

vagetablechicken commented Apr 18, 2024

feat: add nameserver auth #3835

feat: add nameserver auth #3835

Conversation

oh2024 commented Mar 26, 2024

github-actions bot commented Mar 26, 2024 • edited Loading

Linux Test Report

github-actions bot commented Mar 26, 2024 • edited Loading

SDK Test Report

oh2024 commented Apr 16, 2024

vagetablechicken Apr 17, 2024 • edited Loading

Choose a reason for hiding this comment

oh2024 Apr 18, 2024

Choose a reason for hiding this comment

vagetablechicken left a comment • edited Loading

Choose a reason for hiding this comment

aceforeverd Apr 18, 2024 • edited Loading

Choose a reason for hiding this comment

oh2024 Apr 18, 2024

Choose a reason for hiding this comment

vagetablechicken commented Apr 18, 2024

github-actions bot commented Mar 26, 2024 •

edited

Loading

github-actions bot commented Mar 26, 2024 •

edited

Loading

vagetablechicken Apr 17, 2024 •

edited

Loading

vagetablechicken left a comment •

edited

Loading

aceforeverd Apr 18, 2024 •

edited

Loading