From 321ee7e5840ba4e254de4645b3e40676af5ede77 Mon Sep 17 00:00:00 2001 From: Elisacci Lapo Date: Tue, 22 Feb 2022 07:53:50 +0100 Subject: [PATCH 1/7] docs: intent for klip-62: ksqlDB Ruby client --- design-proposals/README.md | 3 +- .../klip-62-ksqldb-ruby-client.md | 53 +++++++++++++++++++ 2 files changed, 55 insertions(+), 1 deletion(-) create mode 100644 design-proposals/klip-62-ksqldb-ruby-client.md diff --git a/design-proposals/README.md b/design-proposals/README.md index 3c9b3a639d0d..29ab8b428142 100644 --- a/design-proposals/README.md +++ b/design-proposals/README.md @@ -37,7 +37,7 @@ This is the guts of our improvement proposal process: The progression of statuses should be: Proposal, Discussion, Approved, Merged -Next KLIP number: **62** +Next KLIP number: **63** | KLIP | Status | Community Release | CP Release | Discussion PR | |----------------------------------------------------------------------------------------------------------------------------------|:--------:|:-----------------:|:----------:|--------------------------------------------------------------| @@ -102,3 +102,4 @@ Next KLIP number: **62** | [KLIP-59: JSON functions](klip-59-json-functions.md) | Approved | | | [Discussion](https://github.com/confluentinc/ksql/pull/8550) | | [KLIP-60: Support input topics with multiple message types](klip-60-multiple-message-types.md) | Proposal | | | | | [KLIP-61: Bootstrapped TABLE subscriptions](klip-61-bootstrapped-subscriptions.md) | Proposal | | | | +| [KLIP-62: ksqlDB Ruby Client](klip-62-ksqldb-ruby-client.md) | Proposal | | | [Discussion](https://github.com/confluentinc/ksql/pull/8794) | diff --git a/design-proposals/klip-62-ksqldb-ruby-client.md b/design-proposals/klip-62-ksqldb-ruby-client.md new file mode 100644 index 000000000000..10c184c04b92 --- /dev/null +++ b/design-proposals/klip-62-ksqldb-ruby-client.md @@ -0,0 +1,53 @@ +# KLIP 62 - ksqlDB Ruby Client + +**Author**: Lapo Elisacci (@LapoElisacci) | +**Release Target**: TBD | +**Status**: _In Discussion_ | +**Discussion**: TBD + +**tl;dr:** Develop a Ruby HTTP2 Client to request ksqlDB REST API + +## Motivation and background + +Allow Ruby / Ruby on Rails developers to get started with ksqlDB. + +## What is in scope + +* A lightweight but complete and configurable Ruby client. +* Support for both synchronous and asynchronous queries. +* Documentation and examples + +## What is not in scope + +* Anything that concerns the ksqlDB development itself. +* Defining an ORM to dynamically build SQL Statements. (It will get handled by a future project) + +## Value/Return + +To be able to easily include ksqlDB in any Ruby / Ruby on Rails applications. + +## Public APIS + +N/A + +## Design + +* A Ruby class will implement the logic to request all ksqlDB REST API available endpoints. +* Request responses will get wrapped inside objects to easily manipulate the returned data. + +## Test plan + +Both unit and integration tests. +All currenlty available ksqlDB statements will get properly tested. + +## LOEs and Delivery Milestones + +TBD + +## Compatibility Implications + +The client will be compatible with Ruby >= 2.6 + +## Security Implications + +The client will support all protocols supported by the ksqlDB REST API. From cba6a77985cb9488ebb0b288bb2ed8324b37e2ca Mon Sep 17 00:00:00 2001 From: Lapo Date: Tue, 8 Mar 2022 16:51:55 +0100 Subject: [PATCH 2/7] Remove KLIP from proposal PR --- .../klip-62-ksqldb-ruby-client.md | 53 ------------------- 1 file changed, 53 deletions(-) delete mode 100644 design-proposals/klip-62-ksqldb-ruby-client.md diff --git a/design-proposals/klip-62-ksqldb-ruby-client.md b/design-proposals/klip-62-ksqldb-ruby-client.md deleted file mode 100644 index 10c184c04b92..000000000000 --- a/design-proposals/klip-62-ksqldb-ruby-client.md +++ /dev/null @@ -1,53 +0,0 @@ -# KLIP 62 - ksqlDB Ruby Client - -**Author**: Lapo Elisacci (@LapoElisacci) | -**Release Target**: TBD | -**Status**: _In Discussion_ | -**Discussion**: TBD - -**tl;dr:** Develop a Ruby HTTP2 Client to request ksqlDB REST API - -## Motivation and background - -Allow Ruby / Ruby on Rails developers to get started with ksqlDB. - -## What is in scope - -* A lightweight but complete and configurable Ruby client. -* Support for both synchronous and asynchronous queries. -* Documentation and examples - -## What is not in scope - -* Anything that concerns the ksqlDB development itself. -* Defining an ORM to dynamically build SQL Statements. (It will get handled by a future project) - -## Value/Return - -To be able to easily include ksqlDB in any Ruby / Ruby on Rails applications. - -## Public APIS - -N/A - -## Design - -* A Ruby class will implement the logic to request all ksqlDB REST API available endpoints. -* Request responses will get wrapped inside objects to easily manipulate the returned data. - -## Test plan - -Both unit and integration tests. -All currenlty available ksqlDB statements will get properly tested. - -## LOEs and Delivery Milestones - -TBD - -## Compatibility Implications - -The client will be compatible with Ruby >= 2.6 - -## Security Implications - -The client will support all protocols supported by the ksqlDB REST API. From 10e97e2bb555c024b6f259284dc23c63c49615cb Mon Sep 17 00:00:00 2001 From: Lapo Date: Tue, 8 Mar 2022 21:54:04 +0100 Subject: [PATCH 3/7] KLIP 62 Ruby client discussion --- .../klip-62-ksqldb-ruby-client.md | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 design-proposals/klip-62-ksqldb-ruby-client.md diff --git a/design-proposals/klip-62-ksqldb-ruby-client.md b/design-proposals/klip-62-ksqldb-ruby-client.md new file mode 100644 index 000000000000..10c184c04b92 --- /dev/null +++ b/design-proposals/klip-62-ksqldb-ruby-client.md @@ -0,0 +1,53 @@ +# KLIP 62 - ksqlDB Ruby Client + +**Author**: Lapo Elisacci (@LapoElisacci) | +**Release Target**: TBD | +**Status**: _In Discussion_ | +**Discussion**: TBD + +**tl;dr:** Develop a Ruby HTTP2 Client to request ksqlDB REST API + +## Motivation and background + +Allow Ruby / Ruby on Rails developers to get started with ksqlDB. + +## What is in scope + +* A lightweight but complete and configurable Ruby client. +* Support for both synchronous and asynchronous queries. +* Documentation and examples + +## What is not in scope + +* Anything that concerns the ksqlDB development itself. +* Defining an ORM to dynamically build SQL Statements. (It will get handled by a future project) + +## Value/Return + +To be able to easily include ksqlDB in any Ruby / Ruby on Rails applications. + +## Public APIS + +N/A + +## Design + +* A Ruby class will implement the logic to request all ksqlDB REST API available endpoints. +* Request responses will get wrapped inside objects to easily manipulate the returned data. + +## Test plan + +Both unit and integration tests. +All currenlty available ksqlDB statements will get properly tested. + +## LOEs and Delivery Milestones + +TBD + +## Compatibility Implications + +The client will be compatible with Ruby >= 2.6 + +## Security Implications + +The client will support all protocols supported by the ksqlDB REST API. From 5a4e2195867961d0c1f0e7957db4973e82902b62 Mon Sep 17 00:00:00 2001 From: Lapo <50866745+LapoElisacci@users.noreply.github.com> Date: Wed, 16 Mar 2022 22:49:11 +0100 Subject: [PATCH 4/7] Remove README.md klip reference --- design-proposals/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/design-proposals/README.md b/design-proposals/README.md index 29ab8b428142..d3587cc20b4f 100644 --- a/design-proposals/README.md +++ b/design-proposals/README.md @@ -102,4 +102,3 @@ Next KLIP number: **63** | [KLIP-59: JSON functions](klip-59-json-functions.md) | Approved | | | [Discussion](https://github.com/confluentinc/ksql/pull/8550) | | [KLIP-60: Support input topics with multiple message types](klip-60-multiple-message-types.md) | Proposal | | | | | [KLIP-61: Bootstrapped TABLE subscriptions](klip-61-bootstrapped-subscriptions.md) | Proposal | | | | -| [KLIP-62: ksqlDB Ruby Client](klip-62-ksqldb-ruby-client.md) | Proposal | | | [Discussion](https://github.com/confluentinc/ksql/pull/8794) | From bd0bfdd2d6ee365f9569a5ee601515908ecca84d Mon Sep 17 00:00:00 2001 From: Lapo <50866745+LapoElisacci@users.noreply.github.com> Date: Wed, 16 Mar 2022 23:25:45 +0100 Subject: [PATCH 5/7] Scopes update --- .../klip-62-ksqldb-ruby-client.md | 36 +++++++++++++++++-- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/design-proposals/klip-62-ksqldb-ruby-client.md b/design-proposals/klip-62-ksqldb-ruby-client.md index 10c184c04b92..60209b1b492b 100644 --- a/design-proposals/klip-62-ksqldb-ruby-client.md +++ b/design-proposals/klip-62-ksqldb-ruby-client.md @@ -13,8 +13,24 @@ Allow Ruby / Ruby on Rails developers to get started with ksqlDB. ## What is in scope -* A lightweight but complete and configurable Ruby client. -* Support for both synchronous and asynchronous queries. +* A lightweight but complete Ruby client, kslqDB host and port will be configurable. +* The Client will handle all the operations the Java one already does: + +- Receive query results one row at a time +- Receive query results in a single batch +- Terminate a push query +- Insert a new row into a stream +- Insert new rows in a streaming fashion +- Create and manage new streams, tables, and persistent queries +- List streams, tables, topics, and queries +- Describe specific streams and tables +- Get metadata about the ksqlDB cluster +- Manage, list and describe connectors +- Define variables for substitution +- Execute Direct HTTP Requests + +* Pull and push queries will be performed against the `/query-stream` endpoint, and inserts to the `/inserts-stream` endpoint. All other requests to the `/ksql` endpoint. + * Documentation and examples ## What is not in scope @@ -28,7 +44,21 @@ To be able to easily include ksqlDB in any Ruby / Ruby on Rails applications. ## Public APIS -N/A +* A class will expose a `config` method to configurate the connection between the client and ksqlDB, like so: + +```Ruby + KsqlDB::Client.configure do |config| + config.host = 'http://localhost' + config.port = 8088 + end +``` + +* A class method like `streamQuery()` will be available to query results one row at a time. +* A class method like `executeQuery()` will handle batched results queries. +* A class method like `terminatePushQuery()` will allow push queries termination. + +More yet to come... + ## Design From a6ba3e7576baa75ca2cc0fc73f16414743ff3e17 Mon Sep 17 00:00:00 2001 From: Lapo <50866745+LapoElisacci@users.noreply.github.com> Date: Thu, 17 Mar 2022 18:19:54 +0100 Subject: [PATCH 6/7] KLIP Updates --- .../klip-62-ksqldb-ruby-client.md | 33 +++++++++++-------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/design-proposals/klip-62-ksqldb-ruby-client.md b/design-proposals/klip-62-ksqldb-ruby-client.md index 60209b1b492b..d9c40951ccc4 100644 --- a/design-proposals/klip-62-ksqldb-ruby-client.md +++ b/design-proposals/klip-62-ksqldb-ruby-client.md @@ -26,10 +26,6 @@ Allow Ruby / Ruby on Rails developers to get started with ksqlDB. - Describe specific streams and tables - Get metadata about the ksqlDB cluster - Manage, list and describe connectors -- Define variables for substitution -- Execute Direct HTTP Requests - -* Pull and push queries will be performed against the `/query-stream` endpoint, and inserts to the `/inserts-stream` endpoint. All other requests to the `/ksql` endpoint. * Documentation and examples @@ -47,18 +43,21 @@ To be able to easily include ksqlDB in any Ruby / Ruby on Rails applications. * A class will expose a `config` method to configurate the connection between the client and ksqlDB, like so: ```Ruby - KsqlDB::Client.configure do |config| - config.host = 'http://localhost' - config.port = 8088 + Ksql.configure do |config| + config.host = 'http://localhost:8088' end ``` -* A class method like `streamQuery()` will be available to query results one row at a time. -* A class method like `executeQuery()` will handle batched results queries. -* A class method like `terminatePushQuery()` will allow push queries termination. - -More yet to come... +### Methods +- **ksql** - Allows statement to be requested against the /ksql endpoint. +- **close_query** - Allows closing persistent queries by requesting the /close-query endpoint. +- **stream** - Allows to perform push queries, processing messages one at a time in a streaming fashion. +- **query** - Allows to perform pull queries. +- **cluster_status** - Allows to introspect the ksqlDB cluster status. +- **health_check** - Allows to check the health of the ksqlDB server. +- **info** - Allows to get information about the status of a ksqlDB Server. +- **terminate** - Allows to terminate the cluster and clean up the resources, or to delete Kafka Topics. ## Design @@ -68,11 +67,17 @@ More yet to come... ## Test plan Both unit and integration tests. -All currenlty available ksqlDB statements will get properly tested. +CREATE, DESCRIBE, DROP, INSERT, SELECT (both push and pull), SHOW and TERMINATE statements will get properly tested, to ensure the client always behaves as expected. + +The standard Ruby RSpec test suite will do. +Testing requests will get performed against ksqlDB >= 0.22 ## LOEs and Delivery Milestones -TBD +The Gem's under development and most of the endpoints are already covered. +A first release is expected to be live by the end of March 2022. + +Any meaningfull future bug will get handled within a week. ## Compatibility Implications From e482824cd22972ad6117e885a385a6b8f3bc5fa5 Mon Sep 17 00:00:00 2001 From: Lapo <50866745+LapoElisacci@users.noreply.github.com> Date: Thu, 17 Mar 2022 18:25:00 +0100 Subject: [PATCH 7/7] KLIP Update --- design-proposals/klip-62-ksqldb-ruby-client.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/design-proposals/klip-62-ksqldb-ruby-client.md b/design-proposals/klip-62-ksqldb-ruby-client.md index d9c40951ccc4..d7d79314dd11 100644 --- a/design-proposals/klip-62-ksqldb-ruby-client.md +++ b/design-proposals/klip-62-ksqldb-ruby-client.md @@ -14,13 +14,12 @@ Allow Ruby / Ruby on Rails developers to get started with ksqlDB. ## What is in scope * A lightweight but complete Ruby client, kslqDB host and port will be configurable. -* The Client will handle all the operations the Java one already does: +* The Client will handle most of the operations the Java one does: - Receive query results one row at a time - Receive query results in a single batch - Terminate a push query - Insert a new row into a stream -- Insert new rows in a streaming fashion - Create and manage new streams, tables, and persistent queries - List streams, tables, topics, and queries - Describe specific streams and tables