docs(adrs): init WIP ClickHouse ADR #179
Conversation
This ADR is not finished, but is open for comment, especially wrt what about this decision needs the most explanation.
|
Note Reviews pausedUse the following commands to manage reviews:
WalkthroughWalkthroughThis change introduces ClickHouse as an additional backend for the GraphQL interface of the ar-io-node project, aiming to improve scalability and performance over the existing SQLite implementation. The documentation evaluates various database options, highlighting ClickHouse's strengths in batch loading and query performance, while also addressing its operational complexity. The structure provides a clear analysis of the decision-making process and future implications for the project's architecture. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant GraphQL Server
participant ClickHouse
Client->>GraphQL Server: Query Data
GraphQL Server->>ClickHouse: Fetch Data
ClickHouse-->>GraphQL Server: Return Data
GraphQL Server-->>Client: Send Response
Recent review detailsConfiguration used: CodeRabbit UI Files selected for processing (1)
Additional context usedLanguageTool
Additional comments not posted (2)
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configuration File (
|
This matches what we're doing in other repos.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
Outside diff range, codebase verification and nitpick comments (1)
docs/madrs/001-clickhouse-gql.md (1)
71-71: Consider an alternative to "very simple to run".The phrase "very simple to run" can be replaced with a more precise alternative.
- `+` Very simple to run (embedded like SQLite) + `+` Extremely easy to run (embedded like SQLite)Tools
LanguageTool
[style] ~71-~71: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...## DuckDB -+Embedded version -+Very simple to run (embedded like SQLite) --No ...(EN_WEAK_ADJECTIVE)
docs/madrs/001-clickhouse-gql.md
Outdated
| - Status: draft | ||
| - Deciders: ... | ||
| - Date: 2024-07-26 | ||
| - Authors: [David] |
There was a problem hiding this comment.
Complete the deciders field.
The deciders field is currently incomplete. Ensure to list all the decision-makers involved in this ADR.
docs/madrs/001-clickhouse-gql.md
Outdated
|
|
||
| ## Decision Outcome | ||
|
|
||
| TODO: summarize the decision |
There was a problem hiding this comment.
Complete the decision outcome section.
The Decision Outcome section is currently a TODO placeholder. Ensure to summarize the decision made.
docs/madrs/001-clickhouse-gql.md
Outdated
|
|
||
| ## Pros and Cons of the Options | ||
|
|
||
| TODO: replace or augment with a table |
There was a problem hiding this comment.
Complete the pros and cons table.
The Pros and Cons of the Options section contains a TODO placeholder for a table. Ensure to replace or augment the list with a table.
docs/madrs/001-clickhouse-gql.md
Outdated
|
|
||
| ## Links | ||
|
|
||
| TODO: references, etc. |
There was a problem hiding this comment.
Complete the links section.
The Links section is currently a TODO placeholder. Ensure to add references and other relevant links.
docs/madrs/001-clickhouse-gql.md
Outdated
|
|
||
| ## Notes | ||
|
|
||
| TODO: add high level thoughts and conclusions here |
There was a problem hiding this comment.
Complete the notes section.
The Notes section is currently a TODO placeholder. Ensure to add high-level thoughts and conclusions.
Tools
LanguageTool
[uncategorized] ~96-~96: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...: references, etc. ## Notes TODO: add high level thoughts and conclusions here(EN_COMPOUND_ADJECTIVE_INTERNAL)
docs/madrs/001-clickhouse-gql.md
Outdated
| TODO: expand on the specific the specific issues with SQLite and the problem | ||
| constraints |
There was a problem hiding this comment.
Remove the duplicated phrase.
The phrase "the specific" is duplicated.
- TODO: expand on the specific the specific issues with SQLite and the problem
+ TODO: expand on the specific issues with SQLite and the problemCommittable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| TODO: expand on the specific the specific issues with SQLite and the problem | |
| constraints | |
| TODO: expand on the specific issues with SQLite and the problem | |
| constraints |
Tools
LanguageTool
[grammar] ~19-~19: This phrase is duplicated. You should probably use “the specific” only once.
Context: ... for this new backend. TODO: expand on the specific the specific issues with SQLite and the problem cons...(PHRASE_REPETITION)
| - Status: draft | ||
| - Deciders: ... | ||
| - Date: 2024-07-26 | ||
| - Authors: [David] |
There was a problem hiding this comment.
Complete the deciders field.
The deciders field is currently incomplete. Please list the individuals or teams responsible for this decision.
| ## Context and Problem Statement | ||
|
|
||
| The initial version of the ar-io-node used SQLite as its primary database | ||
| backend (KV stores are also used in specific cases). While SQLite generaly | ||
| works well, it has some limitations. It is difficult to scale up and out, has | ||
| mediocre batch loading performance, and does not have native suport for Parquet | ||
| (needed for our indexing protocol). In order to address these issues, we intend | ||
| to implement a new GQL backend that can be used in combination with SQLite (and | ||
| other future DB backends). This decision record documents the evaluation of | ||
| database options for this new backend. | ||
|
|
||
| TODO: expand on the specific the specific issues with SQLite and the problem | ||
| constraints |
There was a problem hiding this comment.
Elaborate on the specific issues with SQLite.
The TODO comment indicates that more details are needed on the specific issues with SQLite and the problem constraints.
Do you want me to help expand this section or open a GitHub issue to track this task?
Tools
LanguageTool
[style] ~14-~14: Consider a shorter alternative to avoid wordiness.
Context: ...uet (needed for our indexing protocol). In order to address these issues, we intend to impl...(IN_ORDER_TO_PREMIUM)
[grammar] ~19-~19: This phrase is duplicated. You should probably use “the specific” only once.
Context: ... for this new backend. TODO: expand on the specific the specific issues with SQLite and the problem cons...(PHRASE_REPETITION)
| ## Decision Outcome | ||
|
|
||
| TODO: summarize the decision |
There was a problem hiding this comment.
Summarize the decision outcome.
The TODO comment indicates that the decision summary is missing. Please provide a summary of the decision outcome.
Do you want me to help summarize the decision outcome or open a GitHub issue to track this task?
| ## Pros and Cons of the Options | ||
|
|
||
| TODO: replace or augment with a table | ||
|
|
||
| ### ClickHouse | ||
|
|
||
| - `+` Native Parquet ingest support | ||
| - `+` Extremely fast batch ingest (millions of rows per second) | ||
| - `+` Table sorting | ||
| - `+` Easily horizontally scalability for reads | ||
| - `+` Excellent vertical scalability | ||
| - `+` Relatively simple ops (easy single node setup) | ||
| - `-` Horizontal write scaling is doable, but more complex | ||
| - `-` No easy to use embedded version | ||
|
|
||
| ### DuckDB | ||
|
|
||
| - `+` Embedded version | ||
| - `+` Very simple to run (embedded like SQLite) | ||
| - `-` No table sorting | ||
| - `-` Limited scalability (single node) | ||
|
|
||
| ### Druid | ||
|
|
||
| - `+` Extremely scalable horizontal reads and writes | ||
| - `-` Only supports sorting by time | ||
| - `-` Complex to run and operate | ||
|
|
||
| ### Elastic/OpenSearch | ||
|
|
||
| - `+` Bulk loading API | ||
| - `+` Horizontally scalabilit | ||
| - `+` | ||
| - `-` Limited indexing flexibility (difficult to tailor to specific use cases) | ||
| - `-` Limited sorting capabilities | ||
| - `-` No native Parquet support | ||
|
|
There was a problem hiding this comment.
Consider using a table for better readability.
The pros and cons section provides a good overview but could be improved with a table for better readability.
Do you want me to help create a table or open a GitHub issue to track this task?
Tools
LanguageTool
[style] ~71-~71: As an alternative to the over-used intensifier ‘very’, consider replacing this phrase.
Context: ...## DuckDB -+Embedded version -+Very simple to run (embedded like SQLite) --No ...(EN_WEAK_ADJECTIVE)
| ## Links | ||
|
|
||
| TODO: references, etc. |
There was a problem hiding this comment.
Add references and links.
The TODO comment indicates that references and links are missing. Please provide references and links to relevant resources.
Do you want me to help add references or open a GitHub issue to track this task?
| ## Notes | ||
|
|
||
| TODO: add high level thoughts and conclusions here |
There was a problem hiding this comment.
Add high-level thoughts and conclusions.
The TODO comment indicates that high-level thoughts and conclusions are missing. Please provide high-level thoughts and conclusions.
Do you want me to help add high-level thoughts and conclusions or open a GitHub issue to track this task?
Tools
LanguageTool
[uncategorized] ~96-~96: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...: references, etc. ## Notes TODO: add high level thoughts and conclusions here(EN_COMPOUND_ADJECTIVE_INTERNAL)
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #179 +/- ##
========================================
Coverage 67.40% 67.40%
========================================
Files 29 29
Lines 7146 7146
Branches 383 383
========================================
Hits 4817 4817
Misses 2328 2328
Partials 1 1 ☔ View full report in Codecov by Sentry. |
|
@coderabbitai pause |
This ADR is not finished, but is open for comment, especially wrt what about this decision needs the most explanation.