docs(adrs): init WIP ClickHouse ADR #179
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # ClickHouse as an additional GraphQL backend | ||
|
|
||
| - Status: draft | ||
| - Deciders: ... | ||
| - Date: 2024-07-26 | ||
| - Authors: [David] | ||
|
|
||
| ## Context and Problem Statement | ||
|
|
||
| The initial version of the ar-io-node used SQLite as its primary database | ||
| backend (KV stores are also used in specific cases). While SQLite generaly | ||
| works well, it has some limitations. It is difficult to scale up and out, has | ||
| mediocre batch loading performance, and does not have native suport for Parquet | ||
| (needed for our indexing protocol). In order to address these issues, we intend | ||
| to implement a new GQL backend that can be used in combination with SQLite (and | ||
| other future DB backends). This decision record documents the evaluation of | ||
| database options for this new backend. | ||
|
|
||
| TODO: expand on the specific the specific issues with SQLite and the problem | ||
| constraints | ||
|
Comment on lines
+8
to
+20
There was a problem hiding this comment. Elaborate on the specific issues with SQLite. The TODO comment indicates that more details are needed on the specific issues with SQLite and the problem constraints. Do you want me to help expand this section or open a GitHub issue to track this task? ToolsLanguageTool
|
||
|
|
||
| ## Decision Drivers | ||
|
|
||
| - Horizontal and vertical scalability | ||
| - Batch loading performance | ||
| - Parquet support (for batch loading convenience and indexing protocol) | ||
| - Table sorting (for optimal query performance) | ||
| - Operational simplicity | ||
| - Data compression | ||
| - Query performance | ||
|
|
||
| ## Considered Options | ||
|
|
||
| - ClickHouse | ||
| - Druid | ||
| - DuckDB | ||
| - Elastic/OpenSearch | ||
|
|
||
| ## Decision Outcome | ||
|
|
||
| TODO: summarize the decision | ||
|
Comment on lines
+39
to
+41
There was a problem hiding this comment. Summarize the decision outcome. The TODO comment indicates that the decision summary is missing. Please provide a summary of the decision outcome. Do you want me to help summarize the decision outcome or open a GitHub issue to track this task? |
||
|
|
||
| ### Positive Consequences | ||
|
|
||
| - Fast batch Parquet loading | ||
| - Greater custom indexing flexibility | ||
| - Improved query performance | ||
|
|
||
| ### Negative Consequences | ||
|
|
||
| - Increased configuration and operational complexity | ||
|
|
||
| ## Pros and Cons of the Options | ||
|
|
||
| TODO: replace or augment with a table | ||
|
|
||
| ### ClickHouse | ||
|
|
||
| - `+` Native Parquet ingest support | ||
| - `+` Extremely fast batch ingest (millions of rows per second) | ||
| - `+` Table sorting | ||
| - `+` Easily horizontally scalability for reads | ||
| - `+` Excellent vertical scalability | ||
| - `+` Relatively simple ops (easy single node setup) | ||
| - `-` Horizontal write scaling is doable, but more complex | ||
| - `-` No easy to use embedded version | ||
|
|
||
| ### DuckDB | ||
|
|
||
| - `+` Embedded version | ||
| - `+` Very simple to run (embedded like SQLite) | ||
| - `-` No table sorting | ||
| - `-` Limited scalability (single node) | ||
|
|
||
| ### Druid | ||
|
|
||
| - `+` Extremely scalable horizontal reads and writes | ||
| - `-` Only supports sorting by time | ||
| - `-` Complex to run and operate | ||
|
|
||
| ### Elastic/OpenSearch | ||
|
|
||
| - `+` Bulk loading API | ||
| - `+` Horizontally scalabilit | ||
| - `+` | ||
| - `-` Limited indexing flexibility (difficult to tailor to specific use cases) | ||
| - `-` Limited sorting capabilities | ||
| - `-` No native Parquet support | ||
|
|
||
|
Comment on lines
+53
to
+89
There was a problem hiding this comment. Consider using a table for better readability. The pros and cons section provides a good overview but could be improved with a table for better readability. Do you want me to help create a table or open a GitHub issue to track this task? ToolsLanguageTool
|
||
| ## Links | ||
|
|
||
| TODO: references, etc. | ||
|
Comment on lines
+90
to
+92
There was a problem hiding this comment. Add references and links. The TODO comment indicates that references and links are missing. Please provide references and links to relevant resources. Do you want me to help add references or open a GitHub issue to track this task? |
||
|
|
||
| ## Notes | ||
|
|
||
| TODO: add high level thoughts and conclusions here | ||
|
Comment on lines
+94
to
+96
There was a problem hiding this comment. Add high-level thoughts and conclusions. The TODO comment indicates that high-level thoughts and conclusions are missing. Please provide high-level thoughts and conclusions. Do you want me to help add high-level thoughts and conclusions or open a GitHub issue to track this task? ToolsLanguageTool
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Complete the deciders field.
The deciders field is currently incomplete. Please list the individuals or teams responsible for this decision.