Clickhouse import process #59
Conversation
sheridancbio
force-pushed
the
clickhouse-dependent-import-process
branch
from
e59f5f4
to
6648c06
Compare
sheridancbio
force-pushed
the
clickhouse-dependent-import-process
branch
from
6648c06
to
901afad
Compare
also bugfixes added and TODO notes
sheridancbio
force-pushed
the
clickhouse-dependent-import-process
branch
from
8966e72
to
aa61d3b
Compare
| if [ "$database_to_transfer" == "blue" ] ; then | ||
| clickhouse_destination_database_name="${my_properties['clickhouse_blue_database_name']}" | ||
| mysql_source_database_name="${my_properties['mysql_blue_database_name']}" | ||
| else |
There was a problem hiding this comment.
Here we could check explicitly that the database_to_transfer is green or give an error if it isn't blue or green.
| get_genetic_profile_id_list_query="SELECT genetic_profile_id FROM genetic_profile WHERE genetic_alteration_type NOT IN ('GENERIC_ASSAY', 'MUTATION_EXTENDED', 'STRUCTURAL_VARIANT')" | ||
| query_argument_template="--query={0}" | ||
| query_argument = query_argument_template.format(get_genetic_profile_id_list_query) | ||
| clickhouse_client_obtain_genetic_profile_id_list = ["clickhouse", "client", "--config-file=clickhouse_client_config_2024-10-14-09-03-02.yaml", query_argument] |
There was a problem hiding this comment.
Should the config-file always hardcoded to this?
There was a problem hiding this comment.
I see below you have a TODO to deal with that.
| @@ -0,0 +1,36 @@ | |||
| DROP TABLE IF EXISTS sample_to_gene_panel_derived; | |||
| DROP TABLE IF EXISTS gene_panel_to_gene_derived; | |||
There was a problem hiding this comment.
Aren't these commands stored somewhere else?
There was a problem hiding this comment.
Yes. This file was a placeholder. I need to add calls to the script which processes the .sql files from the github repo (clickhouse.sql and materialized_views.sql) and bursts them into individual sql statements (1 per file). Also, we need to detect and skip over the special cases (genetic_alteration_derived, generic_assay_data_derived).
So this file will be going away
| return 1 | ||
| fi | ||
| update_management_database_name="${my_properties['mysql_update_management_database']}" | ||
| ### TODO : fix this |
There was a problem hiding this comment.
I think this needs adding back.
| return 1 | ||
| fi | ||
| update_management_database_name="${my_properties['mysql_update_management_database']}" | ||
| ### TODO : fix this |
Scripts, property files, and documentation for the blue-green deployment strategy to enable MySQL/Clickhouse database updates during cancer study import.