Issue #108 - Create CLI for csvqb #157
Conversation
…nerating CSV-W qb-flavoured outputs.
| [required] | ||
| -o, --out OUT_DIR Location of the CSV-W outputs. [default: | ||
| out] | ||
| --fail-when-validation-error / --ignore-validation-errors |
There was a problem hiding this comment.
Why do we need to ignore validation errors?
There was a problem hiding this comment.
Because someone might need to make a decision that they know better than the validation errors at some point. It's usually a good idea to let humans override machines.
| print(f"{Style.DIM}CSV: {csv_path.absolute()}") | ||
| print(f"{Style.DIM}info.json: {info_json.absolute()}") | ||
| data = pd.read_csv(csv_path) | ||
| assert isinstance(data, pd.DataFrame) |
There was a problem hiding this comment.
Does it mean that we need just a Pd.DataFrame of tidy data with no Cube’s class at the end of data transformation?
There was a problem hiding this comment.
That's the hope, we should get there eventually with this.
| maybe_config = column_mappings.get(col_title) | ||
| defined_columns.append( | ||
| _get_column_for_metadata_config(col_title, maybe_config, data[col_title]) | ||
| ) | ||
|
|
||
| columns_missing_in_data = set(column_mappings.keys()) - set(column_titles_in_data) |
There was a problem hiding this comment.
What is column_mappings.keys() and column_titles_in_data? And why do we need a set difference between those two?
There was a problem hiding this comment.
column_mappings.keys() are the names of the columns with mappings defined in the info.json file. We do this set different to find column mappings which are provided in the info.json but don't exist in the actual dataframe.
This highlights that the user has made some kind of error and has probably missed out a column from their CSV.
|
@robons Thanks for clarifying the doubts. |
Are you happy enough to approve the PR, @santhosh-thangavel? |
Satisfies issue #108.
Command Line Interface
Build
Build the qb-flavoured CSV-Ws from an info.json V1 file.
Usage: csvqb build [OPTIONS] TIDY_CSV_PATH Build a qb-flavoured CSV-W from a tidy CSV. Options: -c, --config CONFIG_PATH Location of the info.json file containing the QB column mapping definitions. [required] -o, --out OUT_DIR Location of the CSV-W outputs. [default: out] --fail-when-validation-error / --ignore-validation-errors Fail when validation errors occur or ignore validation errors and continue generating a CSV-W. [default: fail-when-validation- error] --validation-errors-to-file Save validation errors to an `validation- errors.json` file in the output directory. [default: False] -h, --help Show this message and exit.example:
Outputs resulting CSV-Ws into the
./outdirectory (which is created if it does not already exist).