Skip to content

Latest commit

 

History

History
1435 lines (958 loc) · 35.9 KB

cli.md

File metadata and controls

1435 lines (958 loc) · 35.9 KB
Raw

Installation

Install globally for the command line tools:

npm install -g datakit

Help

Use --help for each command to display help for it on the command line.

For example:

map --help

Commands

batch

Outputs the input dataset as an array of arrays, each sub-array containing the specified number of records.

Syntax

batch <batch-size> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • batch-size - Specifies the size for each batch.
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, batches the records into groups of 5 and writes to standard output

command-that-produces-json | batch 5

Reads data from a file, batches the records into groups of 5 and writes to standard output

batch 5 input-file.csv

Reads data from a file, batches the records into groups of 5 and writes output to another file

batch 5 input-file.csv output-file.csv

Reads JSON data from standard input, applies the transformation and writes output to another file

command-that-produces-json | batch 5 - output-file.csv

concat

Creates an output dataset by concatenating multiple input datasets. Works like array.concat in JavaScript.

Syntax

concat ...<input-file>

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON formatted data on standard output

Arguments

  • ...input-file - One or more input file names (json, csv or yaml) or a hypen to indicate reading JSON data from standard input.

Notes

  • This command (unlike most others in Datakit) isn't able to output directly to a file. Use redirection to write a file, shown in the example above.

Examples

Concatenates the data from multiple files and writes the result to standard output

concat input-file1.json input-file2.json input-file3.json

Reads JSON data from standard input concatenates it with a file and writes the result to standard output

command-that-produces-json | concat - input-file.json

Concatenates the data from multiple files and writes the result to a file

concat input-file1.json input-file2.json input-file3.json > output-file.json

distinct

Returns the set of distinct values from the input dataset. Removes duplicate values from the dataset.

Syntax

distinct [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, removes duplicate values and writes to standard output

command-that-produces-json | distinct -

Reads data from a file, removes duplicate values and writes to standard output

distinct input-file.csv

Reads data from a file, removes duplicate values and writes output to another file

distinct input-file.csv output-file.csv

filter

Creates an output dataset by filtering the input dataset through the predicate function. Works just like array.filter in JavaScript.

Syntax

filter <predicate-fn> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • predicate-fn - A JavaScript predicate function that is passed each record in the dataset and returns true/truthy to keep the record or false/falsy to rmeove the record. Specifying a file name will load the JavaScript code from the file.
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, applies the filter and writes to standard output

command-that-produces-json | filter "record => record.id === '1234'"

Reads data from a file, applies the filter and writes to standard output

filter "record => record.id === '1234'" input-file.csv

Reads data from a file, applies the filter and writes output to another file

filter "record => record.id === '1234'"  input-file.csv output-file.csv

Reads JSON data from standard input, applies the filter and writes output to another file

command-that-produces-json | filter "record => record.id === '1234'" - output-file.csv

Loads a JavaScript file for the filter function

filter --file my-filter.js input-file.csv output-file.csv

flatten

Flattens a nested dataset by 1 level. Works just like array.flat in JavaScript with an argument of 1 or the flatten function in Lodash.

Syntax

flatten [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Notes

  • There is no reason to use this command with CSV data, because CSV data can't be nested. Use this command with JSON and YAML data.

Examples

Reads JSON data from standard input, flattens it and writes to standard output

command-that-produces-json | flatten

Reads data from a file, flattens it and writes to standard output

flatten input-file.json

Reads data from a file, flattens it and writes output to another file

flatten input-file.json output-file.json

Reads JSON data from standard input, flattens it and writes output to another file

command-that-produces-json | flatten - output-file.json

format-table

Formats data to table rendered in ascii.

Syntax

format-table [<input-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • A table rendered in ascii output to standard output.

Arguments

  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.

Examples

Reads JSON data from standard input, renders the data to an ascii table on standard output

command-that-produces-json | format-table -

Reads data from a JSON file, renders the data to an ascii table on standard output

format-table input-file.json

Reads data from a YAML file, renders the data to an ascii table on standard output

format-table input-file.yaml

format-tree

Formats data to tree rendered in ascii.

Syntax

format-tree [<input-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • A tree rendered in ascii output to standard output.

Arguments

  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.

Examples

Reads JSON data from standard input, renders the data to an ascii tree on standard output

command-that-produces-json | format-tree -

Reads data from a JSON file, renders the data to an ascii tree on standard output

format-tree input-file.json

Reads data from a YAML file, renders the data to an ascii tree on standard output

format-tree input-file.yaml

from-csv

Converts data from the CSV (comma separated values) data format to the JSON data format.

Syntax

from-csv [<csv-input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • CSV file
  • CSV formatted data on standard input

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • input-file - Can be an input file name (must be a CSV file) or a hypen to indicate reading CSV data from standard input.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads CSV data from standard input and writes JSON data to standard output

command-that-produces-csv | from-csv -

Reads data from a CSV file and writes JSON data to standard output

from-csv input-file.csv

Reads data from a CSV file and writes output to a YAML data file

from-csv input-file.csv output-file.yaml

Reads data from a CSV file and writes output to a Json data file

from-csv input-file.csv output-file.json

from-yaml

Converts data from the YAML data format to the JSON data format.

Syntax

from-yaml [<yaml-input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • YAML file
  • YAML formatted data on standard input

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • input-file - Can be an input file name (must be a YAML file) or a hypen to indicate reading YAML data from standard input.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads YAML data from standard input and writes JSON data to standard output

command-that-produces-yaml | from-yaml -

Reads data from a YAML file and writes JSON data to standard output

from-yaml input-file.yaml

Reads data from a YAML file and writes output to a CSV data file

from-yaml input-file.yaml output-file.csv

Reads data from a YAML file and writes output to a JSON data file

from-yaml input-file.yaml output-file.json

group

Organises records from an input dataset into groups based on a key.

Syntax

group <key-selector-fn> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • key-selector-fn - A JavaScript function to select the grouping key for each record of the input dataset. Specifying a file name will load the JavaScript code from the file.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, groups by "department" and writes the groups to standard output

command-that-produces-json | group "record => record.department"

Reads data from a file, groups by "department" and writes the groups to standard output

group "record => record.department" input-file.csv

Reads data from a file, groups by "department" and counts total sales, writing the output to a file

group input-file.csv "r => r.department" | map - "g => ({ department: g.key, totalSales: g.records.length })" output-file.csv

intersect

Aggregates two data sets with common keys kind of like an SQL join.

Syntax

intersect <left-key-selector-fn> <left-input-file> <right-key-selector-fn> <right-input-file> <merge-fn> [<output-file>]

Inputs

Input can be 2 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • left-key-selector-fn - A JavaScript function to select the join key for each record of the left dataset. Specifying a file name will load the JavaScript code from the file.
  • left-input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • right-key-selector-fn - A JavaScript function to select the join key for each record of the right dataset. Specifying a file name will load the JavaScript code from the file.
  • right-input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • merge-fn - A JavaScript function to merge records from left and right datasets. Specifying a file name will load the JavaScript code from the file.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Notes

  • You can only read input data from standard input from one of the left or right datasets.

Examples

Reads two JSON files and merges the datasets based on the "email" field, writes output to a JSON file

intersect "r => r.email" left-input.json "r => r.email" right-input.json "(left, right) => ({ ...left, ...right })" output.json

length

Gets the number of records in a dataset. Works just like array.length in JavaScript.

Syntax

length [<input-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • Prints the number of records in the input dataset.

Arguments

  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input.

Examples

Reads JSON data from standard input, outputs the number of records

command-that-produces-json | length -

Reads data from a file, outputs the number of records

length input-file.csv

map

Creates an output dataset by calling the transformer function on every record of the input dataset. Works just like array.map in JavaScript.

Syntax

map <transformer-fn> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • transformer-fn - A JavaScript function to transform each record of the input dataset. Specifying a file name will load the JavaScript code from the file.
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, applies the transformation and writes to standard output

command-that-produces-json | map "record => record.x"

Reads data from a file, applies the transformation and writes to standard output

map "record => record.x" input-file.csv

Reads data from a file, applies the transformation and writes output to another file

map "record => record.x"  input-file.csv output-file.csv

Reads JSON data from standard input, applies the transformation and writes output to another file

command-that-produces-json | map "record => record.x" - output-file.csv

Loads a JavaScript file for the transformation

map --file my-transformation.js input-file.csv output-file.csv

omit

Creates a new dataset by omitting the specified fields from the input dataset.

Syntax

omit <fields> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • fields - Comma-separated list of field names to omit from the input data
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Notes

  • The input data can be an array of objects or a single object.

See also

Examples

Reads JSON data from standard input, omits some fields and writes to standard output

command-that-produces-json | omit ColumnA,ColumnB,ColumnC

Reads data from a file, omits some fields and writes to standard output

omit ColumnA,ColumnB,ColumnC input-file.csv

Reads data from a file, omits some fields and writes output to another file

omit ColumnA,ColumnB,ColumnC input-file.csv output-file.csv

Reads JSON data from standard input, omits some fields and writes output to another file

command-that-produces-json | omit ColumnA,ColumnB,ColumnC - output-file.csv

orderBy

Sorts the input dataset by the requested criteria and outputs the sorted dataset. Works a bit like array.sort in JavaScript, but really it's way more advanced.

Syntax

orderBy (<sort-fn> [<sort-direction>])+ [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • sort-fn - A JavaScript function to select the sort key from each record of the input dataset. Specifying a file name will load the JavaScript code from the file.
  • sort-direction - Optional sort direction that may be "ascending" or "descending". Defaults to "ascending".
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Notes

  • The sort function and sort direction can be stacked up to create nested levels of sorting.

Examples

Reads JSON data from standard input, sorts by email and writes to standard output

command-that-produces-json | orderBy "record => record.email"

Reads data from a file, sorts by email and writes to standard output

orderBy "record => record.email" input-file.csv

Reads data from a file, sorts by email and writes output to another file

orderBy "record => record.email" input-file.csv output-file.csv

Loads the sort function from a JavaScript file

orderBy --file my-sort-fn.js input-file.csv

Reads JSON data from standard input, sorts by name and then by age (a nested sort) and writes to standard output

orderBy "r => r.email" "r => r.age" - output-file.csv

Reads JSON data from standard input, sorts by age (oldest to youngest) and writes to standard output

orderBy "r => r.age" descending - output-file.csv

pick

Creates a new dataset by picking the specified fields from the input dataset.

Syntax

pick <fields> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • fields - Comma-separated list of field names to pick from the input data
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Notes

  • The input data can be an array of objects or a single object.

See also

Examples

Reads JSON data from standard input, picks some fields and writes to standard output

command-that-produces-json | pick ColumnA,ColumnB,ColumnC

Reads data from a file, picks some fields and writes to standard output

pick ColumnA,ColumnB,ColumnC input-file.csv

Reads data from a file, picks some fields and writes output to another file

pick ColumnA,ColumnB,ColumnC input-file.csv output-file.csv

Reads JSON data from standard input, picks some fields and writes output to another file

command-that-produces-json | pick ColumnA,ColumnB,ColumnC - output-file.csv

reduce

Reduces or aggregates an input dataset to some output value by repeatedly calling the reducer function on every record of the input. Works just like array.reduce in JavaScript.

Syntax

reduce <reducer-fn> <seed-value> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • reducer-fn - A JavaScript "reducer" function called for each record of the input dataset. Specifying a file name will load the JavaScript code from the file.
  • seed-value - JSON value that is used as the initial accumulator value for the reduction.
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, applies the reduction to compute total sales and writes to standard output

command-that-produces-json | reduce "(a, r) => a + r.sales" 0

Reads data from a file, applies the reduction and writes to standard output

reduce "(a, r) => a + r.sales" 0 input-file.json

Reads data from a file, applies the reduction and writes output to another file

reduce "(a, r) => a + r.sales" 0 input-file.csv output-file.csv

Reads JSON data from standard input, applies the reduction and writes output to another file

command-that-produces-json | reduce "(a, r) => a + r.sales" 0 - output-file.csv

Loads a JavaScript file for the transformation

reduce input-file.yaml my-reducer.js 0

run

Execute a command for each record in the input dataset.

Syntax

run <cmd-selector-fn> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • cmd-selector-fn - A JavaScript function to transform each record of the input dataset that creates a command to execute.
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, creates a command and executes the command for each record.

command-that-produces-json | run "record => `echo ${record.x}`"

Reads data from a CSV or JSON file, creates a command and executes the command for each record.

run "record => `echo ${record.x}`" input-file.csv

Reads data from a file, creates a command and executes the command for each record, writing the output to another file.

run "record => `echo ${record.x}`"  input-file.csv output-file.csv

Reads JSON data from standard input, creates a command and executes the command for each record, writing the output to a file.

command-that-produces-json | run "record => `echo ${record.x}`" - output-file.csv

Loads a JavaScript file that provides the function that creates the command.

run --file my-transformation.js input-file.csv output-file.csv

skip

Skips the first X records of the input dataset and writes the remaining records to the output dataset.

Syntax

skip <skip-number> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • skip-number - The number of records to skip.
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, skips 3 records and writes remaining records to standard output

command-that-produces-json | skip 3

Reads data from a file, skips 3 records and writes remaining records to standard output

skip 3 input-file.csv

Reads data from a file, skips 3 records and writes remaining records to another file

skip 3  input-file.csv output-file.csv

Reads JSON data from standard input, skips 3 records and writes remaining records to a file

command-that-produces-json | skip 3 - output-file.csv

take

Takes the first X records of the input dataset and writes them to the output dataset.

Syntax

take <take-number> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • take-number - The number of records to take.
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, takes 3 records and writes them to standard output

command-that-produces-json | take 3

Reads data from a file, takes 3 records and writes them to standard output

take 3 input-file.csv

Reads data from a file, takes 3 records and writes them to another file

take 3 input-file.csv output-file.csv

Reads JSON data from standard input, takes 3 records and writes them to a file

command-that-produces-json | take 3 - output-file.csv

to-csv

Converts data from the JSON data format to the CSV data format.

Syntax

to-csv [options] [<input-file>] [<csv-output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • CSV file
  • CSV formatted data on standard output

Arguments

  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • csv-output-file - The name of a file (must be a CSV file) to output the resulting dataset to. Omitting this causes CSV data to be written to standard output.

Options

  • --columns, --c=<column-names>
    • Sets the columns (and their order) that will be included in the output CSV
    • Example: --columns=ColumnA,ColumnB,ColumnC

Examples

Reads JSON data from standard input and writes CSV data to standard output

command-that-produces-json | to-csv -

Reads JSON data from standard input and writes a CSV data file

command-that-produces-json | to-csv output-file.csv

Reads a JSON data file and writes a CSV data file

to-csv input-file.json output-file.csv

Reads a YAML data file and writes a CSV data file

to-csv input-file.yaml output-file.csv

Writes a CSV file setting the column names

to-csv input-file.json output-file.csv --columns ColumnA,ColumnB,ColumnC

to-object

Creates a JSON object from key/value pairs extracted from the input dataset.

Syntax

to-object <key-selector-fn> <value-selector-fn> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • key-selector-fn - A JavaScript function to select the key from each record of the input dataset. Specifying a file name will load the JavaScript code from the file.
  • value-selector-fn - A JavaScript function to select the value from each record of the input dataset. Specifying a file name will load the JavaScript code from the file.
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Notes

  • Unlike many other datakit commands the to-object command cannot output to the CSV format.

Examples

Reads JSON data from standard input and writes the JSON object to standard output

command-that-produces-json | to-object "r => r.key" "r => r.value"

Reads data from a file and writes the JSON object to standard output

to-object "r => r.key" "r => r.value"  input-file.csv

Reads data from a file and writes the JSON object to a file

to-object "r => r.key" "r => r.value" input-file.csv output-file.json

Loads JavaScript files for the key and value selector functions

to-object --file my-key-selector.js --file my-value-selector.js input-file.csv

Reads JSON data from standard inputand writes the JSON object to a file

to-object "r => r.key" "r => r.value" - output-file.json

to-yaml

Converts data from the JSON data format to the YAML data format.

Syntax

to-yaml [<input-file>] [<yaml-output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • YAML file
  • YAML formatted data on standard output

Arguments

  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • yaml-output-file - The name of a file (must be a YAML file) to output the resulting dataset to. Omitting this causes YAML data to be written to standard output.

Examples

Reads JSON data from standard input and writes YAML data to standard output

command-that-produces-json | to-yaml

Reads JSON data from standard input and writes a YAML data file

command-that-produces-json | to-yaml - output-file.yaml

Reads a JSON data file and writes a YAML data file

to-yaml input-file.json output-file.yaml

Reads a CSV data file and writes a YAML data file

to-yaml input-file.csv output-file.yaml

transform

Transforms an entire dataset through a user defined function.

Syntax

transform <transformer-fn> [<input-file>] [<output-file>]

Inputs

Input can be 1 of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted array on standard input.

Outputs

Output can be one of the following:

  • JSON file
  • CSV file
  • YAML file
  • JSON formatted data on standard output.

Arguments

  • transformer-fn - A JavaScript function to transform the input dataset. Specifying a file name will load the JavaScript code from the file.
  • input-file - Can be an input file name (json, csv or yaml) or a hypen to indicate reading JSON data from standard input. Can be omitted if there are no further arguments.
  • output-file - The name of a file (json, csv or yaml) to output the resulting dataset to. Omitting this causes JSON output to be written to standard output.

Examples

Reads JSON data from standard input, applies the transformation and writes to standard output

command-that-produces-json | transform "dataset => transform(dataset)"

Reads data from a file, applies the transformation and writes to standard output

transform "dataset => transform(dataset)" input-file.csv

Reads data from a file, applies the transformation and writes output to another file

transform "dataset => transform(dataset)" input-file.csv output-file.csv

Reads JSON data from standard input, applies the transformation and writes output to a file

command-that-produces-json | transform "dataset => transform(dataset)" - output-file.csv

Loads a JavaScript file for the transformation

transform --file my-transformation.js input-file.csv