This is a little command line tool meant to eat newline-delimited JSON (CouchDB documents) on stdin and POSTing them to the _bulk_docs endpoint of a CouchDB server.
This is a fork from couchdb-bulk, with the following modifications:
- posts 1000 docs per bulk request (instead of only 1) when the input is newline-delimited JSON
- drops supports for anything else than newline-delimited JSON (1 doc per line), assuming that getting to that data format is the job of another tool, such as jq
- drops supports for the module interface, only the CLI mode remains
- modernized code and dependencies
npm install -g couchdb-bulk2couchdb-bulk2 url [file]The [file] argument is optional, if its missing (or if its '-'), input is expected to be piped via stdin
Example:
cat ./test/fixtures/docs.ndjson | couchdb-bulk2 http://localhost:5984/testdb
// OR
couchdb-bulk2 http://localhost:5984/testdb ./test/fixtures/docs.ndjsoncouchdb-bulk2 expects the input to be newline-delimited JSON.
See http://jsonlines.org for more info on this format.
Each line should be a single doc:
{ "_id": "one" }
{ "_id": "two" }
{ "_id": "three" }This newline-delimited JSON format can easily be obtained from a JSON document containing an array of docs using a tool such as jq
cat view_reponse.json | jq -c '.docs[]' | couchdb-bulk2 http://localhost:5984/testdb-l, --batch-length <number>: set the number of documents to be sent in bulk to CouchDB per batch (default: 1000)-s, --sleep <milliseconds>: defines the amount of time (in milliseconds) to wait once a batch was sent before sending a new one (default: 0)
couchdb-bulk2works great in combinaison withndjson-apply: see data transformation workflow in the Inventaire project